Apache Spark MLlib and Machine Learning

Question 1

What is the primary high-level API for machine learning in Apache Spark?

Accepted Answer

spark.ml (DataFrame-based API)

Answer

spark.ml is the primary, DataFrame-based API for machine learning in Spark, while spark.mllib is the older RDD-based library.

Question 2

What is a Pipeline in Spark MLlib?

Accepted Answer

A sequence of stages (Transformers and Estimators) that form an ML workflow

Answer

A Pipeline chains multiple Transformers and Estimators into a single ML workflow, simplifying training and prediction.

Question 3

What is the difference between a Transformer and an Estimator in Spark ML?

Accepted Answer

A Transformer applies a transformation to a DataFrame; an Estimator fits data to produce a Transformer

Answer

An Estimator has a fit() method that trains on data and returns a Transformer, which has a transform() method.

Question 4

Which Spark MLlib class is used to convert categorical string labels to numeric indices?

Accepted Answer

StringIndexer

Answer

StringIndexer encodes a string column of labels to a column of label indices, ordered by label frequencies.

Question 5

Which Spark MLlib algorithm is used for collaborative filtering-based recommendations?

Accepted Answer

ALS (Alternating Least Squares)

Answer

ALS (Alternating Least Squares) is Spark's collaborative filtering algorithm for building recommendation systems.

Apache Spark Practice Test