Apache Spark MLlib and Machine Learning 2

Question 1

Which class in Spark ML is used for hyperparameter tuning via cross-validation?

Accepted Answer

CrossValidator

Answer

CrossValidator performs k-fold cross-validation across a parameter grid to select the best model hyperparameters.

Question 2

What does the BinaryClassificationEvaluator use by default to evaluate model performance in Spark ML?

Accepted Answer

Area Under ROC Curve (AUC-ROC)

Answer

BinaryClassificationEvaluator defaults to Area Under ROC Curve (areaUnderROC) as its evaluation metric.

Question 3

Which Spark MLlib algorithm performs dimensionality reduction?

Accepted Answer

PCA (Principal Component Analysis)

Answer

PCA (Principal Component Analysis) reduces the dimensionality of feature vectors by projecting them onto principal components.

Question 4

How do you save a trained Spark ML model to disk?

Accepted Answer

model.write().save('path')

Answer

Spark ML models are saved using model.write().save('path'), which persists the model in Parquet format.

Question 5

What is the purpose of StandardScaler in Spark MLlib?

Accepted Answer

Normalizes feature vectors to have zero mean and/or unit standard deviation

Answer

StandardScaler standardizes features by removing the mean and scaling to unit variance, improving convergence for gradient-based algorithms.

Apache Spark Practice Test