Apache Spark MLlib and Machine Learning 2 — Questions and Answers
Question 1: Which class in Spark ML is used for hyperparameter tuning via cross-validation?
- CrossValidator (Correct answer)
- ParamGridSearch
- HyperTuner
- ModelSelector
Correct answer: CrossValidator
CrossValidator performs k-fold cross-validation across a parameter grid to select the best model hyperparameters.
Question 2: What does the BinaryClassificationEvaluator use by default to evaluate model performance in Spark ML?
- Accuracy
- F1 Score
- Area Under ROC Curve (AUC-ROC) (Correct answer)
- Precision
Correct answer: Area Under ROC Curve (AUC-ROC)
BinaryClassificationEvaluator defaults to Area Under ROC Curve (areaUnderROC) as its evaluation metric.
Question 3: Which Spark MLlib algorithm performs dimensionality reduction?
- KMeans
- PCA (Principal Component Analysis) (Correct answer)
- Naive Bayes
- GBTClassifier
Correct answer: PCA (Principal Component Analysis)
PCA (Principal Component Analysis) reduces the dimensionality of feature vectors by projecting them onto principal components.
Question 4: How do you save a trained Spark ML model to disk?
- model.export('path')
- model.save('path')
- model.write().save('path') (Correct answer)
- pickle.dump(model, 'path')
Correct answer: model.write().save('path')
Spark ML models are saved using model.write().save('path'), which persists the model in Parquet format.
Question 5: What is the purpose of StandardScaler in Spark MLlib?
- Encodes categorical features as numeric values
- Normalizes feature vectors to have zero mean and/or unit standard deviation (Correct answer)
- Splits the dataset into training and test sets
- Converts sparse vectors to dense format
Correct answer: Normalizes feature vectors to have zero mean and/or unit standard deviation
StandardScaler standardizes features by removing the mean and scaling to unit variance, improving convergence for gradient-based algorithms.
Question 6: Which evaluation metric does MulticlassClassificationEvaluator use by default in Spark ML?
- AUC-ROC
- F1 Score (Correct answer)
- Accuracy
- Precision
Correct answer: F1 Score
MulticlassClassificationEvaluator defaults to the F1 score as the evaluation metric for multi-class classification.
Apache Spark MLlib and Machine Learning 2
Which class in Spark ML is used for hyperparameter tuning via cross-validation?