AML Model Evaluation & Optimization Techniques 1

Question 1

What is the purpose of a validation set in model evaluation?

Accepted Answer

To tune hyperparameters and assess model performance during training

Answer

The purpose of a validation set in model evaluation is to tune hyperparameters and assess the model's performance during the training process, providing an unbiased estimate of its generalization ability. It helps in making decisions about model architecture and training parameters without contaminating the final test set. This prevents overfitting to the training data and ensures the model performs well on unseen examples.

Question 2

Which metric is commonly used for binary classification performance?

Accepted Answer

Accuracy

Answer

Accuracy is a commonly used metric for evaluating binary classification performance, representing the proportion of correctly classified instances (both true positives and true negatives) out of the total number of instances. While other metrics like precision, recall, and F1-score offer more nuanced insights, accuracy provides a straightforward measure of overall correctness. Mean Absolute Error and Root Mean Squared Error are used for regression tasks.

Question 3

What does cross-validation help to prevent?

Accepted Answer

Overfitting

Answer

Cross-validation is a robust technique used to assess how well a model generalizes to an independent dataset and, crucially, to prevent overfitting. By dividing the data into multiple folds and training/testing the model on different subsets, it provides a more reliable estimate of model performance than a single train-test split. This helps ensure the model learns generalizable patterns rather than memorizing the training data.

Question 4

What is the ROC curve used for?

Accepted Answer

To analyze classification model performance

Answer

The ROC (Receiver Operating Characteristic) curve is a graphical plot used to analyze the performance of a classification model, particularly for binary classification. It illustrates the trade-off between the True Positive Rate (sensitivity) and the False Positive Rate (1-specificity) at various classification thresholds. The Area Under the Curve (AUC-ROC) is a common metric derived from it, indicating the model's overall ability to distinguish between classes.

Question 5

Which metric evaluates regression model performance?

Accepted Answer

Mean Squared Error

Answer

Mean Squared Error (MSE) is a widely used metric for evaluating the performance of regression models. It calculates the average of the squared differences between the predicted values and the actual observed values. MSE provides a measure of the average magnitude of the errors, with larger errors being penalized more heavily due to the squaring, making it sensitive to outliers.

(AML) Advanced Machine Learning Certification Practice Test

(AML) Advanced Machine Learning Certification Practice Test

AML Model Evaluation & Optimization Techniques 1