Time Series Analysis Forecasting Methods and Evaluation 1 — Questions and Answers
Question 1: What is the 'naive' forecasting method?
- Using the most recent observation as the forecast for all future periods (Correct answer)
- Using the historical average as the forecast
- Using linear regression over all data
- Using exponential smoothing with α=0.5
Correct answer: Using the most recent observation as the forecast for all future periods
The naive method sets every future forecast equal to the last observed value, serving as a simple baseline for comparison.
Question 2: What is Mean Absolute Percentage Error (MAPE)?
- The average of the absolute percentage differences between forecasts and actual values (Correct answer)
- The mean of squared errors divided by the mean
- The ratio of MAE to RMSE
- The percentage of forecasts within 5% of actuals
Correct answer: The average of the absolute percentage differences between forecasts and actual values
MAPE = mean(|actual - forecast| / |actual|) × 100%, providing a scale-independent accuracy measure expressed as a percentage.
Question 3: What is the key difference between in-sample (training) error and out-of-sample (test) error?
- In-sample error measures fit on data used for estimation; out-of-sample measures true predictive accuracy on unseen data (Correct answer)
- In-sample error is always larger
- They measure the same thing
- Out-of-sample error is computed before training
Correct answer: In-sample error measures fit on data used for estimation; out-of-sample measures true predictive accuracy on unseen data
In-sample error can be overly optimistic due to overfitting; only out-of-sample evaluation on a held-out test set truly measures forecast performance.
Question 4: What is 'walk-forward validation' (time series cross-validation)?
- Sequentially expanding the training window and evaluating forecasts on subsequent observations (Correct answer)
- Randomly shuffling time series observations for cross-validation
- Using k-fold cross-validation without regard to time order
- Splitting the data once into 80% train and 20% test
Correct answer: Sequentially expanding the training window and evaluating forecasts on subsequent observations
Walk-forward validation respects temporal order by training on data up to time t and forecasting time t+h, then advancing t and repeating.
Question 5: What does the Diebold-Mariano test compare?
- Whether two forecasting models have statistically equal predictive accuracy (Correct answer)
- Whether a series is stationary
- Whether ARIMA or ETS fits better based on AIC
- Whether residuals are normally distributed
Correct answer: Whether two forecasting models have statistically equal predictive accuracy
The Diebold-Mariano test formally tests if the difference in forecast accuracy between two models is statistically significant.
Question 6: What is the Mean Absolute Scaled Error (MASE)?
- MAE scaled by the in-sample MAE of the naive forecast, giving a scale-free accuracy metric (Correct answer)
- MAE divided by the mean of the series
- Absolute error divided by the standard deviation
- The ratio of RMSE to MAE
Correct answer: MAE scaled by the in-sample MAE of the naive forecast, giving a scale-free accuracy metric
MASE normalizes MAE against the naive in-sample error, producing a benchmark-relative metric that avoids MAPE's zero-division problem.
Time Series Analysis Forecasting Methods and Evaluation 1
What is the 'naive' forecasting method?