MTS Assessment & Evaluation Design 3 — Questions and Answers
Question 1: Which statistical measure indicates how closely test scores cluster around the mean, reflecting precision of the assessment?
- Standard deviation (Correct answer)
- Median
- Mode
- Kurtosis
Correct answer: Standard deviation
Standard deviation measures the spread of scores around the mean; a smaller standard deviation indicates scores are more tightly clustered and the test is more precise.
Question 2: When constructing a multiple-choice item, which practice most directly reduces construct-irrelevant variance?
- Using longer answer options
- Ensuring all distractors are plausible but clearly incorrect (Correct answer)
- Writing stems at a reading level above the target audience
- Including at least five answer options
Correct answer: Ensuring all distractors are plausible but clearly incorrect
Plausible distractors ensure the item measures the target knowledge rather than test-taking savvy or reading comprehension of implausible options.
Question 3: In the Phillips ROI Methodology, Level 5 requires converting training benefits to monetary value and comparing them to program costs. What is the formula for ROI?
- (Benefits − Costs) / Benefits × 100
- (Costs / Benefits) × 100
- ((Benefits − Costs) / Costs) × 100 (Correct answer)
- (Benefits × Costs) / 100
Correct answer: ((Benefits − Costs) / Costs) × 100
ROI (%) = ((Net Benefits − Program Costs) / Program Costs) × 100, showing the return for every dollar invested.
Question 4: A formative evaluation conducted mid-course is primarily intended to:
- Certify learner competency at course end
- Identify instructional problems and make real-time improvements (Correct answer)
- Compare learner performance against national norms
- Measure long-term return on investment
Correct answer: Identify instructional problems and make real-time improvements
Formative evaluation occurs during development or delivery to identify weaknesses and guide continuous improvement before final deployment.
Question 5: Which data collection method is most appropriate when the evaluator needs to observe subtle interpersonal dynamics during a leadership simulation?
- Written post-test
- Structured behavioral observation checklist (Correct answer)
- End-of-course satisfaction survey
- Computer-adaptive testing
Correct answer: Structured behavioral observation checklist
A structured behavioral observation checklist allows trained raters to systematically capture specific behaviors as they occur in real time.
Question 6: Which term describes the degree to which an assessment consistently measures what it is intended to measure across administrations?
- Validity
- Reliability (Correct answer)
- Objectivity
- Fidelity
Correct answer: Reliability
Reliability refers to the consistency of measurement — the same instrument should produce similar results under similar conditions.
Question 7: A training coordinator notices that passing scores on a performance test vary widely depending on which rater scores the task. This is a problem of:
- Low content validity
- Inter-rater reliability (Correct answer)
- Test-retest reliability
- Construct underrepresentation
Correct answer: Inter-rater reliability
Inter-rater reliability concerns agreement between different scorers; high variability between raters undermines the consistency and fairness of the assessment.
MTS Assessment & Evaluation Design 3
Which statistical measure indicates how closely test scores cluster around the mean, reflecting precision of the assessment?