PrepL Evaluation and Simulations 3 — Questions and Answers
Question 1: Which statistical measure is used to determine the reliability of scores across multiple raters in a PrepL simulation evaluation?
- Pearson correlation
- Inter-rater reliability (IRR) (Correct answer)
- Standard deviation
- Alpha coefficient
Correct answer: Inter-rater reliability (IRR)
Inter-rater reliability measures the degree of agreement between different evaluators scoring the same performance.
Question 2: A PrepL simulation candidate scores at the cut score. How should this result typically be interpreted?
- The candidate has failed and must retake the exam
- The candidate has marginally passed at the minimum competency threshold (Correct answer)
- The candidate is in the top tier of performers
- The candidate's score is invalid and must be reviewed
Correct answer: The candidate has marginally passed at the minimum competency threshold
A cut score represents the minimum level of competency required to pass; scoring at this point means meeting, not exceeding, the standard.
Question 3: During an OSCE (Objective Structured Clinical Examination) simulation, standardized patients are used primarily to:
- Reduce exam cost
- Provide consistent, scripted clinical scenarios for fair evaluation (Correct answer)
- Test candidates' medical knowledge only
- Replace written examination components
Correct answer: Provide consistent, scripted clinical scenarios for fair evaluation
Standardized patients deliver scripted, reproducible scenarios ensuring every candidate is assessed under equivalent conditions.
Question 4: A licensing simulation evaluator uses a behaviorally anchored rating scale (BARS). What is the main advantage of BARS?
- It is faster to score than other methods
- It links specific observable behaviors to rating levels for greater objectivity (Correct answer)
- It requires only one rater
- It eliminates the need for standardized patients
Correct answer: It links specific observable behaviors to rating levels for greater objectivity
BARS reduces subjectivity by tying each rating level to concrete, observable examples of performance.
Question 5: In evaluation terminology, 'norm-referenced' scoring means a candidate's performance is compared to:
- A fixed set of content standards
- The performance of other candidates in a reference group (Correct answer)
- The candidate's own prior scores
- Expert clinician benchmarks
Correct answer: The performance of other candidates in a reference group
Norm-referenced scoring ranks individuals relative to a defined comparison group rather than against absolute content standards.
Question 6: Which scenario best illustrates a 'high-stakes' simulation in the PrepL context?
- A practice quiz with unlimited retakes
- A scored simulation whose results determine licensure eligibility (Correct answer)
- An unscored role-play for skill development
- A peer observation exercise with no rubric
Correct answer: A scored simulation whose results determine licensure eligibility
High-stakes simulations carry significant consequences, such as determining whether a candidate receives their license.
Question 7: A simulation is described as 'criterion-referenced.' This means candidates are evaluated against:
- Each other's performance
- A predetermined standard of competency (Correct answer)
- National average scores
- The highest-performing candidate in the cohort
Correct answer: A predetermined standard of competency
Criterion-referenced evaluations measure performance against a fixed standard, not relative to other test-takers.
PrepL Evaluation and Simulations 3
Which statistical measure is used to determine the reliability of scores across multiple raters in a PrepL simulation evaluation?