CODESP Structured Oral Examination Design 5 — Questions and Answers
Question 1: A jurisdiction wants to convert its structured oral examination scores to a rank-ordered list. Which psychometric consideration is most important?
- Ensuring the exam has at least 50 questions to support fine-grained ranking
- Verifying that score differences are statistically meaningful before treating ranks as distinct (Correct answer)
- Ranking candidates solely on their highest single-question score
- Averaging panelist scores before converting to percentile ranks
Correct answer: Verifying that score differences are statistically meaningful before treating ranks as distinct
Rank-ordering requires that score differences exceed measurement error; candidates with statistically indistinguishable scores should be treated as tied.
Question 2: In a structured oral examination, 'situational questions' differ from 'experience-based questions' primarily in that situational questions:
- Are scored on a pass/fail basis rather than a multi-point scale
- Ask candidates what they would do in a hypothetical future scenario (Correct answer)
- Require candidates to describe a specific past behavior they demonstrated
- Are reserved for senior-level positions only
Correct answer: Ask candidates what they would do in a hypothetical future scenario
Situational (or hypothetical) questions present a future scenario and ask intended behavior, while experience-based (behavioral) questions ask for past actual behavior.
Question 3: Which documentation practice is most essential for defending a structured oral examination against a legal challenge?
- Retaining only the final averaged scores for each candidate
- Keeping records of the job analysis, question development rationale, rater training, and individual scores (Correct answer)
- Publishing all examination questions publicly after the exam is complete
- Storing only the passing candidates' materials for five years
Correct answer: Keeping records of the job analysis, question development rationale, rater training, and individual scores
Comprehensive documentation of the entire development and administration process provides the evidentiary trail needed to demonstrate job-relatedness and fairness.
Question 4: When a structured oral examination panel includes a subject matter expert (SME), what is their primary role during scoring?
- To advocate for candidates they personally observed performing well on the job
- To evaluate response quality against job-relevant technical and competency benchmarks (Correct answer)
- To replace the HR analyst as the official score recorder
- To negotiate a consensus score on behalf of the entire panel
Correct answer: To evaluate response quality against job-relevant technical and competency benchmarks
SMEs provide domain expertise to judge whether candidate responses reflect the knowledge, skills, and abilities actually required on the job.
Question 5: A CODESP oral examination yields an inter-rater reliability coefficient of 0.55. What action is most appropriate?
- Accept the results because any positive correlation is sufficient for defensibility
- Average the scores and proceed since raters rarely agree perfectly
- Investigate rater training gaps and scoring anchor clarity before using the exam operationally (Correct answer)
- Replace all panelists with AI scoring tools to eliminate human variance
Correct answer: Investigate rater training gaps and scoring anchor clarity before using the exam operationally
A reliability coefficient of 0.55 is below the generally accepted threshold of approximately 0.70, signaling that rater calibration and anchor definition need improvement.
Question 6: Which practice best ensures that structured oral examination questions remain legally defensible over time?
- Using the same question bank indefinitely to maintain historical comparability
- Periodically re-linking questions to updated job analyses as job duties evolve (Correct answer)
- Classifying all questions as confidential and never reviewing them externally
- Replacing questions only when a candidate formally challenges them
Correct answer: Periodically re-linking questions to updated job analyses as job duties evolve
Jobs change over time, so questions must be periodically validated against current job analysis data to maintain content validity and legal defensibility.
Question 7: In a CODESP structured oral examination, the 'board administrator' role is distinct from 'rater' because the board administrator primarily:
- Assigns final scores based on a holistic impression of each candidate
- Manages procedural consistency—timing, question reading, and form distribution—without scoring (Correct answer)
- Determines the passing score after reviewing all candidate responses
- Conducts post-exam interviews with candidates to explain their scores
Correct answer: Manages procedural consistency—timing, question reading, and form distribution—without scoring
The board administrator is responsible for standardizing exam procedures and logistics, keeping their role separate from the evaluative scoring function.
CODESP Structured Oral Examination Design 5
A jurisdiction wants to convert its structured oral examination scores to a rank-ordered list.
Which psychometric consideration is most important?