CODESP Structured Oral Examination Design 2 — Questions and Answers
Question 1: Before administering a structured oral examination, rater training should specifically include which activity?
- Reviewing scoring anchors and practicing rating sample responses (Correct answer)
- Reviewing the personal files of candidates to understand their histories
- Discussing which candidates the department prefers to hire
- Ranking candidates based on their application materials before interviews begin
Correct answer: Reviewing scoring anchors and practicing rating sample responses
Calibration practice with scoring anchors before the exam reduces inter-rater variance.
Question 2: Which of the following is a common rater error that structured oral examination design seeks to minimize?
- Halo effect, where a strong first impression inflates ratings across all dimensions (Correct answer)
- Primacy effect resulting from reading the job description first
- Statistical regression caused by large candidate pools
- Criterion contamination from using too many scoring dimensions
Correct answer: Halo effect, where a strong first impression inflates ratings across all dimensions
The halo effect causes raters to let one positive trait unduly raise scores on all other dimensions.
Question 3: After a structured oral examination, which statistical measure is most useful for evaluating how consistently raters scored candidates?
- Inter-rater reliability coefficient (Correct answer)
- Standard deviation of candidate ages
- Mean salary of the candidate pool
- Cronbach's alpha for the written supplemental application
Correct answer: Inter-rater reliability coefficient
Inter-rater reliability quantifies the degree of agreement among raters, indicating scoring consistency.
Question 4: In a structured oral exam for a public agency, the number of dimensions rated should ideally be determined by which factor?
- The number of critical KSAs identified in the job analysis (Correct answer)
- The number of candidates expected in the applicant pool
- The number of HR staff available to review results
- The time allotted for each interview session
Correct answer: The number of critical KSAs identified in the job analysis
Each rated dimension should correspond to a critical KSA so the exam measures what the job actually requires.
Question 5: What is the main risk of allowing raters to take extensive personal notes about candidate appearance during a structured oral examination?
- It can introduce bias based on non-job-related characteristics and create legal liability (Correct answer)
- It helps raters remember candidate answers more accurately after the day concludes
- It improves the final score's accuracy by capturing nuance
- It is required by most public agency open-meeting laws
Correct answer: It can introduce bias based on non-job-related characteristics and create legal liability
Notes about appearance or non-job factors can be evidence of discriminatory consideration in legal challenges.
Question 6: When is it appropriate to use a structured oral examination as the sole selection tool for a public agency position?
- When the job's critical KSAs are primarily interpersonal, communication-based, and best demonstrated orally (Correct answer)
- Whenever it is faster and cheaper than written exams
- Only when fewer than five candidates apply for the position
- When the hiring manager prefers a personal interview over written tests
Correct answer: When the job's critical KSAs are primarily interpersonal, communication-based, and best demonstrated orally
An oral exam is best suited as a sole tool when the essential job competencies are oral in nature and well-captured through structured questioning.
CODESP Structured Oral Examination Design 2
Before administering a structured oral examination, rater training should specifically include which activity?