CODESP Structured Oral Examination Design 4 — Questions and Answers
Question 1: In a structured oral examination, what is the primary purpose of using an 'anchored rating scale'?
- To ensure all panelists independently assign scores without discussing responses
- To provide behaviorally defined benchmark responses at each score level for consistent evaluation (Correct answer)
- To anchor the exam to a fixed passing score determined before administration
- To prevent candidates from referencing notes or written materials during responses
Correct answer: To provide behaviorally defined benchmark responses at each score level for consistent evaluation
Anchored rating scales define specific behavioral indicators at each score point, reducing subjectivity and improving inter-rater reliability.
Question 2: Which approach best reduces the 'similar-to-me' bias in structured oral examination panels?
- Using a single expert rater instead of a panel
- Allowing panelists to discuss candidate responses before scoring
- Training raters on common biases and requiring independent scoring (Correct answer)
- Selecting panelists who share demographic backgrounds with candidates
Correct answer: Training raters on common biases and requiring independent scoring
Rater training on cognitive biases combined with independent scoring before any discussion is the most effective mitigation strategy.
Question 3: A CODESP-developed structured oral examination uses 'probing questions.' What is their appropriate use?
- To introduce entirely new competency dimensions not covered by the main question
- To clarify ambiguous responses or elicit additional relevant information within the same competency (Correct answer)
- To give lower-scoring candidates a second attempt with easier questions
- To replace the original question if a candidate appears anxious
Correct answer: To clarify ambiguous responses or elicit additional relevant information within the same competency
Probing questions are pre-approved follow-ups used to draw out more complete information within the same competency area, not to introduce new topics.
Question 4: When establishing content validity for a structured oral examination, which step is most critical?
- Administering a pilot exam to at least 500 candidates
- Linking each examination question directly to KSAOs identified in a job analysis (Correct answer)
- Ensuring all questions are drawn from published civil service textbooks
- Having the HR director personally approve all final questions
Correct answer: Linking each examination question directly to KSAOs identified in a job analysis
Content validity depends on a demonstrable linkage between exam questions and job-relevant KSAOs identified through systematic job analysis.
Question 5: In CODESP structured oral exam administration, why are panelists typically prohibited from sharing scores until all panelists have scored independently?
- To comply with open-meeting laws requiring secret ballots
- To prevent anchoring bias where early scores influence subsequent raters (Correct answer)
- To allow each panelist to submit scores to different supervisors
- To speed up the scoring process by avoiding group discussion
Correct answer: To prevent anchoring bias where early scores influence subsequent raters
Sequential scoring discussions create anchoring effects; independent scoring preserves the statistical independence needed for reliable inter-rater agreement.
Question 6: Which factor most directly threatens the reliability of a structured oral examination administered across multiple sites on different days?
- Variation in the physical layout of examination rooms
- Inconsistent question reading, timing, or probing practices across different panels (Correct answer)
- Differences in candidate travel time to each site
- Using the same scoring form at all locations
Correct answer: Inconsistent question reading, timing, or probing practices across different panels
Procedural inconsistencies in how questions are presented and probed across panels directly undermine standardization and score comparability.
Question 7: Under EEO guidelines relevant to CODESP exams, adverse impact in a structured oral examination is most appropriately evaluated using:
- The 80% (four-fifths) rule applied to pass rates of protected groups versus the highest-scoring group (Correct answer)
- The total number of minority candidates who fail regardless of group proportions
- A visual inspection of score distributions without statistical testing
- The percentage of panelists who belong to protected classes
Correct answer: The 80% (four-fifths) rule applied to pass rates of protected groups versus the highest-scoring group
The UGESP four-fifths rule compares selection rates: if a protected group's rate is less than 80% of the top group's rate, adverse impact is indicated.
CODESP Structured Oral Examination Design 4
In a structured oral examination, what is the primary purpose of using an 'anchored rating scale'?