Structured Oral Examination Design Flashcards

In a structured oral examination, what is the primary purpose of using an 'anchored rating scale'?

Answer: To provide behaviorally defined benchmark responses at each score level for consistent evaluation

Anchored rating scales define specific behavioral indicators at each score point, reducing subjectivity and improving inter-rater reliability.

Which approach best reduces the 'similar-to-me' bias in structured oral examination panels?

Answer: Training raters on common biases and requiring independent scoring

Rater training on cognitive biases combined with independent scoring before any discussion is the most effective mitigation strategy.

A CODESP-developed structured oral examination uses 'probing questions.' What is their appropriate use?

Answer: To clarify ambiguous responses or elicit additional relevant information within the same competency

Probing questions are pre-approved follow-ups used to draw out more complete information within the same competency area, not to introduce new topics.

When establishing content validity for a structured oral examination, which step is most critical?

Answer: Linking each examination question directly to KSAOs identified in a job analysis

Content validity depends on a demonstrable linkage between exam questions and job-relevant KSAOs identified through systematic job analysis.

In CODESP structured oral exam administration, why are panelists typically prohibited from sharing scores until all panelists have scored independently?

Answer: To prevent anchoring bias where early scores influence subsequent raters

Sequential scoring discussions create anchoring effects; independent scoring preserves the statistical independence needed for reliable inter-rater agreement.

Which factor most directly threatens the reliability of a structured oral examination administered across multiple sites on different days?

Answer: Inconsistent question reading, timing, or probing practices across different panels

Procedural inconsistencies in how questions are presented and probed across panels directly undermine standardization and score comparability.

Under EEO guidelines relevant to CODESP exams, adverse impact in a structured oral examination is most appropriately evaluated using:

Answer: The 80% (four-fifths) rule applied to pass rates of protected groups versus the highest-scoring group

The UGESP four-fifths rule compares selection rates: if a protected group's rate is less than 80% of the top group's rate, adverse impact is indicated.

A jurisdiction wants to convert its structured oral examination scores to a rank-ordered list. Which psychometric consideration is most important?

Answer: Verifying that score differences are statistically meaningful before treating ranks as distinct

Rank-ordering requires that score differences exceed measurement error; candidates with statistically indistinguishable scores should be treated as tied.

In a structured oral examination, 'situational questions' differ from 'experience-based questions' primarily in that situational questions:

Answer: Ask candidates what they would do in a hypothetical future scenario

Situational (or hypothetical) questions present a future scenario and ask intended behavior, while experience-based (behavioral) questions ask for past actual behavior.

Which documentation practice is most essential for defending a structured oral examination against a legal challenge?

Answer: Keeping records of the job analysis, question development rationale, rater training, and individual scores

Comprehensive documentation of the entire development and administration process provides the evidentiary trail needed to demonstrate job-relatedness and fairness.

When a structured oral examination panel includes a subject matter expert (SME), what is their primary role during scoring?

Answer: To evaluate response quality against job-relevant technical and competency benchmarks

SMEs provide domain expertise to judge whether candidate responses reflect the knowledge, skills, and abilities actually required on the job.

A CODESP oral examination yields an inter-rater reliability coefficient of 0.55. What action is most appropriate?

Answer: Investigate rater training gaps and scoring anchor clarity before using the exam operationally

A reliability coefficient of 0.55 is below the generally accepted threshold of approximately 0.70, signaling that rater calibration and anchor definition need improvement.

Which practice best ensures that structured oral examination questions remain legally defensible over time?

Answer: Periodically re-linking questions to updated job analyses as job duties evolve

Jobs change over time, so questions must be periodically validated against current job analysis data to maintain content validity and legal defensibility.

In a CODESP structured oral examination, the 'board administrator' role is distinct from 'rater' because the board administrator primarily:

Answer: Manages procedural consistency—timing, question reading, and form distribution—without scoring

The board administrator is responsible for standardizing exam procedures and logistics, keeping their role separate from the evaluative scoring function.