CBT Psychometric Principles and Analysis 2 — Questions and Answers
Question 1: A test developer notices that item difficulty values (p-values) cluster around 0.90 for most items. What is the primary concern?
- The test lacks ceiling effect protection
- Items are too easy, reducing score variance and reliability (Correct answer)
- The sample size is too small to judge difficulty
- High p-values indicate strong discriminability
Correct answer: Items are too easy, reducing score variance and reliability
When most items have p-values near 0.90, nearly everyone answers correctly, compressing score variance and lowering reliability coefficients.
Question 2: Which of the following best describes a 'speeded test' in psychometric terms?
- A test where all examinees answer all items given unlimited time
- A test where time limits prevent most examinees from attempting all items (Correct answer)
- A test scored solely on response accuracy regardless of speed
- A test that measures only processing speed constructs
Correct answer: A test where time limits prevent most examinees from attempting all items
A speeded test is one in which the time limit is short enough that few or no examinees complete all items, so speed itself influences scores.
Question 3: In confirmatory factor analysis (CFA), a chi-square goodness-of-fit test yields p = 0.40. What does this suggest?
- The model fits poorly and should be rejected
- The model fits the data well since the result is non-significant (Correct answer)
- The sample size is likely too large
- Factor loadings are not statistically significant
Correct answer: The model fits the data well since the result is non-significant
A non-significant chi-square (p > .05) in CFA indicates the model-implied covariance matrix does not significantly differ from the observed matrix, suggesting acceptable fit.
Question 4: What is the purpose of an anchor item set in an equating study?
- To measure examinees' motivation levels
- To provide common reference items that link two different test forms (Correct answer)
- To increase the overall difficulty of the new form
- To replace items that show differential item functioning
Correct answer: To provide common reference items that link two different test forms
Anchor items appear on both the old and new test forms and serve as a statistical bridge to place scores from different forms on a common scale.
Question 5: A researcher reports Cronbach's alpha = 0.55 for a 5-item scale. Which action would most directly increase this reliability estimate?
- Collecting data from a more homogeneous sample
- Increasing the number of response options per item
- Adding more items that measure the same construct (Correct answer)
- Converting the scale from Likert to dichotomous format
Correct answer: Adding more items that measure the same construct
The Spearman-Brown prophecy formula shows that adding parallel items measuring the same construct increases the number of inter-item covariances and raises alpha.
Question 6: Under Item Response Theory, what does the 'c' parameter in the three-parameter logistic (3PL) model represent?
- Item difficulty on the theta scale
- Item discrimination slope
- Lower asymptote representing guessing probability (Correct answer)
- Upper asymptote representing maximum item information
Correct answer: Lower asymptote representing guessing probability
The c parameter, or pseudo-guessing parameter, sets the lower asymptote of the item characteristic curve, representing the probability of a correct response for very low-ability examinees.
Question 7: Which validity evidence type is most directly assessed by examining a test's factor structure?
- Criterion-related validity
- Content validity
- Internal structure validity (Correct answer)
- Consequential validity
Correct answer: Internal structure validity
According to AERA/APA/NCME Standards, evidence based on internal structure (e.g., factor analysis) evaluates whether item relationships align with the proposed construct.
CBT Psychometric Principles and Analysis 2
A test developer notices that item difficulty values (p-values) cluster around 0.90 for most items.
What is the primary concern?