Psychological Testing I: Types of Tests, Item Writing and Item Analysis, and Standardization — Reliability, Validity and Norms, Worked
2. Test construction: item writing and item analysis
Construction proceeds from a plan (purpose, content, a blueprint or table of specifications crossing content with objectives), to item writing, a try-out on a sample like the target group, item analysis, selection of the best items, and finally standardization. Good items are clear, unambiguous, free of clues (such as "always", "never" or a longer correct option), independent of one another, and at a suitable reading level; multiple-choice distractors must be plausible. Personality items may use true/false, Likert-type or forced-choice formats (the Edwards Personal Preference Schedule uses forced choice to reduce social desirability).
| Index | Formula | Reading it |
|---|---|---|
| Item difficulty (p) | p = number answering correctly ÷ number attempting (or R_U + R_L ÷ n_U + n_L) | Ranges 0–1; a HIGHER p means an EASIER item. Items near 0.5 discriminate best; for a four-option MCQ the optimum is raised for guessing to about (1 + 0.25) ÷ 2 ≈ 0.63. |
| Discrimination (D) | D = (R_U − R_L) ÷ n, using the upper and lower 27 per cent of total scorers (Kelley) | Ranges −1 to +1. Ebel's guide: 0.40 and above very good; 0.30–0.39 reasonably good; 0.20–0.29 marginal, needs revision; below 0.20 poor. A negative D means low scorers do better — the key or item is faulty. |
| Correction for guessing | S = R − W ÷ (k − 1), with k options | 40 right and 15 wrong on four-option items: S = 40 − 15 ÷ 3 = 35. |
Worked item analysis. With 40 candidates in each of the upper and lower groups, 30 in the upper and 10 in the lower group answer an item correctly. Difficulty p = (30 + 10) ÷ 80 = 0.50; discrimination D = (30 − 10) ÷ 40 = 0.50 — an item of ideal difficulty that discriminates very well. Distractor analysis checks that each wrong option attracts some lower-group candidates and fewer upper-group ones. The item–total point-biserial correlation is an alternative discrimination index, and item response theory replaces these sample-dependent indices with the parameters of an item characteristic curve — difficulty (b), discrimination (a) and guessing (c); the Rasch model uses difficulty alone.
3. Reliability
Reliability is consistency of measurement. In classical test theory an observed score is true score plus error (X = T + E), and the reliability coefficient is the proportion of observed-score variance that is true-score variance. Each method estimates consistency against a different source of error.
| Method | Procedure | Error it reflects |
|---|---|---|
| Test–retest | Same test twice, correlate (coefficient of stability) | Time sampling; affected by memory and practice |
| Parallel (alternate) forms | Two equivalent forms, correlate (coefficient of equivalence) | Content sampling |
| Split-half | Correlate odd and even halves, then correct with Spearman–Brown: r_tt = 2r_hh ÷ (1 + r_hh) | Content sampling within one sitting |
| Kuder–Richardson (KR-20, KR-21) and Cronbach's alpha | Inter-item consistency; KR-20 for dichotomous items, alpha for any item format | Content heterogeneity |
| Inter-rater (scorer) | Agreement between scorers; Cohen's kappa corrects for chance agreement | Scorer differences |
Worked. A split-half correlation of 0.60 gives a full-test reliability of 2(0.60) ÷ 1.60 = 0.75 — halving the test had lowered the estimate, and Spearman–Brown restores it. The standard error of measurement, SEM = SD√(1 − r_tt), converts reliability into score units: with SD = 10 and r_tt = 0.91, SEM = 10 × √0.09 = 3, so a person scoring 60 has a true score between 57 and 63 about 68 per cent of the time and between about 54 and 66 (±1.96 SEM) 95 per cent of the time. Reliability rises with test length, heterogeneity of the group and item quality, and falls for speeded tests measured by split-half (which inflates it) — use test–retest or parallel forms for speed tests.
4. Validity and norms
| Type of validity | Question it answers | Evidence |
|---|---|---|
| Content validity | Do the items representatively sample the domain? | Expert judgement against a blueprint. (Face validity — looking valid to test-takers — is not validity in the technical sense.) |
| Criterion-related: concurrent | Does the test agree with a criterion measured at the same time? | Correlation with a current criterion (for example diagnosis) |
| Criterion-related: predictive | Does the test forecast a future criterion? | Correlation with later performance (the validity coefficient) |
| Construct validity | Does the test measure the theoretical trait? | Convergent (high correlation with measures of the same construct) and discriminant (low correlation with different constructs) evidence — Campbell and Fiske's multitrait–multimethod matrix (1959); factorial validity; group differences and experimental effects. |
A test cannot be more valid than it is reliable: the validity coefficient cannot exceed the square root of the reliability. Norms are the performance of a defined standardization sample, used to interpret an individual score. Standardization requires uniform administration and scoring and a large, representative sample. Kinds of norms: age and grade equivalents; percentile ranks (ordinal, unequal units); and standard scores — z (mean 0, SD 1), T (50, 10), stanines (mean 5, SD about 2, a nine-point scale whose bands hold 4, 7, 12, 17, 20, 17, 12, 7 and 4 per cent), stens (mean 5.5, SD 2, ten points) and the deviation IQ (mean 100, SD 15 on the Wechsler scales). Worked: z = +1 is a T of 60, a deviation IQ of 115 and a stanine of 7.
Key takeaways
- Tests are classified by administration, response mode, speed versus power, scoring, norm- versus criterion-referencing, maximum versus typical performance and culture loading.
- Item difficulty p = proportion correct (higher p = easier; about 0.5 is ideal); discrimination D = (R_U − R_L) ÷ n on the upper and lower 27 per cent, with 0.40 and above very good.
- Reliability is estimated by test–retest, parallel forms, split-half with Spearman–Brown (2r ÷ (1 + r)), KR-20 and Cronbach's alpha, and inter-rater agreement; SEM = SD√(1 − r).
- Validity is content, criterion-related (concurrent and predictive) and construct (convergent and discriminant, the multitrait–multimethod matrix); validity cannot exceed the square root of reliability.
- Norms convert raw scores through age, grade and percentile equivalents and standard scores: z, T (50, 10), stanine (5, 2), sten (5.5, 2) and deviation IQ (100, 15).
Practice questions (10)
Attempt each one before opening the answer. Every explanation names the tempting wrong option as well as the right one, because that is where marks are lost.
In the upper and lower groups of 40 candidates each, 30 in the upper group and 10 in the lower group answer an item correctly. What is the discrimination index D? Type a number.
Numerical answer — type the value.
Show answer
Answer: 0.5
D = (R_U − R_L) ÷ n = (30 − 10) ÷ 40 = 20 ÷ 40 = 0.5. By Ebel's guide, 0.40 and above is a very good discriminator.Of 200 candidates who attempt an item, 50 answer it correctly. What is the item's difficulty index p? Type a number.
Numerical answer — type the value.
Show answer
Answer: 0.25
p = number correct ÷ number attempting = 50 ÷ 200 = 0.25. Because only a quarter pass, this is a fairly difficult item — a higher p would mean an easier one.The correlation between the odd and even halves of a test is 0.60. Using the Spearman–Brown formula, what is the reliability of the full test? Type a number.
Numerical answer — type the value.
Show answer
Answer: 0.75
r_tt = 2r_hh ÷ (1 + r_hh) = (2 × 0.60) ÷ (1 + 0.60) = 1.20 ÷ 1.60 = 0.75. The correction is needed because each half is only half as long as the real test.A test has a standard deviation of 10 and a reliability of 0.91. What is its standard error of measurement? Type a number.
Numerical answer — type the value.
Show answer
Answer: 3
SEM = SD√(1 − r_tt) = 10 × √(1 − 0.91) = 10 × √0.09 = 10 × 0.3 = 3 score points.On a test of four-option multiple-choice items, a candidate gets 40 right and 15 wrong and omits the rest. What is the score corrected for guessing, S = R − W ÷ (k − 1)? Type a number.
Numerical answer — type the value.
Show answer
Answer: 35
S = 40 − 15 ÷ (4 − 1) = 40 − 5 = 35. The formula assumes that wrong answers come from blind guessing among the k options.A new anxiety scale correlates 0.70 with an established anxiety scale and 0.05 with a vocabulary test. These two findings provide evidence of:
Show answer
Answer: A — Convergent and discriminant validity, both aspects of construct validity
A high correlation with a measure of the same construct is convergent evidence and a near-zero correlation with an unrelated construct is discriminant evidence; together they support construct validity, as in Campbell and Fiske's multitrait–multimethod logic.Which method of estimating reliability is INAPPROPRIATE for a highly speeded test because it inflates the coefficient?
Show answer
Answer: B — Odd–even split-half
On a speed test almost every attempted item is correct, so odd and even halves are nearly identical up to the point reached and the split-half correlation is spuriously high. Test–retest, separately administered forms or separately timed halves avoid this.A candidate is exactly one standard deviation above the mean on a normally distributed test. Which of the following standard scores are correct for this candidate? Select all that apply.
Show answer
Answer: A — T-score of 60; B — Stanine of 7; C — Deviation IQ of 115
z = +1 gives T = 50 + 10 = 60, deviation IQ = 100 + 15 = 115 and stanine = 5 + 2 = 7. The percentile rank is about 84, not 99.Assertion (A): A test can be highly reliable without being valid. Reason (R): The validity coefficient of a test cannot exceed the square root of its reliability coefficient. Choose the correct option.
Show answer
Answer: B — Both A and R are true, but R is not the correct explanation of A
Both are true, but R does not explain A. R sets an upper limit on validity given reliability (unreliability caps validity); it says nothing about why consistent measurement may still measure the wrong thing, which is what A asserts.Interpreting a pupil's score by whether it reaches a fixed standard of mastery, rather than by comparison with other pupils, is:
Show answer
Answer: C — Criterion-referenced interpretation
A criterion-referenced test compares performance with a defined standard (for example 80 per cent of objectives mastered). Norm-referenced interpretation and percentile ranks compare with a group, and ipsative scores compare a person with themselves across traits.