Psychometric Properties: Reliability, Validity, and Norms

1. Architectural Blueprint & Classical Test Theory (CTT)

Psychological measurement rests on the foundation of Classical Test Theory (CTT), pioneered by Charles Spearman (1904) and formalized by Gulliksen (1950) and Lord & Novick (1968). CTT provides the mathematical framework for decomposing observed test scores into underlying true capability and measurement noise.

The Axioms and Assumptions of Classical Test Theory

1. Axiom of Additivity: Any observed raw score ($X$) is the linear sum of an unobservable True Score ($T$) and an unsystematic Error Score ($E$): $X = T + E$
2. Expected Error is Zero: Over an infinitely large number of repeated administrations under identical conditions, random errors cancel out: $E(E) = 0 \quad \text{or} \quad \mu_E = 0$
3. Independence of True Score and Error: The magnitude of error is completely uncorrelated with the individual's true ability level: $r_{TE} = 0 \quad \text{or} \quad \text{Cov}(T, E) = 0$
4. Independence of Errors Across Occasions: Random error on Test 1 shares zero correlation with random error on Test 2: $r_{E_1 E_2} = 0 \quad \text{or} \quad \text{Cov}(E_1, E_2) = 0$
5. Independence of Error and Cross-Test True Score: Error on Test 1 is unrelated to the true score on Test 2: $r_{T_1 E_2} = 0$

Variance Decomposition & Psychometric Definition of Reliability

Because True scores and Error scores are uncorrelated ($\text{Cov}(T, E) = 0$), the total observed score variance ($\sigma_X^2$) equals the simple sum of True score variance ($\sigma_T^2$) and Error variance ($\sigma_E^2$):

$\sigma_X^2 = \sigma_T^2 + \sigma_E^2$

Psychometrically, Reliability ($r_{xx}$) is defined as the ratio of True score variance to Total observed score variance:

$r_{xx} = \frac{\sigma_T^2}{\sigma_X^2} = \frac{\sigma_T^2}{\sigma_T^2 + \sigma_E^2} = 1 - \frac{\sigma_E^2}{\sigma_X^2}$

• Theoretical Range: $0.00 \le r_{xx} \le 1.00$.
• If a test has zero error ($\sigma_E^2 = 0$), $r_{xx} = 1.00$ (perfect reliability).
• If a test score is pure noise ($\sigma_T^2 = 0$), $r_{xx} = 0.00$ (zero reliability).

The Standard Error of Measurement ($SEM$)

The Standard Error of Measurement ($SEM$) represents the standard deviation of an individual's observed scores around their true score over infinite testings. It quantifies the absolute margin of error in a test's score metric:

$SEM = \sigma_X \sqrt{1 - r_{xx}}$

Where:

• $\sigma_X$ = Standard deviation of the total test scores.
• $r_{xx}$ = Reliability coefficient of the test.

Calculating Confidence Intervals for True Scores:

To communicate score imprecision ethically, examiners construct confidence intervals around the observed score ($X$):

• $68\%$ Confidence Interval: $X \pm 1.00(SEM)$
• $95\%$ Confidence Interval: $X \pm 1.96(SEM)$
• $99\%$ Confidence Interval: $X \pm 2.58(SEM)$

2. Methods of Estimating Reliability

Different empirical designs tap different sources of measurement error:

Comparative Reliability Matrix

Key Reliability Equations

1. The Spearman-Brown Prophecy Formula

When a test is split into halves, the Pearson correlation between the two halves ($r_{hh}$) reflects the reliability of a test only half as long. Because test length directly impacts reliability, the Spearman-Brown formula estimates the reliability of the full-length test ($k = 2$):

$r_{\text{total}} = \frac{2 r_{hh}}{1 + r_{hh}}$

General Formula for any length factor $k$: $r_{kk} = \frac{k \cdot r_{11}}{1 + (k - 1)r_{11}}$

Where $k = \frac{\text{New Length}}{\text{Old Length}}$ (e.g., if a 20-item test is lengthened to 60 items, $k = 3$).

Formula to determine the factor $k$ required to attain a target reliability $r_{kk}$: $k = \frac{r_{kk}(1 - r_{11})}{r_{11}(1 - r_{kk})}$

2. Kuder-Richardson Formulas (Dichotomous Items)

• $KR\text{-}20$ (Unequal item difficulty): $KR\text{-}20 = \frac{N}{N - 1} \left(1 - \frac{\sum pq}{\sigma_X^2}\right)$ Where $N$ = number of items, $p$ = proportion passing item, $q = 1 - p$, and $\sigma_X^2$ = variance of total test scores.
• $KR\text{-}21$ (Equal item difficulty shortcut): $KR\text{-}21 = \frac{N}{N - 1} \left(1 - \frac{\bar{X}(N - \bar{X})}{N \sigma_X^2}\right)$ Where $\bar{X}$ = mean total test score. $KR\text{-}21$ always yields an estimate equal to or lower than $KR\text{-}20$.

3. Cronbach's Alpha ($\alpha$) for Non-Dichotomous Data

For rating scales, attitude surveys, and Likert-type questionnaires (e.g., 1 to 5 scale):