Reliability - A Class for MA Students

Main ideas / lessons (Reliability in language testing)

• Reliability vs. validity

  • Validity: whether test scores mean what we think they mean for the intended inference/decision context.
  • Reliability: whether the test actually measures consistently—i.e., the psychometric consistency of scores (a component of validity, but not identical).

• Core definition of reliability

  • Reliability is dependability/consistency of scores if you measure repeatedly under similar conditions.
  • Key point: reliability does not guarantee validity.
    • A test can be reliable (consistent) even if it measures the “wrong” thing for the decision context.

---

Reliability concepts covered

1) Classical reliability framing

• Test score decomposition:
  • Observed score (X) = True score + Error
• Error can come from factors like:
  • different test forms
  • different raters

---

2) Reliability as internal consistency (Cronbach’s Alpha)

When it’s used

• Items are closed-response (e.g., right/wrong).
• Items are intended to measure the same construct.

Cronbach’s Alpha (as used in language testing / applied linguistics)

• Question it addresses:
  • Do items designed to measure the same construct tend to be answered consistently?
• Requirements/assumptions (as presented):
  • Typically uses 0/1 scoring (correct/incorrect).
• Tooling:
  • Alpha can be calculated with software (e.g., spreadsheet or SPSS).
  • The instructor emphasizes Alpha is usually not computed “by hand” because it’s tedious.

Interpreting Alpha values (rule-of-thumb ranges)

• > 0.9: excellent (very high internal consistency)
• 0.8–0.9: good
• 0.7–0.8: acceptable/common in language testing releases
  • The instructor is not personally convinced that 70–80% is truly sufficient, since it implies substantial error.
• ~0.6 or lower (e.g., ~0.59): poor / needs improvement
  • Even long tests can inflate Alpha, so item quality still matters.

How to improve Alpha (two main levers)

• Improve quality of items.
• Increase the number of items (test length) to collect more information.
  • However, poor items can keep Alpha low even when length increases.

Important limitation

• Alpha should only be applied when items measure the same construct.
  • If you mix items measuring different constructs (e.g., different reading skills), internal consistency drops and Alpha can fall.

---

3) Reliability as agreement (correlation-based approaches)

Used when reliability is assessed as agreement/consistency across:

• administrations/times, or
• raters (for performance assessments: speaking/writing).

Agreement between administrations (two times)

• If learners’ scores correlate strongly across time, error from administration conditions is reduced.
• Correlation interpretation
  • r ranges from -1 to +1
  • r = 0: no relationship
  • r = 1: perfect direct relationship (scores increase together)
• Strength is often interpreted via r² (squared correlation):
  • r² = proportion of shared variance
  • remaining variance is treated as disagreement/error

Agreement between raters (inter-rater reliability)

• Example: two raters scored 20 scripts on a 1–6 scale.
• Reported:
  • Pearson r ≈ 0.83
  • r² ≈ 0.69 → about 69% shared variance, ~31% error/disagreement
• Typical workflow:
  • Compute correlation (Pearson product-moment correlation in SPSS)
  • Check statistical significance (e.g., p < .05 as significant association)

---

Standard Error of Measurement (SEM): uncertainty around scores

Main purpose emphasized

• Reliability estimates (especially Alpha) support calculation of the error around a test score.
• This supports risk-aware interpretation and decision-making.

SEM formula (as presented)

• SEM = SD × √(1 − reliability)
  • SD = standard deviation of scores
  • reliability = Alpha (example values used in lecture)

95% confidence band around a score

• Multiply SEM by 1.96 to get the margin for a 95% confidence interval:
  • observed score ± (SEM × 1.96)
• Instructor’s key message:
  • Even with high reliability, error range can still be meaningfully large, so uncertainty must be considered when making decisions.

---

Decision scenarios illustrating why uncertainty matters

Scenario 1: Emma (lower-stakes academic decision)

• Admissions target: reading score 12
• Emma scores 11
• If error range is about ±3, then 11 is within the plausible error range
• Suggested action:
  • Accept Emma (lower consequence; support exists)

Scenario 2: Joe (high-stakes employment safety decision)

• Minimum reading score: 13
• Joe scores 14
• With error range ±3, retest could be as low as 11
• Suggested action:
  • Treat 14 as potentially unreliable for cut-score decisions
  • Recommend further evaluation (e.g., retest or supervised work trial)
  • If resources are absent, the risk may justify not hiring

---

Methodology / workflow presented (detailed)

A) Internal consistency reliability (Cronbach’s Alpha) via 0/1 item data

1. Prepare data
   • Encode responses as 0/1:
     • 0 = incorrect
     • 1 = correct
2. Run reliability analysis
   • Use an Alpha calculation tool (spreadsheet) or SPSS
   • SPSS path described:
     • Analyze → Scale → Reliability Analysis
     • Select items and move them into the items box
     • Choose Cronbach’s Alpha
     • Under “Statistics,” request descriptive stats if desired
3. Read outputs
   • Cronbach’s Alpha
   • Standard error of measurement (SEM)
   • Item descriptive statistics (e.g., facility/means are mentioned)
4. Item-level follow-up
   • Conduct item analysis (the instructor references point-by-serial correlation as a next step)
   • Revise problematic items to improve Alpha

---

B) Standard error and 95% confidence interval

1. Compute SEM:
   • SEM = SD × √(1 − Alpha)
2. Compute 95% band:
   • Margin = SEM × 1.96
3. Interpretation:
   • Repeated-test score expected range:
     • observed score ± Margin
   • Use that range to manage cut-score risk decisions
   • (Assumes no learning/real change occurs between administrations.)

---

C) Inter-rater reliability / agreement using correlation (Pearson)

1. Collect ratings
   • Two raters score scripts/samples on a rubric scale
2. Compute correlation
   • SPSS path described:
     • Analyze → Correlate → Bivariate
     • Use Pearson correlation
3. Interpret results
   • r: association/consistency strength
   • r²: shared variance percentage (used as “agreement interpreted as reliability”)
   • Report significance (e.g., p < .05)
4. Visual support
   • Make a scatter plot
     • SPSS: Graphs → Legacy Dialogs → Scatter/Dot → Simple Scatter
     • Put Rater 1 and Rater 2 on axes
   • Fit a best-fit line (chart editor option)
   • Export for reporting (with r² visible/usable as needed)

---

Recap of reliability “critical issues”

• Left-side recap (what to calculate/consider)

  • Reliability is often computed as internal consistency for closed-response tests.
  • Cronbach’s Alpha is the most important/internal-consistency measure commonly used.
  • Standard Error (SEM) is crucial; the instructor criticizes literature for not reporting it.
  • “Agreement/high association” is treated as consistency; “lack of consistency” is treated as error, especially:
    • agreement across forms
    • agreement across administrations
    • agreement across raters

• Right-side recap (why it matters)

  • Reliability is about estimating uncertainty.
  • Without uncertainty estimates, it’s hard to manage risk in:
    • interpreting scores
    • setting and using cut scores
    • making high-stakes decisions
    • designing responsible decision procedures

---

Speakers / sources featured

• Robert Lado — cited for the definition of testing/reliability (language testing; book published 1961).
• No other specific speakers/sources are explicitly named beyond the instructor’s references (examples like “COVID test and trace” are mentioned, but no additional credited authors are given).