Video Summary - Washback and Impact

Main ideas, concepts, and lessons

1) “Impact” and “Washback”: how language tests shape society

Impact (older concept) frames how testing and assessment systems affect society at large—sometimes even functioning as an engine to control educational systems and help engineer a desired society.
Washback (more recent, especially from the 1980s) focuses on the effects of tests on classroom teaching and learning behavior—what teachers and students do because of the test.

2) Historical roots: education/testing as governance

Plato (e.g., Republic) is presented as an early example: only those who pass trials/tests become Guardians/philosopher-kings, shaping leadership and society.
Imperial China’s examinations are another example: exams were used to select bureaucracy while reducing aristocratic influence by placing their children in key roles.

3) Validity theory as the bridge to “washback” (Messick)

The lecture links washback to validity—specifically what tests mean and what consequences follow.

Consequential validity / washback within Messick’s validity framework

Key claim: Validity includes not only test-internal evidence, but also consequences for stakeholders (learners, schools, teachers, etc.).
The lecturer notes ongoing debate about what exactly Messick meant by consequential validity—whether it includes only negative consequences or broader societal/political impacts.

Core validity components described (from Messick via the 1996 paper)

Messick’s view breaks validity into these aspects:

Construct validity (central concept), made up of:
- Content aspect
  - What the test contains (e.g., topics, text types).
- Substantive aspect
  - Whether test scores reflect the constructs intended to be assessed.
- Structural aspect
  - The internal structure of the test.
  - Example: if testing multiple skills/constructs (e.g., reading subskills), show via:
    - correlation analysis or factor analysis that components reflect distinct constructs.
- Generalizability aspect
  - Whether scores from one test form generalize to other forms of the test.
  - Example: if security requires multiple versions—do scores remain comparable?
- External aspect (concurrent or predictive)
  - The relationship between test performance and an external criterion (often real-world ability/performance).
  - The lecturer calls this likely “most important” for assessing how test scores predict/represent real competence.
Consequential aspect (added by Messick)
- The effects of test use on stakeholders:
  - Learners
  - Schools
  - Teachers
- (And potentially wider social/policy effects, per the lecturer’s interpretation.)

4) How washback is defined and studied (Alderson & Wall)

Alderson & Wall (1993): definition of washback

Washback is defined as what students and teachers do that they would not otherwise do because of the test.
This implies washback research should ideally include:
- a baseline comparison (what they would have done without the test).

Why earlier washback research was limited

Many studies only:

asked participants what they do under current test conditions, then
hypothesized what they “might have done” without the test.

The lecturer criticizes this as insufficient because washback requires a true comparison.

Washback hypotheses (Alderson & Wall; “15 hypotheses”)

The lecture highlights four hypotheses most researched:

The test influences what teachers teach and how they teach
- Includes:
  - materials,
  - topics,
  - language features/functions emphasized.
- Example: a dissertation comparing two schools in the same village:
  - one preparing for a particular test,
  - one not.
- Result: a radical difference in grammar taught—teachers in the test-focused school chose grammar to raise scores and shaped the syllabus accordingly.
The test influences learner behavior and acceptance
- Includes learners’ choices, study practices, and strategies.
- Example: learners using past test items/past papers to learn “how to do” the test.
Potentially positive or negative washback outcomes
- Debate framed as:
  - Popham: well-designed tests can positively influence teaching/learning, especially when test coverage is broad/valid.
  - Alana Shami: argues washback is “almost wholly negative.”
- The lecturer presents this as an active research debate.
Washback affects curriculum and learning focus (within broader frameworks)
- Treated as part of the broader system influence—whether curriculum becomes test-driven.

5) Stakeholders and system-level washback (Hughes model)

Washback affects many stakeholders.
Hughes lists concerned participants, including:
- teachers
- learners
- administrators
- materials writers
- curriculum designers
Effects expand when test results are tied to ranking and funding:
- administrators respond to league tables by making decisions affecting:
  - teacher hiring,
  - teacher evaluation,
  - school improvement actions.
Materials/curriculum development is described as strongly influenced by assessments.
The lecturer asserts that the Common European Framework (CEFR) has had an extensive effect on materials writers and curriculum designers, and personally views it negatively.
Hughes is said to omit a major stakeholder category:
- policy makers, who are strongly influenced when tests affect national standings.

Policy influence example: PISA

PISA is framed as assessing countries (not individuals) for political/economic purposes.
The implication:
- poor ranking → pressure for radical education-policy intervention
- → major washback at the policy level.

6) Mediating variables: why washback differs across contexts

Washback is mediated by complex variables. For a washback study, consider:

Resources
- access to textbooks, internet, etc.
- especially important in socio-economically deprived areas
Teachers
- qualifications, experience, and language assessment literacy
- the lecturer suggests assessment-literate teachers might mediate washback more positively (noted as under-researched)
Time
- frequency and timing of tests
- most research targets high-stakes end-of-program exams, but the lecturer asks whether more frequent tests change washback patterns
Learners
- motivation
- first-language aptitude
- gender
- background culture
- attitudes toward tests
- learning styles and affective factors
- the lecturer notes many studies are descriptive/small-sample and don’t establish systematic mediating effects
Parents / socioeconomic status (SES)
- parent involvement and SES can mediate impacts
Country context
- especially for international tests: do national attitudes and practices change washback?
- requires cross-country comparisons

7) Research methods in washback studies (including “baseline”)

The lecture presents methodology options and preferred designs.

Ideal designs

Baseline studies
- measure behavior without test influence, then compare.

If baseline is impossible

Compare new vs old tests
- Example design:
  - study under the old testing system,
  - then repeat once the new system begins,
  - compare results to infer washback changes.

Other approaches mentioned

observations
diaries
interviews with:
- learners,
- teachers,
- test writers
- (officials could also be studied, though it’s not commonly done)
university admissions offices as a somewhat investigated case:
- administrators interviewed on how they believe tests affect their work
document analysis:
- textbooks, lesson plans, materials under different washback conditions
surveys and questionnaires:
- how teachers/learners respond to assessment systems

Core methodological lesson

Pure description of one context/small group is useful, but to learn about washback “itself,” research needs comparison (across time, tests, or contexts).

8) The state of washback research (around 2012)

The lecture describes a shift in thinking:
- Earlier question: Does washback exist?
- Later question: Is washback a theory?
Conclusion:
- washback exists, but it is not yet a predictive theory because the field is not close to reliably testable predictions.

9) Political controversies around standards-based testing and “curriculum independence”

The controversy centers on:

Standards-based testing using external standards/frameworks (e.g., CEFR, U.S. ACTFL frameworks).
Dispute:
- whether teaching becomes driven by the test (“teaching to the test”)
- whether truly curriculum-independent tests can exist (tests assessing skills/constructs not directly taught)
The lecture suggests:
- without knowing enough about washback, we can’t confidently create tests that avoid unintended effects.

“Insider trading” controversy in test production

Some agencies:
- produce tests,
- produce teaching materials,
- and train teachers to teach toward the test.
The lecturer criticizes this as:
- monopoly-like influence,
- potential bias toward test-prep solutions.
Proposed “condition” for legitimacy (per the lecturer):
- the test designer/producer should not also produce learning materials or train teachers for test preparation.

10) From “impact” to “certification”: why tests can be socially powerful

Why tests are used

Safety/indirectness
- tests are indirect decision mechanisms (unlike direct workplace performance supervision)
Cost-effectiveness
- assess many individuals without monitoring them in workplaces
Predictive function
- tests predict current or future abilities/conditions
- tied to external validity (predicting performance on criteria beyond the test)

Social outcome: certification

Tests certify individuals for roles/opportunities (and may require re-certification).

11) Examples of negative impact (impact, not washback)

Army Alpha and Beta (1917; Robert Yerkes and colleagues)

Presented as an early large-scale English language testing system.
Immigrant recruits were tested.
Outcomes determined assignment:
- language development battalions vs active battalions.
The lecture claims results were interpreted through the era’s hereditary theories, influencing immigration/social policy and attempts to restrict immigration from countries associated with lower test outcomes.
Early marking technologies are highlighted (e.g., acetate overlays) for grading at scale.

PISA + Cambridge language tests (2020)

The lecture cites an article (El Gazette, Oct 9, 2020) stating:
- Cambridge English language tests would be included in OECD PISA.
A quoted claim (paraphrased by the lecturer):
- the purpose is to create league tables for language learning at the country level, not individual assessment.
The lecturer argues this is politically unacceptable because policy makers can use rankings to guide education policy.

12) Philosophical debate: tests as fairness vs tests as control

Foucault’s critique (via “panopticon” and examination)

Panopticon (Jeremy Bentham):
- prison design where prisoners feel watched.
Foucault argues exams function similarly:
- “surveillance,”
- “normalizing judgment,”
- visibility, classification, and punishment.
Exams are described as ritualized mechanisms combining:
- power + knowledge,
- truth-claims,
- objectification of those assessed.

Competing view: meritocracy and fairness

Tests are argued to provide:
- meritocracy,
- fairness,
- equal chances without privilege or bias.
The contrast:
- tests enable opportunity (mobility, education, job access)
- vs enforce elite values and maintain social status.

Michael Young on meritocracy (1950s)

Meritocracy described by Young as pejorative:
- “meritocratic dictatorship”
- test skill becomes the route to power
- those already advantaged can prepare better and maintain advantage across generations.

The “abolish test” thought experiment

The lecture frames a moral “big red button”:
- abolish all tests/assessments.
Implicit questions:
- Would society still work fairly without mechanisms?
- Would rights/opportunities be truly equal?

Speakers / sources featured (identified in subtitles)

Plato
Samuel Messick
Charles Alderson
Diana Wall
Charles Popham
Alana Shami
Hughes (author of a washback model; first name not given in subtitles)
Bart Davis
Fred Davidson
Jeremy Bentham
Michel Foucault (spelled “Fuku” in subtitles)
Michael Young
John Stuart Mill
Cambridge Assessment
OECD (Organisation for Economic Co-operation and Development)
Robert Yerkes (listed as “Robert Yorks” in subtitles)
(U.S. Army) Army Alpha/Beta test developers (specific names not provided in subtitles)
The El Gazette
ACTFL (American Council on the Teaching of Foreign Languages)
CEFR (Common European Framework of Reference for Languages)

Washback and Impact

Key takeaways