Introduction to Language Testing and Assessment

Main ideas, concepts, and lessons

• Language testing is a central Applied Linguistics discipline (not dreary).

  • It connects to many areas teachers and applied linguists use (e.g., reading, speaking/listening, and integrated skills).
  • In educational contexts such as English for Academic Purposes, tests often involve tasks like reading texts, listening to lectures, and then producing assignments—so test quality depends on what exactly is being assessed.

• Language testing has three key components:

  1. Theory / construct definition: what the test is meant to measure and how.
  2. Practice: how testers actually build the tests.
  3. Impact: what happens in the real world when test-takers receive scores that others use to make decisions.

• Why tests exist: assessments enable decisions.

  • Scores are given so that someone else can decide about the person taking the test.
  • The “stakes” of those decisions vary:
    • Low-stakes tests: for learning/pedagogic decisions (e.g., planning curriculum/syllabus, remedial classes).
    • High-stakes tests: major consequences for learners (e.g., university admission; ability to live/work in another country).

• Testing has grown more important because of high-stakes pressures—especially cheating.

  • Examples of cheating tactics and enforcement described:
    • Hidden equipment in clothing (hidden recording devices; communication to someone outside the test hall).
    • Drones used to detect radio/cheating signals (China’s national college entrance exam enforcement; also mentioned: penalties and reduced ability to retake for years).
    • India example: exam takers reportedly asked to remove clothing to prevent hidden materials (addressing hidden “cheat” items).
  • Underlying reason for cheating: in high-stakes contexts, test scores become valuable “currency” for individuals and institutions (education access, school evaluation, investment, even school closure risks).

• Extreme score-manipulation examples illustrate the power of tests.

  • Florida FCAT “placebo” story: students told that chocolate Power Bars would improve performance on a computer-adaptive test; the video uses this to show how tests can drive unusual real-world behaviors.

• Testing has long historical roots (centuries).

  • Testing used to run educational systems and shape society (example: Plato and the Republic).
  • Vernon Spolsky is referenced as relating to testing history and merit-based selection.
  • Historical evidence from China (civil service examinations dating to ~2nd–3rd centuries A.D.):
    • Strict exam rules (limited restroom use, no exchanging papers, no looking at others’ work, restrictions on noise, sealed envelopes for finished papers).
  • Lesson: testing serves a social policy / fairness function, so language testers and teachers matter greatly.

---

Methodology / “test design cycle” (detailed bullet points)

When designing and conducting language tests, the speaker presents four core assumptions and then a test design cycle.

A) Four basic assumptions behind language tests

• 1) Task relevance assumption

  • The tasks learners do must be relevant to the decisions the test is intended to support.
  • Typically, tasks should be drawn from the domain of interest (e.g., for an academic English test, use tasks from academic contexts).

• 2) Language relevance assumption

  • The produced language/performance is assumed to be relevant to what’s being measured (i.e., the responses contain information needed for the inference).

• 3) Score-to-ability relationship assumption

  • After scoring performance, there is assumed to be a direct relationship between the score (a reductive summary, e.g., 60/75/A grade) and the test-taker’s ability/knowledge.

• 4) Score sufficiency/necessity assumption (for decision-making)

  • The score can be used as:
    • Necessary but not sufficient (e.g., admissions may require a language score but also check transcripts/references), or
    • Another role depending on how the decision is made.

B) Test design cycle (step-by-step)

• 1) Definition of purpose of the test

  • Why the test is being given
  • What decisions will be made
  • Who the intended test audience is
  • What predictions are required

• 2) Criterion (real-world performance target)

  • The real-world ability/behavior to predict
  • Example given: ability to read first-year academic texts in a business context

• 3) Construct definition

  • The underlying skill/abilities the test is intended to measure
  • Example: reading skills such as:
    • reading to integrate information
    • skimming for key facts
  • Key point: constructs are abstract; they must guide task design.

• 4) Task and item design (operationalization)

  • Create tasks/items intended to measure the constructs within the criterion
  • Also includes writing test specifications:
    • descriptions (often with sample items) so others could produce similar items/tasks

• 5) Prototyping and piloting

  • Trial items/tests with small groups (often 4–5 learners)
  • Protocol analysis: learners explain what they’re doing/thinking while answering
  • Goal: check whether the learner processes match the test designer’s assumptions (linked to validity).

• 6) Field testing

  • Administer the near-final/final test to a large group
  • Collect statistical data (e.g., whether items discriminate among learners as expected)
  • Enables stronger justification of inferences from scores.

• 7) Make inferences and support decisions

  • Use score interpretations to make decisions consistent with the original purpose
  • The entire cycle exists because the test must support valid inferences for its intended decision use.

---

Validity example: operationalizing a construct and using a performance level

Construct + operationalization

• Construct example: “integration of knowledge and ideas” (from 2010 Common Core standards)
• Operationalization example:
  • Translate quantitative/technical information expressed in words into visual information (e.g., tables/charts)
  • Translate information expressed visually or mathematically into words

Task example (facility survey recommendation)

• Learners see three proposed facilities (coffee shop, something like a disk/tech-related option, and a reading room).
• Learners must:
  • act as a student union representative
  • do survey research using provided department preference data (first and second choices)
  • write a short recommendation to the student union justifying which option to build

Scoring level example (Level 4)

• Level 4 criterion: the test taker can clearly describe decision-making processes and methods and produce clear short text.
• Example response behavior described:
  • create a chart
  • tally preferences
  • weight first choices more than second choices (e.g., first choice counted twice as heavily)
  • state the winning facility and explain why

Validity point (core lesson)

• Validity question: whether the score from this response genuinely supports an inference about the intended construct.
• Key claim: you wouldn’t rely on one task only; you’d use multiple tasks to justify the construct inference.
• For classroom use, full-scale research may not be required, but careful design and specifications are still emphasized.

---

Final lesson

• There is a high need for language assessment literacy:
  • Teachers and applied linguists need skills to use tests and assessments effectively for:
    • classroom improvement
    • research
    • making important decisions about learners’ futures
• The speaker points viewers to a website for additional materials.

---

Speakers / sources featured

• Glenn Fulcher (speaker; Professor of Applied Linguistics / language assessment, University of Leicester, UK)
• Vernon Spolsky (referenced regarding the history of testing)
• Plato (referenced via The Republic, including testing-based selection in his ideal society)
• Mashable (referenced as the source reporting on drone use for detecting cheating in China)
• 2010 Common Core Standards (United States) (source of the construct/operationalization example)
• FCAT / Florida computer adaptive test (referenced in the placebo/power bar example)