Video summary

Найм в IT сломан. Я придумал, как его починить: Методология TBSS

Main summary

Key takeaways

Business

Overview

The speaker argues that IT hiring is “broken” because the current process relies on:

  1. Unreliable resume screening
  2. Weak/unstructured interviews
  3. Manual throughput problems causing biased filtering
  4. Outdated job requirements (e.g., “3 years experience”)

They propose a data-driven hiring methodology called TBSS (Threshold-Based Sequential Sampling).

The 4 hiring problems (diagnosis)

1. Resumes are a poor selection tool

  • Resumes are treated as evidence of skills, but the speaker frames them as marketing brochures.
  • Recruiter agreement on the same resume: only ~50–60%.
  • In an analysis of 1,000 rejected resumes across major ATS systems:
    • 57% rejections were due to true qualification mismatch
    • 43% due to formatting/parsing errors and arbitrary filters
  • 34–37% of resumes contain factual misrepresentations
  • 49% recruiters caught outright lies
  • AI keyword stuffing makes this worse: candidates mirror job keywords, increasing “fiction” in resumes.

2. Interviews often don’t predict job performance

  • 2024 meta-analysis (30,000+ people): interviews explain ~9% of variation in future job performance.
  • ScienceDirect study: decisions based on unstructured interviews were less accurate than those based only on tests.
  • Claim: 77% of developers believe interview algorithm tasks don’t reflect real job skills.
  • Examples of mismatch:
    • Experienced engineers may fail due to forgetfulness of niche topics
    • Recent grads may pass due to recent prep
  • LLMs can solve similar tasks quickly (e.g., JPT for coding-style prompts), raising questions about what interviews actually test.

3. Too many candidates → biased shortcuts

  • IT/remote roles can get 2,000+ responses.
  • If recruiters spend 1 minute per resume: ~4 full workdays per vacancy.
  • Result: heavy automation + shallow screening:
    • “filter out half of good candidates”
    • open only the first 50–100 resumes and pick “unscientifically”
  • “Ghost jobs” / fake demand:
    • 43% of hiring managers post jobs without real intent (Clarify Capital)
  • Hiring outcomes dropped:
    • ~80% hired in 2020 → ~40% now (as stated from Rivio Laps)

4. Job requirements/criteria are inflated and misaligned

  • “Frankenstein” job descriptions with outdated criteria.
  • Harvard study (“Hidden Workers”): 88% of employers admit their systems filter out qualified candidates due to strict JD criteria.
  • Meta-analysis (Schmitt & Hunter, 10 years of research):
    • “Years of experience” correlates with performance at ~0.18
    • Education correlates ~0.10
    • Structured interviews correlate ~0.51 (much higher)
  • Updated meta-analysis (2022): years of experience ranked 20th+? in overall criteria (stated as #23, with structured interview #1).
  • Conclusion: “3 years experience” is near the bottom of predictive power, yet drives most ATS filtering.

Proposed framework: TBSS (Threshold-Based Sequential Sampling)

Core idea: stop guessing from resumes, then test uniformly and hire the first candidate who passes an absolute bar, using random sampling to avoid biased filtering.

TBSS playbook (process steps)

  1. Remove resumes as a selection filter
    • Only keep strict legal gating (e.g., location/work authorization).
  2. Define an absolute “bar” before interviewing
    • Write specific passing criteria.
    • Use two independent experts to validate the criteria:
      • if they can’t agree on pass/fail, the criteria are not usable—redo it.
  3. Take a random candidate sample
    • Example: 15 candidates selected randomly (via RNG/random sorting).
    • Do not sample by response time, ATS score, recruiter preference, etc.
  4. Run structured interviews
    • Interview all 15 using:
      • single structure (same questions/tasks/evaluation)
      • same evaluation criteria for everyone
      • no relative ranking—only pass/fail vs the absolute bar
  5. Hire the first candidate who exceeds the bar
    • If nobody passes: draw a new random group and repeat.
  6. Recalibration rule
    • After 5 groups (75 interviews) with no hire:
      • don’t lower the bar automatically
      • reassess job description, candidate pool, and/or bar (e.g., impossible profile, paying too little, or searching in the wrong place)

Why random sampling (the speaker’s math/assumptions)

  • Assumption example: candidate pool has ~12% truly qualified (mid-level IT average cited).
  • Sample size: 15 candidates.
  • Probability at least one qualified in a group: ~86.5%
  • Probability of missing qualified candidates in two groups: <2%
  • Insists pool size doesn’t matter much when sampling randomly (500 vs 3,000 yields similar probability if sample size is the same).
  • Sample size 15 is claimed to be “optimal” for maximizing chance while minimizing interviews.

Why structured interviews (the speaker’s evidence + tactics)

The speaker argues interviews must become a precise assessment tool, not conversation.

Structured interview rules

  • Single structure
    • Same questions/tasks for all candidates.
  • “AI/stable signals”
    • Use tasks that can’t be easily spoofed with memorized responses:
      • reasoning through ambiguous debugging in real time
      • explaining/defending past solutions under pressure
      • reacting to changing conditions mid-solution
    • Suggested formats:
      • system design + coding in a test project
      • avoid “coding from scratch under stress” (stress + missing references ≠ real work)
      • avoid offline test assignments due to vague evaluation
  • Independent assessment to prevent anchoring
    • Multiple interviewers write assessments before discussion; then compare.

Concrete example: “improve an existing broken project” test

  • In a prior company (iOS team):
    • Candidates were given a primitive single-screen app with deliberate errors and poor practices.
    • They were asked to:
      • identify what’s wrong
      • propose improvements
      • implement improvements
  • Benefit:
    • candidates can demonstrate level quickly
    • junior success doesn’t require finishing all architectural changes (seniority differentiation)

KPIs / targets / decision thresholds (explicit numbers stated)

  • Candidate sampling size: 15 per group
  • Max attempts before reassessment: 5 groups = 75 interviews
  • Expected interview workload vs old process
    • Current typical process described:
      • ~2,000 resumes received
      • ~1,900 not reviewed
      • ~50 screened
      • ~10 in-depth interviews
      • hire the “best seen”
    • TBSS described:
      • 15 interviews per group
      • likely 1–3 groups to close (stated as 15–45 interviews)
  • Bad hire cost estimate (for cost justification)
    • SHRM: bad hire cost = 30% to 200% of annual salary depending on level
    • Speaker’s numeric example:
      • Senior Russian salary: 3.6M RUB/year
      • Bad hire assumed ~100% → 3.6M RUB
      • Interviewer cost example: 10,000 RUB/hour
        • interview cost estimated ~150,000 RUB
      • Argument: even multiplying interview costs by 10 still doesn’t beat bad-hire loss
  • Hiring failure rate referenced: 46% of employees fail to cope in first 18 months (used rhetorically as evidence of current approach failure)

Actionable recommendations (what to implement)

  • Replace resume ranking with random sampling
    • Only apply legal filters pre-sampling.
  • Create an “absolute pass bar”
    • Validate with two independent experts before running interviews.
  • Standardize interviews
    • Same questions/tasks and evaluation criteria for all candidates.
  • Use structured, live-thinking assessments
    • Prefer debugging/improvement tasks in an existing project over pure memorization/coding-from-scratch.
  • Add independent scoring to avoid anchoring
    • Score first, discuss later.
  • Use the “no hire in 75 interviews” rule to redesign the system
    • Don’t keep interviewing with the same bar—adjust pool/requirements/bar.

High-level business impact claims

TBSS is positioned as:

  • More honest: less bias from resume/ATS artifacts
  • More effective: structured interviews better predict performance
  • More efficient: avoids manually reviewing 2,000 resumes

The speaker frames it as discontinuing “broken” hiring components rather than adding complexity.

Sources / presenters mentioned (as named in subtitles)

  • Anton Nazarov (credited/blamed for “ruining” resume-hiring behavior)
  • Job White (2024 data) (used for 12% qualified assumption)
  • Clarify Capital (ghost jobs: 43% posted without intent to hire)
  • Rivio Laps (vacancy-to-hire ratio: ~80% in 2020 → ~40% now)
  • SHRM (bad hire cost: 30%–200% of annual salary)
  • Harvard study “Hidden Workers” (88% employers admit filters exclude qualified candidates)
  • Schmitt & Hunter meta-analysis (10 years; correlation figures; years of experience vs structured interviews)
  • Polo Sackett and colleagues (2022) meta-analysis (rankings of criteria)
  • Berkeley University (structured interviews predict success nearly 2x better; +29% for job-task-based interviews)
  • ScienceDirect (unstructured interviews less accurate than test results)
  • Kiran (77% developers believe algorithmic interview tasks don’t reflect job skills)
  • Mentions of JPT Chat (example of AI solving coding tasks quickly)
  • References to Hunter and Hunter (1984 meta-analysis) (used to argue cognitive ability predicts success for candidates without experience)

Original video