Video summary

11 Things That Separate Vibe-Coded Toys From Real Apps

Main summary

Key takeaways

Technology

Main idea / framework

The video contrasts “vibe-coded toys” vs real apps, arguing the difference is process and vocabulary, not just talent or tools.

It uses a “gate vs net” mental model:

  • Gates: prevent bad changes from reaching production (e.g., specs, validation, CI checks).
  • Nets: catch problems after something slips through (e.g., testing, error handling, observability).

1) Spec-driven development (core production gate)

Key concept: Define behavior using written specs so the model produces verifiable, testable outputs instead of “wild” changes.

Tools mentioned

  • OpenSpec (daily driver)
  • GitHub Spec Kit

Why it helps

  • Specs become a contract the model must satisfy.
  • Breaks large tasks into smaller chunks (models can fail on huge asks).
  • Provides traceability/lineage (artifacts, issues, commits logged; can debug weeks later).

Process style (OpenSpec: “actions,” not phases)

  1. generate a proposal
  2. produce concrete specs
  3. create system design/context
  4. generate tasks
  5. implement
  6. verify

It integrates test-driven development during task generation.

Claim: Avoid vibe coding without these guardrails, which leads to frequent production breakage.

2) Project documentation hooked into Claude Markdown

Key concept: Keep agent instructions fresh and contextual, not generic.

What to document in Claude/agent markdown

  • Known anti-patterns (things the model should not do)
  • Non-inferables (critical conventions/knowledge not obvious from the codebase)
  • Context scoping and pointers via nested markdown files per directory (API layer, UI layer, etc.)

Tool example

  • Intent Layers skill generates custom agent markdown for big directories (the author notes it “doesn’t actually work anymore” per some users; points to a GitHub repo).

Warning

  • Stale documentation is a major failure mode: agents may follow outdated architecture notes without realizing.

3) Version control as the safety system

Key concept: Version control is required for production readiness because it enables safe iteration and rollback.

Recommendations

  • Atomic commits: one logical scoped change per commit, with commit messages explaining why.
  • Branching + PR flow: never commit directly to main; enforce protected flows (e.g., main only updated after tests).
  • Rollbacks and recovery strategy: assume something will break and plan how to revert.

4) Testing (guard rails for agent + humans)

Key concept: Tests constrain what “correct” means and guide the coding agent toward valid implementations.

TDD approach

  • Red–green–refactor
    • write a failing test
    • minimal code to pass
    • refactor
  • Recommended to combine with OpenSpec task generation.

Test types

  • Unit tests (fast, isolated)
  • Integration tests (multiple components)
  • End-to-end tests (browser-driven; example: Playwright)

Concrete end-to-end focus

  • money/critical paths (e.g., Stripe signup and upgrades)

Regression testing

  • Whenever a bug appears, add a test so it won’t reoccur.

Model warning

  • Language models may generate tests that can never fail unless you enforce anti-pattern prevention.

5) Authentication vs authorization (security gate)

Definitions

  • Authentication = “who are you?”
  • Authorization = “what can you do?”

Claims about models

  • Authorization is especially hard for models to infer correctly without explicit review.

Recommendations

  • Audit data schemas and access patterns so the agent understands entity permissions.
  • Enforce authorization in multiple places:
    • Server/API checks
    • Database row-level security (defense-in-depth)

Example

  • Supabase Row Level Security (RLS)
    • recommendation: enable RLS by default; only disable for specific tables.

6) Error handling (containment instead of app-wide failure)

Key concept: Decide whether failures are contained or crash the whole app.

Approach

  • Handle expected errors with user-friendly messages (e.g., “retry” for mail failures).
  • For unexpected errors, route to global error handlers at boundaries.

Avoid

  • exposing stack traces/sensitive details to users
  • swallowing errors silently

Tools

  • Sentry for monitoring so you learn about issues before widespread user impact.

7) Input validation + retries (reduce happy-path assumptions)

Recommendations

  • Validate inputs on the front end (reject wrong types early).
  • Wrap external API calls with timeouts and retries to prevent error-rate explosions and abuse.

8) Databases (production gotchas)

1) Migrations are required

  • Avoid raw schema updates; use a migration system (schema versioning + rollback/forward safety).

2) Performance impacts are not automatic

  • Indexing: add indexes for frequently queried/sorted fields.
  • Avoid N+1 queries: agent code often causes excessive DB round-trips; consolidate into fewer queries.

3) RLS again

  • Database-level security is part of the database section.

9) Security (beyond basic auth)

Recommended security tools

  • DeepSeek (agent security harness; author says it finds obscure issues; likely via Claude Code/Codeex license)
  • Trail of Bits repo/skills (auditing plugins, security-focused diff review, static analysis)

Checklist concepts

  • OWASP Top 10 as a recurring checklist
  • Secrets management: store secrets securely (not in frontend code; not leaked via env files)
  • Pre-commit secret scanners to avoid committing secrets to git history

Business logic security warning

  • Even non-auth issues can become security vulnerabilities (e.g., “fail open” leading users to access paid features).

Example

  • Trail of Bits skill “Sharp Edges” flags error-prone APIs, dangerous configurations, and “foot-gun” design.

10) Hosting + environment separation

Hosting recommendation principle

  • Choose based on:
    1. engineering experience level
    2. the app’s cost model

Preference

  • Prefer platform-as-a-service like Vercel to avoid complex VPS/AWS/Google Cloud decisions early due to cost/complexity.

Environment separation

  • use preview deployments for staged testing vs production.

Scaling/cost planning

  • ask the agent to analyze codebase usage (edge functions, DB usage, expected token usage) and do cost-benefit calculations.
  • goal: avoid surprising bills (example: unintended $20k cost scenario).

11) Deployment pipeline + safe database migration order

CI/CD pipeline

  • Use GitHub Actions
    • on PR: lint, typecheck, run tests, then build
    • deploy preview only if checks pass

Migration-safe deployment convention

expand → migrate → contract

  1. Expand: add new columns/fields first
  2. Migrate: backfill + ship code reading expanded schema
  3. Contract: remove old schema only after new code is proven

Safe releases

  • test in staging/preview before promoting to main/production
  • ensure migrations run before code that depends on them

Final section: Observability (nets for post-deploy detection)

Goal: identify what’s happening in production without relying on user screenshots.

Three pillars

  1. Logs: centralized events with request IDs/user IDs (e.g., Sentry)
  2. Metrics: dashboards + alerts (e.g., spike in 500s from OpenAI integration)
  3. Traces: end-to-end request flow to see how data moved before errors

Operational example

  • alert on any new unhandled errors caught by global handlers.

Main speakers / sources (as referenced)

  • Speaker/author: the video narrator (no separate person identified explicitly).
  • Sources/tools mentioned:
    • OpenSpec
    • GitHub Spec Kit
    • Ora’s Superpowers (referenced for TDD and testing skills)
    • Claude/Claude Code (implied by “claude markdown” and “Claude/agents”)
    • Intent Layers (skills/library)
    • Supabase (RLS)
    • Sentry
    • Playwright
    • DeepSeek
    • Trail of Bits
    • OWASP Top 10
    • Vercel
    • GitHub Actions

Original video