Programmer tests Claude vs GPT vs Grok: I'm surprised!

Tech/Product Focus: Comparing Coding Agents and “Model Fit”

The speaker argues that the main problem in today’s model ecosystem is “model comparison fatigue.” Even so, the video is framed as a practical guide for choosing between:

• Claude (Fable/Opus family)
• GPT (5.6)
• Grok (4.5)

…especially for coding agents, whether they run interactively or as part of automated workflows.

---

Task “Buckets” for Coding Agents (Core Framework)

The video categorizes where the model will be used to decide what to pick.

1) Interactive Agents

These involve hands-on use, such as:

• Running models via CLI
• Using IDE agents like Cursor / Warp

a) Scoped Work (requirements known upfront)

Typical examples:

• Migrating architecture
• Fixing bugs with testable criteria

Framing: goal-oriented “hill climbing” toward passing tests (mentions “slash goal”).

b) Unscoped Work (success criteria unclear)

Typical examples:

• Landing page ideation with multiple options
• Prototyping “the vibes” before details are finalized

---

2) Automated Agents / Cloud Automations

Examples:

• GitHub Actions triggers
• cron jobs

Within this bucket:

• Code review (speaker mentions building/rolling their own review agent)
• Triage (filling in details on a task board)
• Implementation agents that convert tasks into a scoped solution
• Mentions weekly architecture reviews as unscoped, but out of scope

---

Evaluation Metrics (How the Models Are Judged)

The speaker evaluates models along three axes:

• Cost
  • Token usage and subscription/plan limits
  • Claude/Codex/Grok differ substantially
• Intelligence
  • Ability to reach working solutions without excessive prompting
• Taste
  • Not just “correctness,” but whether the model:
    • fills missing requirement gaps
    • challenges assumptions
    • produces better UX/design decisions
  • The speaker specifically associates this strongly with Claude/Fable

---

Model-by-Model Findings (Speaker Anecdotes)

1) Fable (Claude)

Cost

• Reported as very expensive
• Highest observed session cost: ~$400 (subsidized by Claude subscription)
• Another “not super intensive” conversation: ~$76

Intelligence & Taste

• Claimed to be especially strong in “taste” (design/UX feel and edge-case handling)
• Examples include:
  • Reviewing styling of a PR for a terminal button in the Notes app “Hubble”
    • Uses HTML visualizations + a built-in terminal
  • Identifying UX issues such as:
    • icon weight / visual heaviness
    • missing empty-state considerations
• Another example: implementing a “chat about this note” workflow using:
  • an environment variable containing the file name
  • pre-injection of prompt content for the CLI
  • user-configurable behavior

Benchmark Reference

• For “intelligence,” the speaker points to an “artificial analysis bench,” claiming Fable wins across many benchmarks.

Takeaway

• Best for unscoped work needing many options and nuanced UX/design judgment.

---

2) GPT 5.6

Cost Efficiency

• Emphasized as much more cost efficient than Fable
• For a major scoped task (migrating to the React compiler):
  • cost was less than an expensive Fable session
  • even while delegating parts to Codex
• Reported deeper comparison landing in the ~$20 range for a scoped implementation

Intelligence

• Rated as highly intelligent, especially for end-to-end implementation
• Uses a “slash goal” / test-driven “hill climbing” approach, including:
  • removing items like useMemo / useCallback
  • ensuring tests pass
  • click-based testing (via Codex “computer use” mentioned)

It also handled:

• expanding scope (web + desktop migration)
• producing results that “just worked” per the speaker

Taste

• Taste is acknowledged as improving, but not as strong as Fable
• Example failure mode:
  • it migrated the desktop app but did not migrate shared UI packages
  • these shared packages were used by both web and desktop
  • issue was attributed to treating instructions too literally rather than asking clarifying questions or inferring implied scope

Takeaway

• Best for scoped work when requirements are known and you want efficient, correct implementation.

---

3) Grok 4.5

Cost/Token Efficiency and Speed

• Positioned as the king of token efficiency among the compared models (relative to frontier models)
• Emphasizes speed and cost per task
• Examples include running tasks on Warp:
  • debugging bash commands with a script for about 1.6 credits (roughly “five cents” magnitude)
  • refining automated workflow debugging

Intelligence

• “Pretty freaking high,” though not as strong as frontier leaders in raw capability

Taste

• Has “some” taste:
  • enough to ask clarifying questions that reveal gaps
• Uses a “grill me” skill (attributed to Matt Pocock) to repeatedly challenge/clarify the approach until it’s understood.

Code Quality

• Produces good-quality code, though it may be more verbose

Takeaway

• Best for cloud automations / cloud-run workflows where efficiency matters, such as:
  • triage
  • code review pipelines
  • frequently run, well-scoped implementations
• Also recommended for individuals on a budget (Twitter Premium credits can extend usage).

---

Deployment Guidance: Where Each Model “Shines”

• Unscoped work (options, ideation, design judgment): Fable (Claude)
• Scoped work (requirements-matching implementation):
  • GPT 5.6 (winner)
  • Grok as a cost-efficient alternative
• Cloud runners / automated coding workflows:
  • Grok 4.5 (best match: token efficient, fast, practical for repeated tasks)

---

Mentioned Advanced Pattern: Delegated Orchestration (Fable Delegating)

The speaker discusses an “emergent” approach to combine Claude + GPT:

• Use an orchestration agent (Fable) that delegates subtasks based on task type:
  • GPT for scoped implementation
  • Sonnet (mentioned) for unscoped front-end brainstorming
• Mentions a Claude MD skill/instruction from Theo (Obsidian fame) to enable delegation to sub-agents
• They tried it and observed high costs, even when delegation was happening:
  • examples: ~$76 and ~$400 runs
  • suggests more experimentation/benchmarking is needed

Intended Benefit

• Improve overall code quality and cost efficiency by splitting responsibilities across models:
  • taste (Claude)
  • intelligence (GPT)
  • efficiency (Grok)

---

Main Speakers/Sources

Primary speaker

• The YouTube narrator/reviewer (not explicitly named in the subtitles)

Referenced sources/tools/skills

• Matt Pocock (the “grill me” skill)
• Theo (Obsidian-related delegation/orchestration skill/instruction)
• Coding agents/tools mentioned:
  • Cursor
  • Warp
  • a CLI-based setup (plus Codex computer use)
• App/tool mentioned:
  • Hubble (Notes app with HTML visualizations + built-in terminal)
• Benchmark pointer:
  • “artificial analysis bench” (publisher not named in subtitles)