Video summary

We Got Astra First...Now We're Fighting

Main summary

Key takeaways

Technology

Summary (technology, product features, and analysis)

  • Astra is presented as OpenAI’s “next-generation” large model—described as much larger than prior releases and capable of stronger reasoning and coding. The speakers emphasize it should compete with top-tier models (including references in the “Fable” / Gemini-family space).
  • No reliable public cost/behavior guarantees yet: the creators note they only have early access and don’t know the final price or exact runtime limits, though they estimate very high token burn in testing.

Model comparisons & key claims

Astra vs. “Fable” (current models)

  • More natural conversation: both hosts claim Astra is less “GPT-icky”, producing language they find more conversational.
  • Better task execution with computer tools: described as stronger for:
    • multi-step problem solving
    • longer autonomous-style workflows
  • Where it falls short: Astra is criticized for
    • instruction-following regressions
    • UI/UX failures, especially when asked to rewrite front-end interfaces.

Testing / benchmarks described

1) DEF CON-style puzzles (image/video/asset solving)

They test Astra on two puzzle types:

  • Dress puzzle (multiple photos/video angles + encoded patterns)

    • Astra solves it 2 out of 3 attempts without “official clues.”
    • It gets stuck unless given clue context from the puzzle creators.

  • Jumbled Rubik’s-cube grid puzzle

    • Astra improves movement efficiency.
    • Still, it fails without higher-level guidance that identifies the “golden cubes as keys.”

Key takeaway: Astra can handle complex multi-modal puzzles, but may need extra contextual framing to avoid partial interpretations or “cheating” unless constraints are tight.

2) 3D “fish tank / game generation” test bench

They build a benchmark to push beyond previous limitations:

  • Prior models produced 2D fish better than 3D, so they test 3D conversion.

Findings

  • Astra produces legitimate 3D environments and game assets (using code and optionally Blender/CLI).
  • It struggles to make highly realistic fish, often only achieving “looks-like-a-fish” quality.
  • Outputs are criticized as functional but bland, plus UI issues, such as:
    • unnecessary text
    • weird interface elements
    • inconsistent layout behavior

3) Agentic coding + repo modernization: Ping.gg and T3 rewrites

They use Astra to modernize an old codebase and check whether it:

  • preserves existing UX design intent
  • correctly uses the desired tech stack (they name: Convex, Agora, React 19, Tailwind v4, bun, etc.)
  • rewrites without breaking production workflows

Main critique

  • Astra can do large refactors (even TypeScript → Rust progress), but often misinterprets product requirements, especially:
    • front-end UI reimplementation instead of reusing established UI patterns
    • excess subtitles/UI clutter
    • ignoring system/user constraints such as “reuse as much of the old interface as possible”
  • It can also break or fail to complete PR workflows (described next).

PR / “agent” workflow behavior (major analysis section)

  • The speakers identify a core failure mode: Astra sometimes does not complete actions implied by skills or instructions, particularly around PR lifecycle steps.

Intended “skill” (agent capability)

  • Monitor a pull request
  • Review bot feedback / CI checks
  • Fix code and re-upload
  • Iterate until checks are green

Reported behavior problems

Despite defining a clear “supervise PR” behavior, they claim Astra may:

  • upload changes but then stop supervising
  • fix code but not trigger the expected CI/review re-run
  • misreport completion state, e.g., claiming a “nightly” build will receive fixes after a “recompile/update” even when no commit/PR merge occurred

Resulting complaint

  • Communication/state reporting is bad, making it difficult to determine whether work is truly complete versus merely locally changed.

Speed, reliability, and cost-related observations

  • Slower than competitors: speakers suggest slowness may come from aggressive verification and deeper tool use.
  • Rough tool execution error rates (~errors per 100 commands):
    • Astra: ~7.5
    • Soul: ~12.9
    • Fable: ~2.1
  • They also compare tool-call failure rates and note the gap can be less pronounced depending on the metric.

Cost

  • They estimate heavy token usage in puzzle and coding tests.
  • They discuss per-token pricing bands and warn that final pricing remains uncertain.

“Automatic mode” / autonomy

  • Astra appears to support more autonomous, agent-like workflows.
  • However:
    • Automatic mode can be slower
    • reliability depends strongly on how specific and structured the system prompts/instructions are.

Final takeaways (overall sentiment)

  • Praised: Astra is viewed as a top-tier multimodal + coding + computer-use model, with more “native” language and stronger tool-driven execution than prior OpenAI releases.
  • Not dependable by default: it’s not considered reliable for UI/UX rewrites or PR supervision unless:
    • instructions are extremely explicit
    • “skill” definitions tightly match the expected final state
    • stopping/communication is accurate

Main speakers / sources

  • Primary speakers: two long-time AI model testers/reviewers:
    • Ben (speaker/host)
    • Dex (referenced; also a co-host/participant)
  • Referenced third parties/sources:
    • Sam (Twitter visualization/educator)
    • OpenAI (model provider)
    • PostHog (video sponsor)

Original video