Video summary

I Tested New GLM-5 vs Opus and Sonnet. Wow.

Main summary

Key takeaways

Technology

Summary of the Test (Fairness + Setup)

  • Purpose of test / fairness: The speaker compares the newly released GLM 5 against Claude code models. They initially aim for a fair comparison vs Sonnet (Claude Sonnet), but note that Zeth.ai’s own benchmark was mainly against Opus (older Opus generations mentioned: Opus 4.5 and GPT 5.2), not Sonnet.

  • Project used for benchmarking (Laravel codegen task): A teammate used Sonnet to implement phase one of a project in Laravel + Filament. The LLM test focuses on phase one: generating Laravel database models + migrations for tables, including columns and their logic.

  • What the specification requires: More than migrations—LLMs must produce:

    • migrations
    • models
    • factories
    • seeders

  • Task structure: The task is split into six total sub-phases, making it a good “longer task” test for whether an LLM can complete everything without supervision.

How GLM 5 Was Run (OpenRouter via VS Code)

  • No direct GLM subscription: The speaker uses VS Code with an installed client extension, selecting GLM through OpenRouter.

  • Prompt cost observations:

    • “shown” cost around $2.53
    • actual OpenRouter activity shows $1.27 (nearly ~2x discrepancy)
    • caveat: pricing/limits vary by subscription

Speed Comparison (Same Prompt, Different Latency Effects)

  • Sonnet: ~10 minutes
  • Opus: ~7 minutes
  • GLM 5: ~23 minutes

The speaker initially suspects:

  • potential system load / speed issues
  • added latency from multiple layers (VS Code + extension + OpenRouter)

Later, they find an explanation tied to the amount of generated code.

Key point: GLM 5’s slowness is likely not only infrastructure latency—it correlates with producing more output.

Code Quality Comparison (Laravel Outputs)

1) Eloquent Model Output

  • GLM 5 (left):

    • includes fillable fields, casts, relationships
    • generates many scope/helper methods (e.g., “active now,” “get CTR,” “record impression,” plus other helpers)
    • described as producing more “useful for the future” methods than Sonnet

  • Sonnet (middle):

    • also includes fillables/casts/relationships
    • generates fewer methods, appearing to under-generate relative to GLM

  • Opus (right):

    • baseline similar: fillables/casts/relationships/scopes
    • over-delivers by generating enum classes
    • but produces fewer helper methods overall, making the model less feature-rich than GLM

Conclusion (models): GLM 5 is the clear winner, with a stronger helper/scope set than Sonnet and a more feature-rich model than Opus (despite Opus’s enum classes).

2) Factory Output (Fake Data + States)

  • GLM 5:

    • generates a larger, more granular factory
    • uses logic such as fake values and optional fields
    • produces many factory states (e.g., approved/pending/rejected/archived) and additional future-friendly state methods
    • considered to improve test realism

  • Sonnet:

    • includes logic/state variations, but with less “over-delivery” than GLM

  • Opus:

    • strong point: reuses enums (noted as good PHP practice)
    • has factory states, but fewer than GLM
    • cited as roughly 97 lines for Opus vs ~150 lines for GLM

Explanation for GLM slowness: GLM generates more code (more detailed states), which contributes to longer runtime.

3) Seeder Output (Using Factories for Realistic Test Data)

  • GLM 5:

    • described as granular and realistic
    • creates listings with mixed statuses
    • attaches random tags
    • creates explicitly approved, pending, and rejected records
    • uses the same helper/state logic from factories

  • Sonnet:

    • much more minimal
    • mainly creates listings and random tags
    • lacks granularity across statuses/use cases

  • Opus:

    • similar intent to GLM (approved/pending/etc.)
    • noted for being more readable/structured, using patterns like:
      • factory and then each over collections
      • reusable parts like users/categories
    • GLM uses separate statements instead, which may be more/less readable depending on Laravel experience

Overall Conclusions from the Test

  1. Long-running autonomy: GLM 5 is described as capable of completing long tasks without supervision if the prompt/spec is detailed enough. The speaker suggests they could step away and return after a few minutes to review results.

  2. Why Zeth.ai focuses on Opus: The speaker concludes GLM’s code output is more similar to Opus than to Sonnet, especially in overall type/structure.

  3. Community perception + pricing claims: Social media reactions described GLM 5 as a “killer,” emphasizing it is cheaper than Opus. The speaker references claims like “close to Opus/GPT but 10x cheaper,” while noting real costs depend on subscription/usage.

Guides / Setup Notes Mentioned

  • Practical setup: Use VS Code + a client extension + OpenRouter. Select GLM as the model and provide an OpenRouter API key.

  • Cost observation method: Compare the “quoted” prompt cost vs the actual OpenRouter activity cost.

Main Speakers / Sources Referenced

  • Main speaker: The YouTube creator running the GLM 5 vs Sonnet/Opus test (including references like “my teammate” and “I tested”).

  • Referenced sources/organizations:

    • Zeth.ai (benchmark claims and comparisons)
    • OpenRouter (model access and cost reporting)
    • AI Coding Daily (premium content mention)

Original video