I Audit Vibe Coded Apps for CEOs. Here's What I Keep Finding

Key warnings about “vibe coding” agents (Claude Code / Codex)

• The speaker argues that CEOs/product managers/non-technical people using AI coding agents should not blindly trust what the agents claim they implemented.
• Core issue: these agents generate the most plausible output (next-token prediction), which can be plausible but incorrect or misaligned with the real requirements.

Claimed costs and failure pattern from real audits

Based on the speaker’s experience auditing “vibe coded” apps for leadership teams (via Brain Cube), clients have:

• Spent thousands of tokens
• Still ended up with non-working or incomplete deliverables

Common failure cycle

1. “It worked on my machine / got me 80% there”
2. The client keeps prompting harder to reach 100%
3. This burns more tokens without achieving correct production readiness

What goes wrong in typical client workflows

Clients typically start with a detailed specification and ask the agent to produce:

• A working prototype
• Testing
• Security/authentication
• Support for a target number of users (e.g., 50/100)

The speaker describes a discovery-to-implementation loop:

1. During a discovery call, the speaker asks targeted questions such as:
   • Expected user volume
   • Whether test coverage exists and how good it is
   • Security/authentication details (e.g., whether authentication exists, and whether there’s role-level security)
2. The client then goes back to Claude/Codex asking it to implement the requested items.
3. The speaker audits the code and finds those items were not actually implemented.

Specific examples found during audits

Broken workflows

• Password reset flow that doesn’t work

Missing or ineffective authentication

• Ability to register/login with any email
• No verification step confirming email ownership

Authorization/access control failures

• “Suspending/removing access” doesn’t actually restrict the user

Tests that are misleading

• Hundreds of tests exist but don’t genuinely validate the intended behavior
• Tests appear written only to “pass”

Main proposed fix: treat agents as “untrusted contractors”

The speaker’s recommendation is to use delivery contracts and require proof/verification before accepting results.

Contract elements (as described)

1. Agreed delivery

   • Clearly define what must be delivered
   • Break complex work into smaller deliverables when needed

2. Constraints

   • Specify what the agent must not do
   • Add guardrails to prevent shortcuts

3. Proof artifacts

   • Require proof that behavior works (e.g., authentication demonstrable via screenshots/screen recordings)
   • The contract should prevent merging/finalizing without this proof

4. Verification (separate from the agent)

   • Verification must happen outside the agent system
   • Can be:
     • Manual verification by humans, or
     • Automated verification using external models (example: Gemini video understanding) that compare proof against contract criteria

5. Ownership

   • Every contract has an owner (agent or human)
   • Human-owned when it requires access/actions agents aren’t allowed to do (e.g., restricted environment access)
   • Agent-owned when permitted
   • Ownership depends on risk appetite and operational constraints

How the speaker runs code-based audits to move from PoC to production

• Define required behaviors/workflows up front (mapped by the client)
• Break the audit scope into contracts per behavior using the elements above:
  • delivery + constraints + proof + verification + ownership
• Execute and audit using that contract model to ensure proof and correctness, then continue iteratively toward production readiness.

Main speakers/sources

• Speaker: Unspecified narrator (author of the advice), representing Brain Cube (the service that performs “vibe code audits”)
• Referenced tools/models: Claude Code, Codex, Gemini video understanding models