Summary of "Don't Make This Mistake: Painful Learnings of Applying AI in Security"

Summary of “Don’t Make This Mistake: Painful Learnings of Applying AI in Security“

Security efforts often slow down business and development because fixing vulnerabilities is time-consuming and resource-intensive.
The vulnerability remediation backlog is huge; fixing a single vulnerability can take 5-7 hours, which translates to thousands of hours for companies with many vulnerabilities.
Developers generally dislike vulnerability scanning and remediation tasks since these activities do not contribute positively to their professional growth or business impact.

Help developers fix security vulnerabilities efficiently without slowing down feature development.
Minimize mean time to remediate (MTTR) vulnerabilities.
Enable fixing vulnerabilities at scale.

Simple prompts to fix vulnerabilities (e.g., XSS in JavaScript) sometimes worked but often failed due to lack of context or incorrect assumptions.
AI-generated fixes had mixed results:
- About 29% were good fixes that resolved issues.
- 90% sometimes introduced new vulnerabilities or only partially fixed problems.
- 52% were outright bad, introducing broken code or irrelevant changes.
- Many fixes were generic templates requiring manual implementation.
AI hallucination increased with more input data, counterintuitively causing less reliable fixes.
AI often produced persuasive but incomplete or incorrect fixes (e.g., improper escaping for header manipulation, partial encryption fixes).
Blindly applying AI fixes at scale is risky due to false positives and incomplete context.

AI lacks full context of the application and codebase, leading to partial or incorrect fixes.
Fixes often affect only one part of the code, ignoring dependencies and integration points.
AI-generated fixes sometimes break application logic or do not fully resolve the vulnerability.
Managing false positives from static analysis tools (SAST) complicates automated remediation.
Large-scale automated remediation requires precise, context-aware, and validated fixes.

Combine AI with traditional static analysis and code parsing techniques.
Use Abstract Syntax Trees (AST) to parse and understand code structure precisely.
Provide AI with very focused, custom prompts targeting specific vulnerabilities and code patterns.
Validate AI output programmatically and integrate fixes into templates or algorithms to maintain consistency.
This approach reduces hallucinations and improves repeatability and scalability of fixes.
Still requires human supervision and validation before merging fixes.

Tested AI fixes on known vulnerable apps (WebGoat, Juice Shop) and commercial SaaS tools.
Examples of AI tools tested include ChatGPT (versions 3, 3.5, 4), GitHub Advanced Security’s automatic remediation.
GitHub claims about 90% of findings can be fixed automatically, but fixes still require review.
Discussion on emerging MCP (Model Control Plane) workflows that validate vulnerabilities through live testing before generating fixes and pull requests.
Emphasis on the need to prove vulnerabilities exist before fixing them automatically.

AI tools are like eager junior developers: they try to help but require oversight.
Companies forbidding AI use may fall behind competitively.
Vibe coding (AI-assisted coding from product specs) is gaining traction in B2B software companies.
AI can save development time but never fully replaces human judgment.
Security teams must take responsibility to verify and fix vulnerabilities thoroughly.
Zero trust and continuous verification remain essential principles.

Limit input context to avoid overwhelming the AI.
Separate system prompts (trusted instructions) from user prompts to prevent prompt injection.
Use stepwise prompting or retrieval-augmented generation (RAG) methods for complex fixes.

Primary Speaker: Atan (co-founder of Mob, former IBM application security engineer)
Other Contributors:
- Kevin (referenced but details incomplete)