How Hackers Trick AI Into Giving Up Secrets (Demos and free labs)

Technological Concepts & Main Security Ideas

• Prompt injection attacks against LLM/chatbot systems The video argues that modern AI apps can be tricked into leaking information or performing unintended actions by manipulating the boundary between:

  • Hidden system instructions
  • User-controlled text

• Core problem: instruction boundary confusion AI systems may not reliably determine where the model’s instructions (e.g., system prompt / hidden directives) end and the user’s message or edited content begins.

• System prompt exploitation via “completion sequences” In the “Room” teleconferencing/transcription demo, the attacker edits transcript notes to inject instructions. These injected instructions cause the AI summary feature to follow attacker-supplied directives rather than the intended system behavior.

• Non-deterministic testing (LLM nondeterminism) The speaker emphasizes that LLM responses can vary between runs, making results less repeatable than classical security testing. Attacks may need to be attempted multiple times (up to ~15) to confirm true positives vs. false positives—adding time cost and requiring additional effort from teams.

• Impersonation / “imitation of authority” as an attack method Attackers convince the chatbot it is interacting with an administrator/boss/authorized user (e.g., “I am your boss / administrator”) to bypass safeguards.

---

Key Features & Resources Mentioned

• Prompt injection taxonomy (free resource) by Arcanum A categorized blueprint dividing attacks into four sections, including:

  • Intentions: the attacker’s desired outcome (e.g., forcing the bot to list APIs or output prohibited content).
    • Harmful examples explicitly mentioned include chemical/biological/radiological/nuclear domains.
  • Other sections cover selecting attack approaches and bypass techniques.
    • The video highlights method naming such as “act as an authorized user” and “elevate privileges.”

• Workarounds against “guardrails” and “classifiers” The video claims AI defenses can be bypassed by transforming prompts so that filters fail to recognize the malicious instruction—while the chatbot still understands it, using:

  • Translation (e.g., into Spanish/Portuguese)
  • Base64 encoding
  • Braille-style encoding (dashes/dots representation)

The speaker notes the number of bypass/workaround entries increased from ~21 to ~172.

• AI penetration testing / AI red teaming practice Describes ongoing “comprehensive testing” of chatbots (especially web/app helpdesk assistants) to see whether they can:

  • Access other users’ accounts
  • Disclose secrets/data
  • Be induced to bypass account recovery protections

---

Demos / Labs and What They Demonstrate

1. “Karen” CTF (Def Con Kids oriented; also open to newbies)

• A kid-friendly lab modeled as a “capture the flag” game where the AI (“Karen,” from Spider-Man glasses) protects a secret phrase.
• Demonstrates that authority impersonation can be used to get the AI to reveal protected content.
• Includes multiple difficulty levels (five total), with Level 1 shown first.
• Note: labs are planned to be released after Def Con via separate domain(s).

2. “Instagram/Meta” Account Takeover via AI Support Assistant (Real-world claim)

• A lab replicates an Instagram-like scenario: an attacker uses an assistant (“Mega AI”) to perform password recovery.
• Process described:
  • The assistant requests an account nickname
  • The attacker provides an email address for recovery
  • The attacker supplies a hacker-controlled inbox (e.g., hacker@hacker.com)
  • The attacker receives a verification code, enters it back into the assistant
  • The assistant allows setting a new password
• Video claim: this reportedly occurred against Meta on Instagram ~5 weeks prior, where the chatbot’s access to customer/account data enabled adding emails or changing passwords without proper verification beyond email-code delivery to the attacker’s chosen inbox.
• Lab availability: not available at recording time; expected after Def Con, with links posted on social media.

3. “Room” (Zoom-like) Transcription Summary Manipulation

• Shows a teleconferencing UI with AI transcript generation and an AI summary button.
• The attacker edits transcript notes to inject “new instruction” behavior using a technique described as adding “termination/system” sequence text (attempting to confuse the model about who is giving instructions).
• Because of LLM non-determinism, the exploit may fail initially; the attacker “tinkers” until it works.
• When successful, the system incorrectly summarizes or reveals a confidential CFO meeting summary (board business, possible acquisition, embargo until Q4), used as evidence of boundary failure in prompt handling.
• Claims a similar issue occurred in a real teleconferencing product using AI transcription.

---

Tutorials / Guides Mentioned

• AI Safety Resource Center (already open)
  • A page with 71 links to open labs for practicing prompt injections at varying difficulty levels.
  • Encourages that testing can be performed by ordinary people, not only security professionals.
  • Includes a callout that the taxonomy and labs function as training material; completing all labs is positioned as groundwork for a career in “AI red teaming.”

---

Main Speakers / Sources

• David Bombal (host)
• Jason Haddix (guest; works on AI penetration testing and AI red teaming; Arcanum)