Video summary

Qwen 3.8 Flash Next + HERMES AGENT = AWESOME LOCAL AI AGENTS!

Main summary

Key takeaways

Technology

Overview

The video demonstrates how to set up local AI agents using Qwen 3.8 Flash Next (with vision/VLM support) in a Hermes agent-style workflow, with a focus on performance tuning for quad RTX 3090 systems. It also notes portability to other GPU configurations.

What’s being built / goal

  • The creator needs Qwen 3.8 Flash Next fast enough inside a Hermes agent to “produce things” the speaker cares about.
  • A step-by-step playbook and a copyable “recipe” are provided, including “twists” and tuning notes.
  • The tutorial also showcases an agent-assisted web game (a “farm abduction” shooter) that the model helps develop or iterate on.

Key technical setup parameters (VLLM run block highlights)

The speaker highlights several performance-critical configuration items:

  • Max model length: 131072
  • Max number of sequences: 2
  • Matchbox / token setting: 2048
  • GPU memory utilization: 0.96 (aggressive)
  • KV cache type: DT type auto for KV cache
  • CUDA graph mode: full
  • Prefix caching: enabled
  • Auto tool choice: enabled
  • Tool calling parser: Qwen 3XML
  • No async scheduling: explicitly set to off (important for stability/performance)

Hardware / resource requirements (recommendations)

The runtime depends on very large context/RAM loading.

  • System RAM (recommended): 128 GB
    • The speaker notes it may push near / under ~100 GB, with 128 GB treated as a hard target.
  • GPU VRAM (recommended): at least 96 GB for quad 3090 setups
  • Observed utilization:
    • CPU: ~40% with 16 CPUs
    • Memory: ~103 GB used (out of 192 GB available on their host)
    • GPU memory at load: ~22 GB per 3090
  • Monitoring: uses an NVT top / dashboards to track each GPU’s activity.

Performance / quality observations

  • The speaker compares token speeds to earlier setups (including a note that llama.cpp is slower for prompt processing).
  • Reported throughput/token rates include:
    • Prompt processing: roughly 399 → 516 tokens (as shown on screen)
    • Later runs reportedly around ~59.4 TG (and another around ~55 tokens/sec), emphasizing “better and better performance” and “going faster.”
  • Even with INT4 / Q4-ish quantization, the speaker claims output quality does not degrade much, supporting practical agent use.

Quantization / model choice notes

  • The system uses a vision-enabled INT4 quant.
  • A contributor is credited with describing it as “W4 A16” (with vision support).
  • The speaker emphasizes that vision support matters for the flash lineup and for the demonstrated agentic behavior.
  • They explicitly recommend the INT4 option for quad 3090 owners over an alternative “Llama” approach.

Stability / “twists” (important caveats)

The speaker warns that mismatched async scheduling settings/patches can:

  • reduce throughput (claimed difference of about ~10 tokens/sec), and
  • in some setups, eventually crash.

A workflow is provided that uses a script (e.g., serve-flash-next.sh) to construct the run block and run the VLM/Qwen vision use case.

Where the tutorial/resources are

  • digitalspaceport: the full post/requirements
  • arcade.digitalspaceport.com: the playable abduction game and prompts

The game prompts can be copied and reused, including:

  • a “one mega prompt” and
  • a secondary prompt.

Output / game demonstration (what the agent accomplishes)

The showcased project is a web-based 16-bit shooter involving:

  • abducting items/animals from farms
  • avoiding obstacles/enemies (e.g., birds, farmers)
  • difficulty and gameplay balancing (including troubleshooting when visual/gameplay behavior doesn’t match)

Key iteration approach:

  • It ties changes to recalling prior state from an “abductem game folder”
  • The agent launches and evaluates progress, then gives follow-up instructions to:
    • adjust gameplay difficulty and
    • fix visual/gameplay bugs

Noted troubleshooting examples:

  • beam/UFO alignment issues
  • debugging via visual processing + play rate

Broader conclusions (value of local AI agents)

The speaker argues local AI has reached “seriously useful local AI”:

  • not limited to “magician prompting,” but
  • enabling Hermes agents to assist alignment and produce better outcomes

They also tease:

  • Qwen 4 coming soon
  • Deepseek V4 flash with vision as another upcoming area
    • though they note supporting components may not be fully ready “down the stack” yet

Main speakers / sources (as stated)

  • Primary speaker: Allen (addresses “Allen, grab the lard…” and speaks throughout)
  • Referenced external contributors:
    • Loktar (X post; quad 3090 owner; claims ~198 tokens/sec)
    • Alexey Fateev
    • Super Alashsh
    • Viney AI (credited for the quantization used, “W4 A16” with vision support)
  • Referenced platforms/hosts:
    • digitalspaceport.com
    • arcade.digitalspaceport.com

Original video