Video summary

Opus 4.8 At Home? - Running Local LLMs for IoT Hackers

Main summary

Key takeaways

Technology

Goal / Question

Matt Brown tests whether local LLMs running on two NVIDIA DGX Spark devices can match the performance and capability of Opus 4.8 for IoT security research tasks, including:

  • reverse engineering
  • extracting firmware
  • vulnerability discovery

Hardware & Networking Setup

The setup uses two DGX Spark devices connected with:

  • Ultra-fast Ethernet to bridge model execution across devices
  • Standard Ethernet for SSH access

He emphasizes this is not a cost-saving approach:

  • power usage is high
  • power measurements weren’t the focus

Primary motivations are:

  • privacy
  • data sovereignty
  • self-hosting

Local LLM Hosting Approach

The deployment uses recipes (from an external lab/creator) that automate:

  • downloading model weights (typically from Hugging Face)
  • deploying models onto the DGX Spark cluster
  • running inference via an OpenAI-compatible interface

For serving, the model server runs vLLM on the Spark devices.

Model Options Tested (Tradeoffs)

Primary experiment model

  • DeepSeek V4 Flash (with Vision)
    • includes vision to interpret images (e.g., circuit board photos)
    • positioned as a practical balance of intelligence vs speed for his workflows

Other models discussed / configured on Spark

  • GLM 5.3 Flash (two-Spark)

    • appears slower, workload-dependent

  • DeepSeek V4.1 Flash (two-Spark)

    • described as more “squeezed/quantized”
    • he believes 3-Spark would be more optimal

  • Other mentioned options:

    • Quinn 3.8 / “flash next”
    • notes suggest performance is workload-dependent

Overall theme: more intelligence can be slower, and fitting larger models into two-device constraints may require heavier quantization/efficiency compromises.

Monitoring / Benchmarking (Grafana & Throughput)

He uses a Spark dashboard/toolbar showing cluster nodes:

  • Spark 1 = master node
  • Spark 2 = slave node

For performance measurement, he relies on Grafana to track token decoding throughput (tokens/second), with emphasis on:

  • how quickly the model outputs tokens
  • response performance during longer tasks

While he calls the benchmarks mostly practical/unscientific, they’re used to compare models against his real workflows.

Agent System & Tooling

Agent choice

He recommends Oh My Pi (OMP), described as a fork/improvement of a Pi coding agent:

  • preference for minimalism and control
  • avoids “too general” agent frameworks

Agent configuration

Includes:

  • a default web search order
  • optional authenticated search via Exa
    • though he notes local operation still works without internet

Skills / tools model

  • A skill is a markdown/YAML definition
  • it loads a tool description into the context window only when selected
  • this avoids context bloat from describing every tool upfront

Workflow 1: Firmware / Binary Analysis + Local Tool Use

This workflow extracts and analyzes firmware, for example:

  • extracting filesystem structures (example: SquashFS root filesystem)
  • locating binaries and target functions
  • performing decompilation/deconstruction steps

Key tool:

  • Binary Ninja headless via Python API
    • requires a paid Binary Ninja license

Reported result:

  • identifies a command injection-style vulnerability through decompiling and reasoning about functions

He notes the workflow is fully local.

Workflow 2: Android App + Reverse Engineering from Real Artifacts

In this task, the agent analyzes:

  • an APK
  • logcat messages (from ADB)
  • pcap Bluetooth capture files

Notes:

  • the challenge may fail to “load” due to missing network/conditions
  • the agent can still proceed using the provided artifacts

Tool/skill used:

  • Jadx for decompiling Java/Kotlin code

He reports:

  • decompiler errors at the method level are expected
  • nevertheless, the tool provides enough structure for analysis

Performance Results Compared to Opus 4.8 (Key Numbers)

Large IoT / Android-style challenge

  • Opus 4.8: ~15 minutes
  • DeepSeek V4.1 Flash (two Sparks): 1 hour 11 minutes
  • DeepSeek V2 Flash (favorite locally): ~40 minutes
    • can solve end-to-end (decompile + Bluetooth packet context)

Additional comparisons mentioned:

  • Same model via OpenRouter (cloud): ~22 minutes
  • GLM 5.3 Flash: ~1 hour
    • he’s “not a big fan” based on observed behavior
  • Quinn 3.8 Flash: ~1 hour 13 minutes
    • felt more “chaotic”
  • “Opus 4.0 at home” was slightly slower than expected

Conclusions / “Can you run Opus 4.8 at home?”

  • Local setups can get very close in outcomes (especially with the right model/config)
  • Fully matching Opus 4.8’s speed and behavior is difficult

Core tradeoff:

  • higher performance usually means paying more for more capable hardware

He also mentions:

  • hardware shortages/sellouts (e.g., NVIDIA site, Newegg, etc.)

Outlook:

  • he expects cutting-edge local capability to improve in ~6–9 months with the next hardware generation
  • he highlights the value for users who need private workflows that can’t—or shouldn’t—run on hosted providers

Main speakers / sources

  • Speaker: Matt Brown (described as an IoT pen tester / security researcher)

Referenced external sources

  • Mia AI Lab (X/GitHub)
    • provides recipes, comparisons, and deployment configurations
  • Hugging Face
    • hosts model weights
  • vLLM
    • serves OpenAI-compatible model APIs on the Spark cluster
  • Binary Ninja (headless)
    • decompilation tool used via Python API
  • Jadx
    • Android APK decompiler used via skills/tools
  • Exa
    • web search provider (optional in the agent config)
  • Grafana
    • dashboarding/monitoring for token decoding throughput

Original video