A 32GB GPU for a Third of the Price? Intel's B70 for Local AI

Overview: Local AI GPU Setup (Intel B70 / ASRock, 32GB)

• The creator describes their setup as (by 2026 standards) one of the least expensive 32GB GPU options they could find.
• The system runs fully locally using Quen 3.8, meaning:
  • Prompts and generation do not require sending tokens to cloud services like ChatGPT/Gemini.

Key Performance / Quality Claims with Quen 3.8

• Large context handling

  • The creator claims they can fit the model + substantial context length entirely on the GPU.
  • This is presented as avoiding reliance on RAM/CPU for context.

• Token generation speed

  • Reported rate: ~20–25 tokens/second.
  • Even under heavier context loads, they say it “doesn’t usually” drop below ~20 tokens/second.

• Model type / size

  • Described as a dense model (not mixture-of-experts).
  • Stated size: 26B parameters.

Demo #1: Local Litigation / Case Summarization (DJI case)

• A court case was loaded locally and summarized based on a recent appeals court ruling.
• The model generated structured summaries, including:
  • What points DJI won or lost
  • A chart to follow
• Follow-up questioning was used to test behavior with large context.

Performance vs. Nvidia (Tradeoff)

• The creator acknowledges Nvidia can be faster.
• Their argument: the local setup is “fast enough” for most personal tasks and represents a cost/value sweet spot.
• They also note there are cheaper B70 variants with the same 32GB RAM, but lower compute (B60/B65 mentioned), likely resulting in slower token generation.

Hardware Integration: Power and Noise

• GPU connectivity:
  • Used via PCIe or an Oculink connection (Oculink dock + mini PC).
• OS / platform:
  • Ubuntu 26.04 on a Minis Forum mini PC.
  • The subtitles mention an AMD “stricks” processor.

Power draw and noise observations

• The fan is reported as loud even at idle.
• Power draw noted (excluding the mini PC):
  • ~55W idle when the GPU is loaded / sitting idle.
• Expected impact:
  • Higher overall power usage compared with low-power SBCs (e.g., Raspberry Pi comparison).
• Long-term goal:
  • The creator expresses a desire to solve the power issue for potential 24/7 use.

Optimization / Configuration Workflow

• Initial performance was reported as slow, but the creator says software updates produced significant improvements.
• For Linux/config tweaks, they used an AI coding assistant workflow with a command-line tool:
  • Mentions “Codeex” and “Open AI soul model” credits (as referenced in subtitles).
• Workflow strategy:
  • Use an AI dev partner to iterate faster.
  • Have the AI output a report/log of changes so configurations can be recreated if the OS is wiped.

Demo #2: Local Coding + “Space Invaders” Clones

• The creator tests multiple local models:
  • A Quen mixture-of-experts model on a 16GB card
  • A Gemma 4 dense model (stated 31B parameters)
• Claim:
  • Quen 3.8 produced the best-looking game graphics among the local runs.
• Note:
  • Gameplay/shooting was still described as needing work.

Demo #3: “Gadget Tracker” Automation (RSS + Local Classification)

• A local app workflow:
  • Reads an RSS feed from tech publications.
  • Filters items by using the local Quen model to determine whether an article is about a gadget.
• Storage and refresh loop:
  • Uses a SQLite database (subtitles refer to “SQL light”).
  • Refreshes every 15 minutes:
    • Pull new RSS items
    • Classify them
    • Store results
• UI flow:
  • Lets the user open relevant articles or delete irrelevant ones.
• Extending the project with coding harness:
  • Uses Open Code (command-line coding harness) to add social media images to gadget cards.
  • The update was run locally; the creator watched generation and reports it largely worked on the first attempt.
  • A follow-up prompt fixed a UI issue related to X/delete button visibility.

Demo #4: Web Scraping to Generate a CSV (NYC Council Members)

• Example task:
  • Create a spreadsheet of NYC council member names + email addresses.
• Challenge described:
  • Emails are hidden behind links and are not easily copy/paste.
• Approach:
  • The model fetches the webpage (curl mentioned in subtitles).
  • It finds email addresses in the HTML source.
  • It generates a CSV file saved to local storage.
• Timing and verification:
  • End-to-end time reported as under ~2 minutes.
  • The creator copied the file and spot-checked it as correct.

Main Speakers / Sources

• Lon Cyon (creator speaking throughout)