Don't Buy a GPU: Build This 32GB Free AI Server Instead

Main Idea / Claim

• Shows how to build a free local “AI server” using approximately ~32GB total GPU RAM on Google Kaggle, leveraging two Tesla T4 GPUs (15GB + 15GB).
• Runs a quantized ~27B parameter “Qwen / Queen 3.8”-style model (referenced as Q4/Q1 and “uncensored”).
• Uses Ollama to host the model locally, then benchmarks generation speed (tokens/sec).
• Demonstrates connecting the local Ollama model to:
  • Open WebUI (Open Code) and
  • Hermes Agent

via an OpenAI-compatible endpoint exposed with Cloudflare Quick Tunnel.

---

Tutorial / Setup Steps (Technology + Product Features)

1) Create a Free GPU Account (CGL)

1. Sign up / sign in.
2. Complete phone verification, which is required to receive free GPU access.

2) Use a Provided Kaggle Notebook

• Import the author’s Kaggle notebook from the video description.
• Emphasizes running notebook cells top-to-bottom.

3) Enable GPU in Kaggle

• Kaggle settings → Accelerator → enable GPU T4 x2.

4) Install and Configure Ollama in the Notebook

• Installs required system packages.
• Installs Ollama (the local inference/runtime layer).
• Configures Ollama model storage location.
• Sets the target model to download and run.

5) Download and Run the Base 27B Model

• Ensures Ollama is installed.
• Downloads the specified Qwen / “Queen” 27B ~Q4/Q1 “uncensored” variant into Ollama.
• Confirms installation by checking model size (about ~17GB mentioned).
• Tests the model with a local request (e.g., “reply/think”) to confirm it responds.

6) Observe VRAM Usage

• Monitors GPU memory usage after loading.
• Example behavior mentioned:
  • One GPU might already show ~10GB used
  • Later stages might show ~9GB-ish
  • Exact numbers can vary by step.

7) Create an MTP-Enabled Ollama Model (No Re-download)

• Creates a new Ollama model definition based on the existing one, with MTP enabled.
• Explains MTP conceptually:
  • The model can predict multiple tokens during decoding
  • This may increase generation speed, depending on the workload
• Loads the MTP model into VRAM and uses keep-alive behavior so it stays loaded for future requests.

8) Verify MTP Model Loaded on Both GPUs

• Confirms via:
  • Ollama process info
  • GPU monitoring
• Notes that Ollama runs on both Tesla T4 GPUs.
• Example VRAM usage reported:
  • ~10.5GB and ~10.7GB per GPU (approx).

9) Benchmark Generation Speed

• Runs a fixed prompt test.
• Reported results:
  • Total tokens generated: 281
  • Generation speed: ~16.23 tokens/sec

---

Exposing the Local Server for External Apps

10) Make Ollama Accessible Outside Kaggle with Cloudflare Quick Tunnel

• Ollama is initially accessible only inside Kaggle.
• Uses Cloudflare Quick Tunnel to generate a temporary public HTTPS URL.
• Mentions using the tunnel’s OpenAI-compatible base URL so apps can call it like an OpenAI endpoint.

---

Integration Demos (Product/App Connectivity)

11) Connect to Open Code / Open WebUI

• In Open WebUI, add a custom OpenAI-compatible provider:
  • Provider ID: “Local LLM”
  • Base URL: the tunnel’s OpenAI-compatible endpoint
  • Model name/model ID: the MTP model name used in Ollama
• Sends test prompts and verifies responses work (and are fast after initial setup).

12) Set Up and Test Hermes Agent

• Installs/configures Hermes Agent locally.
• Adds a custom endpoint (OpenAI-compatible URL) with:
  • API type: “Olama”
  • Model autodetection and selection steps
• Runs the agent and verifies it answers queries using the locally hosted MTP model quickly.

---

Key Performance / Review-like Outputs

• Tokens per second benchmark (MTP model): ~16.23 tok/s
• VRAM usage: approximately ~10.5–10.7GB per Tesla T4 GPU during the loaded MTP phase.
• Reports the setup works for:
  • basic local inference, and
  • external-tool inference (Open Code / Hermes) through the tunnel.

---

Main Speakers / Sources

• Main speaker: the video narrator/host (not named in the subtitles)
• Technologies referenced:
  • CGL (free GPU provider for Kaggle setup)
  • Kaggle (GPU-enabled notebook environment)
  • Ollama (local model server/runtime)
  • Qwen / “Queen” 27B model (quantized variant, “uncensored”)
  • Cloudflare Quick Tunnel (public HTTPS / OpenAI-compatible endpoint)
  • Open Code / Open WebUI (custom OpenAI-compatible provider UI)
  • Hermes Agent (agent framework integrated via endpoint)