Run AI Models from your USB Flash Drive Fully Offline - Fully Private and Uncensored

Summary

• Goal / premise: Demonstrates running an LLM fully offline from a USB flash drive, emphasizing privacy and claiming the setup can be uncensored.

Model choice & compute impact

• Uses a 14B-parameter model (larger models generally require more compute).
• Advises selecting model size based on available hardware:
  • CPU/RAM vs GPU

Two executable modes (CPU vs GPU)

The project provides two launcher scripts:

• CPU-only launcher (.bat)
  • Slower, for systems without a GPU.
• GPU-enabled launcher (.bat)
  • Faster, when you have a suitable GPU.

Reported runtime expectations (examples):

• CPU:
  • Roughly 20–25 minutes to get going for the large model
  • First response latency: about 3–5 minutes, then faster follow-up responses
  • Token speed: around ~2.7 tokens/sec
• GPU:
  • Around ~8–10 tokens/sec
  • Faster overall generation
  • Example response time: about ~1–1.5 minutes

---

Install / Download (step-by-step)

1) Download llama.cpp / the “llamafile” binary

1. Search the browser for “llamafile”.
2. Open the first GitHub link.
3. Go to Releases.
4. Download the llamafile binary (~700MB+).
5. Copy the binary to the USB.

2) Download a GGUF model from Hugging Face

1. Go to Hugging Face → Models.
2. Filter for GGUF.
3. Search for a chosen model (example mentioned: “Queen 3”).
4. Download the selected GGUF model file.
5. Copy the GGUF file to the USB.
6. Ensure your CPU can handle it (or choose a smaller model if needed).

3) Prepare the USB to run locally (CPU)

1. Rename the llamafile binary to a preferred name, but ensure it ends with a .exe extension.
2. Create a CPU .bat file that includes:
   • the EXE name
   • the model filename with .GGUF
3. Edit the .bat commands using Notepad and ensure paths/names match your USB layout.

4) GPU support (GPU-enabled .exe + .bat)

• A separate GPU-enabled executable is provided via a link in the video description.
• Create another GPU .bat that uses:
  • the GPU-enabled .exe
  • the same GGUF model file

---

Running the model

• Double-click the appropriate .bat file to start the local server.
• The setup runs on localhost and opens a web UI in your browser.
• The UI includes settings such as:
  • the system prompt/message
  • options for jailbreak-style instructions

Note: The demonstrated model may not be inherently uncensored. The viewer can use an uncensored model and/or add jailbreak prompts.

---

Context length / output control (practical tuning)

Context window observations

• The demonstrated GPU run may have a limited context window.

Adjust generation length

Two ways are shown:

• Use the web UI Sampler setting to change max output
• Or modify the .bat command to include, for example:
  • --max-length 4096
  • (claimed to allow up to roughly a few thousand words / ~3,000 words)

KV cache behavior (general explanation)

• Increasing max lengths increases stored generated data in RAM (KV cache).
• Setting extremely high values (e.g., 16,000) can lead to RAM overuse and crashes.

Resuming generation

• If output stops early, the user can type continue and press Enter.

---

Example use case

• The demo prompts the model to generate a Python program that scans open ports on an IP address (to show offline capability plus generation control).

---

Main speakers / sources

• Speaker: The video’s demonstrator/author (unnamed in the subtitles).
• Primary sources referenced/downloads:
  • GitHub Releases: llamafile repository/binary
  • Hugging Face: Models page filtered to GGUF models