Video summary

I Built My Own LLM Completely From Scratch (for pirates)

Main summary

Key takeaways

Technology

Overview

  • The creator builds an LLM from scratch using only their own training pipeline/data (no Codex/Claude-style coding assistants; mostly “copy-paste” only).
  • Goal: train a tiny GPT-like model that speaks in pirate language.
  • Cost/time claim: about $0.70 GPU cost and roughly 3 days of work (mostly learning + implementation).

Day 1: Architecture & Training Methodology

Learning and references

  • Studies LLM construction at a high level.
  • Mentions Claude as a study reference.

Core building blocks

  • Tokenization / vocabulary building
    • Maps text to token IDs.
    • Emphasizes that vocabulary size cannot change later without retraining.
  • Pretraining objective
    • The neural network predicts the probability of the next token given previous tokens.
  • Instruction tuning / behavior alignment
    • Later “tuning on instructions” uses a smaller learning rate and a smaller curated data subset so the base model isn’t disrupted too much.

Product/repo practices

  • Implements a GPT-like model inspired by nanoGPT / Andrej Karpathy-style approaches.
  • Publishes progress by:
    • Uploading to GitHub
    • Saving checkpoints
    • Pushing checkpoints to Hugging Face
    • Using Weights & Biases (W&B) for training monitoring

Day 2: Data Pipeline & Pirate Transformation

Data strategy (two options considered)

  1. Find pirate-language text datasets directly.
  2. Train on English, then instruction-tune into pirate speech.

Chosen approach

  • Uses an “R” Python library (rule-based) to convert/transform English text into pirate speak.
  • Starts from TinyStories (~2M tiny English stories), then pirate-augments the dataset.

Tokenizer / vocabulary choices

  • Uses BPE via the Hugging Face tokenizer library.
  • Builds a tokenizer with about 8,000 tokens.
  • After transforming/encoding:
    • Reports ~500M tokens
    • Saves the dataset as .bin files

Hardware constraints

  • Debugging/troubleshooting: trains on an M1 MacBook using MPS/CPU.
  • Full training: planned on a rented GPU server.

Training Loop Details (GPT-like)

  • Conceptual flow:
    • Token IDs → embeddings (mentions example embedding dimensionalities)
    • Uses transformer blocks (mentions 6 blocks)
    • Outputs a probability distribution over the full vocabulary (about 8,000 tokens), including an end-of-sentence token
  • Generation behavior:
    • Iteratively predicts the next token, appends it, and repeats.

Training execution

  • Runs on Vast AI.
  • Workflow:
    • Periodically uploads training artifacts/checkpoints to Hugging Face
    • Uses W&B to monitor loss decreasing (mini-batch and overall loss)
  • Suspicion:
    • Training “didn’t take long,” suggesting limited model quality.

Evaluation After Pretraining

Qualitative checks

  • Generates samples that resemble pirate outputs (e.g., starting with “Ahoy…”).

Limitations observed

  • Behaves more like an autocomplete than a true knowledge model.
  • Example failure:
    • Prompting “the capital of France is …” doesn’t work—reasonable since training used TinyStories, not factual sources like Wikipedia.

Next planned stage

  • Move toward a chatbot/instruction-following model.

Fine-tuning / SFT (Instruction Tuning Stage)

SFT dataset

  • Finds a GitHub Q&A dataset with ~15,000 question-answer pairs.
  • The answers already contain pirate speak.
  • References “Dolly”-style data.

SFT procedure

  • Performs SFT (supervised fine-tuning) on the base model:
    • Retokenizes using the same dictionary/tokenizer for consistency
    • Uses a small learning rate to avoid “disturbing” the base model
  • Training details:
    • Runs on Vast AI
    • Takes about 1–1.5 hours
  • Saves checkpoints and deploys for testing via Hugging Face.

Post-SFT Testing & Behavior

  • Tests prompts like: “How are you doing?”
    • Output is pirate-themed and somewhat conversational, but not perfectly aligned.
  • Tests: “Tell me a story, matey”
    • Model tends to reuse pirate vocabulary (e.g., “booty”)
    • Suggests limited instruction depth/coverage due to small-scale training

Packaging the “Finished Product”

Naming and branding

  • Chooses the model name “Nano Beard” (Blackbeard reference).
  • Creates a pirate-themed pixel art logo, explicitly trying not to rely on ChatGPT for the mascot.

Deployment

  • Wraps the model in a Hugging Face Space using Gradio for a simple chat UI.
  • Uses CPU deployment since the model is small.

Main Speakers / Sources

  • Main speaker: The video creator (single author/narrator describing their own build process)
  • Referenced sources/approaches:
    • Claude (for studying LLM architecture/training concepts)
    • nanoGPT / Andrej Karpathy (GPT implementation reference)
    • Hugging Face (tokenizer, checkpoint hosting, Spaces)
    • Vast AI (rented GPU training)
    • Weights & Biases (W&B) (training monitoring)
    • TinyStories (pretraining data)
    • “R” Python library (rule-based pirate conversion)
    • Dolly-style Q&A dataset (SFT instruction/chat data)

Original video