Episode 04 : Tokens Explained | Free AI Engineer Course | 8 Weeks

Main ideas / lessons

• Tokens are “common reusable words” (and smaller word parts) that an LLM treats as known units.

  • The speaker compares this to a street-food vendor preparing common items ahead of time. When an unusual custom order comes (e.g., “chocolate egg Maggi”), the vendor can still assemble it using known ingredients, but won’t predict it as a “common” default.
  • In the same way, an LLM tokenizes input into units it has learned frequently enough to represent efficiently.

• Why tokens exist (problems with raw text → numbers):

  • Computers ultimately process numbers (0/1), so text must be converted into numeric representations.
  • Method 1: letter-by-letter to numbers is inefficient:
    • Produces huge inputs even for simple words.
    • Would make long messages (thousands of lines / lots of words) impractical.
    • Requires complex bookkeeping to preserve word boundaries and sequence information.
  • Method 2: word-by-word using a fixed dictionary (e.g., “Oxford Dictionary”) also fails:
    • Real people use names, slang, typos, new words, company names, and made-up/rare terms.
    • The model can’t assume every word used in prompts exists in a predefined dictionary.
    • So a fixed word list is not enough to cover language variability.
  • Because language is effectively “infinite” (names, spelling variants, combinations), a more flexible approach is needed.

• How tokenization is described:

  • The creator/scientists supposedly scan the internet and identify frequent fragments (common words and common sub-parts).
  • Common words (e.g., “the”, “in”, “walk”) are treated as single tokens; rare names like “Pratyush” are treated as non-common, so they may be split into smaller known parts (e.g., “Pra”, etc.).
  • Complex words are split progressively until the model recognizes pieces (analogy to how children learn and infer meaning from roots and suffixes like -ing).
  • A “token” is not necessarily a whole word or a fixed number of letters; it’s a sub-unit that the model already knows.

• Tokens determine cost and limits:

  • LLMs charge (or measure usage) by token counts:
    • Input tokens: tokens from the prompt.
    • Output/completion tokens: tokens from the model’s response.
    • Total tokens: input + output.
  • Developers can control generation length with max_tokens:
    • If max_tokens is small, the response may be cut mid-way.
    • The example shows checking whether the response finished naturally vs. was truncated.

• Implementation / debugging and coding workflow (from the “laptop/code” portion):

  • Warns about a naming mistake:
    • Creating a Python file named like token.py / token.dotpi can break Python because token-related names are reserved/used internally, causing environment/import errors.
  • Suggests renaming the file to something safe (e.g., using tokens.py or adding a suffix/prefix).
  • Emphasizes that tutorials are “perfect,” but real debugging happens; errors should be handled methodically.
  • Mentions using VS Code with extensions (integrations like Codex/Gemini/Grok) to reduce repeated copy-paste.

• Demonstration: multiple prompts in one run

  • Builds a list of prompts and loops through them to send multiple messages.
  • Collects and prints token usage for each interaction.

---

Methodology / instruction list (as presented)

A) Token concept computation (what to measure)

• When making an API call / LLM request:
  • Compute/inspect usage:
    • usage.prompt_tokens (named as “prep tokens” in the subtitles)
    • usage.completion_tokens (model response tokens)
    • Total tokens = prompt_tokens + completion_tokens

B) Multiple prompt execution (batching)

• Create three prompt messages/variables (conceptually):
  • Part 1: simple prompt (example response: “hi”)
  • Part 2: “explain time travel in detail”
  • Part 3: “write an essay in machine learning” with a length constraint (e.g., “1000 word” style in the explanation)
• Put them in a list, e.g.:
  • prompts = [prompt1, prompt2, prompt3]
• Loop over prompts:
  • For each prompt:
    • Send request to LLM
    • Capture response and token usage
    • Print token counts for prompt and completion

C) Limiting output length

• Use a parameter like:
  • max_tokens = 50
• After the response, check a finish reason / finish reason category:
  • If it ends naturally because the model completed its answer: finish reason indicates stop
  • If it’s cut because token limit was reached: finish reason indicates truncation/length-related finish state
• Increase max_tokens (e.g., 500 then 5000) to observe:
  • Responses end naturally when the limit is high enough.

D) Avoid reserved filenames in Python debugging

• Do not create a Python file named in a way that conflicts with reserved/internal modules (example: token.py).
• If an import/env/circular import error occurs:
  • Rename the file (e.g., tokens.py or token_<something>.py)
  • Re-run the environment.

---

Speakers / sources featured (as identifiable from the subtitles)

• Main speaker / instructor (unnamed; repeatedly addressed as “brother/children” and teaching coding + token concepts).
• AI/model mentioned as the source of responses: “ChatGPT / Grok” (the exact relationship varies by line; “Grok” is used in the code demonstration), but no external human source is explicitly cited.