Video summary

Школа AI агентов на локальных моделях, день #1: AI агент с локальной LLM

Main summary

Key takeaways

Educational

Main ideas / lessons conveyed

  • The video is the Day #1 launch of a course about AI agents using local language models.
  • The instructor frames the whole school around building agents, with only partial emphasis on local model deployment:
    • ~70% agents
    • ~30% local LLMs
  • A course structure is presented to support both beginners and more experienced students, emphasizing:
    • short Q&A
    • increasing amounts of theory
    • hands-on creation of a working project/agent
  • The instructor introduces core concepts needed to build agents:
    • what an agent is (LLM-driven application/workflow)
    • the client–server architecture
    • the role of prompts, knowledge base/RAG, and backend logic

  • The lesson explains how to build a simple end-to-end system:

    1. Run a local LLM in Google Colab
    2. Expose it via FastAPI + Cloudflare Tunnel
    3. Build a frontend with Google Studio that calls the backend
    4. Iterate based on logs/errors (e.g., timeouts, token limits)

Course / schedule (organized as presented)

Overall format

  • 4 lessons total
  • Q&A early in the first lesson (10–15 minutes reserved)
  • Students build a project during the lesson; potential extension if time remains.

Lesson dates & goals

Wednesday — Lesson 1 (mainly basics + first build)

  • Theory
    • local models (high level)
    • agents (core definitions)
    • client-server interaction overview
  • Practical
    • build the first agent/project using:
      • Google Colab (run local model)
      • FastAPI (backend)
      • Cloudflare Tunnel (expose server)
      • Google Studio (frontend)
    • Example project theme: apartment renovation cost estimator
      • can ask clarifying questions, then produce estimate

Friday — Lesson 2 (more powerful deployment + more agents/projects)

  • Use a stronger setup:
    • rent server with a GPU
    • deploy local model via RepLit
  • Example use-case:
    • analyze sales calls / design sales agents (mentioned: children’s party sales)
  • Build two projects with the same general approach.

Monday & Wednesday — Lesson 3 (high-level / “purpose & business”)

  • Focus on “why” and using agents effectively for:
    • selling custom projects
    • subscription services
    • general agent-market / labor-market orientation
  • Mention of analyzing “~70 agents” and choosing use cases.

Bonus lesson

  • Speech and vision modules using local models:
    • speech recognition / speech generation
    • image recognition / image generation
  • Access given at the end of Lesson 3.

Methodology / instruction-style content (hands-on workflow)

A. What an “agent” consists of (conceptual breakdown)

  • Agent = AI-powered project/application
    • Examples given: salesperson bot, dialer/call-surveyor, house/custom constructor, call-center quality analyzer, ad/content generator, etc.
  • Agent core components
    • LLM / speech model (the “engine”)
      • can be OpenAI-like API models or local models (course focus)
    • Application layer (where it runs)
      • includes:
        • Frontend (user interface)
        • Backend (server logic, DB/logs, routing)
    • Prompts
      • many prompts may exist depending on the workflow:
        • instructions for role/behavior
        • quality control, call handling, style control, etc.
    • Knowledge base (optional but important for real quality)
      • described as RAG / retrieval augmentation
      • use company documents/regulations/materials
      • inject relevant context into the LLM

B. Application / platform architecture (client–server + API)

  • The system is described as:
    • Frontend: user inputs → sends request
    • Backend: Python service → calls LLM → returns response
  • Communication happens via API
    • “send request in a format to an endpoint; server responds in a specified format”
    • Mentioned HTTP methods:
      • GET: “give data”
      • POST: “send data / hold data”
  • Implementation approach:
    • FastAPI as the backend framework (Python)
    • backend exposes endpoints that the frontend calls

C. Local model exposure (practical pipeline described)

The instructor builds the first project manually for demonstration:

  1. Run local model in Google Colab
    • load a model (example: an ~8B parameter class model)
    • Colab provides GPU (e.g., Tesla T4 mentioned)
  2. Create a backend in Colab using FastAPI
    • define endpoints (e.g., chat/estimate)
    • backend loads/uses the local model on the GPU
  3. Expose Colab backend externally with Cloudflare Tunnel
    • obtain a public URL (“tunnel link”)
    • frontend uses this link as connection target
  4. Build a frontend in Google Studio
    • add:
      • a “connect” field/button for the tunnel link
      • input fields for project parameters
    • logic:
      • first send parameters → backend → model asks 3 clarifying questions
      • user answers
      • then send answers → backend → backend returns estimate
  5. Iterate using logs and adjust constraints
    • handle issues such as:
      • timeouts
      • generation being too slow
      • token limit too high (example fix: reduce max output tokens)
    • use server logging panel in Google Studio to debug request/response

D. Token / timeout mitigation presented during debugging

  • Problem observed: large generation request caused failure (timeout / too slow).
  • Fix method:
    • reduce generation size, e.g.:
      • reduce “max tokens” (example: 1000 → 300)
    • potentially split into multiple steps/requests:
      • preliminary estimation with one request
      • deeper calculation with subsequent requests

Local models: main concepts and comparative points

Why local models are used (advantages)

  • Confidentiality / privacy
    • data cannot be sent outside company boundaries
    • compliance needs mentioned (government/state/security constraints)
  • Reliability
    • external API services can fail intermittently; local deployment reduces dependency on external uptime
  • Speed
    • instructor notes local models are not always faster than API models; depends on hardware setup

Main “types/grades” of access to models (instructor’s typology)

From most “external/API” to most “local/closed”:

  1. Direct foreign hosted API models (GPT/Gemini/Claude-style)
    • pay per token/request
    • convenient, generally cheap per request
  2. Russian hosted API models
    • payment per tokens; potentially more privacy (still not fully local)
    • mentioned as weaker and often lagging behind by ~1–2 years
  3. Russian server-hosted model services (trusted deployment)
    • company-deployed GPUs in Russia; security still involves a third party
  4. Rent your own server with GPU
    • pay per hour/month
    • more privacy than shared services; cost increases quickly with usage volume
  5. Fully own hardware / closed loop in the office
    • maximum confidentiality and reliability
    • requires large capital expenditure on GPU hardware

Model size explanation

  • Local models differ by parameter count (e.g., 7B, 14B, 32B, 70B…).
  • General guidance:
    • larger models usually cost more (GPU memory, time)
    • quality is not always strictly linear due to architecture choices (e.g., MoE noted)
  • Practical course advice:
    • don’t rely on guesses; test multiple models for your use case
    • run your agent on several candidate models and choose based on analytics

Tools introduced (and how they’re used in the lesson)

  • Google Studio
    • frontend + agent UI (wipe-coding environment)
    • produces a working UI that can chat and call backend endpoints
    • can be designed with themes/layout
  • Google Colab
    • where Python runs and GPUs are provided (for demo)
    • used to deploy local model and run FastAPI backend
    • resources are temporary (instructor emphasizes it turns off)
  • FastAPI
    • Python library to build backend endpoints
  • Cloudflare Tunnel
    • networking tool to expose Colab backend publicly
  • RepLit
    • mentioned for Lesson 2 as an easier deployment option (compared with manual Colab + Tunnel setup)

Homework / submission instructions (explicitly described)

  • Register on the platform (mandatory)
  • Access
    • through a Telegram bot (alternative: contact support)

Homework for this course day (Lesson 1 theme)

  • Create your own local LLM consultant/agent based on the provided template in Colab
  • Deliverables:
    • Link to the Google Studio application (public)
    • Link to the Colab (template run / your project)
    • 3 screenshots (show it working / instructions)
    • short description of what your project does
    • submit via a Google Doc or document upload (PDF/Word also mentioned)

Timing

  • Homework due by Monday (in-class at 5:00 pm mentioned)
  • Participation note:
    • one raffle for those who come to Lesson 3
    • extra raffle for those who submit at least two homework assignments

Key speakers / sources featured

  • Dmitry Romanov — founder/owner of the “University of Artificial Intelligence”; main speaker/host of the lesson.

Original video