Video summary

Школа AI агентов на локальных моделях, день #2: Публикация локальной LLM на своём сервере

Main summary

Key takeaways

Technology

Tech summary (AI agents on local models + server deployment, webinar day #2)

Overall workflow / lesson structure

  • They build two projects in parallel:
    1. Local model on a rented GPU server (for a children’s party salesperson assistant).
    2. OpenAI-based call quality control (for sales department call analytics).
  • Because GPU server provisioning takes time (unpredictable, ~30–40 minutes, sometimes up to ~1 hour), they:
    • start server provisioning first, then
    • run the other project on OpenAI while waiting.
  • They highlight a key “webinar time” constraint: without this ordering, they would waste time idle while the server starts.

Product/tooling approach: “Replit” + agent automation

They use Replit with an agent (referred to as “Replit/rapper” in subtitles) to:

  • Read a technical specification (TZ) written by the speaker (often authored via GPT),
  • Generate/assemble the application,
  • Connect to the rented server,
  • Configure UI elements and runtime behavior,
  • Run autotesting and iteratively improve the solution.

They describe Replit as a “clone/analogue of Google Studio,” while also emphasizing:

  • Replit can behave more autonomously, including server-side operations.
  • The main human work is architecture + correct specification, not writing every line of code.

Renting infrastructure (GPU cloud) + connecting to the server

  • They rent GPU instances from a provider mentioned as “Yers Cloud” / “Cloud.ru” (exact name may be misheard).
  • They discuss GPU options such as:
    • RTX 2080 Ti / 3080 / 4090 / 5090 / A100
  • Pricing is discussed in rubles/hour, with the note that exact figures may be noisy due to subtitles. Examples include:
    • “cheapest” 2080-class instances,
    • much more expensive A100/5090-class options.
  • They choose configuration based on:
    • Disk space (they mention needing ~50 GB+ for model/assets),
    • GPU RAM constraints (example: 2080 Ti with ~11 GB VRAM),
    • and overall cost vs feasibility.

Deployment specifics

  • They generate a key pair (public/private) for server access.
  • They provide the Replit agent with:
    • server IP/address + login + keys
  • They note that billing/charging occurs continuously while the server is provisioned.

Local LLM model deployment + “quantization”

They deploy a small local chat model (referred to as an “8B family,” with mixed subtitles such as ~2.5B / 7B / 8B).

Quantization

  • A core concept: quantization reduces model precision so the model fits in limited VRAM.
  • They claim a common quantization factor:
    • roughly 4× smaller (example described: ~16 GB → ~4 GB effective weights), allowing it to fit in ~11 GB VRAM.
  • Tradeoff:
    • quality may drop slightly (described as “a little dumber”) with increased risk of mistakes.
  • They test whether the app responses remain acceptable, including:
    • UI behavior,
    • and constraints like response language (Russian).

Project A: “Children’s party salesperson” (local model on rented GPU)

App behavior

  • The app is a chat-like sales assistant for children’s party services.
  • It uses:
    • A knowledge base (KB) describing party types, presenters/animators, program options, constraints by age/gender, etc.
    • A prompt that forces the assistant to:
      • ask clarifying questions (to understand client needs),
      • select an appropriate holiday/package,
      • present options and pricing,
      • then collect contact info (name + email/phone; a form appears in the UI).

Prompt design emphasis

  • Ask questions step-by-step (avoid one huge initial message).
  • Constrain output to Russian.

UI planning

  • Layout is described as:
    • left side: conversation/chat
    • right side: visual option selection (“pictures” for choosing plaques/options)

Replit agent responsibilities

  • Connect the agent to the server model,
  • Wire the knowledge base into prompts,
  • Generate UI and potentially backend components.

Project B: OpenAI-based “Sales call quality control”

App workflow

  • The user uploads/selects a call audio (or call file).
  • The system performs:
    1. Speech recognition (transcription),
    2. Personal data filtering/anonymization (mask names/identifiers),
    3. Call evaluation with rubric scoring,
    4. Output:
      • transcription,
      • per-criterion feedback,
      • final grade + recommendations.

Rubric / evaluation criteria

They use explicit criteria with a 0–10 scale per category:

  • Identification of needs (how well the manager understood the client)
  • Presentation (how well the product/school offer was presented)
  • Closing / handling objections (how effectively deal steps and objections were addressed)
  • Flexibility (how well the dialog adapts to client responses)

The output includes:

  • a 10-point scale per criterion,
  • and color gradations (green/yellow/red style) for readability.

They also mention additional measurable metrics:

  • time from sending to response (latency/response speed).

Model orchestration choices

  • They note that speech-to-text and evaluation can use different services/models; subtitles mention:
    • a Whisper-like speech recognition component,
    • and a GPT model for evaluation/judging.

Multimodality / speech pipeline discussion

  • They describe “real-time” conversation as a multi-stage pipeline:
    • speech recognition → LLM response → TTS generation
  • They explain why this approach is hard to make truly fast.
  • They contrast it with “true real-time” speech-to-speech models that can react with ~sub-second latency.
  • They claim top-quality real-time systems are typically available via advanced hosted providers; local real-time is harder.

Testing, model selection, and the “architect role”

They outline a method for choosing which model to deploy:

  • Run autotests: ~100 test questions per model.
  • Use a stronger judge model (mentioned as something like GPT-5.5) to grade outputs.
  • Compute:
    • average score,
    • average response time.
  • Select the best model considering cost and latency.

They frame responsibilities as:

  • humans provide architecture + rubric + specification,
  • the agent handles coding + testing + iteration.

They also mention that under heavy production load they’d need load testing, but for this setup:

  • Replit monitoring/agent iteration is used as a practical safeguard,
  • extreme scaling isn’t the focus.

Prompt engineering + “TZ” concept

  • Replit works best when you provide clear technical specifications (TZ) including:
    • logging requirements,
    • server setup behavior,
    • how prompts should be constructed,
    • how outputs must be formatted.
  • They mention GPT can generate detailed TZ quickly, saving time.
  • Sometimes the agent can rewrite/adjust prompts based on TZ.

Main speakers / sources (as best inferred)

  • Primary speaker: the webinar host (unnamed), referred to as “I/me” and giving step-by-step instructions.
  • Core software/agent sources mentioned:
    • Replit (agent that builds/deploys apps; “replit/rapper” in subtitles)
    • OpenAI (for call quality evaluation)
    • Whisper / speech recognition (STT approach)
    • GPT models (for analysis/judging and TZ generation)
    • GPU rental provider: “Yers Cloud” / “Cloud.ru” / similar (exact provider name may be misheard in subtitles)

Original video