From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

Technological Concepts & Product Features

On-device “agents” and skill execution (Gemma 4-based)

Google is building on-device agent capabilities using a “skill harness” layered on top of AI Core.

• How skills are invoked

  • The model is prompted with skill descriptions.
  • When needed, it selectively loads a specific skill on demand (via a tool/load-skill mechanism).
  • Example: choosing a maps skill when a user asks for an office location.

• Skills can include executable UI/interaction code

  • Skills may contain JavaScript that the app calls to render or handle interactive elements.
  • Example: a custom restaurant “roulette” rendering.

• Developer-created skills

  • Developers can create and publish their own skills, including:
    • Loading from a URL (e.g., GitHub-hosted skills).
    • Sharing via the community, with an easy “load skill” flow in the app.

• Skills used to write skills (self-referential workflow)

  • Mentions using Gemini CLI / code coach.
  • Example scale claim: about 80 skills created.

• Community + official debugging/testing

  • Example: Gemini CLI ADB skill to test with a connected device via ADB.

• Known limitation being worked on

  • The system works well for multiple skill calls across multiple conversation turns.
  • Harder problem: reliably performing multiple skill calls within a single user interaction/response (the model sometimes struggles to chain skills robustly in one turn).

---

“System GenAI” vs. developer-customized options

Google distinguishes between:

• System-level GenAI

  • Pre-installed on-device (e.g., Gemini Nano via AI Core).
  • Optimized for minimal app size impact.

• App GenAI

  • Developer-controlled.
  • Built using LiteRT + LLM runtime.
  • Enables more customization and “boutique tasks,” but requires more developer work.

---

Tiny LLMs (“TLMs”) for on-device use

• Definition: LLMs with < 1B parameters, designed to be embedded into apps.

• Why run locally: latency, privacy, offline use, reliability, and cost savings.

• Core on-device ML stack mentioned

  • MediaPipe
  • LiteRT / LiteRT TLM: runtime for running LLMs in packaged formats (also used with MediaPipe)
  • Runs on CPU, GPU, or NPU, depending on platform.

• Scale / trust claim

  • LiteRT (and TF Lite-like runtime) is built into Android OS.
  • Reported as widely used across 2.7B+ devices.

---

LiteRT TLM model format

• What it is: a single-file packaging format for models, described as bundling the essentials needed to run them.
• Conceptual comparison: positioned as a simpler packaging alternative to .tflite/.task-like formats.
• Important nuance: task files can include non-LLM code as well (e.g., Face Mesh).

---

Guides / Tutorials / Apps Mentioned

Google AI Edge Gallery (open source)

A cross-platform iOS + Android app for running LLMs locally.

• Uses AI Core when available to provide Gemma models to the app.
• Demonstrates:
  • Tiny LLMs and Gemma 4 performance
  • AI chat, image understanding, audio transcription, and example models
  • 3rd-party model support (examples referenced as “q…/phi” style families)
• Open source code:
  • Android code available; built with LiteRT
  • iOS / Swift API mentioned as forthcoming

---

Skills walkthrough demo

A restaurant-related skill demo shows how a simple agent harness + Gemma 4 can power task flows like “select a restaurant” with minimal code/prompt work.

---

Fine-tuning workflow for tiny models (aiming for reliable function calling)

Guidance emphasizes that for small models (roughly 100–200M parameters):

• They often need a narrow, focused task.
• They benefit from fine-tuning using synthetic data (rather than relying purely on prompting).

Export / inference / deployment flow

• Export & quantize (example: quantized Q0.6)
• Test on desktop using LiteRT runtime (example mentions GPU)
• Deploy to target hardware (example mentions Qualcomm NPU acceleration)

Named model and deployment examples

• Fast VLM: ~500M parameters, optimized for Qualcomm NPU
• Function calling model (“Function Gemma”)
  • Based on Gemma 3 tech
  • Size: ~270M parameters
  • Performance claim on Pixel 7:
    • ~2000 tokens/s prefill
    • ~140 tokens/s decode
  • References:
    • YouTube video: “Function Gemma”
    • Function Gemma fine-tuning lab (Hugging Face space) where users define functions, upload data, and fine-tune

---

Synthetic data + fine-tuning results (review/analysis-style claims)

• Experiment approach: synthetic datasets built from app functions (not system prompts), then fine-tuning.
• Reported results:
  • Baseline out-of-the-box success rate for Function Gemma: about ~40–60% (vague subtitle like “40 6% or something”).
  • After fine-tuning: over 90% success for 8/10 functions (with 2 functions lower, but still improved).
• Takeaway: more work than prompting large models, but enables reliable app-scale behavior.

---

Eloquent app example (transcription)

Another app example called Eloquent:

• What it does: offline transcription with personalization using a personal dictionary and keyword handling (e.g., proper nouns and technical jargon).
• Pipeline described:
  • ASR engine (Gemma 3-based)
  • Text polishing engine (Gemma 3-based) to remove fillers like “ums”/“ahs”
• Availability constraint: iOS only and not in Europe yet.

---

Key Quantitative / Benchmark-Style Points

• On-device runtime scale: LiteRT trusted on Android with 2.7B+ devices and high daily invocation counts.
• Function Gemma performance (Pixel 7):
  • ~2000 tokens/s prefill
  • ~140 tokens/s decode
• Fine-tuning improvement (app function calling):
  • Out-of-box: vaguely stated mid-40% to mid-50% success
  • After fine-tuning: >90% success for 8 out of 10 functions
• Skills behavior claim:
  • For a ~4B parameter model, default enabled 8 skills
  • It can select between them reasonably well within a conversation
  • Still being improved: chaining multiple skills in one interaction

---

Main Speakers / Sources

• Cormac Brick (Google) — primary speaker (video title indicates him)
• Mentions Oli / Florina (“Ollie and Florina”) as prior speakers who covered parts of the topic
• Mentions Wei (colleague providing performance data talk tomorrow)
• Mentions Gemma team / DeepMind team collaboration on function calling model and experiments