Qwen 3.5 Small Models Are INCREDIBLE! (Testing 0.8B & 2B On Edge Devices)

Concise summary — focus on tech, features, tests, and practical guide

This is a compact overview of Alibaba’s Qwen 3.5 “small” models (0.8B and 2B), benchmark results, hands‑on tests (desktop and on‑device), practical deployment notes, limitations, and broader significance demonstrated in a hands‑on video/talk.

What Qwen 3.5 small-series are

• Alibaba released native multimodal Qwen 3.5 “small” models: 0.8B and 2B parameters.
• Unified multimodal architecture combining text, vision, and coding capabilities in very small models.
• Very large context window: 262k tokens (can ingest whole PDFs or large codebases).
• Emphasis on “intelligence density”: compressed capabilities that can rival much larger models.

Key benchmarks

• MMLU (general knowledge/reasoning)
  • 2B → 66.5
  • 0.8B → 42.3
  • (Reference: Llama 2 7B ≈ 45.3)
• OCR Bench (vision/OCR)
  • 2B → 85.4
  • 0.8B → 79.1
• Practical implication: strong multimodal/OCR performance relative to parameter count.

Hands-on tests (setup & tools)

• Desktop / server:
  • Host: LM Studio (local inference server).
  • Models: download GGUF versions of 0.8B and 2B, increase context window.
  • Client: Klein in VS Code pointed at the LM Studio server (tested fully offline — airplane mode).
• Mobile / on-device:
  • Native iOS app built in Swift using MLX Swift (Apple’s open-source Metal GPU library) to run models on Apple silicon offline.
  • Repository link shown in the video (not included here).

Practical coding test (offline)

Task: generate a simple café website using only HTML/CSS/JS (no external frameworks to keep it offline).

• 0.8B model
  • Time: ~1 minute to produce code.
  • Output: bland UI, hardcoded Unsplash image URLs (some invalid), struggled with iterative fixes.
• 2B model
  • Time: ~3 minutes; prefers planning before coding.
  • Output: cleaner design (brown theme), attempted cart sidebar but missing add-to-cart UI.
  • Issues: repeated looping/respawning behavior (infinite-loop generation) observed — possibly integration issue (LM Studio/Klein) rather than model-only.
• Takeaway: capable of meaningful prototypes but not reliable for serious/complex production coding.

On-device multimodal tests (iPhone 14 Pro, offline)

• Responsiveness: very fast streamed responses on-device.
• Logical/knowledge test (“car wash” test): both 0.8B and 2B passed.
• Vision tests:
  • Banana image: 0.8B produced odd/incorrect labels (“dog banana”, overripe); 2B more accurate (“fully ripe”).
  • Dog breed: both struggled (0.8B miscounted/didn’t identify breed; 2B incorrectly guessed Pomeranian).
  • OCR / language detection: 0.8B failed to identify Latvian and hallucinated translations; 2B correctly identified Latvian.
• Overall: impressively fast and capable for on‑device multimodal inference, but still prone to hallucinations and inaccurate fine‑grained vision classification.

Practical deployment notes / how-to checklist

• Download GGUF model files (0.8B / 2B).
• Use LM Studio to host a local inference server; set high context length when needed.
• Point the client (Klein / VS Code) to the LM Studio server URL.
• For iOS on Apple silicon:
  • Use MLX Swift for Metal‑accelerated inference on‑device.
  • Build a native app that downloads the GGUF model and runs in offline mode.
• Expect fast inference but:
  • Test for generation loops and instability.
  • Validate accuracy; don’t rely on tiny models for safety‑critical or high‑accuracy tasks.
  • Explicitly instruct models to avoid external resources if fully offline operation is required.

Limitations and caveats observed

• Hallucinations: wrong breed IDs, odd text translations, invented words.
• Generation instability: infinite loops / repeated sections during longer coding tasks in this environment.
• Not ready to replace larger models for complex production use-cases despite strong capabilities.
• Some outputs rely on external assets (e.g., Unsplash) when online — must instruct the model to avoid external resources for fully offline runs.

Practical takeaway: Qwen 3.5 small models demonstrate impressive on‑device multimodal performance and extremely large context windows, but they remain imperfect — fast and capable for prototypes and edge use‑cases, yet not yet fully reliable for production or safety‑critical applications.

Broader context / significance

• Shows that high‑capability multimodal models can be very small and run on older laptops and recent smartphones offline.
• Potentially transformative for edge AI use cases (privacy, offline apps, local processing).
• Caveat: Alibaba reportedly restructuring the Qwen team (key engineers departing), raising questions about future releases and long‑term support for the series.

Video type / contents

Hands‑on review + tutorial/demonstration including:

• Benchmarks overview
• Offline local‑server and VS Code / Klein coding demo
• Native iOS on‑device demo (MLX Swift)
• Practical observations, limitations, and subjective evaluation

Main speakers / sources

• Speaker/tester: Andress from Better Stack (video narrator / tester)
• Technologies & organizations referenced: Alibaba (Qwen 3.5), LM Studio, Klein (client), MLX Swift (Apple), M2 MacBook Pro, iPhone 14 Pro, Llama 2 (benchmark comparison), IBM Granite (context)