Finetune VLMs for Complex Arabic OCR: The Complete 3.5-Hour Masterclass

Technological focus of the masterclass

Problem

Extracting exact Arabic text (OCR-like) from complex scanned documents/images, including:

• Different line styles and layouts
• Watermarks embedded in text
• Page elements like seals, headers, footers, stamps
• Distorted scans / poor digital quality
• Mixed inputs:
  • Some pages in PDF images
  • Others possibly digitally generated text
  • Mixed fonts/layouts
  • Tables / index-like pages

Chosen approach

Instead of relying on traditional OCR only, the instructor uses Vision-Language Models (VLMs) to:

1. Read document images directly
2. Use prompts to output structured data (JSON) describing extracted content and metadata

Local/offline constraint

The project design initially requires no cloud inference (“local model only”), which motivates a full local pipeline:

• PDF → image conversion
• image preprocessing
• run a local VLM
• later, improve quality with fine-tuning

---

Product / model choices and empirical findings

Model families explored

The instructor experimented with several Arabic-capable VLM families, including (notably):

• Gemma 3
• Qwen / Qwens and other Arabic-supportive models
• LLaVA-like variants
• Mentions of Gamma / LavaNext

Key lessons from experiments

• Small models can cause poor extraction and missed exact strings.
• Larger models reduce mistakes but are expensive/slow and may still produce hallucinations (invented or altered text).
• The model must be instructed to perform verbatim extraction (no guessing; preserve original names/scripts).

Selection criteria

Preferences emerged based on:

• Speed
• “Conservatism” (less hallucination, fewer rewriting behaviors)

Some models were rejected because they were:

• too hallucination-prone, or
• inclined to “write/summarize” instead of extract.

Another option was chosen due to faster inference and better behavior after fine-tuning.

---

Core pipeline described (tutorial-like steps)

1) Data sourcing

• Example dataset: ~71 PDF documents (Saudi legal system / ministry-related)
• Checks robots.txt / public access before downloading
• Uses Google Colab for workflow

2) PDF → images

• Converts each PDF page to an image (e.g., a “convert PDF to images” library)
• Creates output directories per PDF
• Uses tunable parameters like image resolution / DPI
  • Smaller images speed up models but risk losing detail

3) Image preprocessing (for better OCR/VLM performance)

• Convert to grayscale
• Resize while preserving aspect ratio
• Adjust contrast (optional but useful)
• Guardrails:
  • do not enlarge small images excessively to avoid pixelation
• Provides a reusable preprocessing function with parameters such as:
  • max width/height constraints
  • min/max logic

4) Local inference with a VLM

• Loads a base VLM (Gemma-family discussed) with HuggingFace Transformers
• Uses a chat/instruction template and processor pipeline:
  • prompt + image → generated output
• Handles:
  • GPU device mapping
  • dtype
  • generation settings (e.g., max new tokens)
• Converts model output back into text / structured format

5) Switch from “text OCR” to “structured document understanding”

The output is reframed as JSON including:

• Document classification (e.g., official letter / regulation / report / certificate)
• Primary language(s)
• Issuer departments / ministries
• Dates (Hijri/Gregorian), deadline/expiry dates
• Physical layout (handwritten vs printed, tables/charts, page numbers)
• Presence/description of seals/signatures
• Table contents (where present)

6) Cloud fallback to improve training data (distillation)

When local models underperform:

• Use a stronger cloud VLM (via OpenRouter / Gemini 3 Flash) to generate “gold-ish” structured outputs
• Save outputs as JSONL (one JSON per image)

7) Fine-tuning local model using knowledge distillation

• Strategy:
  • teacher = cloud VLM
  • student = local VLM fine-tuned on teacher JSON
• Uses LLaMA-Factory
• Uses adapter/LoRA concept:
  • “train an adapter on top of the base model”
• Splits into train/validation sets while attempting to prevent leakage:
  • validation includes images from the same source PDFs to keep distribution consistent

8) Data format and prompt design for fine-tuning

• Converts JSONL into LLaMA-Factory “conversation” format with:
  • <image> placeholders (or image-tag placeholders)
  • system/user messages
  • output as the requested JSON structure
• Emphasized constraints:
  • preserve Arabic script
  • don’t romanize Arabic names
  • use original names exactly
  • fix output formatting issues (including a “JSON repair” step via a json-repair library)

9) Two-stage extraction via two tasks

Instead of one huge output prompt, extraction is split:

• Task 1: extract content (main markdown/text body)
• Task 2: extract structure/metadata (classification, fields, seals/signatures, tables, etc.)

This was reported to improve stability/quality.

10) Training management

• Mentions:
  • checkpoints
  • periodic evaluation
  • careful GPU memory considerations
  • stopping/restarting behavior via checkpoints
  • logging/tracking with W&B

---

Speed optimization and deployment

Inference speed issue

Local inference on high-end VLMs can be slow:

• minutes per set in some cases
• ~tens of seconds per image in other cases

Two speed-related methods

1. Merge model + adapter

   • Use LLaMA-Factory “merge” export to produce a single deployable artifact
   • Goal: faster/cleaner inference

2. Serve via an API

   • Wrap image preprocessing + VLM inference in a local VM server + HTTP endpoint (mentions port ~8000)
   • Include load testing:
     • measure average latency and throughput

OpenE/compatible pipeline idea

• Convert images to Base64
• Send to a VLM endpoint using a JSON-driven prompt
• Return structured JSON results

---

Notable result claims / takeaways

• Local base models alone were insufficient for accurate extraction:
  • tendencies included hallucination and “rewriting”
• Fine-tuning with distillation data + LoRA improved:
  • verbatim extraction behavior
  • handling of Arabic + names + document sections
  • reliability of structured JSON output
• Merging adapters into the model can simplify deployment and may improve speed
• The cloud teacher (Gemini 3 Flash) produced high-quality outputs for distillation, improving downstream student performance

---

Main speakers / sources (as mentioned)

Instructor / presenter

• Narrator of the masterclass (no explicit name in subtitles)

Referenced frameworks / repositories

• HuggingFace Transformers
• LLaMA-Factory
• OpenRouter
• OpenAI/OpenAI-compatible client libraries (via an “OpenE” / OpenAI-like API mention)
• Gemini 3 Flash / Gemma 3 (model families discussed)

Referenced content creator

• “Ahmed’s channel” / “Imam Al-Am” (mentioned regarding an earlier fine-tuning explanation)