Video summary

How to Choose the BEST LLM for YOUR Project: A Methodical Guide

Main summary

Key takeaways

Technology

Summary: How to Choose the Best LLM for Your Project

The video proposes a methodical, time-agnostic approach to selecting the best LLM (Large Language Model) for a specific project. The creator emphasizes that the model landscape is now more complex (many providers, many variants), and shares personal favorite LLMs at the end by use case.

1) Start from your use case

Different LLMs excel at different tasks. Examples mentioned include:

  • Difficult math equation solving
  • Generating Instagram/social posts
  • Document summarization / data processing
  • Mental health assistants
  • Image classification (e.g., birds)
  • Code generation (e.g., autocomplete-style behavior in Cursor)

2) Compare models by raw quantitative intelligence

The video explains that “intelligence” is often measured with quantitative benchmarks such as:

  • General question answering, reasoning, math, coding, science
  • Provider release posts/papers reporting accuracy on standardized tests, including:
    • “competition math,” GPQA, multilingual QA, high-school math, and coding tasks

Recommended tools/sites for intelligence comparisons

  • ArtificialAnalysis.ai Aggregates many benchmark scores into a combined “intelligence index” style comparison.

  • Humanity’s Latest Exam Mentions Gemini 2.5 Pro leading, with O3 close behind.

  • Hugging Face leaderboard Especially useful for open-source/self-hosted models.

3) Important caveats of benchmark-driven selection

The video warns that benchmark results can be misleading:

  • Benchmarks mainly reflect quantitative performance and may not measure qualitative needs like:
    • creativity, empathy, or engaging writing
  • Real-world usefulness may diverge because benchmarks can become overfit/leaked into training data, inflating performance vs. your actual data.
  • More “reasoning-focused” models can increase latency and cost:
    • Reasoning models may emit many “reasoning tokens” (analogous to chain-of-thought), and API pricing may charge for them.
  • The most intelligent models are often closed-source, which trades off privacy.

4) Compare models by price

Key points:

  • Total cost depends on whether you use a hosted API or a cloud/self-host setup.
  • ArtificialAnalysis.ai is again suggested for quick comparisons like cost per token (or cost per X tokens).

Why self-hosting large models is often not cost-effective

  • Example reasoning in the video: a ~78B parameter model would require enormous VRAM.
    • Estimate: on the order of ~78 GB VRAM at fp8
  • The video uses an NVIDIA H100 (80 GB VRAM) example and emphasizes high hourly costs.
  • Smaller/cheaper models can still be highly useful for tasks like summarization, keyword extraction, and simple classification.

5) Compare models by speed/latency

Tradeoff: smaller models usually provide lower latency, making them better for real-time tasks (e.g., translation).

6) Compare models by privacy

  • Maximum privacy typically implies self-hosting.
  • However, self-hosting at scale can be expensive and may reduce flexibility.
  • A “middle ground” mentioned: enterprise/private cloud options like Azure OpenAI, with the claim that providers may not retrain on your data (as stated in the video).

7) Use human preference evaluations for non-benchmarkable qualities

The video highlights LLM Arena:

  • Humans submit a prompt and compare two models’ responses.
  • Models gain/lose ELO-like ratings based on preferences.

It also notes that even if top models lead on intelligence benchmarks, other models may win in qualitative areas such as:

  • Creative writing
  • Text-to-image generation (framed as difficult to evaluate quantitatively)

8) Check popularity/real-world usage via OpenRouter

OpenRouter rankings are described as a practical signal:

  • How many people use each model
  • How preferences vary by prompt type (e.g., programming vs. roleplaying vs. marketing)

9) Account for feature requirements not covered by benchmarks

Select models based on capabilities your application needs, such as:

  • Structured outputs
    • constrained JSON
    • class labels + explanations
    • output parsed into Pydantic models
  • Vision
    • image understanding, OCR
  • Tool calling / function calling
    • calling APIs and returning structured tool results—important for assistants

The video again mentions ArtificialAnalysis.ai for feature comparisons such as JSON mode and function calling.

10) Final recommendation: test with your own prompts/data

Even after reviewing benchmarks and signals above, the creator strongly recommends:

  1. Pick a shortlist of models
  2. Run your own benchmarks on the tasks you care about

They emphasize results can vary by use case and by who is using the model.

Personal favorite LLMs mentioned (by use case)

  • Gemini (Google) APIs
    • Especially Gemini 2.5 Pro and Gemini 2.5 Flash
    • Described as best value and cheap tokens for intelligence
    • Creator increasingly prefers them, including for coding in Cursor
  • Claude / Anthropic (via Cursor)
    • “code models optimized for coding”
  • OpenAI
    • GPT-4.0 / “40 O”: most frequently used; supports image generation in the UI
    • O3: preferred for reasoning, hard math, and difficult coding bugs
  • Open-source (local fine-tuning)
    • Llama 3.1 8B: fits on a MacBook for local running after fine-tuning
  • Open-source via API / deployment
    • DeepSeek: described as the best open-source option; also recommended when extra privacy is needed

Main speakers / sources

  • Main speaker/source: The video creator/host (single person speaking directly in subtitles).
  • Referenced sources/tools/benchmarks:
    • ArtificialAnalysis.ai
    • Humanity’s Latest Exam
    • Hugging Face leaderboard
    • LLM Arena
    • OpenRouter rankings
    • OpenAI (GPT-4.x, O3/O4 mentioned)
    • Google Gemini (Gemini 2.5 Pro/Flash mentioned)
    • Anthropic (Claude) (coding models mentioned)
    • Azure OpenAI (privacy/enterprise option mentioned)
    • Hugging Face / Ollama (self-hosting ecosystem mentioned)
    • DeepSeek, Llama 3.1 (open-source recommendations mentioned)
    • Lambda Labs and NVIDIA H100 (hardware pricing example mentioned)

Original video