Is Fine-Tuning Still Needed? LLMs, RAG, & LoRA

Summary

The video argues that fine-tuning still has a role, but it is no longer the default way to customize a large language model (LLM).

What fine-tuning does

Fine-tuning continues training a base model on a focused dataset, such as legal documents or internal support tickets. This adjusts the model’s weights to improve performance on narrower tasks.

Why its advantage has narrowed

Larger context windows let models use relevant documents directly in prompts. Reasoning models can spend more effort at inference time, and frontier models are improving while becoming cheaper. As a result, a custom model may be overtaken before it is deployed.

The video gives several examples of how results have changed:

• A legal AI company’s fine-tuned model reportedly beat GPT-4 in blind attorney tests 97% of the time in 2023. In the company’s 2025 legal benchmark, however, seven general-purpose models surpassed its custom model.
• Later evaluations found GPT-4 and ChatGPT outperforming BloombergGPT on many financial benchmarks.

Ways to customize without changing weights

• Retrieval-augmented generation (RAG): Retrieve relevant documents at query time and include them in the prompt.
• Context engineering: Assemble useful system instructions, relevant information, and formatting guidance into a prompt.
• Agent skills: Provide on-demand procedural instructions and tool guidance, such as how to query a particular database schema.

Fine-tuning’s costs

Beyond training, teams must collect examples, evaluate quality, prevent regressions, and maintain custom models as frontier models advance.

When fine-tuning may still help

• Latency-sensitive applications: A smaller fine-tuned model may respond faster than a reasoning model, such as in a real-time voice agent.
• Distillation: Use outputs or reasoning traces from a large model to train a smaller one.
• Reinforcement fine-tuning (RFT): Train a model using a grader that scores sampled answers. This is most useful when answers can be graded programmatically and have a definitive right answer.

LoRA, or low-rank adaptation, trains a relatively small adapter while leaving most of the base model’s weights unchanged. The video says many production fine-tuning efforts use LoRA or related parameter-efficient methods.

Practical decision framework

Start with a base model and prompt or context engineering. Add RAG when information is fresh or proprietary, and agent skills when procedural knowledge is missing. Consider fine-tuning only when these approaches do not address a specific bottleneck.

Main speaker and sources

The presenter is identified in the video metadata as Martin Keen of IBM Technology. Examples discussed include an unnamed legal AI company, BloombergGPT, GPT-4 and ChatGPT, and general-purpose frontier models.