AI/ML8 min read

Fine-Tuning LLMs

Adapt a pre-trained LLM to a specific domain or task using supervised fine-tuning (SFT) or PEFT/LoRA - without the cost of training from scratch.

Fine-TuningLLMLoRAHugging FaceMachine Learning

skill: Fine-Tuning LLMs category: AI/ML tags: [Fine-Tuning, LLM, LoRA, Hugging Face, Machine Learning] readingTime: 8

Fine-tuning is warranted when the task requires consistent output format, domain-specific style, or fast inference without long prompts. Before fine-tuning, exhaust prompt engineering and RAG - they're cheaper and reversible. Fine-tuning a 7B model with LoRA costs ~$20-100; training from scratch costs millions.

Workflow

Decide: fine-tune vs. RAG vs. promptingFine-tuning bakes knowledge into weights - good for style/format consistency. RAG retrieves fresh knowledge at inference - good for factual updates. Prompting needs no training - try it first.
Prepare a high-quality datasetData quality beats quantity - 500 excellent examples outperform 10,000 noisy ones. Use chat format for instruction-tuned models.
Fine-tune with LoRA via Hugging Face PEFTLoRA (Low-Rank Adaptation) trains only a tiny adapter on top of frozen model weights - 10-100x fewer trainable parameters than full fine-tuning, fits on a single GPU.
Evaluate with task-specific metricsLoss alone doesn't tell you if the model does what you want. Evaluate on held-out examples using task-appropriate metrics.
Merge and export the adapterLoRA adapters can be merged back into base model weights for single-file deployment - no PEFT dependency at inference time.

Output Format

Fine-tuned adapter or merged model ready for inference, with evaluation metrics and training config documented.

output/
├── adapter_config.json     # LoRA config
├── adapter_model.safetensors
├── training_args.bin
└── trainer_state.json

Merged model (for deployment):
└── merged-model/
    ├── config.json
    ├── tokenizer.json
    └── model.safetensors

Examples

Input

Output

Input

Output

Edge Cases & Gotchas

Data contamination - If your evaluation examples were in the training set, metrics are inflated. Always split data before any preprocessing step.
Learning rate too high - High LR causes catastrophic forgetting. For LoRA fine-tuning, use 2e-4 to 1e-4. For full fine-tuning, use 1e-5.
Overfitting on small datasets - < 500 examples is prone to overfitting. Use early stopping (eval_steps, load_best_model_at_end=True) and watch eval loss diverge from train loss.
Tokenizer mismatch - Using a different tokenizer than the base model causes garbled output. Always save and load the tokenizer from the same model checkpoint.

When to Break These Rules

Use RAG instead of fine-tuning when the knowledge base changes more than monthly - fine-tuning is expensive to re-run for every update.
Use prompt engineering before fine-tuning - a well-designed system prompt with few-shot examples often achieves 80% of fine-tuning quality at zero cost.

Reference Files

https://huggingface.co/docs/peft
https://github.com/huggingface/trl