deploybase

How to Fine-Tune an LLM - Complete Beginner Guide

Deploybase · February 15, 2025 · Tutorials

Contents

Preparation Phase

Step 1: Gather Training Data

Collect labeled examples demonstrating desired behavior.

Format: JSON lines (each line is one example)

{"prompt": "Classify sentiment: The product works great!", "completion": "Positive"}
{"prompt": "Classify sentiment: Terrible experience", "completion": "Negative"}

Minimum: 100 examples (marginal improvement) Recommended: 1,000+ examples (substantial improvement) Optimal: 10,000+ examples (diminishing returns beyond)

Data collection strategies:

  • Annotate customer feedback manually
  • Extract from existing datasets (public datasets, internal logs)
  • Use crowdsourcing (Mechanical Turk, Scale)
  • Generate synthetic examples (cheaper but lower quality)

Budget calculation: At $0.05-$0.15 per example with crowdsourcing, 1,000 examples cost $50-$150. In-house annotation: 5-10 minutes per example, $10-$25 per hour labor = $100-$250.

Step 2: Data Quality Assurance

Check for:

  • Consistency (similar inputs produce similar outputs)
  • Correctness (outputs factually accurate)
  • Diversity (cover edge cases and variations)
  • Format consistency (same structure throughout)

Fix issues:

  • Remove duplicates
  • Correct obvious errors
  • Trim obviously wrong examples
  • Balance class distribution (for classification)

Quality filtering: Remove bottom 10-20% lowest quality examples. Quality matters more than quantity.

Step 3: Split Dataset

Standard split: 70% training, 15% validation, 15% test

Training: 700 examples
Validation: 150 examples
Test: 150 examples

Validation set for hyperparameter tuning. Test set for final evaluation (use only once).

Technical Setup

Option A: Using OpenAI API Fine-Tuning

Easiest approach. No infrastructure setup needed.

Step 1: Format data for OpenAI (JSONL format, max 1GB):

python prepare_data.py --input data.csv --output train.jsonl

Step 2: Create fine-tuning job:

openai api fine_tuning.jobs.create \
  -t train.jsonl \
  -v validation.jsonl \
  --model gpt-3.5-turbo \
  --hyperparameters n_epochs=3

Step 3: Monitor training:

openai api fine_tuning.jobs.retrieve [JOB_ID]

Step 4: Use fine-tuned model:

openai api chat.completions.create \
  -m [FINE_TUNED_MODEL_ID] \
  --messages '[{"role": "user", "content": "Your prompt here"}]'

Cost: $0.008 per 1K tokens training + $0.012 per 1K inference. 1,000 examples @ 100 tokens each = $0.80 training cost. Inference costs higher than base model.

Option B: Self-Hosted Fine-Tuning on GPU

Full control, lower inference costs, higher setup effort.

Requirements:

  • GPU with 24GB+ VRAM (RTX 4090, A100)
  • Python 3.10+
  • PyTorch, Hugging Face Transformers
  • 1-3 days learning curve

Step 1: Rent GPU infrastructure:

RunPod A100 at $1.39/hour. 24-hour fine-tuning session: $33.36

Step 2: Set up environment:

pip install torch transformers datasets peft

git clone https://github.com/huggingface/transformers.git

Step 3: Prepare data:

python prepare_dataset.py \
  --input_file data.jsonl \
  --train_ratio 0.7 \
  --output_dir ./data

Step 4: Configure training:

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./fine_tuned_model",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,
    warmup_steps=100,
    save_strategy="epoch",
    evaluation_strategy="epoch"
)

Step 5: Run training:

python train.py --training_args training_args.json

Training time: 2-48 hours depending on:

  • Model size (7B parameters: 4-8 hours, 70B: 20-48 hours)
  • Dataset size (1,000 examples: 2-4 hours, 10,000: 8-16 hours)
  • Hardware (A100 faster than RTX 4090)
  • Batch size (larger batch = faster but uses more memory)

Step 6: Evaluate on test set:

python evaluate.py --model ./fine_tuned_model --test_file data/test.jsonl

Training Best Practices

Learning Rate Selection

Start with 2e-5 (2×10^-5). Too high = unstable training, loss spikes. Too low = training too slow, suboptimal convergence.

Learning rate schedule: Warm up from 0 to target over first 10% of training. Cosine decay to near-zero at end.

Batch Size Selection

Larger batches: faster training, less memory randomness, smoother loss curves Smaller batches: less memory required, potentially better final accuracy

RTX 4090 (24GB): batch size 4-8 A100 (40GB): batch size 32-64 H100 (80GB): batch size 64-128

Rule of thumb: Maximize batch size until out-of-memory errors appear.

Epoch Optimization

Too few epochs: underfitting, suboptimal performance Too many epochs: overfitting, memorizing training data

Optimal range: 2-5 epochs for most tasks. Validation loss should decrease initially, then plateau or increase (overfitting signal).

Monitor validation loss. Stop training when validation loss increases for 2+ consecutive checkpoints (early stopping).

Gradient Accumulation

Simulates larger batch sizes without more memory:

training_args.gradient_accumulation_steps = 4  # 4x effective batch size

Use when desired batch size exceeds GPU memory. Performance nearly identical.

Cost Optimization

Strategy 1: Quantization-Aware Training

Train using INT8 weights instead of FP32. Reduces memory by 75%.

from peft import prepare_model_for_int8_training

model = prepare_model_for_int8_training(model)

Fine-tune RTX 4090 instead of A100. Cost: $0.34/hour vs $1.39/hour. 24-hour training: $8.16 vs $33.36. Savings: $25.20.

Quality impact: Usually 1-2% accuracy reduction. Often acceptable tradeoff.

Strategy 2: LoRA (Low-Rank Adaptation)

Train only small adapter matrices instead of full model. 1,000x fewer parameters.

from peft import get_peft_model, LoraConfig

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"]
)
model = get_peft_model(model, config)

Memory reduction: 50-70%. Training speed: 2-3x faster. Quality: Comparable to full fine-tuning.

Cost impact: Train on RTX 3090 ($0.22/hour) instead of A100. 12-hour training: $2.64 vs $16.68.

Strategy 3: Distributed Training

Split model or batch across multiple GPUs. RunPod offers multi-GPU pods.

4x A100: 4 × $1.39 = $5.56/hour, but training 3x faster. Effective cost: $1.85/hour per GPU.

Use when time-critical. Adds orchestration complexity.

Strategy 4: Mixed Precision Training

Train in FP16 instead of FP32. Half the memory and computation.

training_args = TrainingArguments(
    fp16=True,
    # ... other args
)

Stability concerns: Loss overflow, gradient scaling needed. Mitigated by amp loss scaling.

Memory savings: 40-50%. Speed improvement: 20-30%. Quality: Identical to FP32.

Deployment

Option 1: API Deployment

Fine-tuned models available immediately through OpenAI API.

openai api completions.create \
  -m [FINE_TUNED_MODEL] \
  -p "Your prompt"

Cost: Higher inference rates than base model. Suitable for low-volume applications.

Option 2: Self-Hosted Inference

Run fine-tuned model on cloud GPU.

Setup minimal: Load model, serve with vLLM or TorchServe.

python -m vllm.entrypoints.openai.api_server \
  --model ./fine_tuned_model \
  --port 8000

Call via HTTP:

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "fine_tuned_model",
    "prompt": "Your prompt",
    "max_tokens": 100
  }'

Cost: RunPod A100 at $1.39/hour. Suitable for high-volume applications with predictable throughput.

Option 3: Serverless Inference

AWS Lambda with GPU, Google Cloud Run, or Hugging Face Inference API.

huggingface-cli upload [USERNAME]/fine_tuned_model ./model

Cost: Per-request pricing or subscription. Suitable for variable load.

Evaluation Metrics

Choose metrics matching the task:

Classification Tasks

Accuracy: (TP + TN) / (TP + TN + FP + FN) Precision: TP / (TP + FP) Recall: TP / (TP + FN) F1: 2 × (Precision × Recall) / (Precision + Recall)

Balance trade-offs. Precision important when false positives costly. Recall important when false negatives costly.

Generation Tasks

BLEU: Matches n-grams with reference (0-1 scale) ROUGE: Recall-oriented metric (0-1 scale) METEOR: Considers synonyms and paraphrases (0-1 scale) Human evaluation: Preferred when possible

Hallucination Testing

Fact-checking: Feed domain-specific facts, verify accuracy Consistency: Ask same question multiple ways, verify answers match Confidence: Check if model admits uncertainty vs false confidence

Common Pitfalls

Pitfall 1: Too little data. Result: Overfitting. Solution: Minimum 1,000 examples for reliable improvement.

Pitfall 2: Poor data quality. Result: Model learns incorrect patterns. Solution: Manual QA, remove outliers.

Pitfall 3: Data leakage. Result: Inflated evaluation metrics. Solution: Strict train/test split, no duplicate examples.

Pitfall 4: Insufficient training. Result: Suboptimal performance. Solution: Monitor validation loss, train until convergence.

Pitfall 5: Catastrophic forgetting. Result: Base model capabilities degrade on original tasks. Solution: Mix task-specific and general examples in training data.

Advanced Fine-Tuning Techniques

Parameter-Efficient Fine-Tuning (PEFT)

LoRA and similar techniques dramatically reduce trainable parameters:

Full fine-tuning: All model weights updated (billions of parameters) LoRA fine-tuning: Only adapter matrices trained (thousands of parameters)

Memory reduction: 50-70% (fit on RTX 4090 instead of A100) Training speed: 2-3x faster Quality: Comparable to full fine-tuning

Technique called Low-Rank Adaptation. Insert small trainable matrices between layers. Multiplicatively combine with base weights. Effective because weight changes likely low-rank.

Instruction Fine-Tuning

Fine-tune on instruction-response pairs rather than raw text.

Format:

Instruction: What's the capital of France?
Response: Paris is the capital of France.

Trains model to follow instructions. Improves reasoning and task-specific behavior.

Preference Fine-Tuning (RLHF Alternative)

Instead of binary correct/incorrect labels, provide ranking of responses.

Example:

Prompt: Explain quantum computing
Response A: [explanation 1]
Response B: [explanation 2]
Ranking: Response A > Response B

Model learns to prefer higher-ranked responses. Technique closer to how humans evaluate quality.

Requires more data but improves output quality significantly.

Data Augmentation Strategies

Synthetic Data Generation

Generate additional training examples programmatically:

Back-translation: Translate text to foreign language and back Paraphrasing: Rephrase existing examples Template-based: Fill templates with variables Model-generated: Use base model to create variations

Synthetic data reduces annotation burden. Balance synthetic and human data.

Typical ratio: 30-50% synthetic, 50-70% human-annotated.

Data Balancing

Imbalanced training data biases model learning.

Example: Classification dataset:

  • Class A: 9,000 examples (positive)
  • Class B: 100 examples (negative)

Model overfits to Class A. Techniques for balance:

  • Undersample majority class
  • Oversample minority class (with variations)
  • Weighted loss function (penalize majority class mistakes less)
  • Balanced batch sampling

Balanced datasets improve generalization significantly.

Inference Optimization Post-Fine-Tuning

Quantization of Fine-Tuned Models

Fine-tune in FP32 for quality, quantize after training:

python train.py --output_dir ./fine_tuned

python quantize.py --model ./fine_tuned --output ./fine_tuned_int8

Inference 4-10x faster. Memory 75% lower. Quality loss minimal (1-2%).

Distillation of Fine-Tuned Models

Train smaller model using fine-tuned large model as teacher:

Large model (70B fine-tuned) → Small model (7B) learning Small model captures 85-95% of performance.

Cost reduction: 10x on inference (smaller model runs on cheaper GPU).

Process:

  1. Fine-tune large model
  2. Generate predictions on unlabeled data
  3. Train small model on predictions
  4. Test small model vs original

Serving Optimization

Deploy using vLLM or similar:

python -m vllm.entrypoints.openai.api_server \
  --model ./fine_tuned_model \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9

Parameters:

  • tensor-parallel-size: Distribute across multiple GPUs if needed
  • gpu-memory-utilization: Maximum GPU memory percentage (0.9 = aggressive)

Fine-Tuning Failure Modes and Recovery

Catastrophic Forgetting

Fine-tuning on narrow task causes model to forget general knowledge.

Recovery:

  • Mix task-specific data with general examples (80/20 split)
  • Use lower learning rates
  • Train for fewer epochs
  • Evaluate on general benchmarks alongside task metrics

Overfitting

Fine-tuning on small dataset (< 100 examples) overfits.

Signs:

  • Training loss decreases, validation loss increases
  • Perfect accuracy on training, poor on test
  • Model memorizes examples

Recovery:

  • Collect more diverse data
  • Apply regularization (dropout, weight decay)
  • Use data augmentation
  • Shorten training (early stopping)

Divergence

Training loss increases instead of decreasing (unstable training).

Causes:

  • Learning rate too high
  • Batch size too small
  • Gradient accumulation misconfigured

Recovery:

  • Reduce learning rate 10x
  • Increase batch size if memory allows
  • Check gradient scaling (mixed precision issues)

Production Deployment Considerations

Monitoring Fine-Tuned Models

Track performance over time:

  • Accuracy/F1 on validation set
  • Latency per request
  • Token output quality (human review)
  • Error rates and types

Alert on degradation:

  • Accuracy drops > 2%
  • Latency increases > 50%
  • Error rate spikes

Scheduled retraining:

  • Retrain weekly/monthly with new data
  • Evaluate before deploying new version
  • Rollback if performance degrades

Versioning and Rollback

Maintain model versions:

  • v1.0: Initial fine-tuned model
  • v1.1: Retrained with new data
  • v1.2: Optimized inference version

Tag production deployments:

  • Track which version in production
  • A/B test new versions (10% traffic initially)
  • Easy rollback if issues emerge

A/B Testing Fine-Tuned Models

Test new model versions on subset of traffic:

Example setup:

  • 95% requests → Current model (stable)
  • 5% requests → New fine-tuned model (test)

Monitor metrics:

  • Success rate (task-specific)
  • User satisfaction (if available)
  • Latency difference
  • Cost difference

If new model better: Gradually increase traffic percentage

FAQ

How much data do we need? Minimum 100 examples. Reliable improvement at 1,000+. Diminishing returns beyond 10,000.

How long does fine-tuning take? OpenAI API: 15-60 minutes Self-hosted A100: 2-24 hours Depends heavily on data size and model size.

Can we fine-tune GPT-4? Yes, as of 2024 OpenAI supports GPT-4o and GPT-3.5-Turbo fine-tuning via the fine-tuning API.

Does fine-tuning hurt general capability? Depends on training data balance. Include general examples to prevent catastrophic forgetting.

What if performance plateaus? Likely overfitting. Try regularization (lower learning rate, more epochs, dropout). Or collect more diverse training data.

Can we fine-tune on dialogue/chat? Yes. Format: conversation_history + response. Works well for chatbots.

Self-Hosted LLM Complete Setup Guide OpenAI API Pricing RunPod GPU Pricing AI Cost Optimization Tips Compare GPU Cloud Providers

Sources

Hugging Face fine-tuning documentation. OpenAI fine-tuning API guide. PyTorch training best practices. Academic research on transfer learning and domain adaptation. Practical experience from thousands of fine-tuning jobs. MLCommons training efficiency benchmarks. GPU cloud pricing as of March 2026.