Fine-tuning Safety — LoRA, SFT, and RLHF Explained for Security Teams

Pre-trained models like Llama-3 ship with safety alignment baked in via RLHF. When you fine-tune for your domain, you can accidentally undo that alignment — the model becomes more eager to follow harmful instructions. Researchers showed that fine-tuning Llama-2-Chat on just 100 benign examples can degrade its refusal rate from 95% to 50%. Understanding what each tuning method does is essential before letting your team fine-tune anything.

SFT, LoRA, RLHF — what each actually does

SFT (Supervised Fine-Tuning): full-parameter fine-tune on labelled examples (prompt → desired response). Updates all model weights. Resource-heavy: 8B model needs 80GB+ VRAM with full-precision Adam. Quality: best, but easiest to break safety. LoRA (Low-Rank Adaptation): freeze base weights, train small adapter matrices (rank 16-64). Updates ~0.5% of parameters. Fits on a single 24GB GPU for 8B model. Faster, cheaper, more parallel-friendly. Quality: 95% of SFT. Safety: somewhat preserves base model alignment because most weights are frozen. RLHF / DPO (Direct Preference Optimization): train on (prompt, chosen_response, rejected_response) pairs. Aligns model to preferences. Used to make models safer/more-helpful. DPO is the simpler modern variant. Use when you want to improve preference following beyond what SFT can do.