Data and Model Poisoning
Poisoning attacks corrupt training data, fine-tuning data, or RAG indices to bias model behaviour. The attacker influences what the model learns — quietly, persistently, and often without detection until the trigger phrase is hit in production.
01What it is
Adversarial manipulation of the data the model consumes. Three flavours: pretraining poisoning (rare but devastating), fine-tuning poisoning (realistic for any team that scrapes data), and RAG poisoning (planting documents in indices the agent retrieves). Effects range from subtle bias to targeted backdoors that activate on specific triggers.
02Why it matters
Most teams build RAG over data they only loosely curate — Confluence wikis, customer-uploaded files, public web scrapes. Any of these can be poisoned. Fine-tunes on public corpora are equally exposed. Once poisoned, behaviour is hard to undo without retraining.
03Attack vectors
- RAG poisoning — adversary uploads a document optimised for retrieval that contains instructions for the LLM.
- Fine-tuning poisoning — slip 0.1% of poisoned examples into a 10K-example dataset; the model learns the backdoor pattern.
- Reward-model poisoning — for RLHF, corrupt the preference dataset that shapes the reward model.
- Web-scraped data — any open dataset built from crawls contains adversarial content.
- Trigger-based backdoors — model behaves normally unless the trigger token appears, then misbehaves on demand.
04Defence patterns
- Data provenance — track every example through curation, label, hash, and version.
- Anomaly detection — cluster the training set; flag outlier clusters before training.
- Differential privacy — bounds memorisation, limits poisoning impact (with accuracy cost).
- Robust training — gradient-based defences (Bagging, RAB) that are robust to a fraction of poisoned examples.
- Eval gates — large red-team suite that runs after every fine-tune; refuse to ship if behaviour drifts on trigger probes.
- RAG hygiene — never auto-ingest user-uploaded documents into the same index your agent retrieves from. Quarantine first.
05Detection
Signals to watch
Behavioural diff between pre-train and post-fine-tune models on a fixed eval. Drift detection on production response distributions. Tripwire prompts that probe for known trigger patterns.
06India context
DPDP · RBI · CERT-In
For regulated entities, model risk management policy must cover training-data provenance. ISO/IEC 42001 (AI management systems) treats training data as a managed asset. DPDP applies if poisoned data contains personal information.