LLM04: Data and Model Poisoning
Risk Level: High
Attackers manipulate training data, fine-tuning data, or embedding data to introduce backdoors, biases, or vulnerabilities into the model. Poisoned models may generate harmful outputs, leak sensitive information, or produce subtly incorrect results that are difficult to detect. This extends beyond pre-training to include RAG data poisoning and fine-tuning attacks.
Attack Example
An attacker contributes thousands of code samples to a public dataset that contain subtle security flaws. When a code-generation LLM is fine-tuned on this data, it learns to suggest insecure coding patterns — such as using eval() for input processing or weak cryptographic functions. Alternatively, an attacker poisons a RAG knowledge base to inject false information into LLM responses.
Mitigations
- Vet and audit training data sources — verify provenance and integrity
- Use data sanitization pipelines to filter malicious or low-quality training samples
- Implement anomaly detection during fine-tuning to flag unusual model behavior changes
- Maintain a data lineage record (AI-BOM) for all training, fine-tuning, and RAG datasets
- Validate RAG data sources and implement integrity checks on knowledge bases