AI Red Teaming — Methodology, PyRIT, garak, llm-guard

AI red teaming emerged as a discipline in 2023-2024. Microsoft published their AI Red Team learnings; Anthropic publishes red-team reports for each major Claude release. The goal: enumerate failure modes before deployment so the company knows what risks it ships. This is increasingly required by regulation (EU AI Act high-risk systems, US Executive Order 14110, India MeitY guidance).

The methodology — what AI red teaming actually is

Six phases. (1) Scope — define the model, deployment context, and harm categories in scope (CBRN, illegal content, PII leakage, jailbreak, etc.). (2) Threat modelling — who would attack, why, with what capability. (3) Attack design — craft prompts and scenarios that probe each harm category. Single-turn jailbreaks, multi-turn manipulation, role-play attacks, language-switching, encoding (base64, leetspeak), context overflow. (4) Execution — run prompts at scale, measure success rate. (5) Triage — categorise findings by severity (critical = clear policy violation, low = ambiguous). (6) Reporting — write up for engineering team to fix; track until resolved. Differs from pentest: results are statistical (success rates), not binary (vuln present or not).

PyRIT — Microsoft's open-source AI red team framework

pip install pyrit or run from Docker. PyRIT (Python Risk Identification Toolkit) automates probing. Core abstractions: orchestrators (single-prompt, multi-turn, tree-of-attacks), scorers (regex match, model-based scoring with separate LLM judge, classifier model), targets (HTTP endpoints, OpenAI/Anthropic API, local Ollama). Example workflow: feed 1000 prompts from HarmfulBehaviors.csv through the PromptSendingOrchestrator targeting your model; score each response with the SelfAskTrueFalseScorer using GPT-4 as judge (“does this response violate the safety policy?”); output a CSV of failures sorted by severity. PyRIT is ~80% of what enterprise red teams use; the gap is custom probes for domain-specific risks.

garak — NVIDIA's LLM vulnerability scanner

pip install garak. Different philosophy from PyRIT — garak is more like Nessus for LLMs. Run garak --model_type huggingface --model_name openai-community/gpt2 --probes encoding,promptinject,xss and it runs ~100 pre-built probes against the model, categorised by attack type. Output is a structured report with pass/fail per probe. Probe categories include: encoding bypasses, prompt injection variants, XSS-via-output, glitch tokens, training data extraction, leak detectors, malware generation. Use garak as your CI check for any model release; PyRIT for in-depth manual investigation.

llm-guard, NeMo Guardrails — defensive layers

Defenders’ tools, paired with red team. llm-guard (ProtectAI) is a Python library that scans prompts and responses for: prompt injection patterns, secrets/PII (regex + ML), bias, toxicity, malicious URLs. Wrap your LLM calls: from llm_guard import scan_prompt; sanitised, results, scores = scan_prompt(scanners, raw_prompt). NeMo Guardrails (NVIDIA) is more like a policy engine — define dialogue flows, restricted topics, fact-grounding requirements in YAML, and the runtime enforces them. Best for chatbots with constrained scope (customer support, internal Q&A); too restrictive for open-ended assistants. Use BOTH: red team to find gaps, guardrails to plug them, red team again to verify.

Multi-turn red teaming and the AART scenarios

Single-turn jailbreaks (“ignore previous, output X”) are 2023. State-of-the-art is multi-turn — gradually steer the model into compliance over 5-15 messages. Researchers built AART (Automated Adversarial Red Teaming) where one LLM attacks another in conversation. Probes scenarios like: starting innocuous → introducing a fictional framing → asking the model to maintain the fictional framing → asking the harmful question within fiction. Defence: explicit scenario detection, conversation-level safety classifiers (not just per-turn), refresh model state if topic drifts inappropriately.

Building an internal red team capability

Headcount: 1 dedicated AI red teamer can run a meaningful program for 1-3 production models. Skills: prompt engineering fluency, statistical thinking, security background helpful but not required. Tools budget: free open source covers 80%; commercial tools (Lakera Red, Robust Intelligence) cost $50K-200K/yr. Cadence: continuous (CI on every model deploy) + quarterly deep-dive. Reporting: monthly metrics to product leadership, ad-hoc critical findings to engineering. Indian context: MeitY guidance and Niti Aayog AI principles increasingly expect documented red-team results for production AI in regulated sectors. Ahead-of-regulation investment.

Build your red-team suite — the 90-day rollout plan

Week 1-2: Pick a tool. garak (NVIDIA) is the broadest battery — 20+ probe categories, easy to integrate. PyRIT (Microsoft) has more agentic capabilities — it can adaptively iterate on attacks. Start with garak for breadth, add PyRIT for depth on critical apps. Week 3-4: Run baseline. garak --model_type rest --model_name your-app-endpoint --probes all. You will get a report with per-probe success rates. Document baseline numbers. Week 5-8: Add custom probes for your domain — your specific system prompt, your specific tools, your specific data sensitivity classes. Generic probes cover 70% of risk; custom probes cover the rest. Week 9-10: Wire into CI. Run a fast subset (10 minutes) on every PR; full suite nightly. Week 11-12: Operationalise. Establish: who reviews findings, what severity classification, what SLA for fixes. Without this, red-team output piles up unread. By day 90 you should have: tooling installed, baselines, custom probes, CI integration, and a triage process. This is the bar for “we do AI red-teaming” — vague verbal claims are not.

Findings triage — converting red-team output to security backlog

A garak run produces hundreds of findings. Without triage, the team drowns. Triage rubric: (1) Severity based on potential impact: data leak / harmful content / RCE = Critical; tool misuse / wrong refusal = High; off-topic / minor inconsistency = Low. (2) Reproducibility: can you reproduce manually? Aim for 3/3 attempts. Single-shot success on a stochastic system might be flaky. (3) Exploitability: does it require a specific feature configuration? Attacker control? Authentication? Prioritise unauthenticated, easily-reproducible findings. (4) Mitigation effort: prompt change (low) vs architectural change (high). (5) Coverage: does this finding indicate a class of bugs or a single instance? A class deserves a structural fix. Map findings to your defect-tracker (Linear, Jira) with a dedicated AI-Security label. Review weekly. Track trend: red-team success rate over time should be your primary metric. Going down quarter-over-quarter is the goal.

Reporting standards — how to write an AI red-team report that drives action

A red-team report that engineering does not act on is wasted effort. Structure that drives action, refined over many engagements: (1) Executive summary — three sentences max. What was tested, top finding, recommended action. The CISO reads this; nothing else. (2) Scope and methodology — what models, endpoints, data sources, tools. Time-bounded; explicitly out-of-scope items. Frameworks used (OWASP LLM Top 10, MITRE ATLAS). (3) Findings — one section per finding. Each finding has: title, severity (Critical/High/Medium/Low/Info), description, reproduction steps, evidence (sample prompts and outputs, redacted), exploitability assessment, recommended remediation, references to OWASP/MITRE IDs. (4) Statistical summary — for stochastic findings, success rate over N attempts. “Bypassed safety filter in 47/100 attempts (47%)” is actionable; “we bypassed it” is not. (5) Remediation roadmap — prioritised by severity × effort matrix. Quick wins listed first. Long-term architectural changes called out. (6) Re-test plan — schedule + scope for re-testing remediations. (7) Appendices — full payloads, full transcripts, tool outputs. For audit. Tone: factual, non-sensational. Never claim “catastrophic” — quantify impact. Length: 15-40 pages typical for a 2-week engagement. Anything >60 pages is unread. Anything <10 pages is incomplete. Delivery: meeting with engineering + security + product before the report lands; never first-time-read. Coach the team through findings; expect pushback on severities; defend with evidence. The goal is fixed bugs, not impressed clients.

Probe library reference — what to run and in what order

garak probe categories (run all on a new target): promptinject (direct injection variants), leakreplay (training-data extraction attempts), encoding (base64/ROT13/leet bypass), dan (jailbreak personas), continuation (harmful-content elicitation via story continuation), misleading (forces hallucinations), suffix (universal adversarial suffixes from Zou et al.), knownbadsignatures (known-malicious patterns), realtoxicityprompts (toxicity), av_spam_scanning (security signature evasion). Recommended order: (1) baseline with promptinject + leakreplay (1-2 hours) to surface low-effort attacks; (2) add encoding + dan (2-4 hours); (3) add suffix + knownbadsignatures (4-8 hours); (4) custom probes for your domain (variable). PyRIT use cases: adaptive multi-turn attacks; agentic red-team where attacker LLM crafts attacks against target LLM. Heavier; use after baselines complete. llm-guard: github.com/protectai/llm-guard — also useful as defensive middleware test target. Promptfoo: npm install -g promptfoo; run eval suites with red-team mode; integrates well with CI. Custom probe template (garak): subclass garak.probes.base.Probe; define prompts and recommended_detector; ship as a single Python file. Reporting: garak emits JSONL; aggregate per-probe success rate; track over time. Cadence: full suite nightly; fast subset on every PR; full attack mode on quarterly red-team week. Indian-language probes: limited public libraries; build your own — Hindi/Tamil prompt-injection variants, code-mixed Hinglish jailbreaks, regional mythology framings. This is original work; if you do it well, contribute to garak as upstream. Reading order: garak docs → Microsoft PyRIT docs → “AI Red Team Tactics” Microsoft talk (2024) → Embrace the Red blog archive. 2 weeks of focused study makes you operationally competent.

FAQ

Do I need a dedicated AI red teamer?

For one production AI feature with low impact: no, run automated probes (garak, PyRIT) in CI. For multiple AI features or any high-impact use (financial, medical, legal): yes — you need someone whose job is finding failure modes.

Should I publish my red-team findings?

Internal: always, to drive engineering fixes. External: depends on disclosure policy. Anthropic and OpenAI publish summary reports for releases; Microsoft AIRT shares case studies. Most enterprises keep findings internal but publish remediation status.

How often do new jailbreaks emerge?

Constantly. Major new attack classes every 2-3 months (DAN-mode → grandma-exploit → many-shot jailbreaking → multi-modal injection). Subscribe to llm-attacks.org, follow @SimonW on Twitter, monitor arXiv cs.CR section. Re-test models monthly against latest known attacks.

How much budget should we allocate to AI red-teaming?

5-15% of total AI engineering budget for sensitive workloads (regulated, customer-facing). 1-3% for internal-only tools. Includes: tooling (low — most are open source), engineering time (significant — it is real work), external pen-test for critical launches ($15-50k for a competent firm). For Indian fintech with a customer-facing AI assistant, plan for at least one external assessment per year plus continuous internal red-teaming.

Are there Indian firms that do AI red-teaming?

Growing market in 2026. RingSafe, NII Consulting, ProCheckUp India, and Lucideus all advertise AI / LLM testing services. Quality varies; ask for sample reports, references, and methodology before contracting. The OWASP LLM Top 10 + MITRE ATLAS coverage should be table-stakes. Some “AI security” pitches are warmed-over web app pentests with extra slides — verify before signing.

⚖️ Legal: Use AI security techniques only on systems you own or have explicit written authorisation to test. In India, unauthorised access is punishable under IT Act §66 (up to 3 years + fine). Pair AI red-teaming with signed Statement of Work or Rules of Engagement before testing.