AI Security 101 — Why ML Systems Break Differently

You probably already understand application security: SQL injection, XSS, broken access control. The OWASP Top 10 maps cleanly to code paths you can grep for. AI security is fundamentally different and unsettling — there is no line of code where the bug lives, no stack trace pointing to the broken function, and the same input that fails today might succeed tomorrow because the model is non-deterministic. If you bring traditional appsec instincts to ML systems and nothing else, you will miss every real attack. This module builds the foundation so the rest of the AI Security track makes sense.

What is an "AI system" actually?

When a security team says “we use AI,” they usually mean one of four things, and the threat model differs for each. Predictive ML is the classical case: a model trained on labelled data that classifies (spam vs not-spam) or regresses (predict sales). Generative LLMs like GPT-4, Claude, Llama, Mistral take a prompt and emit a continuation — they do not retrieve answers, they hallucinate them and sometimes the hallucination happens to be correct. Embedding models turn text/images into vectors used for similarity search and retrieval-augmented generation. Agent systems wrap an LLM with tools, memory, and a loop so it can take real actions (browse, send email, run code). Each architecture has unique failure modes. A SQL-injection mindset applied to an LLM finds nothing; you need a different lens for each. The first job of an AI security person is to ask “what kind of AI?” before any threat-modelling.

Why probabilistic systems break security assumptions

Traditional security depends on determinism. The same input produces the same output. You can write a unit test asserting “given malicious payload X, the system rejects it.” LLMs do not work that way. The exact same prompt, run twice, can produce different output — sometimes the safety filter catches the harmful response, sometimes it does not. Temperature, top-p sampling, and seed values all affect outcomes. This means traditional regression tests fail to catch security bugs that exist 10% of the time. It also means red-team findings are statistical: “this jailbreak works 73% of the time” is a real finding, not a flaky test. Risk acceptance shifts from “is the bug present” to “what is the failure rate at scale.” If your LLM serves 1 million queries per day and a jailbreak works 0.1% of the time, you have 1,000 successful attacks daily.

The four categories of AI-specific attacks

(1) Input attacks — adversarial examples in classifiers, prompt injection in LLMs. The attacker controls model input and exploits how the model processes it. (2) Training-time attacks — data poisoning, backdoors implanted via crafted training samples, model supply-chain compromise. (3) Model-extraction attacks — stealing the model itself by querying it cleverly, or extracting training data through prompt-engineering. (4) Inference-time leaks — the model emits secrets it learned during training (memorised PII, credentials, copyrighted text). Most public AI security incidents in 2025-2026 fell into category 1 (prompt injection in agent systems) or category 4 (Samsung engineers leaking source code into ChatGPT). Categories 2 and 3 require more attacker sophistication but the impact is greater.

OWASP LLM Top 10 (2025) — the practitioner reference

The OWASP LLM Top 10 is the de-facto threat catalogue. Memorise it: LLM01 Prompt Injection, LLM02 Insecure Output Handling, LLM03 Training Data Poisoning, LLM04 Model Denial of Service, LLM05 Supply Chain, LLM06 Sensitive Information Disclosure, LLM07 Insecure Plugin Design, LLM08 Excessive Agency, LLM09 Overreliance, LLM10 Model Theft. The 2025 edition added LLM07/08 because of the explosion in agent frameworks. For India context: DPDP Act 2023 makes LLM06 (sensitive info disclosure) a legal liability — if your support-chatbot leaks one user’s data to another, you owe regulatory notification within hours. Pair OWASP LLM Top 10 with MITRE ATLAS (Adversarial Threat Landscape for AI Systems) for full coverage. ATLAS is more detailed on specific TTPs.

Practical: your first AI security checklist

Before deploying any LLM-powered feature in production, run through this checklist: (1) Is user input ever concatenated into a prompt without delimiters? You have prompt injection risk. (2) Does the LLM call any tools (HTTP requests, database queries, code execution)? You have excessive-agency risk. (3) Does it have access to private data (RAG over internal docs, user records)? You have data leak risk. (4) Is the model self-hosted from a model file you downloaded from Hugging Face? You have supply chain risk. (5) Are model weights, system prompts, or fine-tuned LoRAs treated as secrets? They should be. (6) Do you log prompts and responses? Required for both abuse detection and DPDP-compliant breach investigation. (7) Have you tested a kill-switch — can you globally disable AI features in 60 seconds if something goes wrong? Most teams cannot. Add this to your engineering on-call runbook.

The Indian regulatory landscape in 2026

India’s AI regulation is fragmented but tightening. The MeitY 2024 advisory required all “untested AI” to obtain prior approval (rolled back after backlash but signalled intent). DPDP Act 2023 governs training data and inference outputs that contain personal data — your model is liable if it memorises and emits a PAN/Aadhaar. RBI’s 2024 framework on AI in lending mandates explainability and bias audits for credit-scoring models — banks shipping AI risk-scoring without that face supervisory action. SEBI’s 2025 circular on algo trading covers ML-driven trading desks. For every AI feature you ship, ask: does the data subject know? Is the decision contestable? Can you produce an audit trail? These are not just ethics questions — they are 2026 legal requirements for Indian operators.

Hands-on lab — your first AI threat model in 30 minutes

Pick any AI feature in your own product (or build a toy chatbot if you do not have one). Open a blank document and answer these in writing: (1) Inputs — what flows into the model? User text? Retrieved documents? Image uploads? Function call outputs? (2) Trust boundaries — for each input, who controls it? Authenticated users you trust 70%? Anonymous internet visitors? External systems via webhook? (3) Outputs and side effects — does the model just return text, or does it call tools that change state (send email, run SQL, post to Slack)? Anything with a side effect is your highest-priority risk. (4) Failure modes — for each combination of input × output, ask “what happens if the model is tricked?” Write down the worst plausible outcome. (5) Existing controls — what stops that outcome today? Rate limiting? Output validation? Human review? If the answer is “we trust the model,” you have an unmitigated risk. This exercise is what serious AI security teams do every quarter. Do it once and you immediately understand why agent systems get most of the attention — they have many inputs, broad outputs, and high-stakes side effects.

India 2026 — what regulators, customers, and journalists actually ask

When an Indian enterprise buys your AI feature, three audiences will probe it. The CISO wants the OWASP LLM Top 10 mapped to your controls, evidence of red-teaming, and a SOC 2 / ISO 27001 statement that explicitly covers AI workloads (most do not yet). The DPO / privacy lead wants to know: are personal data flows documented, do you have a DPIA for the AI use case, can data subjects exercise their DPDP rights when the model has memorised their data, and where exactly does inference happen geographically? The journalist (or X / LinkedIn critic) wants to know: have you tested for bias against Indian-language users, low-income users, or specific castes / religions? In 2024-2025, multiple Indian fintechs faced public criticism after their AI credit scoring underweighted applicants from particular regions. Build your AI governance documentation around these three audiences from day one — it is dramatically cheaper than retrofitting under a regulator deadline. The CERT-In incident reporting clock is 6 hours; you cannot draft a policy in time when an incident hits.

Career path — how to actually break into AI security in India

Realistic 12-month plan for an existing security professional. Months 1-3: ML literacy. Andrew Ng’s “Machine Learning Specialisation” (Coursera, ~3 months part-time). Read “Hands-On Large Language Models” (O’Reilly, 2024). Build muscle for the language: tokens, embeddings, fine-tuning, RAG. Months 4-6: AI security specifics. OWASP LLM Top 10 documentation, MITRE ATLAS framework, HackTheBox AI Red Teamer track, Lakera Gandalf challenges. Self-host an Ollama setup at home and try every attack you read about. Document each attempt — that becomes your portfolio. Months 7-9: Specialise. Pick a vertical: red-teaming, MLOps security, AI compliance, defensive AI. Contribute to one open-source project (garak probes, NeMo Guardrails policies, custom Llama Guard fine-tunes). Publish 3-5 deep blog posts on real attacks you reproduced. Months 10-12: Apply. Indian companies hiring AI security in 2026: TCS, Infosys, Wipro, RingSafe, Lucideus, Microsoft India, Google India, Amazon India, every Indian unicorn. Salary expectations: ₹25-60 LPA for Senior; ₹40-90 LPA for Staff/Principal. Remote international roles (often via Turing / Crossover): $100k-$250k. The talent pool is small, demand is large; well-prepared candidates have leverage. Skip generic certs unless your employer pays; portfolio + open-source contributions move the needle more than letters after your name.

Quick reference — frameworks, tools, and starter commands

Bookmark these. Frameworks — OWASP LLM Top 10 (genai.owasp.org), MITRE ATLAS (atlas.mitre.org), NIST AI Risk Management Framework (nist.gov/itl/ai-risk-management-framework), Anthropic Responsible Scaling Policy (anthropic.com/news/anthropics-responsible-scaling-policy), Google Secure AI Framework (safety.google/cybersecurity-advancements/saif). Open-source defensive tools — garak (github.com/leondz/garak), PyRIT (github.com/Azure/PyRIT), llm-guard (github.com/protectai/llm-guard), NeMo Guardrails (github.com/NVIDIA/NeMo-Guardrails), Llama Guard 2 / 3 on Hugging Face. Commercial — Lakera Guard, Protect AI Recon, HiddenLayer, Robust Intelligence (now Cisco). Free practice — Lakera Gandalf (gandalf.lakera.ai), HackTheBox AI Red Teamer track, PortSwigger LLM attacks lab, DEF CON AI Village CTFs (annual). Get-started commands: pip install garak; python -m garak --model_type huggingface --model_name meta-llama/Llama-3.2-3B-Instruct --probes promptinject. Reading order for new joiners: (1) OWASP LLM Top 10 cover-to-cover, 2 hours; (2) Greshake et al. “Indirect Prompt Injection” paper, 1 hour; (3) Anthropic “Sleeper Agents” paper, 2 hours; (4) Simon Willison’s blog archive on LLM tradecraft, 4 hours; (5) one weekend with garak + a local Ollama setup, 6-8 hours. By Monday after that weekend, you have working knowledge that puts you in the top quartile of practitioners. Indian context resources: CERT-In advisories archive, MeitY publications, IndiaAI mission docs (indiaai.gov.in), DPDP Act 2023 full text + 2024 rules. Ongoing signal: subscribe to “ai-safety” tag on arXiv, follow @anthropicai / @openai / @googledeepmind security teams on X, join the OWASP LLM Slack workspace.

FAQ

Is AI security just regular appsec with extra steps?

No. The threat model is fundamentally different — non-deterministic outputs, training-time supply chain, in-context learning meaning the model behaviour can shift mid-conversation. Appsec skills transfer (you still need authn, authz, logging, rate limiting), but you also need ML literacy, statistical thinking, and prompt-engineering tradecraft.

Where should a beginner start practising?

Three free resources: HackTheBox AI Red Teamer track (covers prompt injection, jailbreaks, MITRE ATLAS), Gandalf by Lakera (interactive prompt-injection puzzle game with 7 levels), and OWASP LLM Top 10 official docs. Then run a local Ollama setup and try to break it — the fastest learning path is hands-on.

Do I need to know machine learning math?

For 90% of AI security work — no. You need ML literacy (what training is, what fine-tuning does, what tokens are, why temperature matters), not the linear algebra. The other 10% — adversarial example generation against deep learning classifiers, model extraction — does require understanding gradients. Start with literacy; pick up math when you specialise.

How does AI security overlap with traditional appsec roles?

Heavy overlap. Authn, authz, rate limiting, logging, secrets management, supply chain — all still required. AI adds: prompt-engineering literacy, ML-pipeline awareness, statistical thinking about failure rates, and a new threat catalogue (OWASP LLM, MITRE ATLAS). A senior appsec engineer can pick up AI security in 3-6 months of focused effort.

Should we hire AI security specialists or train existing security staff?

Train existing staff for the next 18 months. The specialist talent pool is tiny and expensive; most teams need 1-2 people with deep AI knowledge plus the broader team upskilled to “AI literate.” Send your senior appsec engineer to the HackTheBox AI Red Teamer track, give them 4 hours/week of dedicated learning time, and they will be productive within a quarter.

⚖️ Legal: Use AI security techniques only on systems you own or have explicit written authorisation to test. In India, unauthorised access is punishable under IT Act §66 (up to 3 years + fine). Pair AI red-teaming with signed Statement of Work or Rules of Engagement before testing.