Defending AI Endpoints — Rate Limit, Content Filters, NeMo Guardrails, Llama Guard

Defence in depth applies to AI endpoints just as to any production service. The layers are different from web apps but the principle holds — assume any single layer can be bypassed; multiple layers raise the cost of a successful attack to non-trivial.

Layer 1 — WAF and edge protection

Cloudflare in front of every public AI endpoint. Configure: (1) bot management — block known scraper bots; (2) geo-block by region if your app is regional (DPDP-Indian apps may legitimately block traffic from sanctioned jurisdictions); (3) rate limit at edge — 100 req/min per IP for /api/chat; (4) WAF rules to block obvious abuse patterns (SQLi, traversal, etc. — even though your LLM endpoint does not have SQL, reduces noise). Cost: $20/month Cloudflare Pro. Stops 80% of automated abuse before it reaches your origin.

Layer 2 — Application rate limit and quotas

Per-user rate limits. Token-budget quotas. Implement in your application middleware, backed by Redis counters. upstash-ratelimit for serverless or limiter-js for Node. Pattern: free users get 100 messages/day, 10/min burst; paid users 1000/day, 30/min. Token quotas backed by Stripe usage records for paid tiers. Critical: rate limit BEFORE calling the LLM, not after — the cost has already been incurred. Track abuse signals: repeated rate-limit hits, payments declining, IP rotation patterns. Auto-suspend accounts hitting >3 thresholds in 24 hours pending human review.

Layer 3 — Input moderation

Before sending user input to the main LLM, run it through a moderation model. Options: (1) OpenAI Moderation API — free, fast (~50ms), catches harmful-content categories (violence, sexual, hate). Limited language coverage. (2) Llama Guard 3 (Meta) — open source, self-hostable, finer-grained taxonomy. Slower (~300ms on CPU). (3) llm-guard library — combines multiple scanners (PII detection via regex+ML, prompt-injection signatures, secret detection, language detection). Good comprehensive coverage. Handle false positives gracefully — return helpful error not just “blocked”; users will be wrongly flagged.

Layer 4 — NeMo Guardrails for dialogue policy

For chatbots with constrained scope, NeMo Guardrails (NVIDIA, open source) enforces policy at the conversation level. Define in YAML: which topics are in/out of scope, what tone to use, what to do on disallowed queries. Example use cases: customer support bot must only discuss your products (off-topic queries get polite redirect); medical app cannot give diagnoses (must defer to professional). Strength: declarative, auditable, version-controllable. Weakness: too restrictive for open-ended assistants; significant latency overhead. Right tool for narrow chatbots; wrong tool for general-purpose assistants.

Layer 5 — Output moderation and PII redaction

Run model output through moderation before returning to user. Catches: (1) safety violations the model produced despite system prompt; (2) PII leakage from training data or RAG context; (3) secrets the model regurgitated from training data (API keys, passwords). Tools: same as input — OpenAI Moderation, Llama Guard, llm-guard. PII redaction tools: presidio (Microsoft), scrubadub, custom regex for India-specific (PAN: [A-Z]{5}\d{4}[A-Z], Aadhaar: \d{4}\s?\d{4}\s?\d{4}). Latency cost: ~300ms; usually parallel with response streaming if implemented correctly.

Layer 6 — Audit log and post-hoc detection

Even with all defences, some abuse gets through. Audit logs let you detect after the fact and ban / patch. Log per request: user_id, timestamp, model, input hash, output hash, moderation flags, cost, latency. Run anomaly detection: users with abnormal cost-per-day, unusually diverse query topics (sign of extraction), repeated flagged content. Daily report to security team. Quarterly: sample 0.1% of conversations for manual review (with appropriate privacy controls). Retention: 90 days for security review; user-deletable on request per DPDP. This layer pays for itself the first time you catch a serious abuser; otherwise it satisfies compliance.

Defence in depth — the 5-layer pattern for production AI endpoints

Layer 1: Edge — Cloudflare WAF rules + bot management filter the worst (script kiddies, common exploit kits, scrapers). Reduces ~80% of garbage at zero per-request cost. Layer 2: API gateway — per-user rate limit, request size cap, content-type validation. Use Upstash for Redis-backed rate limit; Hono/FastAPI middleware for input validation. Layer 3: Pre-LLM filter — Llama Guard 2 or NVIDIA NeMo Guardrails or Lakera scan inputs for known prompt-injection patterns and policy violations. ~20-100ms latency, ~$0.0001 per call. Layer 4: Constrained LLM call — careful system prompt with delimiters, output schema enforcement (function calling with strict JSON schema reduces injection-induced free-form attacks). Layer 5: Post-LLM filter — Llama Guard or NeMo on output before returning to user. Catches: PII leak, harmful content, off-topic abuse. Belt and suspenders. Total added latency: 50-200ms. Total added cost: $0.0005-0.005 per request. Risk reduction: order of magnitude. Skip any layer and the failure mode is predictable.

NeMo Guardrails vs Llama Guard vs Lakera — pick one

NVIDIA NeMo Guardrails: open source, programmable rule language (Colang), heavy but flexible. Best for teams with custom policies. Self-hosted; ~50ms overhead per check. Meta Llama Guard 2: open-source 8B classifier model fine-tuned for harmful-content + prompt-injection detection. Easier to deploy; runs on a single GPU; ~20-50ms inference. Best default for self-hosted stacks. Lakera Guard: SaaS, paid, easy integration via API call. Continuously updated against latest attack patterns. Best when you need quality without operational burden; budget ~$0.0001-0.001 per check. For Indian regulated workloads with data-residency concerns: prefer self-hosted Llama Guard or NeMo. For SaaS startups optimising for time-to-market: Lakera. All three reduce attack success rates 5-10x; pick based on your operational profile.

Cost-aware filter strategy — when each layer is worth the latency

Defence layers add latency and cost; not every app needs every layer. Decision matrix by endpoint type. Internal employee tools, low blast radius (knowledge search, document chat): edge filter only. Output filter optional. Skip pre-LLM filter; trust authenticated employees. Total overhead: ~10ms, ~$0. Customer-facing free-tier chatbot (public marketing assistant, lead-gen): edge + lightweight pre-LLM filter (regex + small classifier) + output PII filter. Skip heavyweight Llama Guard for free tier. Total overhead: ~30ms, ~$0.0001/request. Customer-facing paid product (support automation, content generation): full stack — edge + Llama Guard input + careful prompt + Llama Guard output + audit log. Total overhead: ~100ms, ~$0.0005/request. Regulated / high-stakes (healthcare diagnosis assist, financial advice, legal analysis): everything above + secondary classifier on outputs (independent model voting) + human-in-the-loop sampling (1-5%) + immutable audit log + DPIA documentation. Total overhead: ~200ms, ~$0.002/request. The framework: estimate value per request × success-attack-cost = expected loss without filters. Compare against (filter cost + filter latency-quality cost) per request. If filtering pays for itself in expected losses avoided, deploy it. For most regulated workloads, the math obviously favours full filtering. For low-stakes internal tools, less so. The pattern most teams get wrong: deploying full filters everywhere (slows experience, frustrates users) or no filters anywhere (catches the wrong incident). Match defences to stakes per endpoint.

Reference configurations — Llama Guard, NeMo Guardrails, Lakera Guard

Llama Guard 3 setup. Pull: ollama pull llama-guard3:8b. Use: response = ollama.chat(model="llama-guard3", messages=[{"role":"user", "content": user_input}]); if "unsafe" in response['message']['content'].lower(): block(user_input). Categories: violence, sexual content, criminal planning, weapons, hate, self-harm, sexual/minors, election manipulation, code interpreter abuse, defamation, privacy, intellectual property — 12 hazard types in v3. Customise: fine-tune on your specific policy (llama-guard-customizer available on HF). NeMo Guardrails setup. Install: pip install nemoguardrails. Define rails in Colang: define user ask off topic; define bot refuse off topic; define flow; user ask off topic; bot refuse off topic. Wrap LLM calls: rails = LLMRails(config); response = rails.generate(messages=[{"role":"user","content": q}]). Powerful but complex; budget time for Colang learning. Lakera Guard (commercial API). Setup: POST https://api.lakera.ai/v1/guard with Bearer LAKERA_KEY; pass user input; receive flagged categories. Latency 50-150ms; cost $0.0001-0.001 / call depending on tier. Good for teams without ops bandwidth for self-hosting. Combination pattern: Llama Guard for hard policy enforcement (block); Lakera for nuanced detection (flag); NeMo for complex multi-step conversation flows. Most production stacks pick one + custom rules. Per-category thresholds: violence 0.8, sexual 0.5, privacy 0.3, off-topic 0.6 — calibrate per product; default thresholds are not always right. Bypass-resistant config: run guard on EVERY message (user input AND model output AND tool outputs); never skip “trusted” inputs; assume adversaries probe. Operational: log every block with full context; review weekly to catch false positives + missed attacks; refine continuously.

FAQ

Is OpenAI Moderation enough?

For B2C apps targeting US/EU: usually yes for clear safety violations. For India / multilingual: weak on Indian-language content; supplement with custom classifiers. For B2B with sensitive data: need PII detection on top — Moderation does not flag PII.

Should I use Llama Guard locally vs OpenAI Moderation?

Llama Guard for self-sovereignty (DPDP, on-prem). OpenAI Moderation for free + low-latency at small scale. Hybrid: OpenAI for first-pass; Llama Guard for high-stakes paths.

How much latency do all these layers add?

Realistic budget: 100ms WAF + rate limit, 300ms input moderation (parallel possible), LLM call (1-3s), 300ms output moderation. Total overhead ~500-800ms over the LLM call itself. For chat UX this is acceptable since streaming is used. For real-time agents (voice, gaming), need to optimise (parallel scanners, async logging).

How much latency does input + output filtering add?

Realistic: 50-150ms total at p95 if running on a colocated GPU; 100-300ms if calling a hosted service. Streaming UX hides the input latency (filter while displaying typing indicator). Output filtering is harder to hide; can run progressively on chunks but adds complexity.

Should I run filters in series or parallel?

Input filter must run before LLM (sequential). Multiple input filters can run in parallel via async I/O. Output filter must run on full response, so it is sequential after LLM (or progressive on chunks). Optimise the LLM path as the dominant cost; filter overhead is usually 10-20% of total latency.

⚖️ Legal: Use AI security techniques only on systems you own or have explicit written authorisation to test. In India, unauthorised access is punishable under IT Act §66 (up to 3 years + fine). Pair AI red-teaming with signed Statement of Work or Rules of Engagement before testing.