Building a Production AI Stack — Vector DB, LLM, Auth, Observability

Most “AI app” tutorials show a one-file script that calls OpenAI. Production needs more. This module is the actual architecture you will run, the choices that scale, and the observability that lets you debug at 2am.

Reference architecture

Layer by layer. Edge / CDN: Cloudflare in front for rate limit, WAF, geo-restriction. Application backend: Next.js or FastAPI on Vercel/Railway/Fly.io. Handles auth, request validation, prompt construction. LLM gateway: LiteLLM or Helicone proxy. Routes between providers (OpenAI/Anthropic/self-hosted). Adds caching, retries, fallback. Vector DB: Qdrant (self-hosted) or Pinecone (managed) for RAG. Embedding model: BGE-M3 (self-hosted) or OpenAI text-embedding-3-small. Cache: Redis for prompt-response cache and rate limit counters. Database: Postgres for user data, conversation history, audit logs. Observability: Langfuse (open source) or LangSmith for AI traces; Sentry for errors; Grafana for metrics.

Auth and tenancy

Use a managed auth service. Supabase Auth, Clerk, or NextAuth.js. JWT-based sessions; refresh tokens for long-lived clients. Multi-tenancy isolation matters for B2B: every database query, vector retrieval, and audit-log write must include tenant_id. Test cross-tenant isolation with a dedicated penetration test on your auth surface. Pattern: middleware on every API route extracts tenant_id from JWT, attaches to request context; downstream code MUST use it. RLS in Postgres for defence-in-depth. For consumer apps with single user, the same patterns simplify but do not skip — your roadmap probably includes B2B someday.

The LLM gateway pattern

Do NOT call OpenAI directly from your application. Insert a gateway. Why: (1) Provider switching — swap GPT-4o for Claude or Llama via config; useful for cost optimisation and outage resilience. (2) Caching — same prompt + context returns cached response (10x cost reduction on repeated questions). (3) Retries — providers have 5xx; gateway handles backoff. (4) Logging — single point to instrument all LLM calls. (5) Fallback — primary fails → fallback to secondary. LiteLLM (open source) is the gateway most teams converge on. pip install litellm; litellm --model openai/gpt-4o --port 4000. Your app calls http://localhost:4000/v1/chat/completions; gateway handles the rest.

Observability — what you actually need

Three classes of telemetry. (1) Application: HTTP requests, latency, errors. Standard APM (Sentry, Datadog). (2) LLM-specific: per request — model, version, prompt template, token counts, cost, latency, finish reason, moderation outcome. NEVER log full prompt content in production — log hash + first 50 chars. Tools: Langfuse (open source, self-hostable), Helicone (managed), LangSmith (LangChain). (3) Business metrics: queries per user, completion rate, satisfaction (helpful-feedback widget), conversion funnels. Connect to your analytics (PostHog, Amplitude, GA4). Dashboards to build: cost-per-user, P95 latency, error rate by user-cohort, prompt-version A/B. Set alerts on anomalies (cost spike = bug or abuse).

Cost optimisation patterns

Five techniques that compound. (1) Cascading models — try cheap GPT-4o-mini first; escalate to GPT-4o only if confidence is low. Saves 60-80%. (2) Prompt caching — Anthropic and OpenAI cache repeated context; for chat with long system prompt, cuts input cost 75%. (3) Embeddings cache — embeddings are deterministic per (model, text); cache forever. Save repeated embedding API calls. (4) Quantised self-hosted for high-volume routes — own a 4080 ($1200), serve common queries locally, fall back to API for hard cases. Break-even at ~$1500/mo API spend. (5) Streaming with early stop — let user stop early; UI feedback often suffices in fewer tokens.

Disaster planning — what breaks and what to do

AI dependencies have unique failure modes. (1) Provider outage — OpenAI/Anthropic regularly have hour-long incidents. Multi-provider via gateway with automatic fallback. (2) Rate-limit tier breach — successful product hits Tier limits, throttling kicks in. Pre-arrange Tier upgrades; have fallback provider with capacity. (3) Cost runaway — abuse or buggy code burns $10K in hours. Spend caps in OpenAI dashboard; budget alerts at 50/80/100% in your gateway. (4) Model deprecation — providers deprecate models with 30-90 days notice; need migration plan. Pin model version in code; treat upgrades as release events. (5) Compliance event — DPDP regulator inquiry. Need: full audit log, prompt versions, training data provenance. Build this from day 1; impossible to retrofit.

Deployment recipe — minimum viable production

For a small team launching: Frontend: Next.js on Vercel, Tailwind + shadcn UI. Backend: Same Next.js app router for API; Postgres on Supabase or Neon (free tier ample). Auth: Supabase Auth or Clerk. LLM gateway: LiteLLM Docker on Fly.io (~$5/mo). Vector DB: pgvector in same Postgres (no extra service). Cache: Upstash Redis (free tier). Observability: Langfuse self-hosted on Fly.io ($5/mo) or Langfuse Cloud free tier. Total non-LLM infra: ~$10-30/month at MVP scale. LLM cost depends on usage — set spend caps. This stack handles 100s of concurrent users; outgrows around the 10K MAU mark.

Reference architecture — full-stack AI app, $0 to $10k MRR

For India-based AI startups, the cheapest production-grade architecture in 2026: Frontend Next.js 15 on Vercel free tier or Cloudflare Pages. Backend Hono / FastAPI on Railway / Fly.io ($5-20/mo). Database Supabase (Postgres + auth + storage, free tier covers 50k MAUs). Vector DB pgvector inside the same Supabase Postgres — free, no second system. LLM Anthropic Claude Haiku 3.5 via API (cheap, fast, good); or Llama 3.3 70B via Together AI ($0.88/M tokens). Embeddings OpenAI text-embedding-3-small ($0.02/M tokens) or BGE via Together. Cache Upstash Redis free tier. Auth Supabase Auth (Google + email + magic link). Observability Langfuse self-hosted on Fly.io OR Langfuse cloud free tier. Email Resend free tier. Payments Razorpay (India-native, supports UPI). Total infra cost at zero customers: $0. At 1k MAU: $50-150/mo. At 10k MAU paying customers: $300-1,000/mo. Time from blank repo to demo: 2-4 weekends for an experienced builder.

Scaling — what breaks first and what to do

Predictable failure ladder as you scale. 10-100 users: nothing breaks. 100-1k users: cold-start latency on serverless functions; consider edge runtime or always-on machines. 1k-10k users: LLM cost dominates; introduce caching layers and consider self-hosting smaller workloads. 10k-100k users: vector DB performance degrades; migrate from pgvector to a dedicated vector DB (Turbopuffer, Pinecone, Qdrant) or shard pgvector by tenant. 100k+ users: queue management, multi-region for latency, full observability stack with PII-aware logging, dedicated SRE on-call. The biggest mistake teams make: over-engineering at the 10-user stage. The architecture above gets you to 10k users without redesign; do not pre-build for the 1M-user case. Premature optimisation in AI startups burns runway. Move when the data tells you to.

Operational maturity ladder — assess your AI stack honestly

Self-assessment rubric for production AI systems. Score 0/1 on each; total <5 = early; 5-10 = mid; 11-15 = mature; 16+ = best-in-class. Foundation (5): (1) authentication on every AI endpoint; (2) per-user rate limiting; (3) cost cap per user + global; (4) request and response logging with 90-day retention; (5) basic monitoring (uptime, latency p95, error rate). Quality (5): (6) automated eval suite that runs on every prompt change; (7) regression alerts when eval scores drop; (8) human-rated quality samples reviewed weekly; (9) prompt and model versioning (you can roll back); (10) A/B testing infrastructure for prompt or model changes. Security (5): (11) input filtering (Lakera/Llama Guard/NeMo); (12) output filtering for harmful or PII content; (13) prompt-injection regression suite; (14) red-teaming run quarterly; (15) data-flow documentation reviewed by DPO. Operational (5): (16) feature flag for global AI disable (kill-switch); (17) multi-provider failover via gateway; (18) per-tenant isolation verified; (19) on-call runbook for AI-specific incidents; (20) post-incident review process. Most production AI in 2026 scores 5-10 (foundation + partial quality). 11-15 is where regulated workloads should live. 16-20 is where mission-critical AI lives. The point of the rubric: not to judge, but to expose blindspots. A team scoring 4/20 building “AI for healthcare” should know they are not yet ready; a team scoring 18/20 building “AI for marketing copy” is over-investing. Match maturity to stakes.

Component selection cheat sheet — 2026 picks for each layer

Curated picks for each architectural layer with rationale. Frontend: Next.js 15 (industry default for React SaaS); add Vercel AI SDK for streaming. Backend: Hono (TypeScript, edge-friendly) or FastAPI (Python, ML-native ecosystem). Pick by team language. Auth: Supabase Auth (free, all standard providers, RLS integration) or Clerk ($25/mo+, slicker DX, MFA built in). Database: Supabase Postgres (free tier 500MB) or Neon (free tier 0.5GB, branching). Vector DB: pgvector inside the Postgres above, until 10M+ vectors. LLM: Anthropic Claude Haiku 4.5 ($0.25/$1.25 per M, fast, good); Anthropic Sonnet 4.6 ($3/$15, best general); OpenAI GPT-4o-mini ($0.15/$0.60, cheap); Llama 3.3 70B via Together ($0.88/$0.88, open source). Embeddings: OpenAI text-embedding-3-small ($0.02/M), or Cohere multilingual for Indic. LLM gateway: LiteLLM (open source, multi-provider, prompt caching) or Portkey (commercial, slick UX). Cache: Upstash Redis (free tier 256MB, serverless-friendly). Queue: Inngest (great DX) or Trigger.dev or BullMQ + Redis. Observability: Langfuse (open source, self-host or cloud free tier) — most important non-obvious tool to add early. Analytics: Posthog (open source, comprehensive, free for <1M events/month). Errors: Sentry (free tier generous). Email: Resend (free tier 100/day, devex-focused). Payments: Razorpay for India; Stripe for international. Hosting: Vercel for app (Hobby = $0, Pro = $20/mo per dev); Fly.io or Modal for backend workers. Total monthly cost MVP: $0-50. Total monthly cost at $10k MRR: $200-800. Stack rebuilds in 2026 are cheap: pick this, ship, swap pieces as scale demands. Premature stack debate kills more startups than tech choices.

FAQ

Should I use LangChain or build directly?

For prototypes: LangChain or LlamaIndex save time. For production: most teams hit the abstraction wall and rewrite to direct API calls + LiteLLM. Recommendation: prototype with LangChain, productionise with direct calls + LiteLLM gateway.

Postgres pgvector vs dedicated vector DB?

pgvector for ≤10M vectors and you already have Postgres. Dedicated (Qdrant/Pinecone) for 100M+ vectors, complex filtering, or sub-100ms recall. Most apps live forever in pgvector territory.

Self-host or use OpenAI for an Indian B2B app?

Hybrid. Use OpenAI/Anthropic for non-sensitive features (search, classification). Self-host (Llama via vLLM) for queries containing customer data. Document the data-flow split for DPDP audit. Most enterprises end here in 2026.

Should I use Supabase or build my own auth?

Supabase. In 2026, rolling your own auth is hard to justify. Supabase auth handles email + OAuth + MFA + magic links + JWT issuance, integrates with RLS for fine-grained access, costs $0-25/mo for typical startups. Auth0/Clerk/WorkOS are all valid; pick what your team knows. Building auth yourself wastes months for marginal gain.

When should I move off pgvector?

When p95 query latency exceeds 100ms with proper indexing (HNSW) on optimised hardware, or your data exceeds ~10M vectors. Below that, pgvector is competitive. The migration to a dedicated vector DB is straightforward (export embeddings, re-index) — defer until you actually have the problem.

⚖️ Legal: Use AI security techniques only on systems you own or have explicit written authorisation to test. In India, unauthorised access is punishable under IT Act §66 (up to 3 years + fine). Pair AI red-teaming with signed Statement of Work or Rules of Engagement before testing.