tech . 🎯 1:1 mock interviews and mentorship with Om. ⭐ Star the repo to support the work.
ai-system-design-guide (ombharatiya/ai-system-design-guide)
Searching Database…
ai-system-design-guide README
The project's README from github.com/ombharatiya/ai-system-design-guide. As one markdown file: README.md.
🧠 AI System Design Guide
The Complete Interview & Production Reference
🌐 Instant search at aidaddy.tech. 🎯 1:1 mock interviews and mentorship with Om. ⭐ Star the repo to support the work.
If this guide helps you, follow @ombharatiya on GitHub, X, and LinkedIn to get notified when new chapters, model refreshes, and interview prompts ship.
The living reference for production AI systems. Continuously updated. Interview-ready depth.
A practical, continuously updated guide to AI system design, RAG architectures, LLM engineering, agentic AI, MCP and A2A protocols, and AI engineering interview preparation. Covers production patterns, model selection, evaluation, and real-world case studies from staff-level interviews.
New here? Jump to the 147-question Interview Bank, the RAG Fundamentals chapter, or pick the right LLM for production.
📚 Quick Navigation
| I want to... | Start here |
|---|---|
| Prepare for interviews | Question Bank → Answer Frameworks |
| Practice with a real interviewer | Book a 1:1 mock interview or mentorship (also on Topmate) |
| Learn AI systems fast | LLM Internals → RAG Fundamentals |
| Build production RAG | Chunking → Vector DBs → Reranking → Production RAG |
| Advanced retrieval | Contextual Retrieval → ColBERT → Multi-modal RAG |
| Design multi-tenant AI | Access Control → Case Study |
| Build agents | Agent Fundamentals → MCP & A2A → LangGraph |
| Run self-driving agent loops | Loop Engineering (the four loop levels, termination, budgets, verification, loopmaxxing) |
| Tool-use & computer agents | Landscape → OpenClaw → Safety |
| Autonomous coding agents | Claude Code → OpenCoder Landscape |
| Survive framework version churn | Navigating Framework Churn (stale tutorials, version pinning, what to actually learn) |
| Pick the right model (2026) | Model Taxonomy → Pricing |
| Evaluate AI in production | AI Evals Guide (Phoenix/Langfuse) → AI Evals Guide (LangWatch/Langfuse) |
| Read benchmarks the right way | Benchmarks & Leaderboards (saturation, contamination, harness variance) |
| Track frontier research (2026) | Research Radar (trending papers and what to learn next) |
| Build a voice agent | Real-Time Voice Agents (pipeline vs speech-to-speech vs duplex front end, latency, cost per minute) → Voice AI Questions |
| Route across models / add a gateway | AI Gateways and Model Routing (fallback, rate limits, LiteLLM) |
| Control AI cost | FinOps and Token Economics (caching, batch, attribution, unit economics) |
| Meet AI regulations | AI Governance and Compliance (EU AI Act, NIST RMF, what to implement) |
| Generate images, video, audio | Multimodal Generation (pipelines, provenance, evaluation) |
| Train a reasoning model | RLVR and GRPO (how o-series and R1 are trained) |
| Run models locally | On-Device and Edge Deployment (Ollama vs vLLM, quantization, hardware) |
| Make agents crash-proof | Durable Execution (replay, exactly-once, Temporal) |
| Engineer the data layer | Data Engineering for AI (ingestion, dedup, PII, decontamination) |
| Find the best courses to learn AI | Recommended Courses & Learning Paths |
| Transition from my current role to AI | Role Transition Guide |
| Understand the 2026 AI job market | Job Market Trends |
| Get a quick answer to a common question | FAQ (RAG, agents, models, eval, inference, memory, security) |
| Look up a term | Glossary (every term defined) |
Pick a path
flowchart TD
A[New visitor] --> B{Your goal}
B -->|Interview prep| C[Question Bank]
B -->|Build RAG| D[RAG Fundamentals]
B -->|Build agents| E[Agent Fundamentals]
B -->|Pick a model| F[Model Taxonomy]
B -->|Evaluate AI| G[AI Evals Guide]
C --> H[Answer Frameworks]
D --> I[Chunking + Vector DBs]
E --> J[MCP and Tool Use]
F --> K[Pricing 2026]
G --> L[Phoenix or LangWatch]
🎯 Why This Guide
Traditional books are outdated before they ship. This is a living document: when new models release, when patterns evolve, this updates.
| This Guide | Printed Books |
|---|---|
| October 2026 models (Claude Opus 5.5, Sonnet 5.5, Fable 5.1, GPT-6 Astra/Sol/Luna, GPT-6.1 Sol, Gemini 3.8 Flash, Grok 4.7, Muse Spark 1.3, DeepSeek V4.1-Flash, MiMo-V2.6, GLM-5.3, Kimi K3, Qwen3.8) | Stuck on GPT-4 |
| MCP spec 2026-07-28, A2A v1.0.1, OpenClaw, Computer Use, full-duplex voice agents, Agentic RAG, ColBERT, latent reasoning, MoE serving | Does not exist |
| Real pricing with October 2026 verification dates | Already wrong |
| Staff-level interview Q&A (147 questions through September 2026, including Voice AI) + Job Market Trends | Generic questions |
Quick model picker (October 2026):
- Agentic coding and hard production work: Claude Opus 5.5 ($4/$20 per 1M), the model Anthropic now tells customers to start with and the top scorer on the Artificial Analysis Intelligence Index v4.3.2.
- Production default: the mid tier has converged on $2/$10 with Claude Sonnet 5.5, GPT-6 Sol, and GPT-6.1 Sol. Route agent fleets, chat, and general workloads here.
- Capability ceiling: Claude Fable 5.1 or GPT-6 Astra ($10/$50), only when your evals show Opus 5.5 at high effort falling short.
- Volume: GPT-6 Luna ($0.10/$0.50); Gemini 3.8 Flash at a half-price $0.75/$3.75 through December 31, then $1.50/$7.50, so budget at list; DeepSeek V4.1-Flash for batch work you can schedule off-peak, when it costs half its $0.30/$1.20 peak price.
- Open weights: Xiaomi MiMo-V2.6-Pro (MIT) leads, with GLM-5.3 and Kimi K3 close behind. Read the license first: GLM-5.3 adds a security review only for MaaS operators above US$10B in annual revenue, but Kimi K3 requires a separate agreement for any MaaS operator above US$20M in revenue over 12 months (with affiliates).
- Before you switch: OpenAI bills the whole request at long-context rates above 272K input tokens, the newest Claude models return a 400 on forced
tool_choice, and Opus 5.5 defaults tomediumeffort where Opus 5 usedhigh. Gemini 4 Argon was announced September 30 but is not in the Gemini API yet.
Full breakdown in Model Taxonomy and the Model Selection Guide.
🎯 What This Guide Is (and Is Not)
This guide IS:
- A staff-level reference for designing production AI systems (RAG, agents, MCP, eval pipelines, multi-tenant isolation).
- An interview-prep companion with 147 real questions (including a Voice AI set), answer frameworks with a worked mock transcript, and ten whiteboard exercises through September 2026.
- A living document tracking new model releases, protocol changes, and emerging patterns as they ship.
- Opinionated about tradeoffs: latency vs cost, accuracy vs faithfulness, single-agent vs multi-agent.
- Free, MIT-licensed, and open to PRs from practitioners.
This guide is NOT:
- A tutorial on Python, PyTorch, or basic ML fundamentals (start with a course; see COURSES.md).
- A vendor-neutral hedge; it names specific models, prices, and frameworks because real systems require real choices.
- A replacement for hands-on building; read it alongside a project, not instead of one.
- A research paper digest; it cites papers when they change practice, not for completeness.
📖 Guide Structure
├── 00-interview-prep/ # Questions (147), frameworks, exercises, job-market trends (October 2026)
├── 01-foundations/ # Transformers, attention, embeddings
├── 02-model-landscape/ # Claude Opus 5.5, Sonnet 5.5, Fable 5.1, GPT-6, Gemini 3.8, Grok 4.7, DeepSeek V4.1, open weights, pricing
├── 03-training-and-adaptation/ # Fine-tuning, LoRA, DPO, distillation, RLVR/GRPO
├── 04-inference-optimization/ # KV cache, PagedAttention, vLLM, diffusion LLMs, on-device
├── 05-prompting-and-context/ # Prompt engineering, CoT, Extended Thinking, DSPy, prompt injection
├── 06-retrieval-systems/ # RAG, chunking, GraphRAG, Agentic RAG, ColBERT, Contextual Retrieval, data engineering
├── 07-agentic-systems/ # MCP (spec 2026-07-28), A2A v1.0.1, multi-agent, computer-use, durable execution, loop engineering
├── 08-memory-and-state/ # L1-L3 memory tiers, Mem0, caching
├── 09-frameworks-and-tools/ # LangGraph, DSPy, LlamaIndex, Claude Code, OpenCoder, framework churn
├── 10-document-processing/ # Vision-LLM OCR, multimodal parsing
├── 11-infrastructure-and-mlops/ # GPU clusters, LLMOps, AI gateways, FinOps and cost
├── 12-security-and-access/ # RBAC, ABAC, multi-tenant isolation
├── 13-reliability-and-safety/ # Guardrails, red-teaming, AI governance and compliance
├── 14-evaluation-and-observability/ # RAGAS, LangSmith, benchmarks & leaderboards, drift detection
├── 15-ai-design-patterns/ # Pattern catalog, anti-patterns
├── 16-case-studies/ # Real-world architectures with diagrams
├── 17-tool-use-and-computer-agents/ # OpenClaw, Computer Use, tool agents, safety
├── 18-voice-and-audio-agents/ # Voice agents: pipeline, speech-to-speech, duplex front ends, turn-taking, cost per minute
├── 19-multimodal-generation/ # Image/video/audio generation: pipelines, provenance, evaluation
├── GLOSSARY.md # Every term defined
│
├── ai_evals_comprehensive_study_guide.md # 🔬 Deep-dive: AI Evals (Phoenix + Langfuse)
├── ai_evals_complete_guide_langwatch_langfuse.md # 🔬 Deep-dive: AI Evals (LangWatch + Langfuse)
├── COURSES.md # 🎓 Recommended courses & learning paths
├── TRANSITION_GUIDE.md # 🔄 Transition from Backend/QA/PM/EM to AI roles
└── RESEARCH-RADAR.md # 🛰️ Frontier research radar: trending papers and what to learn next
Chapters by AI System Lifecycle Stage
mindmap
root((AI System Design Guide))
Foundations
LLM Internals
Model Landscape
Training and Adaptation
Build
Prompting and Context
Retrieval Systems
Agentic Systems
Tool Use and Computer Agents
Voice and Audio Agents
Multimodal Generation
Operate
Inference Optimization
Memory and State
Frameworks and Tools
Infrastructure and MLOps
Govern
Security and Access
Reliability and Safety
Evaluation and Observability
Apply
Design Patterns
Case Studies
Interview Prep
🔥 Featured Case Studies
Real interview problems with complete solutions and diagrams:
| Case Study | Problem | Key Patterns |
|---|---|---|
| Real-Time Search | 5-minute data freshness at scale | Streaming + Hybrid Search |
| Coding Agent | Autonomous multi-file changes | Sandboxing + Self-Correction |
| Multi-Tenant SaaS | Coca-Cola and Pepsi on same infra | Defense-in-Depth Isolation |
| Customer Support | 60% auto-resolution rate | Tiered Routing + Escalation |
| Document Intelligence | 50K contracts/month extraction | Vision-LLM + Parallel Extractors |
| Recommendation Engine | Personalized explanations at 50M users | ML Ranking + LLM Explanations |
| Compliance Automation | FDA regulation pre-screening | Claim Extraction + Precedent DB |
| Voice Healthcare | Real-time clinical note generation | ASR Under a BAA + Grounded Notes + Nurse Review |
| Fraud Detection | 100ms decision with explainability | ML + Rules Hybrid |
| Knowledge Management | 2M docs with access control | Permission-Aware RAG |
| Computer-Use Agent | Expense-report automation across 3 legacy UIs | Firecracker VMs + Action Gate + IPI Defense |
| Multi-Tenant Fine-Tuning | 280 tenants on shared base + per-tenant LoRA | LoRA Hot-Swap + Eval-as-PRD per Tenant |
| Eval-Gated CI/CD | Block PRs that regress AI quality | Golden Sets + LLM Judges + Statistical Correction |
| Customer Distillation | Cut a $50K/mo frontier bill to about $9K/mo with under-3-month payback | Trace-Based Distillation + Canary Rollout |
| MCP Knowledge Agent | Cross-system answers from Snowflake/Confluence/Jira/Slack | MCP + OAuth Resource Server + Capability Gating |
🔬 Bonus Deep-Dive Guides
Two companion guides (3,000+ lines each) cover AI evaluation end to end for engineers, PMs, and QAs:
| Guide | Platforms Covered | What's Inside |
|---|---|---|
| AI Evals: Comprehensive Study Guide | Arize Phoenix + Langfuse | LLM-as-a-Judge, RAG eval, multi-turn eval, production safety, statistical correction with judgy, 30-day learning path |
| AI Evals: LangWatch + Langfuse Guide | LangWatch + Langfuse | Same syllabus with LangWatch's 30+ ready-made evaluators, side-by-side platform comparisons, platform choice guidance |
Topics covered across both guides:
- Tracing and observability setup (Phoenix, LangWatch, Langfuse)
- Error analysis: open coding → axial coding → failure mode taxonomy
- Building LLM judges with Train/Dev/Test split and ground truth calibration
- Code-based evaluators (regex, JSON schema, format validators)
- RAG-specific evals: faithfulness, context recall, answer relevance
- Multi-step pipeline evaluation and multi-turn conversation eval
- Production guardrails, safety monitoring, real-time drift detection
- Statistical correction with
judgylibrary - Human annotation best practices and inter-rater reliability
- Cost/latency optimization for eval pipelines at scale
🎓 For Interview Prep
AI engineering and system design interviews ask questions like:
"Design a multi-tenant RAG system where competitors cannot see each other's data."
"Your agent takes 15 steps for a 3-step task. How do you debug it?"
This guide gives you concrete patterns, real tradeoffs, and production failure modes: the depth interviewers expect at senior levels.
➡️ Start with Interview Prep
Want a practice run before the real loop? Reading strong answers builds knowledge, but the loop tests something else: explaining a design out loud, under a clock, while someone pushes back on your tradeoffs. Om runs 1:1 mock AI system design interviews, answer and resume reviews, and longer-term mentorship for people moving into senior AI roles. Book on EngineBogie or Topmate.
❓ Frequently Asked Questions
What is AI system design?
AI system design is the discipline of architecting production-grade systems built around LLMs, retrieval, agents, and evaluation. It covers model selection, RAG pipelines, agent orchestration, memory, observability, and safety. See LLM Internals and AI Design Patterns to get oriented.
How do I prepare for an AI engineering interview?
Start with the Question Bank (147 questions through September 2026), then practice with Answer Frameworks and Whiteboard Exercises. Most senior interviews test RAG design, agent debugging, multi-tenant isolation, and cost/latency tradeoffs, all covered in the Case Studies. Before a real loop, do at least one timed mock with someone who will interrupt and push back; you can book one with Om on EngineBogie or Topmate.
What is RAG (Retrieval-Augmented Generation)?
RAG is a pattern where an LLM retrieves relevant context from an external knowledge source (vector DB, search index, graph) before generating an answer, reducing hallucinations and grounding responses in your data. The full pipeline is covered in RAG Fundamentals and scaled in Production RAG at Scale.
What are AI agents and how are they different from chatbots?
AI agents are LLM-driven systems that plan, call tools, and act over multiple steps to accomplish goals, whereas chatbots typically respond in a single turn. Agents introduce loops, memory, error recovery, and tool-use via protocols like MCP. Start with Agent Fundamentals.
What is MCP (Model Context Protocol) and how does it compare to A2A?
MCP is an open protocol that lets LLM applications discover and call external tools and data sources in a standard way. A2A (Agent2Agent) is the complementary protocol for handing work to an agent you do not control: the remote agent publishes an Agent Card at /.well-known/agent-card.json, the client sends it a message, and the remote agent decides whether to open a long-running task. They solve different layers: MCP is the tool boundary, A2A is the agent boundary, and most systems need MCP long before they need A2A. Both now sit under the Linux Foundation's Agentic AI Foundation (A2A joined in August 2026). Cite versions precisely in interviews: MCP's spec ships as dated revisions, the current one (2026-07-28) made the protocol core stateless, and there is no "MCP 2.0"; A2A's latest spec release is v1.0.1. See Tool Use and MCP.
Which LLM should I use in production: Claude, GPT, Gemini, or open-source?
It depends on latency budget, context length, cost per task (tokens per task times price, not price per token), tool-use quality, and data residency. A sensible October 2026 starting point: Claude Opus 5.5 ($4/$20 per 1M) for agentic coding and hard work, the $2/$10 tier (Claude Sonnet 5.5, GPT-6 Sol, GPT-6.1 Sol) for most production traffic, GPT-6 Luna or Gemini 3.8 Flash for volume, and Claude Fable 5.1 or GPT-6 Astra ($10/$50) only when your evals show the cheaper tiers falling short. The Model Taxonomy and Pricing chapters compare these plus Grok 4.7, Muse Spark 1.3, DeepSeek V4.1-Flash, and open-weight models such as MiMo-V2.6, GLM-5.3, and Kimi K3, and the Model Selection Guide turns them into a decision tree. Run your own evals before switching: vendor scores do not carry across harnesses or effort settings.
How do I evaluate an LLM or RAG system in production?
Combine offline evals (LLM-as-a-judge with ground-truth calibration), online metrics (faithfulness, context recall, answer relevance), and continuous tracing. The companion deep-dives AI Evals: Phoenix + Langfuse and AI Evals: LangWatch + Langfuse walk through this end-to-end.
How do I build a multi-tenant RAG system safely?
Use defense-in-depth: per-tenant indexes or namespaces, query-time access checks, and prompt-layer guards. The Access Control chapter and Multi-Tenant SaaS Case Study cover the patterns that hold up in interviews and production.
What is agentic RAG?
Agentic RAG combines retrieval with an agent loop that can decide what to search, when to re-query, and when to escalate, instead of running a single fixed retrieve-then-generate pass. See Agentic RAG for the architectures and tradeoffs.
Is this guide free? Can I contribute?
Yes, MIT-licensed and free. PRs are welcome; see Contributing Guide. If you have production failure modes, new model benchmarks, or interview questions to add, open a PR.
How often is this guide updated?
Continuously. New model releases, protocol changes (MCP, A2A), and emerging patterns are added as they ship. The October 2026 refresh brought models, prices, protocols, and framework versions up to their state on October 1, 2026. Recent additions include a twelve-question Voice AI section, seven September 2026 questions, a tenth whiteboard exercise, a reworked Real-Time Voice Agents chapter, and the October 2026 Job Market Trends.
Can I use this guide if I am transitioning from backend, QA, PM, or EM into AI?
Yes. The Role Transition Guide maps existing skills to AI engineering, MLE, and AI architect tracks, with reading paths per role. Pair it with COURSES.md for curated learning resources.
🔄 Living Book
This guide tracks:
- New model releases and real-world performance
- Emerging patterns (MCP, Agentic RAG, Flow Engineering)
- Updated pricing and rate limits
- Deprecations and best practice changes
⭐ Star and Watch the repo to get notified when updates are pushed.
🤝 Contributing
Found outdated info? Have production experience to share? PRs welcome. See Contributing Guide.
👋 Stay Connected
If this guide helps you, the easiest way to support it is to follow along where new chapters and refreshes get announced first:
- Website: aidaddy.tech - read the full guide with search, clean navigation, and a mobile-friendly layout.
- GitHub: @ombharatiya - follow for the repo, star the project, and watch for new releases.
- X / Twitter: @ombharatiya - short takes on model releases, MCP, agents, and interviews.
- LinkedIn: ombharatiya - deeper writeups and interview prep tips for senior AI roles.
- Mock interviews and mentorship: EngineBogie or Topmate - book a 1:1 mock AI system design loop, a review of your answers, or ongoing mentorship for a move into AI roles.
📄 License
MIT License. See LICENSE.
Built and maintained by Om Bharatiya · GitHub · Twitter · LinkedIn