ai-system-design-guide README

项目的 README,来自 github.com/ombharatiya/ai-system-design-guide。合并为一个 markdown 文件:README.md。

🧠 AI System Design Guide

The Complete Interview & Production Reference

Read the guide online at aidaddy.tech Book a 1:1 mock interview on EngineBogie Book mentorship on Topmate

🌐 Instant search at aidaddy.tech. 🎯 1:1 mock interviews and mentorship with Om. ⭐ Star the repo to support the work.

Follow on GitHub Follow on Twitter Connect on LinkedIn

If this guide helps you, follow @ombharatiya on GitHub, X, and LinkedIn to get notified when new chapters, model refreshes, and interview prompts ship.

Last commit License PRs Welcome Stars Contributors Open issues

The living reference for production AI systems. Continuously updated. Interview-ready depth.

A practical, continuously updated guide to AI system design, RAG architectures, LLM engineering, agentic AI, MCP and A2A protocols, and AI engineering interview preparation. Covers production patterns, model selection, evaluation, and real-world case studies from staff-level interviews.

New here? Jump to the 193-question Interview Bank, the RAG Fundamentals chapter, or pick the right LLM for production.


📚 Quick Navigation

I want to... Start here
Prepare for interviews Question Bank → Answer Frameworks
Practice with a real interviewer Book a 1:1 mock interview or mentorship (also on Topmate)
Learn AI systems fast LLM Internals → RAG Fundamentals
Build production RAG Chunking → Vector DBs → Reranking → Production RAG
Advanced retrieval Contextual Retrieval → ColBERT → Multi-modal RAG
Design multi-tenant AI Access Control → Case Study
Build agents Agent Fundamentals → MCP & A2A → LangGraph
Run self-driving agent loops Loop Engineering (the four loop levels, termination, budgets, verification, loopmaxxing)
Tool-use & computer agents Landscape → OpenClaw → Safety
Autonomous coding agents Claude Code → OpenCoder Landscape
Survive framework version churn Navigating Framework Churn (stale tutorials, version pinning, what to actually learn)
Pick the right model (2026) Model Taxonomy → Pricing
Evaluate AI in production AI Evals Guide (Phoenix/Langfuse) → AI Evals Guide (LangWatch/Langfuse)
Read benchmarks the right way Benchmarks & Leaderboards (saturation, contamination, harness variance)
Track frontier research (2026) Research Radar (trending papers and what to learn next)
Build a voice agent Real-Time Voice Agents (pipeline vs speech-to-speech vs duplex front end, latency, cost per minute) → Voice AI Questions
Route across models / add a gateway AI Gateways and Model Routing (fallback, rate limits, LiteLLM)
Control AI cost FinOps and Token Economics (caching, batch, attribution, unit economics)
Meet AI regulations AI Governance and Compliance (EU AI Act, NIST RMF, what to implement)
Generate images, video, audio Multimodal Generation (pipelines, provenance, evaluation)
Train a reasoning model RLVR and GRPO (how o-series and R1 are trained)
Run models locally On-Device and Edge Deployment (Ollama vs vLLM, quantization, hardware)
Make agents crash-proof Durable Execution (replay, exactly-once, Temporal)
Engineer the data layer Data Engineering for AI (ingestion, dedup, PII, decontamination)
Find the best courses to learn AI Recommended Courses & Learning Paths
Transition from my current role to AI Role Transition Guide
Understand the 2026 AI job market Job Market Trends
Get a quick answer to a common question FAQ (RAG, agents, models, eval, inference, memory, security)
Look up a term Glossary (every term defined)

Pick a path

flowchart TD
    A[New visitor] --> B{Your goal}
    B -->|Interview prep| C[Question Bank]
    B -->|Build RAG| D[RAG Fundamentals]
    B -->|Build agents| E[Agent Fundamentals]
    B -->|Pick a model| F[Model Taxonomy]
    B -->|Evaluate AI| G[AI Evals Guide]
    C --> H[Answer Frameworks]
    D --> I[Chunking + Vector DBs]
    E --> J[MCP and Tool Use]
    F --> K[Pricing 2026]
    G --> L[Phoenix or LangWatch]

🎯 Why This Guide

Traditional books are outdated before they ship. This is a living document: when new models release, when patterns evolve, this updates.

This Guide Printed Books
November 2026 models (Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1, GPT-6 Astra/Sol/Luna, GPT-6.1 Sol/Terra, Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Muse Wonder, Muse Spark 1.3, DeepSeek V4.1-Flash, Mistral Large 4, MiMo-V2.6, GLM-5.3, Kimi K3.1 Pro, Qwen3.8) Stuck on GPT-4
MCP spec 2026-07-28, A2A v1.0.1, OpenClaw, Computer Use, full-duplex voice agents, Agentic RAG, ColBERT, latent reasoning, MoE serving Does not exist
Real pricing with November 2026 verification dates Already wrong
Staff-level interview Q&A (193 questions through November 2026, including Voice AI and rounds from recent candidate reports) + Job Market Trends Generic questions

Quick model picker (November 2026):

Full breakdown in Model Taxonomy and the Model Selection Guide.


🎯 What This Guide Is (and Is Not)

This guide IS:

This guide is NOT:


📖 Guide Structure

├── 00-interview-prep/           # Questions (193), frameworks, exercises, job-market trends (November 2026)
├── 01-foundations/              # Transformers, attention, embeddings
├── 02-model-landscape/          # Claude Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1, GPT-6 and 6.1, Gemini 4 Argon and 3.8, Grok 4.7, DeepSeek V4.1, open weights, pricing
├── 03-training-and-adaptation/  # Fine-tuning, LoRA, DPO, distillation, RLVR/GRPO
├── 04-inference-optimization/   # KV cache, PagedAttention, vLLM, diffusion LLMs, on-device
├── 05-prompting-and-context/    # Prompt engineering, CoT, Extended Thinking, DSPy, prompt injection
├── 06-retrieval-systems/        # RAG, chunking, GraphRAG, Agentic RAG, ColBERT, Contextual Retrieval, data engineering
├── 07-agentic-systems/          # MCP (spec 2026-07-28), A2A v1.0.1, multi-agent, computer-use, durable execution, loop engineering
├── 08-memory-and-state/         # L1-L3 memory tiers, Mem0, caching
├── 09-frameworks-and-tools/     # LangGraph, DSPy, LlamaIndex, Claude Code, OpenCoder, framework churn
├── 10-document-processing/      # Vision-LLM OCR, multimodal parsing
├── 11-infrastructure-and-mlops/ # GPU clusters, LLMOps, AI gateways, FinOps and cost
├── 12-security-and-access/      # RBAC, ABAC, multi-tenant isolation
├── 13-reliability-and-safety/   # Guardrails, red-teaming, AI governance and compliance
├── 14-evaluation-and-observability/ # RAGAS, LangSmith, benchmarks & leaderboards, drift detection
├── 15-ai-design-patterns/       # Pattern catalog, anti-patterns
├── 16-case-studies/             # Real-world architectures with diagrams
├── 17-tool-use-and-computer-agents/ # OpenClaw, Computer Use, tool agents, safety
├── 18-voice-and-audio-agents/   # Voice agents: pipeline, speech-to-speech, duplex front ends, turn-taking, cost per minute
├── 19-multimodal-generation/    # Image/video/audio generation: pipelines, provenance, evaluation
├── GLOSSARY.md                  # Every term defined
│
├── ai_evals_comprehensive_study_guide.md      # 🔬 Deep-dive: AI Evals (Phoenix + Langfuse)
├── ai_evals_complete_guide_langwatch_langfuse.md  # 🔬 Deep-dive: AI Evals (LangWatch + Langfuse)
├── COURSES.md                   # 🎓 Recommended courses & learning paths
├── TRANSITION_GUIDE.md          # 🔄 Transition from Backend/QA/PM/EM to AI roles
└── RESEARCH-RADAR.md            # 🛰️ Frontier research radar: trending papers and what to learn next

Chapters by AI System Lifecycle Stage

mindmap
  root((AI System Design Guide))
    Foundations
      LLM Internals
      Model Landscape
      Training and Adaptation
    Build
      Prompting and Context
      Retrieval Systems
      Agentic Systems
      Tool Use and Computer Agents
      Voice and Audio Agents
      Multimodal Generation
    Operate
      Inference Optimization
      Memory and State
      Frameworks and Tools
      Infrastructure and MLOps
    Govern
      Security and Access
      Reliability and Safety
      Evaluation and Observability
    Apply
      Design Patterns
      Case Studies
      Interview Prep

Real interview problems with complete solutions and diagrams:

Case Study Problem Key Patterns
Real-Time Search 5-minute data freshness at scale Streaming + Hybrid Search
Coding Agent Autonomous multi-file changes Sandboxing + Self-Correction
Multi-Tenant SaaS Coca-Cola and Pepsi on same infra Defense-in-Depth Isolation
Customer Support 60% auto-resolution rate Tiered Routing + Escalation
Document Intelligence 50K contracts/month extraction Vision-LLM + Parallel Extractors
Recommendation Engine Personalized explanations at 50M users ML Ranking + LLM Explanations
Compliance Automation FDA regulation pre-screening Claim Extraction + Precedent DB
Voice Healthcare Real-time clinical note generation ASR Under a BAA + Grounded Notes + Nurse Review
Fraud Detection 100ms decision with explainability ML + Rules Hybrid
Knowledge Management 2M docs with access control Permission-Aware RAG
Computer-Use Agent Expense-report automation across 3 legacy UIs Firecracker VMs + Action Gate + IPI Defense
Multi-Tenant Fine-Tuning 280 tenants on shared base + per-tenant LoRA LoRA Hot-Swap + Eval-as-PRD per Tenant
Eval-Gated CI/CD Block PRs that regress AI quality Golden Sets + LLM Judges + Statistical Correction
Customer Distillation Cut a $50K/mo frontier bill to about $9K/mo with under-3-month payback Trace-Based Distillation + Canary Rollout
MCP Knowledge Agent Cross-system answers from Snowflake/Confluence/Jira/Slack MCP + OAuth Resource Server + Capability Gating

🔬 Bonus Deep-Dive Guides

Two companion guides (3,000+ lines each) cover AI evaluation end to end for engineers, PMs, and QAs:

Guide Platforms Covered What's Inside
AI Evals: Comprehensive Study Guide Arize Phoenix + Langfuse LLM-as-a-Judge, RAG eval, multi-turn eval, production safety, statistical correction with judgy, 30-day learning path
AI Evals: LangWatch + Langfuse Guide LangWatch + Langfuse Same syllabus with LangWatch's 30+ ready-made evaluators, side-by-side platform comparisons, platform choice guidance

Topics covered across both guides:


🎓 For Interview Prep

AI engineering and system design interviews ask questions like:

"Design a multi-tenant RAG system where competitors cannot see each other's data."

"Your agent takes 15 steps for a 3-step task. How do you debug it?"

This guide gives you concrete patterns, real tradeoffs, and production failure modes: the depth interviewers expect at senior levels.

➡️ Start with Interview Prep

Want a practice run before the real loop? Reading strong answers builds knowledge, but the loop tests something else: explaining a design out loud, under a clock, while someone pushes back on your tradeoffs. Om runs 1:1 mock AI system design interviews, answer and resume reviews, and longer-term mentorship for people moving into senior AI roles. Book on EngineBogie or Topmate.

Book a mock interview on EngineBogie Book mentorship on Topmate


❓ Frequently Asked Questions

What is AI system design?

AI system design is the discipline of architecting production-grade systems built around LLMs, retrieval, agents, and evaluation. It covers model selection, RAG pipelines, agent orchestration, memory, observability, and safety. See LLM Internals and AI Design Patterns to get oriented.

How do I prepare for an AI engineering interview?

Start with the Question Bank (193 questions through November 2026), then practice with Answer Frameworks and Whiteboard Exercises. Most senior interviews test RAG design, agent debugging, multi-tenant isolation, and cost/latency tradeoffs, all covered in the Case Studies. Before a real loop, do at least one timed mock with someone who will interrupt and push back; you can book one with Om on EngineBogie or Topmate.

What is RAG (Retrieval-Augmented Generation)?

RAG is a pattern where an LLM retrieves relevant context from an external knowledge source (vector DB, search index, graph) before generating an answer, reducing hallucinations and grounding responses in your data. The full pipeline is covered in RAG Fundamentals and scaled in Production RAG at Scale.

What are AI agents and how are they different from chatbots?

AI agents are LLM-driven systems that plan, call tools, and act over multiple steps to accomplish goals, whereas chatbots typically respond in a single turn. Agents introduce loops, memory, error recovery, and tool-use via protocols like MCP. Start with Agent Fundamentals.

What is MCP (Model Context Protocol) and how does it compare to A2A?

MCP is an open protocol that lets LLM applications discover and call external tools and data sources in a standard way. A2A (Agent2Agent) is the complementary protocol for handing work to an agent you do not control: the remote agent publishes an Agent Card at /.well-known/agent-card.json, the client sends it a message, and the remote agent decides whether to open a long-running task. They solve different layers: MCP is the tool boundary, A2A is the agent boundary, and most systems need MCP long before they need A2A. Both now sit under the Linux Foundation's Agentic AI Foundation (A2A joined in August 2026). Cite versions precisely in interviews: MCP's spec ships as dated revisions, the current one (2026-07-28) made the protocol core stateless, the next is planned for March 30, 2027, and there is no "MCP 2.0"; A2A's latest spec release is v1.0.1. See Tool Use and MCP.

Which LLM should I use in production: Claude, GPT, Gemini, or open-source?

It depends on latency budget, context length, cost per task (tokens per task times price, not price per token), tool-use quality, and data residency. A sensible November 2026 starting point: Claude Opus 5.5 ($4/$20 per 1M) for agentic coding and hard work, the $2/$10 tier (Claude Sonnet 5.5, GPT-6 Sol, GPT-6.1 Sol) for most production traffic, GPT-6.1 Terra ($1/$5) wherever it holds your quality bar, Claude Haiku 5.5, GPT-6 Luna, or Gemini 3.8 Flash for volume (Haiku 5.5 only on prompts under 100K tokens, where its $0.10/$0.50 price applies), and Claude Fable 5.1 or GPT-6 Astra ($10/$50) only when your evals show the cheaper tiers falling short. The Model Taxonomy and Pricing chapters compare these plus Gemini 4 Argon, Grok 4.7, Muse Spark 1.3, DeepSeek V4.1-Flash, and open-weight models such as MiMo-V2.6, GLM-5.3, Kimi K3.1 Pro (weights since November 4, 2026), and Mistral Large 4, and the Model Selection Guide turns them into a decision tree. Run your own evals before switching: vendor scores do not carry across harnesses or effort settings.

How do I evaluate an LLM or RAG system in production?

Combine offline evals (LLM-as-a-judge with ground-truth calibration), online metrics (faithfulness, context recall, answer relevance), and continuous tracing. The companion deep-dives AI Evals: Phoenix + Langfuse and AI Evals: LangWatch + Langfuse walk through this end-to-end.

How do I build a multi-tenant RAG system safely?

Use defense-in-depth: per-tenant indexes or namespaces, query-time access checks, and prompt-layer guards. The Access Control chapter and Multi-Tenant SaaS Case Study cover the patterns that hold up in interviews and production.

What is agentic RAG?

Agentic RAG combines retrieval with an agent loop that can decide what to search, when to re-query, and when to escalate, instead of running a single fixed retrieve-then-generate pass. See Agentic RAG for the architectures and tradeoffs.

Is this guide free? Can I contribute?

Yes, MIT-licensed and free. PRs are welcome; see Contributing Guide. If you have production failure modes, new model benchmarks, or interview questions to add, open a PR.

How often is this guide updated?

Continuously. New model releases, protocol changes (MCP, A2A), and emerging patterns are added as they ship. The November 2026 refresh brought models, prices, protocols, and framework versions up to their state on November 6, 2026. Recent additions include twelve more questions from recent interview reports (Q174-Q185: security-telemetry agents, a permission-matched code explorer, devboxes, review ingestion under prompt injection, agentic retail search, a required-AI design round), eight November 2026 questions (Q186-Q193: per-end-customer enforcement isolation, server-side message threads, a multi-vendor deprecation program, cyber access programs, open-weight license review, regraded leaderboards, fail-open tenant isolation, agent approval controls), eighteen questions from recent interview reports (Q156-Q173: design-doc review, model distribution, GPU scheduling, KV caches and tokenizers from scratch, agents without a framework, AI-assisted coding rounds, launch readiness, safety screens), eight October 2026 questions (Q148-Q155: agent monitoring, incident classification, gateway supply chain, fine-tune migration, price cliffs, decision models, API-version rollout, patch SLAs), a twelve-question Voice AI section, seven September 2026 questions, a tenth whiteboard exercise, a reworked Real-Time Voice Agents chapter, and the November 2026 Job Market Trends.

Can I use this guide if I am transitioning from backend, QA, PM, or EM into AI?

Yes. The Role Transition Guide maps existing skills to AI engineering, MLE, and AI architect tracks, with reading paths per role. Pair it with COURSES.md for curated learning resources.


🔄 Living Book

This guide tracks:

⭐ Star and Watch the repo to get notified when updates are pushed.


🤝 Contributing

Found outdated info? Have production experience to share? PRs welcome. See Contributing Guide.


👋 Stay Connected

If this guide helps you, the easiest way to support it is to follow along where new chapters and refreshes get announced first:

Follow on GitHub Follow on Twitter Connect on LinkedIn


📄 License

MIT License. See LICENSE.


Built and maintained by Om Bharatiya · GitHub · Twitter · LinkedIn

ai-system-design-guide (ombharatiya/ai-system-design-guide)

正在搜索数据库…

这是什么?一份持续更新的参考指南,涵盖 AI 系统设计、RAG 架构、LLM 工程、智能体 AI、MCP/A2A 协议、评测(evals)以及面向生产级 AI 系统工程师的面试准备。
这是一个可以动手操作的工具,还是只用来阅读的内容?它是一份持续更新的 Markdown 参考文档,由多个章节、精选案例研究、一个面试题库,以及两份超过 3000 行的评测深度指南组成——不是软件,也不是分步课程。
我该怎么开始使用?使用 README 中的“快速导航”表格,它会根据你的目标把你带到对应章节,例如包含 147 道题目的面试题库、RAG 基础,或模型分类(Model Taxonomy)。
这里会从零开始教我 Python 或机器学习基础吗?不会,它默认你已经掌握基础知识;这不是 Python、PyTorch 或机器学习入门教程,初学者会被引导到单独的 COURSES.md 文件。
这份指南适合谁?正在设计生产级 AI 系统的工程师(涉及 RAG、智能体、多租户隔离、评测流水线),以及正在准备 staff 级别 AI 系统设计面试的人。
随着新模型发布,内容会保持更新吗?会的,它被明确描述为一份随模型和协议变化而更新的动态文档;当前版本涵盖 2026 年 10 月的模型,如 Claude Opus 5.5 和 GPT-6 Sol,并附有经过核实的定价。
这份指南包含多少道面试题?截至 2026 年 9 月共 147 道题,其中包括专门的语音 AI(Voice AI)题组,另有答题框架和十个白板练习。
我能和真人一起进行模拟面试吗?可以,作者 Om 提供付费的一对一 AI 系统设计模拟面试、简历和答题审阅,以及长期辅导,可通过 EngineBogie 或 Topmate 预约。
它能帮我选择在生产环境中使用哪个 LLM 吗?可以,它包含模型分类(Model Taxonomy)和定期更新的快速模型选择指南,按旗舰、中端、低价和开放权重模型比较价格与许可注意事项。
大小
4.3 MB
星标
4.4K
更新
提交Nov 10
时长
11个月
提交 · 90 天