ai-system-design README

The project's README from github.com/amitshekhariitbhu/ai-system-design. As one markdown file: README.md.

AI System Design - Learn how to design AI systems built on LLMs, RAG, and AI Agents step by step

AI System Design

AI System Design - A complete guide to learn AI System Design step by step - from LLM inference, GPUs, KV Cache, and caching to RAG, Vector Databases, AI Agents, MCP, Multi-Agent Systems, Voice AI, Guardrails, Evaluation, Observability, Cost Optimization, and a step-by-step framework to crack any AI System Design interview. Everything in one place, explained in simple words, with detailed blogs for every deep dive.

This AI System Design guide is helpful for anyone who wants to become:

  • AI Engineer
  • Gen AI Engineer
  • LLM Engineer
  • Agentic AI Engineer
  • AI Agent Engineer
  • Forward Deployed Engineer
  • AI Solutions Architect
  • AI Platform Engineer
  • Applied AI Engineer
  • Machine Learning Engineer
  • MLOps Engineer
  • LLMOps Engineer
  • Backend Engineer building AI products

Prepared and maintained by the Founder of Outcome School: Amit Shekhar

Follow Amit Shekhar

Follow Outcome School

I teach at Outcome School

Note: AI System Design is moving very fast, so this guide will continue to grow as I write more blogs on new topics. Bookmark it and come back whenever you want a refresher. Keep learning.

Table of Contents

About This AI System Design Guide

In this guide, we will learn about AI System Design, the discipline of putting GPUs, inference servers, caches, vector databases, AI agents, gateways, guardrails, and evals together into one system that is fast, cheap, reliable, and safe. We will also see how an LLM actually runs on a GPU, how prefill and decode shape latency, how caching, routing, and batching cut the cost, how RAG and AI Agents are built for production, how we keep the system safe and measurable, and a step-by-step framework to solve any AI System Design problem in an interview or in real production.

When we use a product like ChatGPT, Cursor, Perplexity, or Claude Code, we see a simple chat box and a streamed response. Behind that simple interface, there is a lot more happening - GPUs, inference servers, vector databases, agent loops, caches, gateways, guardrails, and a long list of design decisions that all have to work together.

This guide is everything we need in one place. We start with the basics like tokens and the inference server, build up through hardware, prefill and decode, scaling, caching, RAG, agentic systems, multi-agent systems, multimodal and voice systems, safety, observability, evaluation, and inference optimization, and finish with a step-by-step framework to solve any AI System Design problem.

Every section is:

This guide focuses on the System Design side of AI. To learn the complete AI Engineering path - Machine Learning, Deep Learning, Transformers, LLMs, Fine-Tuning, RAG, AI Agents, and more - check out the AI Engineering Course.

What is AI System Design?

AI System Design is the discipline of designing the complete system around an AI model, especially a Large Language Model (LLM), so that it can serve real users in a way that is fast, cheap, reliable, safe, and measurable.

In simple words:

AI System Design = System Design + The new constraints of AI models.

The new constraints are GPUs, tokens, long and streamed responses, non-deterministic output, and a real cost on every single request. AI System Design is how we design caches, queues, databases, gateways, retrieval, agents, guardrails, and evals around these constraints.

Let's say we want to build a customer support chatbot. Calling an LLM API in a script takes ten lines of code. But serving 100,000 users with an answer that starts in under a second, stays grounded in our own documents, never leaks private data, and fits within a monthly budget - that is AI System Design.

Who is This AI System Design Guide For?

This AI System Design guide is for:

What Will We Learn in This AI System Design Guide?

In this AI System Design guide, we will learn:

How to Use This AI System Design Guide

AI System Design Learning Path

flowchart TD
    A[Foundations: Tokens, Inference Server, Hardware] --> B[LLM Inference: Prefill and Decode]
    B --> C[Scaling, Estimation, and Load Balancing]
    C --> D[Caching and LLM Routing]
    D --> E[Embeddings, Vector Databases, and RAG]
    E --> F[Context Window Management]
    F --> G[Streaming, Queues, Rate Limiting, AI Gateway]
    G --> H[AI Agents, Tools, MCP, and Memory]
    H --> I[Multi-Agent Systems]
    I --> J[Multimodal and Voice AI]
    J --> K[Guardrails, Safety, and Privacy]
    K --> L[Observability and Evaluation]
    L --> M[Cost, Fine-Tuning, and Inference Optimization]
    M --> N[How to Solve Any AI System Design Problem]

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Why study AI System Design?

When most of us start with AI, we just pick an OpenAI or Anthropic API key, write a small script that calls the API, and get a response back. We feel that our AI app is built.

For a small project or a personal demo, this is good enough.

But the real world is very different.

In the real world, an AI product serves millions of users. Each user sends long prompts. Each response is streamed token by token. Models are slow. GPUs are expensive. Costs add up fast. Hallucinations creep in. Latency matters.

A single API call cannot handle all of this.

To make our AI product reliable, fast, cheap, and safe, we have to think about many things together. This is where AI System Design comes into the picture.

How is AI System Design different from regular System Design?

In regular system design, we deal with CPU, RAM, disk, databases, and network.

In AI System Design, we deal with all of these, and on top of that, we deal with:

So all the regular system design concepts (load balancers, caches, queues, databases) still apply. But we have to use them differently because the workload is different.

LLM Recap

Before we go into AI System Design, let's quickly recap what an LLM is.

LLM stands for Large Language Model. It is a model that takes some text as input and predicts the next token. It does this again and again until the full response is generated.

The text is not directly given to the model. It is first broken into smaller pieces called tokens. Think of it like a chocolate bar. The full bar is the sentence. Each small part we break off is a token. The model processes multiple tokens at a time. Most modern LLMs use BPE (Byte Pair Encoding) to do this tokenization. One token is roughly 4 characters in English. So short common words like "AI" or "Hello" are one token. Longer words like "Bangalore" are 2 tokens. Compound names like "ChatGPT" are 2 to 3 tokens depending on the tokenizer.

LLMs generate text one token at a time. This is called autoregressive generation, which means the model keeps feeding its own output back as the new input. Let's say we give the model this input:

"I love"

The model looks at "I" and "love", and predicts the next token: "teaching". The full sequence becomes "I love teaching". The model now looks at "I", "love", and "teaching" and predicts the next token: "AI". The full sequence becomes "I love teaching AI". This process continues, one token at a time, until the model decides to stop.

At every step, the model does not know the next token for sure. It gives a probability to every possible token, and then one token is picked. Settings like Temperature and Top-k and Top-p Sampling control how this pick happens. This is also why the same prompt can give different outputs, which is one of the biggest reasons AI System Design is different from regular System Design.

Internally, the LLM is a Transformer - a stack of attention and feed-forward layers. The single most important idea inside it is the attention mechanism, where each token converts itself into three vectors - Query (Q), Key (K), and Value (V) - and uses them to figure out which previous tokens matter most for predicting the next one. We have a detailed blog on the math behind Attention - Q, K, and V that goes into the math step by step.

When we use an API like OpenAI, Anthropic, or Google, we send a prompt, and we get a streamed response back, one token at a time.

Examples of LLMs: GPT-5.5, Claude Opus 4.8, Gemini 3.5, Llama 4, Mistral.

If we want to go deep into LLM internals - tokenization, positional encodings, Q/K/V matrices, attention, transformer architecture, KV Cache, Paged Attention, Mixture of Experts - the AI and Machine Learning Program by Outcome School covers all of these from scratch.

This is enough recap. Now, let's move to the actual System Design part.

The Big Picture

Before we go into the details, let's understand the big picture.

An AI system is just a regular system with one new component in the middle: the LLM. Everything else (load balancer, cache, database, queue) is still there. The LLM brings new constraints (slow, expensive, non-deterministic, streaming), and we have to design the rest of the system around those constraints.

In short, an AI system is a regular system with an LLM added in the middle, plus smarter caching, streaming, cost tracking, safety, and evals built around it.

If we keep this in mind, every AI architecture decision becomes easy to reason about.

Inference Server

In regular system design, a server runs our application code and returns a response.

In AI System Design, we have something called the Inference Server.

An inference server is a special kind of server that runs the LLM. It loads the model into the GPU memory and serves prediction requests.

Most teams do not write the inference server from scratch. They use famous open-source ones like:

When we use OpenAI or Anthropic API, we are not running the inference server. We are calling their inference server over the internet.

When we run our own LLM (like Llama or Mistral), we run vLLM or TGI on a GPU machine, and our backend calls this inference server.

So the architecture looks like below:

Client -> Backend Server -> Inference Server (with GPU) -> LLM

The inference server is the heaviest component of an AI system. It needs a GPU. GPUs are expensive. A single H100 GPU costs around $25,000 to $40,000 to buy (depending on the variant, like PCIe vs SXM, and vendor) or roughly $2 to $4 per hour to rent.

Here, the analogy is simple. A regular server is like a kitchen with a few cooks. An inference server is like a kitchen with one very expensive chef (the GPU). We do not want this chef sitting idle for even a second.

This is why in AI System Design, we put a lot of effort into:

Choosing an Inference Engine

Now, let's see which inference engine to use. There are four main inference engines we will see in production. Each one has a sweet spot.

Rule of thumb:

Now, we know which inference engine to use based on our use case.

AI Hardware: GPU, TPU, and LPU

The inference engine is the software. But the software runs on a chip, and the chip decides a lot of our speed and our cost. So, before jumping into latency, we must know the three chips that come up again and again in AI System Design.

Here, the most important lesson is this: while an LLM writes the answer, the bottleneck is memory, not math. To produce every single token, the chip must read the full model weights from memory. A GPU keeps the weights in HBM (High Bandwidth Memory), which sits next to the chip. An LPU keeps the weights in SRAM, inside the chip itself, so the data does not have to travel.

Let me tabulate the differences between a GPU and an LPU for your better understanding.

Point GPU LPU
Built for Any parallel math, training and inference Running a trained language model
Where the weights live HBM, next to the chip SRAM, inside the chip
Memory speed Around 3 TB per second Around 80 TB per second
Memory size per chip Tens of GB Around 230 MB
Chips for a large model A few Hundreds
Speed for a single user Moderate Very high
Batching Needed to get good total output Not needed to get good speed
Training support Yes No
Flexibility Very high Low, everything is planned ahead of time

So, based on our use case:

The bigger lesson goes beyond one chip. Once a workload becomes memory bound, the win comes from moving data less, not from computing more. We will see this same idea again in prefill vs decode, KV Cache, and Flash Attention.

Tokens, Latency, and Throughput

In regular system design, we measure latency in milliseconds and throughput in requests per second.

In AI, we have the same metrics, but we add a few new ones because of how LLMs generate output token by token.

Time to First Token (TTFT)

This is the time it takes from sending the prompt to receiving the very first token back.

Example: We send "Tell me a story" to ChatGPT. It takes 800 ms before the first word "Once" appears on screen. TTFT is 800 ms.

This matters a lot for user experience. If the first token shows up fast, the user feels the app is responsive even if the full answer takes 10 seconds to finish.

Tokens per Second (TPS)

This is the speed at which the model generates tokens after the first one.

Example: After the first token shows up, the model generates 50 tokens per second. So a 500-token response takes 10 seconds to fully appear.

This depends on the model size and the GPU. Smaller models are faster. Bigger models are slower but more accurate.

The same speed is often measured the other way around, as TPOT (Time Per Output Token), which is the time taken to produce each token after the first one. 50 tokens per second means a TPOT of 20 ms.

Throughput

This is the total number of tokens the inference server can produce per second across all users.

Example: If 100 users are using the system at the same time, and each one is getting 20 tokens per second, the total throughput is 2000 tokens per second.

A good AI system has:

These three are in tension with each other. If we batch many requests together, throughput goes up but TTFT goes up too. We will see this trade-off later in continuous batching.

Cost per Token

Every request costs money based on the number of tokens.

Modern LLM APIs now bill on three categories of tokens:

Example: Suppose a model charges $3 per 1 million input tokens and $15 per 1 million output tokens. If a user sends 1000 input tokens and gets 500 output tokens back, the cost of that one request is:

(1000 / 1,000,000) * $3 + (500 / 1,000,000) * $15
= $0.003 + $0.0075
= $0.0105 (about 1 cent)

If we are using a reasoning model, we have to add the cost of reasoning tokens too, which can be much larger than the visible output for hard prompts.

In AI System Design, we watch cost the same way we watch latency. Every dashboard, every alert, every postmortem includes the dollar number.

In short:

Prefill and Decode: The Two Phases of LLM Inference

Now, let's understand why TTFT and TPS behave so differently. The answer lies in how an LLM actually answers a request. Every request goes through two distinct phases.

In simple words, prefill is the model reading our question, and decode is the model writing the answer. We have a detailed blog on Prefill vs Decode that explains both phases step by step.

Both phases use the exact same model and the exact same weights. Only the way the two phases work is different, and that small difference changes everything.

Prefill

Prefill is the phase where the model reads and processes our entire input prompt in one single pass and produces the very first output token.

Let's say a student walks into an exam hall. Before writing anything, the student silently reads the entire question paper. This reading-the-whole-paper part is prefill.

Because the whole prompt is already known up front, the model can process every input token at the same time. While reading the prompt, it also computes and stores the Key (K) and Value (V) of every input token. All of these stored Keys and Values together form the KV Cache (we will learn about it in detail soon). At the end of prefill, the model produces the very first output token.

Prefill is compute-bound. It multiplies big tables of numbers with big tables of numbers, which is exactly what GPUs are built for. The math units of the GPU stay very busy, often approaching 90 percent or more on large prompts.

Longer prompts mean longer prefill. So, faster prefill means sooner we see the first token (lower TTFT).

Decode

Decode is the phase where the model generates the output tokens one at a time, reusing the KV Cache that prefill prepared.

Continuing our exam example, the student now starts writing the answer, word by word. This slow, word-by-word writing is decode.

In decode, only one token is processed per step. The math for one token is tiny. But to produce even that one token, the GPU must still read the entire model weights and the whole growing KV Cache from its memory. So, the GPU spends most of its time waiting for data to arrive, and its math units sit only around 20 to 40 percent busy for a single request.

Decode is memory-bandwidth-bound. Think of a chef who must walk into a giant pantry, grab one ingredient, walk back, chop it, and repeat, one ingredient per trip. The walking back and forth (moving data from memory) takes far longer than the chopping (the tiny math).

For a typical request, prefill finishes in a fraction of a second, but decode can run for several seconds. So, faster decode means faster we see the rest of the answer (lower TPOT).

Let me tabulate the differences between Prefill and Decode for your better understanding.

Aspect Prefill Decode
What it does Processes the whole input prompt Generates output tokens one by one
Parallelism Parallel (all prompt tokens at once) Sequential (one token at a time)
Steps One big step Many small steps
Bottleneck Compute-bound (limited by math speed) Memory-bandwidth-bound (limited by data movement)
GPU utilization High (math units busy, can approach 90 percent or more) Low (math units mostly idle, around 20 to 40 percent)
KV cache action Writes the cache Reads and extends the cache
Latency metric Time To First Token (TTFT) Time Per Output Token (TPOT)
Cost driver Grows with prompt length Streams full weights plus KV cache every step

This split explains a lot of AI System Design. It is why input tokens are cheaper than output tokens. It is why a long prompt makes the first token slow. And it is why so many optimizations focus on the decode phase.

Chunked Prefill

But, here is the catch. When prefill and decode run on the same GPU, they fight with each other. A very long prompt's prefill runs as one giant step. While that giant step runs, it hogs the GPU and freezes the token streaming of other users who are already in their decode phase. Their answers stutter.

Chunked prefill splits one long prompt's prefill into several smaller chunks, and slips small decode steps in between the chunks.

WITHOUT chunked prefill:
  long prompt: [============= ONE BIG PREFILL =============]
  other users: tok ....... (frozen, stuttering) ....... tok

WITH chunked prefill:
  long prompt: [chunk 1][chunk 2][chunk 3][chunk 4][chunk 5]
  other users: tok   tok   tok   tok   tok   tok
                  ^ a decode step slips in between the chunks

Here, we can see that one giant prefill freezes the other users, while chunked prefill lets their tokens keep flowing between the chunks. It is like a chef with one stove. If a customer orders a huge banquet, cooking it all at once freezes every other table's food. Instead, the chef cooks the banquet in small batches, slipping in the quick single dishes between them.

Note: Chunked prefill protects ongoing answers from stalling. It does not make prefill itself faster.

Prefill-Decode Disaggregation

At large scale, we can go one step further. So, here comes Prefill-Decode Disaggregation to the rescue.

Prefill-Decode Disaggregation = Prefill + Decode + Disaggregation, where disaggregation simply means separating things that were joined together.

Prefill-Decode Disaggregation is the technique of running the prefill phase and the decode phase on separate GPUs, so that they never fight for the same hardware.

In simple words, we have one group of GPUs whose only job is to read prompts, and another group of GPUs whose only job is to write answers. There are four main parts:

The flow looks like below:

Request
   |
   v
[Router] --> picks a Prefill worker and a Decode worker
   |
   v
[Prefill worker] --> reads the full prompt
                 --> builds the KV Cache
                 --> produces the first token
   |
   v  (KV Cache is sent over the network)
   |
[Decode worker]  --> loads the KV Cache
                 --> writes token 2, token 3, token 4 ...
   |
   v
Answer streamed to the user

Let's take one request with example numbers, just for the sake of understanding. A user sends a prompt of 4000 tokens and the answer will be 200 tokens long. The prefill worker processes all 4000 tokens in, say, 400 ms and produces the first token along with a KV Cache of, say, 2 GB. The user sees the answer start, so our TTFT is around 400 ms. The 2 GB KV Cache is copied to the decode worker in 20 to 50 ms. The decode worker then produces each token in, say, 20 ms, so the 200 tokens take about 4 seconds. The prefill worker has already moved on to the next user's prompt, and the decode worker is never interrupted by a big prefill.

So, the user gets a quick start (low TTFT) and a smooth flow (low TPOT), both at the same time.

Let me tabulate the differences between co-located serving and disaggregated serving for your better understanding so that you can decide which one to use based on your use case.

Point Co-located Serving Disaggregated Serving
Where prefill and decode run Same GPU Separate prefill and decode workers
Interference between phases Yes, prefill stalls decode No
TTFT and TPOT Trade-off, improving one hurts the other Both can be tuned separately
Scaling Scale both phases together Scale each phase independently
KV Cache transfer Not needed Needed for every request
Network requirement None Fast interconnect is a must
Complexity Low High
Best for Small scale, single GPU, short prompts Large scale, many users, long prompts

For a small setup, chunked prefill on a single GPU is simpler and good enough. For a large setup, disaggregation is the way to go. Serving systems like SGLang support it out of the box.

Scaling AI Systems

When traffic grows, we need to scale.

In regular system design, scaling means more CPU, more RAM, or more machines.

In AI System Design, scaling has more flavors because LLMs are huge. A model like Llama 70B does not even fit in a single GPU. We need to split the model across multiple GPUs.

Let's see the different ways to scale.

Vertical Scaling

Use a bigger machine. In AI, this means a bigger GPU.

Example: Move from an A100 (80 GB) to an H100 (80 GB but faster) or to an H200 (141 GB).

This works only up to a point. The biggest GPU on the market is still limited. Big models do not fit.

Horizontal Scaling

Add more GPU machines and put a load balancer in front of them.

Example: We have 4 inference servers, each running Llama 8B on a single A100. The load balancer distributes user requests across these servers.

This is the most common form of scaling. It works perfectly when the model fits in one GPU.

Tensor Parallelism

When the model does not fit in one GPU, we split the model itself across multiple GPUs.

Example: Llama 70B in 16-bit precision needs around 140 GB. A single A100 has 80 GB. So we split the model across 2 A100s. Each GPU holds half of the model weights. When a forward pass happens, the GPUs talk to each other very fast over NVLink (a high-speed direct connection between NVIDIA GPUs that is much faster than going over a regular network) to compute the result.

This is called tensor parallelism because we split the tensors (weight matrices) of the model across GPUs.

Pipeline Parallelism

In pipeline parallelism, we put different layers of the model on different GPUs.

Example: A model has 80 layers. We put layers 1 to 40 on GPU 1 and layers 41 to 80 on GPU 2. The token first goes through GPU 1, then to GPU 2.

This is good for very large models that have many layers.

Here is a side-by-side picture of the two parallelism styles:

Tensor Parallelism                         Pipeline Parallelism
(split each layer across GPUs)             (split layers across GPUs)

         Input                                    Input
           |                                        |
           v                                        v
   +--------+--------+                         +---------+
   | GPU 1  | GPU 2  |  layer 1                |  GPU 1  |   layers 1..40
   | weight | weight |  (weights split)        +---------+
   | half A | half B |                              |
   +--------+--------+                              v
           |                                   +---------+
           v                                   |  GPU 2  |   layers 41..80
   +--------+--------+                         +---------+
   | GPU 1  | GPU 2  |  layer 2                     |
   |  ...   |  ...   |                              v
   +--------+--------+                            Output
           |
           v
        Output

In tensor parallelism, the GPUs talk to each other inside every layer. In pipeline parallelism, each GPU finishes its layers before passing the token to the next GPU. Both can be combined for very large models.

How to choose

We pick the scaling strategy based on our model size and traffic:

Auto Scaling for AI

This is similar to regular auto scaling, but with one big difference.

In regular auto scaling, we add a new EC2 instance (a virtual server on AWS) when CPU usage crosses a threshold. This takes 30 seconds to 2 minutes.

In AI, when we add a new inference server, the GPU has to load the model first. A 70B model can take anywhere from 1 to 10 minutes to load into GPU memory, depending on whether the weights live on local SSD, network storage, or remote object storage. So we cannot just spin up a new GPU when traffic suddenly spikes.

So, here comes the warm pool to the rescue. We keep some inference servers always warm (always running, model already loaded). When traffic spikes, we route to these warm pools first, and in the background, we spin up more machines.

So in AI, auto scaling is slower and more careful. We always keep extra capacity ready because cold starts are very expensive.

Back-of-the-envelope Estimation for AI

In AI System Design interviews, we estimate three things:

  1. Token Estimation
  2. GPU Estimation
  3. Storage Estimation (for vector DB, embeddings, logs)

Let's take an example. We are building a ChatGPT-like product. Suppose we have 10 million daily active users, and each user has 5 conversations per day with an average of 2000 input tokens and 500 output tokens per conversation.

Token Estimation

Daily input tokens  = 10 million * 5 * 2000 = 100 billion tokens per day
Daily output tokens = 10 million * 5 * 500  = 25  billion tokens per day
Total tokens per day = 125 billion tokens

GPU Estimation

Suppose one H100 GPU can produce 5000 tokens per second on a Llama-70B-class model.

Tokens per H100 per day = 5000 * 60 * 60 * 24 = 432 million tokens per day
Total H100 GPUs needed  = 25 billion / 432 million = ~58 GPUs (just for output)

For input processing, we usually need fewer GPUs because input tokens are processed in parallel (we will see this in continuous batching).

So roughly 60 to 100 H100 GPUs are needed.

Cost Estimation

H100 cost per hour = $3 (rented)
Daily cost  = 60 * 24 * 3 = $4320 per day
Monthly cost = ~$130,000 per month

This is for the inference alone. Storage, vector DB, observability, etc. add more cost.

This kind of estimation is what we do in AI System Design interviews. The numbers here are just examples, but the method is what matters.

Load Balancing for LLM Servers

In regular system design, we use round-robin or least-connection algorithms to distribute requests.

In AI, round-robin is a bad choice. Why?

Because LLM requests are not equal. One request can take 1 second (short answer) and another can take 60 seconds (long answer). If we use round-robin, one server can get stuck on long requests while another sits idle.

So in AI, we use smarter load balancing.

Least Outstanding Tokens

The load balancer tracks how many tokens each server is currently generating. The next request goes to the server with the fewest outstanding tokens. This is much more accurate than counting connections.

Prefix-Aware Routing

When two requests share a long common prefix (like a long system prompt), routing them to the same server is faster because the KV cache (we will learn about this soon) for that prefix is already warm.

Example: A coding assistant has the same 5000-token system prompt for every request. If we route all requests to different servers, each server has to recompute that prefix from scratch. If we route them to the same server, the prefix is already cached.

Sticky Sessions

For a long conversation with the same user, we sometimes want to send all requests of that user to the same server, so the conversation history stays warm in cache.

So in AI, the load balancer is much smarter than in regular system design.

Caching in AI

Caching in AI is much richer than caching in regular systems. The basic shape of any cache is the same. On every request, we check the cache first. If the data is there (a cache hit), we return it without calling the LLM. If not (a cache miss), we call the LLM, store the result in the cache, and return the answer.

Cache Hit (data already cached, skip the LLM)

   Client  --->  Server  --->  Cache
                                  |
                                  | (found)
                                  v
   Client  <---  Server  <-----  Cache


Cache Miss (data not in cache, call the LLM and store)

   Client  --->  Server  --->  Cache
                                  |
                                  | (not found)
                                  v
                  Server  --->  LLM
                                  |
                                  v
                  Server  --->  Cache (store for next time)
                                  |
   Client  <---  Server  <--------+

The win from caching is the size of the gap between a cache hit and a cache miss. In AI, that gap is huge - a hit costs us a few milliseconds in Redis, a miss costs us a full LLM call (seconds and tokens).

We have four major types of cache.

KV Cache

This is the most important cache in AI. We have a detailed blog on KV Cache that explains it from the ground up.

When an LLM generates one token at a time, for each new token, the model has to look at all the previous tokens. If we recompute everything from scratch for every new token, it is extremely slow.

Inside the attention layer, every token is converted into a Query (Q), a Key (K), and a Value (V). The current token uses its Query to compare against the Keys of all previous tokens, and then collects the Values from the relevant ones. So at every step, the model needs the Keys and Values of all previous tokens.

Without caching, the model would recompute Key and Value for every previous token at every step. Let's see this with our running example:

The model is repeating the same work for tokens it has already seen. This is wasted computation.

Do you see the problem? We are paying the GPU to redo work it has already done.

This stored memory is called the KV Cache. The idea is simple: compute the Key and Value for each token once, store them, and reuse them in every future step.

Think of it like taking notes in a meeting. Every time someone new speaks, instead of asking all the previous speakers to repeat what they said, we simply read from our notes and only listen to the new speaker. The notes are the cache.

To put this into perspective with real numbers - if the model is generating 100 tokens, without KV Cache the total Key and Value computations are 2 + 3 + 4 + ... + 100 = 5,049. With KV Cache, it is 2 + 1 + 1 + ... + 1 = 101. That is roughly 50 times fewer computations. The longer the sequence, the bigger the savings.

We cache only the Keys and Values, not the Queries. During autoregressive generation, the Query is only needed for the current token being generated. Once the prediction is done, that Query is no longer needed. But the Key and Value of every past token are needed at every future step.

KV Cache makes generation much faster, but it has to sit in memory the entire time the conversation is active. A long conversation can use gigabytes of KV cache. This is why we cannot serve too many long conversations at the same time on the same machine.

To manage the KV Cache memory more efficiently, modern inference servers use Paged Attention - the same idea operating systems use to manage memory. Instead of reserving one big block of memory upfront for each request (which wastes a lot of memory because we do not know in advance how long the response will be), Paged Attention allocates small fixed-size pages of KV cache as the response grows.

Think of it like a hotel where guests check in but do not know how many nights they will stay. The traditional approach reserves 30 rooms in a row for every guest, even if most stay only 2 nights. Hundreds of rooms sit empty. The paged approach gives guests one room at a time, as needed, wherever a room is available. The hotel keeps a list (the page table) of which rooms belong to which guest. The same idea applies to KV Cache - the system can serve far more users with the same amount of memory.

KV Cache Compression

Paged Attention stops us from wasting memory. But the KV Cache itself is still huge. Let's see how huge with a model close to Llama 2: 32 layers, 32 attention heads, a Key and a Value of 128 numbers each per head, and each number stored in 2 bytes.

layers = 32
heads = 32
head_dim = 128
bytes_per_number = 2

# Key and Value, so multiply by 2
per_token = 2 * layers * heads * head_dim * bytes_per_number
print(per_token / (1024 * 1024))  # 0.5 MB

Here, we are multiplying the number of layers, the number of heads, the size of each head, and the bytes per number. We multiply by 2 because we store both the Key and the Value. So, one token needs 0.5 MB.

A conversation of 4,000 tokens needs 2 GB. A document of 100,000 tokens needs 50 GB - just for the cache of one user. A GPU has 24 GB to 80 GB, and a big part of it is already taken by the model weights. The KV Cache grows with the length of the text and with the number of users, and it eats the memory of the GPU.

So, here comes KV Cache Compression to the rescue. KV Cache Compression is the set of techniques that make the KV Cache smaller while keeping the quality of the model output almost the same.

Let me tabulate the four main approaches for your better understanding.

Quantization Token Eviction Sharing across Heads Low-Rank Compression
How it saves memory Fewer bits per number Fewer tokens stored Fewer Keys and Values per layer Smaller vector per token
Needs retraining No No Yes Yes
Throws away tokens No Yes No No
Typical saving 2x to 4x Fixed cache size 4x to 8x 10x or more
Quality loss Very less Depends on the task Very less Almost none
Example KIVI H2O, StreamingLLM GQA, MQA MLA (DeepSeek)

So, based on our use case:

Note: GQA (Grouped Query Attention) and MQA (Multi-Query Attention) let many attention heads share the same Keys and Values, and MLA (Multi-head Latent Attention) stores a small compressed vector per token instead of the full Keys and Values. We will see GQA in detail in Inference Optimization.

These approaches can be combined. A smaller KV Cache means more users per GPU, longer contexts, and faster decode, and that too without buying a bigger GPU.

Prompt Cache

Many apps send the same long system prompt with every request.

Example: A customer support bot has a 3000-token system prompt that explains the rules. Every user message goes with this same system prompt.

Without prompt cache, the inference server processes those 3000 tokens for every single request.

With prompt cache, the inference server processes the system prompt once and stores the result. For the next request with the same prompt, it reuses the stored result and only processes the new user message.

What is actually stored is the KV Cache of that shared beginning, which is called the prefix. This is why prompt caching is also called prefix caching, and why it cuts the prefill work and the TTFT. It is like a teacher who writes the same long instructions on the board at the start of every class. Instead of rewriting them each time, the teacher keeps a photo of the board and reuses it.

Note: Prompt caching only helps when requests share the exact same beginning. So, we must put the fixed parts (system prompt, tool definitions, long documents) at the start of the prompt, and the changing parts (the user message) at the end.

Both OpenAI and Anthropic support prompt caching. It can cut input cost by up to 90% and reduce TTFT massively. It makes our life easy.

Semantic Cache

This is unique to AI.

Suppose two users ask:

These are different sentences but mean the same thing. In a regular cache, we would call the LLM twice because the input strings are different.

In a semantic cache, we convert each query into an embedding (a vector). We store the embedding and the response. When a new query comes in, we convert it to an embedding and search for similar embeddings in the cache. If we find a close match, we return the cached response without calling the LLM.

This saves cost and latency.

But here is the catch. "Close enough" is not always "the same". We need to set a similarity threshold carefully.

A value around 0.85 to 0.95 works well for many apps, but the right value depends on our use case. We must test it with real questions from our users and adjust it.

A high threshold means fewer matches but safer answers. A low threshold means more matches but a higher chance of a wrong answer.

Note: Never share a semantic cache across tenants or users when the answer depends on private data. "What is my account balance?" must never be served from someone else's cache entry.

Embedding Cache

When we build a RAG system (we will see this soon), we convert lots of documents into embeddings. Computing embeddings is itself an expensive operation.

We cache the embeddings of common queries and documents in Redis or any key-value store. The key is the text (or its hash), and the value is the embedding vector. We have a detailed blog on how an Embedding Cache works that explains how the cache key is built.

Note: The cache key must include the name and version of the embedding model too. The same text gives different vectors in different models, so mixing them silently breaks the search.

Next time we see the same text, we read the embedding from cache instead of recomputing it.

In short:

LLM Routing (Model Routing)

In production AI, we rarely use one single model.

We have many models with different cost, speed, and quality:

For every user request, we ask: which model should we use?

This is model routing, also called LLM Routing.

LLM Routing is the practice of choosing the right LLM for each user query, instead of sending every query to the same LLM.

A simple router can decide based on the question:

The high-level picture looks like below:

                  +----------------+
   user query --> |   LLM Router   |
                  +----------------+
                          |
              +-----------+-----------+
              v           v           v
        +----------+ +----------+ +----------+
        |  Small   | |   Mid    | | Frontier |
        |   LLM    | |   LLM    | |   LLM    |
        +----------+ +----------+ +----------+

A model router is just a regular function, a classifier, or a tiny model that runs before the actual LLM call. It can save 50% to 70% of cost without hurting quality much.

Note: The router itself must be very fast. A good rule of thumb is to keep the router's own latency under 10 percent of the chosen LLM's latency. If the chosen LLM takes 1 second, the router should add at most 100 ms.

There are five common routing strategies:

Cascade routing is the most cost-saving strategy, so let's see it in code:

def cascade(query: str) -> str:
    answer = small_llm(query)
    if is_confident(answer):
        return answer
    answer = mid_llm(query)
    if is_confident(answer):
        return answer
    return frontier_llm(query)

Here, each step either returns the answer or escalates to the next LLM. The is_confident check can be a self-rating from the LLM, a verifier model, or a simple heuristic like the presence of "I do not know". Most queries get solved by the cheapest LLM. The hard queries pay for two or three calls.

Let me tabulate the differences between the routing strategies for your better understanding so that you can decide which one to use based on your use case.

Strategy Speed Cost Flexibility Needs Training
Rule-based Very Fast Very Low Low No
Classifier-based Fast Low Medium Yes
Embedding-based Fast Low Medium No
LLM-as-Router Medium Medium High No
Cascade Slow on hard queries Lowest on average High No

There is no clear winner here. For a small system with a few clear query types, rule-based is enough. For a large product with mixed queries, cascade or LLM-as-router works better.

Common mistakes to avoid:

Note: LLM Routing is not the same as Mixture of Experts. LLM Routing picks one full LLM for a full query, outside the model. Mixture of Experts picks a few expert sub-networks for each token, inside one model. We will see Mixture of Experts later.

Tools like OpenRouter provide model routing as a service.

Vector Database

Regular databases store rows and columns. They are good for exact-match queries.

But in AI, we deal with embeddings. An embedding is a long list of numbers (typically 768, 1024, 1536, or 3072 numbers depending on the model) that represents a piece of text in a way that captures its meaning. Two pieces of text that mean similar things get numbers that sit close together. This is what lets us do Semantic Search, which means searching by meaning instead of by exact words. Images can be turned into embeddings too, and we have a detailed blog on how Image Embeddings work.

When we want to find similar pieces of text, we cannot use SQL WHERE. We have to compute the similarity between two embeddings using a math formula like cosine similarity (which measures how close the two vectors point in the same direction) or dot product (which multiplies and sums the numbers in the two vectors). Both give a single similarity score we can sort by.

A normal database is too slow for this. So we use a Vector Database.

A Vector DB is specially built to:

Examples of Vector DBs: Pinecone, Weaviate, Qdrant, Chroma. Some regular databases also have vector support: PostgreSQL with pgvector, MongoDB, and Redis.

Internally, Vector DBs use special algorithms called vector indexes to make similarity search fast even with millions or billions of vectors.

Comparing the query with every single stored vector one by one is exact, but far too slow at scale. So, vector indexes use Approximate Nearest Neighbor (ANN) search. ANN gives up a tiny bit of accuracy to become thousands of times faster. It finds vectors that are almost certainly the closest, without checking all of them.

Vector Index Types

Now, let's understand the different types of vector indexes. There are four indexes that come up again and again. Each one has a sweet spot based on dataset size and memory budget.

Rule of thumb:

For most apps, pgvector inside PostgreSQL with HNSW is enough. We do not always need a dedicated vector database.

RAG (Retrieval Augmented Generation)

LLMs do not know our company data. They were trained on public data with a cutoff date.

If a user asks our chatbot "What is the refund policy of our store?", the LLM does not know.

To solve this, we use RAG (Retrieval Augmented Generation).

RAG = Retrieval + Augmented + Generation. The idea is simple:

  1. Retrieval: Find the relevant pieces of our private data using a vector search.
  2. Augmented: Add those pieces into the prompt as context.
  3. Generation: Let the LLM generate the answer using that context.

Let's see the flow.

Step 1: Indexing (offline, done once)

Step 2: Query time (online, done for every user query)

The architecture looks like below:

Indexing (offline)
Documents -> Chunker -> Embedding Model -> Vector DB

Query (online)
User -> Backend -> Embedding Model -> Vector DB (top-k chunks)
                                              |
                                              v
                       LLM <- (Prompt + chunks)
                        |
                        v
                     Response

The best way to think about RAG is with a real-world analogy. Suppose we want to find an answer in a big library. Standard RAG is like a librarian who fetches one book. We hand over the question. The librarian picks the book that matches the words best, gives it to us, and the LLM reads it to write the answer. One shot. Done.

The librarian flow is fast but cannot adapt to the question. For more complex questions that need many steps, we need a researcher instead of a librarian. That is Agentic RAG, where an AI Agent drives the retrieval process and decides every step. We will see Agentic RAG in detail below.

RAG is the most popular architecture for AI products today. ChatGPT search, Perplexity, customer support bots, code assistants - all of them use RAG in some form.

Document Parsing and Ingestion

Now, let's discuss the part that most RAG tutorials skip: getting clean text out of real-world documents. In production, our source data is rarely clean Markdown. It is PDFs, scanned images, slides, web pages, spreadsheets, Word documents, and emails.

If we feed messy text into the pipeline, no amount of clever retrieval can save us. The model can only answer from what it was given.

A real document parsing pipeline does the following:

A common pattern is confidence-based routing: high-confidence parses flow straight through, medium-confidence get flagged for spot-check, low-confidence go to manual review.

Tools used: Unstructured, LlamaParse, Mistral OCR, Nemotron Doc Pipeline, AWS Textract, Azure Document Intelligence.

In real RAG projects, 60% to 80% of the engineering effort goes into this step. We must build the document pipeline with the same care we give to the LLM call itself.

Chunking Strategies

The way we cut documents into chunks decides the quality of RAG. A chunk is a small piece of text that we cut out from a bigger document. We have a detailed blog on Chunking Strategies for RAG that explains each strategy with examples.

If chunks are too big, the LLM gets too much noise. If chunks are too small, we lose context and the chunks do not make sense on their own.

Common chunking strategies:

Chunk overlap means we repeat the last few lines of one chunk at the start of the next chunk, so that an idea cut at the boundary still appears whole in at least one chunk. A common setting is an overlap of about 10 to 20 percent of the chunk size. So, for a chunk of 500 tokens, an overlap of 50 to 100 tokens works well.

The right chunk size depends on the kind of answer:

We start with the default and then tune it with evals on our own data.

Vector search is great at finding meaning. But it is bad at exact-match queries like "Order ID 12345" or "Section 4.2.1".

For exact-match, the old keyword search (BM25 - Best Matching 25) is much better.

So in production RAG, we use Hybrid Search: run both vector search and BM25, combine the scores, and return the best results from both.

The two lists have scores on different scales, so we cannot just add them. A popular way to merge them is Reciprocal Rank Fusion (RRF), which ignores the raw scores and only uses the position of each chunk in each list. A chunk that ranks high in both lists ends up at the top.

Tools like Elasticsearch, OpenSearch, Weaviate, and Qdrant support hybrid search natively.

Query Transformation with HyDE

There is one more problem in retrieval. The user's question and the real answer often look very different. A question is short and question-like. The answer in our documents is long and answer-like. So, their embeddings do not always sit close together.

So, here comes HyDE into the picture.

HyDE stands for Hypothetical Document Embeddings. It is a technique where we first ask the AI model to write a fake answer to the question, and then we search using that fake answer instead of the question.

HyDE = Hypothetical + Document + Embeddings. The core idea is simple: an answer looks like an answer. So, if we want to find a real answer, we must search using something that looks like an answer.

We do not show this fake answer to the user. We do not even trust that it is correct. We only use it as a better search key. The final answer is still written from the real documents.

Just for the sake of understanding, let's say closeness is measured from 0 to 1:

Here, we can notice that the fake answer is much closer to the real answer, so the search picks the correct chunk more easily.

Advantage: Better chunks for short, vague, or technical questions, with no extra training.

Disadvantage: One extra LLM call, which means more latency and more cost. The fake answer can also drift off topic.

A good practice is to test both plain search and HyDE on the same set of questions, and keep the one that finds better chunks for our data.

Reranking

The first retrieval is fast but not very accurate. We pull, say, the top 50 chunks. Then we use a smaller, slower, but very accurate model called a reranker to score these 50 chunks again. We keep only the top 5.

A reranker is usually a cross-encoder model. The difference from regular vector search is simple. Vector search encodes the question and the chunk separately and then compares the two vectors - this is fast but a bit shallow. A cross-encoder feeds the question and the chunk together into the model and produces a single relevance score - this is slower but much more accurate because the model can directly compare the words in the question against the words in the chunk.

The two-stage funnel looks like below:

              User Question
                   |
                   v
      +---------------------------+
      |  Stage 1: Retrieval       |
      |  (vector + BM25)          |
      |  Fast, cheap, ~ok recall  |
      +-------------+-------------+
                    |
                    | top 50 chunks
                    v
      +---------------------------+
      |  Stage 2: Reranker        |
      |  (cross-encoder model)    |
      |  Slower, very accurate    |
      +-------------+-------------+
                    |
                    | top 5 chunks
                    v
                   LLM

This two-stage retrieval (cheap retrieval + accurate reranking) is the standard for high-quality RAG. Stage 1 casts a wide net cheaply. Stage 2 picks the best ones precisely.

Examples of rerankers: Cohere Rerank, BGE Reranker, Jina Reranker.

A reranker is one of the simplest changes that gives a big jump in RAG quality. It also helps with the Lost in the Middle problem (we will see it soon), because sending 5 good chunks is better than sending 50 average ones.

ColBERT and Late Interaction

Now, the following question arises. The cross-encoder reranker is accurate but slow. Vector search is fast but blunt, because it squeezes a whole passage into one single vector and loses word-level detail. Can we get both?

So, here comes ColBERT to the rescue. ColBERT keeps one vector per token instead of one vector per passage, and matches them with a cheap step called late interaction.

  1. Cross-encoder  (accurate, slow)

     query + document together  ->  BERT  ->  score
     BERT runs once per pair, nothing can be stored ahead of time


  2. Single-vector dual encoder  (fast, blunt)

     query     ->  BERT  ->  [ one vector ]
                                          >--  one dot product  ->  score
     document  ->  BERT  ->  [ one vector ]


  3. Late interaction, ColBERT  (accurate and fast)

     query     ->  BERT  ->  [ v1  v2  v3 ]
                                          >--  MaxSim  ->  score
     document  ->  BERT  ->  [ v1  v2  v3  v4  v5 ]

Here, we can see the difference clearly. Because the document is encoded without seeing the query, every document's vectors can be computed once, offline. At query time, each query token finds its best matching document token (this is called MaxSim), and the scores are added up.

The price is storage. A 100-token passage needs 100 vectors instead of 1, which is 100 times more storage. So, ColBERT buys its accuracy with disk space. We use it as a strong reranker, or for end-to-end retrieval when quality matters more than storage cost.

Agentic RAG

Standard RAG works for simple questions. But the moment the question gets harder, it breaks down. The librarian only knows how to match words. The librarian cannot think, cannot ask follow-up questions, and cannot decide where to look.

Standard RAG fails when:

So, here comes Agentic RAG to the rescue. We replace the librarian with a researcher.

In simple words:

Agentic RAG = RAG + an AI Agent that controls the retrieval steps.

In Agentic RAG, the agent decides every step:

The Agentic RAG loop:

User Question
     |
     v
+------------------+
|    AI Agent      |
|   (the LLM)      |
+------------------+
     |
     | thinks: "Do I need to retrieve?
     |          Which tool? What query?"
     v
+------------------+
|  Retrieval Tool  |
|  (vector / SQL / |
|   web / graph)   |
+------------------+
     |
     | returns chunks
     v
+------------------+
|    AI Agent      |
|   evaluates:     |
|  "Good enough?"  |
+------------------+
     |
     +--- No  --> retrieve again with a new query
     |
     +--- Yes --> generate the final answer

Common patterns of Agentic RAG: ReAct-style RAG (think, act, observe loop), Self-RAG (the agent grades its own retrievals), and Corrective RAG (CRAG) (the agent recommends a fallback source when the main retrieval is bad).

The trade-off is real: Agentic RAG has higher latency, higher cost, and is harder to debug. We use Agentic RAG only when standard RAG cannot do the job.

Let me tabulate the differences between Standard RAG and Agentic RAG for your better understanding so that you can decide which one to use based on your use case.

Aspect Standard RAG Agentic RAG
Flow Fixed pipeline Dynamic loop
Number of retrievals One Many, as needed
Tools used One (usually vector search) Many (vector, SQL, web, and etc.)
Query rewriting No Yes
Quality check on retrieved chunks No Yes
Multi-hop questions Cannot handle Can handle
Latency Low Higher
Cost Low Higher
Complexity to build Simple More complex

GraphRAG

Now, let's move to a more advanced form of RAG. Vector RAG is great when the answer lives in one chunk. It struggles when the answer requires connecting facts across many chunks. For questions like "How are these two people connected?" or "What policies apply to all products in this category?", vector search just pulls a few related chunks and the LLM has to guess the connections.

So, here comes GraphRAG to the rescue. It solves this problem by building a knowledge graph from the corpus first.

In simple words:

GraphRAG = Knowledge Graph + RAG. Retrieve relationships, not just chunks.

The flow has two stages.

Stage 1: Indexing (offline)

Stage 2: Query time

A small example of a knowledge graph extracted from a corpus looks like below:

                +-----------+
                | Jane Doe  |
                +-----------+
                      |
                      | is_CEO_of
                      v
                +-----------+      acquires       +-----------+
                |  Globex   | -----------------> |   Acme    |
                |           |     (in 2023)      +-----------+
                +-----------+
                      |
                      | has_report
                      v
                +-----------+
                | Q4 2023   |
                | revenue   |
                +-----------+

Question: "Q4 revenue of the company that acquired Acme in 2023?"
Traversal: Acme <- (acquires) <- Globex -> (has_report) -> Q4 2023 revenue

The arrows read in active voice: Globex acquires Acme, Globex has_report Q4 2023 revenue, Jane Doe is_CEO_of Globex. To answer the question, we walk backward along the acquires edge from Acme (which gives us Globex), then forward along the has_report edge (which gives us Q4 2023 revenue).

Vector RAG would never make this connection in a single retrieval. GraphRAG walks the edges and finds the answer by following the relationships.

GraphRAG is much better at:

The trade-off: GraphRAG is much more expensive to index (every document goes through LLM extraction) and harder to update incrementally. We use it when relationships matter and the data is fairly stable.

Microsoft Research's GraphRAG project is the most well-known implementation. Other tools: LlamaIndex Knowledge Graph Index, Neo4j LLM Graph Builder.

In short:

In production, the two are often used together.

Vectorless RAG

Now, let's see the opposite direction. Vector RAG has some real problems:

So, here comes Vectorless RAG to the rescue.

Vectorless RAG = RAG without the vectors. Instead of searching by number similarity, we let the LLM itself look at the structure of the document and decide which part to open, exactly the way a human finds an answer in a book.

Suppose we buy a washing machine and want to know about the warranty. We get a 200-page manual. We do not measure the similarity of anything. We open the table of contents, see "Chapter 9: Warranty and Service", jump to page 154, and read it. That is the whole idea.

It works in two phases:

Let me tabulate the differences between Vector RAG and Vectorless RAG for your better understanding so that you can decide which one to use based on your use case.

Vector RAG Vectorless RAG
Needs an embedding model and a vector database. Needs neither.
Documents are cut into fixed size chunks. Documents are kept whole and read section by section.
Search by measuring distance between numbers. Search by the LLM reading titles and summaries.
No reasoning during the search. Reasoning at every step of the search.
Very fast and very cheap for each question. Slower and costlier, because it needs a few LLM calls.
Hard to explain why a chunk was picked. Easy to explain, because the LLM gives its reason.
Scales very well to millions of documents. Works best on long and well structured documents.
Struggles with exact codes, numbers, and negation. Handles exact codes, numbers, and negation well.

We must use Vector RAG for a very large number of short documents where speed and low cost matter, like a help center with a million articles. We must use Vectorless RAG for long, well structured documents where a wrong answer is expensive, like insurance policies, legal contracts, annual reports, and technical manuals. Many strong systems use both: a cheap vector or keyword search narrows millions of documents down to twenty, and then tree search finds the exact section inside those twenty.

To learn RAG, Vector Databases, embeddings, and retrieval pipelines end to end with hands-on projects, check out the AI and Machine Learning Program by Outcome School.

Context Window Management

Every LLM has a maximum context window. GPT-5.5 has up to 400k tokens. Claude Sonnet 4.6 and Opus 4.8 have up to 1 million tokens. Some models go even further. But all of them are finite.

In production, we hit context limits all the time:

But there is a deeper problem. The maximum context window is the size we are allowed to use, not the size we must use.

Context Rot, Lost in the Middle, and RoPE Decay

As we fill the context with more tokens, three things happen, and they all hurt quality.

1. Context Rot. As the number of tokens in the context grows, the model's ability to recall information from anywhere in that context drops. Long-context evaluations (such as Needle-in-a-Haystack style tests and the more recent RULER and LongBench benchmarks) show that even models that advertise very long context windows start losing accuracy well before they hit that limit. So a large advertised context does not mean an equally large effective context.

2. Lost in the Middle. Research by Liu et al. (2023) showed that models pay the most attention to the start and the end of the context, and the least attention to the middle. If we plot accuracy against the position of the fact, we get a U-shaped curve. So a critical fact placed in the middle of a long context is more likely to be ignored than the same fact placed at the top or bottom. This silently breaks RAG systems that dump 50 chunks into the prompt. The fix is to send less but send the right thing (use a reranker and keep the top few chunks), and to place the most relevant chunks at the start and the end.

3. RoPE Positional Decay. Modern LLMs use Rotary Position Embedding (RoPE) to encode the position of each token. RoPE makes the attention score between two tokens depend only on the relative distance between them, and as two tokens get farther apart, their attention tends to decay. Newer models have softened this, but no production model has eliminated it.

The practical implication: where we place information in the prompt matters as much as the information itself. Important context should go at the top of the system prompt or right next to the question, not buried in the middle.

Context Engineering

Now, let's understand context engineering. Because of these effects, the discipline of Context Engineering has emerged.

Context Engineering is the practice of designing, organizing, and managing everything that goes into an LLM's context window so that the model can do its task reliably.

It is the next layer above prompt engineering. Prompt engineering is about the wording of one instruction. Context engineering is about everything the model sees: the system prompt, the retrieved chunks, the tool definitions, the tool results, the memory, and the conversation history. Context engineering decides:

A useful way to think about this: the context window is a small workspace, and we are the one deciding what goes on it. We choose what to put in, what to leave out, what to put at the top, and what to push to the bottom. The model only sees what we place there.

We need a strategy to manage the context window.

Truncation

Drop the oldest messages when we are about to overflow. Simple but loses information.

Summarization

When the conversation is long, summarize the older messages into a shorter summary and put that summary in the prompt. This is what most chat products do.

Sliding Window

Keep only the last N messages and the system prompt. Forget everything in between. Works for chat where only the recent context matters. The same idea exists inside some models too, where each token only attends to a fixed window of recent tokens. We have a detailed blog on how Sliding Window Attention works.

Compaction

Context compaction is the technique of shrinking the old conversation into a short summary, so the important facts stay while the context window gets free space again. For agent loops, we also compact tool results that are no longer needed. Keep the question, the decisions, and the important findings. Drop the verbose tool outputs. Good compaction never summarizes away the system instructions, the user's main goal, or the very latest messages. Only the messy middle gets compressed. For example, an hour of a coding agent's work can become a short note like "Goal: fix the login bug. Already changed auth.py and login.py. Tests are passing." This is how an agent works on a long task for hours without losing the plot.

Hierarchical Memory

Store the full history in a vector DB and only retrieve relevant parts when needed.

Choosing the right strategy depends on the use case. For chat, summarization works well. For agents, compaction works well. For Q&A, hierarchical memory works well.

To learn how to build an AI Coding Agent from scratch and implement different layers of compaction in it, check out the AI and Machine Learning Program by Outcome School.

Streaming Responses

In regular APIs, the server sends one response after the work is done.

In AI, we cannot wait for the full response. A long answer can take 30 seconds. The user cannot stare at a blank screen for 30 seconds.

So we stream the response. As soon as the LLM generates a token, we send it to the user. The user sees the text appear word by word. This is called Token Streaming.

The difference between a regular response and a streamed response looks like below:

Regular HTTP (no streaming)

   Client                       Server
     |                            |
     | --- request -------------> |
     |                            |  (model generates 500 tokens)
     |                            |  ...
     |                            |  ~10 seconds later
     | <-- full response -------- |
     |     (all 500 tokens)       |


Streaming (SSE)

   Client                       Server
     |                            |
     | --- request -------------> |
     | <-- "Once" --------------- |  ~800 ms (TTFT)
     | <-- " upon" -------------- |
     | <-- " a" ----------------- |
     | <-- " time" -------------- |
     |        ...                 |
     | <-- " end" --------------- |  ~10 seconds total
     | <-- [DONE] --------------- |
     |                            |

In streaming, the user sees the first token at TTFT (under a second) and watches the response build up. In a regular HTTP response, the user stares at a blank screen for 10 seconds and then everything appears at once. Streaming is what makes a slow LLM feel fast. It works perfectly.

There are two main ways to stream. We have a separate detailed blog on HTTP Request, Long Polling, WebSocket, and SSE that explains these in depth.

Server-Sent Events (SSE)

The most popular way. The client opens an HTTP connection. The server keeps it open and writes chunks as they come. Each chunk is a small message that starts with data:, and the stream ends with a special [DONE] marker. This is what ChatGPT uses on the web.

WebSockets

A two-way streaming connection. Used when we also need to receive messages from the client mid-stream (like cancel a request or send more context).

Streaming changes a lot of things in our backend:

Async Processing for Long AI Tasks

Some AI tasks are too long to stream. For example:

For these, we do not stream. We do async processing.

The flow happens in two stages:

Stage 1: Client kicks off the job and gets a job_id back

   Client  ---request--->   Backend   ---enqueue--->  Queue
   Client  <--job_id, "processing"--   Backend


Stage 2: Worker runs the task in the background

   Queue   ---pull--->   Worker   ---run task--->   Database
                                  (saves result)


Stage 3: Client polls for the result (or gets a webhook)

   Client  ---poll job_id--->   Backend   ---read--->   Database
   Client  <---result/status---  Backend

Step by step:

Step 1: The client sends the request.

Step 2: The backend immediately responds with { "job_id": "12345", "status": "processing" }.

Step 3: The backend puts the job in a message queue.

Step 4: A background worker picks up the job and runs it.

Step 5: When done, the worker stores the result in a database.

Step 6: The client polls the status using the Job ID, or gets a webhook callback.

This is the same pattern as regular async processing, but the tasks are AI tasks (LLM calls, embeddings, transcription, etc.).

Message Queues in AI Systems

Message queues are heavily used in AI for the same reasons as regular systems: decoupling, reliability, and retry.

But in AI, the messages are heavier (long prompts, large documents, audio files). And the workers are slower (LLM calls, transcription).

Common usage:

Tools used: AWS SQS, RabbitMQ, Apache Kafka, Redis Streams.

Kafka shines when many independent consumers need to read the same stream of messages (for example, an indexing worker and an analytics worker both reading every uploaded document). SQS and RabbitMQ shine when each message should be processed by exactly one worker (a classic job queue). We pick based on the consumption pattern, not the AI part.

Rate Limiting in AI

In regular APIs, we rate-limit by requests per second. Like 100 requests per minute per user.

In AI, this is not enough. Two requests can be very different in cost. One request can be 100 tokens, another can be 100,000 tokens.

So in AI, we rate-limit on multiple dimensions.

Tokens Per Minute (TPM)

We track how many tokens a user has used in the last minute and block them after a limit.

Requests Per Minute (RPM)

Same as before, but in addition to TPM.

Concurrent Requests

How many requests is a user running at the same time? This matters because LLM requests are long-running.

Cost-Based Rate Limiting

In some apps, we directly track the dollar cost a user is consuming and apply a cost cap.

Tools like Kong, Tyk, and Portkey provide multi-dimensional rate limiting for AI workloads.

Here, we have rate-limited on tokens, requests, concurrency, and cost together. This is critical because a buggy client can otherwise burn through our entire budget in minutes. Now, our AI system can stay safe under sudden load and abusive traffic.

AI Gateway

In real production, our app does not call OpenAI, Anthropic, or our self-hosted vLLM directly. We put a single layer in between, called the AI Gateway (also called LLM Gateway). I will highly recommend treating the AI Gateway as one of the first pieces of infrastructure to build, because every other concern (cost, rate limiting, caching, guardrails, observability) becomes much easier once we have it.

Put simply, an AI Gateway is a reverse proxy specialized for LLM traffic.

A regular API gateway routes HTTP requests. An AI Gateway also understands tokens, prompts, streaming, and cost. It is the single point where every LLM call in our company flows through.

A typical AI Gateway sits like this:

+--------+         +-------------+         +-------------------+
|  App   | ------> | AI Gateway  | ------> | OpenAI / Claude   |
+--------+         |             | ------> | Gemini / Llama    |
                   |             | ------> | Self-hosted vLLM  |
                   +-------------+         +-------------------+

What does the AI Gateway give us?

Tools used: Portkey, LiteLLM, Kong AI Gateway, Envoy AI Gateway, TrueFoundry, AWS Bedrock (acts as a gateway too).

A few things we must get right when designing an AI Gateway:

Once we have an AI Gateway, every other concern in this blog (rate limiting, cost, caching, guardrails, observability) gets a single home. This is why an AI Gateway is one of the first pieces of infrastructure any serious AI team builds.

Embeddings Pipeline

For RAG and semantic search, we need to embed lots of documents.

If we have a million documents and each one has 10 chunks, we have 10 million embeddings to compute. This is a big batch job.

The embedding pipeline looks like below:

Source Data (S3, DB, web)
    |
    v
Chunker (split documents into smaller pieces)
    |
    v
Embedding Model (compute embedding for each chunk)
    |
    v
Vector DB (store embeddings with metadata)

This pipeline runs as a batch job in the background. We use:

We also handle re-embedding:

A good embedding pipeline is incremental. It only processes what is new or changed, not the whole dataset every time.

Choosing an Embedding Model

Now, let's pick the right embedding model. The embedding model decides the quality of our retrieval. A bad embedding model means even the best vector DB returns junk. Common production options in 2026:

Matryoshka Embeddings

A useful trick worth knowing. Matryoshka embeddings are trained so that we can truncate them to a smaller dimension without retraining. So we can store the full 3072-dimensional embedding once, but at query time use only the first 256 or 512 dimensions for a fast first-pass search, and then re-rank with the full vector.

This gives us a memory-vs-quality knob without having to maintain two embedding models. OpenAI text-embedding-3 and Cohere both support this.

A good rule: pick the embedding model based on the task type, the language, the budget, and the privacy needs. Switching the embedding model means re-embedding everything, so this decision has gravity.

AI Agents and Agentic Systems

An AI Agent is a system that can take a goal from us, figure out the steps to achieve it, use tools when needed, and keep working until the goal is done. It is not just an LLM that answers one question. It is an LLM wrapped in a loop, with tools and memory, where the LLM is the decision-maker - it decides what action to take next, picks the right tool, reads the result that the loop feeds back, and plans the next move.

In simple words:

AI Agent = An LLM + Instructions + Tools + Memory + A loop that runs until the goal is achieved.

Think of an AI Agent like a new intern on their first day. They do not do the whole job in one go. They read the task, think about what to do first, use a tool (open a file, send a message, run a query), check the result, and decide the next step. They repeat this until the job is done. An AI Agent does the same thing - just faster and without breaks.

A plain LLM gives us one answer and stops. A chatbot gives many answers. An AI Agent talks and acts, many times, until the task is done.

The Five Core Parts

Every AI Agent, no matter how complex, has the same five core parts.

+------------------------------------------------+
|                      Loop                      |
|                                                |
|   Instructions --->  +-------+                 |
|                      |       |                 |
|   Memory       --->  |  LLM  | <-----+         |
|                      +-------+       |         |
|                          |           |         |
|                          | pick      | result  |
|                          v           |         |
|                      +-------+       |         |
|                      | Tools | ------+         |
|                      +-------+                 |
+------------------------------------------------+

1. The LLM (the brain). This is the reasoning engine. It reads the current situation and decides the next action - either pick a tool for the loop to call, or return a final answer. Every meaningful decision in the agent goes through the LLM. The LLM does not call tools itself - it only recommends which tool to use and what inputs to pass. The loop is the one that actually calls the tool.

2. The Instructions (the system prompt). This tells the LLM what its job is, what tools it has, and what rules to follow. Without good instructions, the agent is confused.

3. The Tools (the hands). These are the actions the agent can take - search, calculate, read files, call APIs, send emails, run code. Each tool has a name, a description, and an input schema.

4. The Memory. This is what lets the agent keep track of progress and avoid repeating work. Short-term memory is the conversation history of the current run. Long-term memory is anything we want the agent to remember across runs - user preferences, past decisions, learned facts. We will go deeper on memory in the next section.

5. The Loop (the runtime). This is the code that keeps the agent going. Send prompt to LLM. Read the LLM's response. If the LLM picked a tool, run that tool. Feed the result back. Repeat until the LLM says "done." The loop is what makes it an agent, not just a chatbot. The loop is also where tool calls actually happen - the LLM only decides which tool to use.

Going back to the intern analogy - the LLM is the intern's brain, the Instructions are the job description the manager handed over, the Tools are the laptop and email and office systems they can use, the Memory is the notebook where they track progress, and the Loop is the workday itself that keeps them going from one task to the next.

Remove any one of these, and it is no longer an AI Agent.

How an AI Agent Works End to End

   +----------------+
   |  User's Goal   |
   +----------------+
            |
            v
   +----------------+        +-------------+
   |      LLM       |------->|    Tool     |
   |  (decides:     |        +-------------+
   |   pick a tool  |              |
   |   or return    |              v
   |   final answer)|        +-------------+
   |                |<-------| Observation |
   +----------------+        +-------------+
            |
            | when goal is achieved
            v
   +----------------+
   |  Final Answer  |
   +----------------+

The goal comes in at the top. The LLM looks at it and decides one of two things - which tool the loop should call next, or the final answer directly. When the LLM picks a tool, the loop runs it and produces an observation, which is fed back to the LLM. The LLM then decides again. This little loop between the LLM, the Tool, and the Observation keeps running until the LLM decides the goal is achieved and returns the final answer.

This loop is also called the agent loop (the think, act, observe cycle), and the code around the LLM that runs it is called the harness. We have a detailed blog on Harness Engineering in AI that explains the harness in depth.

The whole skeleton of an AI Agent looks like this in Python:

async def run_agent(user_goal, tools, max_steps=10):
    messages = [{"role": "user", "content": user_goal}]
    step = 0

    while True:
        # Safety stop: bail out if we hit the step limit
        if step >= max_steps:
            return "Reached step limit without completing the task."
        step += 1

        # Ask the LLM what to do next
        response = await call_llm(messages, tools)

        # If the LLM says "I am done", stop the loop and return the final answer
        if response.is_done:
            return response.final_answer

        # Otherwise, run each tool the LLM picked and feed the result back
        for tool_call in response.tool_calls:
            result = await call_tool(tool_call.name, tool_call.arguments)
            messages.append({
                "role": "tool",
                "name": tool_call.name,
                "content": str(result),
            })

That is the entire AI Agent in about 20 lines of Python. A while loop, an LLM call inside it, a check for the stop signal, and a tool-call branch that feeds results back. Everything else we add on top - memory, parallel tool calls, retries, logging - is polish around this core loop.

Types of AI Agents

Not all AI Agents look the same. Most reuse a small set of patterns:

These are not mutually exclusive. A real system often combines patterns - for example, a ReAct agent that uses Agentic RAG for knowledge lookups, with a Reflection step at the end for quality.

Computer Use and Browser Agents

Now, let's go deeper into computer-use agents. This new class of agents deserves its own note because it changes the system design in several ways.

A regular tool-calling agent talks to APIs. A computer-use agent has no APIs to call. It looks at the screen the way a human does and clicks the same buttons a human would click. This is how it can drive any web app, any desktop app, even systems that have no API at all.

The architecture looks like below:

   +---------------+
   |  User's Goal  |
   +---------------+
            |
            v
   +---------------+    screenshot    +-----------------+
   |     LLM       | <--------------- | Browser /       |
   | (vision +     |   click, type    | Desktop / OS    |
   |  reasoning)   | ---------------> |                 |
   +---------------+                  +-----------------+
            |
            v
   +---------------+
   | Final Answer  |
   +---------------+

The model sees the screen as an image, decides on a UI action (click at coordinates, type text, scroll), the runtime executes that action, and the next screenshot comes back. Loop until done.

Design implications:

Computer-use agents work well for tasks like filling out forms, scraping data from web apps without APIs, navigating legacy enterprise UIs, and running back-office workflows. They replace traditional RPA (Robotic Process Automation) with something far more flexible.

Common Failure Modes

AI Agents are powerful, but they fail in specific ways:

Production-grade AI Agents are mostly about handling these failure modes well. Getting a single demo to work is easy. Keeping it stable across thousands of varied real user requests is the hard part.

AI Orchestration vs AI Agents

Now, the next big question is: do we even need an agent? Many real AI products do not.

AI Orchestration is the process of coordinating multiple AI components, such as LLMs, tools, data sources, and agents, to work together to finish a complex task. Think of it like a conductor in an orchestra. The conductor decides who plays when, who plays louder, and who stops.

The key difference is who controls the flow. In AI Orchestration, the developer defines the steps, and the LLM does the work inside each step. In an AI Agent, the LLM itself decides what to do next.

Let me tabulate the differences between AI Orchestration and AI Agents for your better understanding.

Aspect AI Orchestration AI Agents
Who controls the flow The developer The LLM
Flow type Fixed, defined upfront Dynamic, decided at runtime
Predictability High Low
Best for Known workflows Open-ended tasks
Cost control Easier Harder
Debugging Easier Harder

The common orchestration patterns are Sequential (step after step), Parallel (independent steps at the same time), Conditional (a branch based on the output), Loop (repeat until a check passes), and Orchestrator-Worker (one LLM splits the task and hands pieces to workers). A simple chain of steps like this is also called Prompt Chaining, where the output of one prompt becomes the input of the next prompt.

In real systems, we often use both together. We use AI Orchestration for the high-level flow, and an AI Agent inside one of the steps when that step is open-ended. Start with a fixed workflow. Reach for an agent only when the steps cannot be known in advance.

Loop Engineering

When we do use an agent, the loop is where most of the quality comes from. Loop Engineering is the practice of designing the repeating cycle that an AI agent runs, so that the agent keeps making real progress on a task and stops at the right moment with the right result.

These are the parts of the loop that we, the engineers, must decide:

Here, we can see why the goal matters so much. AI works best when a machine can check whether the task is done. We have a detailed blog on why AI is only as good as our definition of done.

In short: Prompt Engineering fixes what the model hears, Context Engineering fixes what the model knows, and Loop Engineering fixes what the model is allowed to do over time.

Graph Engineering

Graph Engineering is the practice of designing an AI system as a graph, where every step of the work is a node and every path from one step to another step is an edge. A shared state travels through the graph, conditional edges take decisions, and cycles let the system do the work again when a check fails.

Let me tabulate the differences between Loop Engineering and Graph Engineering for your better understanding.

Point Loop Engineering Graph Engineering
Who decides the next step The model decides Our code decides, using the model's output
Structure One loop of think and act Many nodes joined by edges
Path on every run Different every time Same path for the same situation
Testing Hard, because the path keeps changing Easy, we can test one node at a time
Best for Open ended and unknown tasks Known work with clear stages

Frameworks like LangGraph are built exactly on this idea. LangChain gives us the building blocks (prompts, chains, retrievers, and output parsers) to connect an LLM with our data and tools.

To learn AI Agents, Agentic AI, Tool Use, Memory, MCP, Subagents, and Multi-Agent Systems with real projects (including building our own AI Agent), check out the AI and Machine Learning Program by Outcome School.

Tool Calling

Tool calling is how the LLM tells us "please call this function for me".

The LLM gets a list of available tools as part of its prompt. Each tool has a name, a description, and an input schema.

Here is what a tool definition looks like in practice:

{
  "name": "web_search",
  "description": "Search the web for pages matching the query. Use this tool when the user asks for current information, news, or anything not available in the model's training data.",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "The search query to run."
      },
      "max_results": {
        "type": "integer",
        "description": "Maximum number of results to return. Default is 5."
      }
    },
    "required": ["query"]
  }
}

Here, the LLM reads the name to identify the tool, the description to decide when to recommend it, and the input_schema to know what arguments to pass and in what format. The description is the most important field - every word matters, because this is what the LLM uses to pick between tools. A vague description leads to the wrong tool being picked. A precise description leads to the right one.

When the LLM decides to recommend a tool, it returns a structured output that names the tool and gives the arguments. Our code parses this, runs the actual search, and feeds the result back as the next message. It works perfectly.

This is called function calling or tool calling. Both names mean the same thing. It is the foundation of all agentic AI.

Note: The LLM never runs the tool itself. It only returns the name of the tool and the arguments. Our code runs the tool. This is an important security boundary, because it means our code can validate every argument and refuse any action before it happens.

In an agentic system, designing good tools is more important than the model choice. Bad tools (vague names, unclear parameters, noisy outputs) make even the smartest model behave badly.

Model Context Protocol (MCP)

In the section above, we wrote our own JSON tool definitions and our own loop to call them. That works for a small project. But what if we want our agent to talk to GitHub, Slack, Google Drive, our own database, and 20 other systems? We do not want to write 20 different tool integrations in 20 different ways.

So, here comes MCP (Model Context Protocol) to the rescue.

In plain words, MCP is a universal standard for connecting AI applications to tools and data sources.

The popular analogy is: MCP is the USB-C port for AI. Just as USB-C is one standard plug that any device can use, MCP is one standard interface that any AI application can use to plug into any tool or data source.

MCP was introduced by Anthropic in late 2024 and has since been adopted by OpenAI, Microsoft, and parts of the Google and AWS stacks. As of 2026, the ecosystem has thousands of MCP servers covering developer tools, business apps, databases, and search.

How MCP works

MCP has two sides:

+----------------+        +-------------+        +----------------+
|   AI Agent     |        |             |        | GitHub Server  |
| (MCP Client)   | <----> | MCP         | <----> | (MCP Server)   |
+----------------+        | Standard    |        +----------------+
                          |             |        +----------------+
                          |             | <----> | Slack Server   |
                          |             |        | (MCP Server)   |
                          |             |        +----------------+
                          |             |        +----------------+
                          |             | <----> | Database       |
                          +-------------+        | (MCP Server)   |
                                                 +----------------+

When our agent wants to use a tool, it does not care whether the tool talks to GitHub, Slack, or a database. It just speaks MCP. The MCP server on the other side takes care of the actual integration. The problem is solved.

Why MCP matters for AI System Design

Without MCP, every team writes its own tool layer. With MCP:

When to use MCP

When MCP is overkill

MCP is now the default way to do tool calling in production AI systems. If we are designing a serious AI System Design problem in 2026, mentioning MCP is expected.

Note: An MCP server is code that runs with real permissions. We must only connect to MCP servers we trust, give each server the smallest set of permissions it needs, and treat every tool result as untrusted text (we will see why in Prompt Injection).

Agent Skills

MCP gives the agent reach. But reach is not enough. The agent also needs to know our way of doing things - our steps, our format, our checklists. Stuffing all of these into the system prompt fills the context window fast.

So, here comes Agent Skills to the rescue.

An Agent Skill is a folder of instructions, and optionally scripts and reference files, that an AI agent loads by itself only when the task actually needs it.

Agent Skills were introduced by Anthropic in October 2025, and the format is open. Consider a new employee joining our team. We do not retrain them from zero. We hand them our team handbook with separate chapters - one for filing an expense, one for releasing software, one for writing a report. They open one chapter only when they face that job. A Skill is exactly one such chapter.

Inside the folder, there is one required file called SKILL.md with a name, a description, and the instructions. The magic is progressive disclosure, which means revealing information in stages, only as much as is needed right now:

   Level 1  ->  name + description only          (always loaded, tiny)
   Level 2  ->  the full SKILL.md instructions   (loaded when it matches)
   Level 3  ->  extra files in the folder        (loaded only if needed)

Level 1 costs only about a hundred tokens for each Skill. Level 2 is normally kept under five thousand tokens, and we pay that only for the one Skill we are actually using. So, fifty Skills at Level 1 cost about the same as the body of one single Skill. The agent behaves as if it knows fifty procedures in depth, while its context stays almost empty.

Let me tabulate the differences between Agent Skills and MCP for your better understanding.

Point Agent Skills MCP
What it gives Know-how, our steps and our format Reach, a connection to outside systems
What it is made of A folder of instructions, and sometimes scripts A running program that offers tools and data
Question it answers How should this be done? What can the agent reach?
Needs a server running No Yes
Best for Procedures, formats, style, checklists Databases, apps, live data, actions

In real work, we use both together. MCP connects the agent to our issue tracker, and a Skill tells it how our team writes an issue. There is also a third piece built almost exactly like a Skill: OKF (Open Knowledge Format), which writes down the knowledge about our data as a folder of plain markdown files, so any agent knows what our tables actually mean.

Structured Output

Many times, we do not want free-form text from the LLM. We want a clean JSON object that we can use directly in our code.

Example: We are building a resume parser. We want the LLM to read a resume and return:

{
  "name": "Amit Shekhar",
  "email": "[email protected]",
  "skills": ["Python", "Go", "Kotlin"],
  "experience_years": 10
}

If we just ask the LLM "give me JSON", sometimes it returns proper JSON, sometimes it adds extra text like "Sure, here is the JSON:", and sometimes it returns broken JSON.

To fix this, modern APIs support structured output (also called JSON mode or constrained decoding):

Examples: OpenAI's response_format with JSON Schema, Anthropic's tool-use mode, vLLM's guided decoding, Outlines library.

The problem is solved. Now, the next step in our pipeline can trust the JSON without writing fragile parsers.

Structured output is critical when the LLM output goes into the next step of a pipeline. Without it, we have to handle a lot of edge cases.

Memory for AI Agents

An LLM by itself is stateless. Every time we call it, it is like talking to someone who has never met us before. AI Agent Memory is everything we build around the LLM to give it the feeling of remembering. The LLM does not actually remember. The agent remembers on its behalf.

Put plainly, Memory is the system that lets a stateless LLM act like it remembers across turns, sessions, and users.

The Memory Stack

AI Agent Memory is not a single thing. It is a stack of layers, each with a different lifespan and purpose.

+--------------------------------------------+
|              AI Agent Memory               |
|                                            |
|  Layer 1: Context Window (in the prompt)   |
|     - In the current prompt                |
|     - Lost when context overflows          |
|                                            |
|  Layer 2: Short-Term Memory (session)      |
|     - Scratchpad, current task state       |
|     - Lost when the session ends           |
|                                            |
|  Layer 3: Long-Term Memory (persistent)    |
|     - Facts about the user                 |
|     - Past conversations                   |
|     - Learned preferences                  |
|                                            |
|  Layer 4: External Knowledge (tools/RAG)   |
|     - Documents, databases, APIs           |
|     - Queried, not maintained by agent     |
+--------------------------------------------+

Layer 1 - Context Window. This is the text the LLM sees in a single call. It is the fastest memory because the LLM reads it directly. But it is also the smallest. Once it is full, older messages have to be dropped or summarized.

Layer 2 - Short-Term Memory. This is the scratchpad the agent uses during one task or session. It holds the plan, the current step, and the intermediate results. When the session ends, this memory is usually gone.

Layer 3 - Long-Term Memory. This is the persistent memory that survives across conversations and sessions. It stores user facts, preferences, past conversation summaries, and learned behaviors. This is what makes the agent feel like it "knows" the user. Within Long-Term Memory, there are further sub-types - episodic (past experiences), semantic (facts and concepts), and procedural (how to do things).

Layer 4 - External Knowledge. This is everything the agent can query but does not maintain - documents, databases, APIs, the web. The agent reads from these sources but does not write back to them as its own memory.

All four layers work together. The context window is the LLM's desk. Short-term memory is the notepad next to the desk. Long-term memory is the personal diary in the drawer. External knowledge is the library across the street. We want to keep our desk clean, our notepad organized, our diary trustworthy, and the library indexed.

The Four Core Operations

No matter how many layers we have, every memory system does only four things:

1. Write. Save new information. After the user says "I prefer code in Go", we write "preferred language: Go" to long-term memory.

2. Read. Pull relevant information back when needed. This is usually the hardest part. We do not want everything, only what matters right now.

3. Update. Modify existing memories when new information contradicts them. If the user's favorite language was Python last month and is now Go, the memory must be updated.

4. Forget. Remove memories that are stale, wrong, or no longer relevant. Without forgetting, the memory store grows forever and retrieval gets worse.

Memory System = Write + Read + Update + Forget.

If any one of these is broken, the whole memory system breaks.

How Memory Flows at Runtime

                   User Message
                        |
                        v
          +----------------------------+
          |        READ Memory         |
          |  - Long-Term (user facts,  |
          |    preferences, summaries) |
          |  - Short-Term (plan,       |
          |    recent steps)           |
          +-------------+--------------+
                        |
                        v
          +----------------------------+
          |       Build Prompt         |
          |  (system prompt +          |
          |   retrieved memories +     |
          |   user message)            |
          +-------------+--------------+
                        |
                        v
          +----------------------------+    +---------------------+
          |            LLM             |<-->| External Knowledge  |
          |    (reasons, replies)      |    | (tools, RAG, APIs)  |
          +-------------+--------------+    +---------------------+
                        |
                        v
          +----------------------------+
          |        WRITE Memory        |
          |  - Short-Term (new step)   |
          |  - Long-Term (if worth     |
          |    keeping)                |
          +-------------+--------------+
                        |
                        v
                     Response

The same flow in plain words: on every turn, the agent retrieves relevant memories, builds the prompt, calls the LLM (which may recommend a tool that the loop then runs), writes new memories, and responds. **Read -> Build prompt -> Respond -> Write, on

ai-system-design (amitshekhariitbhu/ai-system-design)

Searching Database…

705 ·GitHub
Ai-system-design
Что это такое?Бесплатный open-source гайд, который шаг за шагом учит AI System Design: как объединить GPU, инференс-серверы, кеши, векторные базы данных, AI-агентов, guardrails и evals в одну систему, которая работает быстро, дёшево, надёжно и безопасно.
Это интерактивный курс или просто длинный документ для чтения?Это письменный гайд, который живёт в README репозитория: оглавление, где каждая тема объяснена простыми словами, а там, где тема заслуживает более глубокого разбора, даётся ссылка на подробную отдельную статью в блоге.
Как мне её проходить — от начала до конца или по частям?Если вы новичок, читайте сверху вниз — каждый раздел опирается на предыдущий. Для подготовки к собеседованиям начните с фреймворка 'How to Solve Any AI System Design Problem', а затем вернитесь к базовым строительным блокам.
Мне нужно заранее знать машинное обучение, чтобы разобраться?Нет. Каждый раздел написан для новичков: без жаргона и без предположений о ваших знаниях, а каждый термин объясняется до того, как используется, поэтому гайд можно спокойно читать с нуля.
Это полный курс AI-инжиниринга или узко сфокусированный материал?Гайд сфокусирован на системном дизайне AI. Для полного пути AI-инженера — машинное обучение, deep learning, трансформеры, fine-tuning — автор отсылает к своему отдельному курсу AI Engineering Course.
Для кого это написано?Для софтверных инженеров, переходящих в AI-инжиниринг, бэкенд-, мобильных и фронтенд-разработчиков, создающих AI-продукты, ML-инженеров, выводящих модели в продакшен, менеджеров и архитекторов, студентов и всех, кто готовится к собеседованиям по AI system design.
Могу ли я свободно читать, использовать и делиться этим?Да — это open source под лицензией Apache-2.0, читать можно бесплатно; в конце гайда есть раздел License.
Какие темы реально раскрывает гайд?От основ LLM-инференса (токены, железо GPU/TPU/LPU, prefill против decode, TTFT) через масштабирование и параллелизм, KV- и семантическое кеширование, LLM-роутинг, RAG (чанкинг, гибридный поиск, reranking, GraphRAG), context engineering, AI-агентов (MCP, Agent Skills, память, мультиагентность, A2A), голосовой AI, guardrails, observability, evals, оптимизацию затрат и оптимизацию инференса вроде квантизации и speculative decoding.
Какие реальные системы он разбирает?Три кейс-стади: как устроен Claude Code, как устроен Cursor и как спроектировать голосового AI-агента реального времени — рядом с фреймворком из восьми шагов для решения любой задачи по AI system design и списком типичных вопросов с собеседований.
Гайд уже закончен или продолжает расти?Продолжает расти. Автор отмечает, что AI system design развивается быстро, продолжает добавлять статьи на новые темы и советует добавить гайд в закладки и возвращаться, когда захотите освежить знания.
Чем AI System Design отличается от обычного System Design?По формуле автора, это обычный системный дизайн плюс новые ограничения AI-моделей: GPU, токены, длинные стриминговые ответы, недетерминированный результат и реальная стоимость каждого отдельного запроса.
What are similar projects?ai-engineering-interview-questions-company-wise, ai-agents-the-definitive-guide, llm_interview_note, ai-engineer-notebooks, ai-agent-book, llm-inference-engineering, ai-engineering-hub, AgentGuide
Which topics is it in?system-design-interview, ai-engineering, flash-attention, cost-optimization, graphrag, guardrails, a2a, voice-ai, ai-gateway, speculative-decoding, mixture-of-experts, language-model
Creator
amitshekhariitbhu
Size
0.1 MB
Stars
705
Updated
CommitOct 14
Age
23d
Commits · 90 Days