The AI gateway,
from the ground up

1.1  The 10,000-ft view: what are we even building?

Before any single idea makes sense, you need the map. Everything in this course is one block inside the picture below. We will keep coming back to it and zooming in.

The thing we are building is an AI gateway: a piece of infrastructure that sits between users and a large-language-model service, and inspects every prompt on the way in (to block data leaks and attacks) and every response on the way out. It runs on Kubernetes. Follow one request from the top of the figure to the bottom.

The two halves: control plane vs data plane

The single most important split in the whole system is between the data plane and the control plane. Almost every confusion later dissolves once this is clear.

The data plane is the part that touches every single request. In our picture that is Envoy (a proxy written in C++). If a million prompts arrive, the data plane runs a million times. So it must be brutally fast — microseconds per request — and it must never block. This is why it is C++ and why later topics (zero-copy parsing, non-blocking I/O, caches) all live here: they are all in service of "do the per-request work in microseconds."

The control plane is the part that decides the rules but does not touch individual requests. In our picture that is a Go service. When an operator changes a policy ("also block credit-card numbers"), they change it in the control plane. The control plane runs rarely — seconds or minutes apart — so it can be written in a comfortable, higher-level language (Go) and do slow, careful work.

What is written where — and what you can change

A fair question a learner always has but rarely gets answered: where does all this "config" actually live, and what can I touch? Here is the honest chain:

• You write YAML. Desired state — "run 3 replicas of the model service," "route /v1/chat to it," "block SSNs" — is written as YAML files (Kubernetes objects, and Custom Resource Definitions the mesh adds).
• Kubernetes stores it in etcd. When you kubectl apply a YAML file, it is validated and saved into etcd, the cluster's single source of truth (a distributed key-value database). Nothing is "running" yet — you've only recorded intent.
• Controllers read it and act. Controllers (including the mesh's control plane) watch etcd, notice the new desired state, and make reality match — including translating your intent into low-level Envoy config.
• The control plane pushes to the data plane over xDS. Envoy never reads your YAML directly. The control plane converts it and streams it to every running Envoy over a gRPC API called xDS, live, with no restart.

So "can I modify it?" — yes: you edit YAML and apply it. "What does it take?" — a valid change to a Kubernetes object, which flows through etcd → controller → xDS → data plane in seconds, without dropping a connection. You almost never edit Envoy's raw config or restart anything by hand; that's the whole point of the control plane.

1.2  TLS and mTLS: how the traffic is protected, and how the gateway reads it anyway

The problem TLS solves

The network between a user and your server is hostile. Anyone on the path — a coffee-shop router, an ISP, a compromised switch — can by default read your traffic, modify it, or impersonate either end. TLS (Transport Layer Security, the "S" in HTTPS) fixes all three: it gives you confidentiality (eavesdroppers see gibberish), integrity (tampering is detected), and authentication (you know you're really talking to the server you meant).

How the handshake works — and why each step exists

Every HTTPS connection begins with a handshake. Walk it slowly, because every step answers a specific attack:

• ClientHello. The client says which TLS versions and cipher suites it supports, and which host it wants (SNI). Why: the two sides must agree on algorithms before they can speak securely.
• ServerHello + certificate. The server picks a cipher and sends its certificate (its public key, signed by a CA). Why: the client needs the server's public key to establish a secret, and needs proof of identity so it isn't talking to an impostor.
• The client verifies the certificate chain. It checks the CA's signature up to a CA it already trusts, checks the hostname matches, and checks it hasn't expired or been revoked. Why: this is the step that stops impersonation. Skip it — or click through the warning — and every other guarantee collapses.
• Key exchange. Using asymmetric crypto, both sides derive a shared symmetric session key that no eavesdropper can compute. Why: asymmetric crypto is slow, so it's used only to agree on a fast symmetric key, which then encrypts the actual data. Best of both worlds.
• Encrypted application data. From here the request and response flow under fast symmetric encryption.

Termination: why the gateway can read the prompt

"TLS termination" means a proxy holds the private key for the hostname and ends the encrypted tunnel at itself: it decrypts the request, and for a moment the prompt exists as plaintext in the proxy's memory. That plaintext moment is the only reason inspection is possible at all — you cannot scan ciphertext. The proxy then opens a separate TLS connection onward to the model and re-encrypts. Two tunnels, not one.

mTLS: from one-way trust to mutual trust

Normal TLS is one-way: the server proves who it is, the client stays anonymous. That's right for the public internet (your bank shouldn't need a certificate from you). But inside a cluster, where dozens of services call each other, you want the opposite default: every service should prove its identity to every other, and no service should trust another just because it's "on the network." That is mTLS (mutual TLS): the client also presents a certificate, and the server verifies it too.

1.3  Bloom filters: how to ask "have I seen this?" in constant time

The problem

The inspection pipeline must, for every request and within a microsecond budget, ask a cheap screening question: "could this text contain any of the thousands of known-bad patterns we watch for?" The obvious approaches are too slow or too big. Comparing against every pattern is O(n) in the number of patterns. Keeping an exact set of everything seen costs a lot of memory. We want something that is constant-time and tiny, and we're willing to accept a small, controlled imperfection to get it.

What a Bloom filter is

A Bloom filter is a probabilistic set. It can tell you two things about membership: "definitely not in the set" (with certainty) or "possibly in the set" (with a small chance of being wrong). It never says "definitely yes," and — crucially — it never produces a false negative: if something really is in the set, the filter will never say no.

How it works, step by step

You keep a bit array of m bits, all zero to start, and k independent hash functions. To add an item, hash it with all k functions to get k positions, and set those bits to 1. To query an item, hash it the same way and look at those same k bits:

• If any of the k bits is 0, the item was never added — definitely not present. (Adding it would have set that bit.)
• If all k bits are 1, the item is possibly present — but those bits might have been set by other items that happened to hash there. That's a false positive.

Why it's O(1) — the part that's usually skipped

Evidence: the false-positive rate is a knob, not a mystery

Where it lives in our architecture

Back to the map. Inside the data plane's INSPECT block, the Bloom filter is the cheap first gate. Because ~99% of real traffic is clean, the filter answers "definitely no known-bad pattern here" for the vast majority of requests in a handful of nanoseconds, and they skip the expensive scan entirely. Only the small slice that comes back "maybe" pays for the real, exact matcher (Aho-Corasick, in Part 3). The Bloom filter's job is not to be the final judge — it's to make the common case nearly free.

2.1  L4 vs L7: where the proxy operates decides what it can do

The problem

A proxy has to make decisions: where does this go, is it allowed, can I inspect it? What it's able to decide depends entirely on how far up the stack it reads. And here's the catch that ties back to Part 1: the HTTP request is inside the TLS encryption. So a proxy that doesn't terminate TLS literally cannot see the URL or body — no matter how much it wants to.

How each works — and why the difference is fundamental

An L4 proxy operates on the connection. It reads the IP/port 4-tuple, picks a backend, and then just shovels bytes back and forth. It never parses HTTP; it often never decrypts anything. That makes it extremely cheap and protocol-agnostic — it'll happily proxy HTTP, a database wire protocol, or anything else. Why it's fast: forwarding a byte stream is almost no work per request.

An L7 proxy operates on the request. To do that it must terminate TLS (Part 1), then parse the HTTP — find the method, the path, the headers, the body. Now it can route by URL, add or check headers, retry failed requests, enforce authentication, and — the whole point of our gateway — inspect the content. Why it costs more: TLS termination plus parsing every request is real per-request work.

2.2  Rate limiting: how Nginx and Envoy actually build it

The problem

A backend has finite capacity. Without limits, one buggy client or one attacker can flood it and take everyone down, and there's no fairness between clients. Rate limiting caps how much any one client can consume, sheds excess load early (before it reaches the expensive parts), and gives you a knob for fairness and abuse control.

How it works — the two classic algorithms

Almost every real limiter is a variation of two ideas:

• Token bucket. A bucket refills with tokens at a steady rate R and holds at most B. Each request removes one token; if the bucket is empty, the request is rejected (or delayed). Why it's the favourite: it cleanly separates the two concerns — R sets the sustained rate, B sets how big a burst you'll absorb. A client that's been quiet has a full bucket and can burst; a client hammering you drains it and gets throttled to R.
• Leaky bucket. Requests pour into a queue that drains at a fixed rate; overflow is dropped. This smooths output to a constant rate — good when the backend needs an even flow rather than bursts.
• Fixed vs sliding window. The naive "count per calendar minute" is a fixed window; it's simple but allows nearly 2× the limit across a window boundary (a burst at 11:59:59 and another at 12:00:00). A sliding window weights the current and previous windows to smooth that edge.

How Nginx builds it — the limit_req module

Nginx's limit_req implements a leaky-bucket limiter. You declare a shared-memory zone keyed by some variable (usually the client IP), give it a rate, then apply it:

# define a 10 MB shared zone keyed by client IP, sustaining 10 req/sec
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

server {
  location /v1/ {
    # allow a burst of 20 queued requests; nodelay = serve the burst
    # immediately instead of spacing it out, then enforce 10r/s
    limit_req zone=perip burst=20 nodelay;
    proxy_pass http://model_upstream;
  }
}

How Envoy builds it — local vs global

Envoy offers two rate-limit filters, and the difference matters enormously at scale.

Local rate limiting is a token bucket inside each Envoy instance — no external dependency, just config:

token_bucket:
  max_tokens: 100          # bucket capacity B (burst)
  tokens_per_fill: 100     # add 100 tokens...
  fill_interval: 1s        # ...every second → sustained R = 100/s

It's fast and simple, but it's per instance. Run three Envoys and your "100/s" limit is really 300/s — usually not what you meant.

Global rate limiting fixes that. Each Envoy, instead of deciding locally, sends the request's descriptors (key–value labels it extracts, e.g. {client_id: acme}) to an external Rate Limit Service (RLS) over gRPC. The RLS — typically backed by Redis for a shared counter — returns OK or over-limit, so all Envoys share one consistent budget.

3.1  Aho-Corasick: matching thousands of patterns in one pass

The problem

Our inspection needs to check every request against a big dictionary of literal patterns — known secret prefixes (like an API-key header shape), PII formats, banned strings — thousands of them, within a microsecond budget. Scanning the text once per pattern is far too slow. We want to find every occurrence of every pattern in essentially one pass over the text.

What Aho-Corasick is, and how it works

Aho-Corasick compiles the entire pattern set into a single automaton once, up front, and then scans any text through it in linear time. It has two ingredients:

• The trie (goto edges). Build a trie of all patterns. Walking the text down the trie matches a pattern the moment you reach an accepting node.
• Failure links (the clever part). When the next character doesn't continue the current path, you must not throw away progress and restart. A failure link points from each node to the node representing the longest proper suffix of what you've matched so far that is also a valid prefix of some pattern. So after matching "sh" then seeing "e", if that branch dead-ends you jump to the "he" state — because "he" is a suffix already sitting in your input — instead of re-reading characters.

3.2  The full pipeline: normalize → Bloom → Aho-Corasick → verdict

The problem

Every prompt must get a verdict — allow, redact, or block — before it reaches the model, within about a millisecond, safely, and without adding latency the user can feel. No single technique does all of that, so we stage them.

How it works — and why each stage is where it is

• 1 · Normalize (first, always). Unicode-fold, decode any encodings, strip zero-width characters, lowercase. Why first: if you scanned before normalizing, an attacker would bypass every literal pattern with a homoglyph or a percent-encoding. Canonicalizing first collapses those tricks into the plain form the matcher understands.
• 2 · Bloom prefilter (cheap gate). Ask the O(1) filter "could this contain any watched pattern?" Why second: ~99% of real traffic is clean, and the Bloom filter clears it in nanoseconds, so the expensive scan runs only on the small "maybe" slice. Cheapest filter before costly one — always.
• 3 · Aho-Corasick (exact scan). On the survivors, run the automaton to find the actual matches. Why: the Bloom filter only says "maybe"; this confirms real hits and tells you which patterns and where.
• 4 · Verdict → action. Map matches to a decision: nothing found → allow; a redactable match (a PII span) → redact it and continue; a hard-block match → return 403, and the model never sees the prompt.

4.1  The cluster: a control plane that decides, nodes that run

The problem, and the answer: two kinds of machine

Turning a heap of servers into one declarative system needs a division of labour. Kubernetes splits every cluster into a control plane (the brain — decides and records) and worker nodes (the muscle — actually run your pods). This is the same brain/muscle split as the Envoy control-plane/data-plane idea from Part 1, one level down: it's the pattern Kubernetes is built on.

What each control-plane piece does — and why it's separate

• API server — the single front door. Everything — your kubectl, the controllers, every node's kubelet — talks only to it. Why one door: one place to authenticate, validate, and audit every change. Nothing writes cluster state except through here.
• etcd — the source of truth: a distributed key-value store (kept consistent by the Raft protocol) holding all desired and actual state. Why: the whole system needs one authoritative, consistent record; lose etcd and you've lost the cluster's memory.
• Scheduler — watches for pods with no node yet and picks one, weighing free CPU/memory, affinity rules, and constraints. Why separate: placement is a genuine optimization problem, isolated so it can evolve on its own.
• Controller manager — runs the reconcile loops (Deployment → ReplicaSet → Pod, and dozens more), continuously nudging actual state toward desired. Why: this is the self-healing engine you saw in the reconcile-loop topic, living here.

On each worker node: the kubelet watches the API server for pods assigned to its node, starts their containers via the container runtime, and reports health back; kube-proxy programs the node's networking for Services (next section).

4.2  Services: a stable address over ever-changing pods

The problem

Your gateway needs to call "the model service," which is really 3 pods that keep getting replaced with new IPs. If it dialed a pod IP directly, the first reschedule would break it. Something must provide a fixed address that always lands on a live pod.

What a Service is, and how it works

A Service is that fixed address: a stable VIP and DNS name that load-balances across a dynamic set of pods chosen by a label selector.

apiVersion: v1
kind: Service
metadata: {{ name: model }}
spec:
  selector: {{ app: model }}     # which pods this Service fronts
  ports: [{{ port: 80, targetPort: 8080 }}]
  type: ClusterIP            # internal-only stable VIP

• The selector (app: model) continuously matches pods; a controller keeps an EndpointSlice — the current list of live pod IPs — in sync as pods come and go.
• kube-proxy on every node reads those endpoints and programs the node's kernel (via iptables, IPVS, or eBPF) so any packet sent to the VIP is transparently rewritten to a healthy pod.
• CoreDNS resolves model.default.svc.cluster.local to the VIP, so callers use a name, not an IP.

4.3  VirtualServices: L7 traffic rules a Service can't express

The problem

A plain Service can't do the things real traffic management needs: send 10% of requests to a new version (canary), route /v2/ somewhere else, match on a header for A/B tests, add retries and timeouts, or mirror traffic for testing. All of those are L7 decisions, and a Service is L4. You want to express them declaratively, without touching app code.

What a VirtualService is, and how it works

A VirtualService (Istio; the vendor-neutral equivalent is the Gateway API's HTTPRoute) is a CRD describing L7 routing for a host: match rules and weighted destinations. A companion DestinationRule names the subsets (e.g. v1, v2) it can split between.

kind: VirtualService
spec:
  hosts: [ model ]
  http:
  - route:
    - destination: {{ host: model, subset: v1 }}
      weight: 90            # 90% to the stable version
    - destination: {{ host: model, subset: v2 }}
      weight: 10            # 10% canary to the new version

5.1  Why C++ is fast — and the evidence, not just the claim

The problem with the claim

We've said repeatedly "the data plane is C++ because it's fast." That's lazy unless we can say why and by how much. So let's break down the real sources of C++'s speed, put numbers on them, and — importantly — be honest about where the advantage is small.

Where the speed actually comes from

• Native code, no warmup. C++ compiles ahead of time to machine code; there's no bytecode interpreter and no JIT that must observe the program running before it optimizes. Full speed from the first request.
• No garbage collector. Memory is freed deterministically (RAII, Part on C++ memory), so there are no stop-the-world GC pauses. This is the single biggest win for tail latency.
• Control over memory layout — the deep one. You decide exactly how data sits in memory, so you can keep it contiguous and cache-friendly.
• Zero-cost abstractions. Templates and inlining compile away; a well-written abstraction generates the same code you'd write by hand — you don't pay at runtime for the structure.
• No mandatory safety overhead. No forced bounds or null checks on every access (a double-edged sword we'll pay for in the attack section).

5.2  string_view: the borrow that makes parsing free — and the trap

The problem

Parsing one HTTP request means pulling out many substrings — method, path, protocol, and every header name and value. If each of those is a std::string, each one allocates a heap buffer and copies bytes that are already sitting in memory in the receive buffer. That's dozens of tiny allocations per request, multiplied by your entire request volume — death by a thousand mallocs, right on the hot path we just spent 5.1 protecting.

What string_view is, and how it works

A std::string_view is a non-owning pair: a const char* pointer and a size_t length — about 16 bytes total. It points into characters that live somewhere else and copies none of them.

• No allocation, ever. Constructing a view stores a pointer and a length. That's it — no heap, no copy.
• Substring ops are O(1). remove_prefix, substr, trimming — they just move the pointer or shrink the length. No characters are touched.
• One type for all sources. A string_view can be built from a std::string, a C string, or a raw char[], so a function taking one parameter reads from any of them without conversions.
• Why it's fast: zero allocations, zero copies, and it points into a buffer that's already hot in cache — it inherits the cache-locality win from 5.1.

6.1  The kill chain: one attacker, edge to core

The problem with piecemeal defense

Each earlier part hardened one stage. But an attacker doesn't attack "a stage" — they look for the cheapest complete path from the outside to your data. If any single stage is a soft bypass, the strength of the others may not matter. So we walk the chain from the wire inward, and at each stop name the move and the control that stops it.

Stage by stage

6.2  The three principles that answer the whole playbook

Every defense in the walk above is an instance of just three ideas. If you internalize these, you can reason about a threat you've never seen.

• Defense in depth. Stack independent controls so an attacker must beat all of them, and each one shrinks the blast radius of a failure in another. Beating the edge doesn't beat the parser; beating the parser doesn't beat inspection.
• Fail closed. Every check denies by default when it errors or times out. This is what stops an attacker from turning a crash into a bypass — a broken scanner blocks traffic rather than waving it through.
• Zero trust. No layer trusts another because of where it sits. Identity is proven cryptographically on every hop (mTLS), and every actor gets the least privilege it needs (RBAC). A foothold in one place can't impersonate another.