Kubernetes,
from the satellite view
to the wire.

What is Kubernetes, and why does anyone need it?

Before any YAML, one honest question: what problem was so painful that Google, and then the whole industry, built this thing?

The one sentence that defines Kubernetes

Hold onto that picture. A Deployment is a reconciliation loop. The scheduler is a reconciliation loop. The kubelet is a reconciliation loop. Istio's control plane is a reconciliation loop. Once you see the loop, Kubernetes stops being 40 confusing words and becomes one repeated idea.

The data center: where your pod physically lives

People learn Kubernetes as if it floats in the cloud. It doesn't. Every pod is a Linux process on a real machine, in a real rack, drawing real power, connected by real cables. Understanding the physical layout explains why zones, topology spread, and network latency exist.

The hierarchy: region → zone → data center → rack → server

The network inside: spine-leaf (Clos) fabric

This table is the entire reason multi-region architecture exists. "Sub-millisecond response" is only possible for the part of the journey you control — inside the data center. So the game is: get the user's packet to the nearest region fast (anycast), then never leave the building.

Docker: what a container really is

Kubernetes runs containers, so we must be brutally clear on this: a container is not a small virtual machine. It is a normal Linux process wearing three disguises.

The three disguises (this is the whole trick)

Images and layers — why builds are fast (or slow)

A production multi-stage Dockerfile, line by line

Dockerfile# ---------- STAGE 1: build (has JDK + Maven; thrown away) ----------
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src

# Copy ONLY the dependency manifest first. If pom.xml is unchanged,
# this layer (and the slow dependency download) comes from cache.
COPY pom.xml .
RUN mvn -q dependency:go-offline

# Now copy source — the only part that changes on every commit.
COPY src ./src
RUN mvn -q package -DskipTests

# ---------- STAGE 2: runtime (tiny, what actually ships) ----------
FROM eclipse-temurin:21-jre-alpine AS runtime

# Never run as root. Create an unprivileged user.
RUN addgroup -S app && adduser -S app -G app
USER app
WORKDIR /app

# Pull ONLY the jar from stage 1. Maven, JDK, source: gone.
COPY --from=build --chown=app:app /src/target/payment-api.jar app.jar

# Document the port (metadata only — does not open anything).
EXPOSE 8080

# Container-level health check (K8s uses its own probes instead,
# but this helps plain-Docker and docker compose setups).
HEALTHCHECK --interval=15s --timeout=3s --retries=3 \
  CMD wget -qO- http://localhost:8080/healthz || exit 1

# exec form (JSON array) = java becomes PID 1 and receives SIGTERM
# directly. Shell form ("java -jar ...") would wrap it in /bin/sh,
# which swallows signals → broken graceful shutdown.
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]

docker compose — the whole dev environment in one file

docker-compose.ymlservices:
  payment-api:
    build:
      context: .
      dockerfile: Dockerfile
      target: runtime          # build only up to the runtime stage
    ports:
      - "8080:8080"            # hostPort:containerPort
    environment:
      SPRING_PROFILES_ACTIVE: local
      DB_HOST: postgres        # service name IS the DNS name
      REDIS_HOST: redis
    depends_on:
      postgres:
        condition: service_healthy  # wait until pg healthcheck passes
      redis:
        condition: service_started
    deploy:
      resources:
        limits: { cpus: "2", memory: 1g }   # cgroups, same idea as K8s limits

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: payments
      POSTGRES_PASSWORD: localdev   # local only — never in prod files
    volumes:
      - pgdata:/var/lib/postgresql/data # named volume: data survives restarts
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10

  redis:
    image: redis:7-alpine
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]

volumes:
  pgdata:

Kubernetes architecture: the machine that runs the loop

Now open the hood. A cluster is just two kinds of machines: a few control plane nodes (the brain) and many worker nodes (the muscle). Every component below is one of those thermostats from Altitude 01 — each guards a different promise.

Each component, and the promise it guards

Life of a pod: from kubectl apply to running process

This sequence is the most asked "explain Kubernetes" interview question. Walk it slowly — every arrow is a separate actor doing its own small job:

Pod lifecycle & the termination dance (where outages hide)

Every core object, full YAML, every field explained

This is the working library. One realistic app — payment-api in namespace payments — expressed as complete, production-shaped manifests. Read the comments; they carry the "why" for each field.

1 · Namespace + ResourceQuota — the apartment walls

00-namespace.yamlapiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    team: payments-platform
    istio-injection: enabled      # Istio auto-injects Envoy sidecars here
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: payments-quota
  namespace: payments
spec:
  hard:
    requests.cpu: "200"          # team may request at most 200 cores total
    requests.memory: 400Gi
    limits.cpu: "400"
    limits.memory: 800Gi
    pods: "500"

2 · ConfigMap — configuration that is NOT secret

10-configmap.yamlapiVersion: v1
kind: ConfigMap
metadata:
  name: payment-api-config
  namespace: payments
data:
  # Simple keys → injected as env vars below
  DB_HOST: "payments-db.payments.svc.cluster.local"
  DB_POOL_SIZE: "40"
  CACHE_TTL_SECONDS: "300"
  FEATURE_NEW_LEDGER: "false"
  # Whole file → mounted as a volume below
  application.yaml: |
    server:
      shutdown: graceful
    management.endpoints.web.exposure.include: health,prometheus

3 · Secret — the same idea, but access-controlled

11-secret.yamlapiVersion: v1
kind: Secret
metadata:
  name: payment-api-secrets
  namespace: payments
type: Opaque
stringData:                       # stringData = write plain, stored base64
  DB_PASSWORD: "REPLACED-BY-EXTERNAL-SECRETS"
  STRIPE_API_KEY: "REPLACED-BY-EXTERNAL-SECRETS"
---
# Registry credential so kubelet can pull private images
apiVersion: v1
kind: Secret
metadata:
  name: regcred
  namespace: payments
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6ey4uLn19   # base64 of docker config

4 · ServiceAccount + RBAC — identity and permissions (auth inside the cluster)

20-rbac.yamlapiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-api-sa
  namespace: payments
automountServiceAccountToken: false  # don't mount API creds unless needed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role                          # Role = namespaced; ClusterRole = cluster-wide
metadata:
  name: payment-api-role
  namespace: payments
rules:
- apiGroups: [""]                  # "" = core API group
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["payment-api-secrets"]  # ONLY this one secret
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding                   # glue: who ← gets → which role
metadata:
  name: payment-api-binding
  namespace: payments
subjects:
- kind: ServiceAccount
  name: payment-api-sa
  namespace: payments
roleRef:
  kind: Role
  name: payment-api-role
  apiGroup: rbac.authorization.k8s.io

5 · Deployment — the centerpiece (read every comment)

30-deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: payments
  labels: { app: payment-api, version: v1 }
spec:
  replicas: 6                       # HPA will own this number later
  revisionHistoryLimit: 5           # keep 5 old ReplicaSets for rollback
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%                # may run 25% extra pods during rollout
      maxUnavailable: 0            # NEVER dip below desired capacity
  selector:
    matchLabels: { app: payment-api }   # must match template labels; immutable
  template:                         # ↓ the Pod recipe stamped per replica
    metadata:
      labels: { app: payment-api, version: v1 }
      annotations:
        prometheus.io/scrape: "true"       # telemetry: scrape me
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      serviceAccountName: payment-api-sa
      imagePullSecrets:
      - name: regcred
      terminationGracePeriodSeconds: 45  # > preStop(8) + slowest request

      # ---- spread replicas so one failure domain can't take us out ----
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone   # spread across zones
        whenUnsatisfiable: DoNotSchedule
        labelSelector: { matchLabels: { app: payment-api } }
      affinity:
        podAntiAffinity:                              # avoid same node
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector: { matchLabels: { app: payment-api } }
              topologyKey: kubernetes.io/hostname

      # ---- pod-level security: applies to all containers ----
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile: { type: RuntimeDefault }

      # ---- init container: runs to completion BEFORE the app ----
      initContainers:
      - name: wait-for-db
        image: busybox:1.36
        command: ["sh","-c","until nc -z $DB_HOST 5432; do echo waiting; sleep 2; done"]
        env:
        - name: DB_HOST
          valueFrom: { configMapKeyRef: { name: payment-api-config, key: DB_HOST } }

      containers:
      - name: payment-api
        image: registry.tmo.example/payments/payment-api:1.4.2  # pinned, never :latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8080

        # ---- config in: env from ConfigMap + Secret ----
        envFrom:
        - configMapRef: { name: payment-api-config }
        - secretRef:    { name: payment-api-secrets }
        env:
        - name: POD_NAME                       # downward API: pod tells app who it is
          valueFrom: { fieldRef: { fieldPath: metadata.name } }
        - name: NODE_NAME
          valueFrom: { fieldRef: { fieldPath: spec.nodeName } }

        # ---- resources: THE most important 6 lines for stability ----
        resources:
          requests:            # scheduler reserves this much (bin packing)
            cpu: "1"
            memory: 1Gi
          limits:               # cgroup hard caps
            memory: 1Gi        # limit == request → predictable, no OOM surprises
                                # NOTE: no cpu limit → avoids CFS throttling latency;
                                # many latency-sensitive teams do exactly this

        # ---- the three probes ----
        startupProbe:           # protects slow JVM boot from liveness kills
          httpGet: { path: /actuator/health/liveness, port: http }
          failureThreshold: 30
          periodSeconds: 2    # up to 60s allowed to start
        readinessProbe:         # gate for traffic; fail = removed from LB, NOT restarted
          httpGet: { path: /actuator/health/readiness, port: http }
          periodSeconds: 5
          failureThreshold: 2
        livenessProbe:          # deadlock detector; fail = container restarted
          httpGet: { path: /actuator/health/liveness, port: http }
          periodSeconds: 10
          failureThreshold: 3  # be forgiving; liveness restarts cause cascades

        # ---- graceful shutdown (see timeline above) ----
        lifecycle:
          preStop:
            exec: { command: ["sh","-c","sleep 8"] }

        # ---- container-level hardening ----
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities: { drop: ["ALL"] }

        volumeMounts:
        - name: app-config
          mountPath: /app/config
          readOnly: true
        - name: tmp                # app needs SOME writable dir with RO rootfs
          mountPath: /tmp

      volumes:
      - name: app-config
        configMap: { name: payment-api-config, items: [{ key: application.yaml, path: application.yaml }] }
      - name: tmp
        emptyDir: {}

6 · Service — a stable virtual IP over churning pods

40-service.yamlapiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments
  labels: { app: payment-api }
spec:
  type: ClusterIP            # internal-only; Ingress/mesh will expose it
  selector: { app: payment-api }   # pods with this label = backends
  ports:
  - name: http               # NAMED port: Istio uses names for protocol detection
    port: 80                 # what callers dial
    targetPort: http         # the container port name
    protocol: TCP

7 · HorizontalPodAutoscaler — the elastic waistband

50-hpa.yamlapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-api
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  minReplicas: 6
  maxReplicas: 120
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 60 }  # % of REQUESTS
  - type: Pods                # custom metric via Prometheus Adapter:
    pods:                     # scale on real load, not just CPU
      metric: { name: http_requests_per_second }
      target: { type: AverageValue, averageValue: "800" }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # react instantly to spikes
      policies: [{ type: Percent, value: 100, periodSeconds: 30 }]  # may double / 30s
    scaleDown:
      stabilizationWindowSeconds: 300   # cool down slowly — avoid flapping

8 · PodDisruptionBudget — armor against "helpful" maintenance

51-pdb.yamlapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
  namespace: payments
spec:
  minAvailable: 80%
  selector: { matchLabels: { app: payment-api } }

9 · NetworkPolicy — firewalls between pods

60-networkpolicy.yaml# 1) default deny everything in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: payments }
spec:
  podSelector: {}              # all pods here
  policyTypes: [Ingress, Egress]
---
# 2) allow only: gateway → payment-api, payment-api → db + dns
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: payment-api-allow, namespace: payments }
spec:
  podSelector: { matchLabels: { app: payment-api } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: istio-ingress } }
    ports: [{ port: 8080, protocol: TCP }]
  egress:
  - to: [{ podSelector: { matchLabels: { app: payments-db } } }]
    ports: [{ port: 5432, protocol: TCP }]
  - to: [{ namespaceSelector: {}, podSelector: { matchLabels: { k8s-app: kube-dns } } }]
    ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]  # never forget DNS!

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: payments-db, namespace: payments }
spec:
  serviceName: payments-db-headless   # gives payments-db-0.payments-db-headless DNS
  replicas: 3
  selector: { matchLabels: { app: payments-db } }
  template:
    metadata: { labels: { app: payments-db } }
    spec:
      containers:
      - name: postgres
        image: postgres:16
        ports: [{ containerPort: 5432, name: pg }]
        volumeMounts: [{ name: data, mountPath: /var/lib/postgresql/data }]
  volumeClaimTemplates:               # EACH replica gets ITS OWN PersistentVolume
  - metadata: { name: data }
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: fast-ssd
      resources: { requests: { storage: 200Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: payments-db-headless, namespace: payments }
spec:
  clusterIP: None                     # headless: DNS → pod IPs directly
  selector: { app: payments-db }
  ports: [{ port: 5432, name: pg }]

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: node-exporter, namespace: monitoring }
spec:
  selector: { matchLabels: { app: node-exporter } }
  template:
    metadata: { labels: { app: node-exporter } }
    spec:
      tolerations: [{ operator: Exists }]   # run even on tainted nodes
      hostNetwork: true                       # see the real node network
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.8.1
        ports: [{ containerPort: 9100 }]

apiVersion: batch/v1
kind: CronJob
metadata: { name: settlement-report, namespace: payments }
spec:
  schedule: "0 2 * * *"              # 2 AM daily
  concurrencyPolicy: Forbid          # never overlap runs
  jobTemplate:
    spec:
      backoffLimit: 3                 # retry failed pods 3×
      activeDeadlineSeconds: 3600     # kill if running > 1h
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: report
            image: registry.tmo.example/payments/settlement:2.1.0
            args: ["--date=yesterday"]

1,000,000,000 requests — and answering in sub-millisecond time

First, kill the fear with arithmetic. A billion requests per day is:

Ninety pods is a Tuesday for Kubernetes. The hard part is never the count — it's the latency budget and the failure math. Now the honest physics:

The server-side microsecond budget (where 1 ms actually goes)

Total ≈ 800 µs. Notice what is absent: no relational DB on the hot path (a Postgres query is 1–10 ms by itself — it alone blows the budget), no cross-zone hop (+1–2 ms), no TLS full handshake (session resumption/TLS 1.3 0-RTT keep it off the steady state), no cold JIT. The budget is a list of things you're not allowed to do.

The full journey of one request (follow the packet)

Why anycast? (the trick that shrinks the internet)

Layered load balancing — each layer answers a different question

Istio, Envoy, Nginx — the traffic layer, demystified

Istio YAML: gateway, routing, resilience — complete set

70-istio-gateway.yaml# Gateway: which ports/hosts the ingress Envoy listens on (L4–L6)
apiVersion: networking.istio.io/v1
kind: Gateway
metadata: { name: payments-gw, namespace: payments }
spec:
  selector: { istio: ingressgateway }   # which gateway pods implement this
  servers:
  - port: { number: 443, name: https, protocol: HTTPS }
    hosts: ["pay.tmo.example"]
    tls:
      mode: SIMPLE
      credentialName: pay-tmo-example-tls   # Secret from cert-manager (next altitude)
  - port: { number: 80, name: http, protocol: HTTP }
    hosts: ["pay.tmo.example"]
    tls: { httpsRedirect: true }           # force HTTPS

71-virtualservice.yaml# VirtualService: L7 routing rules — the "if this request, then where"
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payment-api-vs, namespace: payments }
spec:
  hosts: ["pay.tmo.example"]
  gateways: [payments-gw]
  http:
  - name: canary-internal-testers       # header-based canary first
    match:
    - headers: { x-canary: { exact: "true" } }
    route:
    - destination: { host: payment-api.payments.svc.cluster.local, subset: v2 }
  - name: weighted-rollout              # then 95/5 traffic split
    route:
    - destination: { host: payment-api.payments.svc.cluster.local, subset: v1 }
      weight: 95
    - destination: { host: payment-api.payments.svc.cluster.local, subset: v2 }
      weight: 5
    timeout: 2s                         # hard deadline for the whole call
    retries:
      attempts: 2
      perTryTimeout: 800ms
      retryOn: connect-failure,refused-stream,503  # retry ONLY safe failures

72-destinationrule.yaml# DestinationRule: what happens AFTER routing — subsets, LB, circuit breaking
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: payment-api-dr, namespace: payments }
spec:
  host: payment-api.payments.svc.cluster.local
  subsets:                              # subsets = label-selected pod groups
  - { name: v1, labels: { version: v1 } }
  - { name: v2, labels: { version: v2 } }
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST            # better tail latency than round robin
      localityLbSetting: { enabled: true }  # prefer same-zone pods (the 10× trick)
    connectionPool:
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 0    # 0 = unlimited; keep HTTP/2 conns hot
      tcp: { maxConnections: 200 }
    outlierDetection:                   # circuit breaker: eject sick pods
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50           # never eject > half the fleet

73-security.yaml# Enforce mTLS for every workload in the namespace
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: strict-mtls, namespace: payments }
spec:
  mtls: { mode: STRICT }              # plaintext connections rejected
---
# L7 authorization: WHO may call payment-api, and how
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: payment-api-authz, namespace: payments }
spec:
  selector: { matchLabels: { app: payment-api } }
  action: ALLOW
  rules:
  - from:
    - source:
        principals:               # SPIFFE identity from the mTLS cert!
        - cluster.local/ns/istio-ingress/sa/ingress-gw-sa
    to:
    - operation: { methods: [GET, POST], paths: ["/api/*"] }
  - from:
    - source: { principals: [cluster.local/ns/payments/sa/fraud-svc-sa] }
    to:
    - operation: { methods: [GET], paths: ["/internal/risk/*"] }
---
# End-user auth at the edge: validate JWTs (e.g. from Entra ID / OIDC)
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata: { name: jwt-auth, namespace: istio-ingress }
spec:
  selector: { matchLabels: { istio: ingressgateway } }
  jwtRules:
  - issuer: "https://login.microsoftonline.com/TENANT/v2.0"
    jwksUri: "https://login.microsoftonline.com/TENANT/discovery/v2.0/keys"
    audiences: ["api://payment-api"]

The Nginx alternative — simpler edge, no mesh

75-nginx-ingress.yamlapiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payment-api-ingress
  namespace: payments
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "5"
    nginx.ingress.kubernetes.io/limit-rps: "100"       # per-client rate limit
    nginx.ingress.kubernetes.io/enable-cors: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod  # auto-TLS (next altitude)
spec:
  ingressClassName: nginx
  tls:
  - hosts: [pay.tmo.example]
    secretName: pay-tmo-example-tls
  rules:
  - host: pay.tmo.example
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service: { name: payment-api, port: { name: http } }

How it works underneath: the ingress-nginx controller pod watches Ingress objects and endpoints, generates an nginx.conf with an upstream block containing the pod IPs, and hot-applies it (backend changes via a Lua layer without full reloads). Choose Nginx ingress when you need a solid HTTP edge; choose Istio when you also need mesh features (mTLS everywhere, per-service traffic policy, canaries between internal services).

Certificates & CAs: how trust actually works

The mTLS handshake between two sidecars (watch it happen)

cert-manager — certificates as Kubernetes objects

80-cert-manager.yaml# The issuer: HOW to get certificates (Let's Encrypt via ACME here;
# swap for Vault/Venafi/private PKI in enterprises)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata: { name: letsencrypt-prod }
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform-team@tmo.example
    privateKeySecretRef: { name: le-account-key }
    solvers:
    - dns01:                       # DNS-01: prove ownership by writing a TXT
        route53: { region: us-west-2 }   # record; works for wildcards too
---
# The certificate: WHAT you want; controller keeps the Secret fresh
apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: pay-tmo-example, namespace: istio-ingress }
spec:
  secretName: pay-tmo-example-tls   # referenced by the Istio Gateway earlier
  issuerRef: { name: letsencrypt-prod, kind: ClusterIssuer }
  dnsNames: [pay.tmo.example]
  duration: 2160h                   # 90 days
  renewBefore: 720h                 # renew with 30 days to spare
  privateKey: { algorithm: ECDSA, size: 256 }  # smaller, faster handshakes

Telemetry: seeing 35,000 req/s without drowning

The four golden signals (Google SRE) — your dashboard skeleton

Wiring it: ServiceMonitor, alert rule, OTel collector

90-telemetry.yaml# Prometheus Operator: "scrape every service labeled app=payment-api"
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: payment-api, namespace: payments }
spec:
  selector: { matchLabels: { app: payment-api } }
  endpoints:
  - port: http
    path: /actuator/prometheus
    interval: 15s
---
# Alert on the symptom users feel — not on CPU
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: payment-api-alerts, namespace: payments }
spec:
  groups:
  - name: payment-api.slo
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_server_requests_seconds_count{app="payment-api",status=~"5.."}[5m]))
        / sum(rate(http_server_requests_seconds_count{app="payment-api"}[5m])) > 0.01
      for: 2m            # must persist 2 min — kills flappy pages
      labels: { severity: page }
      annotations:
        summary: "payment-api 5xx > 1% for 2m"
    - alert: P99LatencyBudgetBurn
      expr: |
        histogram_quantile(0.99, sum by (le)
          (rate(http_server_requests_seconds_bucket{app="payment-api"}[5m]))) > 0.25
      for: 5m
      labels: { severity: page }
---
# OpenTelemetry Collector: one pipeline for traces (as a ConfigMap excerpt)
apiVersion: v1
kind: ConfigMap
metadata: { name: otel-collector-conf, namespace: monitoring }
data:
  config.yaml: |
    receivers:
      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 } } }
    processors:
      batch: {}
      tail_sampling:            # the smart part: decide AFTER seeing the trace
        policies:
        - { name: errors,    type: status_code, status_code: { status_codes: [ERROR] } }
        - { name: slow,      type: latency, latency: { threshold_ms: 250 } }
        - { name: baseline,  type: probabilistic, probabilistic: { sampling_percentage: 1 } }
    exporters:
      otlp/tempo: { endpoint: tempo.monitoring:4317, tls: { insecure: true } }
    service:
      pipelines:
        traces: { receivers: [otlp], processors: [tail_sampling, batch], exporters: [otlp/tempo] }

Touchdown: the complete deployment, start to finish

Everything above, assembled. Here is the repo layout, the exact order to apply it, and how to verify each layer. This is the runbook you'd actually follow.

repo layoutpayment-api/
├── Dockerfile                  # multi-stage (Altitude 03)
├── .dockerignore
├── docker-compose.yml          # local dev
├── src/ · pom.xml
└── k8s/
    ├── 00-namespace.yaml       # Namespace + ResourceQuota
    ├── 10-configmap.yaml
    ├── 11-secret.yaml          # (in reality: ExternalSecret manifest)
    ├── 20-rbac.yaml            # SA + Role + RoleBinding
    ├── 30-deployment.yaml
    ├── 40-service.yaml
    ├── 50-hpa.yaml
    ├── 51-pdb.yaml
    ├── 60-networkpolicy.yaml
    ├── 70-istio-gateway.yaml
    ├── 71-virtualservice.yaml
    ├── 72-destinationrule.yaml
    ├── 73-security.yaml        # PeerAuthentication + AuthorizationPolicy + JWT
    ├── 80-cert-manager.yaml
    └── 90-telemetry.yaml       # ServiceMonitor + PrometheusRule + OTel

Build, ship, deploy — the exact commands

# 1 · build & push the image (CI does this on every merge)
docker build -t registry.tmo.example/payments/payment-api:1.4.2 .
docker push registry.tmo.example/payments/payment-api:1.4.2

# 2 · foundation: namespace, quota, identity, config
kubectl apply -f k8s/00-namespace.yaml
kubectl apply -f k8s/20-rbac.yaml
kubectl apply -f k8s/10-configmap.yaml -f k8s/11-secret.yaml

# 3 · workload + stable address
kubectl apply -f k8s/30-deployment.yaml -f k8s/40-service.yaml
kubectl -n payments rollout status deploy/payment-api     # waits for readiness

# 4 · resilience & scaling
kubectl apply -f k8s/50-hpa.yaml -f k8s/51-pdb.yaml -f k8s/60-networkpolicy.yaml

# 5 · edge & mesh
kubectl apply -f k8s/80-cert-manager.yaml
kubectl apply -f k8s/70-istio-gateway.yaml -f k8s/71-virtualservice.yaml \
              -f k8s/72-destinationrule.yaml -f k8s/73-security.yaml

# 6 · observability
kubectl apply -f k8s/90-telemetry.yaml

Verify each layer (the debugging ladder)

# Is the pod healthy? (events section = 90% of answers)
kubectl -n payments get pods -o wide
kubectl -n payments describe pod payment-api-xxxxx
kubectl -n payments logs deploy/payment-api --previous   # logs of the CRASHED container

# Is the Service actually selecting pods? (empty ENDPOINTS = label mismatch or readiness failing)
kubectl -n payments get endpointslices -l kubernetes.io/service-name=payment-api

# Can I reach it from inside the cluster? (tests Service + DNS + NetworkPolicy)
kubectl -n payments run tmp --rm -it --image=curlimages/curl -- \
  curl -s http://payment-api.payments.svc.cluster.local/actuator/health

# Is mTLS on? Is routing right? (istioctl reads the live Envoy config)
istioctl x describe pod payment-api-xxxxx -n payments
istioctl proxy-config routes deploy/istio-ingressgateway -n istio-ingress

# Is the cert real & fresh?
kubectl -n istio-ingress get certificate pay-tmo-example -o wide

# Is scaling behaving?
kubectl -n payments get hpa payment-api --watch

# Emergency exits
kubectl -n payments rollout undo deploy/payment-api      # instant rollback
kubectl -n payments rollout history deploy/payment-api

Best-practices checklist (print this)

Say it back three ways (the forgetting-proof recap)

If you can produce these three explanations from memory, the whole page is yours:

— END OF DESCENT · re-read one altitude per day; explain each diagram out loud before moving on —