Master distributed system resilience, the Four Golden Signals, SLOs & error budgets, distributed tracing, high-concurrency Kubernetes operations, chaos experiments, and blameless incident response.
Engineering Foundations
Coined by Google's VP of Engineering Ben Treynor Sloss: "SRE is what happens when you ask a software engineer to design an operations team."
DevOps is a holistic organizational culture bridging developers and operations. SRE is an opinionated, concrete implementation of DevOps using software algorithms, metrics, and automated tooling to eliminate operational toil.
Every mission-critical distributed service is evaluated on Latency (response time), Traffic (demand in QPS), Errors (rate of 5xx failures), and Saturation (system fraction used, e.g. memory/CPU limits).
100% uptime is virtually impossible and counter-productive. SREs establish Service Level Objectives (SLOs), providing product teams an acceptable 'error budget' to innovate quickly without violating customer trust.
When outages happen, human error is never the root cause; human error is a symptom of system vulnerability. SREs lead structured, blameless postmortems with verifiable action items to prevent recurrence.
Curriculum Roadmap
Expand each monthly phase below to inspect core technical topics, estimated time commitment, practical labs, and checkpoint drills.
/proc and /sys filesystemsstrace, ltrace, and introductory eBPF concepts40-50 hours
Write a lightweight Linux diagnostic daemon that parses /proc/meminfo and /proc/stat to log memory pressure spikes and high system-load events.
Brendan Gregg's Linux Performance, Linux Kernel Documentation, and Computer Systems: A Programmer's Perspective.
Diagnose a CPU-bound runaway process using perf top, verify kernel memory allocations with vmstat 1, and trace syscall latency using strace -c.
tcpdump, tshark, and Wiresharkss -s, netstat), MTU path discovery, iptables/nftables packet filtering45-55 hours
Simulate a DNS failover outage on a multi-tier web application and trace socket drops and TCP retransmissions using tcpdump.
High Performance Browser Networking by Ilya Grigorik, Cloudflare Learning Center, and RFC 793 (TCP).
Capture a complete TLS 1.3 handshake packet trace, explain SNI encryption, and resolve a simulated TIME_WAIT socket exhaustion issue.
os, sys, subprocess, requests, psutil, and signal handling50-60 hours
Build a production-grade URL health-checker CLI in Go that concurrently probes 50+ endpoints, calculates P95 latency percentiles, and emits Prometheus metrics.
Automate the Boring Stuff with Python, Go by Example, and Google's Site Reliability Engineering Workbook.
Write an automated script that securely handles API authentication, retries transient 503 errors using jittered backoff, and logs structured traces.
45-55 hours
Build an ultra-minimal Distroless Go microservice image (< 20MB), implement a graceful SIGTERM shutdown handler, and scan for CVEs in CI.
Liz Rice's "Containers from Scratch", Docker Documentation, and OCI Image Specs.
Inspect container namespaces manually using lsns and nsenter, and demonstrate container memory limits triggering OOM kills without affecting the host.
50-60 hours
Architect a multi-AZ VPC on AWS with private application subnets, an Application Load Balancer with TLS termination, and auto-scaling EC2 instances.
AWS Well-Architected Framework: Reliability Pillar, Google Cloud Architecture Center, Envoy Proxy Documentation.
Simulate the sudden loss of an entire Availability Zone and verify that the load balancer seamlessly re-routes traffic with zero dropped connections.
50-60 hours
Create a modular Terraform repository that provisions a hardened VPC, Bastion host, and encrypted storage with automated state locking and drift notifications.
Terraform Up & Running by Yevgeniy Brikman, HashiCorp Learn tutorials, Spacelift IaC Guides.
Simulate manual out-of-band cloud modifications and write an automated pipeline to detect and reconcile the infrastructure drift cleanly.
60-70 hours
Deploy a microservices cluster configured with PodDisruptionBudgets, anti-affinity across node zones, and automated HPA triggered by P99 request latency.
Kubernetes.io production guides, Mumshad Mannambeth CKA series, and "Kubernetes Up & Running".
Perform a non-disruptive rolling upgrade of a multi-replica service under heavy simulated load, verifying zero HTTP 5xx errors.
50-60 hours
Install Istio on a Kubernetes cluster, enforce strict mTLS between services, and configure a circuit breaker that trips when backend latency exceeds 500ms.
Istio Official Docs, Envoy Proxy Architecture Guide, CNCF Service Mesh whitepapers.
Configure a canary release that automatically routes 10% of user traffic to a new release and rolls back if error rate exceeds 1%.
rate(), irate(), histogram_quantile() for P50/P90/P99 latency percentiles55-65 hours
Deploy a complete LGTM (Loki, Grafana, Tempo, Mimir/Prometheus) observability stack collecting metrics, logs, and distributed traces from microservices.
Prometheus Up & Running by Brian Brazil, OpenTelemetry.io Documentation, Grafana Labs Tutorials.
Write PromQL alert expressions for the 4 Golden Signals (Latency, Traffic, Errors, Saturation) with proper hysteresis to eliminate alert flapping.
50-60 hours
Build an automated Slack bot that monitors Prometheus error budget burn rates and alerts the on-call engineer when 2% of the budget burns in 1 hour.
The Site Reliability Workbook (Chapters 2 & 3 on SLOs and Alerting), PagerDuty Incident Response Documentation.
Write a comprehensive blameless postmortem for a simulated production outage, identifying systemic fixes rather than human mistakes.
55-65 hours
Run an automated Chaos Mesh experiment against a staging Kubernetes cluster that randomly injects 200ms latency and kills pods during a 5,000 QPS load test.
Principles of Chaos Engineering (principlesofchaos.org), Chaos Mesh Documentation, k6.io Performance Guides.
Prove that your distributed microservice can withstand random pod terminations during peak load without violating its latency SLO.
60-70 hours
Deliver the complete SRE Production Capstone: a multi-region active-active distributed architecture with automated failover, Prometheus telemetry, SLO dashboards, and chaos testing suites.
Designing Data-Intensive Applications by Martin Kleppmann, System Design Interview by Alex Xu, Google SRE Interview Guides.
Successfully conduct a 45-minute simulated SRE interview walkthrough: diagnose an obscure live outage, explain TCP kernel tuning, and design a resilient global service.
Visual Tech Tree
A hierarchical visual map of foundational knowledge layers required for senior Site Reliability Engineering.
Understand kernel scheduling, memory management, syscall execution, page caches, cgroups, and filesystem drivers.
TCP/IP stack, connection backlogs, TIME_WAIT socket states, TLS handshakes, DNS caches, and low-level packet capture.
Write high-performance daemons, concurrent CLI probes, API adapters, and self-healing automation tools.
Master containerd, runc, Linux namespaces, distroless multi-stage compilation, and container security scanners.
Multi-AZ network topology, L4/L7 load balancing, Anycast DNS routing, Envoy proxy, and database failovers.
Immutable infrastructure, automated drift reconciliation, state locking, and secrets management with Vault.
etcd clustering, Pod Disruption Budgets, anti-affinity rules, HPA autoscaling on custom metrics, and node drain hygiene.
Zero-trust mTLS encryption, canary traffic shifting, circuit breaking, and network fault injection.
Prometheus time series, PromQL quantiles, Grafana dashboards, OpenTelemetry distributed tracing, and Loki logs.
Transform business expectations into mathematical SLOs, monitor multi-window burn rates, and eradicate alert fatigue.
Inject deliberate failures using Chaos Mesh, conduct high-throughput load tests with k6, and organize team game days.
Lead live emergency incidents, coordinate cross-team communications, and write blameless post-incident reviews.
Curated Training
Select any domain tab below to access vetted, 100% free official documentation, books, and interactive sandboxes.
Platform: Google SRE (sre.google/books)
Duration: Self-paced (3 complete books)
Difficulty: Intermediate to Advanced
The foundational canon of modern reliability engineering. Includes 'Site Reliability Engineering', 'The Site Reliability Workbook', and 'Building Secure and Reliable Systems'.
Platform: BrendanGregg.com & Netflix Tech Blog
Duration: Self-paced
Difficulty: Advanced
Authoritative performance methodology: USE method (Utilization, Saturation, Errors), flame graphs, CPU/memory profiling, and eBPF tracing tools.
Platform: O'Reilly / hpbn.co
Duration: Self-paced
Difficulty: Intermediate to Advanced
Master TCP handshake dynamics, congestion control algorithms, TLS 1.3 performance optimization, HTTP/2 multiplexing, and HTTP/3 QUIC protocol details.
Platform: Go.dev
Duration: 2-3 weeks
Difficulty: Beginner to Intermediate
Hands-on tutorials covering the language powering Kubernetes, Docker, Terraform, and Prometheus: goroutines, channels, interfaces, and networking packages.
Platform: Kubernetes.io Documentation
Duration: Self-paced
Difficulty: Intermediate
Learn how to tune Kubernetes clusters for production: etcd disaster recovery, PodDisruptionBudgets, cluster autoscaling, and zero-downtime upgrades.
Platform: Prometheus.io & OpenTelemetry.io
Duration: Self-paced
Difficulty: Intermediate
Comprehensive guides on metrics, Prometheus Query Language (PromQL), Alertmanager, distributed tracing with OpenTelemetry, and Grafana panels.
Platform: principlesofchaos.org & Chaos Mesh
Duration: Self-paced
Difficulty: Intermediate to Advanced
Understand how to design controlled chaos experiments, define hypotheses around steady state, minimize blast radius, and automate recurring failure injections.
Platform: PagerDuty (response.pagerduty.com)
Duration: Self-paced
Difficulty: Beginner to Intermediate
The industry-standard incident response handbook. Teaches the Incident Commander role, stakeholder updates, severity levels, and how to conduct blameless postmortems.
Video Lectures
Top-rated video courses from senior infrastructure engineers and Google SRE practitioners.
Channel: Google Cloud Tech
Duration: Full Series
Skill Level: Beginner to Intermediate
Google engineers explain how SRE works in practice: SLOs, error budgets, toil reduction, and incident response.
Watch Google SRE Series →Speaker: Brendan Gregg
Duration: 1.5 Hours
Skill Level: Advanced
Deep-dive conference talk into Linux kernel performance analysis, CPU profiling, and using BPF tools.
Watch Masterclass →Channel: TechWorld with Nana
Duration: 2 Hours
Skill Level: Beginner to Intermediate
Clear walk-through of Prometheus architecture, metrics types (Counter, Gauge, Histogram), and writing PromQL queries.
Watch Prometheus Tutorial →Channel: CNCF [Cloud Native Computing Foundation]
Duration: Curated Talks
Skill Level: Intermediate
Learn how to instrument microservices with OpenTelemetry to track requests across distributed backend topologies.
Visit CNCF Channel →Channel: freeCodeCamp
Duration: 3.5 Hours
Skill Level: Beginner to Intermediate
Master Go syntax, goroutine concurrency patterns, and standard library HTTP packages for building SRE tools.
Watch Full Go Course →Channel: Chaos Mesh Community
Duration: 1 Hour
Skill Level: Intermediate to Advanced
Step-by-step hands-on guide injecting network partitions, pod failures, and clock skews into Kubernetes clusters.
Watch Chaos Tutorial →Portfolio Blueprints
Practical systems you can build, stress-test, and present to hiring managers during technical interviews.
Estimated Time: 6-8 hours
Skills: Linux /proc, Python/Go, Systemd, Signals
Develop a background service that continuously monitors CPU load averages, disk inode saturation, and memory pressure. Automatically alerts when memory exhaustion is imminent.
Outcome: A reliable systems probe running as a native systemd unit.
Linux • Python • Systemd • Bash
Estimated Time: 6-8 hours
Skills: Nginx/Envoy, Active Health Checks, TLS
Configure an Envoy or Nginx reverse proxy routing traffic across multiple backend web containers with passive health checks, graceful reload on configuration updates, and HTTP/2 support.
Outcome: Seamless traffic routing that automatically sheds unhealthy backends.
Envoy • Nginx • Docker • TLS
Estimated Time: 5-7 hours
Skills: PostgreSQL Dump, AWS S3 Glacier, Cron, Integrity Verification
Write an automated script that takes encrypted database snapshots, uploads them to off-site object storage, verifies cryptographic checksums, and tests restoration in an ephemeral container.
Outcome: A verified, testable disaster recovery pipeline with automated validation.
PostgreSQL • AWS S3 • Bash • GPG
Estimated Time: 10-15 hours
Skills: Prometheus, PromQL, Grafana, OpenTelemetry, Jaeger
Instrument a multi-service web application to export the 4 Golden Signals. Build a unified Grafana dashboard tracking P50/P90/P99 latency percentiles alongside distributed traces.
Outcome: End-to-end trace correlation allowing sub-second root-cause diagnosis.
OpenTelemetry • Prometheus • Grafana • Jaeger
Estimated Time: 12-16 hours
Skills: Kubernetes, PDB, TopologySpreadConstraints, HPA
Deploy a microservices cluster across 3 simulated failure zones. Configure PodDisruptionBudgets, anti-affinity rules, and an HPA scaled on custom Prometheus P95 latency metrics.
Outcome: A cluster that survives involuntary node terminations with zero downtime.
Kubernetes • Helm • Prometheus • HPA
Estimated Time: 10-14 hours
Skills: Google SRE Workbook SLO Algorithms, Alertmanager, Webhooks
Implement the Google SRE multi-window multi-burn-rate alerting algorithm. Configure alerts that distinguish between rapid critical budget depletion (page on-call) and slow erosion (ticket).
Outcome: Zero-noise alerting system that saves engineering teams from alert fatigue.
Prometheus • Alertmanager • Slack API • Python
Estimated Time: 15-20 hours
Skills: Chaos Mesh, k6 Load Testing, Automated Recovery, CI/CD
Construct an automated resilience validation pipeline that generates 10,000 QPS load with k6 while Chaos Mesh injects network latency and pod crashes. Verifies that the latency SLO never breaches.
Outcome: Continuous resilience testing validating system availability before release.
Chaos Mesh • k6 • Kubernetes • GitHub Actions
Estimated Time: 14-18 hours
Skills: Kubernetes Operator / Go, Webhooks, PagerDuty, Runbooks
Build an automated remediation bot that listens for specific Prometheus alert webhooks (e.g. disk space 95% full or deadlocked JVM thread) and executes audited, safe diagnostic runbooks automatically.
Outcome: Cuts Mean Time to Recovery (MTTR) by 80% through automated triage.
Go • Kubernetes API • PagerDuty • Docker
Estimated Time: 20-25 hours
Skills: Terraform, Route53 Anycast, Multi-Region Aurora/CockroachDB, Global Envoy
Architect and provision an active-active deployment across two distinct cloud regions. Implement global latency routing with automated DNS health checks and distributed database replication.
Outcome: Enterprise-grade fault-tolerant cloud platform resilient to regional failure.
Terraform • AWS / GCP • Aurora • Route 53
Industry Credentials
Validated industry benchmarks that demonstrate deep systems knowledge, cloud fluency, and cluster operational skills.
Difficulty: Advanced
Prerequisites: GCP Basics & SRE Principles
Preparation Time: 6-8 weeks
Recommended After: Month 10
Official Certification →Difficulty: Advanced (100% Hands-on CLI)
Prerequisites: Linux & Kubernetes
Preparation Time: 8-10 weeks
Recommended After: Month 7
Official Certification →Difficulty: Intermediate (Performance-based)
Prerequisites: Linux Administration
Preparation Time: 4-6 weeks
Recommended After: Month 3
Official Certification →Difficulty: Intermediate
Prerequisites: AWS Fundamentals
Preparation Time: 6-8 weeks
Recommended After: Month 5
Official Certification →Difficulty: Intermediate
Prerequisites: IaC Concepts
Preparation Time: 3-5 weeks
Recommended After: Month 6
Official Certification →Technical Drills
Real-world troubleshooting workflows, architectural questions, and on-call interview scenarios.
A 502 Bad Gateway means the reverse proxy (e.g. Nginx, ALB, or Envoy) received an invalid response or connection refusal from the upstream application backend. Step 1: Check the proxy access and error logs to identify the exact upstream IP and port. Step 2: Test network reachability and port listening from the proxy to the backend using curl -Iv, nc -zv, or telnet. Step 3: Log onto the backend host/pod and inspect process status (ps aux or kubectl get pods) to see if the application crashed or was killed by the OOM Killer (dmesg -T | grep -i oom). Step 4: Check system resource saturation (top, free -m, iostat -xz 1). Step 5: If the process is alive but non-responsive, inspect socket queues with ss -lntp to identify if the TCP listen backlog is saturated.
SLI (Service Level Indicator): A carefully defined metric representing service performance (e.g., "percentage of HTTP requests returning < 200ms latency over the last 30 days").
SLO (Service Level Objective): A target goal agreed upon by engineering and product teams (e.g., "99.9% of requests must meet the SLI").
SLA (Service Level Agreement): The legal contract with external customers that specifies financial or contractual consequences if the SLO is breached (e.g., 10% credit refund if uptime drops below 99.5%).
An Error Budget is the mathematical inverse of the SLO. If an SLO is 99.9%, the error budget is 0.1% of allowable failures over a rolling window (e.g. 30 days). When a service operates safely within its error budget, product teams have the green light to push new features and experiment rapidly. However, if an outage consumes the error budget, feature releases are automatically paused or gated, and engineering effort is redirected entirely toward reliability hardening and technical debt reduction.
When physical RAM and swap are exhausted, the kernel invokes the OOM Killer. It assigns an oom_score to every process based on memory consumption, CPU utilization, and oom_score_adj, terminating the process with the highest score to prevent kernel panic. In Kubernetes, pods have memory requests and limits. If a pod exceeds its specified memory limit, the container runtime cgroup sends a SIGKILL (exit code 137). SREs prevent this by setting realistic memory requests based on historical P99 profiling, tuning JVM/Go heap allocations below container limits, and setting up Horizontal/Vertical Pod Autoscalers.
Latency measures the duration it takes to service a request. It is crucial to measure successful requests separately from failed requests (failed requests often fail very quickly, which could artificially deflate average latency).
Saturation measures how full a resource is. Most systems degrade non-linearly before reaching 100% capacity (e.g., a disk or CPU queue might introduce severe latency spikes when exceeding 80% saturation). Tracking saturation provides early warnings before latency or errors begin to climb.
Career Trajectory
Seniority milestones, technical scope, and industry compensation expectations across the reliability engineering ladder.
Experience: 0-2 Years
Salary Range: ₹6 - ₹10 LPA
Core Focus: Linux CLI navigation, writing maintenance automation scripts in Python/Go, responding to tier-1 monitoring alerts, and shadowing on-call engineers.
Experience: 2-5 Years
Salary Range: ₹10 - ₹18 LPA
Core Focus: Primary on-call rotation, managing Kubernetes clusters, configuring Prometheus/Grafana telemetry, writing Terraform modules, and conducting blameless postmortems.
Experience: 5-8 Years
Salary Range: ₹18 - ₹35+ LPA
Core Focus: Architecting multi-region failover topologies, designing mathematical SLOs & error budget policies, leading disaster recovery drills, and mentoring teams.
Experience: 8-12 Years
Salary Range: ₹35 - ₹65+ LPA
Core Focus: Enterprise-wide reliability architecture, building internal developer platforms (IDPs), defining zero-trust service mesh policies, and executive technical strategy.
Experience: 12+ Years
Salary Range: ₹65 - ₹1.2+ Cr
Core Focus: Organization-wide resilience standards, cloud cost efficiency, multi-cloud strategy, compliance governance, and engineering executive leadership.
Frequently Asked Questions
Common questions from aspiring Site Reliability Engineers transitioning from software engineering or system administration.
Traditional SysAdmins historically performed repetitive manual interventions (configuring servers by hand, restarting processes, running manual backups). SREs treat operations as a software problem: any repetitive manual task (known as "toil") is capped at 50% of an SRE's time, with the remaining 50%+ spent writing software, creating automated self-healing tools, and engineering distributed systems.
Both have vital roles. Python is ideal for scripting, rapid prototyping, and data analysis. Go (Golang) is the lingua franca of cloud infrastructure (Kubernetes, Docker, Terraform, Prometheus, and Envoy are all written in Go). We recommend starting with Python for systems scripting (Months 1–3) and progressing to Go for concurrent infrastructure tooling (Month 3 onward).
On-call involves carrying an escalation device (phone/pager) during scheduled shifts to respond to critical production incidents. In a mature SRE organization, only critical user-impacting SLO breaches page the on-call engineer. High-noise warnings are handled via automated tickets during work hours. Furthermore, SRE culture mandates that every page must have an actionable runbook and be followed by a blameless postmortem to permanently fix the root cause.
Yes, in fact, this is the most common path! Software developers bring strong coding and architecture skills (needing to build Linux kernel and networking fluency), while DevOps engineers bring CI/CD and cloud experience (needing to deepen their coding, algorithm, and SLO/telemetry practices).