The Operate blog
Notes from the frontline of AI ops
Patterns, incidents, and lessons from running AI SRE in production — for teams that own their infrastructure.
We write about root cause analysis, on-call and incident response, postmortems of real outages, observability, and what it takes to let AI agents investigate production safely — with read-only access, verified evidence, and a human approving every change.
How to Check HAR Files Before Sending a Customer Support Escalation
We review HAR capture scope, changed conditions and evidence limits, then prepare a useful engineering handoff with a separate sharing decision.
What Does AI SRE Data Retention Cover After Deleting an Investigation?
Evaluate AI SRE data retention with a worksheet for threads, derived memory, exports and backups. Verify deletion scope without mistaking silence for erasure.
Did Your AI SRE Query a Caught-Up Replica Before Finding No Order?
Test if an AI SRE mistakes an empty replica read for a failed write. Use seven evidence cards covering replay, snapshots, query errors and uncertainty.
Whose Permissions Does an AI SRE Use When Support Asks in Slack?
Check whose permissions an AI SRE uses from Slack with a practical worksheet for requester access, source scope, saved results and output audiences.
Can a Log Line Change Your AI SRE’s Diagnosis?
Test whether customer-controlled log text can steer an AI SRE’s diagnosis with six synthetic cases, evidence checks, and a buyer review worksheet.
How Much Does an AI SRE Cost When One Incident Triggers Several Investigations?
Compare AI SRE pricing per incident, including repeat runs, credit bundles, query fees and human review. Use a ledger to test your own pilot costs.
Managed SRE Services Should Answer Who Owns Production Work
Compare managed SRE services by the work they own, coverage, permissions and handoffs. Use a responsibility matrix before choosing a provider.
Can Your AI SRE Explain Yesterday’s Feature-Flag Decision?
Evaluate AI SRE tools with five feature-flag evidence tests. Check historical accuracy, honest uncertainty and review effort before a production pilot.
How Much Do Database Monitoring Tools Cost for a Team With 12 Database Hosts
We priced database monitoring tools for 12 hosts from six vendors' own pages, then listed the costs the per-host price leaves out.
What Is the Impact of Flaky Tests on Your Team (and How to Measure It)
We break the impact of flaky tests into four cost lines, explain why published numbers disagree, and show how to measure your own team's cost in two weeks.
How to Run Root Cause Analysis in Software Without Stopping at the First Why
We walk through root cause analysis in software with six evidence steps, method choice, and a fault tree of Cloudflare's 2025 outage.
What Is an AI SRE (and Which Parts of the SRE Job It Actually Does)
We define an AI SRE against the real SRE job, grade vendor definitions on Google's five autonomy levels, and list what to settle before one reads production.
Self-Hosted vs Cloud AI Agents, Which One Passes a Production Security Review?
Compare self-hosted and cloud AI agents that read production, with five deployment shapes, eight security review questions, and where LLM prompts go.
What Is a Good MTTR for a Software Team (and How to Set Your Own Target)
What a good MTTR looks like for software teams, using DORA's recovery bands and our target from the error budget and incident rate.
How to Retry Flaky Tests Without Letting a Real Regression Through
We show what retries do to flaky tests and real regressions, how six test runners report a pass on retry, and which failures should fail the build.
Which Postgres Monitoring Tools Keep the Evidence You Need After an Incident?
We compare six Postgres monitoring tools by the evidence they keep after an incident: blocking chains, query history, wait events and captured plans.
I-PASS On-Call Handoff Lessons From Hospital Shift Changes
A structured handoff cut hospital errors 23 percent with no added time. What I-PASS teaches on-call teams about handoffs, with a template.
Salesforce Outage Began With a Login Service That Stalled
Salesforce's September 2026 outage began with a login service that stalled rather than failed. Why slow dependencies do more damage and what to test.
Kafka Consumer Lag Shapes That Tell You Not to Scale Out
Kafka consumer lag rising while consumers idle at low CPU? Read it per partition. Five lag shapes, the check that confirms each, and when scaling hurts.
Read-Only Production Access Is a Capacity Change Too
Read-only production access stops writes, not load. Locks, temp files, open snapshots and replica conflicts still hit production, and most guardrails ship off.
Postgres ALTER TABLE Lock Queues Can Block Every Read
A metadata-only Postgres ALTER TABLE still waits for an exclusive lock, and every read queues behind it. How the lock queue works and how to bound it.
Terraform force-unlock Is the Easy Part: Proving the Lock Is Orphaned, and Finding What the Dead Apply Already Built
terraform force-unlock takes ten seconds. Proving the lock is orphaned, not live, takes six metadata fields and one audit log query. Plus finding what the dead apply already built.
A Pull Request Review Gate Sits After the Write
A pull request review gate guards the merge, but opening the pull request already needed write access. Five gaps to audit and what a patch file changes.
Ariane 501 Shows Redundancy Without Diversity Fails Together
Both Ariane 501 inertial units failed 0.05 seconds apart because they were identical. What the Lions report says about redundancy, dead code and diversity.
EMFILE vs ENFILE vs Inotify for Too Many Open Files
Three unrelated Linux limits print too many open files. How EMFILE, ENFILE and the inotify limits differ, and where to read each one in a minute.
Exactly Once Delivery Is a Property of the Write
Exactly once delivery stops at the broker. Most duplicates come from a commit whose acknowledgement was lost, so the guarantee belongs in the write itself.
Incident Case Definition Lessons From Disease Outbreak Work
Outbreak investigators define who counts as a case before hunting for the cause. Incident channels skip that step, and the blast radius pays for it.
Partition Pruning Fails Silently, So Prove It in the Plan
A query can return correct rows and still read every partition. How to prove partition pruning in Postgres, MySQL, BigQuery and Snowflake, and what to fix.
The Amazon AI Outage Began With a Stale Wiki, Not Bad Code
Amazon traced its one AI-related incident to advice inferred from an outdated wiki. Why diff review cannot catch a stale premise and what control can.
TimescaleDB Compression Made Empty Results the Slowest Query
At VesselAPI, not-found requests used 71% of database time because TimescaleDB compression removed the index a miss needed. How it happened and what to check.
TLS Handshake Timeout Is Usually Not a TLS Problem
A TLS handshake timeout means the client heard silence after TCP connected, not a certificate fault. Seven ordered checks prove where the wait began.
Data Warehouse Cost Optimization Is a Routing Problem
Data warehouse cost optimization fails when the bill reaches finance and the query plan stays with engineers. How to route cost drift to people who can fix it.
Silent Failure Detection Needs an Expectation Held Outside
A silent failure is an absence, and the missing component cannot log its own absence. Four shapes of external expectation, and what each one catches.
How to Fix ImagePullBackOff by Reading the Registry Error
ImagePullBackOff is a retry timer, not a cause. Read the registry error in the pod events, match it to the failing layer, and force a clean retry safely.
Interviewing for Diagnosis Versus Construction in SRE Hiring
Hiring loops measure construction while capacity goes to diagnosis. What Google's SRE book, selection research and debugging interviews actually support.
SIGKILL After Thirty Seconds Erases Scheduled Work in Memory
Kubernetes sends SIGKILL when the 30-second grace period ends, and work scheduled in process memory dies with no error. Where it hides and how to fix it.
504 Gateway Timeout Means a Timer Expired, Not a Slow Hop
A 504 gateway timeout names the hop whose timer expired, not the slow one. Build the timeout ladder for Nginx, your load balancer and database to find it.
Maintenance Window Risk and the Nowlan and Heap Patterns
Nowlan and Heap found 89 percent of items had no wear-out zone and 68 percent failed early. What that means for scheduling a maintenance window.
Agent Observability Reaches 94% but Online Evals Only 45%
LangChain's survey of 1,340 practitioners shows production agents are widely traced and less often scored online. What that gap hides and what to add first.
Retry Storm Diagnosis Starts With the Amplification Factor
A retry storm usually amplifies an outage rather than causing it. Measure the amplification factor, find the layer retrying, and tell it from a traffic spike.
Escalation Metrics Should Ask If the Escalation Was Needed
Escalation metrics track the handoff, never the need. Answerability rate fills the gap with a small two-sided sample review and three buckets you can act on.
Booking.com's Capacity Test Cannot Be Left Switched On: Every Command It Sends Expires By Itself
Booking.com's Kafka capacity tests expire on their own rather than being cleaned up afterwards. That design rule applies to every automated change you make to production.
Postgres Autovacuum Diagnosis That Starts From the Symptom
Start a Postgres autovacuum investigation at the alert. Four symptoms, five look-alikes to rule out, five vacuum failures to confirm and the wraparound runbook.
Blameless Postmortems Record the Answer and Lose the Search
A blameless postmortem usually keeps the timeline and the cause, and drops the search that took most of the time. What STELLA found and four fields to add.
No Space Left on Device Errors That Are Not a Full Disk
No space left on device does not always mean a full disk. An ordered ENOSPC procedure covering inodes, open deleted files, quotas, containers and inotify.
A Service Ownership Model Fails When It Is Confidently Wrong
Stale ownership data misroutes incidents and can trigger destructive automation. What GitHub learned, the low water mark it added and what to measure.
Vulnerability Prioritization Needs VEX to Stop Repeat Triage
Vulnerability prioritization fails when the scanner reopens a finding you already disproved. The five VEX justifications and how to record the answer once.
Why Exit Zero Is Not Evidence a Batch Job Worked
Clinical labs never accept a run because the instrument reported no error. They put a known answer through it. What control samples teach batch job monitoring.
Cloud Cost Optimization Spends Reliability No One Owns
Cloud cost optimization books a saving with an owner and spends reliability nobody tracks. Five things to attach to every cost change, and when to skip them.
Incident History Retrieval Was Not the Three Mile Island Gap
A near-identical sequence happened at Davis-Besse eighteen months before Three Mile Island. What it says about searching incident history versus acting on it.
Liveness Probe vs Readiness Probe and What a Restart Deletes
A readiness probe stops traffic and a liveness probe restarts the container, deleting the evidence. When each fits, and four questions before adding liveness.
Flink Autoscaler Migration at Netflix and the Wrong Unit
Netflix is moving more than 30,000 Flink jobs to the open-source autoscaler. The old one worked, and the new one changes the unit of the decision.
Reliability Reporting Cannot Detect a Ten Percent Gain
A Google simulation on real incident data shows a genuine 10 percent MTTR gain goes unseen up to 40 percent of the time, and noise can look like a win.
Exit Code 137 Means SIGKILL, Not Always Out of Memory
Exit code 137 means a SIGKILL arrived, not always that memory ran out. The six senders, the evidence each leaves, and how to prove which one did it.
Canary Deployment Gates That Catch Wrong Answers, Not Errors
Canary deployment analysis gates on errors and latency, so a release that returns wrong but valid answers passes. Five correctness gates that catch it.
Your LLM Fallback Was Not Independent on September 3
ChatGPT, Claude and Grok failed the same morning with three different stated causes. Why an LLM fallback assumes hidden independence and what to measure.
Incident Severity Levels When the Impact Is Still Unknown
Incident severity levels get declared before impact is known. A procedure to grade from evidence, re-grade on set triggers and audit severity calls later.
The GitHub Outage in August Was Capacity, Not a Change
GitHub says neither August 2026 outage came from a code or config change. What the root cause shows and how to add a saturation branch to your triage.
Alert Actionability Lessons From an Emergency Room Study
An emergency department study found 0.8 percent of monitor alarms changed clinical management. How to measure alert actionability on your own on-call.
Connection Reset by Peer Means a Component Chose to Reset
Connection reset by peer means a TCP RST arrived. Learn the six components that send one, their timing signatures, and how to prove which one did.
Certificate Expiration Monitoring After the New 200-Day Rule
TLS certificates are now capped at 200 days, with 100 days due in March 2027. Where certificate expiration monitoring misses certificates and what to fix.
Self-Healing Infrastructure Lessons From the 1990 AT&T Crash
AT&T's 1990 collapse began in recovery code. The self-healing machinery spread the fault across 114 identical switches. What it means for automated remediation.
Context Deadline Exceeded Names the Caller, Not the Slow Hop
Context deadline exceeded is logged by the caller that gave up, not the slow hop. How the deadline budget shrinks and how to find the hop that spent it.
Root Cause Analysis at Meta Scale and What DrP Really Costs
Meta's DrP runs 50,000 automated analyses a day for 300+ teams. The overlooked figure is 2,000 hand-written analyzers, and what that costs to copy.
Your Triage Agent Solved Everything Nobody Needed Help With
A triage agent scored on total cases handled looks great on easy alerts. Score it on the residual, the hard incidents, where evidence and verification decide.
CrashLoopBackOff Triage in the Order That Narrows the Cause
CrashLoopBackOff says a container keeps restarting, not why. Triage it in order, by blast radius, exit code and crash versus kill, before reading the logs.
On-Call Onboarding Should Report Time to First Page
Time to first page turns on-call onboarding into a number. How to define it, the two failure modes it exposes, five gates from Google and what to report.
Postgres Query Plan Changed With No Deploy and p99 Tripled
A Postgres query plan can change under an identical query. Stale statistics, the generic plan switch and lost partition pruning, each with a check and a fix.
Runbook Automation Speeds Up Execution but Not Diagnosis
Runbook automation compresses the execution half of an incident. Measure match rate, verified recency and diagnosis share first to see which half is slow.
Log Retention Policy Lessons From the FAA 25-Hour Rule
The FAA moved cockpit recorders from 2 to 25 hours because evidence was overwritten before anyone knew. Your log retention policy may have the same flaw.
The AI Productivity Paradox Shows Up as Incidents per PR
Faros telemetry shows PRs per developer up 16% while incidents per PR rose 243%. What the AI productivity paradox costs on-call teams and what to measure.
Too Many Open Files Is Rarely Fixed by Raising the Limit
Too many open files usually means a leak, a sizing gap or a slow dependency, not a low limit. How to tell which in five commands before you raise anything.
Postgres Index Bloat Grows From Indexes Nobody Reviews
Postgres index bloat is often a set of indexes that only grows. How to measure it, read scan counters safely and drop indexes on a live table.
CI/CD Pipeline Failure That Stays Green and Deploys Nothing
The worst CI/CD pipeline failure is a green run that did nothing. Eight steps that succeed by doing nothing, the assertion for each and a six-question audit.
Configuration Drift Causes Incidents Weeks After the Change
Configuration drift stays harmless until a deploy or restart reads it. Where drift hides, how to sort it, and a five-step triage to run during an incident.
Cascading Failure Through the Lens of Normal Accidents
Perrow's Normal Accidents theory explains the October 2025 AWS cascade through tight coupling, interactive complexity and redundant automation.
DevOps for Startups When One Person Carries the Pager
DevOps for startups with one person on call. The five signals to collect, a fixed triage order, a one-page runbook and where AI should stay read-only.
Transactional Outbox Pattern Problems That Start on Day Two
The transactional outbox pattern fixes dual writes and adds relay stalls, lag, churn, poison messages and skipped rows. How to detect each one in production.
Incident Recurrence Rate Tells You What Closure Rate Hides
Closure rate rewards silencing alerts. Incident recurrence rate shows whether fixes held. How to build it from failure signatures and what to report beside it.
AI Agent Guardrails for Production Come Down to Five Rules
AI agent guardrails for production operations come down to five rules. Read-only access, evidence, an independent check, a human gate and an audit trail.
What p99 Latency Shows That Your p50 and Averages Hide
p99 latency is the response time 99% of requests beat. Why tails regress without a deploy, who hits them, and a four-step workflow to find the cause.
The Incident Commander Doctrine Software Only Half Borrowed
Software kept the incident commander title from fire command but dropped most of the doctrine. Span of control, command transfer and what does not carry over.
Build vs Buy AI for Operations After the DIY Copilot Wave
Teams are building their own ops copilots on self-hosted models. What they get right, where internal copilots stall, and a build vs buy AI framework to use.
A Dead Letter Queue Needs Owners, Alerts and a Redrive Rule
A dead letter queue only prevents data loss if someone watches it. The metrics, alert thresholds, failure triage and safe redrive steps that keep it honest.
Code Yellow in Software Engineering Is a Lagging Indicator
A code yellow pauses roadmap work to fix a serious reliability problem. Where the term came from, how to run one and the signals that appear months earlier.
ReadWriteOnce vs ReadWriteMany Is About Nodes, Not Pods
ReadWriteOnce vs ReadWriteMany is a node rule, not a pod rule. Why the misconfig passes every test, fails on a node drain, and how to audit and fix it.
How to Catch the N+1 Query Problem in Production
The N+1 query problem passes review and tests, then grows with your data. How to catch it in production telemetry, fix it safely and keep it from returning.
Silent Failures in Software and the 2003 Blackout Alarm
In 2003 a control room alarm system stalled without a trace and its backup inherited the stall. The same failure mode lives in cron jobs, queues and alerts.
Unplanned Work Is the Capacity Your Roadmap Never Counts
Unplanned work takes a measurable share of every engineering week. What the data really shows, capacity math for a 20-engineer team and a two-week measure.
AWS Outage Postmortem Lessons Your Provider Will Not Write
AWS's latest public postmortem covers October 2025. How to write the customer-side AWS outage postmortem your provider never will, step by step.
A Missing Timeout Let a Slow Service Stall CircleCI
CircleCI's 2 July outage began when API threads waited on a slow data service with no timeout. The timeline, thread starvation and five changes that bound it.
Measure Time to Diagnose as Its Own Incident Clock
Time to diagnose hides inside MTTR. Four timestamps to record per incident, what counts as a cause identified, and how to read the intervals honestly.
How to Reduce On-Call Alert Fatigue by Triaging First
On-call alert fatigue comes from pages that need no action and carry no evidence. Five steps to cut it without hiding real incidents, and how to measure it.
Agentic Database Changes Need More Than Replay Checks
Replaying traffic on a database fork proves answers match, not that a change is safe in production. Why agentic database changes should be reviewable diffs.
Knight Capital's 97 Unread Emails and AI Agent Controls
Knight Capital lost about $460 million in 45 minutes. The SEC order reads like a control checklist for any human or AI agent with production write access.
Idle in Transaction Postgres Sessions That Can Block Vacuum
Idle in transaction Postgres sessions hold locks and, if they wrote data, block vacuum while dashboards stay green. Find them, end them, and set the timeout.
A Flaky Tests Runbook That Treats Each Flake as an Incident
Treat flaky tests like small incidents. Detect them from CI data, quarantine with exit dates, classify the cause, and verify the fix with enough green runs.
Second Opinions in Root Cause Analysis and the 21% Problem
A Mayo Clinic study found 21% of referred diagnoses were distinctly different on review. Root cause analysis fails the same way, and a second read catches it.
Dashboards Green but System Down Comes From Edge Failures
When every health check passes and users still fail, the fault is usually between components. Four edge classes with a documented cause, and a path test.
Stuck State Outlives the Outage at Railway and CircleCI
Railway and CircleCI both published July 2 postmortems where state captured during instability outlived the fix. Four stuck states and how to expire them.
Stale Connections After Network Recovery Kept Railway Down
Railway's 2 July outage outlived its network fault because storage connections stayed pinned and mesh tunnels held a wrong address. What to audit in your stack.
Monzo's On-Call Rotation Models Broke Twice as It Grew
Monzo published three on-call rotation models from 2018 to 2022. We read what broke at about 50 and 150+ engineers, and what each change asked of every team.
AI Root Cause Analysis Without Write Access to Production
AI root cause analysis can read logs, databases and code without write access. Where to draw the line, how to check answers, and what to ask a vendor first.
Automated Root Cause Analysis Tools Judged by Ten Questions
Automated root cause analysis tools all promise speed. Ten questions that test scope, evidence, calibration and write access, plus how to run a fair trial.
Run a Root Cause Analysis Pilot in Shadow Mode First
Back-test automated root cause analysis on closed incidents, then run it in shadow mode and score every finding. A pilot protocol that stays read-only.
Verifying AI Root Cause Analysis the Way Pilots Cross-Check
Verifying AI root cause analysis needs a checker that did not produce the answer. Pilot monitoring shows how to build one that reads raw evidence and says no.
Connection Pool Exhaustion at WorkOS Was a Hold-and-Wait Bug
WorkOS added capacity and nothing improved. How a nested transaction caused connection pool exhaustion, the three signals that identify it and the fixes.
When to Hire an SRE and What Must Exist Before You Do
When to hire an SRE depends on signals you can check and four prerequisites the first hire needs. What one SRE cannot fix and the cheaper steps to try first.
SRE Toil Has Risen Two Years Running While AI Spread
Catchpoint's SRE Report medians for toil went from 25% to 30% to 34% as AI spread. A survey shows association, so we test three hypotheses for why.
Zero Downtime Database Migration Steps the Guides Leave Out
A zero downtime database migration fails outside the pattern. Six expand and contract steps for PostgreSQL, four skipped failure modes and the safety checks.
AI Audit Trail for Infrastructure Changes Starts in the Repo
AI-assisted Terraform and pipeline changes ship fast, but the reasons die with the chat session. Five fields that keep an AI audit trail in your repository.
Which Flaky Test Causes Point at Bugs in the System
Flaky test causes split into test bugs and system bugs. What the research says about each cause, and five checks to sort a flake before you retry it.
Kubernetes Requests and Limits Can Be Cut Without the Blame
Padded Kubernetes requests and limits survive because cutting them is a personal risk. What a request controls, four real risks and a blame-safe rollout.
Delhi Data Center Fire Blast Radius Audit for Your Stack
A June 2026 fire at a Delhi data center disrupted Google Cloud for weeks and threatened decades of customer data. Four audit questions for your blast radius.
AI Agent Monitoring Misses Failures That Return 200 OK
AI agent monitoring that trusts 200 OK misses agents that do nothing. Five silent failure shapes, why dashboards stay green, and what to record instead.
Schema Drift Turns the Next Deploy Into an Incident
Schema drift is the gap between your migrations and the live database. How it gets in, what Snowflake's December 2025 outage shows, and how to triage it.
MTTR Median vs Mean and Why the Average Misleads
1.8 million real outages show a 1.9-minute median and a long tail. Why a single average MTTR misleads, and the percentile view to report instead.
Observability Cost Is a Hedge Against Bad Investigations
Observability cost is mostly a hedge, paying to store answers you rarely read. Split telemetry into detection, transient state and data you can query later.
How to Reduce MTTR by Shortening Diagnosis, Not Detection
How to reduce MTTR once alerting is already fast. Time each incident phase, then shrink diagnosis with rollbacks, one evidence timeline and a second check.
Is AWS Down or Is It Us, Answered in Fifteen Minutes
Is AWS down, or did you break it? Status pages trail impact by an hour. A five-step protocol to settle attribution in the first fifteen minutes of an outage.
Andon Cord Software Practice Misreads How Toyota Works
Toyota's andon cord rarely stopped the line, and it summoned help in seconds. What the real mechanics teach ops teams about escalation and cheap signals.
What we write about
The Operate blog covers the practical side of keeping production systems healthy. That includes diagnosing Kubernetes failures like CrashLoopBackOff and ImagePullBackOff, Postgres lock queues and query plan regressions, retry storms, connection pool exhaustion, and the operational lessons in public postmortems from AWS, GitHub, and others.
We also write about incident response as a discipline: severity levels, incident command, alert actionability, MTTR, and what aviation and emergency medicine can teach software teams about diagnosing problems under pressure.
AI in operations, carefully
A recurring theme is how to let AI agents investigate production safely. That means read-only access, evidence a human can verify, independent cross-checks of every conclusion, and an engineer approving any change. Our posts are candid about where automated root cause analysis helps and where it still needs human judgement.