The Operate blog
Notes from the frontline of AI ops
Patterns, incidents, and lessons from running AI SRE in production — for teams that own their infrastructure.
We write about root cause analysis, on-call and incident response, postmortems of real outages, observability, and what it takes to let AI agents investigate production safely — with read-only access, verified evidence, and a human approving every change.

Incident and Problem Management Change Mean Time to Resolution
Compare incident and problem management, preserve restoration evidence, and correct cause links without distorting your resolution-time reports.

How to Choose Logs or Traces During a Production Investigation
We compare logs and traces, validate correlation IDs, diagnose missing spans and preserve the evidence needed for a production debugging review.

How Does Workflow Monitoring Differ With Automation or Orchestration?
We compare automation and orchestration through decision ownership, cutover checks and monitoring so operations teams can choose a clear control boundary.

Whose Permissions Does an AI SRE Use When Support Asks in Slack?
Check whose permissions an AI SRE uses from Slack with a practical worksheet for requester access, source scope, saved results and output audiences.

How Much Does an AI SRE Cost When One Incident Triggers Several Investigations?
Compare AI SRE pricing per incident, including repeat runs, credit bundles, query fees and human review. Use a ledger to test your own pilot costs.

How to Run Root Cause Analysis in Software Without Stopping at the First Why
We walk through root cause analysis in software with six evidence steps, method choice, and a fault tree of Cloudflare's 2025 outage.

What Is an AI SRE (and Which Parts of the SRE Job It Actually Does)
We define an AI SRE against the real SRE job, grade vendor definitions on Google's five autonomy levels, and list what to settle before one reads production.

Self-Hosted vs Cloud AI Agents, Which One Passes a Production Security Review?
Compare self-hosted and cloud AI agents that read production, with five deployment shapes, eight security review questions, and where LLM prompts go.

What Is a Good MTTR for a Software Team (and How to Set Your Own Target)
What a good MTTR looks like for software teams, using DORA's recovery bands and our target from the error budget and incident rate.

Which Postgres Monitoring Tools Keep the Evidence You Need After an Incident?
We compare six Postgres monitoring tools by the evidence they keep after an incident: blocking chains, query history, wait events and captured plans.

Read-Only Production Access Is a Capacity Change Too
Read-only production access stops writes, not load. Locks, temp files, open snapshots and replica conflicts still hit production, and most guardrails ship off.

Postgres ALTER TABLE Lock Queues Can Block Every Read
A metadata-only Postgres ALTER TABLE still waits for an exclusive lock, and every read queues behind it. How the lock queue works and how to bound it.

Terraform force-unlock Is the Easy Part: Proving the Lock Is Orphaned, and Finding What the Dead Apply Already Built
terraform force-unlock takes ten seconds. Proving the lock is orphaned, not live, takes six metadata fields and one audit log query. Plus finding what the dead apply already built.

EMFILE vs ENFILE vs Inotify for Too Many Open Files
Three unrelated Linux limits print too many open files. How EMFILE, ENFILE and the inotify limits differ, and where to read each one in a minute.

Exactly Once Delivery Is a Property of the Write
Exactly once delivery stops at the broker. Most duplicates come from a commit whose acknowledgement was lost, so the guarantee belongs in the write itself.

TLS Handshake Timeout Is Usually Not a TLS Problem
A TLS handshake timeout means the client heard silence after TCP connected, not a certificate fault. Seven ordered checks prove where the wait began.

504 Gateway Timeout Means a Timer Expired, Not a Slow Hop
A 504 gateway timeout names the hop whose timer expired, not the slow one. Build the timeout ladder for Nginx, your load balancer and database to find it.

Postgres Autovacuum Diagnosis That Starts From the Symptom
Start a Postgres autovacuum investigation at the alert. Four symptoms, five look-alikes to rule out, five vacuum failures to confirm and the wraparound runbook.

No Space Left on Device Errors That Are Not a Full Disk
No space left on device does not always mean a full disk. An ordered ENOSPC procedure covering inodes, open deleted files, quotas, containers and inotify.

Exit Code 137 Means SIGKILL, Not Always Out of Memory
Exit code 137 means a SIGKILL arrived, not always that memory ran out. The six senders, the evidence each leaves, and how to prove which one did it.

Connection Reset by Peer Means a Component Chose to Reset
Connection reset by peer means a TCP RST arrived. Learn the six components that send one, their timing signatures, and how to prove which one did.

Context Deadline Exceeded Names the Caller, Not the Slow Hop
Context deadline exceeded is logged by the caller that gave up, not the slow hop. How the deadline budget shrinks and how to find the hop that spent it.

CrashLoopBackOff Triage in the Order That Narrows the Cause
CrashLoopBackOff says a container keeps restarting, not why. Triage it in order, by blast radius, exit code and crash versus kill, before reading the logs.

Postgres Query Plan Changed With No Deploy and p99 Tripled
A Postgres query plan can change under an identical query. Stale statistics, the generic plan switch and lost partition pruning, each with a check and a fix.

Transactional Outbox Pattern Problems That Start on Day Two
The transactional outbox pattern fixes dual writes and adds relay stalls, lag, churn, poison messages and skipped rows. How to detect each one in production.

Build vs Buy AI for Operations After the DIY Copilot Wave
Teams are building their own ops copilots on self-hosted models. What they get right, where internal copilots stall, and a build vs buy AI framework to use.

A Dead Letter Queue Needs Owners, Alerts and a Redrive Rule
A dead letter queue only prevents data loss if someone watches it. The metrics, alert thresholds, failure triage and safe redrive steps that keep it honest.

How to Catch the N+1 Query Problem in Production
The N+1 query problem passes review and tests, then grows with your data. How to catch it in production telemetry, fix it safely and keep it from returning.

Unplanned Work Is the Capacity Your Roadmap Never Counts
Unplanned work takes a measurable share of every engineering week. What the data really shows, capacity math for a 20-engineer team and a two-week measure.

How to Reduce On-Call Alert Fatigue by Triaging First
On-call alert fatigue comes from pages that need no action and carry no evidence. Five steps to cut it without hiding real incidents, and how to measure it.

Idle in Transaction Postgres Sessions That Can Block Vacuum
Idle in transaction Postgres sessions hold locks and, if they wrote data, block vacuum while dashboards stay green. Find them, end them, and set the timeout.

AI Root Cause Analysis Without Write Access to Production
AI root cause analysis can read logs, databases and code without write access. Where to draw the line, how to check answers, and what to ask a vendor first.

How to Reduce MTTR by Shortening Diagnosis, Not Detection
How to reduce MTTR once alerting is already fast. Time each incident phase, then shrink diagnosis with rollbacks, one evidence timeline and a second check.
What we write about
The Operate blog covers the practical side of keeping production systems healthy. That includes diagnosing Kubernetes failures like CrashLoopBackOff and ImagePullBackOff, Postgres lock queues and query plan regressions, retry storms, connection pool exhaustion, and the operational lessons in public postmortems from AWS, GitHub, and others.
We also write about incident response as a discipline: severity levels, incident command, alert actionability, MTTR, and what aviation and emergency medicine can teach software teams about diagnosing problems under pressure.
AI in operations, carefully
A recurring theme is how to let AI agents investigate production safely. That means read-only access, evidence a human can verify, independent cross-checks of every conclusion, and an engineer approving any change. Our posts are candid about where automated root cause analysis helps and where it still needs human judgement.