The Operate blog

    Notes from the frontline of AI ops

    Patterns, incidents, and lessons from running AI SRE in production — for teams that own their infrastructure.

    We write about root cause analysis, on-call and incident response, postmortems of real outages, observability, and what it takes to let AI agents investigate production safely — with read-only access, verified evidence, and a human approving every change.

    Category:
    Incident and Problem Management Change Mean Time to Resolution
    Incident Response
    Nov 16, 2026·13 min read

    Incident and Problem Management Change Mean Time to Resolution

    Compare incident and problem management, preserve restoration evidence, and correct cause links without distorting your resolution-time reports.

    How to Choose Logs or Traces During a Production Investigation
    Reliability Engineering
    Oct 30, 2026·14 min read

    How to Choose Logs or Traces During a Production Investigation

    We compare logs and traces, validate correlation IDs, diagnose missing spans and preserve the evidence needed for a production debugging review.

    How Does Workflow Monitoring Differ With Automation or Orchestration?
    Software Operations
    Oct 30, 2026·12 min read

    How Does Workflow Monitoring Differ With Automation or Orchestration?

    We compare automation and orchestration through decision ownership, cutover checks and monitoring so operations teams can choose a clear control boundary.

    Whose Permissions Does an AI SRE Use When Support Asks in Slack?
    Engineering Operations
    Oct 11, 2026·15 min read

    Whose Permissions Does an AI SRE Use When Support Asks in Slack?

    Check whose permissions an AI SRE uses from Slack with a practical worksheet for requester access, source scope, saved results and output audiences.

    How Much Does an AI SRE Cost When One Incident Triggers Several Investigations?
    Engineering Operations
    Oct 9, 2026·15 min read

    How Much Does an AI SRE Cost When One Incident Triggers Several Investigations?

    Compare AI SRE pricing per incident, including repeat runs, credit bundles, query fees and human review. Use a ledger to test your own pilot costs.

    How to Run Root Cause Analysis in Software Without Stopping at the First Why
    Incident Response
    Oct 2, 2026·16 min read

    How to Run Root Cause Analysis in Software Without Stopping at the First Why

    We walk through root cause analysis in software with six evidence steps, method choice, and a fault tree of Cloudflare's 2025 outage.

    What Is an AI SRE (and Which Parts of the SRE Job It Actually Does)
    Reliability Engineering
    Oct 1, 2026·14 min read

    What Is an AI SRE (and Which Parts of the SRE Job It Actually Does)

    We define an AI SRE against the real SRE job, grade vendor definitions on Google's five autonomy levels, and list what to settle before one reads production.

    Self-Hosted vs Cloud AI Agents, Which One Passes a Production Security Review?
    Engineering Operations
    Sep 30, 2026·14 min read

    Self-Hosted vs Cloud AI Agents, Which One Passes a Production Security Review?

    Compare self-hosted and cloud AI agents that read production, with five deployment shapes, eight security review questions, and where LLM prompts go.

    What Is a Good MTTR for a Software Team (and How to Set Your Own Target)
    Reliability Engineering
    Sep 28, 2026·12 min read

    What Is a Good MTTR for a Software Team (and How to Set Your Own Target)

    What a good MTTR looks like for software teams, using DORA's recovery bands and our target from the error budget and incident rate.

    Which Postgres Monitoring Tools Keep the Evidence You Need After an Incident?
    Engineering
    Sep 24, 2026·10 min read

    Which Postgres Monitoring Tools Keep the Evidence You Need After an Incident?

    We compare six Postgres monitoring tools by the evidence they keep after an incident: blocking chains, query history, wait events and captured plans.

    Read-Only Production Access Is a Capacity Change Too
    Engineering Management
    Sep 22, 2026·9 min read

    Read-Only Production Access Is a Capacity Change Too

    Read-only production access stops writes, not load. Locks, temp files, open snapshots and replica conflicts still hit production, and most guardrails ship off.

    Postgres ALTER TABLE Lock Queues Can Block Every Read
    Database Engineering
    Sep 22, 2026·10 min read

    Postgres ALTER TABLE Lock Queues Can Block Every Read

    A metadata-only Postgres ALTER TABLE still waits for an exclusive lock, and every read queues behind it. How the lock queue works and how to bound it.

    Terraform force-unlock Is the Easy Part: Proving the Lock Is Orphaned, and Finding What the Dead Apply Already Built
    Infrastructure & Devops
    Sep 22, 2026·7 min read

    Terraform force-unlock Is the Easy Part: Proving the Lock Is Orphaned, and Finding What the Dead Apply Already Built

    terraform force-unlock takes ten seconds. Proving the lock is orphaned, not live, takes six metadata fields and one audit log query. Plus finding what the dead apply already built.

    EMFILE vs ENFILE vs Inotify for Too Many Open Files
    Engineering
    Sep 21, 2026·10 min read

    EMFILE vs ENFILE vs Inotify for Too Many Open Files

    Three unrelated Linux limits print too many open files. How EMFILE, ENFILE and the inotify limits differ, and where to read each one in a minute.

    Exactly Once Delivery Is a Property of the Write
    Engineering & Operations
    Sep 20, 2026·11 min read

    Exactly Once Delivery Is a Property of the Write

    Exactly once delivery stops at the broker. Most duplicates come from a commit whose acknowledgement was lost, so the guarantee belongs in the write itself.

    TLS Handshake Timeout Is Usually Not a TLS Problem
    Engineering
    Sep 19, 2026·11 min read

    TLS Handshake Timeout Is Usually Not a TLS Problem

    A TLS handshake timeout means the client heard silence after TCP connected, not a certificate fault. Seven ordered checks prove where the wait began.

    504 Gateway Timeout Means a Timer Expired, Not a Slow Hop
    Engineering Operations
    Sep 17, 2026·10 min read

    504 Gateway Timeout Means a Timer Expired, Not a Slow Hop

    A 504 gateway timeout names the hop whose timer expired, not the slow one. Build the timeout ladder for Nginx, your load balancer and database to find it.

    Postgres Autovacuum Diagnosis That Starts From the Symptom
    Engineering
    Sep 15, 2026·10 min read

    Postgres Autovacuum Diagnosis That Starts From the Symptom

    Start a Postgres autovacuum investigation at the alert. Four symptoms, five look-alikes to rule out, five vacuum failures to confirm and the wraparound runbook.

    No Space Left on Device Errors That Are Not a Full Disk
    Reliability Engineering
    Sep 14, 2026·10 min read

    No Space Left on Device Errors That Are Not a Full Disk

    No space left on device does not always mean a full disk. An ordered ENOSPC procedure covering inodes, open deleted files, quotas, containers and inotify.

    Exit Code 137 Means SIGKILL, Not Always Out of Memory
    Engineering & SRE
    Sep 11, 2026·10 min read

    Exit Code 137 Means SIGKILL, Not Always Out of Memory

    Exit code 137 means a SIGKILL arrived, not always that memory ran out. The six senders, the evidence each leaves, and how to prove which one did it.

    Connection Reset by Peer Means a Component Chose to Reset
    Engineering
    Aug 25, 2026·12 min read

    Connection Reset by Peer Means a Component Chose to Reset

    Connection reset by peer means a TCP RST arrived. Learn the six components that send one, their timing signatures, and how to prove which one did.

    Context Deadline Exceeded Names the Caller, Not the Slow Hop
    SRE & Engineering
    Aug 11, 2026·10 min read

    Context Deadline Exceeded Names the Caller, Not the Slow Hop

    Context deadline exceeded is logged by the caller that gave up, not the slow hop. How the deadline budget shrinks and how to find the hop that spent it.

    CrashLoopBackOff Triage in the Order That Narrows the Cause
    Engineering Guides
    Aug 11, 2026·9 min read

    CrashLoopBackOff Triage in the Order That Narrows the Cause

    CrashLoopBackOff says a container keeps restarting, not why. Triage it in order, by blast radius, exit code and crash versus kill, before reading the logs.

    Postgres Query Plan Changed With No Deploy and p99 Tripled
    Database Reliability
    Aug 11, 2026·9 min read

    Postgres Query Plan Changed With No Deploy and p99 Tripled

    A Postgres query plan can change under an identical query. Stale statistics, the generic plan switch and lost partition pruning, each with a check and a fix.

    Transactional Outbox Pattern Problems That Start on Day Two
    Engineering Operations
    Aug 1, 2026·9 min read

    Transactional Outbox Pattern Problems That Start on Day Two

    The transactional outbox pattern fixes dual writes and adds relay stalls, lag, churn, poison messages and skipped rows. How to detect each one in production.

    Build vs Buy AI for Operations After the DIY Copilot Wave
    Engineering Leadership
    Jul 27, 2026·10 min read

    Build vs Buy AI for Operations After the DIY Copilot Wave

    Teams are building their own ops copilots on self-hosted models. What they get right, where internal copilots stall, and a build vs buy AI framework to use.

    A Dead Letter Queue Needs Owners, Alerts and a Redrive Rule
    Engineering Ops
    Jul 26, 2026·11 min read

    A Dead Letter Queue Needs Owners, Alerts and a Redrive Rule

    A dead letter queue only prevents data loss if someone watches it. The metrics, alert thresholds, failure triage and safe redrive steps that keep it honest.

    How to Catch the N+1 Query Problem in Production
    Engineering Persistence
    Jul 23, 2026·10 min read

    How to Catch the N+1 Query Problem in Production

    The N+1 query problem passes review and tests, then grows with your data. How to catch it in production telemetry, fix it safely and keep it from returning.

    Unplanned Work Is the Capacity Your Roadmap Never Counts
    Engineering Leadership
    Jul 21, 2026·9 min read

    Unplanned Work Is the Capacity Your Roadmap Never Counts

    Unplanned work takes a measurable share of every engineering week. What the data really shows, capacity math for a 20-engineer team and a two-week measure.

    How to Reduce On-Call Alert Fatigue by Triaging First
    SRE & Incident Management
    Jul 16, 2026·10 min read

    How to Reduce On-Call Alert Fatigue by Triaging First

    On-call alert fatigue comes from pages that need no action and carry no evidence. Five steps to cut it without hiding real incidents, and how to measure it.

    Idle in Transaction Postgres Sessions That Can Block Vacuum
    Postgres
    Jul 15, 2026·9 min read

    Idle in Transaction Postgres Sessions That Can Block Vacuum

    Idle in transaction Postgres sessions hold locks and, if they wrote data, block vacuum while dashboards stay green. Find them, end them, and set the timeout.

    AI Root Cause Analysis Without Write Access to Production
    Engineering
    Jul 9, 2026·11 min read

    AI Root Cause Analysis Without Write Access to Production

    AI root cause analysis can read logs, databases and code without write access. Where to draw the line, how to check answers, and what to ask a vendor first.

    How to Reduce MTTR by Shortening Diagnosis, Not Detection
    Engineering Leadership
    Jun 19, 2026·10 min read

    How to Reduce MTTR by Shortening Diagnosis, Not Detection

    How to reduce MTTR once alerting is already fast. Time each incident phase, then shrink diagnosis with rollbacks, one evidence timeline and a second check.

    What we write about

    The Operate blog covers the practical side of keeping production systems healthy. That includes diagnosing Kubernetes failures like CrashLoopBackOff and ImagePullBackOff, Postgres lock queues and query plan regressions, retry storms, connection pool exhaustion, and the operational lessons in public postmortems from AWS, GitHub, and others.

    We also write about incident response as a discipline: severity levels, incident command, alert actionability, MTTR, and what aviation and emergency medicine can teach software teams about diagnosing problems under pressure.

    AI in operations, carefully

    A recurring theme is how to let AI agents investigate production safely. That means read-only access, evidence a human can verify, independent cross-checks of every conclusion, and an engineer approving any change. Our posts are candid about where automated root cause analysis helps and where it still needs human judgement.