https://operatex.dev/blog Field notes on AI SRE, root-cause analysis, and running self-hosted AI in production. en-us Sun, 19 Jul 2026 18:53:10 GMT https://operatex.dev/blog/stuck-state-july-2-postmortem-analysis https://operatex.dev/blog/stuck-state-july-2-postmortem-analysis Sun, 19 Jul 2026 06:49:46 GMT SRE & Platform Engineering Railway and CircleCI both published July 2, 2026 postmortems with the same hidden failure mode: state captured during instability that never corrected itself. https://operatex.dev/blog/reduce-mttr-diagnosis-first-playbook https://operatex.dev/blog/reduce-mttr-diagnosis-first-playbook Sat, 18 Jul 2026 17:11:06 GMT SRE & Engineering Efficiency About 80 percent of MTTR is spent finding the cause, not fixing it. A diagnosis-first playbook for reducing MTTR across code, database, infra, logs, and CI. https://operatex.dev/blog/on-call-alert-fatigue-solutions-triage-evidence https://operatex.dev/blog/on-call-alert-fatigue-solutions-triage-evidence Thu, 16 Jul 2026 19:41:11 GMT SRE & Incident Management On-call alert fatigue is a triage problem, not a threshold problem. A practical guide to cutting page volume by investigating alerts to root cause across the whole stack. https://operatex.dev/blog/agentic-database-autonomy-verification-risks https://operatex.dev/blog/agentic-database-autonomy-verification-risks Thu, 16 Jul 2026 00:15:24 GMT Engineering Strategy A response to 'Agentic databases aren't agentic': replaying traffic on a fork proves answer-equivalence, not production safety. Why the reviewable diff, not the autonomy boundary, is the unit of trust. https://operatex.dev/blog/automated-root-cause-analysis-safety-guide https://operatex.dev/blog/automated-root-cause-analysis-safety-guide Wed, 15 Jul 2026 08:07:43 GMT Engineering Leadership A practical, vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production. https://operatex.dev/blog/knight-capital-97-unread-emails-ai-agent-era https://operatex.dev/blog/knight-capital-97-unread-emails-ai-agent-era Wed, 15 Jul 2026 02:47:38 GMT Postmortems Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes. https://operatex.dev/blog/idle-in-transaction-postgres https://operatex.dev/blog/idle-in-transaction-postgres Wed, 15 Jul 2026 02:28:00 GMT Postgres What idle in transaction means in Postgres, why it silently blocks autovacuum while dashboards stay green, the 60-second diagnosis SQL, and the one-line timeout that prevents it. https://operatex.dev/blog/flaky-test-incident-runbook https://operatex.dev/blog/flaky-test-incident-runbook Tue, 14 Jul 2026 19:04:58 GMT Engineering Operations A practical runbook that treats flaky tests as incidents: detection thresholds, a five-cause taxonomy, quarantine exit criteria, and the math of verifying a fix. https://operatex.dev/blog/root-cause-analysis-second-opinion-21-percent-problem https://operatex.dev/blog/root-cause-analysis-second-opinion-21-percent-problem Tue, 14 Jul 2026 00:21:50 GMT Engineering Leadership When Mayo Clinic re-examined referred diagnoses, 21% changed completely. Root cause analysis has the same failure mode and none of the safeguards. https://operatex.dev/blog/dashboards-green-but-system-down-failure-class https://operatex.dev/blog/dashboards-green-but-system-down-failure-class Mon, 13 Jul 2026 04:52:38 GMT Engineering & SRE Pod healthy, service healthy, endpoint healthy - and the request times out. Four real incidents where every dashboard was green, and a cross-layer protocol for the failure class between components. https://operatex.dev/blog/stuck-state-postmortem-railway-circleci-outage https://operatex.dev/blog/stuck-state-postmortem-railway-circleci-outage Sun, 12 Jul 2026 14:50:55 GMT Engineering & SRE Railway and CircleCI both went down on July 2, 2026. The interesting failures came after recovery: connections and workflows that silently kept bad state. https://operatex.dev/blog/railway-outage-stale-connections-network-recovery https://operatex.dev/blog/railway-outage-stale-connections-network-recovery Sat, 11 Jul 2026 23:01:53 GMT Incident Analysis Railway's July 2026 outage report shows three systems that captured bad state during 20 minutes of instability and held it silently. Here's the failure class. https://operatex.dev/blog/monzo-on-call-rotation-models-scaling https://operatex.dev/blog/monzo-on-call-rotation-models-scaling Thu, 09 Jul 2026 20:33:21 GMT Engineering Management Monzo published three on-call models at three company sizes - with pay figures and honest failure notes. A teardown of the scaling cliffs every org hits. https://operatex.dev/blog/automated-root-cause-analysis-no-ai-write-access https://operatex.dev/blog/automated-root-cause-analysis-no-ai-write-access Thu, 09 Jul 2026 08:49:05 GMT Engineering Manual RCA eats 30-40% of engineering time. Here's how to automate the investigation across your whole stack without giving AI write access to production. https://operatex.dev/blog/automated-root-cause-analysis-2026-vendor-neutral-guide https://operatex.dev/blog/automated-root-cause-analysis-2026-vendor-neutral-guide Thu, 09 Jul 2026 08:32:46 GMT Engineering A vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production. https://operatex.dev/blog/the-buyers-guide-to-automated-root-cause-analysis-what-the-ai-answers-leave-out-7gmv https://operatex.dev/blog/the-buyers-guide-to-automated-root-cause-analysis-what-the-ai-answers-leave-out-7gmv Wed, 08 Jul 2026 14:52:23 GMT Engineering Leadership A vendor-neutral buyer's guide to automated root cause analysis: the features to look for, whole-stack coverage, and why the safe default is patch-file-only, no write access. https://operatex.dev/blog/verifying-ai-root-cause-analysis-aviation-cross-check https://operatex.dev/blog/verifying-ai-root-cause-analysis-aviation-cross-check Mon, 06 Jul 2026 09:12:12 GMT Engineering Leadership Airbus builds redundant flight computers with different chips, teams, and languages. Aviation's independence principle is the standard AI root-cause analysis should be held to. https://operatex.dev/blog/workos-database-connection-pool-exhaustion-analysis https://operatex.dev/blog/workos-database-connection-pool-exhaustion-analysis Sat, 04 Jul 2026 05:23:06 GMT Postmortem Analysis WorkOS added capacity and nothing improved. Hold-and-wait connection-pool starvation, why it self-sustains, and the audit to run on your own critical path. https://operatex.dev/blog/when-to-hire-an-sre-framework https://operatex.dev/blog/when-to-hire-an-sre-framework Sat, 04 Jul 2026 00:33:34 GMT Engineering Leadership Everyone ranking for 'when to hire an SRE' is selling the answer. A disinterested framework: the real signals, what must exist before the hire, and the alternatives to try first. https://operatex.dev/blog/sre-toil-ai-paradox-2026 https://operatex.dev/blog/sre-toil-ai-paradox-2026 Fri, 03 Jul 2026 18:55:32 GMT Engineering Leadership Survey toil fell for years, then reversed as AI adoption went mainstream. The mechanism: verification burden, new ops surfaces, and AI aimed at the wrong loop. https://operatex.dev/blog/zero-downtime-database-migration-guide https://operatex.dev/blog/zero-downtime-database-migration-guide Thu, 02 Jul 2026 09:57:00 GMT Engineering Operations The expand-contract playbook plus what guides skip: foreign keys, stored procedures, backup mismatches, and how to verify a live migration is safe step by step. https://operatex.dev/blog/ai-infrastructure-changes-audit-trail-documentation https://operatex.dev/blog/ai-infrastructure-changes-audit-trail-documentation Wed, 01 Jul 2026 22:19:25 GMT Engineering Strategy AI-assisted terraform and pipeline changes ship fast, but the reasoning dies with the chat session. A practical convention for keeping the why in your repo. https://operatex.dev/blog/flaky-tests-system-nondeterminism-not-noise https://operatex.dev/blog/flaky-tests-system-nondeterminism-not-noise Sat, 27 Jun 2026 02:29:41 GMT Reliability Engineering Google found 84% of CI pass-to-fail transitions are flakes. Retrying hides the race conditions and shared state that later cause production incidents. https://operatex.dev/blog/rightsizing-kubernetes-requests-guide https://operatex.dev/blog/rightsizing-kubernetes-requests-guide Thu, 25 Jun 2026 02:14:44 GMT Engineering Management Your cluster runs at 25 percent because nobody wants to own the next latency blip. A rollout playbook for cutting padded Kubernetes requests safely. https://operatex.dev/blog/delhi-data-center-fire-blast-radius-lessons https://operatex.dev/blog/delhi-data-center-fire-blast-radius-lessons Wed, 24 Jun 2026 03:11:21 GMT Incident Response A fire in one Delhi data center destroyed decades of customer data and degraded Google Cloud for weeks. Eight audit questions to ask about your own blast radius. https://operatex.dev/blog/silent-failures-ai-agent-pipelines-200-ok https://operatex.dev/blog/silent-failures-ai-agent-pipelines-200-ok Mon, 22 Jun 2026 23:22:30 GMT Engineering & SRE Agent pipelines fail while every dashboard stays green because 200 OK measures transport, not work. Five shapes of silent failure and how to detect each. https://operatex.dev/blog/database-schema-drift-operational-guide https://operatex.dev/blog/database-schema-drift-operational-guide Mon, 22 Jun 2026 17:18:36 GMT Database Operations Schema drift turns deploys, failovers, and restores into incidents. How drift happens, what Snowflake's 13-hour outage shows, and how to detect and triage it. https://operatex.dev/blog/mttr-median-vs-mean-outage-data-study https://operatex.dev/blog/mttr-median-vs-mean-outage-data-study Mon, 22 Jun 2026 05:18:16 GMT Engineering Management Fresh data from 1.8 million outages: median resolution 1.9 minutes, mean 21.9. The gap explains why your MTTR dashboard misleads, and what to report instead. https://operatex.dev/blog/knight-capital-incident-lessons-ai-agents https://operatex.dev/blog/knight-capital-incident-lessons-ai-agents Sun, 21 Jun 2026 22:35:44 GMT Engineering Leadership Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes. https://operatex.dev/blog/observability-costs-investigation-hedge https://operatex.dev/blog/observability-costs-investigation-hedge Sun, 21 Jun 2026 20:40:02 GMT Engineering Management Every cost guide says collect less telemetry. The real lever is why you store it: an investigation model that can only query pre-collected data. Fix that instead. https://operatex.dev/blog/how-to-reduce-mttr-diagnosis-bottleneck https://operatex.dev/blog/how-to-reduce-mttr-diagnosis-bottleneck Fri, 19 Jun 2026 15:15:57 GMT Engineering Leadership Most MTTR guides optimize detection. The clock is lost in diagnosis. A phase-by-phase breakdown of where incident time goes and how to shrink the 80%. https://operatex.dev/blog/third-party-outage-triage-protocol-aws-july-2026 https://operatex.dev/blog/third-party-outage-triage-protocol-aws-july-2026 Fri, 19 Jun 2026 09:23:09 GMT Incident Response AWS's July 16 CloudFront outage took 59 minutes to reach the status page. A triage protocol for deciding, in the first fifteen minutes, whether an incident is yours or upstream. https://operatex.dev/blog/andon-cord-software-development-myth https://operatex.dev/blog/andon-cord-software-development-myth Thu, 18 Jun 2026 16:35:54 GMT Engineering Operations Toyota's andon cord rarely stopped the line—it summoned help in seconds. What the real mechanics teach ops teams about escalation, freezes, and cheap signals.