https://operatex.dev/blog Field notes on AI SRE, root-cause analysis, and running self-hosted AI in production. en-us Fri, 07 Aug 2026 20:24:17 GMT https://operatex.dev/blog/too-many-open-files-field-guide https://operatex.dev/blog/too-many-open-files-field-guide Fri, 07 Aug 2026 13:00:23 GMT Troubleshooting Too many open files is rarely a limit problem. How to tell a descriptor leak from a sizing or latency issue, across Linux, nginx, Postgres and Kubernetes. https://operatex.dev/blog/postgres-index-bloat-ratchet-review https://operatex.dev/blog/postgres-index-bloat-ratchet-review Thu, 06 Aug 2026 13:00:30 GMT Database Reliability One production table carried 17 GB of index over 11 GB of heap, and every index was justified when it was added. Why index sets only grow and how to run a review. https://operatex.dev/blog/ci-cd-pipeline-failure-green-no-op-taxonomy https://operatex.dev/blog/ci-cd-pipeline-failure-green-no-op-taxonomy Wed, 05 Aug 2026 13:00:27 GMT Engineering Operations Pipeline config bugs almost always fail toward doing less, so they never raise an error. A taxonomy of steps that succeed by doing nothing and how to catch them. https://operatex.dev/blog/configuration-drift-incident-triage-guide https://operatex.dev/blog/configuration-drift-incident-triage-guide Tue, 04 Aug 2026 13:00:41 GMT Engineering Operations Most configuration drift guides stop at infrastructure as code. This one covers the drift IaC never sees, how to tell legitimate drift from dangerous drift, and a triage runbook for incidents. https://operatex.dev/blog/normal-accidents-cloud-scale-cascading-failure https://operatex.dev/blog/normal-accidents-cloud-scale-cascading-failure Mon, 03 Aug 2026 13:00:35 GMT Engineering Strategy Perrow's Normal Accidents predicted the Oct 2025 AWS cascade: tight coupling plus interactive complexity, triggered by the redundancy itself. An engineer's guide. https://operatex.dev/blog/solo-founder-devops-playbook https://operatex.dev/blog/solo-founder-devops-playbook Sun, 02 Aug 2026 13:00:32 GMT Operations Strategy AI made shipping cheap; operating is still expensive. A triage order, minimum telemetry list, and one-page runbook for solo founders and two-person teams. https://operatex.dev/blog/transactional-outbox-pattern-production-guide https://operatex.dev/blog/transactional-outbox-pattern-production-guide Sat, 01 Aug 2026 13:00:39 GMT Engineering Operations Every outbox pattern tutorial stops at the happy path. A day-2 guide to relay stalls, outbox lag, table bloat, poison messages, and the metrics that catch them. https://operatex.dev/blog/incident-closure-rate-vs-recurrence-rate https://operatex.dev/blog/incident-closure-rate-vs-recurrence-rate Fri, 31 Jul 2026 13:00:21 GMT Incident Management Ticket closure rate rewards silencing symptoms, and automation industrializes it. A replacement scorecard: recurrence rate, time-to-recurrence, verified-fix ratio. https://operatex.dev/blog/ai-ops-architecture-consensus-five-constraints https://operatex.dev/blog/ai-ops-architecture-consensus-five-constraints Thu, 30 Jul 2026 13:00:36 GMT Engineering Leadership Six unaffiliated engineers building AI for operations converged on the same five constraints. What that consensus is, why it happened, and how to test vendors against it. https://operatex.dev/blog/p99-latency-field-guide-tail-regressions https://operatex.dev/blog/p99-latency-field-guide-tail-regressions Wed, 29 Jul 2026 13:00:36 GMT SRE & Performance What p99 latency is, why tails regress without a deploy, and a workflow for finding the cause: histograms, exemplars, per-route views, and SLOs that catch tails. https://operatex.dev/blog/incident-commander-ics-doctrine-software-sre https://operatex.dev/blog/incident-commander-ics-doctrine-software-sre Tue, 28 Jul 2026 13:00:40 GMT Incident Management Software took the incident commander title from fire service ICS and dropped the doctrine. What actually transfers - span of control, transfer of command - and what doesn't. https://operatex.dev/blog/everyone-is-building-an-ops-copilot-diy-wave-notes https://operatex.dev/blog/everyone-is-building-an-ops-copilot-diy-wave-notes Mon, 27 Jul 2026 13:00:39 GMT Engineering Leadership Practitioners are assembling self-hosted LLM ops stacks. What the DIY wave gets right, where it stalls, and a build-vs-buy framework for AI operations tooling. https://operatex.dev/blog/dead-letter-queue-operating-manual https://operatex.dev/blog/dead-letter-queue-operating-manual Sun, 26 Jul 2026 13:00:39 GMT Engineering Ops Every broker doc explains configuring a dead letter queue. This is the missing operating manual: metrics, alert thresholds, failure triage, and safe redrives. https://operatex.dev/blog/code-yellow-engineering-escalation-guide https://operatex.dev/blog/code-yellow-engineering-escalation-guide Sat, 25 Jul 2026 13:00:41 GMT Engineering Management A Code Yellow is the bill for months of undiagnosed degradation. When to declare one, how to run it, and the signals that fire long before you need the protocol. https://operatex.dev/blog/readwriteonce-vs-readwritemany-kubernetes-misconfig https://operatex.dev/blog/readwriteonce-vs-readwritemany-kubernetes-misconfig Fri, 24 Jul 2026 13:00:42 GMT Engineering ReadWriteOnce limits a volume to one node, not one pod. Why this Kubernetes misconfig passes every test and fails during a node drain, and how to audit for it. https://operatex.dev/blog/n-plus-1-query-problem-production-detection https://operatex.dev/blog/n-plus-1-query-problem-production-detection Thu, 23 Jul 2026 13:00:37 GMT Engineering Persistence The N+1 query problem rarely shows up in code review. How it ships silently, how to detect it in production telemetry, and how to fix it without regressions. https://operatex.dev/blog/silent-failures-software-2003-blackout-monitoring https://operatex.dev/blog/silent-failures-software-2003-blackout-monitoring Wed, 22 Jul 2026 13:00:41 GMT Reliability Engineering A millisecond race condition silently killed the alarms before the 2003 blackout. The same failure mode lives in your cron jobs, queues, and alert pipelines. https://operatex.dev/blog/capacity-math-unplanned-work-engineering https://operatex.dev/blog/capacity-math-unplanned-work-engineering Tue, 21 Jul 2026 13:00:37 GMT Engineering Leadership Accelerate, Stripe, and real team data on unplanned engineering work, turned into capacity math you can run on your own org in two weeks. https://operatex.dev/blog/aws-outage-postmortem-lessons-2025-2026-streak https://operatex.dev/blog/aws-outage-postmortem-lessons-2025-2026-streak Mon, 20 Jul 2026 13:00:39 GMT Engineering Management Four major AWS incidents in eight months, one public postmortem. A customer-side postmortem template for turning provider outages into your own reliability review. https://operatex.dev/blog/stuck-state-july-2-postmortem-analysis https://operatex.dev/blog/stuck-state-july-2-postmortem-analysis Sun, 19 Jul 2026 06:49:46 GMT SRE & Platform Engineering Railway and CircleCI both published July 2, 2026 postmortems with the same hidden failure mode: state captured during instability that never corrected itself. https://operatex.dev/blog/reduce-mttr-diagnosis-first-playbook https://operatex.dev/blog/reduce-mttr-diagnosis-first-playbook Sat, 18 Jul 2026 17:11:06 GMT SRE & Engineering Efficiency About 80 percent of MTTR is spent finding the cause, not fixing it. A diagnosis-first playbook for reducing MTTR across code, database, infra, logs, and CI. https://operatex.dev/blog/on-call-alert-fatigue-solutions-triage-evidence https://operatex.dev/blog/on-call-alert-fatigue-solutions-triage-evidence Thu, 16 Jul 2026 19:41:11 GMT SRE & Incident Management On-call alert fatigue is a triage problem, not a threshold problem. A practical guide to cutting page volume by investigating alerts to root cause across the whole stack. https://operatex.dev/blog/agentic-database-autonomy-verification-risks https://operatex.dev/blog/agentic-database-autonomy-verification-risks Thu, 16 Jul 2026 00:15:24 GMT Engineering Strategy A response to 'Agentic databases aren't agentic': replaying traffic on a fork proves answer-equivalence, not production safety. Why the reviewable diff, not the autonomy boundary, is the unit of trust. https://operatex.dev/blog/automated-root-cause-analysis-safety-guide https://operatex.dev/blog/automated-root-cause-analysis-safety-guide Wed, 15 Jul 2026 08:07:43 GMT Engineering Leadership A practical, vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production. https://operatex.dev/blog/knight-capital-97-unread-emails-ai-agent-era https://operatex.dev/blog/knight-capital-97-unread-emails-ai-agent-era Wed, 15 Jul 2026 02:47:38 GMT Postmortems Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes. https://operatex.dev/blog/idle-in-transaction-postgres https://operatex.dev/blog/idle-in-transaction-postgres Wed, 15 Jul 2026 02:28:00 GMT Postgres What idle in transaction means in Postgres, why it silently blocks autovacuum while dashboards stay green, the 60-second diagnosis SQL, and the one-line timeout that prevents it. https://operatex.dev/blog/flaky-test-incident-runbook https://operatex.dev/blog/flaky-test-incident-runbook Tue, 14 Jul 2026 19:04:58 GMT Engineering Operations A practical runbook that treats flaky tests as incidents: detection thresholds, a five-cause taxonomy, quarantine exit criteria, and the math of verifying a fix. https://operatex.dev/blog/root-cause-analysis-second-opinion-21-percent-problem https://operatex.dev/blog/root-cause-analysis-second-opinion-21-percent-problem Tue, 14 Jul 2026 00:21:50 GMT Engineering Leadership When Mayo Clinic re-examined referred diagnoses, 21% changed completely. Root cause analysis has the same failure mode and none of the safeguards. https://operatex.dev/blog/dashboards-green-but-system-down-failure-class https://operatex.dev/blog/dashboards-green-but-system-down-failure-class Mon, 13 Jul 2026 04:52:38 GMT Engineering & SRE Pod healthy, service healthy, endpoint healthy - and the request times out. Four real incidents where every dashboard was green, and a cross-layer protocol for the failure class between components. https://operatex.dev/blog/stuck-state-postmortem-railway-circleci-outage https://operatex.dev/blog/stuck-state-postmortem-railway-circleci-outage Sun, 12 Jul 2026 14:50:55 GMT Engineering & SRE Railway and CircleCI both went down on July 2, 2026. The interesting failures came after recovery: connections and workflows that silently kept bad state. https://operatex.dev/blog/railway-outage-stale-connections-network-recovery https://operatex.dev/blog/railway-outage-stale-connections-network-recovery Sat, 11 Jul 2026 23:01:53 GMT Incident Analysis Railway's July 2026 outage report shows three systems that captured bad state during 20 minutes of instability and held it silently. Here's the failure class. https://operatex.dev/blog/monzo-on-call-rotation-models-scaling https://operatex.dev/blog/monzo-on-call-rotation-models-scaling Thu, 09 Jul 2026 20:33:21 GMT Engineering Management Monzo published three on-call models at three company sizes - with pay figures and honest failure notes. A teardown of the scaling cliffs every org hits. https://operatex.dev/blog/automated-root-cause-analysis-no-ai-write-access https://operatex.dev/blog/automated-root-cause-analysis-no-ai-write-access Thu, 09 Jul 2026 08:49:05 GMT Engineering Manual RCA eats 30-40% of engineering time. Here's how to automate the investigation across your whole stack without giving AI write access to production. https://operatex.dev/blog/automated-root-cause-analysis-2026-vendor-neutral-guide https://operatex.dev/blog/automated-root-cause-analysis-2026-vendor-neutral-guide Thu, 09 Jul 2026 08:32:46 GMT Engineering A vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production. https://operatex.dev/blog/the-buyers-guide-to-automated-root-cause-analysis-what-the-ai-answers-leave-out-7gmv https://operatex.dev/blog/the-buyers-guide-to-automated-root-cause-analysis-what-the-ai-answers-leave-out-7gmv Wed, 08 Jul 2026 14:52:23 GMT Engineering Leadership A vendor-neutral buyer's guide to automated root cause analysis: the features to look for, whole-stack coverage, and why the safe default is patch-file-only, no write access. https://operatex.dev/blog/verifying-ai-root-cause-analysis-aviation-cross-check https://operatex.dev/blog/verifying-ai-root-cause-analysis-aviation-cross-check Mon, 06 Jul 2026 09:12:12 GMT Engineering Leadership Airbus builds redundant flight computers with different chips, teams, and languages. Aviation's independence principle is the standard AI root-cause analysis should be held to. https://operatex.dev/blog/workos-database-connection-pool-exhaustion-analysis https://operatex.dev/blog/workos-database-connection-pool-exhaustion-analysis Sat, 04 Jul 2026 05:23:06 GMT Postmortem Analysis WorkOS added capacity and nothing improved. Hold-and-wait connection-pool starvation, why it self-sustains, and the audit to run on your own critical path. https://operatex.dev/blog/when-to-hire-an-sre-framework https://operatex.dev/blog/when-to-hire-an-sre-framework Sat, 04 Jul 2026 00:33:34 GMT Engineering Leadership Everyone ranking for 'when to hire an SRE' is selling the answer. A disinterested framework: the real signals, what must exist before the hire, and the alternatives to try first. https://operatex.dev/blog/sre-toil-ai-paradox-2026 https://operatex.dev/blog/sre-toil-ai-paradox-2026 Fri, 03 Jul 2026 18:55:32 GMT Engineering Leadership Survey toil fell for years, then reversed as AI adoption went mainstream. The mechanism: verification burden, new ops surfaces, and AI aimed at the wrong loop. https://operatex.dev/blog/zero-downtime-database-migration-guide https://operatex.dev/blog/zero-downtime-database-migration-guide Thu, 02 Jul 2026 09:57:00 GMT Engineering Operations The expand-contract playbook plus what guides skip: foreign keys, stored procedures, backup mismatches, and how to verify a live migration is safe step by step. https://operatex.dev/blog/ai-infrastructure-changes-audit-trail-documentation https://operatex.dev/blog/ai-infrastructure-changes-audit-trail-documentation Wed, 01 Jul 2026 22:19:25 GMT Engineering Strategy AI-assisted terraform and pipeline changes ship fast, but the reasoning dies with the chat session. A practical convention for keeping the why in your repo. https://operatex.dev/blog/flaky-tests-system-nondeterminism-not-noise https://operatex.dev/blog/flaky-tests-system-nondeterminism-not-noise Sat, 27 Jun 2026 02:29:41 GMT Reliability Engineering Google found 84% of CI pass-to-fail transitions are flakes. Retrying hides the race conditions and shared state that later cause production incidents. https://operatex.dev/blog/rightsizing-kubernetes-requests-guide https://operatex.dev/blog/rightsizing-kubernetes-requests-guide Thu, 25 Jun 2026 02:14:44 GMT Engineering Management Your cluster runs at 25 percent because nobody wants to own the next latency blip. A rollout playbook for cutting padded Kubernetes requests safely. https://operatex.dev/blog/delhi-data-center-fire-blast-radius-lessons https://operatex.dev/blog/delhi-data-center-fire-blast-radius-lessons Wed, 24 Jun 2026 03:11:21 GMT Incident Response A fire in one Delhi data center destroyed decades of customer data and degraded Google Cloud for weeks. Eight audit questions to ask about your own blast radius. https://operatex.dev/blog/silent-failures-ai-agent-pipelines-200-ok https://operatex.dev/blog/silent-failures-ai-agent-pipelines-200-ok Mon, 22 Jun 2026 23:22:30 GMT Engineering & SRE Agent pipelines fail while every dashboard stays green because 200 OK measures transport, not work. Five shapes of silent failure and how to detect each. https://operatex.dev/blog/database-schema-drift-operational-guide https://operatex.dev/blog/database-schema-drift-operational-guide Mon, 22 Jun 2026 17:18:36 GMT Database Operations Schema drift turns deploys, failovers, and restores into incidents. How drift happens, what Snowflake's 13-hour outage shows, and how to detect and triage it. https://operatex.dev/blog/mttr-median-vs-mean-outage-data-study https://operatex.dev/blog/mttr-median-vs-mean-outage-data-study Mon, 22 Jun 2026 05:18:16 GMT Engineering Management Fresh data from 1.8 million outages: median resolution 1.9 minutes, mean 21.9. The gap explains why your MTTR dashboard misleads, and what to report instead. https://operatex.dev/blog/knight-capital-incident-lessons-ai-agents https://operatex.dev/blog/knight-capital-incident-lessons-ai-agents Sun, 21 Jun 2026 22:35:44 GMT Engineering Leadership Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes. https://operatex.dev/blog/observability-costs-investigation-hedge https://operatex.dev/blog/observability-costs-investigation-hedge Sun, 21 Jun 2026 20:40:02 GMT Engineering Management Every cost guide says collect less telemetry. The real lever is why you store it: an investigation model that can only query pre-collected data. Fix that instead. https://operatex.dev/blog/how-to-reduce-mttr-diagnosis-bottleneck https://operatex.dev/blog/how-to-reduce-mttr-diagnosis-bottleneck Fri, 19 Jun 2026 15:15:57 GMT Engineering Leadership Most MTTR guides optimize detection. The clock is lost in diagnosis. A phase-by-phase breakdown of where incident time goes and how to shrink the 80%. https://operatex.dev/blog/third-party-outage-triage-protocol-aws-july-2026 https://operatex.dev/blog/third-party-outage-triage-protocol-aws-july-2026 Fri, 19 Jun 2026 09:23:09 GMT Incident Response AWS's July 16 CloudFront outage took 59 minutes to reach the status page. A triage protocol for deciding, in the first fifteen minutes, whether an incident is yours or upstream. https://operatex.dev/blog/andon-cord-software-development-myth https://operatex.dev/blog/andon-cord-software-development-myth Thu, 18 Jun 2026 16:35:54 GMT Engineering Operations Toyota's andon cord rarely stopped the line—it summoned help in seconds. What the real mechanics teach ops teams about escalation, freezes, and cheap signals.