# Operate

> Operate is a self-hosted AI SRE. It watches production, investigates alerts and silent failures,
> finds the root cause with evidence, and drafts the fix as a pull request. It runs inside your own
> infrastructure with read-only access, and nothing ships without human approval.

## Product

- [Home](https://operatex.dev/): what Operate is and how the four-agent pipeline works
- [Get started](https://operatex.dev/docs/get-started): Docker deployment and read-only credentials
- [Security](https://operatex.dev/security): self-hosted, read-only, human-gated architecture
- [Open source RCA](https://operatex.dev/open-source): free root cause analysis on public repositories

## Blog

- [Too Many Open Files Is Almost Never a ulimit Problem: A Production Field Guide to File Descriptor Exhaustion](https://operatex.dev/blog/too-many-open-files-field-guide) (2026-08-07): Too many open files is rarely a limit problem. How to tell a descriptor leak from a sizing or latency issue, across Linux, nginx, Postgres and Kubernetes.
- [Seven Indexes, Larger Than the Table: Why Index Sets Rot and Nothing Forces a Review](https://operatex.dev/blog/postgres-index-bloat-ratchet-review) (2026-08-06): One production table carried 17 GB of index over 11 GB of heap, and every index was justified when it was added. Why index sets only grow and how to run a review.
- [The Pipeline Was Green and It Deployed Nothing: A Taxonomy of Steps That Succeed by Doing Nothing](https://operatex.dev/blog/ci-cd-pipeline-failure-green-no-op-taxonomy) (2026-08-05): Pipeline config bugs almost always fail toward doing less, so they never raise an error. A taxonomy of steps that succeed by doing nothing and how to catch them.
- [Configuration Drift Is Not a Hygiene Problem: Triaging the Drift That Actually Causes Incidents](https://operatex.dev/blog/configuration-drift-incident-triage-guide) (2026-08-04): Most configuration drift guides stop at infrastructure as code. This one covers the drift IaC never sees, how to tell legitimate drift from dangerous drift, and a triage runbook for incidents.
- [Normal Accidents at Cloud Scale](https://operatex.dev/blog/normal-accidents-cloud-scale-cascading-failure) (2026-08-03): Perrow's Normal Accidents predicted the Oct 2025 AWS cascade: tight coupling plus interactive complexity, triggered by the redundancy itself. An engineer's guide.
- [Alarms Are the Cheap Half: An Ops Playbook for Teams of One](https://operatex.dev/blog/solo-founder-devops-playbook) (2026-08-02): AI made shipping cheap; operating is still expensive. A triage order, minimum telemetry list, and one-page runbook for solo founders and two-person teams.
- [The Transactional Outbox Pattern on Day 2: Failure Modes the Tutorials Skip](https://operatex.dev/blog/transactional-outbox-pattern-production-guide) (2026-08-01): Every outbox pattern tutorial stops at the happy path. A day-2 guide to relay stalls, outbox lag, table bloat, poison messages, and the metrics that catch them.
- [Closed Is Not Fixed: Why Closure Rate Is the Wrong Incident Metric](https://operatex.dev/blog/incident-closure-rate-vs-recurrence-rate) (2026-07-31): Ticket closure rate rewards silencing symptoms, and automation industrializes it. A replacement scorecard: recurrence rate, time-to-recurrence, verified-fix ratio.
- [Five Strangers, One Architecture: The Emerging Spec for AI in Production Operations](https://operatex.dev/blog/ai-ops-architecture-consensus-five-constraints) (2026-07-30): Six unaffiliated engineers building AI for operations converged on the same five constraints. What that consensus is, why it happened, and how to test vendors against it.
- [What Is p99 Latency? A Field Guide for Teams Whose p50 Looks Fine](https://operatex.dev/blog/p99-latency-field-guide-tail-regressions) (2026-07-29): What p99 latency is, why tails regress without a deploy, and a workflow for finding the cause: histograms, exemplars, per-route views, and SLOs that catch tails.
- [Span of Control Is Five to Seven: What Fire Command Doctrine Actually Says About Running Software Incidents](https://operatex.dev/blog/incident-commander-ics-doctrine-software-sre) (2026-07-28): Software took the incident commander title from fire service ICS and dropped the doctrine. What actually transfers - span of control, transfer of command - and what doesn't.
- [Everyone Is Building an Ops Copilot: Notes From the DIY Wave](https://operatex.dev/blog/everyone-is-building-an-ops-copilot-diy-wave-notes) (2026-07-27): Practitioners are assembling self-hosted LLM ops stacks. What the DIY wave gets right, where it stalls, and a build-vs-buy framework for AI operations tooling.
- [Your Dead Letter Queue Is Where Failures Go to Be Forgotten: An Operating Manual](https://operatex.dev/blog/dead-letter-queue-operating-manual) (2026-07-26): Every broker doc explains configuring a dead letter queue. This is the missing operating manual: metrics, alert thresholds, failure triage, and safe redrives.
- [A Code Yellow Is a Lagging Indicator: The Signals That Fire Months Earlier](https://operatex.dev/blog/code-yellow-engineering-escalation-guide) (2026-07-25): A Code Yellow is the bill for months of undiagnosed degradation. When to declare one, how to run it, and the signals that fire long before you need the protocol.
- [ReadWriteOnce Does Not Mean One Pod: The Misconfig That Waits for a Node Drain](https://operatex.dev/blog/readwriteonce-vs-readwritemany-kubernetes-misconfig) (2026-07-24): ReadWriteOnce limits a volume to one node, not one pod. Why this Kubernetes misconfig passes every test and fails during a node drain, and how to audit for it.
- [The N+1 Query Problem in Production: Why It Ships Silently and How to Catch It](https://operatex.dev/blog/n-plus-1-query-problem-production-detection) (2026-07-23): The N+1 query problem rarely shows up in code review. How it ships silently, how to detect it in production telemetry, and how to fix it without regressions.
- [Green Dashboards, Dark Grid: The 2003 Blackout as a Silent-Failure Postmortem](https://operatex.dev/blog/silent-failures-software-2003-blackout-monitoring) (2026-07-22): A millisecond race condition silently killed the alarms before the 2003 blackout. The same failure mode lives in your cron jobs, queues, and alert pipelines.
- [30-40% of Your Engineering Capacity Is Missing. Here Is the Math.](https://operatex.dev/blog/capacity-math-unplanned-work-engineering) (2026-07-21): Accelerate, Stripe, and real team data on unplanned engineering work, turned into capacity math you can run on your own org in two weeks.
- [Your Provider's Postmortem Is Not Your Postmortem](https://operatex.dev/blog/aws-outage-postmortem-lessons-2025-2026-streak) (2026-07-20): Four major AWS incidents in eight months, one public postmortem. A customer-side postmortem template for turning provider outages into your own reliability review.
- [Stuck state: the failure mode behind both July 2 postmortems](https://operatex.dev/blog/stuck-state-july-2-postmortem-analysis) (2026-07-19): Railway and CircleCI both published July 2, 2026 postmortems with the same hidden failure mode: state captured during instability that never corrected itself.
- [Reduce MTTR by attacking the 80 percent: diagnosis, not repair](https://operatex.dev/blog/reduce-mttr-diagnosis-first-playbook) (2026-07-18): About 80 percent of MTTR is spent finding the cause, not fixing it. A diagnosis-first playbook for reducing MTTR across code, database, infra, logs, and CI.
- [Beating On-Call Alert Fatigue: Triage by Evidence, Not by Threshold-Tuning](https://operatex.dev/blog/on-call-alert-fatigue-solutions-triage-evidence) (2026-07-16): On-call alert fatigue is a triage problem, not a threshold problem. A practical guide to cutting page volume by investigating alerts to root cause across the whole stack.
- [Replay-Verified Is Not Production-Safe: Where the Agentic Database's Autonomy Ladder Breaks](https://operatex.dev/blog/agentic-database-autonomy-verification-risks) (2026-07-16): A response to 'Agentic databases aren't agentic': replaying traffic on a fork proves answer-equivalence, not production safety. Why the reviewable diff, not the autonomy boundary, is the unit of trust.
- [Automating Root Cause Analysis Without Giving AI Write Access to Production](https://operatex.dev/blog/automated-root-cause-analysis-safety-guide) (2026-07-15): A practical, vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production.
- [Knight Capital's 97 Unread Emails: A $460M Lesson for the AI-Agent Era](https://operatex.dev/blog/knight-capital-97-unread-emails-ai-agent-era) (2026-07-15): Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes.
- [Idle in Transaction in Postgres: What It Means, How to Find It, and Why Autovacuum Can't Save You](https://operatex.dev/blog/idle-in-transaction-postgres) (2026-07-15): What idle in transaction means in Postgres, why it silently blocks autovacuum while dashboards stay green, the 60-second diagnosis SQL, and the one-line timeout that prevents it.
- [The flaky test runbook: detect, quarantine, diagnose, verify](https://operatex.dev/blog/flaky-test-incident-runbook) (2026-07-14): A practical runbook that treats flaky tests as incidents: detection thresholds, a five-cause taxonomy, quarantine exit criteria, and the math of verifying a fix.
- [The 21% Problem: Why Second Opinions are Critical for Root Cause Analysis](https://operatex.dev/blog/root-cause-analysis-second-opinion-21-percent-problem) (2026-07-14): When Mayo Clinic re-examined referred diagnoses, 21% changed completely. Root cause analysis has the same failure mode and none of the safeguards.
- [Every Component Is Green and the System Is Down](https://operatex.dev/blog/dashboards-green-but-system-down-failure-class) (2026-07-13): Pod healthy, service healthy, endpoint healthy - and the request times out. Four real incidents where every dashboard was green, and a cross-layer protocol for the failure class between components.
- [When Recovery Doesn’t Recover: Railway, CircleCI, and Stuck State](https://operatex.dev/blog/stuck-state-postmortem-railway-circleci-outage) (2026-07-12): Railway and CircleCI both went down on July 2, 2026. The interesting failures came after recovery: connections and workflows that silently kept bad state.
- [Recovery Is Not Convergence: Anatomy of an Outage That Outlived Its Root Cause](https://operatex.dev/blog/railway-outage-stale-connections-network-recovery) (2026-07-11): Railway's July 2026 outage report shows three systems that captured bad state during 20 minutes of instability and held it silently. Here's the failure class.
- [Every On-Call Model Has an Expiry Headcount: Lessons from Monzo’s Public Record](https://operatex.dev/blog/monzo-on-call-rotation-models-scaling) (2026-07-09): Monzo published three on-call models at three company sizes - with pay figures and honest failure notes. A teardown of the scaling cliffs every org hits.
- [Automating Root Cause Analysis Without Giving AI Write Access to Production](https://operatex.dev/blog/automated-root-cause-analysis-no-ai-write-access) (2026-07-09): Manual RCA eats 30-40% of engineering time. Here's how to automate the investigation across your whole stack without giving AI write access to production.
- [Automated Root Cause Analysis in 2026: A Vendor-Neutral, Whole-Stack Guide for Engineering Leaders](https://operatex.dev/blog/automated-root-cause-analysis-2026-vendor-neutral-guide) (2026-07-09): A vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production.
- [The Buyer's Guide to Automated Root Cause Analysis: What the AI Answers Leave Out](https://operatex.dev/blog/the-buyers-guide-to-automated-root-cause-analysis-what-the-ai-answers-leave-out-7gmv) (2026-07-08): A vendor-neutral buyer's guide to automated root cause analysis: the features to look for, whole-stack coverage, and why the safe default is patch-file-only, no write access.
- [The Monitoring Pilot Never Flies: Aviation's Rule for Verifying AI Diagnoses](https://operatex.dev/blog/verifying-ai-root-cause-analysis-aviation-cross-check) (2026-07-06): Airbus builds redundant flight computers with different chips, teams, and languages. Aviation's independence principle is the standard AI root-cause analysis should be held to.
- [They Added Capacity and It Did Not Help: What WorkOS's Connection-Pool Starvation Says About Hold-and-Wait Failures](https://operatex.dev/blog/workos-database-connection-pool-exhaustion-analysis) (2026-07-04): WorkOS added capacity and nothing improved. Hold-and-wait connection-pool starvation, why it self-sustains, and the audit to run on your own critical path.
- [When to Hire an SRE (According to People Not Selling You One)](https://operatex.dev/blog/when-to-hire-an-sre-framework) (2026-07-04): Everyone ranking for 'when to hire an SRE' is selling the answer. A disinterested framework: the real signals, what must exist before the hire, and the alternatives to try first.
- [Toil Went Up, Not Down: The AI-Ops Paradox in the Survey Data](https://operatex.dev/blog/sre-toil-ai-paradox-2026) (2026-07-03): Survey toil fell for years, then reversed as AI adoption went mainstream. The mechanism: verification burden, new ops surfaces, and AI aimed at the wrong loop.
- [Zero-Downtime Database Migrations: The Playbook and the Failure Modes Nobody Writes Down](https://operatex.dev/blog/zero-downtime-database-migration-guide) (2026-07-02): The expand-contract playbook plus what guides skip: foreign keys, stored procedures, backup mismatches, and how to verify a live migration is safe step by step.
- [AI-Assisted Infra Changes Need a Paper Trail, Not a Chat Log](https://operatex.dev/blog/ai-infrastructure-changes-audit-trail-documentation) (2026-07-01): AI-assisted terraform and pipeline changes ship fast, but the reasoning dies with the chat session. A practical convention for keeping the why in your repo.
- [Retries Are Eating Your Signal: Flaky Tests Are System Bugs Wearing a Costume](https://operatex.dev/blog/flaky-tests-system-nondeterminism-not-noise) (2026-06-27): Google found 84% of CI pass-to-fail transitions are flakes. Retrying hides the race conditions and shared state that later cause production incidents.
- [Nobody Will Cut the Requests: A Blame-Safe Playbook for Rightsizing Kubernetes at 25 Percent Utilization](https://operatex.dev/blog/rightsizing-kubernetes-requests-guide) (2026-06-25): Your cluster runs at 25 percent because nobody wants to own the next latency blip. A rollout playbook for cutting padded Kubernetes requests safely.
- [The Delhi Data Center Fire Is a Blast-Radius Audit You Haven't Run Yet](https://operatex.dev/blog/delhi-data-center-fire-blast-radius-lessons) (2026-06-24): A fire in one Delhi data center destroyed decades of customer data and degraded Google Cloud for weeks. Eight audit questions to ask about your own blast radius.
- [The Pipeline Returned 200 OK and Did Nothing: The Agent Failure Class Your Monitoring Reports as Healthy](https://operatex.dev/blog/silent-failures-ai-agent-pipelines-200-ok) (2026-06-22): Agent pipelines fail while every dashboard stays green because 200 OK measures transport, not work. Five shapes of silent failure and how to detect each.
- [Schema Drift Is an Incident Waiting for a Deploy](https://operatex.dev/blog/database-schema-drift-operational-guide) (2026-06-22): Schema drift turns deploys, failovers, and restores into incidents. How drift happens, what Snowflake's 13-hour outage shows, and how to detect and triage it.
- [Your Average MTTR Is a Fiction: What 1.8 Million Outages Say About the Tail](https://operatex.dev/blog/mttr-median-vs-mean-outage-data-study) (2026-06-22): Fresh data from 1.8 million outages: median resolution 1.9 minutes, mean 21.9. The gap explains why your MTTR dashboard misleads, and what to report instead.
- [Knight Capital's 97 Unread Emails: A $460M Lesson for the AI-Agent Era](https://operatex.dev/blog/knight-capital-incident-lessons-ai-agents) (2026-06-21): Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signals, unverified fixes.
- [Your Observability Bill Is a Hedge Against Bad Investigations](https://operatex.dev/blog/observability-costs-investigation-hedge) (2026-06-21): Every cost guide says collect less telemetry. The real lever is why you store it: an investigation model that can only query pre-collected data. Fix that instead.
- [How to Reduce MTTR: Fix the 80% of the Clock Nobody Instruments](https://operatex.dev/blog/how-to-reduce-mttr-diagnosis-bottleneck) (2026-06-19): Most MTTR guides optimize detection. The clock is lost in diagnosis. A phase-by-phase breakdown of where incident time goes and how to shrink the 80%.
- [Is It Us or Is It AWS? Triage for the First Fifteen Minutes of an Upstream Outage](https://operatex.dev/blog/third-party-outage-triage-protocol-aws-july-2026) (2026-06-19): AWS's July 16 CloudFront outage took 59 minutes to reach the status page. A triage protocol for deciding, in the first fifteen minutes, whether an incident is yours or upstream.
- [The Andon Cord Almost Never Stops the Line: What Software Got Wrong About Toyota's Most Borrowed Idea](https://operatex.dev/blog/andon-cord-software-development-myth) (2026-06-18): Toyota's andon cord rarely stopped the line—it summoned help in seconds. What the real mechanics teach ops teams about escalation, freezes, and cheap signals.

## Legal

- [Privacy policy](https://operatex.dev/privacy)
- [Terms of service](https://operatex.dev/terms)