Build vs Buy AI for Operations After the DIY Copilot Wave

The build vs buy AI question for operations is the choice between assembling an internal copilot for incident work and adopting a commercial tool. It usually comes up after someone has already wired a self-hosted model to the logs and shown it answering a real question.

The DIY instinct is sound. It comes from real constraints around data, permissions and trust, and those constraints make a good requirements list for any tool, built or bought.

In this post we cover why engineers build their own, what the builders get right, where internal copilots stall, a framework for deciding, when building is clearly the right call and what to do next.

Why Engineers Build Their Own Ops Copilot

The first driver is data. Logs and database rows routinely contain customer data, and many teams cannot send them to a third-party model without a legal review that takes longer than the project.

A self-hosted open-source model removes that blocker on day one. It also lets the team feed the model internal runbooks, service ownership and architecture notes that no general product knows about.

The second driver is trust. When a tool suggests restarting a database, an engineer wants the log line or query plan behind the suggestion, and a black box that only returns a conclusion does not get used twice.

• The Permissions Instinct Is Correct

Builders almost always start read-only, and we think that instinct is right. The OWASP guidance on excessive agency describes the risk as "damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM", and traces it to excessive functionality, permissions and autonomy.

An internal copilot that can only read logs, query a replica and read the repository cannot take a damaging action, however wrong its answer. That property is worth keeping whatever you decide.

What the DIY Builders Get Right

Well-built internal copilots share three habits that many commercial tools still lack. Each one is worth writing into a requirements document before any vendor conversation.

• They keep data inside the network, by hosting the model next to the systems it reads or by choosing the provider deliberately.
• They hold read-only credentials and leave every action to a human, so the copilot advises and engineers decide.
• They show evidence, quoting the log line, metric or code path behind each claim so a reviewer can check it.

The same OWASP entry recommends that teams "Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken." Internal builders tend to arrive at that rule by instinct, because they are the ones paged when the tool is wrong.

We set out the same principles as five guardrails in the rules for AI agents in production, and the deployment-shape side in self-hosted versus cloud AI agents.

Four Places Internal Ops Copilots Stall

The first demo is easy. The problems arrive in the second quarter, when the copilot has users and the people who built it have other work.

1. Verification Is Harder Than Generation

Most internal copilots ask the same model to check its own answer. Research by Huang and colleagues at ICLR 2024, Large Language Models Cannot Self-Correct Reasoning Yet, found that "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction."

An independent check needs a different model or a deterministic rule set, reading the original evidence rather than the first answer. We explain how to build that check in verifying AI root cause analysis the way pilots cross-check, and it is usually the piece a side project never finishes.

2. The Copilot Becomes a Production Service

Once people rely on it, the copilot needs monitoring, upgrades, model changes, credential rotation and an owner on call. The Google SRE book defines toil as work that is "manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows", and an under-staffed internal tool generates exactly that kind of work.

That maintenance comes out of the same capacity the copilot was meant to free. We look at how unplanned work eats a roadmap in our capacity math for unplanned work.

3. Knowledge Leaves With the Author

Internal copilots are often one engineer's project, with prompts, adapters and evaluation sets that live in their head. When that person changes team, the tool stays running and stops improving.

A copilot nobody understands is worse than none, because people keep trusting answers that nobody can explain or fix. We would treat a named second owner as a precondition for rolling one out widely.

4. Coverage Grows Slower Than the Estate

Every new data source needs an adapter, credentials, a permission review and tests. A team that started with one log source and one repository often finds the copilot blind to the database or the deploy history behind the next incident.

That gap is invisible until the incident that needed the missing source. We count adapters as part of the build cost, not as later polish.

A Build vs Buy AI Framework for Operations Tooling

We use the same five requirements for both paths. The question is not which path is cheaper on paper, but which one meets the bar with the staff you actually have.

Requirement	Building means	Buying requires
Data location	Host the model in your network, or pick a provider and document the data flow	Self-hosted deployment and a clear statement of what reaches the vendor
Permissions	Read-only credentials for every source you connect	Read-only adapters, with no commit, merge or deploy rights
Verification	A second model or rule set that reads the raw evidence	An independent check you can inspect, with its limits stated
Auditability	A log of every query and every answer, kept and searchable	The same log, exportable to your own tooling
Ownership	A named team, an on-call rotation and an upgrade plan	A vendor who maintains adapters as your stack changes

Score each row honestly for both paths. If buying cannot meet the first four rows, keep building. If building cannot meet the last row, the copilot will decay, however good the first version was.

We also price the decision in engineer time rather than licence cost. A copilot that needs a quarter of one engineer forever is a real cost even when the software is free.

The data row deserves the most care, because it is where the two paths differ most in practice. A self-hosted LLM keeps investigation context in your network, while an external provider receives whatever context each question needs. Many commercial AI SRE tools now let you bring your own model, so the honest comparison is your internal copilot against a product running the same model you would have chosen.

The verification row is the one teams most often score too generously. A checker that reads the first model's answer instead of the raw evidence will agree with it for the same wrong reason, so we ask to see a case where the check rejected an answer before we give the row a pass.

• Questions to Ask Any Vendor

Before a trial, we ask four things. Where does the context an investigation needs actually go, and can we keep it in our network?

What exactly can the tool change in production, and how is that enforced? How is each answer checked, and what does the tool say when it is unsure?

The last question is the evidence trail. A useful answer comes with links to the log lines, queries and code behind it, which is the standard we hold an AI agent that investigates with evidence to as well.

When Building Your Own Copilot Is the Right Call

Building is the better choice in some cases, and we say so plainly. If no vendor can run inside your network and your data cannot leave it, there is nothing to buy.

It is also right when the copilot's value is mostly your own operational knowledge, such as a domain-specific runbook system or a check only your business understands. A small team with a narrow scope and a committed owner can keep such a tool healthy for years.

The warning sign is scope creep. A copilot that began as a log summariser and now wants adapters for every system, a verification layer and a chat bot for three teams has become a product, and products need product staffing.

Settle the Copilot Decision With One Real Incident

Take one past incident your team already understands, run it through the internal copilot and through any tool you are considering, and compare the evidence each one cites. Then score both against the five rows above, including who will own the result next year.

We think that test settles more build vs buy AI debates than any feature list. Operate runs inside your own infrastructure through read-only adapters, and a separate agent on a different model checks each proposed root cause against the evidence.