What actually changes after launch
Four things move under a live system. Model providers deprecate versions and ship new ones that behave differently on your prompts. Your own content changes, so retrieval starts returning documents that were not there at handover. Usage shifts as more of the organization finds the system, and the questions get harder than the ones it was tested on. And the business itself changes: a new product, a new regulation, a new definition of the right answer. None of this announces itself. It shows up as answers that are slightly worse than they used to be.
Evaluations decide what ships
Every system we build carries an evaluation suite: a graded set of real cases with known good answers, owned by your team as much as by ours. In managed operations that suite becomes the control. A model upgrade, a prompt revision, a retrieval change or a new data source is proposed, run against the suite, then compared with the version currently in production. If it does not hold, it does not ship. You see the comparison itself rather than a summary of it, so accepting a change is your decision and it is made on evidence.
Monitoring, and what happens when something breaks
We watch the system inside your own observability stack, not a console only we can see: failure rates, latency trends, retrieval quality, cost of usage as volume grows, and the specific ways this system is known to fail. When something breaks, the path is defined before it is needed. A named engineer picks it up, you are told what is affected and what we are doing about it, and the fix is followed by a written account of the cause and the change that prevents a repeat.
A standing block of senior engineering time
Each period carries an agreed allocation of senior engineering time, and you direct where it goes. Part of it is upkeep that simply has to happen: model migrations, dependency and security updates, new evaluation cases for failure modes discovered in real use. The rest is yours to spend on improvement, whether that is a workflow the system does not cover yet, an integration your team keeps asking for, or work to make an expensive path cheaper to run. The engineers doing it are the ones who built the system, so nothing is relearned at your expense.
Support that is part software and part people
Some systems need somebody watching them, not only something. Where that is true we run the support layer as two halves that belong together. AI agents handle the volume, working the same tools and the same records a person would, and our engineers sit behind them to take the escalations, judge the answers the agents were not confident about, and feed what they learn back into the evaluations so the next month is better than the last. Neither half works alone. Agents without people accumulate quiet errors nobody is accountable for, and people without agents are a cost that grows in a straight line with your traffic. What we will not do is put a support layer around a system whose correct behaviour has never been written down, because the escalations then have nothing to be judged against.
The roadmap review, and staying able to leave
On a cadence set in your contract, a founding partner sits down with your team and works through the same three questions: what the evaluations show, what has changed underneath the system since the last review, and what is worth doing next. It is a working session, not a status deck. That review is also where you can decide to take the system back in house. Runbooks, documentation and training are kept current for exactly that reason, so ending a managed operations engagement is a decision rather than a project.
