Support, MLOps and monitoring
AI in production needs ongoing monitoring: response times, costs, errors, answer quality, drift and model changes. We monitor, alert and fix so your agents and search engines stay reliable over time.
Updated October 11, 2026
The big picture
of agentic AI projects will be canceled by the end of 2027, according to Gartner: rising costs, unclear value, poorly controlled risks (forecast).
Source: Gartner, June 25, 2025
of notice before OpenAI retires a generally available model. Models change, and your solutions need to keep up.
Source: OpenAI, deprecation policy
minimum log retention for deployers of high-risk AI systems, along with monitoring of their operation (AI Act, Article 26).
Source: AI Act, Article 26
What we monitor
Language models often fail silently: an answer can arrive with no technical error and still be wrong. That's why we track both the technical side and quality.
Typical response time and spikes, time to first token.
Pitfall: An average hides the slowdowns on difficult cases.
Input and output tokens consumed, cost per session or per user.
Pitfall: A poorly tuned retry loop can multiply the bill.
Provider outages and wrong answers with no error message.
Pitfall: A technically successful call can still contain a made-up answer.
Faithfulness to sources, answer relevance and safety, tracked over time.
Pitfall: Scores are assessed by a model, so we track the trend.
Changes in incoming questions, answers and retrieved documents.
Pitfall: Quality declines gradually and goes unnoticed without measurement.
For a search engine over your documents, we also check that the right passages are retrieved. Sources: OpenTelemetry · LangWatch · Splunk.
The monitoring loop
Every call is traced: model, duration, tokens, errors.
An alert fires when a threshold is crossed or a trend worsens.
The trace shows which step is at fault: model, tool, retry, retrieval.
Adjusting the prompt, the documents, the model or the agent's access rights.
A fixed set of test questions confirms there is no regression.
Drift and model changes
Incoming questions and documents evolve: what the solution sees today differs from what it saw at launch.
A business rule changes (new procedure, new pricing): the same questions now call for different answers.
OpenAI commits to at least 6 months' notice for generally available models, and much less for preview versions. With some providers, the endpoint stays the same while the model behind it changes. We pin the version in use, test the replacement on a fixed set of questions before any migration, and alert on discrepancies.
Sources: OpenAI · Databricks · Deepchecks.
Controlling API costs
Examples taken from Anthropic's documentation, shown relative to the base price. Pricing changes: we check the provider's rate card at the time of the project.
Baseline
−50% for non-urgent tasks
0.1× the base price (down to 0.025× depending on the model)
Instructions and documents repeated from one request to the next are cached: reading them again costs about ten times less. The cache lasts 5 minutes by default and is extended each time it is used.
Deferred workloads (analyses, evaluations, reports) run in batches: half price, and most batches finish in under an hour. The discounts stack.
Simple requests go to a smaller model, difficult ones to a more powerful model. We measure the savings on your cases, because they vary.
Sources: Anthropic, prompt caching · Anthropic, batch processing · Anthropic, Building effective agents.
Traceability and compliance
For high-risk AI systems, the AI Act requires deployers to monitor how the system operates and to keep its logs for at least six months, subject to other applicable rules, particularly on personal data. Well-designed monitoring provides these records without collecting more than necessary.
General information, not legal advice. Source: AI Act, Article 26.
Our approach
Trace every call: model, duration, tokens, errors.
Define alerts with you on latency, costs, errors and quality.
A representative set of questions to track quality and catch regressions.
A periodic review of quality, costs and usage with your teams.
New versions of models, documents or instructions, without regressions.
FAQ
It is the continuous tracking of a solution in production: response times, costs, errors, answer quality and drift. It lets you catch a problem before your teams or customers feel it.
Because a model can respond without any technical error and still give a wrong answer. So quality has to be measured too: faithfulness to sources, relevance, safety.
OpenAI commits to at least 6 months' notice for generally available models. We pin the version in use, test the replacement model on a fixed set of questions, and only migrate if quality holds.
By caching repeated instructions, batching non-urgent tasks, choosing the right model for each task and setting spending caps. At Anthropic, cache reads cost 0.1 times the base price and batch processing 50% less.
No. They can be reduced by grounding answers in your documents and detected through regular evaluations. The rate depends on the model, the task and the language: we measure it on your own cases.
Yes, provided you define what is kept and for how long. The AI Act requires at least six months of logs for certain high-risk systems, subject to other rules such as personal data protection.
Go further
Tell us about your solution in production: we'll identify what needs monitoring and how.