In a meeting with the IT leadership of a manufacturing company, the team walked me through an AI strategy they had been developing for close to a year. They had a roadmap, a maturity model, two proofs of concept and support from the board. When I asked what was in production, the answer was nothing yet.
In the months leading up to this article, I had variations of that conversation around fifteen times: in manufacturing in Austria, insurance in Munich and the public sector in Vienna. The teams had done substantial work. They were still trying to establish a route from a promising prototype to something the organization could rely on.
I find that frustrating because the missing work often receives attention late. Someone needs to define what acceptable performance looks like, decide who can authorize actions, provide evidence for a review and prepare the team that will operate the system. Those responsibilities do not resolve themselves when the model improves.
I call this the AI production gap. In the engagements I see, engineering and organizational ownership are often a larger part of it than model capability alone.
Planning for a changing technology
Gartner’s Generative AI Immaturity Model, published in September 2024, describes the difficulty of defining a general maturity model while the technology and its practices are still changing. I recognize that problem in roadmaps that assume a predictable progression from pilot to scale.
A roadmap remains useful, but it needs room for what the team learns. A new model may alter the cost or quality of a workflow. A data-access issue may change its scope. An operational review may expose work that the prototype did not include.
The teams making progress in my experience treat those discoveries as part of delivery. They keep evaluating the system and make deliberate changes to the architecture, rather than wait for a point at which every uncertainty has disappeared.
Specifications leave too much implicit
When an initiative stalls, the first explanation is often that it needs a better model, more data or another round of prompt tuning. Sometimes that is right. I also find gaps between the problem a stakeholder describes and the decisions an implementation has to make.
Experienced engineers often close those gaps through conversations and domain knowledge. An agent can ask questions, but it cannot be relied on to recognize every assumption that needs one. If a missing rule is resolved incorrectly, the mistake can spread through code and tests before someone from the business sees it.
My colleague Manuel Klein discusses this in Why we created the Intent Engineer. Our approach makes the business problem, measurable success criteria, domain rules and constraints explicit enough to guide implementation. We want the team to agree on what success means while leaving room to choose how to achieve it.
That work is particularly useful when someone later needs to explain why a system behaves as it does. The relevant legal and supervisory requirements depend on the sector and use case; DORA, MaRisk and the EU AI Act have different scopes. An intent document can support the evidence trail, but it does not by itself satisfy those obligations.
Technical metrics leave part of the question unanswered
Accuracy, latency and token cost help an engineering team understand a system. The business also needs to know whether it improved the process it was built for.
I want that connection defined early. What is the baseline? Which outcome are we trying to change? How will we distinguish useful automation from work that somebody has to correct afterward?
The evaluation also needs a response. A change in the data may reduce quality. A new model or prompt may handle familiar cases differently. The team needs both the instrumentation to notice and the authority to investigate, restrict or revise the system.
Without that arrangement, people can know something is going wrong and still struggle to get it fixed.
The team receiving the system needs to understand it
This part comes from my time running Corporate IT. I inherited systems built by external teams who understood them well. That understanding did not always transfer with the handover.
We received software that we could not confidently debug, evaluate or modify. We limited what it was allowed to do because we did not know enough to take responsibility for broader use. That was a reasonable operational decision, even though it reduced the value of the system.
I see a similar problem with AI projects. Documentation may describe the components without giving operations a way to investigate an unexpected answer or action. The receiving team needs working examples, relevant evaluation cases, access to the records and a clear support route. They also need time to learn the system before responsibility changes hands.
I would involve them while the workflow is being built. Their questions are part of finding out whether the design is ready for production.
AI arrives through several routes
Gartner’s AI Technology Sandwich describes AI entering enterprises through existing products, platforms and new development. That is a useful way to think about the scope of governance.
A team may be building an internal agent while its SaaS vendors add AI features and other departments buy services independently. Each use can involve different data, terms, costs and permissions. Reviewing the organization’s flagship project covers only part of that activity.
I would maintain a view of the services in use and the people responsible for them. Shared capabilities can help where the needs repeat, but the individual workflow still needs an assessment. A common platform cannot make two different uses equivalent simply because they run through the same infrastructure.
Build the operational capabilities once they recur
Across engagements in banking, insurance, the public sector, health and manufacturing, I have seen teams repeatedly need similar supporting capabilities. Reusing them can reduce work and make the result easier to operate.
An agent framework can provide tool registration, permissions, action limits, escalation and audit events. Its job is to make it clear what an agent can do, on whose authority, and how someone can intervene.
An LLMOps layer can manage prompt and model versions, evaluation runs, cost monitoring, routing, caching and release checks. It should help the team investigate a change in behavior and identify which configuration produced a particular result.
A knowledge layer can provide connectors, retrieval, reranking and evaluation of the sources returned. Access permissions and freshness need to survive the path from source system to model input. These details are easy to overlook in a prototype built around a small, curated document set.
My starting point is to deploy within the customer’s infrastructure and work with its governance requirements. That still requires checking model calls, connectors and other external services. Hosting the application in the customer’s environment does not, on its own, establish where every piece of data is processed.
I would build these shared capabilities around concrete needs. The return depends on how much the organization reuses them and what they cost to maintain. A platform should make the next use case easier to deliver without becoming a prerequisite for every experiment.
Testing delivery with a smaller team
The delivery model I have been developing and testing is the Focused Impact Pair: an Intent Engineer and a Systems Engineer working with AI agents.
The Intent Engineer works with the domain experts to describe the problem, constraints and success criteria. The Systems Engineer makes architectural decisions, coordinates agent work and takes responsibility for technical quality. Agents can contribute code, tests and documentation, including longer implementation tasks.
We tested the model in an internal weekend hackathon, with pairs building AI engineering platforms. The results encouraged me to keep working on the approach. Clear intent and suitable tools allowed small teams to make substantial progress.
In client work, I have also seen time saved in implementation disappear into repeated corrections when the specification was weak. Defining the problem well has become more valuable as generating code has become faster.
The pair still needs access to the relevant domain, security and operational expertise. Those responsibilities remain part of delivery, even when fewer people are writing the implementation.
Make the benefit visible to the people funding it
I have watched promising AI programs lose funding because the team could not demonstrate the benefit in terms finance could assess. The system was doing useful work, but the evidence was missing or disconnected from the business case.
That is why I want evaluation included from the start. If a project aims to reduce processing time, establish the baseline and measure the resulting workflow, including review and correction. If it is meant to improve quality, agree on how that quality will be assessed.
I described a related approach in my post on governable agents: evaluate retrieval, rule application and explanation separately. That makes it easier to identify the source of a failure and to show which parts of the process are dependable.
A business owner and the finance team should be able to question the benefit calculation. The operational team should be able to investigate the system’s behavior. Both conversations need evidence, though they may use different views of it.
What I would do before the next production review
I would start with the people who know the process and the team expected to run the system. Agree on a useful first scope, the conditions for release and who can make the remaining decisions.
Then I would check whether the supporting work has an owner: evaluation, permissions, monitoring, cost controls, data access and support. Where the organization already has a suitable capability, use it. Where it does not, make the work visible in the plan.
Finally, I would rehearse an ordinary failure. Take an incorrect recommendation or a failed action and ask the receiving team to investigate it. Can they find the inputs and configuration? Can they correct the issue or contain it? Do they know who to call?
That exercise often tells me more about readiness than another demonstration of the successful path. It also gives the team a concrete list of work to complete before people depend on the system.
Sources and further reading
- Gartner, A Generative AI Immaturity Model (September 2024).
- Gartner, AI Technology Sandwich: A Conceptual Framework for Executing AI (October 2024).
- Gartner, A Journey Guide to Realizing Value from Data, Analytics and AI (November 2024).
- Manuel Klein, Why we created the Intent Engineer.
