Most agentic AI demos work brilliantly until they hit production. We learned this the hard way running goal-driven agents inside Civil Defense, police forces, and chemical plants. The failure modes are predictable. The fixes are unglamorous. Here's what three years of deployment taught us.
The first failure: tool hallucination
When an LLM picks a tool that doesn't exist, you don't find out until production. Schema-strict tool registries prevent this — every tool has a typed signature, and the planner can only emit calls that match. We rejected pure function-calling APIs in favor of typed action grammars that the planner generates against, not in spite of.
Lesson
If your agent framework lets the LLM invent tool names, you don't have a framework. You have a chatbot with extra steps.
The second failure: policy drift
Approval policies that live in prompt text decay. The model summarizes them, paraphrases them, eventually forgets them. We moved every policy into structural guardrails — code that runs after the model proposes an action and before the system executes it. The model can't bypass what isn't in its execution path.
The third failure: multi-agent gridlock
When two agents share a workspace, they fight. We watched a dispatch agent and a maintenance agent argue over the same vehicle for 14 minutes in a staging environment. The fix isn't smarter coordination — it's explicit lock acquisition in the object graph. Whoever holds the lock wins. Conflicts surface as events, not as silent races.
The fourth failure: silent regression
Agents drift. Last quarter's "good enough" model becomes this quarter's liability. We run shadow agents on every production decision — the live agent acts, the shadow agent predicts. When divergence crosses threshold, the shadow gets promoted or the live agent gets retrained.
A production agent isn't a model. It's a model, a typed action grammar, structural policy, lock-protected state, and shadow evaluation — together. Anything less and you're running a demo.
What we won't do anymore
- Prompt-only policy enforcement.
- Unbounded tool registries.
- Shared mutable state without locks.
- Trusting demo performance.
None of this is novel computer science. All of it is the difference between an agent that works in a demo and an agent that runs Dubai Police's critical-asset chain-of-custody.