Model-To-System Gap
The model-to-system gap is the distance between a capable model and a reliable product experience.
Use it when a demo works, but the production question is still unclear: what context is assembled, what tools are allowed, what state is remembered, what costs are acceptable, and what proof exists that the workflow works for the intended job?
Rubric
| Layer | Question | Evidence to look for |
|---|---|---|
| Context | What does the model see, and what is deliberately excluded? | Source policy, retrieval evals, context order tests. |
| Tools | What can the system do, and where are approvals required? | Permission map, tool logs, rollback behavior. |
| Memory | What persists across sessions, and how is it corrected? | Memory schema, user controls, audit trail. |
| Reliability | What failure modes are expected and measured? | Evals, incident notes, fallback paths. |
| Cost | What latency and spend budgets shape the design? | Budget envelope, model routing, capacity assumptions. |
| Proof | Which claims are backed by visible evidence? | Claims table, proof inventory, caveats. |
Operator takeaway
Do not judge an AI product by model access alone. Judge the operating system around the model.