[{"data":1,"prerenderedAt":72},["ShallowReactive",2],{"blog-category-ai-in-production":3},[4,24,34,48,61],{"id":5,"slug":6,"body":7,"html":8,"title":9,"description":10,"category":11,"tags":12,"author":17,"date":18,"year":19,"month":20,"quarter":21,"status":22,"featured":23},"2026\u002F07\u002Fai-in-production\u002Fdocument-intelligence-in-regulated-workflows","document-intelligence-in-regulated-workflows","\nRegulated workflows run on documents: identity documents, company registries, financial statements, invoices, contracts, permits, medical referrals, supplier certificates, audit reports. Extracting data from them is among the most valuable uses of AI, and among the easiest to get subtly wrong.\n\nA demo extracts ten fields from a clean PDF perfectly. Production brings scans, photos, handwriting, multiple languages, unusual layouts and documents that are simply the wrong document.\n\n## The production pattern\n\n**1. Classify first.** Before extracting anything, determine what the document is. A bank statement sent where a trade licence was expected should be caught at the door.\n\n**2. Extract to a schema.** Every document type has a defined schema of fields, types and formats. The model's output is validated against it, and anything that doesn't conform is rejected.\n\n**3. Validate against rules and sources.** Cross-check extracted values: totals that should add up, dates that should be in order, registration numbers that should exist in a registry, names that should match the application.\n\n**4. Carry confidence and provenance.** Every extracted field records where it came from on the page and how confident the extraction is. Reviewers see the source next to the value.\n\n**5. Route by confidence and risk.** High-confidence, low-risk fields flow straight through. Low-confidence or high-risk fields go to a human review queue. The thresholds are business decisions, not model defaults.\n\n**6. Learn from corrections.** Every human correction is recorded and becomes evaluation data for the next model or prompt change.\n\n## Where it appears across the Atlas\n\nDocument intelligence isn't a product on its own. It's a capability inside many application families:\n\n- **Onboarding and KYC\u002FKYB:** identity and company documents\n- **Case management:** evidence submitted by applicants ([AI-assisted case management](\u002Fblog\u002Fai-assisted-case-management))\n- **Referrals and pre-authorization** in healthcare ([care coordination](\u002Fblog\u002Freferrals-and-care-coordination))\n- **Permits** in the built environment ([permitting and inspections](\u002Fblog\u002Fpermitting-and-inspections-for-the-built-environment))\n- **Supplier assurance:** SOC reports and certificates ([third-party risk](\u002Fblog\u002Fthird-party-and-supplier-risk-reviews))\n- **Finance:** remittances and statements ([reconciliation](\u002Fblog\u002Freconciliation-and-exception-workbenches))\n\n## Controls designed in\n\n- Original documents retained, unaltered, with hashes\n- Extracted values linked to their source location\n- Every human override recorded, with the reviewer and reason\n- Access to sensitive documents restricted by role and logged\n- The model provider and hosting chosen to meet data-residency requirements\n\n## Measuring it honestly\n\nField-level accuracy on a held-out test set per document type, straight-through processing rate, review queue volume and correction rate. Agree the thresholds with the business and compliance owners before launch. See [evaluation and guardrails](\u002Fblog\u002Fevaluation-and-guardrails-before-production).\n\n## Arabic and bilingual documents\n\nIn the GCC, many documents are Arabic, English or both, and include stamps, signatures and handwriting. Test sets must reflect that mix from day one. Performance on English samples says little about performance on the documents you'll actually receive.\n\n[Bring us the document types](\u002Fcontact) that slow your workflow down.\n","\u003Cp>Regulated workflows run on documents: identity documents, company registries, financial statements, invoices, contracts, permits, medical referrals, supplier certificates, audit reports. Extracting data from them is among the most valuable uses of AI, and among the easiest to get subtly wrong.\u003C\u002Fp>\n\u003Cp>A demo extracts ten fields from a clean PDF perfectly. Production brings scans, photos, handwriting, multiple languages, unusual layouts and documents that are simply the wrong document.\u003C\u002Fp>\n\u003Ch2>The production pattern\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>1. Classify first.\u003C\u002Fstrong> Before extracting anything, determine what the document is. A bank statement sent where a trade licence was expected should be caught at the door.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>2. Extract to a schema.\u003C\u002Fstrong> Every document type has a defined schema of fields, types and formats. The model's output is validated against it, and anything that doesn't conform is rejected.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>3. Validate against rules and sources.\u003C\u002Fstrong> Cross-check extracted values: totals that should add up, dates that should be in order, registration numbers that should exist in a registry, names that should match the application.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>4. Carry confidence and provenance.\u003C\u002Fstrong> Every extracted field records where it came from on the page and how confident the extraction is. Reviewers see the source next to the value.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>5. Route by confidence and risk.\u003C\u002Fstrong> High-confidence, low-risk fields flow straight through. Low-confidence or high-risk fields go to a human review queue. The thresholds are business decisions, not model defaults.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>6. Learn from corrections.\u003C\u002Fstrong> Every human correction is recorded and becomes evaluation data for the next model or prompt change.\u003C\u002Fp>\n\u003Ch2>Where it appears across the Atlas\u003C\u002Fh2>\n\u003Cp>Document intelligence isn't a product on its own. It's a capability inside many application families:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Onboarding and KYC\u002FKYB:\u003C\u002Fstrong> identity and company documents\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Case management:\u003C\u002Fstrong> evidence submitted by applicants (\u003Ca href=\"\u002Fblog\u002Fai-assisted-case-management\">AI-assisted case management\u003C\u002Fa>)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Referrals and pre-authorization\u003C\u002Fstrong> in healthcare (\u003Ca href=\"\u002Fblog\u002Freferrals-and-care-coordination\">care coordination\u003C\u002Fa>)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Permits\u003C\u002Fstrong> in the built environment (\u003Ca href=\"\u002Fblog\u002Fpermitting-and-inspections-for-the-built-environment\">permitting and inspections\u003C\u002Fa>)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Supplier assurance:\u003C\u002Fstrong> SOC reports and certificates (\u003Ca href=\"\u002Fblog\u002Fthird-party-and-supplier-risk-reviews\">third-party risk\u003C\u002Fa>)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Finance:\u003C\u002Fstrong> remittances and statements (\u003Ca href=\"\u002Fblog\u002Freconciliation-and-exception-workbenches\">reconciliation\u003C\u002Fa>)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Controls designed in\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>Original documents retained, unaltered, with hashes\u003C\u002Fli>\n\u003Cli>Extracted values linked to their source location\u003C\u002Fli>\n\u003Cli>Every human override recorded, with the reviewer and reason\u003C\u002Fli>\n\u003Cli>Access to sensitive documents restricted by role and logged\u003C\u002Fli>\n\u003Cli>The model provider and hosting chosen to meet data-residency requirements\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Measuring it honestly\u003C\u002Fh2>\n\u003Cp>Field-level accuracy on a held-out test set per document type, straight-through processing rate, review queue volume and correction rate. Agree the thresholds with the business and compliance owners before launch. See \u003Ca href=\"\u002Fblog\u002Fevaluation-and-guardrails-before-production\">evaluation and guardrails\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2>Arabic and bilingual documents\u003C\u002Fh2>\n\u003Cp>In the GCC, many documents are Arabic, English or both, and include stamps, signatures and handwriting. Test sets must reflect that mix from day one. Performance on English samples says little about performance on the documents you'll actually receive.\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"\u002Fcontact\">Bring us the document types\u003C\u002Fa> that slow your workflow down.\u003C\u002Fp>\n","Document intelligence in regulated workflows: extraction with verification","Extracting data from documents with AI is easy to demo and hard to trust. How to build extraction with validation, confidence and human review.","ai-in-production",[13,14,15,16],"document-intelligence","human-in-the-loop","evidence","production","fazezero-editorial","2026-07-23T00:00:00.000Z",2026,7,3,"published",false,{"id":25,"slug":26,"body":27,"html":28,"title":29,"description":30,"category":11,"tags":31,"author":17,"date":33,"year":19,"month":20,"quarter":21,"status":22,"featured":23},"2026\u002F07\u002Fai-in-production\u002Fagentic-automation-with-human-checkpoints","agentic-automation-with-human-checkpoints","\nAgents, meaning AI systems that plan and take multi-step actions with tools, are the most exciting and the most dangerous AI capability in the enterprise. An agent that gathers context from five systems before an analyst opens a case saves real time. An agent that closes accounts, moves money or emails customers on its own is a governance incident waiting to happen.\n\nThe answer isn't to avoid agents. It's to put them inside a workflow with **checkpoints**.\n\n## Design principles\n\n**1. Bounded tools.** An agent can only call tools that the application explicitly exposes to it, each with a narrow purpose and validated inputs. No general shell, no arbitrary API access.\n\n**2. Read before write.** Most value comes from read-only work: gathering context, correlating records, drafting. Make read-only the default and treat every write as a separate, higher-risk capability.\n\n**3. Explicit approval for consequential actions.** Anything that changes a record of consequence, contacts a customer, moves value or changes access requires a person to approve. Some actions require two people.\n\n**4. Identity and least privilege.** The agent acts with its own service identity or on behalf of a user, never with broader permissions than the user who invoked it.\n\n**5. Deterministic workflow state.** The workflow engine, not the model, decides what state a case is in and what happens next. The agent proposes, and the workflow disposes.\n\n**6. Untrusted input.** Content the agent reads (emails, documents, alerts, web pages) is data. Instructions embedded in it are ignored, and attempts are logged.\n\n**7. Full traceability.** Every plan, tool call, input, output, approval and rejection is recorded, so reviewers can reconstruct why something happened.\n\n## Where agents earn their keep\n\n- **Case preparation:** assemble customer, transaction and history context before a human opens the case. See [AI-assisted case management](\u002Fblog\u002Fai-assisted-case-management).\n- **Security enrichment:** read-only lookups across security tools. See [AI in the SOC](\u002Fblog\u002Fai-in-the-soc-triage-and-investigation).\n- **Document workflows:** extract, validate and route documents, and escalate what fails validation.\n- **Operations recovery:** generate and score recovery options for a controller to choose from. See [operations control](\u002Fblog\u002Foperations-control-and-disruption-management).\n- **Reconciliation:** propose matches and classify breaks for an analyst to confirm.\n\nIn each case, the agent compresses the time *before* a human decision. It doesn't replace the decision.\n\n## What to measure\n\nTime saved before the decision point, how often agent proposals are accepted unchanged, the rejection reasons, how often approval gates fire, and incidents caused by agent actions. That last number should be zero, and the design should make it hard to be anything else.\n\n## How it fits the architecture\n\nIn our application foundations, agent tools are ordinary application services with contracts, authorization and tests. That's the same discipline as any other API. This is the practical meaning of [AI accelerates the implementation, architecture governs it](\u002Fblog\u002Fai-accelerates-architecture-governs).\n\n[Bring us a workflow](\u002Fcontact) where an agent could prepare the decision, and we'll scope the checkpoints with you.\n","\u003Cp>Agents, meaning AI systems that plan and take multi-step actions with tools, are the most exciting and the most dangerous AI capability in the enterprise. An agent that gathers context from five systems before an analyst opens a case saves real time. An agent that closes accounts, moves money or emails customers on its own is a governance incident waiting to happen.\u003C\u002Fp>\n\u003Cp>The answer isn't to avoid agents. It's to put them inside a workflow with \u003Cstrong>checkpoints\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Ch2>Design principles\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>1. Bounded tools.\u003C\u002Fstrong> An agent can only call tools that the application explicitly exposes to it, each with a narrow purpose and validated inputs. No general shell, no arbitrary API access.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>2. Read before write.\u003C\u002Fstrong> Most value comes from read-only work: gathering context, correlating records, drafting. Make read-only the default and treat every write as a separate, higher-risk capability.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>3. Explicit approval for consequential actions.\u003C\u002Fstrong> Anything that changes a record of consequence, contacts a customer, moves value or changes access requires a person to approve. Some actions require two people.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>4. Identity and least privilege.\u003C\u002Fstrong> The agent acts with its own service identity or on behalf of a user, never with broader permissions than the user who invoked it.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>5. Deterministic workflow state.\u003C\u002Fstrong> The workflow engine, not the model, decides what state a case is in and what happens next. The agent proposes, and the workflow disposes.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>6. Untrusted input.\u003C\u002Fstrong> Content the agent reads (emails, documents, alerts, web pages) is data. Instructions embedded in it are ignored, and attempts are logged.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>7. Full traceability.\u003C\u002Fstrong> Every plan, tool call, input, output, approval and rejection is recorded, so reviewers can reconstruct why something happened.\u003C\u002Fp>\n\u003Ch2>Where agents earn their keep\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Cstrong>Case preparation:\u003C\u002Fstrong> assemble customer, transaction and history context before a human opens the case. See \u003Ca href=\"\u002Fblog\u002Fai-assisted-case-management\">AI-assisted case management\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Security enrichment:\u003C\u002Fstrong> read-only lookups across security tools. See \u003Ca href=\"\u002Fblog\u002Fai-in-the-soc-triage-and-investigation\">AI in the SOC\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Document workflows:\u003C\u002Fstrong> extract, validate and route documents, and escalate what fails validation.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Operations recovery:\u003C\u002Fstrong> generate and score recovery options for a controller to choose from. See \u003Ca href=\"\u002Fblog\u002Foperations-control-and-disruption-management\">operations control\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Reconciliation:\u003C\u002Fstrong> propose matches and classify breaks for an analyst to confirm.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>In each case, the agent compresses the time \u003Cem>before\u003C\u002Fem> a human decision. It doesn't replace the decision.\u003C\u002Fp>\n\u003Ch2>What to measure\u003C\u002Fh2>\n\u003Cp>Time saved before the decision point, how often agent proposals are accepted unchanged, the rejection reasons, how often approval gates fire, and incidents caused by agent actions. That last number should be zero, and the design should make it hard to be anything else.\u003C\u002Fp>\n\u003Ch2>How it fits the architecture\u003C\u002Fh2>\n\u003Cp>In our application foundations, agent tools are ordinary application services with contracts, authorization and tests. That's the same discipline as any other API. This is the practical meaning of \u003Ca href=\"\u002Fblog\u002Fai-accelerates-architecture-governs\">AI accelerates the implementation, architecture governs it\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"\u002Fcontact\">Bring us a workflow\u003C\u002Fa> where an agent could prepare the decision, and we'll scope the checkpoints with you.\u003C\u002Fp>\n","Agentic automation with human checkpoints","How to use AI agents in enterprise workflows safely: bounded tools, read-before-write, explicit approvals and an audit trail of every step.",[32,14,16,15],"agents","2026-07-14T00:00:00.000Z",{"id":35,"slug":36,"body":37,"html":38,"title":39,"description":40,"category":11,"tags":41,"author":17,"date":45,"year":19,"month":20,"quarter":21,"status":22,"featured":23,"series":46,"seriesOrder":47},"2026\u002F07\u002Fai-in-production\u002Fdeployment-ready-is-not-production","deployment-ready-is-not-production","\n“Production-ready” is one of the most abused phrases in enterprise software. It usually means “the demo worked.” We avoid it on purpose.\n\nWe describe every application in one of three stages. Each stage describes what has actually happened, not what we hope will happen.\n\n## 1. Deployment-Ready Application Foundation\n\nThis is where every application in our inventory starts. A foundation has:\n\n- been generated and scaffolded on the factory architecture\n- a runnable core with domain services and adapters\n- an API structure defined by contracts\n- a web application\n- identity and authorization patterns\n- automated tests\n- a deployment baseline\n- a known, consistent architectural structure\n\nA foundation is real software, not a mock-up. But it hasn't met your data, your identity provider, your integrations or your policies. Calling it “production” would be misleading.\n\n## 2. Customer-Configured Application\n\nThe foundation has been adapted to one organization:\n\n- the customer's requirements and business workflows\n- their data and data platforms\n- integrations with their systems of record\n- AI and model choices, including providers, hosting and evaluation criteria\n- their identity environment\n- their cloud platform and regional constraints\n- their policies and approval rules\n- their operating environment\n\nThis is what an [AI Production Sprint](\u002Fservices\u002Fai-production-sprint) delivers. It is a working application on a production path, but it isn't in production yet.\n\n## 3. Production Deployment\n\nThe application has completed the customer-specific work that production actually requires:\n\n- integration testing against live systems\n- security hardening and review\n- cloud deployment in the customer's environment\n- operational testing\n- customer acceptance\n- observability and alerting\n- a support design\n- production controls\n\nOnly then do we call it production.\n\n## Why the distinction matters\n\n**For buyers**, it sets honest expectations. A foundation shortens the path to production. It doesn't remove the path. Integration, hardening and acceptance still take real effort, and a vendor who claims otherwise is either underestimating your environment or overestimating their template.\n\n**For risk and security teams**, it gives a shared vocabulary. A deployment-ready foundation can be reviewed architecturally. A customer-configured application can be reviewed against your policies. A production deployment has passed your gates.\n\n**For partners**, it prevents over-selling. A systems integrator who promises a “production-ready AI app in two weeks” is setting up the relationship to fail.\n\n## How the stages map to engagements\n\n| Stage | How you get there |\n|---|---|\n| Deployment-ready foundation | Already in the inventory, or generated by the factory |\n| Customer-configured | [Solution Definition Sprint](\u002Fservices\u002Fsolution-definition-sprint), then [AI Production Sprint](\u002Fservices\u002Fai-production-sprint) |\n| Production deployment | Completion of the production roadmap, often inside an [Application Family Program](\u002Fservices\u002Fapplication-family-program) |\n\n## What we won't do\n\nWe won't label a listing “production” because it runs. We won't describe a customer-configured application as a production deployment before acceptance. And we won't publish customer names, metrics or badges we can't evidence.\n\nIt's a small discipline. It also happens to be the one enterprise buyers trust most.\n\nRead more about [the factory](\u002Ffactory), or [bring us a use case](\u002Fcontact).\n","\u003Cp>“Production-ready” is one of the most abused phrases in enterprise software. It usually means “the demo worked.” We avoid it on purpose.\u003C\u002Fp>\n\u003Cp>We describe every application in one of three stages. Each stage describes what has actually happened, not what we hope will happen.\u003C\u002Fp>\n\u003Ch2>1. Deployment-Ready Application Foundation\u003C\u002Fh2>\n\u003Cp>This is where every application in our inventory starts. A foundation has:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>been generated and scaffolded on the factory architecture\u003C\u002Fli>\n\u003Cli>a runnable core with domain services and adapters\u003C\u002Fli>\n\u003Cli>an API structure defined by contracts\u003C\u002Fli>\n\u003Cli>a web application\u003C\u002Fli>\n\u003Cli>identity and authorization patterns\u003C\u002Fli>\n\u003Cli>automated tests\u003C\u002Fli>\n\u003Cli>a deployment baseline\u003C\u002Fli>\n\u003Cli>a known, consistent architectural structure\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>A foundation is real software, not a mock-up. But it hasn't met your data, your identity provider, your integrations or your policies. Calling it “production” would be misleading.\u003C\u002Fp>\n\u003Ch2>2. Customer-Configured Application\u003C\u002Fh2>\n\u003Cp>The foundation has been adapted to one organization:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>the customer's requirements and business workflows\u003C\u002Fli>\n\u003Cli>their data and data platforms\u003C\u002Fli>\n\u003Cli>integrations with their systems of record\u003C\u002Fli>\n\u003Cli>AI and model choices, including providers, hosting and evaluation criteria\u003C\u002Fli>\n\u003Cli>their identity environment\u003C\u002Fli>\n\u003Cli>their cloud platform and regional constraints\u003C\u002Fli>\n\u003Cli>their policies and approval rules\u003C\u002Fli>\n\u003Cli>their operating environment\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>This is what an \u003Ca href=\"\u002Fservices\u002Fai-production-sprint\">AI Production Sprint\u003C\u002Fa> delivers. It is a working application on a production path, but it isn't in production yet.\u003C\u002Fp>\n\u003Ch2>3. Production Deployment\u003C\u002Fh2>\n\u003Cp>The application has completed the customer-specific work that production actually requires:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>integration testing against live systems\u003C\u002Fli>\n\u003Cli>security hardening and review\u003C\u002Fli>\n\u003Cli>cloud deployment in the customer's environment\u003C\u002Fli>\n\u003Cli>operational testing\u003C\u002Fli>\n\u003Cli>customer acceptance\u003C\u002Fli>\n\u003Cli>observability and alerting\u003C\u002Fli>\n\u003Cli>a support design\u003C\u002Fli>\n\u003Cli>production controls\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Only then do we call it production.\u003C\u002Fp>\n\u003Ch2>Why the distinction matters\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>For buyers\u003C\u002Fstrong>, it sets honest expectations. A foundation shortens the path to production. It doesn't remove the path. Integration, hardening and acceptance still take real effort, and a vendor who claims otherwise is either underestimating your environment or overestimating their template.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>For risk and security teams\u003C\u002Fstrong>, it gives a shared vocabulary. A deployment-ready foundation can be reviewed architecturally. A customer-configured application can be reviewed against your policies. A production deployment has passed your gates.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>For partners\u003C\u002Fstrong>, it prevents over-selling. A systems integrator who promises a “production-ready AI app in two weeks” is setting up the relationship to fail.\u003C\u002Fp>\n\u003Ch2>How the stages map to engagements\u003C\u002Fh2>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Stage\u003C\u002Fth>\n\u003Cth>How you get there\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Deployment-ready foundation\u003C\u002Ftd>\n\u003Ctd>Already in the inventory, or generated by the factory\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Customer-configured\u003C\u002Ftd>\n\u003Ctd>\u003Ca href=\"\u002Fservices\u002Fsolution-definition-sprint\">Solution Definition Sprint\u003C\u002Fa>, then \u003Ca href=\"\u002Fservices\u002Fai-production-sprint\">AI Production Sprint\u003C\u002Fa>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Production deployment\u003C\u002Ftd>\n\u003Ctd>Completion of the production roadmap, often inside an \u003Ca href=\"\u002Fservices\u002Fapplication-family-program\">Application Family Program\u003C\u002Fa>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Ch2>What we won't do\u003C\u002Fh2>\n\u003Cp>We won't label a listing “production” because it runs. We won't describe a customer-configured application as a production deployment before acceptance. And we won't publish customer names, metrics or badges we can't evidence.\u003C\u002Fp>\n\u003Cp>It's a small discipline. It also happens to be the one enterprise buyers trust most.\u003C\u002Fp>\n\u003Cp>Read more about \u003Ca href=\"\u002Ffactory\">the factory\u003C\u002Fa>, or \u003Ca href=\"\u002Fcontact\">bring us a use case\u003C\u002Fa>.\u003C\u002Fp>\n","Deployment-ready is not production: an honest maturity model","Why we describe applications in three stages (deployment-ready foundation, customer-configured, production deployment) and never call a template 'production'.",[16,42,43,44],"application-factory","implementation","governance","2026-07-07T00:00:00.000Z","the-application-factory",5,{"id":49,"slug":50,"body":51,"html":52,"title":53,"description":54,"category":11,"tags":55,"author":17,"date":58,"year":19,"month":59,"quarter":60,"status":22,"featured":23},"2026\u002F06\u002Fai-in-production\u002Fevaluation-and-guardrails-before-production","evaluation-and-guardrails-before-production","\nTraditional software has a comforting property: the same input produces the same output, so a passing test suite means something. AI features don't behave that way. The same prompt can produce different answers, a model upgrade can change behaviour silently, and a content change can make a previously correct answer wrong.\n\nSo AI features need their own form of testing, **evaluation**, and it has to be a delivery gate, not a one-off exercise before a demo.\n\n## Four layers of evaluation\n\n**1. Task quality.** Does the feature do its job? For extraction, field-level accuracy against labelled documents. For classification, precision and recall per class. For summarization, coverage of required facts. For retrieval, whether the right sources come back.\n\n**2. Groundedness.** For anything generated from sources, is every claim supported by the retrieved material, and are citations correct? An ungrounded answer is a defect even when it happens to be true.\n\n**3. Safety and policy.** Does the feature refuse what it should: out-of-scope questions, requests for data the user can't access, instructions hidden in documents (prompt injection)? Does it avoid prohibited content and claims?\n\n**4. Regression.** Every change to prompts, models, retrieval settings or content is re-evaluated against the same test sets, and the results are compared with the last accepted baseline.\n\n## Building the test sets\n\nGood test sets come from the workflow, not from the vendor:\n\n- real questions and documents from the pilot, anonymized where needed\n- edge cases the business owner worries about\n- known-hard cases collected from production feedback\n- adversarial cases: injection attempts, ambiguous requests, missing data\n\nEvery case has an expected outcome defined by a person who owns the domain.\n\n## Guardrails in the application\n\nEvaluation tells you how the feature behaves. Guardrails constrain it in production:\n\n- **Grounding rules:** answer only from retrieved, authorized sources, or say you don't know.\n- **Output validation:** structured outputs checked against schemas and business rules before use.\n- **Allow-lists:** an AI can only reference entities that exist. It can't invent a product, a customer or a case number.\n- **Human checkpoints:** consequential outputs are drafts until a person accepts them.\n- **Untrusted-input handling:** document and user content is treated as data, never as instructions.\n- **Fallbacks:** if the model is unavailable or uncertain, the workflow continues deterministically.\n\n## Monitoring after launch\n\nIn production, keep measuring: acceptance and edit rates on AI drafts, user flags, drift in evaluation scores on a sampled stream, and latency and cost. Those signals feed the next round of test cases.\n\n## How the factory handles it\n\nIn our architecture, evaluation sits alongside the automated test suite. Every application foundation that includes AI features ships with an evaluation harness, and a Production Sprint doesn't close until the agreed evaluation thresholds are met. It's one of the [quality gates](\u002Fservices\u002Fai-production-sprint) we use to decide whether something is done.\n\nRelated: [from AI pilot to production application](\u002Fblog\u002Ffrom-ai-pilot-to-production-application) and [AI model governance as an application](\u002Fblog\u002Fai-model-governance-as-an-application).\n\nHave a pilot that's never been evaluated properly? [Bring it to us](\u002Fcontact).\n","\u003Cp>Traditional software has a comforting property: the same input produces the same output, so a passing test suite means something. AI features don't behave that way. The same prompt can produce different answers, a model upgrade can change behaviour silently, and a content change can make a previously correct answer wrong.\u003C\u002Fp>\n\u003Cp>So AI features need their own form of testing, \u003Cstrong>evaluation\u003C\u002Fstrong>, and it has to be a delivery gate, not a one-off exercise before a demo.\u003C\u002Fp>\n\u003Ch2>Four layers of evaluation\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>1. Task quality.\u003C\u002Fstrong> Does the feature do its job? For extraction, field-level accuracy against labelled documents. For classification, precision and recall per class. For summarization, coverage of required facts. For retrieval, whether the right sources come back.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>2. Groundedness.\u003C\u002Fstrong> For anything generated from sources, is every claim supported by the retrieved material, and are citations correct? An ungrounded answer is a defect even when it happens to be true.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>3. Safety and policy.\u003C\u002Fstrong> Does the feature refuse what it should: out-of-scope questions, requests for data the user can't access, instructions hidden in documents (prompt injection)? Does it avoid prohibited content and claims?\u003C\u002Fp>\n\u003Cp>\u003Cstrong>4. Regression.\u003C\u002Fstrong> Every change to prompts, models, retrieval settings or content is re-evaluated against the same test sets, and the results are compared with the last accepted baseline.\u003C\u002Fp>\n\u003Ch2>Building the test sets\u003C\u002Fh2>\n\u003Cp>Good test sets come from the workflow, not from the vendor:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>real questions and documents from the pilot, anonymized where needed\u003C\u002Fli>\n\u003Cli>edge cases the business owner worries about\u003C\u002Fli>\n\u003Cli>known-hard cases collected from production feedback\u003C\u002Fli>\n\u003Cli>adversarial cases: injection attempts, ambiguous requests, missing data\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Every case has an expected outcome defined by a person who owns the domain.\u003C\u002Fp>\n\u003Ch2>Guardrails in the application\u003C\u002Fh2>\n\u003Cp>Evaluation tells you how the feature behaves. Guardrails constrain it in production:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Grounding rules:\u003C\u002Fstrong> answer only from retrieved, authorized sources, or say you don't know.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Output validation:\u003C\u002Fstrong> structured outputs checked against schemas and business rules before use.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Allow-lists:\u003C\u002Fstrong> an AI can only reference entities that exist. It can't invent a product, a customer or a case number.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Human checkpoints:\u003C\u002Fstrong> consequential outputs are drafts until a person accepts them.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Untrusted-input handling:\u003C\u002Fstrong> document and user content is treated as data, never as instructions.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Fallbacks:\u003C\u002Fstrong> if the model is unavailable or uncertain, the workflow continues deterministically.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Monitoring after launch\u003C\u002Fh2>\n\u003Cp>In production, keep measuring: acceptance and edit rates on AI drafts, user flags, drift in evaluation scores on a sampled stream, and latency and cost. Those signals feed the next round of test cases.\u003C\u002Fp>\n\u003Ch2>How the factory handles it\u003C\u002Fh2>\n\u003Cp>In our architecture, evaluation sits alongside the automated test suite. Every application foundation that includes AI features ships with an evaluation harness, and a Production Sprint doesn't close until the agreed evaluation thresholds are met. It's one of the \u003Ca href=\"\u002Fservices\u002Fai-production-sprint\">quality gates\u003C\u002Fa> we use to decide whether something is done.\u003C\u002Fp>\n\u003Cp>Related: \u003Ca href=\"\u002Fblog\u002Ffrom-ai-pilot-to-production-application\">from AI pilot to production application\u003C\u002Fa> and \u003Ca href=\"\u002Fblog\u002Fai-model-governance-as-an-application\">AI model governance as an application\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>Have a pilot that's never been evaluated properly? \u003Ca href=\"\u002Fcontact\">Bring it to us\u003C\u002Fa>.\u003C\u002Fp>\n","Evaluation and guardrails: how to test AI features before production","AI features need evaluation as a delivery gate, just like tests: test sets, groundedness checks, safety checks and regression on every change.",[56,16,57,14],"evaluation","ai-governance","2026-06-30T00:00:00.000Z",6,2,{"id":62,"slug":63,"body":64,"html":65,"title":66,"description":67,"category":11,"tags":68,"author":17,"date":70,"year":19,"month":59,"quarter":60,"status":22,"featured":23,"series":46,"seriesOrder":71},"2026\u002F06\u002Fai-in-production\u002Ffrom-ai-pilot-to-production-application","from-ai-pilot-to-production-application","\nThere is a question we ask early in almost every conversation:\n\n> **How many AI use cases have you identified or prototyped that are not yet operating as production applications?**\n\nThe answer is rarely zero. Often it's a backlog: copilots that impressed a steering committee, agents that worked on sample data, proofs of concept that never got a production owner. The models were fine. Everything *around* the models was missing.\n\n## What a pilot proves, and what it doesn't\n\nA pilot proves that a model can do a task on representative data in a controlled setting. That is worth knowing. It does **not** prove that:\n\n- real users can reach it through enterprise identity with the right permissions\n- it integrates with the systems of record the workflow depends on\n- someone owns the output and the decisions made with it\n- failures, exceptions and edge cases have somewhere to go\n- quality is measured continuously, not once\n- the organization can audit what happened last Tuesday\n- it can be deployed, monitored and supported in the customer's cloud\n\nThose are the things production is made of.\n\n## Seven changes between pilot and production\n\n**1. From a notebook to an application.** Production AI lives inside a workflow application with a domain model, APIs and a user interface, not in a script or a chat window.\n\n**2. From shared keys to enterprise identity.** Users authenticate through the organization's identity provider. Authorization decides who can see which data and approve which actions. Multi-tenant deployments isolate data by design.\n\n**3. From sample data to integrations.** The application reads and writes through adapters to core systems, document stores and data platforms, with error handling, retries and idempotency.\n\n**4. From a demo to a human-accountable workflow.** AI drafts, summarizes, classifies and recommends. People approve, decide and remain accountable, with clear checkpoints in the workflow.\n\n**5. From one-off testing to evaluation as a gate.** Automated tests cover the application. AI evaluations cover the model behaviour, and both run on every change. See [evaluation and guardrails](\u002Fblog\u002Fevaluation-and-guardrails-before-production).\n\n**6. From “it worked” to evidence.** Every significant action produces an audit event. Controls produce evidence as a by-product of the work.\n\n**7. From a laptop to a deployment baseline.** Compute, API gateway, data, secrets, identity, AI services, observability and CI\u002FCD are defined for the target cloud.\n\n## Why most pilots stall at step two\n\nPilots are usually built to answer a capability question quickly, which is the right way to run a pilot. The trouble starts when the pilot code becomes the starting point for production. Identity, integration and controls then have to be retrofitted into a structure that never expected them, and that is where timelines collapse.\n\nWe take the opposite route. The pilot's *learning* carries forward: the prompts, the evaluation data, the workflow insight. The pilot's *code* usually doesn't. The production application starts from a deployment-ready foundation that already has steps 1, 2, 5, 6 and 7 built in, so the engineering effort goes into integration and the customer-specific workflow.\n\n## A practical path\n\n1. Pick the pilot with a named business owner and a workflow that runs weekly.\n2. Map it to the closest application foundation in the [Atlas](\u002Fatlas).\n3. Define the delta in a [Solution Definition Sprint](\u002Fservices\u002Fsolution-definition-sprint): integrations, identity, controls, evaluations and data.\n4. Build it in an [AI Production Sprint](\u002Fservices\u002Fai-production-sprint).\n5. Once it is running, add the adjacent workflows through an [Application Family Program](\u002Fservices\u002Fapplication-family-program).\n\nIf you have a backlog of pilots, [bring us the one that matters most](\u002Fcontact).\n","\u003Cp>There is a question we ask early in almost every conversation:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>How many AI use cases have you identified or prototyped that are not yet operating as production applications?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>The answer is rarely zero. Often it's a backlog: copilots that impressed a steering committee, agents that worked on sample data, proofs of concept that never got a production owner. The models were fine. Everything \u003Cem>around\u003C\u002Fem> the models was missing.\u003C\u002Fp>\n\u003Ch2>What a pilot proves, and what it doesn't\u003C\u002Fh2>\n\u003Cp>A pilot proves that a model can do a task on representative data in a controlled setting. That is worth knowing. It does \u003Cstrong>not\u003C\u002Fstrong> prove that:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>real users can reach it through enterprise identity with the right permissions\u003C\u002Fli>\n\u003Cli>it integrates with the systems of record the workflow depends on\u003C\u002Fli>\n\u003Cli>someone owns the output and the decisions made with it\u003C\u002Fli>\n\u003Cli>failures, exceptions and edge cases have somewhere to go\u003C\u002Fli>\n\u003Cli>quality is measured continuously, not once\u003C\u002Fli>\n\u003Cli>the organization can audit what happened last Tuesday\u003C\u002Fli>\n\u003Cli>it can be deployed, monitored and supported in the customer's cloud\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Those are the things production is made of.\u003C\u002Fp>\n\u003Ch2>Seven changes between pilot and production\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>1. From a notebook to an application.\u003C\u002Fstrong> Production AI lives inside a workflow application with a domain model, APIs and a user interface, not in a script or a chat window.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>2. From shared keys to enterprise identity.\u003C\u002Fstrong> Users authenticate through the organization's identity provider. Authorization decides who can see which data and approve which actions. Multi-tenant deployments isolate data by design.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>3. From sample data to integrations.\u003C\u002Fstrong> The application reads and writes through adapters to core systems, document stores and data platforms, with error handling, retries and idempotency.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>4. From a demo to a human-accountable workflow.\u003C\u002Fstrong> AI drafts, summarizes, classifies and recommends. People approve, decide and remain accountable, with clear checkpoints in the workflow.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>5. From one-off testing to evaluation as a gate.\u003C\u002Fstrong> Automated tests cover the application. AI evaluations cover the model behaviour, and both run on every change. See \u003Ca href=\"\u002Fblog\u002Fevaluation-and-guardrails-before-production\">evaluation and guardrails\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>6. From “it worked” to evidence.\u003C\u002Fstrong> Every significant action produces an audit event. Controls produce evidence as a by-product of the work.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>7. From a laptop to a deployment baseline.\u003C\u002Fstrong> Compute, API gateway, data, secrets, identity, AI services, observability and CI\u002FCD are defined for the target cloud.\u003C\u002Fp>\n\u003Ch2>Why most pilots stall at step two\u003C\u002Fh2>\n\u003Cp>Pilots are usually built to answer a capability question quickly, which is the right way to run a pilot. The trouble starts when the pilot code becomes the starting point for production. Identity, integration and controls then have to be retrofitted into a structure that never expected them, and that is where timelines collapse.\u003C\u002Fp>\n\u003Cp>We take the opposite route. The pilot's \u003Cem>learning\u003C\u002Fem> carries forward: the prompts, the evaluation data, the workflow insight. The pilot's \u003Cem>code\u003C\u002Fem> usually doesn't. The production application starts from a deployment-ready foundation that already has steps 1, 2, 5, 6 and 7 built in, so the engineering effort goes into integration and the customer-specific workflow.\u003C\u002Fp>\n\u003Ch2>A practical path\u003C\u002Fh2>\n\u003Col>\n\u003Cli>Pick the pilot with a named business owner and a workflow that runs weekly.\u003C\u002Fli>\n\u003Cli>Map it to the closest application foundation in the \u003Ca href=\"\u002Fatlas\">Atlas\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Define the delta in a \u003Ca href=\"\u002Fservices\u002Fsolution-definition-sprint\">Solution Definition Sprint\u003C\u002Fa>: integrations, identity, controls, evaluations and data.\u003C\u002Fli>\n\u003Cli>Build it in an \u003Ca href=\"\u002Fservices\u002Fai-production-sprint\">AI Production Sprint\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Once it is running, add the adjacent workflows through an \u003Ca href=\"\u002Fservices\u002Fapplication-family-program\">Application Family Program\u003C\u002Fa>.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>If you have a backlog of pilots, \u003Ca href=\"\u002Fcontact\">bring us the one that matters most\u003C\u002Fa>.\u003C\u002Fp>\n","From AI pilot to production application: what actually changes","Most enterprise AI pilots never reach production. The gap is not the model. It is identity, integration, controls, evaluation and ownership.",[16,43,69,56],"enterprise","2026-06-25T00:00:00.000Z",4,1790080512622]