Evaluation and guardrails: how to test AI features before production
AI features need evaluation as a delivery gate, just like tests: test sets, groundedness checks, safety checks and regression on every change.
Traditional software has a comforting property: the same input produces the same output, so a passing test suite means something. AI features don't behave that way. The same prompt can produce different answers, a model upgrade can change behaviour silently, and a content change can make a previously correct answer wrong.
So AI features need their own form of testing, evaluation, and it has to be a delivery gate, not a one-off exercise before a demo.
Four layers of evaluation
1. Task quality. Does the feature do its job? For extraction, field-level accuracy against labelled documents. For classification, precision and recall per class. For summarization, coverage of required facts. For retrieval, whether the right sources come back.
2. Groundedness. For anything generated from sources, is every claim supported by the retrieved material, and are citations correct? An ungrounded answer is a defect even when it happens to be true.
3. Safety and policy. Does the feature refuse what it should: out-of-scope questions, requests for data the user can't access, instructions hidden in documents (prompt injection)? Does it avoid prohibited content and claims?
4. Regression. Every change to prompts, models, retrieval settings or content is re-evaluated against the same test sets, and the results are compared with the last accepted baseline.
Building the test sets
Good test sets come from the workflow, not from the vendor:
- real questions and documents from the pilot, anonymized where needed
- edge cases the business owner worries about
- known-hard cases collected from production feedback
- adversarial cases: injection attempts, ambiguous requests, missing data
Every case has an expected outcome defined by a person who owns the domain.
Guardrails in the application
Evaluation tells you how the feature behaves. Guardrails constrain it in production:
- Grounding rules: answer only from retrieved, authorized sources, or say you don't know.
- Output validation: structured outputs checked against schemas and business rules before use.
- Allow-lists: an AI can only reference entities that exist. It can't invent a product, a customer or a case number.
- Human checkpoints: consequential outputs are drafts until a person accepts them.
- Untrusted-input handling: document and user content is treated as data, never as instructions.
- Fallbacks: if the model is unavailable or uncertain, the workflow continues deterministically.
Monitoring after launch
In production, keep measuring: acceptance and edit rates on AI drafts, user flags, drift in evaluation scores on a sampled stream, and latency and cost. Those signals feed the next round of test cases.
How the factory handles it
In our architecture, evaluation sits alongside the automated test suite. Every application foundation that includes AI features ships with an evaluation harness, and a Production Sprint doesn't close until the agreed evaluation thresholds are met. It's one of the quality gates we use to decide whether something is done.
Related: from AI pilot to production application and AI model governance as an application.
Have a pilot that's never been evaluated properly? Bring it to us.