Practice 04 / Applied AI

Applied AI systems.

We put language models into production behind measurements you can trust. Retrieval, serving, evaluation, and guardrails, built so you can prove the system is correct before it faces a user. When a task cannot be made measurable and safe, we say so.

What this practice is

AI you can measure, not a demo you have to trust.

Most AI features look convincing in a notebook and fall apart under real inputs. We treat a language model like any other dependency in a production system: it has a latency budget, an error rate, a blast radius, and a rollback plan. Before anything reaches users, we build the evaluation harness that tells us whether it actually works, and we keep running it after launch.

The work spans four connected areas: retrieval systems that ground answers in your own data, model serving that holds its latency under concurrency, evaluation and guardrails that catch bad output, and the ML platform underneath that makes all of it repeatable. We build the ones you need and leave out the ones you do not.

Retrieval / RAG Triton Evaluation Guardrails Vector search MLOps Observability

Typical targets we hold

Eval coverage400+ graded cases
p95 inferenceUnder 300 ms
Guardrail checksInput and output
GroundingCited or refused
Human reviewDefined checkpoints
Regression gateBlocks on eval drop
RollbackSingle command
Shipped behind measurable guardrails

What we build

Four building blocks, assembled to fit.

Each block stands on its own and connects to the others. We start from the outcome you need and work backward to the smallest system that delivers it under measurement.

01 / Retrieval

Retrieval and RAG systems

Answers grounded in your own documents, tickets, and records instead of the model's training data. We build the ingestion, chunking, and embedding pipeline, tune hybrid retrieval (dense vectors plus keyword), and add reranking so the right passages reach the prompt. Every answer carries citations, and the system refuses when it cannot find support.

  • Grounded generationCitations on every claim, with a refusal path when retrieval comes up empty.
  • Hybrid retrievalVector plus keyword search with reranking, tuned against a labelled query set.
  • Fresh indexesIncremental re-embedding so the index tracks source changes, not last quarter's snapshot.
02 / Serving

Model serving and inference infra

The layer that turns a model into a service with a latency budget. We serve on Triton with dynamic batching, run models on GPU or CPU depending on cost and load, and put a queue in front so a traffic spike degrades gracefully instead of timing out. Autoscaling is tied to real queue depth, and every request is traced end to end.

  • Batched inferenceDynamic and continuous batching on Triton to lift throughput without wrecking tail latency.
  • Right-sized computeQuantized and distilled models where accuracy allows, to cut GPU spend.
  • BackpressureQueues, timeouts, and load shedding so overload is bounded, not a cascade.
03 / Evaluation

Evaluation, guardrails, and observability

The part most teams skip and later regret. We build a graded evaluation set from your real cases, wire it into CI so a change that drops quality cannot merge, and run input and output guardrails at request time to block prompt injection, unsafe content, and unsupported claims. In production we log inputs, retrieved context, and outputs so every answer can be audited.

  • Graded eval setsLabelled cases with pass thresholds, run on every change and on a schedule against live traffic.
  • Runtime guardrailsInjection detection, content policy, and grounding checks on input and output.
  • Full traceabilityPrompt, context, and response logged so any answer can be reconstructed and reviewed.
04 / Platform

ML platform and MLOps

The plumbing that makes AI repeatable rather than a one-off heroics project. Versioned datasets and prompts, a model registry, reproducible training and fine-tuning jobs, and deployment through the same review gates as the rest of your code. When a model or prompt changes, you can see exactly what changed, why, and how the evaluation numbers moved.

  • Versioned everythingDatasets, prompts, and models tracked in a registry with lineage from data to deploy.
  • Reproducible jobsFine-tuning and batch inference as pipelines you can rerun and audit, not notebooks.
  • Gated rolloutCanary and shadow traffic with automatic rollback on an evaluation or latency regression.

How a model reaches production

From a promising idea to a gated release.

01

Frame the task and the failure modes

We define what a correct answer looks like, what a wrong one costs, and where a human has to stay in the loop. If the task cannot be graded, we stop here and tell you why, before you spend a budget on it.

02

Build the eval before the feature

We assemble a graded set from your real cases and set pass thresholds. This is the ruler we measure every later change against, so improvements are visible and regressions are caught in CI, not in front of users.

03

Serve, guard, and observe

We put the model behind Triton with batching and backpressure, wrap it in input and output guardrails, and instrument it so p95 latency, refusal rate, and eval scores are on a dashboard from day one.

04

Roll out behind a gate

Shadow traffic first, then a small canary, with automatic rollback if the eval score or latency budget slips. Once it holds, we widen the rollout and hand you the runbook, or run it for you under an SLO.

Under the hood

Evaluation and serving as configuration.

An illustrative serving and evaluation config. The eval gate is not a suggestion: if graded accuracy drops below the threshold or p95 latency blows the budget, the release is blocked and the previous version stays live. Guardrails run on every request, not only in tests.

  • Declared budgetsLatency and accuracy thresholds live in version control next to the code.
  • Fails closedA failed guardrail returns a safe refusal, never an unchecked answer.
serving.eval.yaml
# triage-assistant, retrieval + eval gate
serving:
  runtime: triton
  dynamic_batching: true
  max_batch_size: 32
  latency_budget_p95_ms: 300
retrieval:
  mode: hybrid # dense + keyword
  rerank: true
  require_citation: true
guardrails:
  input: [injection, pii]
  output: [grounding, policy]
  on_fail: refuse
eval_gate:
  suite: triage_v3 # 412 graded cases
  min_accuracy: 0.92
  block_on_regression: true
> eval triage_v3 passed 0.94 > 0.92, gate open

Where we say no

We ship AI where it is measurable and safe.

Not every problem is an AI problem, and not every AI feature belongs in production. If a task has no gradable notion of correct, if the cost of a wrong answer is high and cannot be contained by review, or if a smaller deterministic system would do the job better, we will tell you plainly. We would rather lose the line item than ship something we cannot measure and you cannot trust.

Measurable

If we cannot grade it, we do not ship it. Every feature ships with an eval set and a threshold.

Safe

Guardrails on input and output, a refusal path, and a human checkpoint wherever the stakes require one.

Honest

When a simpler system beats a model, we build the simpler system and say so up front.

Related work

This is what it looks like in production.

A retrieval assistant we built and evaluated behind measurable guardrails, running against real claims every day.

Health / Retrieval

Retrieval assistant for claims triage

Verta Health handled a growing backlog of claims by hand. We built a retrieval assistant that pulls the relevant policy clauses and prior decisions for each claim, drafts a triage recommendation with citations, and refuses when the evidence is thin. Every draft is graded against a labelled set, and a reviewer signs off before anything is actioned.

71%Faster triage
100%Cited or refused

What made it work

  • Evidence, then draftRetrieval grounds every recommendation in the policy and prior decisions it cites.
  • Reviewer in the loopA human signs off, and their corrections feed back into the eval set.
  • Measured continuouslyAccuracy is graded against real claims, not a fixed test written once.
The assistant never guesses. It shows the clauses it relied on, and it flags what it is unsure about. That is the only reason our reviewers trust it.
CV
Clara Voss
Product Director, Verta Health

Questions we get

Honest answers on the hard parts.

We build a graded set from your real cases, with each case labelled by someone who knows the domain, and a clear pass threshold. That suite runs in CI on every change and on a schedule against live traffic, so a prompt tweak or a model swap that drops quality is caught before it reaches users. We track accuracy, refusal rate, and latency together, because a system that is accurate but too slow, or fast but wrong, is still broken. As reviewers correct outputs in production, those corrections feed back into the eval set, so the ruler gets sharper over time.

You cannot fully stop a model from inventing things, so we design around it. Answers are grounded in retrieved evidence and must cite the source they used, an output guardrail checks that the claim is actually supported by that source, and if nothing supports it the system refuses rather than guessing. For anything where a wrong answer carries real cost, a human reviews before the output is actioned. The goal is not a model that is always right, it is a system where an unsupported answer cannot reach the user unchecked.

Both, chosen on the constraints rather than fashion. Hosted API models get you to a working system fastest and are often the right call for lower-volume or exploratory work. We move to self-hosted models on Triton when data residency, cost at high volume, tail latency, or the need to fine-tune on your own data make it worth the operational weight. We keep the serving interface stable so you can switch providers or models without rewriting the application, and we benchmark both against the same eval set so the decision is made on numbers.

Your data stays under your control. For sensitive workloads we run self-hosted models inside your own environment, so prompts and documents never leave your boundary. Where a hosted API is appropriate, we use providers with no-training and retention terms that fit your obligations, strip or tokenize sensitive fields before they leave, and document exactly what is sent and stored. Retrieval indexes, logs, and traces live where you decide, with access controls and retention policies set to your rules, not defaults. See our privacy policy for how we handle information during an engagement.

Start

Have an AI feature that has to be right?

Tell us the task and where a wrong answer would hurt. We will tell you honestly whether it can be made measurable and safe, and how we would build it if it can.