Build an evaluation set from your own historical cases and score it in CI on every change. It tells you when a model, prompt or index change made answers worse. Tracing with OpenTelemetry into a self-hosted Langfuse and Grafana shows what happened on each live request. Both run inside your environment, because prompts, retrieved passages and answers are the data that the private deployment exists to protect.
Why the tooling stays inside your boundary
A trace of a retrieval augmented generation (RAG) request holds the question, the passages that the system retrieved and the answer. In a bank or an insurer, those passages contain customer records. The OpenTelemetry GenAI conventions warn that the output messages attribute “is likely to contain sensitive information including user/PII data”. A hosted tracing service receives that content and stores it outside your environment. For personal information, that raises Privacy Act questions, and APP 8 sets out the steps an entity must take before it discloses personal information overseas. The sovereignty and residency article covers that test. The rest of this article assumes that the evaluation code, the trace store and the dashboards all run on your infrastructure.
An evaluation set from historical cases
Start with work that a person already finished and signed off. Examples are closed service tickets, answered policy queries, claims files with a recorded decision and past audit queries. Each case records:
- The question, as a user typed it
- The documents and pages that the person used to answer it
- The reference answer that the person gave
- The role of the user, for permission tests
- Tags for document type, business unit and difficulty
Add two kinds of negative case. The first kind has no answer in the corpus, and the system must say so. The second kind is a permission case, where the test user must get no chunks from a restricted document. The RAG reference stack explains why that test matters.
The evaluation set is a copy of production data. APP 11 applies to it like any other store of personal information. Keep it in the same environment, restrict who can read it, and de-identify the cases that do not need real names.
Size the set with arithmetic. With n cases and a pass rate p, the standard error is the square root of p x (1 - p) / n. Assume p = 0.9. At n = 200 cases, the standard error is the square root of 0.09 / 200, which is 0.021. The 95% interval is about ±4.2 points. At n = 800, it falls to 0.011, or about ±2.1 points. A 2 point drop is not visible at 200 cases. Compare the old and new version on the same cases, and count the cases that went from pass to fail. The paired count removes the case-to-case variation that two separate averages carry.
Metrics for retrieval and generation
Score retrieval and generation separately. A wrong answer from good passages is a model or prompt problem. A wrong answer from bad passages is a parsing, chunking or search problem.
| Metric | What it measures | What it needs | How it is computed |
|---|---|---|---|
| Recall@k | Whether a reference chunk is in the top k results | Reference chunk IDs | Deterministic code |
| Context precision | Whether relevant chunks rank above irrelevant ones | Question, retrieved chunks, reference | LLM judge |
| Context recall | How many claims in the reference answer the retrieved chunks support | A reference answer | LLM judge |
| Faithfulness | How many claims in the answer the retrieved chunks support | Answer and retrieved chunks | LLM judge |
| Answer relevancy | Whether the answer addresses the question | Question and answer | LLM judge plus embeddings |
| Citation validity | Whether every cited chunk ID was in the retrieved set | Answer and retrieved IDs | Deterministic code |
| Permission leakage | Whether any restricted chunk reached the model | Test user, ACLs, retrieved IDs | Deterministic code |
The Ragas definitions are specific. Faithfulness “measures how factually consistent a response is with the retrieved context”. It is the number of supported claims divided by the total number of claims, from 0 to 1. Context precision “evaluates the retriever’s ability to rank relevant chunks higher than irrelevant ones”. Context recall “always requires a reference to compare against”. Answer relevancy works “without evaluating factual accuracy”, so a fluent wrong answer can score well, and it must not be the only generation metric.
The LLM-judged metrics need a judge model. Run the judge on your own vLLM endpoint so that the evaluation data stays inside. Ragas builds its judge from an OpenAI client object (docs). vLLM documents the same client pointed at a local base_url (docs):
from openai import OpenAI
from ragas.llms import llm_factory
client = OpenAI(base_url="http://vllm-judge.internal:8000/v1", api_key=JUDGE_KEY)
judge = llm_factory("judge-model-name", client=client)A judge is a model, and it makes mistakes. Score 50 cases by hand, compare the hand scores with the judge scores, and keep the comparison with the release record.
Regression gates in CI
Every change that can move answers goes through the same gate. That includes a new model or model revision, a prompt template, the chunker, the embedding model and the index configuration. Use pytest, whose built-in parametrize decorator runs one test function per case.
Split the gates into two kinds. Hard gates fail the build on any single failure: permission leakage, invalid JSON, a citation to a chunk that was not retrieved, or an answer to a case that has no answer. Soft gates compare the new version with the stored baseline: for example, fail the build if more than a set number of cases flip from pass to fail on faithfulness.
The CI runner needs the corpus, the vector index and a GPU. A GitHub-hosted runner has none of them. A self-hosted runner “can be physical, virtual, in a container, on-premises, or in a cloud”, so it can sit inside your network. The host must be able to make outbound HTTPS connections over port 443 to GitHub. Workflow logs appear in GitHub, so the tests print case IDs and scores only, never case content, and upload no artefacts that hold case text.
jobs:
rag-eval:
runs-on: [self-hosted, linux, x64, gpu]
steps:
- uses: actions/checkout@v7
- run: pytest tests/eval --junitxml=eval-results.xmlWrite every run to a results table in PostgreSQL or MLflow, keyed by commit, model revision and index version. The audit trail for a release is that table.
Tracing with OpenTelemetry
The OpenTelemetry semantic conventions for generative AI give standard names to spans and attributes. As of October 2026, they have status “Development” and live in a separate repository. Names can still change, so pin the version that your instrumentation uses.
One request becomes one trace. It holds a retrieval span, a rerank span, a chat span for the generation call and a span for output validation. The convention names a span {gen_ai.operation.name} {gen_ai.request.model}. Attributes include gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.retrieval.top_k.
Content attributes such as gen_ai.input.messages, gen_ai.output.messages and gen_ai.retrieval.documents are opt-in. The conventions list three patterns. The default is to record no instructions, inputs or outputs. The second records content on spans, for cases where “privacy regulations do not apply or the telemetry storage complies with them, for example, in pre-production environments”. The third stores content externally and records references. The spec recommends this pattern “in production environments where telemetry volume is a concern or sensitive data needs to be handled securely”. External storage “enables separate access controls”. For a regulated business, use the third pattern in production.
Self-hosting Langfuse
Langfuse is an open source tracing and evaluation tool for LLM applications. Its self-hosting guide lists two application containers (web and worker) and four storage components: PostgreSQL, ClickHouse, Redis or Valkey, and S3 or blob storage. The guide describes Docker Compose as a “Single VM without high availability, scaling, or backups” and lists Kubernetes with Helm among the production options.
The licence file puts everything outside the ee directories under the MIT Expat licence. Some add-on features need a licence key. Check the features you need against that list before you plan the build.
A self-hosted instance sends a small amount of deployment telemetry to Langfuse. The page states that it “does not send raw traces, prompts, observations, scores, or dataset contents”. For the open source edition, set TELEMETRY_ENABLED=false on all application containers. The page says enterprise telemetry cannot be disabled. An egress deny rule blocks it in either case.
Langfuse accepts OpenTelemetry traces on /api/public/otel. It supports OTLP over HTTP, and the page states that “gRPC is not supported yet”. Configure the OpenTelemetry Collector exporter for HTTP. The Langfuse SDKs also accept a masking function to “redact sensitive information before trace data leaves your application”.
Dashboards in Grafana
vLLM exposes Prometheus metrics on /metrics, including vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:num_requests_waiting and vllm:kv_cache_usage_perc. The vLLM repository has a Prometheus and Grafana example under examples/observability/prometheus_grafana. Grafana has the Prometheus data source preinstalled, and it ships with a PostgreSQL data source for the evaluation results table.
One dashboard covers operations: p95 latency, queue depth, KV cache use and error rate per model. A second covers quality: evaluation scores per release, the refusal rate, the rate of failed citation checks and user feedback. Alert on the quality panels as well as on latency.
What to log for an audit trail, and what not to log
An audit record for each request lets a reviewer reconstruct what the system did and why. Record these fields:
- Request ID and timestamp
- A pseudonymous user ID and the access groups resolved for the request
- Model name and revision, prompt template version and index version
- Retrieved chunk IDs and document versions
- Validation results and any human review decision
- The output, or a hash of it with the full text in a restricted store
Do not put raw prompts or answers in general application logs. Do not copy retrieved passages into the trace store, because the IDs point to the source system. Never log secrets, API keys, tax file numbers or card numbers, and mask them in the SDK before export.
APP 11 requires an entity to take reasonable steps to protect personal information from misuse, interference, loss and unauthorised access. The OAIC guide to information security says that APP 11.3 makes those steps include “technical and organisational measures”, for information held after 11 December 2024. The same guide covers APP 11.2: destroy or de-identify personal information once you no longer need it, unless a law requires you to keep it. Set a retention period for traces and evaluation data, enforce it in ClickHouse and PostgreSQL, and protect audit logs from change.
What to check next
Check three things. Is there an evaluation set of real cases, sized for the change that you need to detect? Does every change to the model, prompt or index pass a CI gate before release? Can you show a reviewer the trace and the audit record for one request from last month, without any of it leaving your environment? The air-gapped AI article covers the same tooling with no outbound network at all, and the hardening article covers the controls around it.
Zinto sets up evaluation, tracing and audit logging in on-prem AI deployment and LLM integration and RAG work.