This page is an example of how we run a private LLM build for a bank, and it does not describe a client.
The situation
The credit team at a mid-size bank reads facility agreements, covenant schedules, borrower financial statements and past credit memos every day. An analyst who wants to see how a covenant was worded on a similar facility searches shared drives or asks a colleague. Public AI tools are out, because the documents hold borrower data. APRA’s CPS 234 requires the bank to protect its information assets with controls that match how critical and sensitive they are. The standard also covers information assets that a third party manages. The bank has GPU capacity and a platform team, but no engineers with experience in LLM systems.
What we build
We run this as a forward deployed team. Forward deployed engineers are our engineers who work inside your team, on your systems and under your access rules. They sit with the bank’s platform team and build three parts.
The first part is ingestion. A pipeline extracts the text of each document page by page. It stores the text in PostgreSQL with pgvector, an extension that searches text by meaning. Each document keeps the permissions it has in its source system. An analyst who cannot open a file gets no answer from it.
The second part is the model. We serve an open-weight model such as Llama or Mistral with vLLM, an inference server. It runs on GPU nodes inside the bank’s VPC or in its own data centre. This is a private AI deployment, and no prompt, document or answer goes to an outside service.
The third part is the tool the analysts use. An analyst asks a question in plain words, for example how gearing covenants were set on past facilities in one sector. The answer cites the document and the page, and a click opens that page. The tool says so when the documents hold no answer.
Langfuse and OpenTelemetry trace every question, retrieval step and answer into the bank’s own monitoring. We also build an evaluation set with the credit team, which is a fixed list of real questions with known answers and source pages. The evaluation set runs in pytest on every change, and a change that makes answers worse does not ship.
What your team owns
The bank owns every part. The code sits in the bank’s repositories, the infrastructure is in Terraform, and releases go through the bank’s own pipelines. The platform team runs the model servers, the ingestion pipeline and the permission mapping. We write runbooks for the common jobs: add a document source, change a permission rule, swap the model and roll back a release. Our engineers do each job with the bank’s engineers before handover, and our access ends at handover.
What to expect
Analysts get answers inside the bank’s network, with the source page next to each one. The risk team can read the traces to see who asked what, which documents the system read and what it answered. The credit team and the risk team judge the system against the same evaluation set. The bank’s own engineers can add a new document set or use case on the same platform.