RAG (retrieval augmented generation) means the system first finds the passages in your documents that match a question, then the language model writes an answer from those passages. The answer cites each passage it used. The model does not learn your documents, and they stay in your own index.

What stops it answering from outside the documents?

The model is instructed to answer only from the passages it found and to say so when they do not hold the answer. The evaluation set includes questions the documents cannot answer, and we score how often the system declines them. Any answer can be checked by opening the cited source.

Which vector database do you use?

PostgreSQL with pgvector when you already run Postgres and the library fits in it. Qdrant when the index is large or the search load is high. Elasticsearch when you already run it for keyword search. The choice comes out of the evaluation scores on your own documents.

Can it respect who is allowed to see which documents?

Yes. Permissions come from the system each document lives in, such as SharePoint or your document management system, and the search applies them before the model reads anything. A staff member gets answers only from documents they can already open.

Do our documents go to a public AI service?

No. The model and the index run inside your own cloud account or on your own servers, and no document or question leaves that environment. The hosting is the same setup as our private AI deployment service.

LLM integration and RAG

Every answer cites the page it came from

Retrieval only searches the documents the person asking is allowed to open.

Question

How did we set gearing covenants on past property facilities?

Filter to documents this person can open
Search the index by meaning and keyword
Pass the best passages to the model

Answer

Past property facilities set a maximum gearing ratio in the covenant schedule, tested each half year ¹. Two later facilities added a cure period ².

1Facility agreement, covenant schedule, page 14
2Credit memo, section 4

Illustration. Every answer links to the page it came from.

You leave with staff asking questions of your own documents

01
Staff asking questions of your own documents
02
A cited source on every answer
03
Document permissions applied to every search

What it is

LLM integration means connecting a large language model (software that reads and writes text) to the systems your business already runs. RAG is the most common form of it. The system searches your own documents for the passages that answer a question, and the model writes the answer from those passages and cites them.

We provide RAG consulting and development for Australian businesses that cannot send their documents to a public AI service. The model and the index run in your own cloud account or on your own servers. On-prem AI deployment covers how that hosting works.

Where the work goes

Most of a RAG build is search rather than the model. A library of policy wordings, a set of loan agreements or ten years of job files holds long documents. Those documents carry defined terms, cross references and versions that replace each other. A search that cuts them into even blocks of text loses the clause structure, and the model then answers from the wrong passage.

Architecture

The ingestion pipeline runs in LlamaIndex. It extracts the text and splits each document along its own structure: sections, clauses, schedules and defined terms. Each passage keeps its document, date, client, version and permissions. An embedding model turns each passage into a vector, a list of numbers that captures its meaning.

Search is hybrid. Vector search in PostgreSQL with pgvector or in Qdrant finds passages by meaning. Keyword search in Elasticsearch finds exact terms, because a policy number or a clause reference has to match exactly. Filters narrow the search to one client, one fund or one job before the model reads anything. The model runs on vLLM inside your environment and writes the answer from the top passages.

Superseded documents stay in the index and are marked as superseded. A question about the current terms gets the current version. A question about the terms that applied in 2023 gets the 2023 version.

Citations and permissions

Each answer names the document and the page or clause it came from, and the reader can open it in one click. A credit analyst, an underwriter or a project manager can then check the answer before acting on it. When the documents do not hold the answer, the system says so and shows the closest passages it found.

Permissions are read from the system each document lives in and stored on each passage. The search applies them as a filter on every query. Deal team folders, client files and HR records stay inside the groups that own them.

Evaluation and logging

We build a set of real questions from your staff before go live. Each question has an answer and a source that a person in the business agrees is correct. An evaluation harness is a test program that scores the system against that set. Ours follows the Ragas style and scores retrieval on whether the right passage came back. It scores the answer on whether it matches and on whether every claim in it is supported by a cited passage. The harness runs in pytest on every change to the documents, the search or the model. A change that scores lower does not go live.

Langfuse records every question, the passages retrieved, the model version, the answer and the person who asked. Your team can replay a bad answer and see which step failed.

In practice

For a Sydney builder on Procore we made about 10,000 job files, site notes and tenders askable on a private server they rent. A private model answers the questions, and every answer cites the documents behind it. Procore stayed the system of record, and nothing was migrated out of it. The same document layer later fed their estimating automations. The document search page covers this use in more detail.

Beyond search

LLM integration also covers work where the model reads and writes inside a system rather than answering a person. One example is a claim file summary written into the claims system. Others are a comparison of two agreements, or details from an email posted to the CRM. When that work runs as a process with several steps and review points, it becomes workflow automation.

Handover

Your team gets the ingestion and sync code, the evaluation set and a runbook for adding a new document source, in your own repositories.

llm-integration-rag.stack.yaml

# Tools on this service. Same stack across every Zinto build.
service: LLM integration and RAG
runs_on: your servers, on premises
stack:
  - LlamaIndex
  - PostgreSQL with pgvector
  - Qdrant
  - Elasticsearch
  - vLLM
  - Langfuse
  - pytest

Other services

All seven services →

U1Strategy and readiness auditA ranked list of what to build, what to buy and what to leave, checked against your data, systems and security rules U2On-prem AI deploymentCustom GPU servers in your building or data centre, running open-weight models on bare metal, with every request logged in your environment U3Forward deployed engineersSenior engineers inside your team and your systems until the work is live and handed over U4Agents and workflow automationAgents that read, check and post work inside your systems of record, with a person where an error costs money U5Data engineeringThe data layer your agents and reports read from U7Fine-tuningAn open model trained on your data for one narrow task