Private RAG stack: parsing, hybrid search and permissions

A private RAG system has six parts that you run yourself: a parser, a chunker, an embedding model, a search index, a reranker and a generation model. Retrieval augmented generation (RAG) means the system finds passages in your documents and gives them to the model together with the question. Parsing and permissions decide what the model can see, so those two layers get the first engineering effort.

The reference stack

LlamaIndex or LangGraph holds the steps together. Every component in the table runs inside your VPC or data centre, and no document text, question or vector leaves it. The deployment architecture article covers where those servers sit.

Parsing scanned PDFs and tables

A born-digital PDF has a text layer. A scanned PDF is a stack of page images, so it needs OCR (optical character recognition, which turns an image of text into text). Most document stores hold both kinds in the same folder.

Tables are the second problem. A plain text extractor reads across the page line by line and merges cells from different columns. A premium schedule or a bill of quantities then becomes a row of numbers with no headers.

Docling is one open source parser that handles both. Its README lists “Advanced PDF understanding incl. page layout, reading order, table structure”, “Extensive OCR support for scanned PDFs and images” and “Local execution capabilities for sensitive data and air-gapped environments”. The codebase is MIT licensed, and its models carry their own licences. A hosted parsing API sends every page to a third party, which is a Privacy Act disclosure question covered in the sovereignty and residency article.

Keep three outputs for every element: the text, the page number and the bounding box. Before you index a new document type, compare the parser output with 20 sample pages by eye. Also count pages that produced no text, because a scanned page with zero characters means OCR did not run on it.

Chunking

A chunk is the unit that you embed and retrieve. The main choices are:

• Fixed token windows with an overlap
• Structure-aware chunks that follow headings, sections and clauses
• One chunk per table, with the table header repeated in the chunk
• Parent and child chunks, where the system retrieves a small chunk and gives the model its parent section

In LlamaIndex, the SentenceSplitter “attempts to split text while respecting the boundaries of sentences”. The HierarchicalNodeParser builds parent and child chunks for its AutoMergingRetriever. Policy documents, contracts and procedures have clear structure, so structure-aware chunks are a sensible first choice. Pick the chunk size with your evaluation set.

Every chunk carries metadata: document ID, document version, page number, section heading, effective date and the access groups from the source system. Retrieval filters and citations both depend on it.

Embedding models you can host

All three models below publish weights on Hugging Face, and vLLM serves them through its pooling runner on an OpenAI-compatible /v1/embeddings endpoint. Figures are from the model cards as of October 2026.

Test two candidates on your own evaluation set before you choose. A change of embedding model later means you embed the whole corpus again, because vectors from two models are not comparable. Store the model name and revision on every row so that a mixed index is visible. Check the weight file format before approval, as the hardening article describes.

The pgvector README says each vector takes 4 x dimensions + 8 bytes. Assume 20,000 documents, 15 pages each and two chunks per page. That is 600,000 chunks. At 1024 dimensions, each vector takes 4,104 bytes, so the raw vectors need about 2.5 GB. The half precision halfvec type takes 2 x dimensions + 8 bytes, which is about 1.2 GB. The HNSW index and the chunk text come on top of that figure.

Vector store options

The default choice is the store your team already operates and backs up. A second database engine is a second thing to patch, monitor and restore.

Hybrid search and reranking

Dense retrieval matches meaning. BM25 (a ranking formula based on term frequency) matches exact terms such as policy numbers, clause references, ABNs and product codes. Two clauses with similar wording get similar vectors, so dense search alone can rank clause 14.3(c) above clause 14.3(b). The stack runs both methods and fuses the two ranked lists.

Reciprocal rank fusion (RRF) gives each document a score of 1 / (k + rank) for each list it appears in, and adds the scores. Elasticsearch defaults k to 60. Take a chunk that ranks 1st in BM25 and 10th in dense search: 1/61 + 1/70 = 0.0307. A chunk that ranks 3rd in both scores 2/63 = 0.0317 and ranks higher, because RRF rewards agreement between the two lists.

A reranker then reads the question and each candidate together and returns a relevance score. This type of model is a cross-encoder. It costs more compute per candidate than a vector comparison, so it runs only on the top candidates. bge-reranker-v2-m3 is Apache 2.0 licensed, and its card says the score “can be mapped to a float value in [0,1] by sigmoid function”. vLLM serves cross-encoders on a /rerank endpoint. Fuse to 50 to 100 candidates, rerank, and give the top 5 to 10 to the model. Then tune both numbers against the evaluation set. The bge-m3 card recommends the same shape: “hybrid retrieval + re-ranking”.

Permission-aware retrieval

The index must return only the chunks that the person who asks can open in the source system. The OWASP Top 10 for LLM Applications 2025 lists this under LLM08 Vector and Embedding Weaknesses and recommends “permission-aware vector and embedding stores”.

The pattern has four parts:

1. Copy the access groups from the source system (SharePoint, Procore, a document management system) onto every chunk at ingestion.
2. Resolve the user’s groups on the server from the identity provider. Never take them from the prompt or the client.
3. Filter inside the search query, before ranking.
4. Run a sync job for permission changes and deletions, and test it with cases that must return nothing.

A system prompt line such as “only use documents the user can see” is not a control, because the restricted text is already in the prompt.

pgvector needs care here. Its README says “filtering is applied after the index is scanned”. It gives the example that a condition which matches 10% of rows, with the default hnsw.ef_search of 40, returns only 4 rows on average. Turn on iterative index scans, and consider partial indexes or partitions for large groups.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id           bigserial PRIMARY KEY,
  document_id  bigint  NOT NULL,
  doc_version  int     NOT NULL,
  page_number  int     NOT NULL,
  acl_groups   text[]  NOT NULL,
  body         text    NOT NULL,
  body_tsv     tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
  embedding    vector(1024) NOT NULL
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);
CREATE INDEX ON chunks USING gin (body_tsv);
CREATE INDEX ON chunks USING gin (acl_groups);

-- Per query: $1 is the question vector, $2 the caller's groups
SET hnsw.iterative_scan = relaxed_order;
SELECT id, document_id, page_number
FROM chunks
WHERE acl_groups && $2
ORDER BY embedding <=> $1
LIMIT 100;

<=> is cosine distance. With relaxed_order, results can be slightly out of order, and the reranker sorts them again. Row-level security can enforce the same filter inside Postgres, but table owners bypass it unless you set FORCE ROW LEVEL SECURITY. For an APRA-regulated entity, this filter falls under paragraph 21 of CPS 234. That paragraph requires controls commensurate with “the criticality and sensitivity of the information assets”.

Citations to the source page

Give each chunk an ID in the prompt and ask the model to return the IDs that it used. vLLM structured outputs can force a JSON schema such as {answer, citations: [chunk_id]}. Code then checks that every cited ID was in the retrieved set and drops any answer that cites nothing. The interface turns each ID into a link to the document version and page, with the bounding box highlighted. OWASP’s prompt injection guidance recommends the same thing: “request detailed reasoning and source citations, and use deterministic code to validate adherence to these formats”.

We built this for a Sydney builder with about 10,000 PDFs next to Procore. The documents became askable on a private server they rent, with a private model, and every answer cites its source documents, as the case study describes.

Serving with vLLM

vLLM runs three separate deployments in this stack: the generation model, the embedding model and the reranker. Its HTTP server “implements OpenAI’s Completions API, Chat API, and more”, so LlamaIndex and LangGraph connect to it like a hosted API. Each deployment needs its own GPU budget, and the GPU sizing article shows the arithmetic. A full corpus embedding run competes with live traffic, so run it on a separate replica or out of hours.

Failure modes

What to decide next

Decide three things before you write code. Pick the vector store your team already runs. Pick two embedding models to test. Write down how access groups flow from each source system into the index, including deletions. Then build the evaluation set described in the evaluation and observability article, because every number in this stack gets tuned against it. Source data with no clean owner or format is a data engineering job that comes first.

Zinto designs and builds this stack on your infrastructure as part of LLM integration and RAG.