Securing a self-hosted LLM: controls mapped to Essential Eight

A self-hosted LLM stack is a set of Linux servers with GPUs. Most of its controls are the ones you already run on other servers. It adds three risks: model files that can execute code when loaded, documents that carry instructions to the model, and model output that other systems trust. This article sets out the controls layer by layer. It maps them to the ASD Essential Eight and APRA CPS 234, the frameworks that Australian risk teams already assess against.

The threat model

The OWASP Top 10 for LLM Applications is the common reference. The current edition, as of October 2026, is the 2025 list. Six entries apply directly to a private RAG deployment:

• LLM01:2025 Prompt Injection
• LLM03:2025 Supply Chain
• LLM05:2025 Improper Output Handling
• LLM06:2025 Excessive Agency
• LLM08:2025 Vector and Embedding Weaknesses
• LLM10:2025 Unbounded Consumption

Self-hosting removes the third-party API from the threat model. It also moves the work of patching, isolation and model provenance onto your team.

Network: no public endpoints and default deny egress

The inference server never gets a public address. Users reach it through an internal gateway that authenticates them and allowlists the API paths that the application uses.

The vLLM security guide explains why the gateway matters. The --api-key option protects only endpoints “under the /v1, /v2, /inference, and /cohere path prefixes”, and the guide says “Do not rely exclusively on --api-key for securing access to vLLM.” It also says that communication between nodes in a multi-node deployment is “insecure by default” and that “Messages are sent unencrypted”. Put the GPU nodes on a dedicated network segment, and put vLLM “behind a reverse proxy (such as nginx, Envoy, or a Kubernetes Gateway)”.

Egress is denied by default. In Kubernetes, the documented default deny egress policy is short:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress

Two notes from the same page apply. The policy also blocks DNS, so add an allow rule for the cluster DNS service. A NetworkPolicy also does nothing unless the network plugin enforces it: “Creating a NetworkPolicy resource without a controller that implements it will have no effect.” Allow traffic only to named internal services such as the vector store, the artefact registry and the trace collector. A compromised process that tries to reach an external host then fails at the network layer.

Identity and least privilege

Users sign in through your identity provider. At Essential Eight Maturity Level Two, the maturity model requires that “Multi-factor authentication used for authenticating users of systems is phishing-resistant”. Level Three adds MFA for “users of data repositories”, which includes the vector store.

The model holds no credentials. Each component (gateway, retriever, vector store, inference server) runs as its own service account with access to its own resources only. OWASP LLM01 says to “Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model.” The user’s identity flows to the retriever so that the search returns only documents that the user can open, as the RAG reference stack describes.

Administrators reach GPU nodes through a jump server. At Level Two, “Administrative activities are conducted through jump servers” and “Privileged access to systems and applications is disabled after 45 days of inactivity.”

Secrets

The Kubernetes documentation warns that Secrets are “by default, stored unencrypted in the API server’s underlying data store (etcd)”. It lists four steps: enable encryption at rest, apply least-privilege RBAC to Secrets, restrict Secret access to specific containers, and consider an external secret store. Essential Eight Level Two requires that service account credentials are “long, unique, unpredictable and managed”.

Keep secrets out of prompts and system instructions. A secret in a prompt reaches traces, logs and, through prompt leakage, users.

Model supply chain

Treat a model file as untrusted code until the intake pipeline checks it. The Hugging Face pickle scanning page says that pickle “is the default format for PyTorch model weights”. It also says “There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file.” The PyTorch torch.load reference says “Never load data from an untrusted source.” Safetensors is “a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy)”.

Check the format before you approve a model. As of October 2026, the main branch of BAAI/bge-m3, a common embedding model, ships its weights as pytorch_model.bin and has no safetensors file. Convert it in an isolated build job with no network and no secrets, or choose another model.

The model intake pipeline runs outside the serving cluster:

1. Download from a pinned commit. The Hugging Face download guide notes that a commit hash “must be the full-length hash”.
2. Download only *.safetensors and configuration files with allow_patterns.
3. Record the SHA256 of each file. The Hub shows it on the page of each large file, for example the bge-m3 weights page.
4. Check the hashes again after transfer, and store the files in your internal artefact registry.
5. Serve from the registry only, with --trust-remote-code left at its default of False. If a model needs custom code, review that code and pin it with --code-revision.

OWASP LLM03 recommends “third-party model integrity checks with signing and file hashes” and points to OWASP CycloneDX for AI and ML bills of materials. The model selection article covers licence and provenance checks before a model reaches this pipeline.

Prompt injection from retrieved documents

The OWASP LLM01 page defines the indirect form: “Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files.” In a RAG system, every retrieved chunk is such an input. An emailed invoice can carry white text that tells the model to mark the invoice as approved. The same page says that RAG and fine-tuning “do not fully mitigate prompt injection vulnerabilities”.

Design on the assumption that an injection will succeed, and limit what it can do:

• Wrap retrieved text in clear delimiters and label it as untrusted data. OWASP says “Separate and clearly denote untrusted content”.
• Give the model no tool that moves money, sends email or changes a record without a person who approves the action.
• Put attack documents in the evaluation set, so every release is tested against them. The evaluation article covers the CI gate.
• Scan documents at ingestion for instruction-like text and hidden text. A scanner finds only the patterns it knows, so it supports the other controls rather than replacing them.

Output validation

OWASP LLM05 says “Treat the model as any other user, adopting a zero-trust approach”. Every downstream system checks model output in the same way as input from a web form:

• Force a JSON schema with vLLM structured outputs, then check it again in code.
• “Use parameterized queries or prepared statements for all database operations involving LLM output.”
• Encode output for the place it is shown, and apply a strict Content Security Policy.
• Block images and links to external hosts in rendered answers. A rendered image URL can carry data out in its query string.
• Set a maximum output length and per-user rate limits, which address LLM10 Unbounded Consumption.

Logging and detection

Send gateway authentication events, admin actions, model deployments, egress denials and validation failures to your central log platform. From Level Two, several Essential Eight strategies require that “Event logs are protected from unauthorised modification and deletion”. The evaluation article lists what to keep out of logs under APP 11.

CPS 234 paragraph 23 requires “robust mechanisms in place to detect and respond to information security incidents in a timely manner”. Paragraph 35 requires notice to APRA “no later than 72 hours” after the entity becomes aware of a material incident. Paragraph 36 sets 10 business days for a material control weakness that the entity cannot remediate in a timely manner. These are from the July 2019 standard. The incident plan for the LLM stack includes model rollback and index rollback.

Patching

The patch surface is the operating system, the NVIDIA driver and firmware, CUDA libraries, container images, Python packages, vLLM and the vector store. The maturity model sets timeframes:

• At all levels, internet-facing servers get operating system patches within 48 hours for critical or exploited vulnerabilities, and within two weeks otherwise.
• At Levels One and Two, operating systems of non-internet-facing servers are patched within one month.
• From Level Two, applications other than office software, browsers, email clients, PDF software and security products are patched within one month, and scanned at least fortnightly.
• At Level Three, non-internet-facing servers, drivers and firmware get the 48 hour rule for critical vulnerabilities, and one month otherwise.

GPU drivers fall under the driver requirements. Rebuild images on a schedule, roll the change one replica at a time, and run the evaluation gate before the new image takes traffic.

The Essential Eight was designed for “internet-connected information technology networks”, and several requirements name Microsoft products. For a Linux GPU cluster, map each control to the intent of the strategy and record the mapping for your assessor.

Checklist

What to check next

Take the checklist to your security team and mark each row as in place, planned or not applicable, with the evidence. Start with egress, model intake and the gateway, because those three close the paths that are specific to an LLM stack. The air-gapped AI article covers the case where the stack has no outbound network at all. For banking and insurance clients, the CPS 234 column is the one the internal audit team will ask for.

Zinto builds and hardens these stacks as part of on-prem AI deployment.