On-prem, VPC or hybrid: where a private LLM should run

A private LLM deployment takes one of three shapes. On-prem puts the GPUs, the model and the logs in a data centre you control. VPC runs the same stack on GPU instances in your own cloud account in an Australian region. Hybrid splits the stack between the two and joins the halves with a private circuit.

The layers are the same in all three: a serving engine, a retrieval store, a gateway, identity, logging, a network boundary and egress controls. The topology decides who owns the hardware, where the boundary sits and which third parties your risk team must assess.

The three topologies

On-prem means GPU servers in your own data centre or in a colocation cage that you lease. You buy or lease the hardware, run Kubernetes on it, and carry the spare capacity for failures. The hardware choice is covered in sizing GPU hardware for LLM inference.

VPC means GPU instances in private subnets of your own cloud account. A VPC (virtual private cloud) is the private network inside that account. The model weights sit in your own object storage, and the instances have no public IP address.

Hybrid has two common forms. In the first, documents, identity and retrieval stay on-prem and inference runs in a VPC, connected over AWS Direct Connect or Azure ExpressRoute. In the second, an open-weight model handles most requests. The gateway sends selected tasks to a hosted model, such as Amazon Bedrock or Azure OpenAI in an Australian region. The regional options for hosted models are in Bedrock and Azure OpenAI in Australian regions.

Layer by layer

Serving

vLLM is an open source inference server that loads a model onto GPUs and answers requests. It provides an HTTP server that implements OpenAI’s Completions API, Chat API, and more. Applications call the same API in all three topologies. A move from VPC to on-prem does not change application code.

The vLLM documentation states: “Do not rely on –api-key alone to secure vLLM.” Put vLLM on a private network behind the gateway and accept traffic from the gateway only. vLLM also collects anonymous usage data by default. Set VLLM_NO_USAGE_STATS=1 to turn it off. Set HF_HUB_OFFLINE=1 so that no HTTP calls will be made to the Hugging Face Hub, and load weights from your own store.

Retrieval store

The retrieval store holds document chunks and their embeddings. An embedding is a list of numbers that represents the meaning of a chunk. PostgreSQL with pgvector or Qdrant are the usual choices. The store sits inside the same boundary as the model, and it filters results by the permissions of the user at query time. The design is in RAG on private infrastructure.

In the first hybrid form, the store stays next to the source documents on-prem. Only the retrieved chunks cross the circuit to the GPUs. Those chunks contain client data. The circuit needs the same encryption and monitoring as any other path for that data.

Gateway

The gateway is the single entry point for every model call. It checks the user token, applies rate limits and quotas for each team, routes the request to a model and writes the request record. In a hybrid deployment, the gateway also holds the routing rules that decide which requests can go to a hosted model. Keep those rules in version control and review changes like a firewall change.

Identity

Users sign in through the identity provider you already run, such as Microsoft Entra ID or Okta. The usual protocol is OpenID Connect, which is an interoperable authentication protocol based on the OAuth 2.0 framework of specifications. The gateway checks the token, reads the group claims and maps the groups to roles. The same groups drive the permission filter in the retrieval store. Batch jobs use their own service accounts. Nobody uses a shared API key.

Logging and tracing

Each request record holds the user, the prompt, the retrieved sources, the model version, the answer and the latency. OpenTelemetry carries traces across the gateway, retrieval and serving. Langfuse records the LLM steps. It can be self-hosted using Docker on your own infrastructure, with PostgreSQL, ClickHouse, Redis and S3 compatible storage behind it.

The logs contain the same personal information as the prompts. APP 11 says an entity must take reasonable steps to protect personal information it holds from misuse, interference and loss. Give the trace store the same access controls, retention period and destruction rules as the source system.

Network boundary

In a VPC, the GPU nodes sit in private subnets. They reach cloud services through private endpoints. With AWS PrivateLink, you do not need to use an internet gateway, NAT device, public IP address, Direct Connect connection, or AWS Site-to-Site VPN connection. Amazon Bedrock exposes its runtime API through an interface endpoint with the service name com.amazonaws.{region}.bedrock-runtime.

On Azure, a private endpoint is a network interface that uses a private IP address from your virtual network. Microsoft notes that a private endpoint does not by itself restrict public network access to the service. Turn off public access on each resource as a separate step. Google Cloud uses Private Service Connect for the same purpose.

On-prem, the boundary is a firewall zone. The GPU nodes sit in their own segment and accept inbound traffic from the gateway only.

Egress

The model servers need no internet path at run time. By default, Kubernetes allows all outbound connections from a pod. Apply a default deny policy to the serving namespace. The policy needs a network plugin that enforces it. The Kubernetes documentation states that creating a NetworkPolicy resource without a controller that implements it will have no effect.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-lockdown
  namespace: llm-serving
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { name: llm-gateway }
      ports:
        - { protocol: TCP, port: 8000 }
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports:
        - { protocol: UDP, port: 53 }
    - to:
        - ipBlock: { cidr: 10.20.30.0/24 }   # internal model store only
      ports:
        - { protocol: TCP, port: 443 }

At the VPC edge, AWS Network Firewall can create allow lists and deny lists with domain names. It matches on the TLS SNI or the host header not the IP addresses, so pair the domain rules with IP rules. More controls are in hardening a self-hosted LLM stack.

Comparison table

Australian rules that shape the choice

CPS 234 applies wherever the stack runs. An APRA regulated entity must assess the security capability of any third party that manages its information assets. The entity must also notify APRA of a material information security incident no later than 72 hours after it becomes aware of it.

CPS 230 decides the paperwork. The current version commences on 1 July 2026. For all material arrangements, an entity must maintain a formal legally binding agreement. It must also notify APRA prior to entering into any material offshoring arrangement. Your cloud provider can already hold a material arrangement with you. A VPC in its Australian region can sit inside that arrangement if the terms cover the new service. Your risk team decides this case by case.

APP 8 sets out the steps an APP entity must take to protect personal information before it is disclosed overseas. A hosted model in hybrid form is the part to check. Check where the provider processes the prompt and where it stores any logs. Data sovereignty and data residency are separate questions, and the hosted model answers them differently from an open-weight model in your account.

The Hosting Certification Framework list shows the Australian regions of AWS, Microsoft Azure and Google Cloud as Strategic certified cloud services. It also lists certified colocation facilities. The same page states that the framework is currently undergoing reform.

When each topology fits

On-prem fits a business that runs its own data centre and has a team that already operates servers at night and on weekends. It also fits when a contract or policy keeps data on premises, and when the load is steady enough to keep the GPUs busy. A fully disconnected build is in air-gapped AI.

VPC fits a bank, insurer or wealth manager that already approved a cloud provider. It lets you test real load before you commit to hardware. Check GPU availability by region before you design. As of October 2026, the AWS instance list shows P5 and P5e families in Asia Pacific (Sydney) and only Trainium families in Asia Pacific (Melbourne). The regional detail is in buying, leasing or renting GPUs in Australia.

Hybrid fits when the source systems must stay on-prem but the GPU capacity is easier to get in the cloud. It also fits when a small set of tasks needs a larger hosted model. Hybrid is the hardest of the three to run. It has two sets of controls, a circuit that carries client data and routing rules that a person must review.

What to decide next

Answer these questions before you choose a topology:

1. Where do the source documents and records live today, and can retrieved chunks leave that location?
2. Which cloud provider already holds a material arrangement under CPS 230?
3. How many people will use the system at the same time, and what does that mean for GPU memory?
4. Can any request go to a hosted model, and who approves the routing rules?
5. Who replaces a failed GPU node, and how fast?

Our on-prem AI deployment service builds this stack in your VPC or on your own NVIDIA servers. The banking page shows how it applies to an APRA regulated entity.