What hardware do we need?

One or more NVIDIA GPU servers. The number and type depend on the size of the model and how many people use it at the same time. We size the servers against your real usage and give you a specification before you buy anything. Our guide to [sizing GPU hardware](/insights/sizing-gpu-hardware-for-llm-inference) shows the arithmetic.

Do we need a data centre?

No. A single GPU server can sit in a server room with enough power, cooling and network capacity. Larger builds usually go into a data centre or a colocation facility. We check power and cooling at the site before the hardware is ordered.

Is a model on our own servers as good as ChatGPT?

Not on every task. Open-weight models perform well on focused work over your own documents, such as finding a clause or answering from a policy, and the largest public models still lead on open-ended reasoning. We score candidate models on your own cases and show you the scores before you choose.

Where does our data go?

It stays on your servers. Documents, prompts, answers and logs stay inside your network, and the model can run with no internet connection. Zinto staff get access for the build and for support only, through accounts you control.

What does it cost to run?

You pay for the servers up front, then for power, cooling and upkeep. The cost per request falls as more of your team uses the system. We estimate the running cost from your expected usage before the build starts, and compare it with a hosted model in an Australian region.

On-prem AI deployment

Every layer runs on hardware you own

From sign in down to the GPUs, nothing on the server calls a public AI service.

PeopleStaff sign in through your identity provider

GatewayAccess rules, rate limits, every request logged

InferencevLLM serving Llama or Mistral

RetrievalPostgreSQL with pgvector on local disk

HardwareNVIDIA GPUs on bare metal

SiteYour server room, data centre or colocation

✕ No route to public AI services

You leave owning the servers and the models on them

01
GPU servers you own, running models inside your own network
02
Candidate models scored on your own cases before go live
03
Your team running, changing and rolling back the system without us

What is on-prem AI?

On-prem AI is an AI system that runs on servers you own, in your own building or data centre. The model, your documents, the questions staff ask and the answers they get never leave your network. People also call it a private LLM, a self-hosted LLM or an on-premise LLM. An LLM (large language model) is the kind of model behind ChatGPT.

We build it on bare metal. Bare metal means the model runs directly on the physical server, with no virtual machine and no cloud provider between your data and the hardware. You hold the servers, the keys and the model files.

Who needs on-prem AI?

Banks, insurers, asset and wealth managers, accounting and audit firms, real estate finance businesses and finance teams hold client records, account data and claims files. Builders, logistics firms, manufacturers and wholesalers hold pricing, tenders and supplier terms. Most of these businesses cannot send that data to a public AI service.

OAIC guidance on AI products recommends that organisations “do not enter personal information, and particularly sensitive information, into publicly available generative AI tools”. APP 8 and section 16C of the Privacy Act make your business accountable when an overseas recipient mishandles personal information that you disclose. A system on your own servers keeps that data inside an environment you already govern.

Custom server builds

We size the hardware from your real workload before you buy anything. The inputs are the model you need, the length of your documents and how many people use it at the same time. The output is a specification: the GPU type and count, CPU, memory, NVMe storage and network. Our guide to sizing GPU hardware shows the arithmetic behind it.

You buy through your usual hardware vendor, or we source the servers for you. We check power, cooling and rack space at the site first, because a GPU server draws far more power than a standard one. The servers go into your server room, your data centre or a colocation facility in Australia. Buying, leasing or renting GPUs covers the trade-offs.

Bare metal set up

Each server gets a Linux operating system, the NVIDIA drivers and a container runtime. The model runs in Docker containers, on a single server or across a Kubernetes cluster when you run more than one. vLLM loads the model onto the GPUs and answers requests over a private API. vLLM is an inference server, the program that runs the model and serves its answers.

The model is open-weight, which means its files are public and you run them yourself. Llama and Mistral are the common families. The endpoint has no public address. For the most sensitive work the whole system runs with no internet connection at all, as air-gapped deployments describes.

Around the model we build search over your documents, connections to your systems of record, and the workflows that use them.

When the cloud fits

Some businesses do not want to run hardware. For them we deploy the same stack on GPU nodes in a VPC in an Australian cloud region, or use a hosted model under your own cloud account. A VPC (virtual private cloud) is the private network in your cloud account. Amazon Bedrock runs in the Sydney region and offers Australian inference profiles that keep requests inside Australian regions. Azure OpenAI offers deployments in Australia East that process prompts and responses inside the Australian geography. We confirm the processing location of each model before anything goes live. Where a private LLM should run compares the options.

Evaluation

We choose the model with you after a test on your own work. The evaluation set comes from your past cases, with answers that your people agree are correct. An evaluation harness, a test program that runs the set against the system, scores each candidate model on accuracy, response time and cost per request. You see the scores before the system carries real work. The set runs again before every model or prompt change, and a change that scores lower does not go live.

Logging and access

Staff sign in through the identity provider you already use, such as Microsoft Entra ID or Okta. Each person sees only the documents their role permits. OpenTelemetry records a trace for every request. Langfuse runs on your servers and stores the prompt, the answer, the cited sources, the model version and the person who asked. That log can feed your existing security monitoring.

Each deployment pins one model version. The previous version stays ready, and the runbook gives the steps to roll back to it.

In practice

A Sydney construction business on Procore had about 10,000 job files, tenders and site notes as PDFs. We made them machine readable on a private server the client rents and put a private model on top. Every answer cites the documents it came from, and no document or question goes to a public AI service. Ten staff find in seconds what took hours of digging. Estimating automations now run on the same foundation, as the case study describes.

Handover

Your infrastructure team gets the hardware specification, the build scripts, the runbook, the evaluation set and training to run, monitor and change the system. You hold the servers, the keys and the model files, and the system keeps running when our engagement ends.

private-ai.stack.yaml

# Tools on this service. Same stack across every Zinto build.
service: On-prem AI deployment
runs_on: your servers, on premises
stack:
  - On-prem NVIDIA GPUs
  - vLLM
  - Llama
  - Mistral
  - Kubernetes
  - Docker
  - OpenTelemetry
  - Langfuse

Other services

All seven services →

U1Strategy and readiness auditA ranked list of what to build, what to buy and what to leave, checked against your data, systems and security rules U3Forward deployed engineersSenior engineers inside your team and your systems until the work is live and handed over U4Agents and workflow automationAgents that read, check and post work inside your systems of record, with a person where an error costs money U5Data engineeringThe data layer your agents and reports read from U6LLM integration and RAGAnswers from your own documents, each with its source U7Fine-tuningAn open model trained on your data for one narrow task