An API bills you for each token, and a GPU bills you for each hour whether it does work or not. Self-hosting costs less only when your traffic keeps the GPU busy enough that its cost per token falls below the API price. Utilisation decides the answer, and you can measure it before you buy anything.
The API side
The cost of a hosted model is the token count multiplied by the price for each token, with input and output priced separately:
API cost per month = (input tokens x input price) + (output tokens x output price)The Amazon Bedrock pricing page lists on-demand prices for the Asia Pacific (Sydney) Region. As of 20 October 2026, it shows these prices for each million tokens:
| Model on Bedrock, Sydney | Input, per 1M tokens | Output, per 1M tokens |
|---|---|---|
| Magistral Small 1.2 | $0.5150 | $1.5450 |
| Mistral Large 3 | $0.5150 | $1.5450 |
The page shows a dollar sign without a currency code on these rows. This article treats them as US dollars. Make sure that your AWS invoice uses the same currency before you reuse the figures. The page also says batch inference costs “50% lower” than on-demand, which matters for overnight work such as document classification.
Both models are open-weight under Apache 2.0, according to the Magistral Small 1.2 and Mistral Large 3 model cards. That gives a fair comparison: the same model through an API or on your own GPU. Bedrock and Azure OpenAI in Australian regions covers what the hosted option does and does not keep in the country.
The self-hosted side
The cost of a self-hosted model has four parts:
Self-hosted cost per month = GPU + power + ops staff time + idle capacityGPU is the rental price or the amortised purchase price. A cloud rental includes power, cooling and the data centre in its hourly rate. The Azure Retail Prices API is Microsoft’s official price source, and Microsoft states that it prices all Azure services in USD. As of 20 October 2026, it returns these Linux pay-as-you-go prices for Australia East:
| Azure VM, Australia East | GPUs | Price per hour |
|---|---|---|
| Standard_NC40ads_H100_v5 | 1 x H100 NVL, 94 GB | US$10.121 |
| Standard_ND96isr_H100_v5 | 8 x H100, 640 GB in total | US$142.60 |
Savings plans and reservations lower the hourly price, and the preview version of the same API returns those rates.
If you buy the hardware, the GPU line is the purchase price less its resale value, divided by the months you expect to use it. Power is then a separate line. NVIDIA lists the H100 SXM at up to 700 W each, and the DGX H100 power specification shows 10.2 kW maximum. At that maximum, one month is 10.2 kW x 730 hours = 7,446 kWh. Multiply by your tariff and by the power usage effectiveness of your data centre (the ratio of total facility power to IT power). Add colocation fees and the hardware support contract. GPU procurement in Australia covers purchase, lead times and the hosting choice.
Ops staff time is the engineer hours for patching, upgrades, monitoring and incidents. Idle capacity is every paid GPU hour that serves no request.
Utilisation as the deciding variable
Utilisation is the share of paid GPU hours that do useful work at your latency target. The effective cost of self-hosting for each million tokens follows from it:
cost per 1M tokens = monthly self-hosted cost / (tokens per second x seconds per month x utilisation / 1,000,000)The API price stays flat as volume changes. The self-hosted price falls as utilisation rises. Break-even is the volume at which they meet. Below it the API costs less. Above it self-hosting costs less, up to the throughput limit of the hardware.
A worked break-even
Every figure below is either a price linked above or an assumption stated in the table. Replace the assumptions with your own.
| Input | Value | Source |
|---|---|---|
| Model | Magistral Small 1.2, 24B parameters | Model card |
| Fits on one GPU | 24 billion x 2 bytes = 48 GB at BF16, inside 94 GB | Arithmetic |
| API price | US$0.515 input, US$1.545 output per 1M tokens | Bedrock, Sydney |
| Token mix | 3 input tokens for each output token | Assumption |
| Blended API price | (3 x 0.515 + 1 x 1.545) / 4 = US$0.7725 per 1M tokens | Arithmetic |
| GPU | 1 x Standard_NC40ads_H100_v5 at US$10.121 an hour | Azure, Australia East |
| Hours | 730 a month, always on | 8,760 hours / 12 |
| Ops staff time | 20 hours a month at US$100 an hour = US$2,000 | Assumption |
The arithmetic:
- GPU: 10.121 x 730 = US$7,388.33 a month.
- Total self-hosted: 7,388.33 + 2,000 = US$9,388.33 a month.
- Break-even volume: 9,388.33 / 0.7725 = 12,153 million tokens, or about 12.2 billion tokens a month.
- As requests: at 4,000 tokens each (3,000 in, 1,000 out), that is about 3.04 million requests a month, or about 100,000 a day.
- As throughput: 12.153 billion tokens / (730 x 3,600 seconds) = about 4,624 tokens a second, around the clock.
The fifth line is the test. If one GPU cannot sustain about 4,600 tokens a second of your traffic at your latency target, the API costs less at every volume. If it can, self-hosting costs less only when real traffic reaches that rate.
Office hours change the result. Assume traffic arrives in 10 hours a day on 22 working days, which is 220 hours a month:
- GPU left on all month: the same 12.2 billion tokens must arrive in 220 hours. That needs about 15,345 tokens a second during those hours (12.153 billion / 792,000 seconds).
- GPU stopped outside office hours: the GPU costs 10.121 x 220 = US$2,226.62, and the total is US$4,226.62. Break-even falls to about 5.5 billion tokens a month, or about 6,908 tokens a second during office hours.
The same method on the larger model shows the scale effect. Mistral Large 3 in NVFP4 runs on a single node of H100s, according to its card. On a Standard_ND96isr_H100_v5 at US$142.60 an hour, one month costs 142.6 x 730 = US$104,098. With the same US$2,000 for ops, break-even is 106,098 / 0.7725 = about 137 billion tokens a month. That is about 52,262 tokens a second around the clock.
Do not guess the throughput figure. Measure it with vllm bench serve on the candidate GPU, with input and output lengths drawn from your own logs. The vLLM benchmark reports total token throughput, time to first token and time per output token. Use the throughput at the concurrency where latency still meets your target. Sizing GPU hardware for LLM inference explains why that number falls as context grows.
Reasons other than cost
A regulated business also weighs these factors.
- Data control. APP 8 sets out steps before personal information is disclosed overseas, and APP 11 requires reasonable steps to protect it. A model on your own GPUs keeps prompts, documents and logs inside your network. Data sovereignty and residency in Australia covers the difference.
- Service provider risk. For an APRA regulated entity, a hosted model provider can be a material service provider under CPS 230. That brings a register return and notification to APRA.
- Rate limits. Bedrock controls inference with quotas on token usage, and AWS says quota increases are not granted automatically. A self-hosted model is limited only by your hardware.
- Latency. A model in the same network as the data removes a round trip to a shared service and a shared queue. Measure both before you count this as a gain.
- Version control. A self-hosted deployment runs the exact weights you evaluated until you choose to change them.
Hidden costs
Each side has costs that the per-token or per-hour price leaves out.
| Cost | Self-hosted | API |
|---|---|---|
| Evaluation | Build and re-run the evaluation set for every model, quantisation and serving change | Re-run it when the provider changes or retires a model version |
| Upgrades | Download, test and roll out new weights, vLLM releases and GPU drivers | Migrate prompts when a model version is retired |
| On call | A roster for the GPU service if people depend on it outside office hours | The provider runs the service, and you still own the application |
| Redundancy | A second GPU for failover doubles the GPU line: 2 x US$7,388.33 = US$14,776.66 a month | Handled by the provider within its service terms and your quotas |
| Security | Patching, access control and logging for the stack (hardening) | Account configuration and data handling terms |
| Token growth | No change in price as prompts grow, until throughput runs out | Retrieval context adds input tokens to every call |
The redundancy line changes the worked example. With two GPUs and the same ops cost, the break-even volume rises to (14,776.66 + 2,000) / 0.7725 = about 21.7 billion tokens a month.
What to decide next
Pull a month of token counts from your current logs or pilot and split them into input and output. Benchmark the candidate model on the candidate GPU with your own prompt lengths. Then put both figures into the formulas above with your own staff rates. If the break-even throughput is above what the hardware sustains, use a hosted model in an Australian region and revisit when volume grows. Our on-prem AI deployment and AI strategy and readiness audit services run this cost model with your numbers.