Open-weight LLMs compared: licences, size and fit in 2026

Pick the smallest open-weight model that passes your own evaluation set. It must also come under a licence your legal team has read and run on hardware you can buy or rent in Australia. An open-weight model is a model whose trained weights you can download and run on your own servers under the publisher’s licence. Public leaderboards help you build a shortlist. They do not tell you how a model handles your documents, your forms or your customers.

The shortlist as of October 2026

The table uses only facts stated on each Hugging Face model card on 20 October 2026. “Not stated” means the card does not say. It does not mean the model lacks the feature. Model cards change, so read each card again on the day you decide.

A mixture of experts (MoE) model splits its layers into many expert blocks and sends each token through a few of them. The Mistral Large 3 card gives two sets of figures: 675B total with 41B active, and 673B with 39B active. The table uses the first set.

Task fit on your own evaluation set

An evaluation set is a fixed list of real inputs from your work, each with the expected output or a marking rubric. It is the only test that decides the choice. Build it before you download a model.

1. Take a few hundred cases from the real workflow: claims notes for an insurer, loan file extracts for a bank, supplier invoices, client emails. Remove or de-identify personal information first.
2. Write the expected answer for each case, or a rubric a reviewer can apply in under a minute.
3. Score what a machine can score: exact match on extracted fields, JSON that passes its schema, the correct source cited. Have a person grade a sample of the rest.
4. Run every candidate with the same prompt, the same serving stack and the same quantisation you plan to deploy.
5. Log each run in MLflow or Langfuse with the model version, the prompt version and the scores.

Keep the hard cases in the set: the scanned page, the email with three requests in it, the form with a field in the wrong place. Average scores hide the failures that reach a customer. Our article on evaluation and observability for self-hosted LLMs covers the harness in more detail.

Licence terms

Each publisher uses different terms, and some publishers use several. Read the licence file in the repository of the exact model you deploy.

Llama 4 uses the Llama 4 Community License Agreement, effective 5 April 2025. A licensee whose products had more than 700 million monthly active users in the month before the Llama 4 release date must request a licence from Meta. A licensee that distributes or makes available the model, or a product that contains it, must display “Built with Llama” on a related website or user interface. A model you distribute that was trained or fine-tuned from Llama materials must have “Llama” at the start of its name. The licence also makes the Acceptable Use Policy part of the agreement. That policy prohibits the unlicensed practice of professions, including financial and legal ones, and discrimination in the provision of credit. For multimodal Llama 4 models, the policy withholds the licence grant from companies with a principal place of business in the European Union. If your group is headquartered in the EU, ask your lawyers about that clause before you use image input.

Mistral publishes under three different regimes:

• Apache 2.0, which permits commercial and non-commercial use. Mistral Small 4, Magistral Small 1.2 and Mistral Large 3 use it.
• A Modified MIT License for Mistral Medium 3.5. It removes all rights from a company whose global consolidated monthly revenue exceeds US$20 million for the preceding month. That is about US$240 million a year (20 x 12). A company above it needs a commercial licence from Mistral, so check your group’s consolidated revenue before you shortlist the model.
• The Mistral Research License for older models such as Mistral Large Instruct 2411, which allows non-commercial use only.

Qwen is split the same way. Qwen3.8-27B is Apache 2.0. The large Qwen3.8-2.4T-A95B uses the Qwen3.8-Max License. Under it, products with more than 100 million monthly active users or US$20 million monthly revenue must display the model name on the user interface. A separate licence is needed for a “Model as a Service” or “AI Work Assistant” business with more than US$50 million revenue in 12 months. Internal use that does not expose the model to third parties is exempt from that condition. Qwen3.8-Flash-Next carries a third tag, qwen-community-1.0, so read its file separately.

Record the licence name, version and file hash in your model register. Add the publisher and the country where it is based, because a risk committee will ask for both.

Parameter count and hardware footprint

The memory for the weights is the parameter count multiplied by the bytes for each parameter. BF16 uses 2 bytes, FP8 uses 1 byte and a 4-bit format uses half a byte. An MoE model holds all its experts in memory. The total count therefore sets the memory, and the active count sets the compute for each token, which drives speed. The KV cache (the stored attention state for each token in context) needs memory on top of the weights. It grows with context length and with the number of users at once.

• Qwen3.8-27B at BF16: 27 billion x 2 bytes = 54 GB. That fits on one 94 GB H100 NVL and leaves about 40 GB for the KV cache.
• Llama 4 Scout at int4: 109 billion x 0.5 bytes = about 55 GB. The card says Scout fits on a single H100 with int4 applied at load time.
• Mistral Small 4 at FP8: 119 billion x 1 byte = 119 GB. The card’s vLLM command splits it across two GPUs.
• Llama 4 Maverick at FP8: about 400 GB. The card says it fits on a single H100 DGX host, which NVIDIA lists with 640 GB of GPU memory.
• Mistral Large 3 at FP8: about 675 GB. The card says it runs on a single node of B200s or H200s.
• Qwen3.8-2.4T at BF16: 2.4 trillion x 2 bytes = 4.8 TB, which needs several nodes.

A model that fits on one GPU is cheaper to run, simpler to patch and easier to make redundant. Sizing GPU hardware for LLM inference works through the KV cache arithmetic, and GPU procurement in Australia covers lead times and suppliers.

Context length

The context window is the maximum number of tokens the model reads in one request. The advertised figure and the useful figure differ. The Magistral Small 1.2 card says performance can degrade past 40k tokens in a 128k window. Each extra token of context also costs KV cache memory for every user at once. Test the candidate at the context length your workflow uses, with the answer placed early, in the middle and late in the input. For document question answering, retrieval of the relevant passages usually costs less than a long prompt. RAG on private infrastructure covers that design.

Languages

In the 2021 Census, 5.8 million people (22.8%) used a language other than English at home. The most common were Mandarin, Arabic, Vietnamese, Cantonese and Punjabi. The Llama 4 card lists 12 supported languages, including Arabic, Hindi and Vietnamese, with no Chinese language in the list. The Mistral Small 4 card lists Chinese, Japanese, Korean and Arabic among “dozens of languages”. The Qwen3.8-27B card gives no count. If your customers write to you in these languages, put their messages in the evaluation set and grade them as you grade English.

Tool calling

Tool calling is when the model returns a structured request to run a function, and your code runs it. It needs support in the model and a matching parser in the serving engine. The Mistral Small 4 card gives this vLLM command:

vllm serve mistralai/Mistral-Small-4-119B-2603 \
  --max-model-len 262144 --tensor-parallel-size 2 \
  --attention-backend FLASH_ATTN_MLA \
  --tool-call-parser mistral --enable-auto-tool-choice \
  --reasoning-parser mistral \
  --max_num_batched_tokens 16384 --max_num_seqs 128 \
  --gpu_memory_utilization 0.8

Score tool calls in the evaluation set. Count the calls that parse, the calls that pick the correct tool and the calls with correct arguments. Also test what the model does when a tool returns an error or a permission is missing. Hardening a self-hosted LLM stack covers how to limit what a tool can do.

Quantisation availability

Quantisation stores weights in fewer bits to cut memory and increase speed, and it can lose some quality. Publisher quantisations, such as Mistral’s FP8 and NVFP4 checkpoints, come from the same source as the weights. NVFP4 is a 4-bit floating point format. The Mistral Large 3 card says its NVFP4 version runs on a single node of H100s or A100s. Community quantisations, such as GGUF files for llama.cpp, are numerous: the Qwen3.8-27B page lists more than 1,500. Treat each one as third-party code. Record who made it, pin the file hash and evaluate that exact artefact. A quantised model is a different model for evaluation purposes.

What to decide next

Choose three candidates from the table that fit your hardware and your licence position. Build the evaluation set from real cases, run all three on the serving stack you will deploy and keep the scores with the licence record. If none of them passes, decide whether the gap is knowledge or behaviour before you consider fine-tuning or RAG. Our on-prem AI deployment service runs this selection with your team inside your own environment.