Open-Weight Models on Your Own GPUs

The frontier-model leaderboards are a genuinely useful thing to read. On raw reasoning, coding, and long-context recall, the best closed APIs from the major labs still sit at the top, and for a lot of work the honest answer is to call one of them and move on.

However, in a regulated enterprise the leaderboard is answering a question you did not ask. The question that decides your architecture is not "which model scores highest on a public benchmark?" It is "which model can I run over a customer's regulated dossier without that dossier leaving my perimeter?" On that question, an open-weight model, Llama, Mistral, Qwen, or one of their descendants, running on GPUs you control, frequently wins. Not because it is smarter. Because it is there, inside your walls, where the frontier API is not.

This is a different axis of comparison than the one the market trains you to use. Capability is one axis. Locus of execution is another. In an unregulated business the first dominates. In a regulated one the second often overrides it.

What "open-weight" actually buys you

Start with the term, because it gets used loosely. An open-weight model is one whose trained parameters you can download and run on your own hardware. It is not necessarily open-source in the full sense: the training data and the training code may be closed, and the license may carry conditions. What you get is the artifact that matters operationally, the weights, so you can load them onto your GPUs and run inference without a network call to anyone.

That single property, inference under your own roof, is what changes the compliance picture.

When you call a frontier API, the prompt and its contents leave your perimeter and are processed on infrastructure you neither own nor homologated. Contractual assurances about non-retention and non-training help, and they matter, but they are promises about someone else's systems. When you run an open-weight model on your own GPUs, there is no egress to promise about. The regulated data is read, reasoned over, and discarded inside a boundary your security function already controls. Data residency stops being a clause you audit in a vendor's contract and becomes a physical fact about where the electrons went.

For work over PII, credit files, medical records, or anything your regulator treats as special-category data, that distinction is not a nuance. It is frequently the whole decision.

The capability gap is real, and it is closing where it matters

Being honest about the tradeoff is the only way this argument is worth making.

On the hardest general reasoning, the best closed frontier models are still ahead of the best open weights, and pretending otherwise sets a project up to fail. But two things soften that gap in a regulated context.

First, the gap is narrowing, and it narrows fastest on exactly the tasks a regulated back office runs: structured extraction, classification against a policy, summarization with citation, constrained reasoning over a bounded document set. These are not open-ended creative tasks where frontier scale shows off. They are narrow, repeatable, and gradeable, which is where a fine-tuned or well-prompted open-weight model closes most of the distance.

Second, in production the model is never the whole system. A mid-tier open-weight model wrapped in retrieval, validation, and a human gate on the hard cases often outperforms a frontier model called naked, because the accuracy that survives to production comes from the architecture around the model, not from the model's raw score. The reframe from the trenches holds here: structure plus a good-enough model beats a great model with no structure.

So the practical question is not "is the open model as smart as the frontier one?" It is "is the open model good enough on this specific, bounded task once it sits inside a system that catches its mistakes?" For a large slice of regulated work, the answer is yes.

The bill you actually pay: ops burden and GPU capacity

Running your own weights is not free, and the cost is not mainly the license. It is operational.

You now own an inference stack. That means GPU capacity planning, model serving, batching and quantization to fit memory budgets, version management as new open weights ship, and the security hardening of the machines the weights run on. The frontier API hid all of that behind someone else's SRE team. Bring the model home and that work is yours.

GPU capacity in particular is a real constraint, not a footnote. Frontier APIs give you effectively elastic throughput on demand. An on-premise cluster has a fixed ceiling, and a workload spike that an API would have absorbed silently becomes a queue you have to plan around. This is a genuine engineering cost, and any honest build accounts for it up front rather than discovering it at the first month-end batch.

None of this argues against on-GPU models. It argues against pretending they are a checkbox. The organizations that succeed treat the inference stack as infrastructure to be run with the same discipline as any other regulated system, and the ones that fail treat a downloaded model as a finished product.

Provider-agnostic is what lets you route, not choose

Here is the part that resolves the tension. The choice is not open-weight versus frontier. It is a routing decision made per workload, and the thing that makes routing possible is a provider-agnostic gateway.

A model gateway is a single internal interface every agent calls, behind which the models are interchangeable. An open-weight model on your GPUs, a frontier API, and an EU-region hosted model all sit behind the same door, and the agent does not know which one answered. The value of that door is that it lets a policy, not a developer's habit, decide where each workload runs.

The rule that policy enforces is routing by data sensitivity, and it fails closed. A step that reads a regulated dossier is pinned to the on-GPU model and physically cannot reach an external API. A step that drafts a generic summary from non-sensitive material, or reasons over already-public information, may be routed to a frontier API for its extra capability, but only because a policy explicitly permitted that class of data to leave. When the classification is uncertain, the data is treated as sensitive and stays home. The default protects you; relaxing it takes a deliberate, logged decision.

That is what makes the capability gap manageable rather than fatal. You do not have to give up frontier reasoning everywhere to keep regulated data inside your walls. You route the sensitive workloads to the model that lives in your perimeter, and the non-sensitive ones to whatever is best for the job, under a rule your compliance function wrote and can prove.

The question worth carrying into your next architecture review, then, is not "which model is best?" It is which of our workloads can never leave the building, and do we have a model good enough to serve those inside our own walls? Answer that per data class, and the frontier-versus-open debate dissolves into a routing table. Skip it, and you are one convenient API call away from sending a regulated dossier somewhere it was never allowed to go.

If you want to map which of your workloads belong on your own GPUs and which can safely route elsewhere, book a meeting and bring us the most sensitive one.