Running LLMs On-Premises: The Realities Nobody Advertises

Running an LLM on hardware you own takes about ten minutes. Download an open-weight model, start a runtime, send a prompt, watch the tokens arrive. The demo works and the project gets funded on the strength of it.

Serving that same model to a department is a different engineering problem, and very little from the ten-minute version carries over. The hard part is not installing the model, it is capacity, and capacity on a GPU comes down to two quantities you can calculate before buying anything, which are how much memory the workload needs and how fast that memory can be read. This article works through both with the arithmetic written out, as the sizing companion to the on-premises AI guide. GB below means 10^9 bytes, which is what the parameter-count multiplication produces, and every figure is an illustrative worked example rather than a benchmark, because the formula is the point.

GPU memory is the binding constraint

Everything else about serving an LLM, including latency, concurrency and usable context length, is downstream of whether the workload fits in VRAM. When it does not fit, the runtime does not slow down gracefully. It either refuses requests and evicts running sequences to recompute them later, or it fails to start at all. Four things consume that memory, which are the weights, the key-value cache, the activations produced while processing a request, and the fixed overhead the runtime holds regardless of load.

Start with the weights

Weight memory is the number everyone gets right, because it is a multiplication.

weight memory = parameter count × bytes per parameter

At FP16 or BF16 that is 2 bytes per parameter, at 8-bit roughly 1 byte, and at 4-bit roughly half a byte, although real 4-bit checkpoints run above that because the group scales and zero points that make 4-bit work are stored alongside the weights. Take a mid-size open-weight model of 14 billion parameters, the class many departments settle on because it fits one card and handles extraction, summarization, classification and retrieval-grounded answering competently.

On a card with 32 GB of VRAM, of the kind specified for the AI processing nodes in VIDIZMO's on-premises Redactor reference build, the BF16 version leaves roughly 6 GB before anything else has been allocated. That sounds workable. It is not, and the next section is why.

Then the key-value cache

Each generated token attends to every token before it. Rather than recomputing keys and values for the whole prefix at every step, the runtime caches them. That cache belongs to one sequence and grows by a token's worth on every decode step, and it is released only when the sequence finishes.

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes per element

The leading 2 covers keys and values and bytes per element is 2 at FP16, while the rest comes straight out of the model's config file. Look hardest at kv_heads, because grouped-query attention lets a model carry many attention heads but few key-value heads, and the cache scales with the small number rather than the large one. Take a shape typical of current models in this class, with 48 layers, 40 attention heads at head dimension 128, and 8 key-value heads.

2 × 48 × 8 × 128 × 2 = 196,608 bytes per token, or about 192 KB. Small per token, and then you multiply by context length.

The last row ends most arguments about long context, because one user with a 128k prompt needs more KV cache than the entire weight set of the model at full precision. And if the same model had 40 key-value heads rather than 8, every figure in that table would be five times larger, and a single 8k sequence would need 7.5 GB. Architecture decides what you can serve, not just parameter count.

Then activations and everything the runtime holds

Before a single weight loads, the CUDA context, the serving framework, the kernel workspaces and the allocator's own bookkeeping take roughly 1 to 2 GB. During prefill the runtime materializes activations for the entire prompt at once, scaling with batch size multiplied by prompt length multiplied by hidden size, which produces a transient peak well above the steady state during decode. Fragmentation takes a further slice that is real and hard to predict. Reserve 10 to 15 percent of the card and do not spend it, because setting a serving framework's VRAM fraction to 0.95 is how a system that ran fine for weeks dies on one unusually long prompt.

What this means on a 32 GB card

Put the pieces together for the 14B example, with roughly 2 GB held back for runtime overhead and headroom.

Two concurrent users on the top row. That is the demo. The person who ran it was alone on the box with a short prompt, and nothing in that experience predicted what happens when the third and fourth requests arrive together. Every lever in that table is available before you buy more hardware, whether quantizing the weights, quantizing the KV cache, capping the context the endpoint accepts, or capping concurrency and queueing the overflow deliberately rather than discovering the cap by accident.

When the model does not fit on one card

The same multiplication applies to the largest models, and only the size of the answer changes. A 250-billion-parameter model is 500 GB of weights at BF16, 250 GB at 8-bit and 125 GB at 4-bit, so against the 80 GB H100 SXM, the 141 GB H200 and the 48 GB L40S you need seven, four or eleven cards respectively at BF16, dropping to two, one or three at 4-bit. Those card counts are why quantization is a procurement decision rather than a tuning detail at this scale.

Four things change once a model spans cards, and none of them appear in the weight arithmetic.

Aggregate memory is not usable memory. Each card carries its own runtime context, kernel workspaces and activation buffers, so the headroom you reserved on the 32 GB example is reserved on every card rather than once across the set. Sizing to the sum of the sticker capacities is the standard way to end up one card short.

The KV cache shards with the model rather than sitting on one card. Under tensor parallelism the attention heads are divided across cards, so each card holds its slice of the cache for every running sequence. That is good news for capacity, because cache headroom scales with the card count, and it means the concurrency arithmetic from the earlier tables applies to the aggregate rather than to a single device.

The link between the cards becomes part of the latency budget, because the cards have to exchange results with each other at every step of producing every token. A set of cards on a high-bandwidth fabric and the same cards communicating over PCIe are not the same machine, and that link is the specification most often missing from a quote.

Two different definitions of a gigabyte can cost you a card. The multiplication above produces decimal GB, while some tooling reports memory in GiB, so 500 GB is about 466 GiB. Roughly seven percent sounds ignorable until you are deciding between four cards and five.

What quantization buys, and what it costs

Quantization stores weights at lower numeric precision, usually with a scale and offset held per small group of values so the reconstruction stays close to the original. Some formats dequantize back to higher precision inside the kernel and others compute directly in the low-precision format, and the names you will meet include GPTQ, AWQ, the GGUF k-quants, FP8, INT8 with activation smoothing, and NF4. KV cache quantization is a separate setting, and on a memory-constrained card both are worth having.

Memory is the obvious gain and throughput is the less obvious one, because decode reads the entire weight set out of memory for every token, so a weight set a third the size is read roughly three times faster. Quantization often does more for responsiveness than for capacity.

The cost is paid in quality, and the published measurements are smaller than the reputation suggests. GPTQ reported four-bit quantization costing 0.25 perplexity or less against full precision on 175-billion-parameter models, and 0.3 to 0.6 points at three bits. What collapses is the naive method of rounding every number to the nearest available value, which took one 176-billion-parameter model from 8.11 perplexity to 571 at three bits. The method matters more than the bit width, which is the practical reason to use a published quantization recipe rather than whatever the runtime does by default.

Those numbers need one caveat, which is that perplexity on general text is not the standard your work will be judged against, and the cost of quantization lands unevenly across different kinds of task. Eight-bit is usually close to indistinguishable from full precision for practical work, and four-bit is where judgment enters. Extraction, classification, summarization and retrieval-grounded answering over supplied text tend to hold up well, because the model is mostly reading rather than recalling from parameters. Multi-step reasoning, code generation, strict adherence to a required output format, and work in lower-resource languages degrade sooner and less predictably. Larger models also tolerate aggressive quantization better than small ones, which is why a larger model at 4-bit often beats a smaller model at 8-bit for the same memory budget while being slower per token.

None of that can be read off a public leaderboard, and all of it can be measured on your own prompts, your own documents and your own acceptance criteria, which is the subject of choosing an on-prem model.

Throughput once more than one person is using it

Prefill and decode are different machines

Prefill processes the whole prompt at once. It is matrix-matrix work that saturates the GPU's arithmetic units, and its cost scales with prompt length. Decode produces one token at a time as matrix-vector work, and for every token it must read every weight the model has out of memory. Decode is bound by memory bandwidth rather than compute, and the arithmetic units sit largely idle while it runs.

An upper bound on single-user decode speed falls straight out of that, by dividing the card's memory bandwidth by the bytes of weights read per token. Bandwidth is a published figure, so this calculation needs no estimation. NVIDIA specifies 3.35TB/s on the 80GB H100 SXM, 4.8TB/s on the 141GB H200, and 864GB/s on the 48GB L40S. Those three numbers span a factor of five and largely determine how fast a given model can generate for one user, which is why the L40S is a capacity card rather than a latency card whatever its memory size suggests. On a card with bandwidth in the region of 1.8 TB per second, a 26 GB BF16 weight set gives a ceiling somewhere near 65 tokens per second, and the same model at 4-bit gives roughly three times that. No real implementation reaches the ceiling, because attention over the KV cache, kernel launch overhead, sampling and scheduler work all consume time the formula ignores, but it tells you the order of magnitude before you sign a purchase order.

Why batching helps, and who pays for it

Because decode is bandwidth-bound, serving several sequences at once is nearly free on the weight side. The weights are read once per step and used for every sequence in the batch, so aggregate tokens per second climbs steeply with batch size at first, then flattens as attention over a growing set of KV caches becomes the bandwidth consumer.

Individual users experience the opposite curve. Their tokens per second falls as the batch grows, because each decode step does more work before their next token appears, and their time to first token rises as well, since their prompt waits behind other prefills. A throughput figure quoted without concurrency and context length attached therefore says nothing. "1,200 tokens per second" at batch 64 with short prompts and "80 tokens per second" for one user at 8k context can be the same card running the same model on the same day.

VIDIZMO's published capacity figure for real-time video is stated the way an inference figure should be, at 32 or more concurrent camera streams per GPU, validated on an RTX 5090, processing every 4th frame by default, with capacity varying by GPU model, frame size, frame sampling rate and stream FPS. Frame sampling and stream rate are to video analytics what concurrency and context length are to text generation, and a capacity number without them is not a number. The deployment side of that decision is worked through in on-prem versus cloud for real-time video AI processing.

Continuous batching in one paragraph

Naive batching collects a set of requests, runs them together, and waits for the longest generation to finish before starting the next set, which leaves the GPU processing a batch of one while a single long answer completes. Continuous batching schedules at the token level instead, admitting new requests into the running batch at each decode step and retiring finished sequences immediately, so the GPU stays full across arrivals of wildly different lengths. Paired with a paged KV cache, which allocates in fixed-size blocks rather than reserving a contiguous maximum-length region per sequence, it removes most of the waste from variable-length traffic.

That waste has been measured, and the figure is large enough to explain why serving systems were redesigned around it. The paper introducing paged attention found that in the serving systems it compared against, "only 20.4% - 38.2% of the KV cache memory is used to store the actual token states". The rest was fragmentation plus space reserved for tokens that were never generated. If you are sizing from the tables earlier in this article, that is the difference between the memory you allocated and the memory you got value from.

The gains from deciding what to run at every token, rather than once for each batch of requests, are larger still. The paper introducing iteration-level scheduling measured a 175-billion-parameter model across 16 GPUs going from 0.185 requests per second to 6.81 at the same median latency, "which is a 36.9× speedup", and reported that the engine alone without the new scheduler accounted for only "up to 47%" of it. Almost all of that improvement came from the order in which work was scheduled rather than from faster low-level code.

Two costs come with those wins, and neither is usually mentioned alongside the speedups. Holding the KV cache in fixed-size blocks "leads to 20-26% higher attention kernel latency" than holding it in one contiguous region, which is a price paid on every token to buy the memory efficiency. Batching the work of reading new prompts alongside the work of generating tokens, which is what keeps throughput high, has been measured to raise the gap between one token appearing and the next "up to 28.3×" against a batch that is only generating (Sarathi-Serve). Throughput is bought with the waiting time your unluckiest requests experience. If your users are people waiting for text to appear rather than a batch pipeline, decide which one you are optimising before you tune anything.

The distance between a demo and a departmental service

Sizing to average load is the most common and most expensive mistake here. A department's usage is not uniform. It clusters at the start of the day and heavily at the end of it, when reports get written and everyone clears their queue before leaving, and scheduled ingestion jobs land in that same window because someone set them to run at close of business years ago.

Queue behaviour turns that concentration into an outage that never shows up as an outage. For simple arrival models the expected wait scales with 1 divided by (1 minus utilization), so the multiplier is 2 at 50 percent utilization, 5 at 80 percent, 10 at 90 percent and 20 at 95 percent. The GPU is not broken at 95 percent, it is busy, and every user experiences busy as broken.

A single card that serves 5 people well can therefore serve 50 badly, and the transition is not linear. The first few extra users cost almost nothing, because batching absorbs them. Past the point where the KV cache is full, the scheduler either preempts running sequences and recomputes them later, cutting throughput exactly when demand is highest, or rejects new requests outright. The system's worst behaviour is reliably scheduled for its busiest hour, and putting a batch transcription pipeline on the same card makes it worse, because batch work can wait and your users cannot.

Which is why production deployments are several machines

A production on-premises AI deployment does not look like a GPU box. It looks like a set of specialised roles. VIDIZMO's on-premises architecture separates the application server, a content processing server handling transcoding, an object detection and computer vision server, a transcription and audio analysis server, a vision description server, an OCR server, a PII detection server, an AI agentic server running chatbot and RAG pipelines, and an AI embedding server generating vectors for search. Those roles have very different resource profiles, and running them on one GPU means the noisiest sets everyone's latency.

The reference hardware for an on-premises Redactor deployment shows the shape of it.

High-speed NVMe storage throughout, on every machine.

Two things there are worth stopping on. Most of the servers carry no GPU, because transcoding, message brokering, database work and the application tier are CPU, memory and I/O problems, and accelerating them spends money the AI nodes needed. And the GPU work is split across two nodes rather than concentrated in one card, which is what allows one queue to back up without stalling the rest of the pipeline.

What the same capacity costs to rent

Anyone approving a capital purchase will ask what renting the equivalent costs, and the list prices are public. Microsoft's retail price feed lists the eight-GPU ND96isr H100 v5 in East US at $98.32 an hour on demand, $18.17 an hour spot, and $551,221 for a one-year reservation. AWS lists p5.48xlarge, also eight H100s, at $55.04 an hour in us-east-1. Google lists a3-highgpu-8g at $88.49 an hour in Iowa, falling to $38.86 on a three-year commitment. Identical silicon, and the on-demand spread runs from $55 to $98 an hour, which tells you the number is commercial rather than technical.

Owning the hardware replaces that with a different bill, and the parts people forget are power and cooling. NVIDIA rates the H100 SXM at up to 700W and the eight-GPU DGX B200 chassis at about 14.3 kW maximum. US industrial electricity averaged 8.71 cents per kilowatt-hour in May 2026, and the state spread in that same table runs from 6.62 cents in Iowa to 22.18 in Rhode Island, so siting moves the operating cost by more than a factor of three before anyone negotiates. Cooling multiplies it again. The Lawrence Berkeley National Laboratory's 2024 data centre energy report puts the US annual average power usage effectiveness at 1.4 in 2023, improved from 1.6 in 2014, and projects 1.15 to 1.35 by 2028. A PUE of 1.4 means four watts of overhead for every ten watts delivered to the silicon.

The comparison that actually decides the question is not price per hour. It is utilization. Rented capacity bills for the hours you use and owned capacity bills for all of them, so a workload that keeps the cards busy favours ownership and a workload with a sharp daily peak and a quiet night does not. Work out your duty cycle before your dollar rate.

What you are actually signing up to operate

The first thing you take on is a set of software versions that all have to agree with one another. The driver, the CUDA toolkit, the container runtime, the serving framework and the kernel libraries each have a supported range, and those ranges do not always overlap. A serving framework release requiring a newer CUDA than the driver your change process approved is routine, and resolving it means upgrading a driver on a machine other teams depend on.

Model updates are not package updates. New weights can arrive with a new tokenizer or a revised chat template, and a changed chat template does not throw an error. It quietly alters output formatting and breaks whatever parses the result three steps downstream. Evaluation is therefore your job now, because a hosted provider improves the model underneath you and absorbs the regression risk of doing so, while self-hosting makes every model change one you justify against a test set built from your own work.

Capacity planning never finishes either. Usage grows, context lengths grow as people learn they can paste more in, and agentic workflows multiply calls per user action, because one request that reads a document, plans, retrieves and drafts is four or more model calls rather than one. Capacity sized for interactive chat does not cover the same team once agents do the work.

In a disconnected environment each of those becomes a manual supply chain, with review and approval for every artefact crossing the boundary, which is covered in air-gapped AI and what still works with no internet at all. And underneath all of it you are the escalation path now, with no provider status page and no support queue absorbing the first hour of an incident. When generation stops at 2am, the person who gets called works for you.

The one structural decision that reduces this load is treating the model as configuration rather than architecture. In AI Intelligence Hub, self-hosted generation runs on the customer's own hardware and the design is model-agnostic, so changing model, or running more than one concurrently for different tasks, is a configuration change rather than a rebuild. That does not remove the evaluation work, but it makes evaluation the entire cost of a model change instead of the cheapest part of it.

Where a hosted API is still the better answer

Some workloads should not run on your own GPUs, and saying so is more useful than pretending otherwise.

• Bursty or unpredictable load is a poor fit, because self-hosting means sizing for peak and paying for that peak continuously, including the hours when the cards do nothing.
• Very long context work is punishing, since the KV cache dominates memory at long context and a hosted provider spreads that cost across many tenants.
• The hardest reasoning and code generation tasks still show a visible gap between the best open-weight models and the frontier, though it narrows each release cycle.
• Small teams without GPU operations capability will find the staff time is the real cost, and it is continuous rather than one-off.
• Prototyping is far cheaper against an API, and finding out whether a use case works at all should happen before a purchase order exists.
• Material you would publish anyway gains nothing here, so sovereignty controls applied to it buy cost without buying security.

Self-hosting relocates the cost of inference rather than removing it. Amortized hardware, power, cooling, rack space and the engineering time to keep the stack current all land per token whether or not anyone counts them that way, and where a commercial platform is involved, on-premises AI usage is metered and carries allowances the same way cloud usage does. Run your own models because the data cannot leave, because the network the system lives on permits nothing else, because the latency budget will not tolerate a round trip, or because the workload is steady enough that owning the capacity is rational. Unlimited free inference is not among the reasons.

A sizing worksheet

Work through this before specifying hardware, and again whenever the workload changes.

1. Write down peak concurrent users rather than total users, and measure a real prompt from the use case to get context length instead of guessing.
2. Calculate weight memory as parameter count multiplied by bytes per parameter, checking the published checkpoint size rather than trusting the arithmetic at 4-bit.
3. Pull layers, kv_heads and head_dim from the model config, then calculate KV bytes per token as 2 × layers × kv_heads × head_dim × bytes per element.
4. Multiply that by target context length and again by peak concurrency for the KV budget you actually need.
5. Add weight memory, KV budget, and 10 to 15 percent of the card for overhead, activations and fragmentation.
6. Compare the total against the card, and if it does not fit, choose deliberately among quantizing weights, quantizing the KV cache, capping context, capping concurrency, or adding a GPU, and record which lever you pulled.
7. Set a latency budget covering time to first token and sustained tokens per second, then load-test at peak concurrency with realistic prompts rather than at a batch size of one.
8. Decide in advance what happens when the queue is full, and get a human to agree to that behaviour before a busy Friday afternoon settles it for you.

Every number in this article is illustrative. The formulas are not, and they will tell you within a few gigabytes whether the deployment you are planning fits the hardware you are about to buy. That afternoon of arithmetic is cheaper than finding the answer in production with a department watching.