Choosing an On-Prem Model: A Practical Evaluation Framework

Choosing which open-weight model to run on your own hardware is a measurement problem rather than a research problem. The decision that holds up is made against a test set built from your own documents and your own queries, scored at the quantization you will actually deploy and the concurrency you will actually see, with someone from legal reading the licence before the pilot rather than after it.

This is the selection half of a self-hosted deployment. The sizing half, GPU memory arithmetic and throughput under load, is in running LLMs on-premises. The wider case for running AI inside infrastructure you control, including where it is the wrong answer, is in on-premises AI.

Why public leaderboards mislead for this decision

Leaderboards are a reasonable way to find out which models are worth a look, and past that point they stop being evidence about your own deployment, for five reasons that stack on top of one another.

A benchmark score can measure recall rather than capability, because public test sets circulate on the open web and models are trained on scraped web text, so an item and its answer are often already in the training data by the time the model is asked the question. The cleanest measurement of this commissioned an entirely new benchmark to compare against an old one. GSM1k was built to mirror the established GSM8k grade-school maths benchmark on solve rate, solution length and answer magnitude, then run against leading open and closed models. The authors observed "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes", and found a positive relationship, Spearman's r² of 0.36, between how likely a model was to generate a GSM8k example verbatim and how far its score fell.

Read the paper's own qualification alongside the finding, because it changes what you do about it. The authors also report that "many models, especially those on the frontier, show minimal signs of overfitting, and all models broadly demonstrate generalization to novel math problems". Contamination is real, it is uneven between model families, and you cannot tell from the leaderboard which family you are looking at. An 8% overstatement is not fatal to a shortlist. It is fatal to a decision made on a two-point gap.

The reason publishers cannot fully rule it out is that the standard defence does not work. Decontamination is usually done by string matching on n-gram overlap, and research on rephrased benchmark samples showed that "simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures", to the point that "a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4". Applying a stronger detection method to real pre-training corpora, the same authors "identified that 8-18% of the HumanEval benchmark overlaps" with widely used datasets. The contamination that matters is the kind the standard tool cannot see, which is why a publisher's decontamination statement is weaker evidence than it sounds.

Benchmark tasks are built so a machine can mark them, which forces them into short self-contained problems with one checkable answer, and almost nothing an organization actually does has that shape. Your work runs over long documents rather than a paragraph, is grounded in retrieved passages, is constrained to a format some downstream system has to parse, and is judged on whether a specialist would put their name to it. Skill at competition mathematics says nothing about pulling a policy number off a scanned claim form without inventing a plausible one.

Preference rankings measure which answer people liked the look of rather than which one was right, and the people who run those boards have measured how large that effect is. Arena-style boards ask which of two answers people prefer, and LMArena's own style-control analysis fitted a coefficient of 0.249 to answer length against 0.031, 0.024 and 0.019 for markdown lists, headers and bold text, concluding that "length was the dominant style factor". Controlling for style moved real models several places, in both directions. The team states its own caveat, that "our analysis is still observational" and confounders such as a genuine correlation between length and quality are not ruled out. The same pattern holds when the voting is done by other models rather than by people, which is now the usual arrangement. SOS-Bench reports that "LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following" and that "LLM-judges have powerful implicit biases, prioritizing style over factuality and safety". Its measurements put style at R=0.999 against the overall verdict and conciseness at R=0.114, and injecting a sarcastic tone cost 96% of the score where making the answer factually wrong cost 13%. Any ranking that penalises tone far harder than error is not measuring what you are buying for.

The model that was ranked is not the model you will run, because published scores describe it at full precision and your hardware will require a quantized version, meaning one whose numbers are stored at lower precision so the whole thing fits in the memory you have. Quantization is not a uniform tax that every model pays at the same rate. Two models with near-identical full-precision scores can degrade very differently at four bits, and the damage concentrates in specific behaviours rather than spreading evenly, commonly long-context recall, adherence to an output schema, arithmetic, and languages that were thin in the training mix. Method and calibration data shift the result again, so a community re-upload and the publisher's own quantization are not interchangeable.

There is a fifth problem that undermines the comparison itself, and it is the one least often mentioned. A model's score moves under changes that carry no meaning at all, such as how the instructions around the question are punctuated and spaced. Research on prompt formatting found widely used open models "extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points" on a single 13-billion-parameter model, with sensitivity persisting as model size, few-shot count and instruction tuning increased. The authors also report that format performance "only weakly correlates between models", which they say "puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format". That is precisely what a leaderboard is. Two models a few points apart on a public board may swap places entirely under your prompt template.

Underneath all five sits the plainest problem of the lot, which is that no public benchmark contains your case notes, your claim forms, your radio traffic or your redaction rules. Use leaderboards to pick four candidates worth testing, then stop looking at them. The corollary is that your own evaluation has to fix its prompt format and report the format it used, because you have just inherited the same instability the leaderboards suffer from.

Start from the task, not the model

Before any candidate gets downloaded, write the task down. A written definition surfaces the disagreements in the room while they are still cheap, and it answers what goes in, what comes out, what a correct output looks like in enough detail that two people would agree, and what a failure costs.

The last two rows do most of the work. Summarizing an internal meeting is recoverable, because the reader attended it and will notice a wrong summary. Flagging personal information in a document about to be released publicly is not, because the failure is found by whoever receives the unredacted material. Recoverable tasks can be optimized on average quality. Unrecoverable ones should be optimized on worst case, which usually means favouring recall and accepting false positives that a reviewer clears.

Write down what the system is allowed to do when it is unsure, because that one permission changes how hard the task is. A model that may decline, or hand the case to a person, has an easier job than one that must produce an answer every time, and scoring the two the same way will mislead you.

Building a representative test set from your own data

The test set is the asset. It outlives every model you evaluate with it, and it is what lets you upgrade later without guessing.

How many examples

Enough that a gap between two candidates means something rather than being noise in the sample. With a pass or fail metric, an accuracy near 80% on 100 examples carries a standard error of about four percentage points, so the ninety-five percent interval is roughly plus or minus eight, and candidates scoring 78% and 84% are not separated by a set that small. Uncertainty falls with the square root of the count, so halving that interval takes around 400 examples.

Scoring every candidate on the same items and comparing per item rather than by headline average cancels out how hard each individual item is, which detects a real difference with far fewer examples. A working target is 100 to 300 labelled examples per distinct task, weighted higher where failures are unrecoverable. Per distinct task matters, since 200 examples spread across five different jobs is 40 each and distinguishes nothing. Keep a smoke set of about 20 for fast iteration during prompt work.

Sampling so the hard cases survive

Random sampling fills the set with whatever your organization handles most of, which is usually the easy case, and the average then hides the failures that matter. Sample in strata instead, which means dividing the inputs into groups first and drawing from each group deliberately. Bucket inputs by what plausibly affects difficulty, usually source system, media type, length, language and input quality, then sample within each bucket and overweight the difficult ones relative to their real frequency, recording the true frequencies so you can reweight back later.

Then go hunting for the inputs nobody wants in the set, because those are the ones that decide whether the system survives contact with real work. Collect low-resolution scans and faxed pages, photographs taken at an angle under bad lighting, handwriting in a form field, tables broken across a page boundary, audio with heavy accents or two people talking over each other, paragraphs where two languages appear together, and files that are technically valid and semantically nonsense. Add a few questions your corpus cannot answer, since whether a candidate says so or invents an answer is one of the higher-value observations available.

The labelling burden, honestly

Labelling 200 examples to a standard a domain expert would defend is days of that expert's time, not an afternoon of an engineer's. Budget it explicitly and get the named expert's manager to agree before the project starts, because this is where evaluation programmes quietly die.

The bill comes down in a few ways. Labels are built once and reused across every candidate, every quantization, every serving engine and every future version. Where a task has many valid answers, mark outputs acceptable or not instead of authoring a reference answer, and where even that is hard, pairwise comparison removes the need for a reference entirely. Freeze the set once built, holding back a slice you never look at during prompt tuning, so something clean remains to catch a prompt overfitted to your own examples.

How to measure

Match the measurement to the task and use the cheapest method the task tolerates.

Deterministic metrics come first, because they run unattended and can therefore run every time anything changes. Field extraction is scored by normalized exact match per field, comparing what the model pulled out against the expected value once both have been written the same way. Transcription is scored by word error rate, computed separately for clean and difficult audio rather than pooled, since one combined figure hides exactly the recordings you were worried about. Entity and PII detection are scored by precision and recall per entity type, meaning how much of what the model flags is correct and how much of what is present it finds, because a model can be strong on names and weak on account numbers while the pooled figure looks fine. Retrieval is scored by hit rate at k, the share of queries where a correct passage lands in the top k results, measured apart from generation so you can tell which half is failing.

Human review is the only option where the output is open-ended, because no automatic check knows what a good answer looks like when several different answers would all be good. Write a rubric with concrete criteria rather than asking for a general quality score, and keep the scale short, since a four-point scale with written anchors produces more consistent labels than a ten-point scale with none.

Model-graded evaluation makes open-ended scoring affordable and has failure modes worth designing around. Judges reward length and confident phrasing, and they favour outputs resembling their own style, which biases any comparison involving the judge's own family. A judge also cannot catch a factual error about your domain unless the supporting material is in its context, so give it the source passage and ask for a verdict grounded in that passage. Validate it against a human-labelled subsample first, because a judge that disagrees with your reviewers is scoring its own preferences rather than your standard.

Pairwise comparison, where a reviewer sees two outputs and picks the better one, is the cheapest reliable option for subjective work, because people are far steadier at choosing between two things than at scoring one thing out of five, and nobody has to write a reference answer first. Blind the reviewer to which candidate produced which output, randomize left and right position, allow a tie, and record which reviewer judged which pair.

Before you act on any of these numbers, check that your own labels are consistent enough to carry them, because a metric can never be more reliable than the judgements underneath it. Have two reviewers independently label an overlapping subsample and compute how often they agree, corrected for the agreement you would expect from chance alone. If they agree only 70% of the time, no metric built on those labels can separate candidates differing by less than that gap, and the honest response is to fix the rubric and relabel rather than report the number. Low agreement usually means the rubric is underspecified rather than that the reviewers are careless.

How expensive this exercise is depends partly on the platform you deploy into, because that decides whether swapping one candidate for another is an infrastructure project or a form field. Where the model is welded into the application, comparing candidates means standing up parallel environments and running them side by side. Where the model is a configuration choice, the comparison costs a setting change. VIDIZMO AI Intelligence Hub is model-agnostic and multiple models can operate concurrently, so candidates can be compared in place and different models can serve different tasks in the same deployment, served from the customer's own hardware. Embedding providers are configured independently of generation, so an embedding model can be held stable while generation candidates are swapped underneath it.

Licence terms vary far more than teams expect

Open weights and open source are different things, and conflating them is the most common legal error in this process. Open weights means the parameters are downloadable. Open source has a published definition, and it is stricter than most people assume. The Open Source Definition sets out ten criteria, of which three decide this question. Criterion 5 requires that a licence "must not discriminate against any person or group of persons". Criterion 6 requires that it "must not restrict anyone from making use of the program in a specific field of endeavor". Criterion 8 requires that rights "must not depend on the program's being part of a particular software distribution". An acceptable-use policy naming prohibited application categories fails criterion 6 on its face, and a user-count threshold above which separate terms apply sits awkwardly with criterion 5. Several widely deployed model licences carry both. They are open weights and they are not open source, whatever the announcement said, which is why legal review belongs in shortlisting rather than in approval.

Permissive licences applied to weights behave the way engineers already expect from open-source code, granting broad rights and asking mainly for attribution in return. The Apache License 2.0, from January 2004 and the most common of them, permits commercial use, modification and redistribution, and its section 4 conditions amount to giving recipients a copy of the licence, marking modified files with "prominent notices stating that You changed the files", retaining copyright, patent, trademark and attribution notices, and carrying forward any NOTICE file. Those are bookkeeping obligations rather than constraints on what you may build.

Community and source-available licences grant broad use while attaching conditions, commonly an acceptable-use policy prohibiting named application categories, a threshold on user count or revenue above which separate terms apply, limits on offering the model as a hosted service to others, and a requirement to pass the same terms downstream to anyone you distribute to. The thresholds are what catch teams out, because terms that are comfortable at pilot scale change character if the deployment succeeds or your organization merges with a larger one.

Other licences constrain the outputs or the derivatives, covering whether outputs may be used to train another model and whether a fine-tuned derivative must carry a particular name, an attribution or the same terms. Read this category closely if you plan to fine-tune on internal data and treat the result as your own asset. Research-only and non-commercial weights are simpler, provided somebody notices before the pilot becomes production.

Two mechanical points apply throughout. Fine-tunes inherit the base model's terms, so the chain has to be walked back to the original. And community re-uploads, including many quantized variants, are not always distributed with the licence intact, so take weights from the original publisher and archive the licence text alongside what you deployed.

A model that fails legal review after six months of prompt engineering costs you the six months, not the model. Government and regulated buyers should look at the publisher's jurisdiction and any export rules attaching to it in the same pass.

The total cost of a model that is free to download

The weights are free to download and almost nothing downstream of them is free. Hardware is the visible cost, covered in the sizing article, where memory rather than raw compute usually turns out to be the binding constraint. Building and labelling the test set is the cost that appears on no quotation and is paid in the time of people who are hard to free up.

Re-evaluation is the cost nobody forecasts, because it arrives again every time anything underneath the system moves. A new model version, a different quantization, a serving-engine upgrade, a changed system prompt and a changed retrieval configuration are all reasons to re-run the suite, and self-hosting means you own the decision to move. An unattended deterministic suite costs almost nothing to repeat, while one needing human reviewers costs the same every time.

Reindexing is the largest single switching cost in a retrieval system, and it attaches to the embedding model rather than the generation model. Vectors from one embedding model are not comparable with another's, so changing embedding models means regenerating every vector in the corpus. That cost scales with corpus size rather than query volume, so it grows quietly for as long as the system runs and is highest exactly when you want to switch. Choose the embedding model with more care than the generation model, and see model portability for where lock-in forms.

Migration overlap belongs in the plan too, since comparing the outgoing and incoming models means holding both in memory for the duration.

The scoring template

Copy this table, set the weights before you look at any results, and agree them with whoever owns the outcome rather than with whoever runs the evaluation. Weights chosen after the scores arrive are not weights, they are a justification.

Tune the weights to your situation. A high-volume interactive assistant pushes weight toward latency, while a batch pipeline running overnight moves that weight to accuracy. Where a criterion does not apply, for example language coverage in a single-language deployment, remove the row and redistribute its weight rather than scoring every candidate a 3. Treat licence fit and memory footprint as gates rather than weights if a failure there is disqualifying, because a weighted average ranks survivors well and expresses a hard constraint badly.

Safety behaviour runs in both directions. Over-refusal is the more common problem in enterprise and public-sector work, where models decline to process incident reports, medical records, investigative material or safety complaints because the surface content resembles what a consumer guardrail was built to block. Test both directions on your own material rather than on generic prompts.

A worked example

Two candidates, scored one to five against the weights above.

Candidate B is the better model on paper and the worse choice here. It wins on full-precision accuracy against the test set, loses most of that lead once quantized to the precision the available GPUs require, and still needs more memory than the hardware comfortably provides. Candidate A gives up a little headline accuracy, keeps it after quantization, fits, and clears legal review without conditions.

The number is not the decision, and what it buys you is a decision somebody can audit and an argument that has moved onto the weights, which is where the argument belongs. Read the low scores individually rather than trusting the total, since any criterion where a candidate scores a 1 deserves a conversation about whether it should have been a gate. Then record the table alongside the model version, the quantization, the serving engine version and the date of the run, because the next person to ask why you chose this model will be asking months from now.