Somewhere between legal, compliance and the board, most organizations that want to use AI have been handed a single question they cannot answer confidently. Where does our data go when we use this, and who else can reach it? The question sounds simple and it is not, because the honest answer has several parts that live in different places, some technical, some contractual, and one of them entirely legal.
On-premise AI is the architecture people reach for when that answer has to be clean. It means running every model in the pipeline inside infrastructure your organization controls, rather than sending prompts and documents to a provider's API. Note the word every. A language model is one model class among several, and in most enterprise workloads it is not the one that touches the raw material first. Speech-to-text sees the whole recording. OCR sees the whole page. Object and face detection see the whole frame. Entity detection sees the text that all of those produced. Embedding generation sees the entire corpus. A deployment that self-hosts the language model and calls a cloud service for any of the rest has moved the sensitive content across the boundary before the language model was ever invoked.
Whether you need any of this is a separate question from whether you want it, and the reflex answer usually reaches for more isolation than the real requirement calls for, and occasionally for less.
This guide covers what sovereignty means once you take it apart, the range of architectures between a shared cloud tenant and a fully disconnected environment, the parts of an AI system that quietly leak even when the language model is local, what self-hosting honestly costs, and a question set you can put in front of any vendor including this one.
Sovereignty is three questions that get collapsed into one
Most conversations about AI sovereignty run into trouble because the word carries several meanings at once and nobody says which one they are using. It is more useful to separate them. This framing is ours rather than a standard definition, but it maps cleanly onto the requirements that show up in real procurement documents.
Where the data physically sits
The geographic question. Which country, which region, which building holds the storage volume. This is the question vendors answer most readily, because it is the easiest to satisfy, and it is why every enterprise platform now offers region selection. It is a genuine control and it is the weakest of the three.
Who controls the compute and the models
The operational question. Who administers the machines that run inference, who can push a software update, who holds the encryption keys, and whose staff could in principle read what passes through. A provider running a managed service in your chosen region still operates that service. Answering this question means either taking operational control yourself or constraining the provider's access contractually and technically, and those two are not equivalent.
Whose laws can compel access
The jurisdictional question, and the one that catches people out. Legal reach follows the provider, not only the hardware. The CLOUD Act's operative provision, 18 U.S.C. § 2713, requires a provider of electronic communication or remote computing service to preserve, back up or disclose customer content and records "within such provider's possession, custody, or control, regardless of whether such communication, record, or other information is located within or outside of the United States." Pinning your tenant to Frankfurt answers the geographic question completely and leaves the jurisdictional one exactly where it was.
Compelled disclosure of a non-US customer's content does happen, and it is rare. Microsoft's law enforcement requests report for the second half of 2025 records 190 requests worldwide for accounts associated with enterprise customers, 94 compelled disclosures, and three cases where it "provided content data to U.S. law enforcement related to 3 non-U.S. enterprise customers whose data was stored outside the U.S." One of those customers was in the EU or EFTA. Three in six months is a number you can plan around. Zero is a number only a different architecture gets you.
That distinction is the single most useful thing on this page for a European or Canadian buyer, and it is worked through properly in data residency and jurisdiction for AI, which separates storage residency from processing residency from support access, and covers which contractual controls fix the problem versus which only document it.
A spectrum, not a binary
The choice is not cloud or on-premises. There are four rungs, each removing a specific exposure and each charging for it.
| Level |
What it removes |
What it costs you |
| Region pinning |
Data leaving a chosen geography |
Nothing technical. Leaves the jurisdictional question open |
| Private or dedicated tenant |
Shared infrastructure and neighbouring tenants |
Higher cost, and the environment is still provider-operated |
| On-premises |
Provider access to running systems |
You own upgrades, capacity planning, and hardware |
| Air-gapped |
Network egress entirely |
Model updates and support become a manual supply chain |
Read the table as a ladder where you stop at the first rung that satisfies your actual obligation. A marketing team worried about confidentiality of campaign drafts is answered by region pinning and a contract that forbids training on their data. A hospital handling PHI is usually answered by a dedicated tenant with the right agreements in place. A police agency processing criminal justice information has less latitude than it might expect, because the CJIS Security Policy v6.1 states that "the storage of CJI, regardless of encryption status, shall only be permitted in cloud environments" that "reside within the physical boundaries of APB-member country" and are "under legal authority of an APB-member agency", which means the United States, its territories, Indian Tribes and Canada. A defense programme on a classified network is answered only by air-gap, because nothing else is permitted to exist on that network.
Climbing higher than your obligation is expensive in ways that are not obvious at purchase. Every rung transfers work from the provider to your team, and that work continues for as long as the system runs. The organizations that regret going on-premises are almost never the ones with a hard legal requirement. They are the ones who chose it on principle and then discovered that patching, capacity and model evaluation are now permanent line items.
VIDIZMO supports each rung of that ladder for the same product, including shared and dedicated SaaS, deployment in your own cloud tenant, full on-premises installation in your data centre, and a fully air-gapped configuration with no external network access at all. That range matters for architecture rather than for pricing. A platform that only offers one rung forces the requirement to bend to the product.
The parts of an AI system that leak even when the language model is local
This is the section that matters most to a security officer, and it is where most sovereignty plans are incomplete. Moving one model inside your walls addresses one component. Several others carry sensitive content and are commonly overlooked.
Start with the other model classes, because that is the gap vendors are quietest about. Ask a vendor whether AI runs on your infrastructure and the answer will be about the language model, because that is the part the market talks about. Then ask separately where speech-to-text runs, where OCR runs, where object and face detection run, where translation runs, where entity and PII detection run, and where embeddings are generated. Each of those is a model, each takes the raw material rather than a summary of it, and each can be wired to a cloud endpoint independently of the language model. A recording sent to a hosted transcription service has left your environment in full, in the original speaker's voice, before any language model was asked anything. The question that discriminates between vendors is not whether they can run a model on your hardware. It is which stages of the pipeline still make a network call.
Prompts and queries carry the payload. The sensitive material is rarely the model weights. It is the case file pasted into the context window, the patient note, the paragraph of an unreleased contract. If retrieval is local but generation calls a hosted API, every retrieved passage travels to that API as part of the prompt. The architecture diagram looks sovereign and the data flow is not.
Embeddings and vector indexes are derived copies of your corpus. An embedding is a lossy numerical representation, which people take to mean it is safe to store anywhere. Vec2Text recovers 92% of 32-token inputs exactly from their embeddings, and against clinical notes from MIMIC-III it recovered 94% of first names, 95% of last names and 26% of the notes verbatim. The attack needs query access to the same embedding model and degrades sharply with passage length, so it is a bounded result rather than a general break. It is more than enough reason to treat the vector index as a copy of the material it was built from and place it under the same controls as the originals.
Telemetry and logs are the quiet channel. Vendor software installed inside your data centre frequently reports usage, errors, crash dumps and diagnostics outward by default. Crash dumps in particular can contain fragments of whatever was in memory. Ask specifically what leaves, on what schedule, to which destination, and whether it can be disabled without losing support.
The model update path is a network dependency that outlives every other control. A system that pulls new model weights, container images or threat signatures from the internet has a permanent outbound route by design. In a disconnected environment that route does not exist, which is why updates become a physical process with its own review and approval steps rather than a background task.
If your deployment will be genuinely disconnected, air-gapped AI and what still works with no internet at all goes through which capabilities survive, which change shape, and which break outright.
Where this sits relative to national sovereign AI
Many readers meet the term sovereign AI first in its national sense, where a government funds domestic GPU capacity, models trained or tuned on local language and local data, and systems that answer only to local law. Those programmes are real and well funded, and they solve two of the three layers a country needs.
What they generally do not fund is the application layer. A national programme produces compute and it produces open-weight models, and it leaves a gap where the software an agency actually wants should be, because most enterprise software cannot run disconnected from its vendor. An agency handed a domestic GPU cluster and a capable open-weight model still cannot run a case management system, an evidence platform or a document processing pipeline on it unless that software was built to operate without calling home.
The same structure applies one level down, at a scale a single CIO can act on. The organizational version of the national question is identical in form and answerable this quarter rather than this decade. National sovereign AI programmes and what they mean for your organization works through what the funded programmes cover and where the constraint actually sits.
Model-agnosticism is the sharpest expression of sovereignty at the scale of a single organization. An architecture where the model is a configurable component, and where a hosted model can be swapped for a self-hosted open-weight one without rebuilding the system around it, buys something more durable than deployment flexibility. It buys the ability to survive your vendor's decisions, including price changes, deprecations, and terms-of-service revisions you had no part in. Where that portability really lives, and why embeddings are the hardest part to move, is covered in model portability and avoiding AI vendor lock-in.
What running AI yourself honestly costs
Open-weight models have closed most of the capability gap for the tasks enterprises actually run. Transcription, extraction, classification, summarization and retrieval-grounded question answering are all served well by models you can download and run. The gap that remains shows up on long-context reasoning and on the hardest coding and analysis work, and it narrows with every release cycle.
The cost is not the model. It is everything around it.
GPU memory is the binding constraint on what you can serve, and the first half of the sizing question is a multiplication anyone can do in their head. Weight memory is the parameter count times the bytes each parameter occupies, and the bytes are set by the precision you serve at. FP32 is 4 bytes, FP16 and BF16 are 2, INT8 is 1, and INT4 is half a byte. Nobody serves at FP32, so treat 2 bytes as the honest starting point and anything below it as a decision you are making about quality.
Work it through on a 250-billion-parameter model. At 2 bytes that is 500 GB of weights, at INT8 250 GB, and at INT4 125 GB. Then divide by what a card actually holds. NVIDIA publishes 80GB per H100 SXM at 3.35TB/s, 141GB per H200 at 4.8TB/s, and 48GB on the L40S.
| Precision |
Weights, 250B model |
H100, 80GB |
H200, 141GB |
L40S, 48GB |
| FP16 or BF16 |
500 GB |
7 cards |
4 cards |
11 cards |
| INT8 |
250 GB |
4 cards |
2 cards |
6 cards |
| INT4 |
125 GB |
2 cards |
1 card |
3 cards |
Those counts are for weights alone, rounded up, and they are a floor rather than an answer. The key-value cache sits on top and grows with context length multiplied by concurrent requests, which is why a model that runs comfortably in a single-user demo falls over when a department uses it. At long context the cache stops being a rounding error and starts rivalling the weights. Activations during prefill and the runtime's own overhead take another slice, and the research behind modern inference servers found that in the serving systems of the time only 20.4% to 38.2% of the memory reserved for the key-value cache actually held token state, so reserved and useful are not the same number. As a planning rule rather than a measurement, size for the weights and then assume you need meaningfully more again before the system serves anyone concurrently. Working out how much more, from your own context lengths and concurrency, is the arithmetic in the cluster article below.
Quantization is therefore the lever that moves card count most, and the quality it costs is smaller than most people expect. On a 175-billion-parameter model, four-bit quantization with a good method cost 0.25 perplexity or less against full precision, while three-bit cost 0.3 to 0.6 points, and naive round-to-nearest quantization collapsed entirely at three bits, taking one 176-billion-parameter model from 8.11 perplexity to 571 (GPTQ, arXiv 2210.17323). Method matters more than bit width. How much quality your workload loses depends on the task, and you only find out by measuring on your own data.
Power is the cost line that surprises people who have only bought software. A single H100 SXM is rated at up to 700W and an eight-GPU chassis at roughly 14.3 kW, before cooling, which the Lawrence Berkeley National Laboratory's 2024 data centre energy report puts at a US average power usage effectiveness of 1.4. Where you site the hardware then moves the bill by more than a factor of three, because industrial electricity in the EIA's May 2026 table runs from 6.62 cents per kilowatt-hour in Iowa to 22.18 in Rhode Island against an 8.71 cent national average. The full arithmetic, including what the same capacity rents for on the major clouds and how to work out concurrency from memory rather than guessing, is in running LLMs on-premises and the realities nobody advertises.
You also inherit evaluation, and there is less published help with it than you would expect. Open-weight developers report benchmark scores on their own model cards, but they report different benchmarks and different variants from each other, and the leading hosted vendors have largely stopped publishing comparable numbers at all. A like-for-like comparison assembled from vendor documentation is therefore not available to you, whatever a procurement template asks for. Self-hosting also means you decide when to move rather than having a provider improve the model underneath you, and deciding requires a test set built from your own work. Choosing an on-prem model covers how to build one that tells you something.
And you inherit operations. Patching, GPU driver management, capacity planning as usage grows, and a model update process that is now yours. For an air-gapped deployment, add a review step for every artefact crossing the boundary.
Set against that, some organizations should not do this at all. If your AI use case is drafting marketing copy, summarizing public documents, or answering questions about information you would publish anyway, use a hosted API and spend the money and the headcount on something that matters more. Sovereign architecture is a control for material where exposure carries legal, safety or national-security consequences. Applying it to everything is a way of spending a security budget without buying security.
Matching the requirement to the data
Regulators rarely mandate a deployment model directly. They mandate controls, and the deployment model is what makes the controls achievable. The mapping below reflects what tends to be expected rather than what any specific statute names.
| Data type |
What is typically expected |
Notes on posture |
| Public or already-published material |
Any deployment model |
Sovereignty controls add cost without adding protection |
| Commercially sensitive internal material |
Region pinning plus contractual limits on training and retention |
Verify the training clause in writing rather than in marketing copy |
| Personal data under GDPR or similar regimes |
Regional processing plus a defensible transfer position |
Geography alone does not answer the jurisdictional question. Article 83(5) puts transfer breaches in the top penalty tier, up to 20,000,000 EUR or 4% of worldwide annual turnover |
| PHI under HIPAA |
Dedicated environment, a BAA where required, and full audit logging |
VIDIZMO provides data-protection, security, redaction and anonymization capabilities that help customers meet HIPAA obligations, and can enter into a BAA where required |
| Criminal justice information |
CJIS Security Policy controls, in a cloud environment inside an APB-member country, commonly on-premises or government cloud |
v6.1 requires FIPS 140-3 certified modules with a symmetric key of at least 256-bit strength for CJI at rest outside physically secure locations, and warns that FIPS 140-2 certificates stop being acceptable after 21 September 2026. VIDIZMO is CJIS Security Policy-aligned. The agency holds the compliance obligation. No vendor holds a CJIS certification, because no certifying body issues one |
| Federal systems requiring FedRAMP |
Deployment on FedRAMP-authorized infrastructure |
VIDIZMO's processes and software are aligned with NIST SP 800-53 Rev. 5, and the platform is deployable on FedRAMP High-authorized infrastructure, either Project Hosts' FedRAMP-authorized environment or Microsoft Azure Government with agency sponsorship. VIDIZMO does not hold its own FedRAMP authorization |
| Unclassified national security systems at IL5 |
Government cloud at the appropriate impact level |
The DoD Cloud Service Provider SRG v1r7 defines Impact Level 5 as unclassified national security systems and national security information rather than simply higher-sensitivity CUI, and states that "per CNSSP 32, the minimum requirement for all unclassified NSS is equivalent to the FedRAMP High baseline with the additional overlays and NSS controls in CNSSI 1253 Appendix D". IL4 and IL5 workloads are supported via Microsoft Azure Government |
| Classified up to SECRET at IL6 |
Air-gapped, or a dedicated accredited environment |
The same SRG allows access "via one or more private SIPRNet connections or approved CNSSP-11 circuits" and states an IL6 offering "is not considered 'commercially available'" |
Two of those rows are worth expanding on their own. What the CJIS Security Policy actually demands of an AI system, as opposed to what vendors claim about it, is in what CJIS actually requires when AI touches criminal justice data. How those control requirements turn into architecture decisions and authorization boundaries is in the sovereign AI compliance architecture guide.
Ten questions to put to any AI vendor
Lift these into a questionnaire or an RFP. They are ordered so that the early answers make the later ones easier to interpret, and the fuller version with the answers that should concern you is in the security questionnaire for AI vendors.
- Where is my prompt processed, and does that location change depending on which feature I use?
- Is any customer content used to train, tune or evaluate your models, and what written commitment covers that?
- What data leaves my environment as telemetry, diagnostics or crash reports, on what schedule, and can it be disabled without losing support?
- Which subprocessors touch my content, in which jurisdictions, and how am I notified when that list changes?
- Where does my vector index live, and is it protected to the same standard as the source documents it was built from?
- Can the system run with the network cable unplugged, and which specific features stop working when it does?
- How do model updates reach a disconnected deployment, who reviews the artefact, and what does the approval trail look like?
- Which model version produced a given output, and is that recorded alongside the output?
- Who holds the encryption keys, and what standing access do your staff have to a running production system?
- If I decide to move to a different model or a different vendor, what specifically comes with me and what has to be rebuilt?
Question eight is the one most often missing from a questionnaire and most often asked later by a court. When an AI output is relied upon in a legal or regulatory proceeding, the record has to show which model and version produced it, from which input, under which prompt, run by whom, and what a human reviewer changed. Chain of custody for AI outputs sets out what that record needs to contain.
How VIDIZMO runs in these environments
The reason this guide can be specific about disconnected operation is that the VIDIZMO platform is built to run that way.
VIDIZMO AI Intelligence Hub deploys as SaaS, in a customer-owned cloud tenant, on-premises in a customer data centre, or fully air-gapped with no external network access. The platform is model-agnostic and can run open-weight models of the customer's choosing, self-hosted, with the model treated as a configuration choice rather than an architectural assumption and multiple models able to operate concurrently. Managed provider APIs remain tenant-configurable options for deployments that want them, which is a different thing from depending on one.
That self-hosting covers every model class rather than the language model alone. Speech-to-text, translation, OCR, object and face detection, entity and PII detection, embedding generation and the language models all run inside the customer's environment, and the default posture across the stack is self-hosted rather than hosted. In an air-gapped deployment all AI processing runs on-premises and no data leaves the environment. The graph-based workflow designer behaves identically there, with no dependency on external APIs.
The hardware footprint is smaller than the rented-cluster figures above suggest, because running a document and media pipeline is not the same job as training. A representative on-premises Redactor build uses a web and database server at 16 cores and 32 to 64GB of RAM, an encoding server at 16 cores with an 8GB-VRAM GPU, two AI processing servers at 8 to 12 cores with 32GB-VRAM GPUs, and a broker at 4 cores, all on NVMe storage. Sizing moves with throughput and with how many model classes you run concurrently, and it is a purchase order rather than a data centre programme.
On the compliance side, VIDIZMO is ISO/IEC 27001:2022 certified under certificate RA-2507091. Audit logs are held in append-only Azure SQL ledger tables that are tamper-evident, so any modification is cryptographically detectable, and are archived under a locked time-based WORM retention policy. VIDIZMO staff hold no standing access to customer environments, with break-glass access only, time-bound, MFA-enforced and logged. AI does not train on customer data by default, and doing so requires explicit written authorization.
For evidence and records work specifically, DEMS produces a chain of custody report covering IP address, username, timestamp, event type and event details, exportable as PDF or CSV, with SHA-384 hash-based tamper detection.
Where to go deeper
The clusters below take one part of this argument each and go considerably further than a pillar can.
On architecture and operations, read running LLMs on-premises for sizing, choosing an on-prem model for selection, air-gapped AI for disconnected operation, agentic AI workflows on infrastructure you control for multi-step automation, and model portability for keeping your options open.
On law and compliance, read data residency and jurisdiction, what CJIS actually requires, the compliance architecture guide, chain of custody for AI outputs, and the AI vendor security questionnaire.
On the wider context, read national sovereign AI programmes and sovereign AI for video and document intelligence.
For deployment economics and vertical applications, read cloud versus on-premises video hosting costs, private cloud digital evidence management, on-premises redaction software architecture, on-prem versus cloud for real-time video AI, and data residency and compliance across deployment models.
