Local language models or cloud models – how do you choose?

A local language model provides greater technical control over where data is processed. An external model via cloud or API can provide higher capacity and faster access to new features. No option is automatically the safest or the best.

The choice should follow information classification, task, quality requirements, response time, cost, and the organization's ability to operate the solution. Often, the result is a hybrid where different models are used for different types of data.

Three common ways to run language models

External API service

An external API service means that the application sends a request to a model operated by the provider. The organization avoids model servers and gains quick access to updates. Control and terms depend on the specific enterprise or API service.

Private cloud environment

A private cloud environment means that the service runs within a more restricted cloud setup with an agreed region, identity, and network controls. The model can still be a managed service, but the application and data flow are integrated into the organization's cloud architecture.

Local or dedicated operation

Local or dedicated operation means that the model and surrounding components run on infrastructure controlled by the organization or its operating partner. This could be in an on-premises data center, a Swedish operating environment, or a dedicated instance.

The boundaries are not absolute. A solution can use a local document index but an external model, or local models for sensitive data and external models for open material.

When is a local model relevant?

Local operation is particularly relevant when information must not leave a controlled environment, when the organization needs full control over retention, or when network isolation is a requirement. It can also provide predictable access for large, stable volumes.

The benefits come with responsibility. The organization needs to manage model files, patching, GPU capacity, scaling, monitoring, and security updates. Furthermore, an open-source model may have license terms and limitations that must be reviewed.

Local operation does not automatically solve security in document indexes, logs, or user interfaces. The entire chain must be covered by identity, authorization, encryption, backup, and incident management.

Secure document analysis with AI in regulated businesses

When is a cloud model a better fit?

Cloud models are suitable when high quality, broad language capability, multimodality, or quick access to new features carry significant weight. They can scale without the organization sizing its own hardware and are often priced per use.

Enterprise offerings can provide terms ensuring that customer data is not used for training. OpenAI, Microsoft, Google Cloud, and AWS publish separate descriptions of data handling for their enterprise and cloud services.[1–4] These need to be read for the specific product and checked against contracts and configurations.

An external service is less suitable if the region, subcontractors, or retention cannot be reconciled with the information classification. It can also create cost risks for very large volumes or vendor lock-in if the application is built tightly around unique features.

The right model is not a matter of brand. It is a consequence of data, task, and requirements.

Data handling and vendor terms

Do not only assess whether input data is used for model training. Map out where prompts, documents, metadata, output data, and logs are processed and stored. Check region, retention, sub-processors, support access, deletion, and export.

For personal data, the organization needs to determine the purpose, legal basis, and responsibility. IMY emphasizes data protection throughout the entire lifecycle.[5] Local operation changes the supply chain but does not remove GDPR or the requirements for data minimization and security.

What happens to corporate data when you use AI?

Quality, capacity, and response time

Larger models often provide higher general capacity but can be slower and more expensive. Smaller models can perform very well in narrow tasks, especially in combination with RAG, structured output, and clear instructions.

Test models on the organization's own representative tasks. Measure accuracy, source grounding, correct refusals, language, response time, and cost. A standardized benchmark from the provider says little about how the model handles your tables, professional terminology, or security requirements.

Capacity planning for local operation needs to take into account concurrent users, context length, and peak loads. For API services, you need to manage quotas, network dependencies, and the provider's availability levels.

What is RAG? How companies use their own data in a language model

Cost and operational responsibility

Compare total cost, not just price per token or server. Local operation includes hardware or operating environment, energy, licenses, operational expertise, monitoring, and spare capacity. Cloud operation includes usage, data transfer, logging, agreements, and cost control.

The cost structure changes with volume. A local model can become economical with high and steady loads, while APIs are better suited for fluctuating usage. Model size, context, and output length affect both alternatives.

Hybrid and model-neutral architecture

A model-neutral architecture separates the application's workflow from a single model. This allows the organization to route tasks according to information classification and requirements, as well as test a new model before it replaces the old one.

This requires a common abstraction layer, version-controlled prompts, a test library, and tracking per model. Switching is not free: output formats and behaviors change. Regression testing must therefore be part of the model as a lifecycle component.

Buy a service, build your own, or use an AI platform?

Read more about Violet AI Platform

Twelve questions for choosing a model

1. What information classification will be processed?

2. Are there personal data or confidentiality requirements?

3. What quality is required and how is it measured?

4. Are multiple languages, images, or audio needed?

5. How many concurrent users and what response time are required?

6. Can the organization accept dependence on the internet or a provider?

7. Which regions and subcontractors are permitted?

8. What retention and logging are needed?

9. Is there expertise and budget for local operation?

10. How does the volume of use vary?

11. How do we switch or restore a model?

12. How are security and quality tested after each change?

AI Act, NIS2, DORA & GDPR – which requirements affect your company's AI solution?

Sources

1. OpenAI, Enterprise privacy:
   https://openai.com/enterprise-privacy/

2. Microsoft, Azure OpenAI data privacy:
   https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy

3. Google Cloud, Zero data retention:
   https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/zero-data-retention

4. AWS, Data protection in Amazon Bedrock:
   https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html

5. IMY, Guidance on GDPR and AI:
   https://www.imy.se/verksamhet/dataskydd/innovationsportalen/vagledning-om-gdpr-och-ai/