Local AI vs API AI: not an ideology choice.
You pick the architecture with the lowest total cost able to meet the required quality, privacy and availability. This is the real map of when a local model adds up and when an API model is the more efficient path.
Criterion-by-criterion comparison, no strawman on either side
| Criterion | Local AI (SLM) | API AI |
|---|---|---|
| Marginal cost per call | Apparently zero, but CAPEX and dedicated staff are paid regardless. | Predictable per token. Lightweight models cost cents per thousand calls. |
| Latency | Very low on the same network. Great when the user is right there. | Good on lightweight models, medium on frontier. Depends on endpoint region. |
| Privacy and compliance | Data does not leave your network. Hard requirement in health, defense, government. | Major providers offer no-retention contracts and EU regions. |
| Quality on complex tasks | Limited. Models that fit on a reasonable server do not rival frontier. | State of the art on demand. No infrastructure. |
| Maintenance and staffing | Requires MLOps, model updates, monitoring, hardware, backups. | The vendor maintains the model. You maintain your orchestration. |
| Availability during network outages | Works without internet. Real edge on shop floor, field, offline. | Depends on the link. Cache and deterministic fallback are mandatory. |
When privacy is a hard requirement, when volume is massive and stable, or when you must operate offline: local AI usually pays off. Otherwise, an API model on a neutral architecture is cheaper to run and faster to evolve.
Frequently asked questions
When does a local AI model (SLM) make sense?
When privacy is a hard requirement (health, defense, public sector, banking), when volume is massive and stable (millions of calls per month on repetitive tasks), or when you must operate without internet (shop floor, field, ship).
When does an API model make sense?
When the task demands state-of-the-art, when volume is variable or low, when the company does not want in-house MLOps staff, or when manageable latency is acceptable. In many cases, API with a no-retention contract is cheaper and more flexible.
Which one is cheaper?
Depends on volume. A lightweight API model costs cents per thousands of calls. A local SLM adds hardware CAPEX plus MLOps and staff OPEX, usually 3 to 5 times the initial hardware price per year.
Which guarantees more privacy?
In the abstract, the local model, because data does not leave the network. In practice, large providers offer no-retention contracts and EU regions, meeting most compliance requirements without local infrastructure.
Can local and API models be combined?
Yes, and it is the usual setup. The model gateway routes each call to the right provider by task, privacy and cost. Architecture decides, not the model.
In SVP the model gateway lets you switch providers without rewriting the product
The local vs API debate is rarely a single answer. Often the answer is both, at different levels of the Semantic Scale. Architecture decides, not the model.