Yann Torrent
Yann Torrent · CTO, co-founder View Yann Torrent's profile

LLM API dependency: the one-week test

LLM API dependency: the one-week test

Most AI features in business software rely on a single LLM API, often OpenAI’s. That is not a problem in itself: these models are good, and their price per token has mostly gone down in recent years.

The problem shows up the day you have to switch, and discover you don’t know how. A deprecated model, a bill that drifts, a customer asking where their data goes: the question is not whether it will happen, but how long it will take you to react.

What lock-in really is

You are locked in when the cost of switching providers exceeds the gain you expect from it. It is not a question of how good the provider is. It is a question of room to manoeuvre.

With an LLM API, that cost is misleading. Changing the endpoint takes an hour: most inference engines now expose an OpenAI-compatible API. What is expensive are the prompts tuned for one specific model, the behaviors your product depends on without knowing it, and having no way to measure what changes.

Four ways to depend on an API

Cost. The risk is not so much a rise in the price per token as its unpredictability. Reasoning tokens, cache pricing, long context: the pricing structure keeps evolving, and your cost per request with it. Your customer contracts, on the other hand, are fixed.

The model. Providers offer dated versions and give notice before retiring them. But they do retire them. On a date you don’t choose, you migrate, and the tone, refusals, accuracy or output format of your feature change.

Data. Every call sends your data out. “Zero data retention” means it is not stored, not that it does not leave. And hosting in a European region does not settle everything: a US provider remains subject to the Cloud Act, which allows US authorities to demand data wherever it is stored. For HR, legal or health data, that is often a deal-breaker.

Availability. Rate limits, incidents, quotas, changing terms of use: your feature’s SLA is capped by your provider’s.

None of these risks is a bug. They all come from the same decision: letting one specific model become an assumption of your product.

The test: could you switch models in a week?

Ask your team: if your model disappeared tomorrow, how long would it take to switch, and how would you know whether quality dropped? The answer puts you on one of these three levels.

LevelWhere the model livesSwitching means…
1Hard-coded
Where the model livesKey, endpoint and model name scattered across your servicesSwitching means…A refactoring project
2Configurable
Where the model livesA parameter in a configuration fileSwitching means…A quick change, but an unmeasured regression
3Abstracted and measured
Where the model livesBehind an internal interface, with an evaluation setSwitching means…A measured switch in a few days

Level 2 is the most misleading. It gives the illusion of freedom: the config line changes in ten seconds, but nobody knows what broke.

What we learned running our own models

At Agora, we serve open-weight models on our own infrastructure, in France, and we switch them as better ones come out. Changing models is not an incident for us, it is routine.

The lesson: changing the endpoint costs nothing. What takes time is knowing whether the new model does as well on your cases, not on a public benchmark.

Three levers to stay in control

1. An interface, not a model. Your services call “generate”, “extract”, “transcribe”, never a model name. Model choice, prompts and parameters live behind that interface. With OpenAI-compatible APIs as the de facto standard, this layer is cheap to build.

2. An evaluation set of your own. A few hundred cases drawn from your real usage, with the expected answer. It is what turns a forced migration into a measured decision. A model you don’t measure is a model you can’t switch.

3. A local option for sensitive data. For data that must not leave, the answer is not a better DPA: it is not sending it at all. An open-weight model served on infrastructure that you or your European provider control solves the problem at the root.

Once these three levers are in place, choosing the best model per task becomes simple: a small, fast model for extraction, a bigger one for writing. And planning a fallback model for when the main one goes down becomes a configuration line, measured.

For a software publisher, it is a sales issue

Your customers are already asking: where does our data go, who can access it, under which jurisdiction? “With a US provider, but in a European region” is no longer enough in public tenders, healthcare or HR.

Your margin depends on a price you don’t negotiate. And your roadmap depends on someone else’s deprecation schedule.

An architecture where the model is interchangeable and measured answers all three. It is not a technical refinement: it is a selling point.

Key takeaways

  • Lock-in is when switching costs more than it brings. With an LLM, the cost is not in the endpoint, it is in the measurement.
  • The test: if your model disappeared tomorrow, could you switch in a week, and prove that quality holds?
  • Three levers: an internal interface, an evaluation set, a local option for sensitive data.

This is the architecture we built for our own products, and the one the Agora platform brings to software publishers integrating AI.

Bring AI into your software with Agora Software.

Let's talk