AI micro-decisions: cutting LLM cost and latency
In short
- Much of the work done by software or an AI agent comes down to micro-decisions: is this ticket urgent, which team should handle it, is this document an invoice?
- Handed to a general-purpose large language model (LLM), they get expensive at scale, take seconds and return free text that has to be interpreted.
- A dedicated “System One” decision engine answers these questions in a few hundred milliseconds, with a typed answer and its probability. The LLM remains available for what requires writing or reasoning.
Gartner predicts that at least 15% of day-to-day work decisions will be made autonomously through agentic AI by 2028, up from none in 2024. In the same press release, the firm expects over 40% of agentic AI projects to be canceled by the end of 2027, first and foremost because of escalating costs.
These two figures are about the same thing. Most of the decisions an agent makes in a day are short and repetitive: filing a message, setting an urgency level, validating a step before acting. Handing all of them to a general-purpose LLM means paying for reasoning on questions that only need a yes or a no.
What is a micro-decision?
Some conditions fit in one line of code: an amount above a threshold, a past due date, an empty field. Others require reading a text or an image to decide, and every type of business software has its own:
- CRM: does this customer e-mail announce a cancellation?
- ERP: is this complaint about a delay, a defective product or a billing error?
- SCM and logistics: does this scanned delivery note carry a handwritten reservation from the recipient?
- HRIS: is this employee question about payroll, leave or health insurance?
- Recruitment: how well does this application match the position, on a scale of 1 to 4?
- Accounting firms: is this document a purchase invoice, an expense report or a credit note?
- Legal: does this contract contain a non-compete clause?
- Maintenance and after-sales: does this photo taken on site show a leak?
A micro-decision has three traits:
- A closed question about a piece of content, text or image.
- An answer of a type known in advance: yes or no, a choice from a list you define, or a score on a scale.
- High volume: thousands to millions a day, across all your clients.
In an agentic application, they add up at every step: the agent has to decide which tool to call, whether the request is out of scope, whether the action it is preparing carries a risk.
What a micro-decision costs when an LLM makes it
A general-purpose LLM can answer these questions. The problem lies in how it does so, multiplied by volume.
The bill follows the volume. Each call bills the tokens sent (instructions, the content to analyze, often examples) and the tokens generated. A single decision costs little. A million decisions a month, spread across all your clients, end up weighing on the product’s margin, and that cost grows with the feature’s success.
Latency adds up. An LLM generates its answer token by token. For a simple question, that commonly takes one to several seconds. In an agent that chains five or six decisions before acting, those seconds stack up, and the user waits.
The answer is free text. Even with strict instructions, the model may add a sentence of explanation, change the case of a category or invent a category close to the requested ones. Your code has to interpret the answer, and every deviation becomes an error case to handle.
Throughput depends on a third party. LLM APIs enforce requests-per-minute quotas. A spike in activity at your clients can hit a limit you do not control, a topic covered in our article on LLM API dependency.
System 1, System 2: sharing out the decisions

In Thinking, Fast and Slow (2011), Nobel laureate psychologist Daniel Kahneman distinguishes two modes of thought: a fast, intuitive system that answers effortlessly, and a slow, deliberate system called on when the question demands it. A human spends most of the day in System 1 and keeps System 2 for the problems that deserve it.
Applied to software, the idea is to hand each decision to the model that suits it. Research has already measured the value of this split. The authors of FrugalGPT (Stanford, 2023) match the performance of the best LLM tested with up to 98% cost reduction, by sending each request to the cheapest model able to answer it. The RouteLLM team (2024) shows that a router between a strong model and a lighter one cuts costs by more than half in some cases, with no drop in answer quality.
A System One decision engine takes this logic further: instead of a smaller model that still generates text, a model designed to decide. It receives the content and the questions, and returns in a single pass one typed answer per question, with its probability. It writes nothing, which is precisely what makes it fast.
What we observed testing Reflex
Reflex is Agora’s System One decision engine. We tested it on November 10, 2026 on the public Reflex demo, under the conditions a visitor would meet.
On six sample documents (an invoice, a signed contract, a bank statement, a payslip, a restaurant receipt and a blurry photo of a letter), Reflex answered three questions per document: the document type, the presence of a handwritten signature and legibility. The six documents were processed in 0.75 seconds in total, with calls in parallel. The five legible documents were classified correctly with a probability of 99 to 100%, and the contract’s signature was detected at 100%.
The blurry photo is the most telling case. Reflex classified it as “Other” with a probability of 30%, and judged it illegible. That is exactly what the probability is for: below a threshold you set, the decision goes back to a human instead of being applied. The engine files what is clear on its own and flags what is not.
On support tickets, the result is similar: “Production server down: no invoices have gone out since 9 am” was classified as an incident at 99.98%, with the user blocked at 99.5%. A dashboard feature request was recognized as such at 99.95%, as non-blocking.
How to share out the decisions in your software
Three steps are enough to find out what a decision engine can take on in your product.
- Find the LLM calls that expect a closed answer. Every prompt ending with “answer only yes or no” or “choose from these categories” is a micro-decision.
- Write the questions and choice lists in natural language. With Reflex, they are sent with each request: no prior training, and you adapt the categories to each client’s taxonomy.
- Set a probability threshold. Above it, the decision is applied. Below it, it goes back to a human, as in our use cases on automatically filing incoming mail or prioritizing requests by urgency.
The LLM keeps everything that requires writing, explaining or untangling an unusual case. It receives fewer calls, and the ones it receives are those that justify its cost.
That leaves the business model. Billed per token, each decision costs a little on every call, and the bill follows the volume. Reflex comes with capacity-based licensing: you subscribe to a peak throughput in decisions per second, the monthly cost is fixed whatever the number of decisions, and data is processed in France, which matters to software publishers whose clients require sovereign AI.
Signs that a decision deserves a dedicated engine
- Your LLM bill is rising while most calls are used to sort, route or validate.
- Your volumes run into thousands a day, or will once the feature is adopted by all your clients.
- The decision has to land in real time, while the user waits.
- Your code parses free-text answers and handles error cases when the model breaks the format.
- Your clients are under NDA or in a regulated sector and require hosting in France.
When several of these signs are present, the simplest step is to test on your own content: the Reflex demo accepts your texts, your images and even a whole folder, and compares its classification with your filing.
Reflex makes the micro-decisions of your software and your agents in a few hundred milliseconds, and leaves to the LLM what requires reasoning. Try it live on your content, or show us a real case.
Bring AI into your software with Agora Software.
Let's talk