Keeping AI features cheap: caching, batching and model choice

Written By: BrainSoft In AI & Research

The first version of an AI feature usually works and then the bill arrives. You wired up an endpoint, it calls a model on every request, and suddenly a chat box on a product page costs more per month than the database that serves the rest of the site. Nothing is broken. It is just wasteful in three places we almost always find on a first pass.

Most of the savings come from three habits: cache identical or near-identical calls, batch work that does not need an answer right now, and use the smallest model that passes your own tests. Below is how we actually do it on client projects, with the trade-offs spelled out.

Cache the calls you are making twice

Before touching models, log your requests for a week and look for repeats. Summaries of the same document, classifications of the same support ticket text, embeddings of the same product description. In most products a surprising share of traffic is the same input arriving again. A cache keyed on a hash of the normalised prompt plus the model name and parameters handles that.

import hashlib, json

def cache_key(prompt: str, model: str, **params) -> str:
    raw = json.dumps({"p": prompt, "m": model, **params}, sort_keys=True)
    return "ai:" + hashlib.sha256(raw.encode()).hexdigest()

Two details matter. First, normalise before hashing: trim whitespace and lowercase where case carries no meaning. Second, set a TTL deliberately. A summary of a stable document can live for days. A classification of a live ticket should expire in minutes or not be cached at all. Stale answers are a real cost too.

Embeddings are the easiest win. They are deterministic for the same input and model, so a long-lived cache is fine, and you avoid paying to re-embed a catalogue every time a job restarts.

Batch what nobody is waiting for

Ask a simple question about each call: does a human see the result within a second? If not, it should not be a synchronous request. Nightly re-tagging, backfilling embeddings, generating alt text for an image library, scoring a batch of leads, summarising yesterday's tickets. Those belong in a queue worker that collects items into batches and sends them together.

• Group by model and prompt template so the shared instructions are sent once.
• Cap the batch size so one failure does not lose an hour of work.
• Store per-item results and retry only the failures, not the whole batch.
• Keep a dead-letter queue; malformed input should not stall the worker.

Even without provider-side batch discounts, batching cuts overhead: fewer connections, fewer retries, and a much lower chance of hitting rate limits that force you onto a bigger, pricier tier. When a provider does offer an asynchronous bulk endpoint at a lower price, this queue is already the right shape to use it.

Pick the smallest model that passes your tests

Teams default to the biggest model and never revisit it. Instead, build a small evaluation set from real inputs you have already seen, with expected outputs you are willing to defend. Fifty to a couple of hundred examples is usually enough to tell models apart for a narrow task. Then run the candidates through it.

for model in ["small", "medium", "large"]:
    score = run_eval(model, cases)   # your own pass/fail or similarity
    print(model, score)

In practice, extraction, classification, routing and short summaries often pass on a small model. Reserve the large one for the step that genuinely needs reasoning, and route per task rather than per product. A common pattern: small model for the cheap first pass, escalate to a bigger one only when confidence is low or validation fails. That fallback path also keeps quality steady when the small model drifts on odd input.

Two more levers sit next to model choice. Trim the prompt; every token of boilerplate you resend is paid for on every call. And cap output length, since generation is usually the expensive half. If you are setting this up across a few products and want a second pair of eyes on the architecture, our services cover exactly this kind of work.

Measure before and after

Track cost per feature, not just total spend. Log token counts, model name, cache hit or miss, and latency on every call, then chart it. Without that, you cannot tell whether a prompt change saved money or quietly doubled the output length. Cache hit rate is the number to watch first; if it sits near zero after a week, your keys are probably too specific.

None of this is exotic. It is the same discipline you would apply to any expensive external call. If you would rather have someone walk through your current setup, get in touch.

Frequently asked questions

#AI & Research