When you don't need RAG

A few months ago I built a retrieval pipeline for VeriStream AI: chunking, embeddings, a vector store, the usual shape. So when a client asked me for a WhatsApp bot that answers questions from his product manuals and policies, retrieval was the design I reached for first.

I left it out. The bot has no vector store, no chunking and no embeddings. All 12 knowledge-base files go into the prompt on every call, and the provider's prompt cache does the work that retrieval would have done for the bill. This post is about when that is the right call, because it is not always.

The arithmetic

The argument is a price list. On the model I use, cached input tokens are billed at a tenth of the normal input rate. My knowledge base is 45,673 characters, and with the instructions around it the prompt prefix comes to about 15,000 tokens.

Per answered message, at list prices:

Retrieval would send a few thousand tokens of selected sections on every call, and those tokens are different each time, so none of them are cached. A few thousand tokens at the full rate cost more than 15,000 at a tenth of it. The whole knowledge base, cached, is cheaper than the relevant slice of it, uncached.

It is also less to get wrong. There is no chunk boundary that cuts a warranty clause in half, no embedding model that misses a Hinglish phrasing, and no question of whether the right section was retrieved. The model sees everything, every time.

These are estimates from list prices, not a bill, and the token counts are conversions from character counts. The bot is pre-launch, so I have no production numbers at all. The arithmetic is the whole case.

Four things that have to be true

The knowledge base fits in one prompt with room to spare. Mine covers five camera models, a handful of accessories and the brand's policies. A catalogue of five thousand products is a different problem.

The prefix is byte-identical on every call. A prompt cache matches on exact bytes from the start of the prompt. So I build the system prompt once when the service starts, not per request. The files are sorted by path and their line endings are normalised before they are joined, so the same content always produces the same bytes. Nothing that varies by customer sits inside it. History and the new message come after.

The content changes rarely. Any edit to the knowledge base or the instructions changes the bytes, and the next call is a cold start at about ten times the price. For product manuals that is fine. For stock levels or prices it would not be, which is one reason the bot links the product page and never states a price.

Everyone gets the same knowledge. One brand, one prefix. If each customer needed their own documents, there would be no shared prefix to cache.

Check the cache before you build on it

I did not take the caching on trust. Before building anything on top of it, I made two calls against the real model and read the usage figures on the second. 8,495 of its 8,512 input tokens came from cache.

One thing that check taught me: there is a minimum size below which the provider does not cache at all, 1,024 tokens on this API. The invented catalogue I use for testing is smaller than that, so caching can only be checked against the real knowledge base.

Prompt size becomes the bill

This is the cost of the design, and I found it the hard way. By my estimate the prefix is 84% of the cost of a cache hit and 98% of a miss. Every paragraph in the knowledge base is paid for on every message, whether or not anyone asks about it.

I scraped the brand's website into the knowledge base one day, and the bill went up. When I looked, the knowledge base stated the warranty four times and carried one paragraph eight times. With retrieval, duplicates are untidy. Here they are a recurring charge. Consolidating took it from 113k characters to 67k with every customer-facing answer kept, and the audit turned up one answer that was simply wrong.

So the prompt now has a budget. A test caps the compiled prompt at an exact character count, and I lower the cap whenever I trim. A new rule has to be paid for with a cut somewhere else. It makes a routine content update into a negotiation, and that is the intent.

When I would go back to retrieval

• The knowledge base outgrows one prompt, or gets close enough that answers start to suffer.
• The content changes often enough that the cache is cold more than it is warm.
• Different users need different documents.
• Traffic is so thin that the cache expires between messages, and most calls are misses anyway.

None of those is true for this bot today. The first could become true if the catalogue grows, and I would rather know the exit than pretend there is not one.