What RAG Actually Costs Per Query: Real Numbers From a Production System

The Short Answer

A RAG query on a production system costs about $0.0023 in inference — under a quarter of a cent. Fixed infrastructure runs about $57/month, and roughly 92% of that is one line item: the managed vector database's plan minimum.

At 1,000 queries a month those two numbers combine to about $59, of which inference is $2.30. Inference is 4% of the bill. Almost every RAG cost discussion optimises the 4%.

The numbers below come from a system we run, at the settings it actually runs at, not from a spreadsheet built on defaults. Where a figure is derived rather than measured, it says so.

What Goes Into One Query

A single query does four things: embed the question, retrieve candidate passages, rerank them, and generate an answer from the ones that survive.

The configuration those numbers assume:

And the cost of each step:

Two design choices hold that number down, and both are worth naming because they are the ones people get wrong.

The reranker runs in-process. It is lexical-overlap scoring, not a hosted cross-encoder. A managed reranking API charges per document reranked, and at five documents per query it would add roughly an order of magnitude to the marginal cost — turning the cheapest step in the pipeline into one of the most expensive.

The context budget is bounded. Five chunks at 100 tokens each puts a hard ceiling on prompt size. Cost per query cannot drift upward as the document corpus grows, which is the failure mode that makes RAG bills unpredictable. Whether your corpus is 10 pages or 10,000, the model sees the same amount of text.

A note on rates: these are list prices — $1.00/$5.00 per million tokens for Haiku 4.5, $0.02 per million for Titan V2. Partner-hosted inference is priced separately. The conclusion survives any plausible variance, because the gap between inference and fixed cost is more than an order of magnitude.

What Ingestion Costs

Almost nothing. Chunking and keyword indexing are local; only embedding costs money.

Indexing an entire enterprise document set costs about the same as a cup of coffee, once.

The real ingestion constraint is wall-clock time, not money. Our first live ingest ran 109 chunks in about 61 seconds. The cause was not embedding throughput — it was that the pipeline awaited four network round-trips per chunk in sequence, and called the single-document embedding API instead of the batch one, so the client's built-in batching never engaged. That model reproduces the measurement exactly: 109 chunks × 4 round-trips × 140ms ≈ 61s.

Restructuring it into phases — bulk existence checks at bounded concurrency, then one batched embedding call, then one bulk upsert — removes most of that. We are deliberately not publishing an "after" figure yet, because the improvement we can currently quote is derived from that same 140ms round-trip rather than measured against a live ingest under real throttling. It will be lower in practice than arithmetic suggests, and a number nobody has measured is not a number worth quoting.

The lesson generalises past our system: if your ingestion is slow, measure round-trips before you blame the embedding model. Latency-bound pipelines look exactly like throughput-bound ones from the outside.

The Line Item That Actually Matters

The vector database is 92% of fixed cost and it is a floor, not a usage charge. You pay it on the first day with an empty index.

That reframes the whole economics question. Serving 1,000 queries a month costs about $59 in total, of which $2.30 is inference. Serving 10,000 costs about $80. The marginal cost of a query is real but small; the cost of existing is what you are actually buying.

What This Means for Design Decisions

Three things follow, and they are the opposite of the usual advice.

Stop optimising the model choice for cost. Moving generation from a small model to a frontier model takes a query from roughly $0.0023 to roughly $0.0117. Across 1,000 queries that is a difference of about $9 a month. If answer quality is what makes the system worth running, that is the cheapest quality improvement available anywhere in the stack. Nobody churns over $9; plenty of people churn over a bad answer.

Treat context budget as a quality dial, not a cost one. Our 100-token chunks with topK=5 give the model roughly 500 tokens of context — often too little to contain a complete answer, which shows up as unnecessary refusals and thin responses long before it shows up on a bill. Raising chunk size to 400 and topK to 8 takes a query to about $0.005. That is still around 5% of a $99/month plan. Under-retrieval costs more in churn than over-retrieval costs in tokens.

Look hard at the vector store, because that is where the money is. A managed serverless vector database with a $50 floor makes sense at scale and makes very little sense at one small index. Postgres with pgvector, on a database you are already running, removes that floor almost entirely. At corpus sizes below a few million chunks the retrieval quality difference is negligible, and the cost difference is the entire fixed bill.

Why These Numbers Are Hard to Find

Published RAG cost analyses tend to model token consumption in isolation, because tokens are the part with a public price list. The result is guidance that is directionally wrong: it optimises the smallest line item and ignores the one that dominates.

If you are building a RAG system and want a single number to carry away, use this one: at low query volumes, your vector database costs more than your language model. Design accordingly.