Deep-dive guide
RAG and Vector Database Cost Simulator: What Each Retrieved Answer Really Costs
Executive industry analysis
Retrieval-augmented generation looks like a small addition to a chat feature: store your documents as vectors, fetch the best passages for each question and let a language model write the answer. The first prototype costs almost nothing. The problem appears at volume, because every answer repeats the same chain of paid steps, and the biggest of them, the tokens in the prompt, is not fixed. It depends on how much text you paste in. A feature that costs a fraction of a cent in testing can cost several cents per answer in production.
The structure of the problem is a per-answer cost stacked on a per-month cost. Per answer you pay for input tokens, output tokens, a possible reranking call and a database read. Per month you pay to keep the vectors in memory, to re-embed documents that change and to run evaluation and tracing. Revenue is attached to answers, so the margin of the product is the gap between income per answer and variable cost per answer, after the monthly layer is spread over the volume.
The domino model follows this in five steps. Context size and the prompt set the tokens per answer. With the model price they set the cost per thousand answers. Compared with revenue per answer that becomes the cost share. After observability, what remains is the contribution margin, and fixed team costs decide the monthly profit. A longer context, a pricier model or a larger share of reranking each raise the second box, and a growth in usage then multiplies the damage rather than diluting it.
The iceberg separates the bill you notice from the ones you do not. Above the waterline are model input tokens and the vector database. Below it are output tokens, which are priced at a multiple of input; reranking calls on each search; the cost of re-embedding documents as they change; and the evaluation, tracing and logging that production systems need. Each is easy to leave out of an early estimate. Use the simulator to find the context size or model price at which your month reaches zero and decide what to trim first.
Formula and methodology blueprint
The model works per month. Revenue is answers times revenue per answer. Each answer sends the retrieved context plus a fixed prompt to the model and receives a fixed-length answer priced at four times the input rate. The vector database is billed for memory: chunks times embedding dimensions times four bytes, with index overhead, priced per gigabyte, plus a read fee per million queries. Reranking is priced per search and applies to the chosen share of queries. Changed documents are re-embedded every month.
- Model input
- I = answers × (context + prompt tokens) ÷ 1e6 × price
- The largest line for most products; it grows with context size and answer volume.
- Model output
- O = answers × 350 tokens ÷ 1e6 × price × 4
- Output tokens cost a multiple of input tokens, so long answers add up quickly.
- Vector storage
- V = chunks × 1,536 × 4 bytes × 1.5 × $5 per GB + reads
- Memory-resident vectors with index overhead, plus a per-query read fee.
- Rerank and re-embed
- R = answers × rerank share × $0.002 + chunks × 8% × 400 tokens × embed price
- A call on each reranked search and the monthly refresh of changed documents.
- Net profit
- Net = revenue − I − O − V − R − observability − fixed
- The monthly result. The breaking point is where this reaches zero.
Cascading elasticity
Context tokens and model price are the most elastic inputs because they multiply each other and then the answer volume. Revenue per answer is equally powerful on the income side. Reranking share and index size are gentler but steady. Answer volume cuts both ways: more answers earn more, but only while income per answer stays above the variable cost of producing it.
Three real-world case studies
Bootstrapped starter
A small support bot with a tidy index
A product team indexes a few hundred thousand chunks of help articles and serves sixty thousand answers a month using a modest model, short context and little reranking. Each answer earns well. Load it to see how affordable retrieval is when the prompt is lean and the knowledge base is small.
Scaling mid-market
A high-volume assistant with efficient prompts
A platform serves two and a half million answers a month over sixty million chunks, but keeps context short and uses a low-cost model. Revenue per answer is thin yet the volume is large. Load it to see how prompt discipline turns scale into profit and how sensitive it is to a pricier model.
Black-swan crisis
A huge context on a premium model
A team pastes fifteen thousand tokens of documents into every prompt, uses a premium model and reranks every search over a hundred million chunks. Each answer earns only two cents. Load it to see model tokens alone exceed revenue and the month turn deeply negative.
Industry benchmark matrix
| Metric | Danger zone | Industry average | Top 10% elite |
|---|---|---|---|
| Model and retrieval cost as share of revenue | Above 60% | 20–40% | Below 15% |
| Context tokens per answer | Above 9,000 | 2,000–5,000 | Below 1,500 |
| Share of searches reranked | 100% on every search | 30–60% | Selective, under 30% |
| Monthly re-embedding share of index | Above 25% | 5–10% | Below 3% |
| Contribution margin per answer | Below 25% | 45–60% | Above 70% |
Variables and levers
Answers per month
Number of questions your product answers in a month, each one a retrieval plus a model call.
Lever: Scale usage only after cost per answer is below revenue per answer, and rate-limit free users that add cost but no income.
Revenue per answer
Average income attributable to one answer, from subscriptions or usage fees divided by answers served.
Lever: Price by usage or add a higher tier for heavy users so income per answer follows the model bill.
Indexed chunks
Number of text chunks stored in the vector database, in millions.
Lever: Deduplicate and prune the index, and keep rarely used content in cheaper storage, because memory-resident vectors are billed every month.
Context tokens per answer
Retrieved passages pasted into the prompt for each answer. This is the largest driver of model cost.
Lever: Retrieve fewer, better passages and trim them before the prompt, since input tokens are paid on every single answer.
Model price per million input tokens
What the language model charges per million input tokens. Output tokens are priced at a multiple of this.
Lever: Route simple questions to a smaller model and cache repeated answers, because price per token multiplies the largest line.
Answers that use reranking
Share of searches that pass through a reranking model to improve the order of retrieved passages.
Lever: Rerank only the ambiguous searches, and use the gain in precision to shorten the context you send to the model.
How to use this simulator
- 1Enter monthly answers and the average income per answer from your billing data.
- 2Add the average tokens of retrieved context and the model's price per million input tokens.
- 3Set the number of indexed chunks and the share of searches that use reranking.
- 4Press Find my breaking point to see which input pushes profit to zero first.
- 5Compare the current design with tighter retrieval over 24 months, including its setup cost.
Mistakes to avoid
- Estimating cost from a prototype. Test traffic uses short prompts and few users; production adds long contexts, retries and repeated questions.
- Forgetting output tokens. Answers are priced higher per token than prompts, and long, chatty responses quietly raise the bill.
- Treating the index as free. Memory-resident vectors are charged every month whether or not anyone queries them.
- Chasing quality with ever larger context. Past a point more passages add cost faster than they add accuracy, and better retrieval is cheaper.
Frequently asked questions
What drives the cost of a RAG answer the most?
Usually the tokens sent to the language model. Each retrieved passage you paste into the prompt is paid for on every answer, so context size is the first lever to examine.
Is the vector database expensive?
Often it is the smaller part. It becomes significant with tens of millions of chunks or when everything is held in memory. Pruning and tiering help.
Does reranking pay for itself?
It can, when better ordering lets you send fewer passages to the model. Test whether the saved input tokens outweigh the extra call.
Why are output tokens in the model?
They are priced at a multiple of input tokens. A verbose answer costs more than a concise one at the same context size.
How do I find the breaking point?
Press Find my breaking point. The tool moves each input toward its worse end and reports the value where monthly profit reaches zero.
How accurate is this?
It is an educational model with stated assumptions, not financial advice. Use your own token logs and invoices, and check current vendor pricing.
This is an educational planning model with stated assumptions. It is not financial, tax or legal advice. Replace the inputs with your own figures and verify important decisions with a qualified professional.