RAG Cost Calculator

RAG pipeline cost breakdown

A production RAG system has three cost components: (1) embedding creation — converting documents into vectors once, plus re-embedding on updates; (2) vector storage — storing vectors in your database monthly; (3) query retrieval — embedding each user query and running similarity search. This calculator makes all three transparent.

Choosing the right vector database

Pinecone Serverless is purpose-built for vector search with managed infrastructure. Supabase pgvector runs on PostgreSQL and is free for reads — ideal if you already use Supabase. Weaviate offers a managed serverless option with good developer tooling. Compare total monthly cost for your actual query volume before committing.

What is a RAG pipeline and why it costs money

Retrieval-Augmented Generation (RAG) is a technique that grounds LLM responses in your own data by retrieving relevant context before generating an answer. A RAG pipeline has four distinct stages, each with its own cost dimension. First, embedding generation converts your documents into high-dimensional vectors using a model like OpenAI text-embedding-3-small. Second, vector storage persists those embeddings in a specialized database such as Pinecone or pgvector so they can be searched later. Third, retrieval queries embed the user's question and perform a similarity search against your stored vectors to find the most relevant chunks. Fourth, LLM inference takes the retrieved context plus the user's question and generates a grounded answer. The total cost of a RAG system is the sum of these four stages. Embedding generation is largely a one-time expense amortized over the life of your corpus, but storage and retrieval are ongoing monthly costs that scale with your data volume and query traffic. LLM inference is typically the most expensive single component, often accounting for 60–80% of total pipeline cost at high query volumes. Understanding where the money goes is the first step to optimizing your spend.

Embedding costs: one-time vs ongoing

Embedding your initial document corpus is a one-time cost, but most production systems also need ongoing embedding for new and updated documents. OpenAI's text-embedding-3-small costs $0.02 per million tokens, while the older ada-002 model costs $0.10 per million tokens — a 5x price difference for comparable quality on most retrieval tasks. The newer text-embedding-3-large costs $0.13 per million tokens and produces 3072-dimension vectors, offering better retrieval accuracy at a significant cost premium. To put these numbers in perspective: a corpus of 10,000 documents averaging 500 tokens each contains roughly 5 million tokens total. Embedding that entire corpus with text-embedding-3-small costs just $0.10 — practically free. However, at enterprise scale with 1 million documents averaging 800 tokens, you are looking at 800 million tokens, which costs $16 with 3-small or $104 with 3-large. The ongoing cost depends on your document churn rate. If 10% of your corpus updates monthly, that is 80 million tokens re-embedded each month — $1.60/month with 3-small or $10.40/month with 3-large. Most teams find that text-embedding-3-small provides sufficient retrieval quality for general-purpose knowledge bases, making it the default recommendation unless you are working with highly specialized technical or multilingual content where the extra dimensions of 3-large measurably improve recall.

Vector database pricing: Pinecone vs Supabase vs Weaviate

Vector database pricing varies dramatically depending on whether you choose a managed, self-hosted, or serverless option. Pinecone is the most popular managed vector database with a starter tier at approximately $70/month for 1 pod with 1 million vectors. Their serverless option charges based on read units ($8.25 per million read units), write units ($2 per million write units), and storage ($0.33 per GB/month), which can be cheaper for bursty workloads but expensive at sustained high query volumes. Supabase with pgvector runs on PostgreSQL and starts at $25/month for their Pro plan, which includes compute, storage, and unlimited queries within your database instance. Since vector similarity search runs as a SQL query, there is no per-read charge — making Supabase extremely cost-effective for read-heavy workloads under moderate scale. The tradeoff is that pgvector performance degrades past a few million vectors without careful index tuning (IVFFlat or HNSW indexes). Weaviate Cloud offers a serverless tier with pay-per-query pricing that starts free for small workloads and scales to roughly $0.06 per 10,000 queries at production volume, plus $0.175 per GB/month for storage. For a typical workload of 50,000 documents and 100,000 monthly queries, expect roughly $70–$95/month on Pinecone, $25–$35/month on Supabase, and $40–$60/month on Weaviate. The optimal choice depends on your query volume pattern, required latency, and existing infrastructure.

Typical RAG pipeline costs by scale

Small-scale deployment (1,000 documents, 5,000 queries/month): Embedding costs are negligible at under $0.01/month for re-embeds. Vector storage on Supabase pgvector costs $25/month (the minimum plan). Query retrieval adds roughly $0.50/month. LLM inference for 5,000 responses at ~1,000 tokens each using GPT-4o-mini costs about $3.75/month. Total: approximately $30/month. Medium-scale deployment (50,000 documents, 100,000 queries/month): Embedding re-embeds cost around $0.80/month with text-embedding-3-small at 10% monthly churn. Vector storage on Pinecone serverless runs $35–$50/month. Query retrieval including embedding queries and read operations costs $15–$25/month. LLM inference for 100,000 responses using GPT-4o-mini costs approximately $375/month, or $75/month with response caching at a 50% hit rate. Total: approximately $450–$550/month. Large-scale deployment (1,000,000 documents, 1,000,000 queries/month): Embedding re-embeds cost $16/month for 10% churn. Vector storage on a dedicated Pinecone pod runs $200–$400/month. Query retrieval costs $150–$250/month. LLM inference at this scale is the dominant cost: $3,750/month with GPT-4o-mini or $750/month with aggressive caching and shorter responses. Total: approximately $4,000–$5,500/month. At every scale, LLM inference dominates the cost structure once you exceed a few thousand queries per month.

How to reduce RAG costs

Five proven strategies can significantly reduce your RAG pipeline costs. First, use a smaller embedding model. Switching from text-embedding-3-large ($0.13/MTok) to text-embedding-3-small ($0.02/MTok) saves 85% on embedding costs with minimal retrieval quality loss for most use cases. Run a retrieval evaluation on your actual queries before and after switching to verify. Second, optimize your chunk size. Smaller chunks (256–512 tokens) produce more vectors and higher storage costs, but they improve retrieval precision and reduce the context sent to the LLM — often lowering inference costs more than storage increases. Conversely, larger chunks (1024–2048 tokens) reduce vector count but may retrieve irrelevant content that inflates your prompt. The sweet spot depends on your content structure: structured documentation works well at 256 tokens; narrative content needs 512–1024. Third, implement hybrid search combining BM25 keyword matching with vector similarity. Hybrid search can return highly relevant results on exact-match queries without needing an expensive reranking step, reducing the number of LLM calls needed for answer synthesis. Tools like Weaviate and Elasticsearch support hybrid search natively. Fourth, cache frequent queries. In most production systems, 20–30% of queries are repeated or semantically identical. Implementing a semantic cache (using embedding similarity with a 0.95+ threshold) can cut your query and inference costs by 20–40% with no quality loss. Redis with vector similarity or GPTCache are popular options. Fifth, use tiered retrieval with a cheap first pass and expensive reranking only when confidence is low. Run an initial retrieval with a fast, cheap model. If the top result's similarity score exceeds your confidence threshold (typically 0.85+), skip reranking entirely. Only invoke a cross-encoder reranker or larger LLM for ambiguous queries. This approach typically saves 40–60% on retrieval costs while maintaining answer quality above 95% of the full-rerank baseline.

RAG vs fine-tuning: cost comparison

RAG and fine-tuning are complementary approaches to grounding LLM outputs in domain-specific knowledge, but their cost structures differ fundamentally. RAG incurs an ongoing per-query cost: every request requires embedding, retrieval, and a larger prompt (retrieved context adds tokens). Fine-tuning requires a significant upfront training cost but produces a model that answers from internalized knowledge without retrieval, resulting in smaller prompts and lower per-query inference cost. RAG is cheaper when your knowledge base changes frequently (weekly or more), your corpus is small to medium (under 100,000 documents), you need source attribution and citations, or your query volume is low to moderate (under 50,000 queries/month). At 10,000 queries/month with a 50,000-document corpus, RAG typically costs $50–$150/month total. Fine-tuning wins when your knowledge is relatively static (quarterly updates at most), query volume is very high (500,000+ queries/month), responses do not need source citations, or you need the lowest possible per-query latency. Fine-tuning GPT-4o costs approximately $25 per million training tokens. A training dataset of 10,000 examples at 1,000 tokens each costs $250 to train. If you retrain quarterly, that is $1,000/year in training costs. However, inference on a fine-tuned model uses shorter prompts (no retrieved context), saving $0.002–$0.01 per query compared to RAG. At 500,000 queries/month, this saves $1,000–$5,000/month in inference costs — far exceeding the training expense. The hybrid approach is increasingly common: fine-tune for stable domain knowledge and use RAG only for frequently changing data. This minimizes retrieval costs while keeping responses current. For most teams starting out, RAG is the lower-risk choice because it requires no training data preparation, produces results immediately, and costs are fully variable — you pay nothing if no one queries the system.