Skip to content
←Back to Blog
August 28, 2026•8 min read•By Dzaki Amri Zaidaan

Production RAG in 2026: Hybrid Search with pgvector, Reciprocal Rank Fusion, and Context Caching

A deep dive into building production-grade RAG pipelines using PostgreSQL: combining dense embeddings with BM25 full-text search, implementing Reciprocal Rank Fusion, and leveraging context caching to reduce latency and cost. Includes a complete SQL/TypeScript implementation and performance trade-offs.

#AI Engineering#RAG#pgvector#PostgreSQL#Hybrid Search
text

Key Takeaway / TL;DR:

  • Naive vector search alone fails on exact keywords, IDs, and rare terms; hybrid search with BM25 + dense embeddings significantly improves retrieval quality.
  • Reciprocal Rank Fusion (RRF) is a simple, robust way to merge heterogeneous relevance scores without complex calibration.
  • Context caching (e.g., Anthropic's prompt caching) can cut RAG latency and cost by up to 90% for repeated system prompts and retrieved contexts.

The Problem & Industry Shift

In 2026, RAG (Retrieval-Augmented Generation) has moved from demos to production, but many pipelines still rely on a single vector index. This approach fails on exact-match queries like product codes, legal citations, or user IDs. Dense embeddings capture semantic similarity but often miss lexical precision. The industry shift is toward hybrid search: combining dense vectors with BM25 full-text search, then fusing results using Reciprocal Rank Fusion (RRF). This is now a standard pattern in production RAG, and PostgreSQL with pgvector is a compelling platform because it supports both vector and full-text indexing in the same database, eliminating the need for a separate vector database.

Architecture & Core Mechanics

A production RAG pipeline involves several stages: ingestion, retrieval, reranking, and generation. Here's the data flow:

[Documents] -> [Chunking] -> [Embedding Model] -> [PostgreSQL: pgvector + tsvector]
                                                          |
                                                          v
[Query] -> [Embedding] -> [Vector Search] -> [RRF Fusion] -> [Reranker] -> [LLM with Context Cache]
                |-> [Full-Text Search] -> [RRF Fusion] -> [Reranker] -> [LLM with Context Cache]

Key components:

  1. Chunking: Overlap and hierarchical chunking to preserve context. For example, chunk size 512 tokens with 50-token overlap.
  2. Dense embeddings: Use a model like text-embedding-3-large to generate vectors.
  3. BM25: PostgreSQL's tsvector/tsquery with websearch_to_tsquery for full-text search.
  4. RRF: Combine ranked lists from both retrievers using the formula: score = Σ 1/(k + rank), where k is typically 60.
  5. Reranking: A cross-encoder model (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) to refine top-k results.
  6. Context caching: Cache the system prompt and static context to avoid re-encoding them for every request.

Production Code Example

Below is a TypeScript implementation using pg and pgvector for a hybrid search query. It assumes a table documents with columns id, content, embedding vector(1536), and tsv tsvector.

import { Pool } from 'pg';
import { embed, generate } from './ai';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

// RRF fusion function
function reciprocalRankFusion(results: { id: string; rank: number }[][], k = 60) {
  const scores = new Map<string, number>();
  for (const list of results) {
    list.forEach((item, index) => {
      const score = 1 / (k + index + 1); // rank starts at 1
      scores.set(item.id, (scores.get(item.id) || 0) + score);
    });
  }
  return Array.from(scores.entries())
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

async function hybridSearch(query: string, topK = 10) {
  // 1. Generate embedding for query
  const queryEmbedding = await embed(query);

  // 2. Run vector and full-text searches in parallel
  const [vectorResults, ftsResults] = await Promise.all([
    pool.query(
      `SELECT id, embedding <=> $1 AS distance
       FROM documents
       ORDER BY embedding <=> $1
       LIMIT $2`,
      [queryEmbedding, topK]
    ),
    pool.query(
      `SELECT id, ts_rank(tsv, websearch_to_tsquery('english', $1)) AS rank
       FROM documents
       WHERE tsv @@ websearch_to_tsquery('english', $1)
       ORDER BY rank DESC
       LIMIT $2`,
      [query, topK]
    ),
  ]);

  // 3. Convert to ranked lists (lower rank is better)
  const vectorRanked = vectorResults.rows.map((r, i) => ({ id: r.id, rank: i + 1 }));
  const ftsRanked = ftsResults.rows.map((r, i) => ({ id: r.id, rank: i + 1 }));

  // 4. Fuse using RRF
  const fusedIds = reciprocalRankFusion([vectorRanked, ftsRanked]);

  // 5. Fetch full documents for reranking
  const docs = await pool.query(
    `SELECT id, content FROM documents WHERE id = ANY($1)`,
    [fusedIds.slice(0, topK)]
  );

  // 6. Rerank with cross-encoder (pseudo-code)
  const reranked = await rerank(query, docs.rows);

  return reranked;
}

Critical decisions:

  • Use <=> for cosine distance (pgvector operator).
  • Use websearch_to_tsquery for robust query parsing.
  • Run both searches in parallel to minimize latency.
  • RRF avoids score normalization issues.

Performance, Cost & Trade-offs

Benchmarks (typical):

  • Vector search alone: Recall@10 ~65% on MS MARCO.
  • Hybrid (vector + BM25 + RRF): Recall@10 ~82% (+17% relative).
  • Adding a reranker: +5-10% further, but adds ~50ms latency per query.

Latency breakdown:

  • Embedding query: ~10ms (local model) or ~100ms (API).
  • Vector search: ~20ms for 1M vectors with HNSW index.
  • Full-text search: ~5ms.
  • RRF: <1ms.
  • Reranking: ~50ms for top 10 with cross-encoder.
  • LLM generation: 1-3s (dominates).

Cost:

  • Context caching can reduce LLM cost by up to 90% for repeated system prompts. For example, Anthropic's prompt caching charges 1.25x for cache writes but 0.1x for cache reads [1].
  • Storing embeddings in PostgreSQL avoids separate vector DB costs.

Trade-offs:

  • Hybrid search requires maintaining both tsvector and embedding columns; use triggers to keep them in sync.
  • RRF is simple but not optimal; weighted RRF can improve results if you have validation data.
  • Context caching increases memory usage; set appropriate TTLs.

Actionable Checklist / Summary

  1. Start with hybrid search: Combine tsvector and pgvector in PostgreSQL. Use RRF for fusion.
  2. Chunk wisely: Use overlapping chunks (e.g., 512 tokens with 50 overlap) and consider hierarchical chunking for long documents.
  3. Index properly: Create HNSW index on embedding and GIN index on tsv.
  4. Rerank: Add a cross-encoder reranker for top 10 results to improve precision.
  5. Cache context: Use prompt caching (e.g., Anthropic's) for system prompts and static retrieved context to reduce latency/cost.
  6. Monitor and evaluate: Track retrieval metrics (Recall@k, MRR) and end-to-end quality.
  7. Consider weighted RRF: Tune k and weights if you have labeled data.

References

← Explore More ArticlesShare on X / Twitter →