Adding RAG to a SaaS product that already has customers

Every RAG tutorial begins the same way: an empty directory, a folder of PDFs, a loop that embeds them. It's a genuinely good way to learn the concepts and almost useless as a guide to putting retrieval into a product that already has customers, permissions and a schema you can't casually restructure on a Tuesday.

Here's the version for that situation.

Decide what question it answers before you index anything

The failure mode specific to existing products is indexing everything, because everything is right there. You end up with a search box that can answer any question mediocrely, which nobody uses twice.

Pick one question your users currently answer by trawling through your interface. "What did we agree with this customer about renewal terms?" "Which of our policies covers this situation?" "Has this issue come up before, and what fixed it?" One question, one corpus, one measurable improvement. Expand after it works.

Tenant isolation is the first design decision, not the last

In a multi-tenant product, retrieval that returns another customer's data is not a bug, it's a breach notification. This has to be enforced structurally, and the choice is between three options.

-- The filter is part of the search, not applied afterwards
SELECT id, content, 1 - (embedding <=> $1) AS score
FROM document_chunks
WHERE tenant_id = $2
  AND deleted_at IS NULL
ORDER BY embedding <=> $1
LIMIT 20;

CREATE INDEX ON document_chunks
  USING hnsw (embedding vector_cosine_ops)
  WHERE deleted_at IS NULL;

Use the database you already have

The default advice is a dedicated vector database. For a team adding retrieval to an existing product, that advice is usually wrong, and the reason has nothing to do with benchmarks.

If you're already running Postgres, pgvector means your embeddings live in the same database as the rows they describe. Which means a document deleted in a transaction has its chunks deleted in the same transaction. No sync job, no eventual consistency, no window in which a deleted contract is still answerable. Tenant filtering reuses the row-level security you've already got, and your backups already cover it.

A dedicated vector store becomes the right answer past roughly ten million chunks, or when the embedding workload starts competing with transactional queries for the same connection pool. Both are good problems and neither is a launch problem. Start in Postgres; you can move later, and you probably won't need to.

Ingestion has to be incremental from day one

The greenfield tutorial does a one-off import. In a live product, data is changing while you index it, and the three cases that matter are the ones a batch script handles worst.

1. 01Creation. New document, new chunks, embedded and indexed — ideally in a background job so a user's save doesn't wait on an embedding API.
2. 02Update. Re-chunk and re-embed the changed document, then replace its chunks atomically. Half-updated documents produce answers stitched from two versions, which is worse than either.
3. 03Deletion. The one that gets missed. A deleted document whose chunks survive means your assistant quotes a contract the customer terminated. Delete on the same transaction boundary as the source row.

Backfill the existing corpus with a throttled job, not a single pass — embedding APIs rate-limit, and a full re-embed of a large corpus is a real cost. Track progress per tenant so you can enable the feature for customers whose data is ready.

Hybrid search, because your users type identifiers

This matters more in an existing product than in a demo, because real users search the way they talk about your product: invoice numbers, ticket IDs, SKUs, surnames, version strings. Vector search is bad at all of those — it returns semantically similar text that contains none of them.

Run Postgres full-text search and vector search in parallel, fuse the results with reciprocal rank fusion, then re-rank the fused set with a cross-encoder. Re-ranking is typically the single biggest accuracy gain in the pipeline and the one most often skipped, because the pipeline appears to work without it.

Cost, per tenant, before you launch

A feature that costs $0.04 per query is free at pilot scale and a margin problem at ten thousand users. Model it against your actual pricing before release rather than after.

The three levers, in order of effect: trim retrieved context (most teams pass far more than the model needs), cache the stable prefix of the prompt, and route straightforward queries to a smaller model. Together these routinely cut generation spend by more than half.

Measure retrieval separately, and before generation

The instinct when answers are poor is to rewrite the prompt, because the prompt is the visible part. It's usually the wrong lever — if the right passage was never retrieved, no wording recovers it.

Take fifty real questions from your support queue with the passages that should answer them, and score retrieval on recall@k before touching generation. Only once the right material is reliably being found is it worth tuning what the model does with it. There's more on both stages in the retrieval section of our AI integration page.

Ship it to one tenant first

Behind a per-tenant flag, to a customer who'll tell you when it's wrong. Log every query with the retrieved chunk ids, whether the user acted on the answer, and whether they rephrased and asked again. Rephrasing is the most useful negative signal you'll get and the easiest to capture.

Those logged queries become your next eval set, which is the point at which the feature starts improving on evidence rather than on opinion.