Our AI search ran inside Postgres, which scanned every chunk a user could see on every question. I moved it, in stages and without downtime, onto a dedicated vector store that keeps the same speed as the data grows and runs inside a customer's own cloud.
pgvector tables in each tenant's schema. Similarity was computed per question across a user's whole document set, and slowed down as companies added documents.
A LanceDB vector service on a managed disk with snapshots, behind one interface, so each tenant can run on LanceDB or Turbopuffer, in our cloud or theirs.
The problem
Search worked for small companies and slowed as customers grew into thousands of documents. The way the queries were written, the vector index couldn’t help, so each question was compared against every chunk the user could access.
The obvious fix, a hosted vector database, came with a catch: it ran in one cloud region, so a customer hosting BHyve in their own cloud would get no AI search at all.
The idea
Treat the vector store as a swappable part. Put every store behind one 9-method interface, run the new engine next to Postgres as an A/B before trusting it, and let each tenant choose where its vectors live.
How I did it
- Built a separate Python RAG service and backfilled the existing vectors into Turbopuffer without re-chunking or re-embedding, with an A/B harness for recall, nDCG and MRR.
- Removed the Postgres fallback on 9 July once the new path held, so there was only one way to answer a question.
- Built a LanceDB vector service as the platform’s first stateful workload: single-writer by design, a storage class that keeps the disk if the pod is deleted, and daily snapshots kept for 14 days.
- Load-tested on real tenant data: vector search 5.3 ms at 100k rows and 6.2 ms at 500k, memory flat near 1 GB, and only disk growing, at about 3.4 KB a row.
- Picked a new embedding model with a blind, position-randomised judge on 935 real production questions, and rolled it out dev first (7 Sep), then staging (10 Sep), then production (12 Sep).
- Helped move one enterprise customer’s search into its own cloud: 172,839 chunks from 3,226 documents, 4.3 GB of LanceDB files, with every batch’s fidelity checked.
The impact
Every production tenant was on LanceDB by 25 October. Query time no longer depends on how many documents a company has, and self-hosted customers get the same AI search as everyone else. The 64 GiB production disk has room for roughly 13–20 million chunks at today’s row size.
One honest footnote: after the switch I measured that compressed vector indexes drop recall@10 to 0.29 on large tenants unless a refine step is added, which brings it back to 0.83 for about 8 ms. That fix is documented and next on the list.
Migrate in stages and keep the old path alive until the new one has been measured. Then keep measuring after the switch: that’s how I found the recall gap.