Vector Databases

Quick Answer

A Vector Database, stores embeddings, numeric representations of meaning, and finds the entries closest to a query in that meaning-space, fast, at scale. It is the retrieval backbone of RAG and semantic search. Whether you need a dedicated one, or a vector extension on the database you already run, is a scale question with a measurable answer.

What is a Vector Database?

Ordinary search matches words. A query for "salary bands" misses the document that only says "compensation structure."

Vector search fixes this by comparing meaning. An embedding model converts each piece of text into a vector, a list of hundreds of numbers positioning it in a space where similar meanings sit near each other. A vector database stores those vectors and, given a query vector, returns its nearest neighbours.

The database part matters because exact nearest-neighbour search across millions of vectors is too slow to do naively. Vector databases make an approximate version of that search fast, and add what production needs around it: metadata filtering, updates, persistence, and access control.

One framing keeps purchasing decisions honest: a vector database is an index, not a source of truth. It holds derived data, re-computable from your documents at any time. That is why the pipeline feeding it matters more than the logo on it.

How Vector Databases Work

1. Embedding: An embedding model turns each chunk into a vector. The choice of embedding model shapes retrieval quality more than the choice of database.
2. Indexing: Vectors enter an approximate nearest neighbor (ANN) index. The dominant structure is HNSW, a layered graph navigated coarse-to-fine, trading a sliver of recall for orders-of-magnitude speed.
3. Query: The question is embedded the same way, and the index returns the closest candidates, typically by cosine similarity.
4. Filter and combine: Production queries mix similarity with metadata filters (department, date, permissions) and keyword search, since hybrid retrieval catches what either method alone misses.

Two techniques most teams have not met yet, and should have:

5. Quantization: Storing vectors at lower precision (int8, even binary) cuts memory dramatically, and binary quantization can shrink the footprint by an order of magnitude or more, with modest recall loss recovered by re-scoring a shortlist at full precision. At scale, this is the difference between fitting in RAM and not.
6. Matryoshka embeddings: Embeddings trained so their first dimensions carry the most meaning (Kusupati et al., 2022) can be truncated to a fraction of their length for a cheap first pass, then re-ranked at full length. Coarse-to-fine search from one embedding.

The word approximate deserves attention. ANN indexes are tunable trade-offs, and an index tuned for speed silently drops relevant results.

Benefits of Vector Databases

1. Meaning-based recall: Finds the compensation doc when the query said salary, across phrasing, synonyms, and languages.
2. Scale with speed: Millions of vectors searched in milliseconds through ANN indexing.
3. Metadata filtering: Similarity constrained by permissions, recency, or source, which real applications always need.
4. Fresh knowledge: New content is embedded and searchable in seconds. No model retraining involved.
5. Shrinkable cost: Quantization and truncation mean the memory bill is a design choice, not a fixed tax.

Where Vector Databases Are Used

1. RAG retrieval: The store every grounded assistant queries first.
2. Semantic search: Over documentation, tickets, and knowledge bases.
3. Recommendation and similarity: Finding items, cases, or customers that resemble this one.
4. Deduplication and clustering: Spotting near-identical content across large corpora.
5. Agent memory: The long-term store agents read and write across sessions.

Common Mistakes With Vector Databases

1. Buying infrastructure before measuring scale: A dedicated store for a corpus Postgres could serve adds an operational system, a sync pipeline, and a security surface for no gained capability.
2. Blaming the database for embedding problems: If retrieval returns junk, the embedding model and the chunking are the usual suspects. Swapping stores rarely fixes it.
3. Vector-only retrieval: Exact identifiers, SKU-B4920, error codes, names, need lexical search. Hybrid is the production default for a reason.
4. Ignoring index tuning: Default ANN parameters are a guess. Recall is measurable, so measure it before trusting it.
5. Letting the index drift: Documents change, embeddings do not update themselves. Staleness here looks like confident answers citing the old policy.

When You Should Not Use a Vector Database

If your corpus is small, thousands of chunks rather than millions, brute-force similarity in memory or in your existing database is fast enough, and the operational simplicity is worth more than the headroom.

If queries are exact-match by nature, IDs, codes, structured fields, classical search and SQL do it better with zero approximation.

And if your retrieval quality problem is upstream, bad parsing, bad chunking, stale content, no store fixes that. The index faithfully serves whatever it was fed.