Vector database questions are now a standard fixture in ML and AI engineer technical screens, driven by the explosion of production RAG pipelines at scale. Reddit’s 340M-vector Qdrant-vs-Milvus evaluation, ZenML’s 1,200-deployment RAG analysis, and Pinecone’s acquisition-market moment all landed in the same 2024–2026 window. The interview has shifted from definition-recall to trade-off judgment: HNSW versus IVF, pre-filter versus post-filter, and pgvector versus a dedicated store.
- Why can’t you just use a regular database for similarity search?
- What are embeddings, and where do the vectors actually come from?
- When would you use cosine similarity versus L2 distance versus dot product?
- What is approximate nearest neighbor search, and why approximate?
- When is exact brute-force search still the right choice?
- Explain how HNSW works. What do M, ef_construction, and ef_search control?
- How does an IVF index work, and what do nlist and nprobe trade off?
- Compare scalar, product, and binary quantization. When would you use each?
- How do you combine metadata filters with vector search without wrecking recall?
- What happens to an HNSW index under heavy streaming writes?
- Design a multi-tenant RAG search system over 5M documents at 300 QPS.
- How would you implement hybrid search, and how does Reciprocal Rank Fusion work?
- How do you isolate tenants and per-document ACLs inside one vector store?
- pgvector or a dedicated vector database — how do you decide?
- How do you shard and query a vector index across hundreds of millions of vectors?
- How do you tune the recall-versus-latency trade-off?
- How would you cut memory usage while keeping recall acceptable?
- What breaks first when a vector search system scales, and how do you see it coming?
- What metrics prove a vector search system is healthy after launch?
What vector database interviews actually test in 2026
Vector database questions appear in ML Engineer, AI Engineer, and Search/Relevance Engineer screens — the roles building RAG pipelines, recommendation systems, and enterprise semantic search. The companion AI engineer interview guide covers the full stack from model serving to orchestration; this one goes deep on retrieval infrastructure and the index trade-offs interviewers probe at the technical screen and system-design stage.
Graham Gillen of Pureinsights, who consults on enterprise search infrastructure, put it plainly: “The question ‘which vector database should I use?’ is increasingly the wrong question. Vectors don’t live in isolation — they work alongside structured data, keyword indexes, filters, and business logic. The platforms that figured that out earliest are the ones pulling ahead.” (Source: Pureinsights, 2026)
Interviewers evaluate three competencies:
- Index selection and trade-off reasoning. Can you choose between HNSW, IVF, flat search, and quantization for a specific scale, recall target, and update frequency — and justify it from first principles rather than naming a default?
- System design under realistic constraints. Multi-tenant RAG pipelines, hybrid keyword+vector retrieval, and distributed sharding at hundreds of millions of vectors are common prompts. The signal they read is component-level thinking and failure-mode awareness, not happy-path data flow.
- Production ops intuition. HNSW graph degradation under streaming writes, the recall-latency curve’s non-linearity above 95%, and what to monitor after launch separate engineers who have shipped from those who have only read the docs.
This guide follows that evaluation funnel: core concepts (recruiter screen) → index internals (technical screen) → system design (hard prompts) → production ops (senior signal). The index-selection decision matrix and production failure-mode checklist with hard numbers are the information-gain wedge no competitor provides for free.
Core-concept screen: embeddings, similarity, and approximate search
Vector search interviews at the concept-screen stage test whether a candidate understands why the entire infrastructure stack exists — not just what a vector database does, but why a relational database fundamentally cannot do it. The B-tree data structure, which makes relational queries fast, organizes data for exact-value lookups and range scans; it has no mechanism for answering “find the 10 most geometrically close points in 1,536 dimensions” without scanning every row. That structural incompatibility is why HNSW, IVF, and the entire ANN index family were built.
This section covers five questions at the recruiter or early technical screen: why regular databases fail for similarity search, what embeddings are and where vectors originate, which distance metric to pick and why, what ANN trades off, and when brute-force is still the right call. Candidates who pass this stage don’t recite definitions — they trace structural constraints to the engineering decisions those constraints forced.
Why can’t you just use a regular database for similarity search?
Concept: Structural incompatibility between relational indexes and similarity search | Difficulty: junior | Stage: recruiter
Direct answer: A vector database exists because relational databases index discrete, low-cardinality values using B-tree or hash structures — queries match exact values or sorted ranges. Similarity search requires a fundamentally different operation: finding K vectors geometrically closest to a query in high-dimensional space.
A B-tree on a 1,536-dimensional embedding column must compare every row to answer “find the 10 most semantically similar documents” — linear brute-force, infeasible at production scale. Approximate Nearest Neighbor (ANN) indexes like HNSW or IVF build graph or cluster structures that let queries skip the majority of the search space, trading a small accuracy loss for orders-of-magnitude speedup. The ACID model relational databases optimize for is also a poor fit: vector workloads need horizontal scale and low latency, pushing toward BASE consistency.
What they’re really probing: Can the candidate trace the performance gap to a structural incompatibility in index design — not just recite “vector databases are better for similarity search”?
Vectors are fixed-size, enabling contiguous storage and O(1) position calculation — a property lost when co-stored with variable-length relational data. Brute-force is feasible only below roughly 5 million vectors; beyond that, ANN index structures become necessary for sub-second latency (Source: Pinecone).
What are embeddings, and where do the vectors actually come from?
Concept: Embedding model as the upstream producer of vectors | Difficulty: junior | Stage: recruiter
Direct answer: Embeddings are dense floating-point arrays — typically 256 to 3,072 dimensions — that encode semantic meaning as produced by an ML encoder model. The vectors don’t come from the database; they come from an encoder model (OpenAI’s text-embedding-ada-002 for text, CLIP for image-text pairs) running outside the database before ingestion.
Geometric distance between vectors approximates semantic similarity: a query about “machine learning frameworks” lands close to TensorFlow and PyTorch documents without shared keywords. The database stores and retrieves K closest neighbors to a query embedding; the embedding model’s quality sets the ceiling on retrieval quality. Swapping models requires re-embedding and re-indexing the entire corpus — a costly operational event interviewers frequently probe as a follow-up.
What they’re really probing: Does the candidate understand that the database and embedding model are separate systems with separate quality ceilings, and that model swaps are expensive operational events?
The embedding model determines both dimensionality and retrieval quality ceiling — a core concept for NLP engineer roles and AI engineers alike. State-of-the-art PDF retrieval models can exceed 100,000 dimensions per page, making dimensionality a hidden computational tax on every distance computation (Source: Qdrant, 2025). Normalize vectors at ingest rather than per query.
When would you use cosine similarity versus L2 distance versus dot product?
Concept: Distance metric selection by model and use case | Difficulty: junior–mid | Stage: recruiter/technical
Direct answer: The choice depends on how the embedding model was trained. Cosine similarity measures the angle between vectors and ignores magnitude — correct for text embeddings where two documents of different lengths can carry identical semantic meaning; most text models are trained to optimize cosine similarity.
L2 (Euclidean) distance measures straight-line distance and is magnitude-sensitive, making it the default for image embeddings and metric-learning models where absolute position carries information. Dot product (inner product) is mathematically equivalent to cosine when vectors are L2-normalized but faster to compute because it skips the normalization step — this is why recommendation systems that pre-normalize at ingest prefer inner product. When uncertain, check the model card: modern models specify the intended metric, and using the wrong one produces measurably degraded retrieval.
What they’re really probing: Whether the candidate knows the mathematical relationship between metrics — especially the cosine ↔ inner-product equivalence under normalization — not just the rule-of-thumb assignment.
When vectors are L2-normalized, cosine is equivalent to inner product; normalizing once at ingest converts all cosine queries to faster dot-product computations (Source: pgvector docs). A common follow-up: “What if your model produces unnormalized vectors?” — cosine without normalization is not equivalent to L2, and behavior shifts with the magnitude distribution.
What is approximate nearest neighbor search, and why approximate?
Concept: ANN trade-off and recall semantics | Difficulty: junior–mid | Stage: recruiter/technical
Direct answer: Approximate Nearest Neighbor (ANN) search trades a small, controlled accuracy loss for orders-of-magnitude latency improvement. Exact K-nearest-neighbor must compare the query to every vector — O(N) complexity infeasible past a few million vectors.
ANN algorithms build data structures (graphs, cluster partitions) at index time that let queries examine only a small fraction of the space. The returned results may not include every true nearest neighbor, and that is acceptable by design. ANN accuracy is measured by recall@K — the fraction of true top-K neighbors returned. The critical insight: the recall-latency curve is non-linear — moving from 90% to 95% recall is cheap; pushing from 97% to 99.9% can multiply latency many times. Define a recall SLO before tuning.
What they’re really probing: Whether the candidate understands ANN systems are probabilistic by design, and that forcing “always return exact matches” semantics onto them is a known production failure mode.
Itamar Syn-Hershko of BigData Boutique flags teams that fight ANN’s intentional probabilistic design as a recurring failure mode. Common keyword-search mental models that degrade ANN performance:
- Scatter-gather across all shards — ANN intentionally skips large regions of the index; querying every shard eliminates the latency benefit.
- Enforcing exact-match guarantees — ANN returns probabilistic neighbors; exact semantics require a brute-force fallback, not an ANN parameter.
- Benchmarking recall on a toy dataset — recall curves are distribution-specific and must be validated on production data before committing to a target SLO.
When is exact brute-force search still the right choice?
Concept: When ANN overhead isn’t justified | Difficulty: junior | Stage: recruiter/technical
Direct answer: Exact brute-force search — comparing the query to every vector — is appropriate when the dataset fits within the latency budget, generally below roughly 5 million vectors depending on dimensions and hardware.
Below that threshold, HNSW or IVF adds build-time cost, memory overhead, and tuning complexity with no meaningful latency benefit, while exact search delivers 100% recall and needs no parameter tuning. Weaviate’s flat index and FAISS’s IndexFlatL2 implement brute-force directly. Weaviate’s dynamic index (v1.25+) automates the transition — flat below 10,000 objects, auto-switching to HNSW above — eliminating the manual call at small scale. The practical rule: if the collection fits in RAM and brute-force passes a load test, don’t reach for ANN.
What they’re really probing: Whether the candidate defaults to the complex tool or reaches for the simplest structure that satisfies the constraint — a signal of engineering judgment over pattern-matching.
Premature quantization — applying index compression before RAM is actually constrained — is the parallel anti-pattern: it introduces approximation error and operational complexity without performance benefit when vectors already fit comfortably in memory. Define actual constraints first, then choose the simplest structure that satisfies them.
Index internals: HNSW, IVF, and quantization
HNSW and IVF are the two dominant ANN index families in production vector search, and understanding how they work internally is what separates engineers who can tune a system from those who can only configure one. HNSW builds a hierarchical graph where each vector connects to its nearest neighbors across multiple layers; search descends the hierarchy, trading a small accuracy loss for O(log N) complexity. IVF partitions the vector space into Voronoi clusters at build time; queries search only the nearest nprobe clusters, so the nlist-to-nprobe ratio governs the recall-latency trade-off.
Beyond the two index families, this section covers quantization compression techniques (scalar, product, binary) and two underappreciated failure modes: the filtering recall problem that motivated filterable HNSW, and the silent quality degradation that accumulates under heavy streaming writes. Engineers who understand these internals can derive parameter tuning recommendations from first principles rather than relying on defaults they don’t understand.
Explain how HNSW works. What do M, ef_construction, and ef_search control?
Concept: HNSW graph structure and runtime-vs-build-time parameters | Difficulty: senior | Stage: technical
Direct answer: HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each vector is a node connected to its nearest neighbors (Source: Malkov & Yashunin, 2018). Search starts at the sparsest top layer, descends via greedy traversal, and narrows to the base graph for a final neighborhood scan.
Three parameters control the trade-offs: M (max edges per node) determines graph connectivity — higher M improves recall and increases memory footprint; ef_construction sets the candidate list at build time — larger means a better-connected graph but more build time, and it is locked in after construction; ef_search controls candidates explored per query, is the primary recall-latency knob after the index is built, and is adjustable at runtime without rebuilding. HNSW achieves O(log N) search complexity.
What they’re really probing: Whether the candidate knows which parameter addresses which concern — specifically that ef_search is runtime-adjustable while M and ef_construction are immutable once built.
Itamar Syn-Hershko of BigData Boutique notes that “ANN indexes assume a relatively stable dataset” — HNSW degrades when new nodes connect based on graph state at ingest rather than the ideal final topology (Source: BigData Boutique, 2026). Qdrant addresses this with a mutable/immutable segment architecture: new data lands in mutable segments immediately; a background optimizer converts them to HNSW indexes on a schedule.
How does an IVF index work, and what do nlist and nprobe trade off?
Concept: IVF partitioning and cluster search parameters | Difficulty: mid | Stage: technical
Direct answer: An IVF (Inverted File) index partitions the vector space into Voronoi cells via k-means clustering at build time. Each vector is assigned to its nearest centroid and stored in that cluster’s inverted list. At query time, the search finds the nprobe nearest centroids and searches only those clusters.
nlist controls how many clusters are built — more clusters means smaller, faster per-cluster searches, but without proportionally increasing nprobe, true nearest neighbors can fall across cluster boundaries and get missed. nprobe controls clusters searched per query — more nprobe means better recall at higher latency. The Milvus starting point: nlist ≈ sqrt(N), then tune nprobe until the recall target is met. IVF tends to be faster than HNSW at very large scales because centroid lookup is simpler than multi-layer graph traversal.
What they’re really probing: Whether the candidate understands that nlist and nprobe are correlated — raising nlist without raising nprobe actually hurts recall, which surprises candidates who assume “more clusters = better search.”
Milvus offers three IVF variants (Source: Milvus docs):
- IVF_FLAT — raw float32 vectors; highest recall, highest memory cost per vector.
- IVF_SQ8 — scalar quantization; ~4x memory compression with modest recall loss.
- IVF_PQ — product quantization; highest compression ratio, suited for billion-scale.
pgvector’s IVFFlat follows the same design; recommended lists = rows/1000 for tables over 1M rows, or sqrt(rows) for smaller collections (Source: pgvector docs).
Compare scalar, product, and binary quantization. When would you use each?
Concept: Quantization compression levels and recall trade-offs | Difficulty: senior | Stage: technical
Direct answer: Quantization compresses floating-point vectors to reduce memory footprint at some cost to recall. Scalar Quantization (SQ8) converts float32 to uint8 — ~4x memory compression with modest recall loss; the lowest-risk first step when an HNSW index is borderline for available RAM.
Product Quantization (PQ) splits each vector into m sub-vectors and quantizes each into a codebook entry; much higher compression ratios make billion-scale search feasible within GPU memory budgets, at the cost of greater recall loss and a required rescoring pass. Binary Quantization (BQ) is the most aggressive: each dimension becomes a single bit, achieving ~32x compression and up to 40x faster distance computation, but BQ requires rescoring against full-precision vectors and works best on high-dimensional embeddings approximately normally distributed around zero.
| Method | Compression | Recall impact | When to use | Rescoring required |
|---|---|---|---|---|
| Scalar (SQ8) | ~4x (float32 → uint8) | Modest loss | Index barely fits in RAM; lowest-risk first step | No |
| Product (PQ) | High (configurable via m sub-vectors) | Moderate–high loss | Billion-scale; GPU memory budgets | Recommended |
| Binary (BQ) | ~32x; up to 40x faster candidate pass | High without rescore | High-dim embeddings (~normally distributed); extreme memory constraints | Required |
What they’re really probing: Can the candidate order the three methods by compression-vs-accuracy trade-off and explain exactly when rescoring is required versus optional?
David Myriel of Qdrant describes the two-phase retrieval pattern as the production-grade approach: quantized vectors for a broad fast candidate pass, then rescore the top-K against full-precision vectors to recover accuracy without paying full-precision cost on every computation. Benchmark recall on your specific model and dataset before deploying — impact varies significantly by data distribution.
How do you combine metadata filters with vector search without wrecking recall?
Concept: Pre-filter, post-filter, and filterable HNSW strategies | Difficulty: senior | Stage: technical
Direct answer: Combining metadata filters with vector search is harder than it appears — the naive approach breaks recall silently: no errors, just degraded results.
Post-filtering runs ANN search on the full index then filters results: it fails when all top-K candidates fall outside the filter predicate, returning fewer than K results. Pre-filtering restricts the candidate pool before search: it fragments the HNSW graph, which was built on the full dataset, breaking small-world connectivity when only a filtered subset is visible. Filterable HNSW (used by Qdrant, Elasticsearch, and OpenSearch’s Lucene engine) solves this by building per-payload-value subgraph links at index time, maintaining connectivity for any filtered query without a full rebuild. Highly selective filters expose the post-filter miss most severely.
| Strategy | How it works | Failure mode | When to use |
|---|---|---|---|
| Post-filter | ANN search on full index, then filter results | Returns fewer than K results when filter is highly selective | Filter matches >10% of corpus; acceptable recall loss |
| Pre-filter | Restrict candidate pool by filter, then search | Fragments HNSW graph; degrades recall on selective filters | Generally avoid for HNSW; viable for IVF with high nprobe |
| Filterable HNSW (in-graph) | Payload-aware links built at index time; filter applied during traversal | Requires payload indexes created before data ingestion | Default for Qdrant; best recall at any filter selectivity |
What they’re really probing: Whether the candidate knows this is an unsolved problem that motivated dedicated vector databases — not a configuration footnote — and which systems implement which strategy.
The Qdrant Academy course on filterable HNSW frames this as the “core challenge of real-world vector search” and maps filter selectivity to query planner strategy. In pgvector, filtered queries can swing from 50ms to 5s depending on filter selectivity and a cost model not tuned for vector similarity (Source: Alex Jacobs).
What happens to an HNSW index under heavy streaming writes?
Concept: HNSW graph quality degradation under continuous inserts | Difficulty: senior | Stage: technical
Direct answer: HNSW graph quality degrades under heavy streaming writes because each inserted vector connects to neighbors based on graph state at that moment, not the ideal final topology. As inserts accumulate, new nodes attach to suboptimal neighbors, increasing traversal depth and reducing recall without triggering any visible error — the index returns results, just less accurate ones.
This differs qualitatively from keyword-index degradation: with HNSW, more writes degrade search quality itself, not just response speed. Itamar Syn-Hershko characterizes this as “ANN indexes assume a relatively stable dataset” — the graph is designed for a batch-built, relatively static collection. The operational remediation is a periodic full index rebuild (“graph refresh”), requiring either planned downtime or a blue-green index swap.
What they’re really probing: Whether the candidate knows that degradation is silent (no errors, just lower recall) and that the response is planned rebuild, not simply more hardware.
Qdrant’s mutable/immutable segment architecture partially mitigates this: new data lands in mutable segments immediately; a background optimizer converts them to immutable HNSW indexes; search merges results across segments during the transition. Batch inserts preserve graph quality better than continuous streaming because the index is built over a stable snapshot.
When to reach for HNSW, IVF, flat, or quantization: an index selection matrix
Index selection in vector search is the highest-leverage decision in a system design interview — the wrong choice at this step propagates through query latency, recall ceiling, operational complexity, and memory cost for the lifetime of the deployment. The matrix below maps five index types to the constraints that favor them: dataset scale, recall target, write pattern, and memory budget. Every cell is backed by a published benchmark or implementation detail; the update-tolerance column for IVF variants is marked qualitative where hard production numbers aren’t in the literature.
The practical read-order for the table: start at your dataset scale, then check write frequency in the update-tolerance column, then verify the memory budget against the footprint column before committing to an index choice. Candidates who internalize this decision sequence can justify any selection from first principles rather than naming a default.
| Index | Best-for dataset size | Recall | Query latency | Memory footprint | Update tolerance | Key knobs |
|---|---|---|---|---|---|---|
| HNSW | 1M–100M vectors | Highest (95–99%+ with tuning); gold standard for recall | O(log N); sub-millisecond when fully RAM-resident (Source: Malkov & Yashunin, 2018) | High — full float32 graph; 1M × 1536-dim ≈ 6 GB plus graph overhead (Source: Qdrant, 2025) | Low — streaming inserts degrade graph quality silently (Source: BigData Boutique, 2026) | M (graph edges per node); ef_construction (build quality); ef_search (runtime recall vs. latency) |
| IVF-FLAT | 1M–100M vectors | High with adequate nprobe; exact within each searched cluster | Faster than HNSW at very large N with low nprobe; cluster lookup is simpler than graph traversal | Full float32 in clusters; similar to HNSW without graph edge overhead | Moderate — new inserts append to nearest centroid bucket without graph degradation; rebuild needed if data distribution shifts significantly | nlist (clusters built); nprobe (clusters searched per query) |
| IVF_SQ8 | 10M–500M vectors | Modest recall loss vs. IVF-FLAT; no rescoring required | Comparable to IVF-FLAT with lower RAM pressure | ~4x lower than IVF-FLAT (float32 → uint8) (Source: Qdrant, 2025) | Moderate (same cluster-append behavior as IVF-FLAT) | nlist; nprobe; no additional quantization knobs |
| IVF_PQ | 100M–1B+ vectors | Lower recall than SQ8; rescore pass required to recover accuracy (Source: FAISS paper, 2024) | Fast first-pass; total latency includes rescore step overhead | Lowest among IVF family; enables billion-scale search within GPU memory budgets | Moderate (same as IVF-FLAT) | nlist; nprobe; m (sub-vectors per vector); nbits per sub-vector codebook |
| Flat / brute-force | <5M vectors (Source: Pinecone) | 100% exact — no approximation error | Linear O(N); fast at small scale, infeasible at large scale | Full float32; no overhead beyond raw vectors | Highest — no index structure to maintain; updates are simple appends | None |
| HNSW + Binary Quantization | 50M–500M vectors | Recoverable to near-HNSW recall with rescoring; up to 40x faster candidate pass (Source: Qdrant, 2025) | ~40x faster initial candidate pass; total latency depends on rescore overhead | ~32x lower than full float32 HNSW; extreme memory reduction | Low — same graph degradation pattern as HNSW under streaming writes | M; ef_construction; ef_search; rescore_top_k (candidates passed to full-precision rescore) |
Under 5M vectors, flat search delivers 100% recall with no tuning overhead — don’t add indexing complexity before there’s a reason. Between 1M and 100M vectors, HNSW is the default: highest recall, fastest query latency when RAM-resident, and runtime-adjustable ef_search (Source: Pinecone). Prefer IVF when the write pattern is continuous streaming — IVF’s cluster-append behavior doesn’t degrade the index structure the way HNSW graph writes do — or when full float32 HNSW exceeds available RAM.
Quantization is a memory lever, not a performance shortcut — apply it only when RAM is the binding constraint, not preemptively. SQ8 is the lowest-risk step: 4x compression with modest recall loss and no rescoring overhead; Binary Quantization delivers 32x compression and a 40x faster candidate pass but requires rescoring against full-precision vectors for acceptable recall. Premature quantization — when vectors already fit in RAM — adds approximation error with no benefit.
In the interview, the evaluator watches for one thing: can the candidate justify the choice from first principles rather than naming a default stack? Strong candidates reason aloud — “I’d start with HNSW at current scale, create payload indexes before ingestion to support filtered search, and evaluate binary quantization if the collection grows past 50M vectors and RAM becomes the binding constraint” — demonstrating operational sequence thinking, not just algorithm naming.
System design: RAG retrieval, hybrid search, and multi-tenant vector stores
RAG pipeline architecture, hybrid search, and distributed sharding at scale are the three problem classes that fill the system design portion of a vector search interview. These questions test a fundamentally different skill than index internals — a candidate must not just know that HNSW offers lower latency than IVF, but understand how to compose the full retrieval stack: chunking and embedding pipelines, ANN index selection for a given QPS and latency SLO, hybrid BM25+dense fusion, multi-tenant ACL enforcement at the retrieval layer, and shard fan-out budget at hundreds of millions of vectors.
Five questions here mirror the prompts that appear in senior ML engineer system design loops in 2026: a 5M-document RAG system at 300 QPS, hybrid search implementation with Reciprocal Rank Fusion, tenant isolation and per-document ACL strategies, the pgvector-vs-dedicated-store trade-off, and distributed sharding with scatter-gather query patterns. Candidates who pass this section can sketch a component-level retrieval architecture and reason through its failure modes before the interviewer asks.
Design a multi-tenant RAG search system over 5M documents at 300 QPS.
Concept: End-to-end RAG pipeline design under production constraints | Difficulty: Hard | Stage: system-design
Direct answer: Reported as an OpenAI Technical Screen question (Source: prachub.com), this prompt tests seven design areas simultaneously. The 5M-document corpus needs an ingestion pipeline: semantic chunking at ~512 tokens, dedup, metadata extraction with tenant ID and ACL tags, and embedding generation before upsert to a tenant-partitioned vector store.
The retrieval layer combines hybrid search — BM25 for exact-match terms (error codes, contract clauses, SKUs) merged with dense ANN via Reciprocal Rank Fusion — with a cross-encoder reranker to narrow 20 candidates to 5 before generation. The 300 QPS at p95 ≤ 1.2s breaks down as 50ms embedding + 50–100ms retrieval + 800ms LLM generation. ACL enforcement must happen at retrieval time — filter by tenant namespace and ACL tags during the ANN query, not post-generation — to prevent context from leaking across tenants.
What they’re really probing: Can the candidate decompose the latency budget, identify the retrieval layer as the correct ACL enforcement point, and name what breaks first at 10x traffic before the interviewer has to ask?
For deeper RAG pipeline coverage, the RAG pipelines guide covers the full stack; the LangChain guide covers the framework layer. Documents embed and upsert to mutable segments immediately; a background optimizer builds HNSW indexes incrementally — the pattern Qdrant’s segment architecture is designed for.
How would you implement hybrid search, and how does Reciprocal Rank Fusion work?
Concept: BM25 + ANN fusion and the RRF algorithm | Difficulty: mid–senior | Stage: technical
Direct answer: Hybrid search combines sparse (BM25) keyword retrieval with dense ANN semantic retrieval and merges results via a fusion algorithm. The standard method is Reciprocal Rank Fusion (RRF): for each document, the score is the sum of 1/(k + rank_i) across all retrieval systems, where k = 60 is the standard constant and rank_i is the document’s rank from system i.
Because RRF is rank-based rather than score-based, it safely merges results from systems with incompatible score scales — BM25 TF-IDF scores and cosine similarity share no common range, but ranks do. Weaviate implements hybrid search with an `alpha` parameter (0.0 = pure BM25, 1.0 = pure vector, 0.75 default); Elasticsearch and OpenSearch expose RRF directly. The production motivation: pure vector search fails on exact-match queries — product SKUs, error codes, and contract clause references require BM25 precision that semantic embeddings consistently miss.
What they’re really probing: Whether the candidate can write out the RRF formula, explain why rank-based fusion outperforms score-based fusion when merging heterogeneous retrieval systems, and name where pure vector search breaks down in enterprise deployments.
Graham Gillen of Pureinsights observed in 2026 that teams who deployed pure vector search and had to add keyword retrieval back found “dedicated vector databases found it harder to add keyword search than Elasticsearch found it to add vector search” (Source: Pureinsights, 2026). Hybrid search consistently outperforms either retrieval mode alone across benchmark datasets and enterprise deployments.
How do you isolate tenants and per-document ACLs inside one vector store?
Concept: Tenant isolation strategies and their operational trade-offs | Difficulty: senior | Stage: system-design
Direct answer: Multi-tenancy means Tenant A’s query must never return Tenant B’s vectors, even when semantically similar. Two primary strategies exist. Namespace-per-tenant (Pinecone’s model) creates a separate partition per tenant, enabling per-tenant upsert, query, delete, and data isolation at the API level; the drawback is collection proliferation and complex cross-tenant operations.
Payload-filter-per-tenant (Qdrant’s recommended approach) stores all tenants in one collection with a tenant_id metadata field and filters every query; this avoids collection proliferation and simplifies shard planning, but requires filtered search to perform well at high selectivity. The Qdrant production checklist warns explicitly: “partition by payload field rather than creating separate collections per user/tenant to avoid shard proliferation” (Source: Qdrant production checklist).
What they’re really probing: Whether the candidate understands the operational cost of namespace-per-tenant proliferation and can articulate the trade-off between isolation guarantees and infrastructure complexity.
Two implementation details separate correct multi-tenancy from near-correct:
- Pinecone namespace isolation — per-namespace upsert, query, delete, and fetch gives each tenant full CRUD isolation without requiring separate indexes (Source: Pinecone docs).
- ACL enforcement at retrieval time — compound filters (tenant_id = X AND user_id IN [allowed_users]) must run during the ANN query, not as a post-fetch application filter; a post-fetch filter can expose intermediate context to unauthorized callers.
pgvector or a dedicated vector database — how do you decide?
Concept: Operational context determining the right tool | Difficulty: mid–senior | Stage: technical
Direct answer: The decision hinges on scale, write throughput, and filter complexity. pgvector is the right answer when the dataset is under a few million vectors, the team already operates Postgres, the workload is read-heavy, and ACID transactions for hybrid relational+vector queries are valuable.
Alex Jacobs identified the specific failure conditions: “None of the blogs mention that building an HNSW index on a few million vectors can consume 10+ GB of RAM or more. On your production database. While it’s running. For potentially hours.” (Source: Alex Jacobs) Dedicated vector databases — Qdrant, Milvus, or managed services like Pinecone — are the right answer when filtered search recall is critical, write throughput is high, or the collection exceeds ~10M vectors; purpose-built filterable HNSW and dedicated hardware layouts outperform general-purpose Postgres at those thresholds.
What they’re really probing: Whether the candidate has a principled decision framework rather than a default preference — and whether they know the specific failure conditions that make pgvector the wrong choice at scale.
pgvector 0.8.0 (October 2024) added iterative scanning for HNSW and IVFFlat: when a filter exhausts the initial candidate set before K results are found, the index expands the search rather than returning a short result (Source: PostgreSQL.org, 2024). This closes the over-filtering recall gap but doesn’t give pgvector purpose-built vector hardware layouts — a functionality fix, not a fundamental architecture fix at scale.
How do you shard and query a vector index across hundreds of millions of vectors?
Concept: Distributed sharding and scatter-gather overhead at scale | Difficulty: senior | Stage: system-design
Direct answer: At hundreds of millions to billions of vectors, the dataset exceeds single-machine capacity, requiring horizontal sharding. The query pattern becomes scatter-gather: fan out to all shards in parallel, collect top-K candidates from each, merge and globally re-rank, then return the final top-K.
The critical insight: at this scale, fan-out latency and result merge overhead dominate over raw distance computation — each shard adds a network round-trip, CPU scheduling, and merge cost that can exceed the ANN search itself. The sharding key matters as much as the fan-out strategy: shard by data, not by query pattern or tenant. ANN indexes require dense, coherent vector spaces; sharding by semantic category or tenant fragments the space, forcing queries to touch more shards to achieve the recall of a coherent single index.
What they’re really probing: Whether the candidate understands that scatter-gather is not free at scale and that naive tenant-based sharding creates a recall-coverage problem that can’t be tuned away.
Reddit’s evaluation at 340M vectors found Milvus’s heterogeneous architecture — separate query nodes, data nodes, and index nodes — scales better at high replication because ingestion and query traffic don’t contend on the same nodes (Source: Milvus/Reddit evaluation). BigData Boutique warns: “querying too many shards ‘just in case’ is a performance anti-pattern” — shard fan-out budget must be treated as carefully as ANN index configuration (Source: BigData Boutique, 2026).
Scaling vector search: RAM residency, recall economics, and what breaks first
Vector search systems fail in predictable ways — and knowing the failure sequence, along with the hard numbers that define each threshold, is what separates production engineers from developers who have only run the tutorials. The three canonical failure modes appear in order of frequency: the HNSW index exceeding available RAM, which produces disk I/O spikes and latency degradation with no obvious error; streaming write accumulation that silently degrades graph quality over weeks; and recall-chasing above 97%, which multiplies latency without proportional quality gain.
David Myriel of Qdrant states: “Issues like memory errors, disk I/O spikes, and search delays underscore how default configurations can fail spectacularly under production loads” (Source: Qdrant, 2025). This section covers four questions interviewers use as senior-level filters: recall-latency tuning against a defined SLO, memory reduction strategies in priority order, failure mode sequencing, and the metrics that reveal degradation before users report it. Candidates who answer these well have shipped production systems, not just read the documentation.
How do you tune the recall-versus-latency trade-off?
Concept: ANN parameter tuning against a defined recall SLO | Difficulty: mid | Stage: technical
Direct answer: Recall-versus-latency tuning works through index-specific parameters: ef_search for HNSW (candidate list at query time — higher value, better recall, higher latency), nprobe for IVF (clusters searched per query), and num_candidates for Elasticsearch-style systems.
The recall curve is non-linear in a way that matters practically: going from 90% to 95% recall is cheap; 95% to 97% costs roughly 20% more latency; 97% to 99.9% can multiply latency many times over. The professional sequence: set a recall SLO first — 90%, 95%, or 97% — then use ANN-Benchmarks or your own harness to find the parameter that hits that target on your specific dataset. Itamar Syn-Hershko of BigData Boutique warns against recall-chasing: teams that over-tune to 99%+ routinely land at latency an order of magnitude above their SLO with no measurable user-experience benefit.
What they’re really probing: Whether the candidate benchmarks on their own dataset rather than trusting published numbers, and whether they understand that the recall curve is non-linear with a cost inflection point above 95%.
ANN-Benchmarks provides standardized recall@10 vs. queries-per-second measurements across datasets including SIFT, GloVe, and NYTimes (Source: ANN-Benchmarks). Published benchmarks are informative starting points, not substitutes for profiling on your own embedding model and production data distribution — dataset characteristics affect HNSW graph quality and IVF cluster separation in ways generic benchmarks won’t capture.
How would you cut memory usage while keeping recall acceptable?
Concept: Quantization sequence and the principle of constrained optimization | Difficulty: mid–senior | Stage: technical
Direct answer: Memory reduction follows a priority sequence — apply compression only under a genuine constraint, not as a default optimization. First, confirm RAM is actually the bottleneck: if vectors fit comfortably in memory, raw float32 is faster and simpler. When RAM is tight, apply Scalar Quantization (SQ8) first: float32 → uint8 cuts memory ~4x with modest recall loss and no rescoring overhead — the lowest-risk step.
If SQ8 is insufficient, Product Quantization (PQ) provides much higher compression at the cost of greater recall degradation and a mandatory rescore pass. For extreme memory constraints with high-dimensional, approximately normal embeddings, Binary Quantization achieves ~32x compression and up to 40x faster candidate pass — but rescoring against full-precision vectors is required to recover recall. A complementary lever: memmap storage handles datasets larger than physical RAM at slightly higher latency; SSDs are required since HDDs introduce unacceptable random-read latency for HNSW graph traversal.
What they’re really probing: Whether the candidate applies compression in a principled sequence — profile first, compress only under an actual constraint — versus immediately reaching for the most aggressive option.
David Myriel of Qdrant describes the two-phase retrieval pattern as the production-grade approach. Decision sequence before applying quantization:
- Confirm RAM is actually constrained — if vectors fit comfortably in memory, raw float32 avoids approximation error with no configuration overhead.
- Apply SQ8 first (float32 → uint8, ~4x compression, no rescoring required) — lowest risk; validate recall on production data before proceeding.
- Use PQ for billion-scale or BQ for extreme memory reduction; both require a rescoring pass against full-precision vectors to recover acceptable recall.
What breaks first when a vector search system scales, and how do you see it coming?
Concept: Production failure mode sequence and leading indicators | Difficulty: senior | Stage: technical
Direct answer: The most common first failure is the HNSW index exceeding available RAM — 1 million 1,536-dimensional embeddings consume approximately 6 GB, and default configurations don’t reserve dedicated WAL paths or pre-size memory allocations; the first symptom is disk I/O spikes and query slowdowns as the index spills to disk.
The second failure is HNSW graph degradation from streaming inserts, which is silent — no error raised, recall decreases quietly over weeks. Third is chasing recall above 97%, which multiplies latency without proportional quality gain. David Myriel states: “default configurations can fail spectacularly under production loads” — the checklist below captures the numbers to anticipate these failures before users report them.
| Signal / Number | What it means in production | Source |
|---|---|---|
| 1M × 1536-dim ≈ 6 GB | Raw float32 storage floor; HNSW adds graph edge overhead on top of this baseline | Qdrant, 2025 |
| 90% → 95% recall: cheap | Low ef_search / nprobe; minor latency impact; standard operating range | BigData Boutique, 2026 |
| 95% → 97% recall: ~+20% latency | Meaningful but often acceptable if SLO requires it; budget for the cost | BigData Boutique, 2026 |
| 97% → 99.9% recall: latency multiplied many times over | Brutally expensive; rarely justified by user-experience improvement | BigData Boutique, 2026 |
| HNSW build: 10+ GB RAM for hours | On a live Postgres production instance, while it serves other queries; causes lock contention | Alex Jacobs |
| pgvector filtered query: 50ms → 5s swing | Query planner cost-model mismatch on vector + filter queries; selectivity-dependent | Alex Jacobs |
| Context rot above 50k–150k tokens | RAG retrieval performance degrades even within the model’s stated maximum context window | ZenML, 1,200 deployments, 2025 |
What they’re really probing: Whether the candidate knows the failure modes are predictable and monitorable — not random events — and can name the leading indicators before the interviewer has to prompt them.
Alex Strick van Linschoten’s analysis of 1,200 production LLM deployments found that teams reaching 80% retrieval quality quickly spent the majority of subsequent effort pushing past 95% — the last stretch consumes disproportionate engineering time (Source: ZenML, 2025). Define the retrieval quality SLO first; stop tuning when you hit it.
What metrics prove a vector search system is healthy after launch?
Concept: Three-layer monitoring for production retrieval systems | Difficulty: mid–senior | Stage: technical
Direct answer: A healthy vector search system requires monitoring at three layers. At the infrastructure layer: RAM utilization (is the HNSW index RAM-resident or spilling to disk?), disk I/O rate (a spike signals index spill), CPU saturation per query node, and WAL lag.
At the retrieval layer: recall@K measured against a ground-truth evaluation set monthly or after each model update — recall is a measured property, not an assumed one — query latency at p50/p95/p99 broken down by filter selectivity, and result set size per query (fewer than K results signals post-filter over-selectivity).
At the RAG application layer: faithfulness (do generated answers cite retrieved passages?), answer relevance, and NDCG@K for ranking quality. Cathy Zhang and Dr. Malini Bhandaru of Intel/Milvus identify load latency — time to reload data and rebuild an ANN index after restart — as a metric most teams neglect until a restart event exposes the gap (Source: Intel/Milvus, 2024).
What they’re really probing: Whether the candidate treats recall as an active measurement requiring a ground-truth evaluation harness, not a passive property assumed to be stable after initial deployment.
Vespa engineering (2025) identifies stale results from batch-oriented indexing as a production gap — a document updated in the source system but not yet re-indexed causes the retrieval layer to serve outdated context without raising an error signal (Source: Vespa, 2025). Index freshness lag — the delay between a source-system update and the vector store reflecting it — should be an explicit SLO alongside recall and latency.
Questions to ask your interviewer (and what each one signals)
Reverse questions in a vector database technical screen aren’t formalities — they’re the last evaluation signal the interviewer sees. The questions below are ranked by signal strength: the first few demonstrate production experience, the later ones fit any mid-level candidate.
- “What index type are you running in production, and what’s the key limitation you’ve hit with it?”
Signals that index selection is an ongoing operational decision for you, not a one-time setup step — and invites the interviewer to share a real constraint, shifting the conversation from evaluation to collaboration. - “How do you measure recall in production — do you have a ground-truth evaluation set running on a schedule?”
Signals production maturity. You know recall drifts after model updates and streaming inserts, and that measuring it requires a purpose-built harness. - “How does your system handle filtered vector search — pre-filter, post-filter, or a filterable HNSW implementation?”
Signals that you’ve worked with the filtering problem and know systems differ substantially here, with the wrong default degrading recall silently. - “Where does end-to-end RAG latency actually live — the embedding step, the retrieval step, or LLM generation?”
Signals system-level thinking: you know that optimizing only the ANN step frequently misses the real bottleneck. - “How are you handling index freshness — batch rebuild pipeline, streaming ingestion, or Qdrant’s mutable/immutable segment architecture?”
Signals that you know HNSW degrades under streaming writes, this is an active ops concern, and you’re familiar with the mitigation strategies different systems offer. - “What drove the original choice between pgvector and a dedicated vector database — and would the team make the same call today?”
Signals awareness that the market has changed fast enough that stacks chosen in 2023 may look different evaluated in 2026. - “What’s the hardest open problem on the retrieval side right now — recall at your current dataset size, memory cost, or the keyword-plus-vector hybrid integration?”
Signals long-term thinking: you’re assessing where the team’s next hard engineering problem is, not just whether the role exists.
A one-week prep plan for vector database interviews
One to two focused hours per day, each targeting a specific competency tier from the evaluation funnel above.
- Days 1–2: Core concepts and hands-on indexing. Read the HNSW paper abstract and Qdrant’s dedicated-vector-search and resource-optimization articles for intuitions behind M, ef_construction, ef_search, and the cosine/L2/dot-product choice; then run IndexFlatL2 versus IndexHNSWFlat on a 50K-vector dataset from ann-benchmarks.com and measure recall at equal query latency. Exit goal: explain all three HNSW parameters and the distance-metric selection rule from memory in 90 seconds.
- Days 3–4: System design and production failure modes. Read BigData Boutique’s “Scaling Vector Search from Millions to Billions” in full, then work through the OpenAI RAG system-design question (5M documents, 300 QPS, p95 ≤ 1.2s, multi-tenant ACLs) with a 10-minute timer — whiteboard the chunking pipeline, retrieval layer, ACL enforcement point, and latency budget decomposition. Read the Milvus/Reddit 340M-vector evaluation for scatter-gather trade-offs in a real production choice.
- Day 5: Tool-specific depth matched to the target stack. For Pinecone: serverless or pod-based architecture and namespace isolation; for Milvus/Qdrant: distributed node types, quantization sequence, and filterable HNSW or post-filter; for pgvector: HNSW build cost on a live instance, iterative scanning, and filtered query planner failure mode. Study the system the job description mentions and one alternative for comparison.
- Days 6–7: Mock interviews and the reverse-question list. Record yourself answering the index-selection matrix question out loud, then practice the recall-latency tuning answer and finalize three reverse questions from the list above. Final check: run through the numbers-to-know once — the 6 GB per 1M-vector baseline and the recall curve breakpoints (90→95 cheap, 97→99.9 expensive) — without consulting the table.
