Design retrieval and ranking for a knowledge assistant

Question

## Scenario

Design the retrieval and ranking system for an assistant that answers questions over a large, frequently updated document collection. The system must return relevant supporting passages with low interactive latency and provide enough evidence for the answer to be inspected.

Assume tens of millions of passages, multiple languages, access-controlled documents, and a stream of new or edited content.

## Goals

- Retrieve relevant passages for both keyword-heavy and semantic queries.
- Rank the best evidence within a latency budget suitable for an interactive product.
- Respect document-level permissions at query time.
- Support indexing updates without long periods of stale or missing content.
- Evaluate retrieval separately from final answer generation.
- Degrade usefully when the highest-quality model or index is unavailable.

## Design discussion

Cover the following areas:

1. Document parsing, chunking, metadata, deduplication, and versioning.
2. Sparse, dense, or hybrid candidate generation.
3. Training examples for a retriever: positive pairs, hard negatives, weak labels, and leakage prevention.
4. Dual-encoder versus cross-encoder ranking, including latency and quality trade-offs.
5. Index topology, update path, caching, and permission filtering.
6. Offline metrics such as recall@k, MRR, and NDCG, plus online product metrics.
7. How retrieved evidence is passed to a generative model and how citations are preserved.

State a concrete scale and latency target, then estimate the main capacity and cost drivers.

## Follow-ups

- Queries shift rapidly after a breaking event and no labeled data exists yet.
- A relevant document is long, but the answer depends on two distant passages.
- Authorization changes must take effect within minutes.
- The generator gives a fluent answer that is unsupported by retrieved passages.