Mine Novel Examples from an Unlabeled Corpus

Question

Design a pipeline that selects rare, useful, and diverse examples from a very large unlabeled text corpus for human labeling.

## Requirements

- Define the downstream task, novelty target, labeling budget, and what counts as success.
- Remove unsafe, low-quality, duplicate, and near-duplicate content while tracking provenance.
- Represent examples for semantic retrieval and compare distance-to-labeled-data, density, uncertainty, and rule-based signals.
- Balance novelty against diversity and relevance so annotators are not flooded with one unusual cluster.
- Build a review workflow with sampling, auditability, and feedback from annotators.
- Evaluate selection quality without assuming unselected data is negative.

Describe offline stages, indexes, scoring, batch construction, and how newly labeled data changes the next round.

## Follow-ups

1. Detect distribution shift over time.
2. Protect privacy and remove memorized personal data.
3. Allocate budget across languages or domains.
4. Compare active learning with novelty mining and random baselines.