Reported interview periodJan 2026–Sep 2026
Aggregate Labels from Noisy Annotators
You receive labels from multiple annotators, with missing entries. Infer one binary label per item, identify annotators who systematically disagree with their peers, remove them, and recompute the labels.
Practice focusAssumptions Rows are annotators, columns are items, labels are 0 , 1 , or -1 for missing. Majority ignores missing entries and ties resolve to 1 . An annotator is bad when agreement with provisional labels on overlapping items is strictly below 0.5 . If gold is supplied, also return accuracy rounded to a stated precision. Requirements - Define behavior for items with no labels and annotators with no overlap. - Avoid treating missing values as votes. - Recompute the final majority after filtering. - State complexity and numerical representation.…