Question
You receive labels from multiple annotators, with missing entries. Infer one binary label per item, identify annotators who systematically disagree with their peers, remove them, and recompute the labels. ```python def majority_labels(annotations: np.ndarray) -> np.ndarray: ... def filter_bad_annotators(annotations: np.ndarray) -> list[int]: ... def aggregate_labels(annotations: np.ndarray, gold=None): ... ``` ## Assumptions Rows are annotators, columns are items, labels are `0`, `1`, or `-1` for missing. Majority ignores missing entries and ties resolve to `1`. An annotator is bad when agreement with provisional labels on overlapping items is strictly below `0.5`. If `gold` is supplied, also return accuracy rounded to a stated precision. ## Requirements - Define behavior for items with no labels and annotators with no overlap. - Avoid treating missing values as votes. - Recompute the final majority after filtering. - State complexity and numerical representation. ## Follow-ups 1. Weight annotators by estimated reliability. 2. Implement an EM-style latent-truth model such as Dawid–Skene. 3. Account for item difficulty and correlated annotators. 4. Compare raw agreement with chance-corrected measures.