Question
You are given a dataset of labeled examples and a helper that scores candidate label completions under a language model. Design a prompt-driven binary classifier, evaluate it, and improve it systematically. **Assumption:** the exact domain and helper signature will be provided during the interview. The core task is to produce a real-valued score for each example, choose a label, and measure quality—not merely call the model once. ## Requirements - Build prompts from row data without leaking the gold label. - Compare the two label candidates using their model scores, converting log scores correctly when a probability or calibrated score is required. - Return both the chosen label and a confidence-like score. - Compute at least accuracy and a loss or calibration metric over the dataset. - Make prompt templates explicit and reproducible. - Handle malformed output and model-call failures without silently counting them as correct. ## Staged follow-ups 1. Establish a simple baseline prompt and evaluation loop. 2. Diagnose errors and revise instructions or examples based on categories, not individual test answers. 3. Tune a decision threshold and explain how it changes precision and recall. 4. Add structured output, retries, and chained calls while controlling latency and cost. 5. Design a held-out evaluation and guard against prompt overfitting.