Reported interview periodSep 2025–Jun 2026
Implement a Longest-Match Tokenizer
Given an input string and a vocabulary, tokenize from left to right using longest-prefix matching.
Practice focusAt each position, emit the integer ID of the longest vocabulary entry matching the remaining text. If nothing matches, emit vocab["UNK"] and advance by one character. With collapse unk=True , collapse consecutive unknown IDs into a single ID; otherwise emit one for each unmatched character. The literal vocabulary entry UNK is reserved and must not match input text as an ordinary token. Assume every token has a distinct integer ID and the vocabulary contains UNK . Requirements and edge cases - Return an empty list for empty input.…