Question
You are given a seed URL and a parser object. Calling `htmlParser.getUrls(url)` returns the links found on that page. Return every reachable URL whose hostname matches the seed URL's hostname, including the seed itself. ```python def crawl(startUrl: str, htmlParser: HtmlParser) -> list[str]: ... ``` ## Requirements - Visit each accepted URL at most once. - Never follow a link to another hostname. - Use the supplied parser rather than implementing HTTP fetching. - Return all reachable accepted URLs; result order is not significant. - Treat complete URL strings as visit identities. Compare parsed hostnames, not hostname substrings; different ports and schemes on the same hostname remain eligible. Assume syntactically valid absolute HTTP(S) URLs. No URL canonicalization, redirect fetching or network access is required. The supplied offline parser returns an empty list for an unknown page. Public checks compare sorted output; implement a correct single-threaded traversal first. ## Design discussion (not graded) - Add bounded concurrency for an I/O-bound parser, preventing duplicate processing and premature termination. - Add per-host rate limits, retries with backoff, cancellation, and partial-failure reporting. - Design a distributed crawler with durable work ownership, deduplication, politeness, and recovery after a worker dies. - Explain when a Bloom filter is acceptable and what false positives would mean.