Public Web Data Collection Checklist: Freeze the Schema First

1. Freeze the schema with sample rows

A field name does not fully define a field. “Price” needs rules for currency, tax, promotions, and ranges. “Company name” needs a choice between brand name, legal name, and display name. “Published date” needs a timezone and a convention for dates that cannot be identified.

Fill three to five sample rows from real pages before bulk collection. Samples reveal field ambiguity, page variants, and transformation cost. Move to scale only after those rows are accepted.

2. Define public sources and access boundaries

“Visible on the web” does not mean “collectable by any method.” Scope should define publicly accessible pages, allowed domains, page types, request rate, and exclusions while respecting applicable terms, technical restrictions, and law. Content behind login, paywalls, personal accounts, or access-control bypasses should not enter a public-web task by default.

A site’s robots.txt is an important machine-readable rule for automated crawlers. It is not access authorization and does not replace site terms or applicable legal review. Pause sources with unclear boundaries for confirmation.

• List allowed domains and explicitly excluded domains or paths.
• Distinguish ordinary public pages, public APIs, downloads, and search result pages.
• Set request rate, retry behavior, and stop conditions.
• Exclude personal data, sensitive information, and access-control bypasses.

3. Make every row traceable

A dataset is maintainable only when an error can be located. Store at least source_url and accessed_at on every row. If a row uses several pages, identify which source supports which fields. Preserve original page values alongside cleaned values whenever possible.

For frequently changing data, access time should match the precision the task needs. For downloaded files, keep the filename, version, or integrity information. When a source changes, the team can then separate real-world change from a collection-rule change.

4. Treat the quality receipt as part of the dataset

A CSV that exported successfully does not prove coverage. The delivery receipt should count requested records, successful records, duplicates, missing fields, conflicting values, inaccessible pages, and items for human review. Every count should map to specific rows or an exception list.

Duplicates do not always belong in the trash. Define the unique key first, then decide whether to merge, retain, or label them. Do not let an executor arbitrarily choose between conflicting values; preserve the sources and apply the frozen priority rule.

• Requested volume, successful volume, and actual coverage.
• Missing rate by field, not only empty-row count.
• The unique key and merge rule used for duplicates.
• Conflicting sources, selection rule, and unresolved count.
• Access failures, page variants, and records needing review.

5. Accept with computable rules

“The data looks good” is not an acceptance rule. A stronger contract specifies required fields, allowed missing rates, uniqueness rules, format rules, source coverage, and sample size. High-value fields may require 100% source traceability, while long-tail fields may allow a stated missing threshold.

Acceptance should also test structure: whether the file opens, encoding is correct, column names match, data types remain stable, and an exception receipt is attached. The dataset is deliverable only when structure, content, and evidence pass together.

6. Ten questions before collection starts

Put these ten items in the task contract to reduce rework and disputes.

• Which decision or downstream workflow will use the data?
• What are the allowed sources, excluded sources, and access boundaries?
• Are field definitions, data types, and formats complete?
• Have sample rows from real pages been accepted?
• How are missing, conflicting, duplicate, and inaccessible records represented?
• What is the unique key and deduplication rule?
• Should original and normalized values both remain?
• Which source and time fields are required per row?
• What are the coverage, missing-rate, and sampling thresholds?
• Which condition pauses execution for reconfirmation?