A dataset becomes harder to use when a familiar field changes meaning without explanation. While building BatScrape’s first Actors, we’re treating output shape as part of the product documentation.
The following is our intended approach. It does not describe a released shared schema package or an existing compatibility guarantee.
Describe each field
For each Actor, we plan to document field names, value types, and missing-value behavior. A collection timestamp should be distinguishable from a source’s publication timestamp. A source identifier should not quietly become a row number.
Example records make those decisions easier to review. They should include ordinary results as well as records with optional fields missing.
Review changes from the consumer’s side
Adding a field and renaming a field have different consequences for someone using a dataset. Before changing an output, check whether an existing export or script would still work.
Our aim is to explain meaningful changes in the Actor’s release notes. The specific policy will be documented with each Actor as it becomes available.
Test shape and meaning separately
A schema check can catch an unexpected value type. It cannot always tell whether the source changed what a number means. Sample comparisons and source-specific checks remain useful even when a record has the right shape.
Predictable output begins with those explicit decisions, not with a promise that a changing website will never change.
BatScrape Team
Building BatScrape
