Pagination turns one request into a sequence of decisions: where to start, when to continue, and when enough data has been collected. Retries add another question: what should happen when a run stops partway through?
These are design questions for BatScrape’s Actors in development. We have not released a universal resume mechanism or a guarantee of free retries.
Define progress for each source
Some sources expose a cursor; others use page numbers or offsets. A progress marker is useful only if it still refers to the intended next result when the run continues. Results may change between requests.
Document the source’s behavior before deciding what to persist. Keep the requested limit separate from the number of records actually collected.
Think about duplicates
A retry can repeat work. Where a source provides a stable identifier, it can help recognize records already seen. Without one, a deduplication strategy needs explicit assumptions and sample tests.
Saving progress and saving output also need careful ordering. A checkpoint that advances beyond durable output can skip records on the next attempt.
Verify the promise
Test interruption before and after a page is written. Check the resulting dataset for gaps and duplicates. Then describe the actual behavior in the Actor’s documentation, including limitations. Run costs and retry behavior should be checked on the published listing rather than inferred from a design article.
BatScrape Team
Building BatScrape
