Ultimate

X_SCRAPE_BODY_TOO_LARGE

Means

the HTTP leg's response body ran past maxBytes

Typical cause

terminal. A response's size is a property of the endpoint, so attempt 2 reads the same bytes and dies identically. The default is 32 MiB (DEFAULT_HTTP_MAX_BYTES) — generous for the JSON endpoint behind a paginated page, far under what OOM-kills a worker. Before this the leg did await response.text(), which materialises the whole body: AbortSignal.timeout bounds TIME, not bytes, so a 30s stream at 50 MB/s was a 1.5 GB allocation and the run persisted nothing

Fix

raise it for the one call — page.fetch(url, { maxBytes: 64 * 1024 * 1024 }) — or narrow the endpoint so it answers a page rather than a corpus. Never raise it globally: the cap is what stops one hostile origin taking the worker down

Listed under Scraping

This page as markdown