An inital scrape can take the better part of a day. Press 'start' to begin. Shift-'start' to do one cycle at a time.
We start scrapes hourly skipping hours if they run long.
Run More Scrapes
We fetch sitemaps for one site and then discover more.
Process Next Site.
We fetch page json to index and inspect for more sites.
Process Next Page.
See Queue Stats while running.
See also Active Sites, Failed Sites, Crazy Business