We will at times defeat the incremental behavior of the Ruby Sitemap Scrape in order to retrieve new kinds of information.
When we defeat the incremental behavior we also lose the activity reporting that comes from the change detection logic. We document these discontinuities by recording our full scrape history here.
# Method
Edit scrape.rb to ignore the 'scraped' file dates.
Edit cron.sh to not record activity.
Edit crontab to not launch scrapes on its schedule.
Launch the modified cron.sh from the command line.
# History
We ran a full scrape from 22:41 July 28 until 1:11 the next day. Incremental scraping will resume at 6:00. We added indexes items.txt and plugins.txt.