Region Scraping and Indexing

Wiki offers local and global versions of search. Global becomes more difficult as our federation grows. We explore possible proliferation of indexes over large regions that would benefit from incremental specialization possible with Outpost Thinking.

We wrote a stand-alone scraper in TypeScript and found the experience encouraging. github

We've moved this code around a bit but now it is located in its own repo, an extension of seran-wiki. github

We embedded the user interface and about pages into the single file that implements the scraper.

We explored implementation of start/stop/step controls realized on the server-side as awaiting on promises return by a single function, step.

We deployed this application as a docker container hosted in a multi-purpose digital ocean droplet. This includes build and deploy on push automation.

We claim the first Seran outpost page forked back into the federation. See Federation Scraper