Restricting Wiki Farm Growth

As part of returning to provisioning wiki sites as part of registering members of a community, we look at stopping the creation of new wiki on first access. github

The wiki-server will by default create the directory structure it requires when it is started. To prevent this from happening we can modify the existing farm code so that the server for a particular wiki is only started if data directory for the wiki exists. We can then rely on a registration process creating at least the top level directory for the new wiki.

# Thinking Through A Solution

Let's add a configuration flag `restricted` so we know that we want to use this new behaviour. We'll set the default to `false`, so that wiki defaults to the current behaviour.

When wiki starts we can easily create a list of the existing wiki data directories.

existingWiki = fs.readdirSync(wikiDir, {withFileTypes: true}) .filter (item) -> item.isDirectory() .map (item) -> item.name

This list can added to the checks in the `allow` function, to check if the any particular wiki in *allowed* in the farm. By adding a simple check.

if argv.restrict and !existingWiki.includes(host) return false

The obvious problem with this is that any wiki added while the server is running will not be in the list, and not allowed.

So, we need to either i) watch the contents of the data directory and add new wiki to the list (and remove any that are deleted), ii) add an extra test to `allow` to check for a data directory for any wiki not in the list, or iii) an internal function to add a new wiki to the list.

We'll rule out checking for new wiki data directories, option (ii).

An internal function looks to be the best option, option (iii). But, will be linked with the eventual registration process so is probably best left until that is better understood.

There are limitations to watching for changes in the farm's data directory, option (i). See fs.watch . But, there is a wrapper, chokidar, which helps address at least some of those limitation.

So, the initial implementation will watch for new/deleted wiki, using `chokidar`. For those environments where the `fs.watch` doesn't work and `fs.watchFile`, which uses polling, is required chokidar uses environment variables to allow the switch to be made.

The code for this.

watcher = chokidar.watch wikiDir, { persistent: true ignoreInitial: true depth: 0 } watcher .on('addDir', (newWiki) -> newWiki = path.basename(newWiki) existingWiki.push(newWiki) .on('unlinkDir', (delWiki) -> delWiki = path.basename(delWiki) _.pull(existingWiki, delWiki)

# Restrictions / Limitations

Some limitation that exist in the initial version.

* We only look at, and watch, the contents of `argv.data` for data directories for existing wiki. *This is not complete, as it is possible to override this parameter by `wikiDomains`. A complete solution will need to scan any `wikiDomains` and create a list of locations to scan/watch for existing wiki.*

* We are using a mainly default configuration for the fs.watch/fs.watchFile/FSEvents wrapper, chokidar. This will default to using `fs.watch` (or FSEvents on macOS). In some environments, typically if using a network file system or virtual environment, it might be necessary to switch to using `fs.watchFile` (backed by polling). To do this set the environment variable `CHOKIDAR_USEPOLLING` to true (1). You will also want to set the fire system polling interval, `CHOKIDAR_INTERVAL`, in milliseconds, default `100`.

* The use of `fs.readdirSync` option `withFileTypes` requires at least Node.js version 10.10.0, so the minimum version has been increased to that. **N.B.** Node.js 8 is end of life on December 31, 2019.

* There is no configuration for showing a custom error page for when somebody tries to access a wiki that does not exist. Current response is a `400` HTTP status and the text `Invalid host`. A better default error page, and the ability to redirect to a custom one are needed.