Network Explorer's Toolkit

We're seeing a lot of interesting new technology that might just all work together. Let me fantasize about how a few of my favorites might connect.

# History

Exploratory Parsing used a web app to build and manage feature extraction from huge datasets such as all of wikipedia. First we consider what has worked well there.

The PEG grammar's backtracking and disambiguation allowed us to describe and analyze what we didn't know.

Piumarta's peg/leg implementation ran extremely fast by collapsing the generated C functions into one giant inline with no dynamic storage allocation. site

A Sinatra app managed the authoring, building and running of batch jobs that spun out rich status information as they progressed.

The web server could shell out to manage an inventory of parsing experiments with sampled results graphically indexed while also managing huge datasets and selective extracts from these supporting ongoing development.

What's not to like? The web app began to feel like handcuffs in that it only did what it had been programmed to do. And enhancing the program was progressively more tedious. This brings me to the compositional model of federated wiki.

# Future

Federated wiki brings three things to this application: hypertext for managing various projects, the journal for tracking improvements on each project, and the lineup for ad hoc rearrangement of its computational pipeline.

Viz.js brings Graphviz's excellent place and route to the clickable interactive world of client-side SVG.

PegJs brings parser generation and execution to Javascript where its byte-offset from-to error diagnostics provide welcome support for casual revision of complex grammars.

ES6 brings modules, promises and lambda notation to the already ubiquitous async language on both client and server side of the network.

Deno brings simple package management with direct web addresses and automatic caching to make the build environment. Maybe it also has sufficient foreign-function interface (FFI) to reach into the resources of the machine without talking through the shell.

WASM and WebWorkers offer access to the compute resources of a multi-core workstation.

Extended BPF offers intimate access to at least network traffic without system call overhead.

Imagine this all configured and then ultimately driving table-top virtual reality. Where to start?

# Plugins

I'm trying to imagine what plugins might look like if there were no npm and Plugmatic to manage them. Could they be as simple as an ES6 module delivered over the internet? This seems to be what Deno wants.

What if I wrote these modules as easily as I once wrote cgi scripts. Open an editor, define some entry points, drop this into a directory of static files to be served to my Deno jobs that need them?

I'm imagining a line of representation. I write in hypertext that I share above the line and javascript plugins that I share below the line. This part of my vision is new. New to me even since I started enumerating the new technologies at my finger tip 60 minutes ago.