How We Connect Data

We've adopted a laissez-faire attitude about communication between plugins. But could the node stream protocol be our unix pipe equivalent?

Dominic Tarr recalls, In node.js’s early days the website featured an example of a tcp echo server. This is the birth of node’s most important api, the Stream. But things were just beginning to take form. post

When Tarr discovered streams and realized they would be useful, he wrote event-stream which applied streams to a use similar to Functional Reactive Programming. He convinced substack and Max Ogden how useful streams were, and then they convinced everyone else.

Streams had been created by the core node developers, but they were then adopted by people who contributed greatly to node’s ecosystem. These two groups had slightly different interpretations of what a stream was.

If node streams teach us anything, it’s that it’s very difficult to develop something as fundamental as streams inside a “core”. You can’t change core without breaking things because things assume core and never declare dependencies.


Part I of Tarr's story suggests we haven't done anything wrong by not incorporating streams into wiki's core yet. Maybe part II will hint when the abstraction will be done enough to do so.

Calvin French-Owen reviews javascript's history and predicts we’ll continue to see the cycle of rapid iteration for the next few years at least. Companies will be forced to update their codebase, or be happy with whatever horse they have. post

The npm async module solves many problems we ignored or dealt with by hand. Caolan McMahon's occasional posts offer direct and related advice.

Node Style and Structure ready for async. post

Tiny Immutables ready for async. post

Stream all the things, with backpressure. post

Doug McIlroy envisioned software development by component assembly and had largely defined what became pipes before asking unix developers to consider adding components to their work.

Typewritten summary of requirements as of 1964. post

McIlroy's retrospective on early years of unix. pdf

Nonuniform Signal Resampled anticipates signal computation in wiki without being wiki.