Functional Programming with Streams in node.js

Who Am I?

  HP webOS Framework Team

  @CrabDude (Twitter, Github, etc…)

  Organizer BayNode & Dallas.node

  Author:

  trycatch – async try/catch in node.js

  stepup – step node.js control flow + trycatch

  tiki – in-browser package manager

The Problem VFS

hermesClient

hermesBuild 3rd Party API

hermesDeploy

Ares (browser)

VFS VFS

hermesClient hermesClient

hermes

The Goal

  Don’t buffer any data

  Don’t write to disk

  Minimize latency

  Allow reuse of functionality (e.g., readdir)

  Functionality may be local or remote (e.g., getFolder)

Why node.js?

“Node's goal is to provide an easy way to build scalable network programs.” – nodejs.org

node.js === JavaScript

JavaScript is the future of the web, and code reuse is its unfair advantage.

Basics

  EventEmitter

  Stream

  Traditional streams (aka non-node.js lazy streams)

EventEmitter: Example

EventEmitter: Extend

Streams

“Streams are to time as arrays are to space.” – Jed Schmidt @ JSConf.eu/2010

Readable Streams

  Events: data, end, error, close

  Methods: pause, resume, end, destroy

Writable Streams

  Events: drain, error, close, pause, resume

  Methods: write, end, destroy

readable.pipe(writeable)

Readable streams can be piped to writable streams (and vice-versa!)

  on(‘data’) => write()

  on(‘end’) => end()

  on(‘drain’) => resume()

  on(‘close’) => destroy()

  on(‘error’) => on(‘error’)

  on(‘pause’) => pause()

readable.pipe(writeable): Example

Lazy Streams

Streams in functional programming…

Lazy Streams: Higher Order Functions

A higher-order function is a function that does at least one of the following:

  Take one or more functions as input   output a function

Lazy Streams: Lists

Got All That?

Hermes: Revisited FTP Dropbox Box.net Filesystem

hermesFtp hermesDropbox hermesBox hermesFilesystem

hermesBuild 3rd Party API

hermesDeploy

Ares (browser)

Still Got All That?

We need…

We need to recursively zip a directory…

  Let’s start with abstract implementation non-specific functional versions of…

  Filter paths from readdir

  Filter directories from paths

  Filter files from paths

  Recursively return all file paths in a directory

We need…

  More “abstract implementation non-specific functional versions” (errr…) of…

  ls (aka readdir)

  getFile (aka readFile)

  getFolder (stream of gets)

We need…

  Yet More AINSFV of...

  Zip a stream of files

  Allow node.js streams to be piped to lazy streams

  pipe(): lazy stream => node.js stream

  pump(): node.js stream => lazy stream

Almost there!

Now we can…

Zip a folder up using an arbitrary…

  getFile / readFile

  ls / readdir

… And stream the response! Yeeeeehaw!

Sweeeeeet.

Now What?

Since all the implementation specific functions are passed in, we can…

Port this to the browser!

Say whaaat?

node-browserify by @Substack does a lot of this already

  Stream, Buffer, Path, etc…

We need…

  Complete buffer implementation (browser has TypedArrays & ArrayBuffer)

  fs (browser has FileSystem API)

  zlib…

  inflate(), deflate(), gzip(), gunzip(), unzip()

Stay Tuned.

I’m currently porting the fs and buffer modules to the browser, and eventually zlib.

http://github.com/crabdude/fs-browserify http://github.com/crabdude/buffer-browserify http://github.com/crabdude/zlib-browserify

We’re HIRING! webOS Framework Team

If you think this is pretty cool… Or you think I’M pretty cool…

Stop by the Enyo booth or talk to me afterwards.

Thank you.

adam.crabtree@palm.com https://github.com/Gozala/streamer/blob/master/readme.js http://en.wikipedia.org/wiki/Higher-order_function https://github.com/mikeal/request http://felixge.s3.amazonaws.com/11/nodejs-streams.pdf https://github.com/substack/node-browserify

Functional Programming in JavaScript: http://igstan.ro/posts/2011-05-02-understanding-monads-with-

javascript.html http://ndc2011.macsimum.no/mp4/Day2%20Thursday/

Track6%201500-1600.mp4 http://drboolean.tumblr.com/ http://osteele.com/sources/javascript/functional/