BEYOND THE CALLBACK:Yield Control with Javascript Generators

JuLy 16, 2015

Darren Cruse darren.cruse@gmail.com

STORY TIME…(put on your jammies)

Once upon a time…

When you were young

• And learning to program at your momma’s knee…

She said

• Break your big long sections of code into little functions that do just one thing each

Let the main routine call the little ones

• And they will each run

• Until they’re done

• And return to the main function

This is called “Control Flow”

• Can you say “control flow”?

(I knew you could)

And then one day…

• (maybe in Colorado)

Momma said

• How about we try something new?

Instead of that function you’ve been used to…

• The one that starts at the top

• And executes till it’s done

• And then just returns

Let’s have us a function

• That’s starts at the beginning like normal

• But returns from the middle

• Not when it’s done done, but maybe when it’s just part way done

• But if we call it again

• Let’s have it start up in the middle again

• And then we can have it do some stuff

• And then maybe return from the middle again

• So when we call it again

• It can start up in the middle again

• And then maybe we could do this three or four times

• And maybe throw some loops in there

• And then eventually when the function is really “done done”

• it can return

• And be finished

• Like we’re used to with our other functions

So what do you think About this new kind of function?

Sound good?

But wait a minute

Was your momma maybe kind of an academic type?

Did she also use words like…

“lazy sequences” !

“coroutines” !

“delimited continuations” !

“callcc”

If so…

Then yo momma wasn’t on “the pipe”.

Yo momma was talking about

Generator Functions!!!

(as added by the ecmascript 6 standard)

And yo momma deserves credit

(as does the es6 committee)

For trying to teach your sorry a** about alternative means of

control flow…

And thereby save you from the fate of heavyweight

multithreading that doesn’t scale, and shared state concurrency with

rampant race conditions

Which is the true…

Enough silliness

In computer science, a generator is a special routine that can be used to control the iteration behaviour of a loop. In fact, all generators are iterators.[1] A generator is very similar to a

function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values.

However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short,

a generator looks like a function but behaves like an iterator.

From wikipedia:

https://en.wikipedia.org/wiki/Generator_(computer_programming)

2.1 Lisp 2.2 CLU 2.3 Icon 2.4 C++ 2.5 Perl 2.6 Tcl

2.7 Haskell 2.8 Racket 2.9 PHP 2.10 Ruby 2.11 Java 2.12 C# 2.13 XL 2.14 F#

2.15 Python

Also from wikipedia:I was surprised to see the number of languages with generators (not all are “native” though)

The typical generator example is very often…

An infinite Fibonacci Sequence

function* fibonacci() { let [prev, curr] = [0, 1]; while (true) { yield curr; [prev, curr] = [curr, prev + curr]; } } !var gen = fibonacci(); console.log(gen.next().value); // 1 console.log(gen.next().value); // 1 console.log(gen.next().value); // 2 console.log(gen.next().value); // 3 console.log(gen.next().value); // 5 console.log(gen.next().value); // 8

again from: https://en.wikipedia.org/wiki/Generator_(computer_programming)

“yield” indicates to yield the value “curr” to the caller. This is

also where the function will resume next time.

Picking it apart…

function* fibonacci() { let [prev, curr] = [0, 1]; while (true) { yield curr; [prev, curr] = [curr, prev + curr]; } }

again from: https://en.wikipedia.org/wiki/Generator_(computer_programming)

The “*” in “function*” indicates this is a generator function

Each call to “gen.next()” runs the generator till

the next yield (or till the function returns), and

gets back an object containing the yielded

(or returned) value.

Picking it apart…

var gen = fibonacci(); console.log(gen.next().value); // 1 console.log(gen.next().value); // 1 console.log(gen.next().value); // 2 console.log(gen.next().value); // 3 console.log(gen.next().value); // 5 console.log(gen.next().value); // 8

again from: https://en.wikipedia.org/wiki/Generator_(computer_programming)

Calling the function does not run the function body. Rather it’s like calling a constructor. It returns a generator object that you can then use to run (and resume) the function body.

{ value: val, done: true }

What gen.next() returns is:

A little object like:

“value” is the value yielded/returned

“done” is true if the generator “returned” (or exited without a “return”).

as opposed to a yield.

once a generator is “done” it should not be “next’ed” (if so it simply returns undefined)

But add the --harmony option on the node command!!

node > v0.11.2

Recent versions of Chrome Recent versions of Firefox

We can run this code as it is in

Also in the browser in:

For other browsers a transpiler is required…

io.js supports it (by default - no option needed)

The most commonly mentioned transpiler for generators is from Facebook and is called

“Regenerator”

https://www.npmjs.com/package/regenerator

npm install -g regenerator regenerator --include-runtime fibonacci-basic.js > fibonacci-basic-es5.js

Regenerator works by converting your code to a state machine…

function fibonacci() { var prev, curr; ! return regeneratorRuntime.wrap( function fibonacci$(context$1$0) { while (1) switch (context$1$0.prev = context$1$0.next) { case 0: prev = 0; curr = 1; case 2: if (!true) { context$1$0.next = 9; break; } context$1$0.next = 5; return curr; case 5: prev = curr; curr = prev + curr; context$1$0.next = 2; break; case 9: case "end": return context$1$0.stop(); } }, marked0$0[0], this); }

babel has support as well (that uses Regenerator to do the magic)

!

Traceur supports them as well.

Even though there’s fibonacci examples everywhere…

• What I found interesting was that almost immediately after generators came on the scene there was a spate of “task runners”.

• These are utilities that invert the relationship of consumer and producer like you see in the fibonacci example

• In order to simplify asynchronous code where it looks like it’s “blocking”

Lots-o-Task-Runners

• co (TJ/Visionmedia)

• Q.async (Q)

• Promise.coroutine (bluebird)

• galaxy

• monocle

• suspend

• genny

• gen-run

• coro

• (and on…)

“Paving the cow paths?”

(how about a stampede!!)

Probably so many because:

• 1. People wanted them

• 2. They’re pretty easy to implement once you have generators.

Remember when you first learned to program how simple it was…

var myNumber = parseInt(Math.random() * 100); !while (true) { var guess = read.question('What is your guess? '); if (guess > myNumber) { console.log('Guess lower.'); } else if (guess < myNumber) { console.log('Guess higher.'); } else { console.log('YOU WIN!'); break; } }

There are cases when blocking code really is easier to follow

var myNumber = parseInt(Math.random() * 100); !rl.setPrompt('What is your guess? '); rl.prompt(); !rl.on('line', function(guess) { if (guess > myNumber) { console.log('Guess lower.'); } else if (guess < myNumber) { console.log('Guess higher.'); } else { console.log('YOU WIN!'); rl.close(); } rl.prompt(); }).on('close', function() { console.log('Thanks for playing.'); process.exit(0); });

Looking at “co” as an example of a “task runner”…

var myNumber = parseInt(Math.random() * 100); !co(function* playgame() { while(true) { var guess = yield prompt('What is your guess? '); if (guess > myNumber) { console.log('Guess lower.'); } else if (guess < myNumber) { console.log('Guess higher.'); } else { console.log('YOU WIN!'); return "Goodbye"; } } })

To clarify - when we used fibonacci we were the consumer of the generator

function* fibonacci() { let [prev, curr] = [0, 1]; while (true) { yield curr; [prev, curr] = [curr, prev + curr]; } } !var gen = fibonacci(); console.log(gen.next().value); // 1 console.log(gen.next().value); // 1 console.log(gen.next().value); // 2 console.log(gen.next().value); // 3 console.log(gen.next().value); // 5 console.log(gen.next().value); // 8

“library” code

“our” code

(=“producer”)

(=“consumer”)

Here things are reversed

var myNumber = parseInt(Math.random() * 100); !co(function* playgame() { while(true) { var guess = yield prompt('What is your guess?'); ! if (guess > myNumber) { console.log('Guess lower.'); } else if (guess < myNumber) { console.log('Guess higher.'); } else { console.log('YOU WIN!'); return "Goodbye"; } } })

We’ve written the “producing” (yielding) code, and hidden inside the “co” routine is the gen.next() “consuming” code.

In this case “prompt” would typically return a promise which is yielded back to “co”, then “co”

will “gen.next()” us once the promise

resolves to a value. !

This is the “secret sauce” for appearing to block yet remaining single threaded as always.

AN EASY METAPHOR…

We “yield” the control flow and let other stuff (like the event loop)

run until the asynchronous result is ready

But there’s another “metaphor” coming in es7…

Async/Await

Basically the story (more or less) is…There was a strawman proposal for async/await

at the same time there was a proposal for generators !

The generators proposal got approved in time for es6 !

The async/await proposal did not but is part of es7 !

The good news is async/await is pretty much the same idea as “task runners” but with slightly simpler syntax

!The syntax is known for being similar to what C# has had - it doesn’t have e.g. function* is has async/await

!(though some javascript async/await implementations are

using generators internally btw) !

Also the async/await standard is designed to complement the new promise standards i.e. to work

well with apis returning promises.

Same control flow - nicer syntax:var prompt = require('./node_modules/prompt-promise'); !var myNumber = parseInt(Math.random() * 100); !// note instead of function* below we have "async" async function playgame() { try { var gameover = false; while(!gameover) { // and instead of yield we "await" var guess = await prompt('What is your guess? '); ! if (guess > myNumber) { console.log('Guess lower.'); } else if (guess < myNumber) { console.log('Guess higher.'); } else { console.log('YOU WIN!'); gameover = true; } } console.log('Thanks for playing.'); } catch(err) { console.log(err); } }

Yay Working Exceptions!!

• Yes those were standard try/catch blocks in that last example!!

• Yay (standard) exceptions work even with asynchronous code!!

• (no special .catch etc. as with promises)

• They work both with the generator “task runners” and with async/await.

• I didn’t mention but in addition to gen.next() there’s also a “gen.throw()” which throws an exception at the point the generator has paused on a “yield”. This is the secret sauce.

SUPER BRIEFLY: SOME INTERESTING USES

OF GENERATORS

Briefly: KOA (http://koajs.com)

• Another TJ/VisionMedia Project (as is “co”)

• Intended as the next generation of “Express” (which TJ/VisionMedia also developed)

• It’s been able to simplify the writing of middleware greatly since you simply “yield” to the downstream middleware:

app.use(function *(next){ var start = new Date; yield next; var ms = new Date - start; console.log('%s %s - %s', this.method, this.url, ms); });

get the time before

yield to downstream

get the time after

Briefly: js-csp

• A “communicating sequential processes” library in javascript - it uses generators in it’s implementation.

• Modeled off of core.async and google go.

• A key concept is that concurrent processes communicate through channels

• And the default behavior of channels is to block (they can buffer if you want but the default is to block).

• Think of the simplicity like you see in our “guess this number” example, but waiting on concurrent processes as opposed to user input.

• See also:

• https://github.com/ubolonton/js-csp

• http://jlongster.com/Taming-the-Asynchronous-Beast-with-CSP-in-JavaScript

• http://swannodette.github.io/2013/08/24/es6-generators-and-csp/

Briefly: js-csp

var {chan,timeout} = csp = require(‘js-csp'); !var ch = chan(); !go { var val; while((val = <- ch) !== csp.CLOSED) { console.log(val); } } !go { ch <- 1; (<- timeout(1000)); ch <- 2; ch.close(); }

This example uses js-csp with sweet.js macros I based on some started by James Long. Note <- is a blocking channel operation (i.e. it’s doing a yield behind it)

a channel to talk over

run this reader guy

concurrently with

this writer guy

wait for the writer

wait for the reader

js-csp in the browser (w/out sweet macros this time) (thanks David Nolen & James Long)

• How about being able to block on a channel of events? Really changes how you think about UI programming!!

function listen(el, type) { var ch = chan(); el.addEventListener(type, function(e) { csp.putAsync(ch, e); }); return ch; } !go(function*() { var el = document.querySelector('#ui1'); var ch = listen(el, 'mousemove'); while(true) { var e = yield take(ch); el.innerHTML = ((e.layerX||e.clientX)+', ' + (e.layerY || e.clientY)); } });

create channel of “type” events on “el”

concurrent with other stuff happening

on the page

wait for mouse moves and display

them

http://jlongster.com/Taming-the-Asynchronous-Beast-with-CSP-in-JavaScript

THANKS FOR COMING