Using Deferreds with jQuery
Deferreds, Futures, Promises - all terms for a concept whose (mainstream) time has come. If you read hacker news in the last week alone several popular articles tackled the proper usage and understanding of Promises in the Javascript world. The ever popular jQuery library has had a Deferred object since version 1.5 but programmers seem to be noticing it a lot more with recent releases.
Arguably jQuery’s implementation is not the best - but Javacript developers can use various implementations of the Promises/A spec like the popular Q library, or other implementations included in frameworks like Dojo. If you haven’t ever grappled with this concept, I’m sensing that now would be a good time to start!
Promises are cool! And they give us new ways to manage asynchronous code, simplify the management of callback chains, and potentially give us an API to create "lazy evaluation" features that some languages have baked in.
I have to confess that my first introduction to the idea of Promises came from the venerable Python twisted library. twisted predates Node.js but as a networking library similarly emphasizes non-blocking asynchronous code. I didn’t react very well to my first exposure to deferreds. Honestly reading twisted’s documentation about its deferred implementation confused me - it felt so complicated and esoteric. What’s wrong with passing a callback function? And an error function if you need one?
Of course that approach doesn’t scale very well when your requirements are a little more involved - the term "Callback Hell" was invented to describe the mess that frequently results from doing something in a callback that requires a callback that requires a callback that requires… Well you get the idea. Particularly in a language like Javascript where it is common to define anonymous functions at the point of usage you frequently end up with "the pyramid of doom" - code marked by a long ascending line of closing braces and parentheses.
So what are Promises? For the moment I’m going to ignore the right way to describe the the concept and stick with jQuery’s implementation, focusing on solving common problems with async/callback oriented code. Don’t worry - we’ll have plenty of assigned reading at the end!
Consider a common in-the-browser sort of problem. I want to download a couple of structured resources containing lists of entries and update my browser display every few seconds with a new entry (animated, of course) until all entries are consumed. With jQuery I might (naively) use callbacks to handle the asynchronous parts of this task and get myself into trouble. My code might look something like:
1: $.getJSON(url1, function (data1) { 2: $.getJSON(url2, function (data2) { 3: var list = data1.results.concat(data2.results); 4: list.sort(function (a, b) { return a.id - b.id; }); 5: 6: var adder = function () { 7: var tweet = list.pop(); 8: var html = $.trim(template(tweet)); 9: var $tweet = $(html); 10: $tweets.append($tweet); 11: $tweet.fadeIn(100, function () { 12: setTimeout(function () { 13: if ($tweets.children().length >= max_tweets) { 14: $tweets.children().first().remove(); 15: } 16: if (list.length > 0) { 17: adder(); 18: } 19: }, 2000); 20: }); 21: }; 22: adder(); 23: }); 24: });
This isn’t very much code but already I’ve gotten myself into trouble. For one thing - the "ajax" (actually JSONP, but let’s not quibble) calls to retrieve data from Twitter use the typical jQuery callback api: passing a function to be executed when the data has been sucessfully retrieved. Lines 1 & 2 should look pretty straightforward - but do you see the bug? I’ve taken two asynchronous operations and serialized them! The second download doesn’t start until the first sucess callback is fired!
Once I’ve got my data it seems like a simple task - every few seconds add new entries to the display and remove old entries as necessary. But the asynchronous nature of timeout’s in Javascript makes our code a little less obvious. What might be a simple synchronous call to sleep(3000) in another language here needs a callback function that re-registers itself with the setTimeout method in a sort of semi-recursive way. And of course the use of callbacks frequently leads to that deeply nested structure with the carefully matched ending parens and braces. This is more than an aesthetic concerne: None of those inner functions are very testable or reusable or composable. Naming wouldn’t help much as the functions would then be bound to one another by name. Is there a better way?
In jQuery $.Deferred offers us an object that can stand as a proxy for a value that may or may not exist yet. This in itself might be useful - async operations like AJAX calls can return a standard object which can interrogate the status of the asynchronous operation.
More interesting for our purposes is that we can use it to aggregate multiple callbacks or separate the attachment of a callback from the invocation of an action. You might think, from jQuery’s point of view at least, that a Deferred object is just an API to manage complicated callback chains.
Deferred objects are meant to handle callbacks - the API starts out simply enough by providing a .then() method that accepts callback functions. It doesn’t fire them until the .resolve() method is called and arguments passed to .resolve() are passed on to the callback functions. So
>>> d = new $.Deferred(); >>> d.then(function(x){ console.log(x);}); // add a callback >>> d.then(function(x){ console.log(x + 1);}); // add another >>> d.resolve(5); 5 6
Importantly Deferred objects in jQuery really exist to manage Promises. Promises are another object that is essentially the read-only version of a deferred. To make a function work with jQuery’s deferred API we just need to return the Promise that represents its underlying callable. The promise can be extracted from a Deferred object by calling its .promise() method.
We can use what we know so far to write a deferrable version of sleep:
var sleep = function (timeout) { var d = new $.Deferred(); //make a deferred setTimeout(d.resolve, timeout); // at some point in the future call its resolve method return d.promise(); // return the promise the deferred object is wrapped around }; var sleep3 = function () {return sleep(3000); };
jQuery includes a utility function $.when() which accepts multiple Deferred objects and creates a Deferred which resolves when all of the deferreds it is based on resolve. That’s a pretty slick idea and we can use it like:
>>> $.when(sleep(3000)).then(function(){console.log('test');}) ... three seconds later ... test
Since many of the asynchronous operations like AJAX retrieval now return a Deferred-like object in addition to accepting the traditional sucess anad failure callbacks you can schedule code to run at the completion of several asynchronous operations by passing their resulting Deferreds to $.when(). So if we give ourselves a very normal looking function which accepts a tweet, uses a template to generate some html, and inserts it into the DOM:
var adder = function ($tweets, tweet) { var html = $.trim(template(tweet)); var $tweet = $(html); $tweets.append($tweet); $tweet.fadeIn(); if ($tweets.children().length > max_tweets) { $tweets.children().first().remove(); } };
We can write less nested code just chaining many sucessive Promises together, starting the chain and letting jQuery figure out which function to call next:
1: var handler = function (e) { 2: var list = []; 3: var append = function (data) { list = list.concat(data.results); }; 4: var promise1 = $.getJSON(url1).then(append); 5: var promise2 = $.getJSON(url2).then(append); 6: $.when(promise1, promise2).then(function () { 7: var d = new $.Deferred(); 8: var d2 = d; // keep a duplicate to original deferred 9: $.each(list, function (i, tweet) { 10: // save promise returned by .then() to continue chaining 11: d2 = d2.then(sleep3).then(function () {adder($tweets, tweet); }); 12: }); 13: d.resolve(); // run the chain! 14: }); 15: };
The only complication to manage is remember that the Deferred is basically a proxy for many Promises. Repeatedly calling .then() on the same promise would build up a big list of functions to execute all at once when the original deferred is resolved. But by calling .then() on each additional promise we attach we end up creating a long chain of execution. Essentially we’re build a queue of functions to execute synchronously in this case even though some of them (like our implementation of sleep) operate synchronously under the covers.
I’m going to stop here - there’s so much more to understand about Deferreds than just scheduling callbacks. Instead of explaining myself, however, Let me send you out to do some reading that has sparked my interest lately. Two related essays point out some of the limitations of jQuery’s $.Deferred object : You’re missing the point of promises and What is the point of Promises?. I also enjoyed the explanation that delved into the functional nature of promises titled Callbacks are Imperative, Promises are Functional. And most specifically to using Promises with jQuery you might watch the video from 2012 I promise to show you when to use Deferreds.
If I’ve missed a good resource let me know in the comments - understanding this model of managing deferred execution seems to be increasingly important and there are good resources in other languages to explore the same idea. In fact if you’ve experienced confusion around the idea of a Monad, understanding Promises might also get you Monads! But that’s a conversation for another day!