Tech Blog :: Exploring the node.js frontier

Oct 16 '11 9:24pm

Exploring the node.js frontier

I have spent much of the last few weeks learning and coding in Node.js, and I'd like to share some of my impressions and lessons-learned for others starting out. If you're not familiar yet, Node.js is a framework for building server-side applications with asynchronous javascript. It's only two years old, but already has a vast ecosystem of plug-in "modules" and higher-level frameworks built on top of it.

My first application is a simple web app for learning Spanish using flashcards. The code is open on Github. The app utilizes basic CRUD (Create-Retrieve-Update-Delete) functionality (of "Words" in this case), form handling, authentication, input validation, and an end-user interface - i.e. the basic components of a web app. I'm using MongoDB for the database and Express.js (which sites on top of Connect, on top of Node) as the web framework. For templating I learned Jade, and for easier CSS I'm using LessCSS.

In the process of building it, I encountered numerous challenges and questions, some solved and many still open; found some great resources; and started to train my brain to think of server-side code asynchronously.

Node is a blank slate

Node "out of the box" isn't a web server like Apache; it's more of a language, like Ruby. You start with a blank slate, on top of which you can code a daemon, an IRC server, a process manager, or a blog - there's no automatic handling of virtualhosts, requests, responses, webroots, or any of the components that a LAMP stack (for example) assumes you want. The node community is building infrastructural components that can be dropped in, and I expect that the more I delve into the ecosystem, the more familiar I'll become with those components. At its core, however, Node is simply an API for asynchronous I/O methods.

No more linear flow

I'm used to coding in PHP, which involves linear instructions, each of them "blocking." Take this linear pseudocode snippet for CRUD operations on a "word" object, for example:

if (new word) {
  render an empty form
else if (editing existing word) {
  load the word
  populate the form
  render the form
else if (deleting existing word) {
  delete the word
  redirect back to list

This is easy to do with "blocking" code. Functions return values, discrete input-output functions can be reused in multiple situations, the returned values can be evaluated, each step follows from the previous one. This is convenient but limits performance: in a high-traffic PHP-MySql application, this flow takes up a server process, and if the database is responding slowly under the load, the whole process waits; concurrent processes quickly hog all the server's memory, and a bottleneck in one part of the stack stalls the whole application. In node, the rest of the operations in the "event loop" continue to run, waiting patiently for the database (or any other I/O) callback to respond.

Coding that way is not so easy, however. If you try to load the word, for instance, you run the query with an asynchronous callback. There is no return statement on the query function. The rest of the code has be nested inside that callback, or else the code will keep running and will never get the response. So that bit would look more like this:

load the word ( function(word) {
  populate the form
  render the form

But deeply nested code isn't as intuitive as linear code, and it can make function portability very difficult. Suppose you have to run 10 database queries to populate the form - nesting them all inside each other gets very messy, and what if the logic needs to be more conditional, requiring a different nesting order in different cases?

There are ways of handling these problems, of course, but I'm just starting to learn them. In the case of the simple "load the word" scenario, Express offers the app.param construct, which parses parameters in the URL before executing the route callback. So the :word token tells the app to load a word with a given ID into the request object, then it renders the form.

No more ignoring POST and GET

In PHP, if there's a form on a page, the same piece of code processes the page whether its HTTP method is POST or GET. The $_REQUEST array even combines their parameters. Express doesn't like that, however - there is an app.all() construct that ignores the method, but the framework seems to prefer separate app.get() and routing. (There's apparently some controversy/confusion over the additional method PUT, but I steered clear of that for now.)

Back to the "word form" scenario: I load the form with GET, but submit the form with POST. That's two routes with essentially duplicate code. I could simply save an entry on POST, or render the form with GET - but what if I want to validate the form, then it needs to render the form when a POST fails validation - so it quickly becomes a mess. Express offers some solutions for this - res.redirect('back') goes back to the previous URL - but that seems like a hack that doesn't suit every situation. You can see how I handled this here, but I haven't yet figured out the best general approach to this problem.

New code needs a restart

In a PHP application, you can edit or deploy the code directly to the webroot, and as soon as it's saved, the next request uses it. With node, however, the javascript is loaded into memory when the app is run using the node command, and it runs the same code until the application is restarted. In its simplest use, this involves a Ctrl+C to stop and node app.js to restart. There are several pitfalls here:

  • Sessions (and any other in-app memory items) are lost every time you restart. So anyone using your app is suddenly logged out. For sessions, this is resolved with a database or other external session store; I can imagine other scenarios where this would be more challenging.
  • An uncaught runtime bug can crash the app, and if it's running autonomously on a server, there's nothing built-in to keep it running. One approach to this is a process manager; I'm using forever, which was built especially for node, to keep processes running and restart them easily when I deploy new code. Others have built tools within Node that abstract an individual app's process through a separate process-managing app.

When should the database connect?

Node's architectural philosophy suggests that nothing should be loaded until it's needed. A database connection might not be needed on an empty form, for instance - so it makes sense to open a database connection per request, and only when needed. I tried this approach first, using a "route middleware" function to connect on certain requests, and separated the database handling into its own module. That failed when I wanted to keep track of session IDs with MongoDB (using connect-mongo) - because a database connection is then needed on every request, and the examples all opened a connection at the top of the app, in the global scope. I switched to the latter approach, but I'm not sure which way is better.

Javascript can get very complicated

  • As logic flows through nested callbacks, variable scope is constantly changing. var and this have to be watched very carefully.
  • Writing functions that work portably across use cases without simple return statements is tricky. (One nice Node convention that covers many of these scenarios is the callback(error, result) concept, allowing calling functions to know if the result came back successfully in a standard way.)
  • Passing logic flow across node's "modules" is also tricky. Closures are helpful here, passing the app object to route modules, for instance. But in many cases, it wasn't clear how to divide the code in a way that was simultaneously logical, preserved variable scope, and worked portably with callbacks.
  • Everything - functions, arrays, classes - is an object. Class inheritance is done by instantiating another class/object and then modifying the new object's prototype. The same object can have the equivalent of "static" functions (by assigning them directly to the object) or instantiated methods (by assigning them to prototype). It's easy to get confused.
  • Javascript is a little clunky with handling empty values. The standard approach still seems to be if (typeof x == "undefined") which, at the very least, is a lot of code to express if (x). I used Underscore.js to help with this and other basic object manipulation shortcuts.
  • Because Express processes the request until there's a clear end to the response, and because everything is asynchronous, it's easy to miss a scenario in the flow where something unpredictable happens, no response is sent, and the client/user's browser simply hangs waiting for a response. I don't know if this is bad on the node side - the hanging request probably uses very little resources, since it's not actively doing anything - but it means the code has to handle a lot of possible error scenarios. Unlike in blocking code, you can't just put a catch-all else at the end of the flow to handle the unknown.

What my Flashcards app does now

The Spanish Flashcards app currently allows words (with English, Spanish, and part of speech) to be entered, shown in a list, put into groups, and randomly cycled with only one side shown, as a flashcard quiz.
The app also integrates with the WordReference API to lookup a new word and enter it - however, as of now, there's a bug in the English-Spanish API that prevents definitions from being returned. So I tested it using the English-French dictionary, and hope they'll fix the Spanish one soon.
It's built now to require login, with a single password set in a plain-text config.js file.

Next Steps for the app

I'd like to build out the flashcard game piece, so it remembers what words have been played, lets the player indicate if he got the answer right or wrong, and prioritizes previously-wrong or unseen words over ones that the player already knows.

Where I want to go with Node.js

I've been working primarily with Drupal for several years, and I want to diversify for a number of reasons: I've become very frustrated with the direction of Drupal core development, and don't want all my eggs in that basket. Web applications are increasingly requiring real-time, high-concurrency, noSQL infrastructure, which Node is well-suited for and my LAMP/Drupal skillset is not. And maybe most importantly, I find the whole architecture of Node to be fascinating and exciting, and the open-source ecosystem around it is growing organically, extremely fast, and seemingly without top-down direction.

Some resources that helped me

(All of these and many more are in my node.js tag on Delicious.)

  • Nodejitsu docs - tutorials on conventions and some best practices.
  • Victor Kane's node introand Lit app, which taught me a lot about CRUD best practices.
  • The API documentation for node.js, connect, and express.js.
  • HowToNode - seems like a generally good node resource/blog.
  • NPM, the Node Package Manager, is critical for sharing portable components, and serves as a central directory of Node modules.
  • 2009 talk by Ryan Dahl, the creator of Node.js, introducing the framework.
  • Forms and express-form, two libraries for handling form rendering and/or validation. (I tried the former and decided not to use it, but they try to simplify a very basic problem.)

Check out the code for my Spanish Flashcards app, and if you're into Node yourself and want to learn more of it together, drop me a line!

Don't open a database connection per request; the overhead of opening a connection is significant enough to create performance issues under any kind of load. HTTP pipelining and websockets are similar in principle; they achieve better efficiency by using a persistent connection.

I'm still very confused about what node.js IS exactly. Perhaps answering this question would help: What tools that I'm now using or familiar with is node a replacement for? Apache? PHP/ruby? Drupal/Rails? Something totally outside my standard LAMP experience?

"Out of the box" it does nothing, the way an empty shell script does nothing. Node.js itself is a low-level framework, more like an API really, for event-driven IO. HTTP, DNS, and other basic protocols are built in, everything else is an add-on. Using Node.js, you could build an HTTP server to replace Apache (especially good for high-concurrency/real-time/interactive apps which Apache is bad at), or write non-web scripts like you might with Ruby. Using the web frameworks built on top of node, you could create a simple website, or an MVC app (Express.js is similar to Sinatra, a lightweight alternative to Rails, and there are other frameworks that try to be more like Rails); and in theory there's no reason why you couldn't make a CMS like Drupal with it (I'll bet some people already have). As a replacement for an existing toolkit, it probably fits more into a Rails or Django developer's paradigm than into a Drupal dev's.

Some use cases node.js could be really good for:
- interactive/HTML5/ajaxy web apps (where long polling can't scale on Apache)
- tailing log files to a browser in real-time with a websocket
- IMAP server with callback hooks
- IRC / XMPP / chat server (standalone or inside a web app)
- Growl-type notifications for a web app (the Drupal-node.js module does this)
- a web spider, or any other app with multiple simultaneous operations that shouldn't block each other.

Coming from years of Drupal development, it's a whole new world. It makes me wish I had learned more of Rails or Django so I'd be more familiar with basic REST principles.