I have spent much of the last few weeks learning and coding in Node.js, and I'd like to share some of my impressions and lessons-learned for others starting out. If you're not familiar yet, Node.js is a framework for building server-side applications with asynchronous javascript. It's only two years old, but already has a vast ecosystem of plug-in "modules" and higher-level frameworks built on top of it.
My first application is a simple web app for learning Spanish using flashcards. The code is open on Github. The app utilizes basic CRUD (Create-Retrieve-Update-Delete) functionality (of "Words" in this case), form handling, authentication, input validation, and an end-user interface - i.e. the basic components of a web app. I'm using MongoDB for the database and Express.js (which sites on top of Connect, on top of Node) as the web framework. For templating I learned Jade, and for easier CSS I'm using LessCSS.
In the process of building it, I encountered numerous challenges and questions, some solved and many still open; found some great resources; and started to train my brain to think of server-side code asynchronously.
Node is a blank slate
Node "out of the box" isn't a web server like Apache; it's more of a language, like Ruby. You start with a blank slate, on top of which you can code a daemon, an IRC server, a process manager, or a blog - there's no automatic handling of virtualhosts, requests, responses, webroots, or any of the components that a LAMP stack (for example) assumes you want. The node community is building infrastructural components that can be dropped in, and I expect that the more I delve into the ecosystem, the more familiar I'll become with those components. At its core, however, Node is simply an API for asynchronous I/O methods.
No more linear flow
I'm used to coding in PHP, which involves linear instructions, each of them "blocking." Take this linear pseudocode snippet for CRUD operations on a "word" object, for example:
if (new word) {
render an empty form
}
else if (editing existing word) {
load the word
populate the form
render the form
}
else if (deleting existing word) {
delete the word
redirect back to list
}
This is easy to do with "blocking" code. Functions return values, discrete input-output functions can be reused in multiple situations, the returned values can be evaluated, each step follows from the previous one. This is convenient but limits performance: in a high-traffic PHP-MySql application, this flow takes up a server process, and if the database is responding slowly under the load, the whole process waits; concurrent processes quickly hog all the server's memory, and a bottleneck in one part of the stack stalls the whole application. In node, the rest of the operations in the "event loop" continue to run, waiting patiently for the database (or any other I/O) callback to respond.
Coding that way is not so easy, however. If you try to load the word, for instance, you run the query with an asynchronous callback. There is no return statement on the query function. The rest of the code has be nested inside that callback, or else the code will keep running and will never get the response. So that bit would look more like this:
load the word ( function(word) {
populate the form
render the form
});
But deeply nested code isn't as intuitive as linear code, and it can make function portability very difficult. Suppose you have to run 10 database queries to populate the form - nesting them all inside each other gets very messy, and what if the logic needs to be more conditional, requiring a different nesting order in different cases?
There are ways of handling these problems, of course, but I'm just starting to learn them. In the case of the simple "load the word" scenario, Express offers the app.param construct, which parses parameters in the URL before executing the route callback. So the :word token tells the app to load a word with a given ID into the request object, then it renders the form.
No more ignoring POST and GET
In PHP, if there's a form on a page, the same piece of code processes the page whether its HTTP method is POST or GET. The $_REQUEST array even combines their parameters. Express doesn't like that, however - there is an app.all() construct that ignores the method, but the framework seems to prefer separate app.get() and app.post() routing. (There's apparently some controversy/confusion over the additional method PUT, but I steered clear of that for now.)
Back to the "word form" scenario: I load the form with GET, but submit the form with POST. That's two routes with essentially duplicate code. I could simply save an entry on POST, or render the form with GET - but what if I want to validate the form, then it needs to render the form when a POST fails validation - so it quickly becomes a mess. Express offers some solutions for this - res.redirect('back') goes back to the previous URL - but that seems like a hack that doesn't suit every situation. You can see how I handled this here, but I haven't yet figured out the best general approach to this problem.
New code needs a restart
In a PHP application, you can edit or deploy the code directly to the webroot, and as soon as it's saved, the next request uses it. With node, however, the javascript is loaded into memory when the app is run using the node command, and it runs the same code until the application is restarted. In its simplest use, this involves a Ctrl+C to stop and node app.js to restart. There are several pitfalls here:
- Sessions (and any other in-app memory items) are lost every time you restart. So anyone using your app is suddenly logged out. For sessions, this is resolved with a database or other external session store; I can imagine other scenarios where this would be more challenging.
- An uncaught runtime bug can crash the app, and if it's running autonomously on a server, there's nothing built-in to keep it running. One approach to this is a process manager; I'm using forever, which was built especially for node, to keep processes running and restart them easily when I deploy new code. Others have built tools within Node that abstract an individual app's process through a separate process-managing app.
When should the database connect?
Node's architectural philosophy suggests that nothing should be loaded until it's needed. A database connection might not be needed on an empty form, for instance - so it makes sense to open a database connection per request, and only when needed. I tried this approach first, using a "route middleware" function to connect on certain requests, and separated the database handling into its own module. That failed when I wanted to keep track of session IDs with MongoDB (using connect-mongo) - because a database connection is then needed on every request, and the examples all opened a connection at the top of the app, in the global scope. I switched to the latter approach, but I'm not sure which way is better.
Javascript can get very complicated
- As logic flows through nested callbacks, variable scope is constantly changing.
var and this have to be watched very carefully.
- Writing functions that work portably across use cases without simple
return statements is tricky. (One nice Node convention that covers many of these scenarios is the callback(error, result) concept, allowing calling functions to know if the result came back successfully in a standard way.)
- Passing logic flow across node's "modules" is also tricky. Closures are helpful here, passing the
app object to route modules, for instance. But in many cases, it wasn't clear how to divide the code in a way that was simultaneously logical, preserved variable scope, and worked portably with callbacks.
- Everything - functions, arrays, classes - is an object. Class inheritance is done by instantiating another class/object and then modifying the new object's
prototype. The same object can have the equivalent of "static" functions (by assigning them directly to the object) or instantiated methods (by assigning them to prototype). It's easy to get confused.
- Javascript is a little clunky with handling empty values. The standard approach still seems to be
if (typeof x == "undefined") which, at the very least, is a lot of code to express if (x). I used Underscore.js to help with this and other basic object manipulation shortcuts.
- Because Express processes the request until there's a clear end to the response, and because everything is asynchronous, it's easy to miss a scenario in the flow where something unpredictable happens, no response is sent, and the client/user's browser simply hangs waiting for a response. I don't know if this is bad on the node side - the hanging request probably uses very little resources, since it's not actively doing anything - but it means the code has to handle a lot of possible error scenarios. Unlike in blocking code, you can't just put a catch-all
else at the end of the flow to handle the unknown.
What my Flashcards app does now
The Spanish Flashcards app currently allows words (with English, Spanish, and part of speech) to be entered, shown in a list, put into groups, and randomly cycled with only one side shown, as a flashcard quiz.
The app also integrates with the WordReference API to lookup a new word and enter it - however, as of now, there's a bug in the English-Spanish API that prevents definitions from being returned. So I tested it using the English-French dictionary, and hope they'll fix the Spanish one soon.
It's built now to require login, with a single password set in a plain-text config.js file.
Next Steps for the app
I'd like to build out the flashcard game piece, so it remembers what words have been played, lets the player indicate if he got the answer right or wrong, and prioritizes previously-wrong or unseen words over ones that the player already knows.
Where I want to go with Node.js
I've been working primarily with Drupal for several years, and I want to diversify for a number of reasons: I've become very frustrated with the direction of Drupal core development, and don't want all my eggs in that basket. Web applications are increasingly requiring real-time, high-concurrency, noSQL infrastructure, which Node is well-suited for and my LAMP/Drupal skillset is not. And maybe most importantly, I find the whole architecture of Node to be fascinating and exciting, and the open-source ecosystem around it is growing organically, extremely fast, and seemingly without top-down direction.
Some resources that helped me
(All of these and many more are in my node.js tag on Delicious.)
- Nodejitsu docs - tutorials on conventions and some best practices.
- Victor Kane's node introand Lit app, which taught me a lot about CRUD best practices.
- The API documentation for node.js, connect, and express.js.
- HowToNode - seems like a generally good node resource/blog.
- NPM, the Node Package Manager, is critical for sharing portable components, and serves as a central directory of Node modules.
- 2009 talk by Ryan Dahl, the creator of Node.js, introducing the framework.
- Forms and express-form, two libraries for handling form rendering and/or validation. (I tried the former and decided not to use it, but they try to simplify a very basic problem.)
Check out the code for my Spanish Flashcards app, and if you're into Node yourself and want to learn more of it together, drop me a line!
Google: TheBuckSt0p
Facebook: BenBuckman
LinkedIn
Github: newleafdigital
@thebuckst0p
Delicious: thebuckst0p
Drupal.org: thebuckst0p