Tech Blog :: Tech Blog

Tech Blog

Oct 16 '11 9:24pm

Exploring the node.js frontier

I have spent much of the last few weeks learning and coding in Node.js, and I'd like to share some of my impressions and lessons-learned for others starting out. If you're not familiar yet, Node.js is a framework for building server-side applications with asynchronous javascript. It's only two years old, but already has a vast ecosystem of plug-in "modules" and higher-level frameworks built on top of it.

My first application is a simple web app for learning Spanish using flashcards. The code is open on Github. The app utilizes basic CRUD (Create-Retrieve-Update-Delete) functionality (of "Words" in this case), form handling, authentication, input validation, and an end-user interface - i.e. the basic components of a web app. I'm using MongoDB for the database and Express.js (which sites on top of Connect, on top of Node) as the web framework. For templating I learned Jade, and for easier CSS I'm using LessCSS.

In the process of building it, I encountered numerous challenges and questions, some solved and many still open; found some great resources; and started to train my brain to think of server-side code asynchronously.

Node is a blank slate

Node "out of the box" isn't a web server like Apache; it's more of a language, like Ruby. You start with a blank slate, on top of which you can code a daemon, an IRC server, a process manager, or a blog - there's no automatic handling of virtualhosts, requests, responses, webroots, or any of the components that a LAMP stack (for example) assumes you want. The node community is building infrastructural components that can be dropped in, and I expect that the more I delve into the ecosystem, the more familiar I'll become with those components. At its core, however, Node is simply an API for asynchronous I/O methods.

No more linear flow

I'm used to coding in PHP, which involves linear instructions, each of them "blocking." Take this linear pseudocode snippet for CRUD operations on a "word" object, for example:

if (new word) {
  render an empty form
else if (editing existing word) {
  load the word
  populate the form
  render the form
else if (deleting existing word) {
  delete the word
  redirect back to list

This is easy to do with "blocking" code. Functions return values, discrete input-output functions can be reused in multiple situations, the returned values can be evaluated, each step follows from the previous one. This is convenient but limits performance: in a high-traffic PHP-MySql application, this flow takes up a server process, and if the database is responding slowly under the load, the whole process waits; concurrent processes quickly hog all the server's memory, and a bottleneck in one part of the stack stalls the whole application. In node, the rest of the operations in the "event loop" continue to run, waiting patiently for the database (or any other I/O) callback to respond.

Coding that way is not so easy, however. If you try to load the word, for instance, you run the query with an asynchronous callback. There is no return statement on the query function. The rest of the code has be nested inside that callback, or else the code will keep running and will never get the response. So that bit would look more like this:

load the word ( function(word) {
  populate the form
  render the form

But deeply nested code isn't as intuitive as linear code, and it can make function portability very difficult. Suppose you have to run 10 database queries to populate the form - nesting them all inside each other gets very messy, and what if the logic needs to be more conditional, requiring a different nesting order in different cases?

There are ways of handling these problems, of course, but I'm just starting to learn them. In the case of the simple "load the word" scenario, Express offers the app.param construct, which parses parameters in the URL before executing the route callback. So the :word token tells the app to load a word with a given ID into the request object, then it renders the form.

No more ignoring POST and GET

In PHP, if there's a form on a page, the same piece of code processes the page whether its HTTP method is POST or GET. The $_REQUEST array even combines their parameters. Express doesn't like that, however - there is an app.all() construct that ignores the method, but the framework seems to prefer separate app.get() and routing. (There's apparently some controversy/confusion over the additional method PUT, but I steered clear of that for now.)

Back to the "word form" scenario: I load the form with GET, but submit the form with POST. That's two routes with essentially duplicate code. I could simply save an entry on POST, or render the form with GET - but what if I want to validate the form, then it needs to render the form when a POST fails validation - so it quickly becomes a mess. Express offers some solutions for this - res.redirect('back') goes back to the previous URL - but that seems like a hack that doesn't suit every situation. You can see how I handled this here, but I haven't yet figured out the best general approach to this problem.

New code needs a restart

In a PHP application, you can edit or deploy the code directly to the webroot, and as soon as it's saved, the next request uses it. With node, however, the javascript is loaded into memory when the app is run using the node command, and it runs the same code until the application is restarted. In its simplest use, this involves a Ctrl+C to stop and node app.js to restart. There are several pitfalls here:

  • Sessions (and any other in-app memory items) are lost every time you restart. So anyone using your app is suddenly logged out. For sessions, this is resolved with a database or other external session store; I can imagine other scenarios where this would be more challenging.
  • An uncaught runtime bug can crash the app, and if it's running autonomously on a server, there's nothing built-in to keep it running. One approach to this is a process manager; I'm using forever, which was built especially for node, to keep processes running and restart them easily when I deploy new code. Others have built tools within Node that abstract an individual app's process through a separate process-managing app.

When should the database connect?

Node's architectural philosophy suggests that nothing should be loaded until it's needed. A database connection might not be needed on an empty form, for instance - so it makes sense to open a database connection per request, and only when needed. I tried this approach first, using a "route middleware" function to connect on certain requests, and separated the database handling into its own module. That failed when I wanted to keep track of session IDs with MongoDB (using connect-mongo) - because a database connection is then needed on every request, and the examples all opened a connection at the top of the app, in the global scope. I switched to the latter approach, but I'm not sure which way is better.

Javascript can get very complicated

  • As logic flows through nested callbacks, variable scope is constantly changing. var and this have to be watched very carefully.
  • Writing functions that work portably across use cases without simple return statements is tricky. (One nice Node convention that covers many of these scenarios is the callback(error, result) concept, allowing calling functions to know if the result came back successfully in a standard way.)
  • Passing logic flow across node's "modules" is also tricky. Closures are helpful here, passing the app object to route modules, for instance. But in many cases, it wasn't clear how to divide the code in a way that was simultaneously logical, preserved variable scope, and worked portably with callbacks.
  • Everything - functions, arrays, classes - is an object. Class inheritance is done by instantiating another class/object and then modifying the new object's prototype. The same object can have the equivalent of "static" functions (by assigning them directly to the object) or instantiated methods (by assigning them to prototype). It's easy to get confused.
  • Javascript is a little clunky with handling empty values. The standard approach still seems to be if (typeof x == "undefined") which, at the very least, is a lot of code to express if (x). I used Underscore.js to help with this and other basic object manipulation shortcuts.
  • Because Express processes the request until there's a clear end to the response, and because everything is asynchronous, it's easy to miss a scenario in the flow where something unpredictable happens, no response is sent, and the client/user's browser simply hangs waiting for a response. I don't know if this is bad on the node side - the hanging request probably uses very little resources, since it's not actively doing anything - but it means the code has to handle a lot of possible error scenarios. Unlike in blocking code, you can't just put a catch-all else at the end of the flow to handle the unknown.

What my Flashcards app does now

The Spanish Flashcards app currently allows words (with English, Spanish, and part of speech) to be entered, shown in a list, put into groups, and randomly cycled with only one side shown, as a flashcard quiz.
The app also integrates with the WordReference API to lookup a new word and enter it - however, as of now, there's a bug in the English-Spanish API that prevents definitions from being returned. So I tested it using the English-French dictionary, and hope they'll fix the Spanish one soon.
It's built now to require login, with a single password set in a plain-text config.js file.

Next Steps for the app

I'd like to build out the flashcard game piece, so it remembers what words have been played, lets the player indicate if he got the answer right or wrong, and prioritizes previously-wrong or unseen words over ones that the player already knows.

Where I want to go with Node.js

I've been working primarily with Drupal for several years, and I want to diversify for a number of reasons: I've become very frustrated with the direction of Drupal core development, and don't want all my eggs in that basket. Web applications are increasingly requiring real-time, high-concurrency, noSQL infrastructure, which Node is well-suited for and my LAMP/Drupal skillset is not. And maybe most importantly, I find the whole architecture of Node to be fascinating and exciting, and the open-source ecosystem around it is growing organically, extremely fast, and seemingly without top-down direction.

Some resources that helped me

(All of these and many more are in my node.js tag on Delicious.)

  • Nodejitsu docs - tutorials on conventions and some best practices.
  • Victor Kane's node introand Lit app, which taught me a lot about CRUD best practices.
  • The API documentation for node.js, connect, and express.js.
  • HowToNode - seems like a generally good node resource/blog.
  • NPM, the Node Package Manager, is critical for sharing portable components, and serves as a central directory of Node modules.
  • 2009 talk by Ryan Dahl, the creator of Node.js, introducing the framework.
  • Forms and express-form, two libraries for handling form rendering and/or validation. (I tried the former and decided not to use it, but they try to simplify a very basic problem.)

Check out the code for my Spanish Flashcards app, and if you're into Node yourself and want to learn more of it together, drop me a line!

Aug 21 '11 4:02pm

Brainstorming: Building an advertising system for

One of our first revenue-generating features for Antiques Near Me (a startup antique sale-finding portal which I co-founded) is basic sponsorship ads, which we've simply been calling "featured listings." On the Boston portal, for example, the featured listing would be an antiques business in Boston, displayed prominently, clearly a sponsor, but organic (not a spammy banner ad).

I've been brainstorming how to build it, and the options span quite a range. I'll lay out some of my considerations:

  • The primary stack running the site is Drupal 6 in LAMP, cached behind Varnish to prevent server-side bottlenecks. We could build the whole system in Drupal, with Drupal-based ecommerce to sell it, and render the ads as part of the page. But if advertisers want to see stats (e.g. how many impressions/clicks has their sponsorship generated), a server-side approach has no single place to track impressions.
  • The ad placement logic doesn't have to be fancy - we want the sponsorships to be exclusive for a given time period - so we don't need all the fancy math of DFP or OpenX for figuring out what ad to place where. But the system will eventually need to handle variable pricing, variable time frames, potential "inventory" to check for availability, and other basic needs of an ad system.
  • We're running AdSense ads through Google's free DFP service, so we could set up placements and ad units for each sponsor there. But that's a manual process, and we want the ad "real estate" to scale (eventually for each city and antiques category); so in the long-run it has to be automated. That requires DFP API integration. I've signed up for access to that API, and the PHP library looks robust, but the approval process is opaque, and I'm not sure this is the right approach.
  • A hybrid Drupal-DFP approach, with flexible ad placements in DFP and client-side variables passed in from Drupal to differentiate the stats, sounds nice. But it's not clear if this is feasible; information I've gotten from a big-biz AdOps guy suggests it's not with the free edition.
  • I could build a scalable, in-house, back-end solution using Node.js and MongoDB. In theory this can handle a lot more concurrent traffic (each request being very small and quick) than Drupal/LAMP. Mongo is already in use on the site and I've wanted to learn Node for a while. This would require learning Node well enough to deploy this comfortably; with a custom bridge between Drupal (still handling the UI and transactions) and Node. This could take a while to roll out, and adds another moving piece to an already complex stack.
  • Maybe there's another 3rd party off-the-shelf service to handle this, that could be easily bridged with Drupal?

I'm curious how other sites handle similar requirements. Any ideas?

Aug 16 '11 12:03am

Launched: KickinKitchen.TV, a kids' cooking show

Yesterday we (New Leaf Digital) launched the new site for KickinKitchen.TV, a brilliant new kids' cooking show. The site (designed by another firm) is built on Drupal 7, and invites kids to interact with the show by sharing recipes, photos and videos, entering contests, and spreading the word about the show and healthy eating on social networks.

On the technical side, the site involved a robust content architecture, access rules allowing editors and users to share content creation, COPPA compliance for the young audience, new Mailchimp-Profile2 integration for real-time tracking of content submissions, and some fun jQuery animations.

I got my start programming from the BASIC page in 3-2-1 Contact magazine as a child, and I love to cook, so I think the mission of this show is vital, and I look forward to seeing the website empower those goals.

Aug 10 '11 4:06pm

Mac shell trick: have Growl notify you when something happens

Let's say you're waiting for some event to happen, and that event is detectable in your unix terminal. Examples: a DNS record to propagate (use ping and grep), a blog post to appear in an aggregator (curl and grep), etc. Instead of checking it every minute, you can have your terminal run a loop until it finds the condition you've set, and when it does, it'll notify you with Growl.

I'll use the DNS-test example here:

To check if is mapped to, we can run,
ping -c1 | grep ""; echo $?

Or to check for the appearance of some HTML on a web page, we can do,
curl -s | grep "some content"; echo $?

The $? at the end checks the exit code - 0 being success, non-0 being error.

Now put that in a loop, running every 30 seconds, with a growl popup when it succeeds:

while true; do
  FOUND=`ping -c1 | grep ""; echo $?`; 
  if [[ "$FOUND" -eq "0" ]]; then growlnotify -t "Alert" -m "FOUND" -s; break; fi; 
  sleep 30; 

Or in one line,
while true; do FOUND=`ping -c1 | grep ""; echo $?`; if [[ "$FOUND" -eq "0" ]]; then growlnotify -t "Alert" -m "FOUND" -s; break; fi; sleep 30; done

Aug 10 '11 3:23pm

Drupal’s increasing complexity is becoming a turnoff for developers

I’ve been developing custom applications with Drupal for three years, a little with 4.7 and 5, primarily with 6, and lately more with 7. Lately I’ve become concerned with the trend in Drupal’s code base toward increasing complexity, which I believe is becoming a danger to Drupal’s adoption.

In general when writing code, a solution can solve the current scenario in front of us right now, or it can try to account for future scenarios in advance. I’ve seen this referred to as N-case or N+1 development. N-case code is efficient, but not robust; N+1 code is abstract and complex, and theoretically allows for an economy of scale, allowing more to be done with less code/work. In practice, it also shifts the burden: as non-developers want the code to accommodate more use cases, the developers write more code, with more complexity and abstraction.

Suppose you want to record a date with a form and save it to a database. You’d need an HTML form, a timestamp (integer) field in your schema, and a few lines of code. Throw in a stable jQuery date popup widget and you have more code but not much more complexity. Or you could imagine every possible date permutation, all theoretically accessible to non-developers, and you end up with the 14,673 lines in Drupal’s Date module.

Drupal is primarily a content management system, not simply a framework for efficient development, so it needs to account for the myriad use cases of non-developer site builders. This calls for abstracting everything into user interfaces, which takes a lot of code. However, there needs to be a countervailing force in the development process, pushing back against increasing abstraction (in the name of end-user simplicity) for the sake of preserving underlying simplicity. In other words, there is an inherent tension in Drupal (like any big software project) between keeping the UI both robust and simple, and keeping the code robust and simple - and increasingly Drupal, rather than trying to maintain a balance, has tended to sacrifice the latter.

User interfaces are one form of abstraction; N+infinity APIs - which I’m more concerned with - are another, which particularly increase underlying complexity. Drupal has a legacy code base built with partly outdated assumptions, and developers adding new functionality have to make a choice: rewrite the old code to be more robust but less complex, or add additional abstraction layers on top? The latter takes less time but easily creates a mess. For example: Drupal 7 tries to abstract nodes, user profiles, actions, etc into “entities” and attach fields to any kind of entity. Each of these still has its legacy ID, but now there is an additional layer in between tying these “entity IDs” to their types, and then another layer for “bundles,” which apply to some entity types but not others. The result from a development cycle perspective was a Drupal 7 release that, even delayed a year, lacked components of the Entity system in core (they moved to “contrib”). The result from a systems perspective is an architecture that has too many layers to make sense if it were built from scratch. Why not, for example, have everything be a node? Content as nodes, users as nodes, profiles as nodes, etc. The node table would need to lose legacy columns like “sticky” - they would become fields - and some node types like “user” might need fixed meanings in core. Then three structures get merged into one, and the system gets simpler without compromising flexibility.

I recently tried to programatically use the Activity module - which used to be a simple way to record user activity - and had to “implement” the Entities and Trigger APIs to do it, requiring hundreds of lines of code. I gave up on that approach and instead used the elegant core module Watchdog - which, with a simple custom report pulling from the existing system, produced the same end-user effect as Activity with a tiny fraction of the code and complexity. The fact that Views doesn’t natively generate Watchdog reports and Rules doesn’t report to Watchdog as an action says a lot, I think, about the way Drupal has developed over the last few years.

On a Drupal 7 site I’m building now, I’ve worked with the Node API, Fields API, Entities API, Form API, Activity API, Rules API, Token API... I could have also worked with the Schema, Views, Exportables, Features, and Batch APIs, and on and on. The best definition I’ve heard for an API (I believe by Larry Garfield at Drupalcon Chicago) is “ the wall between 2 systems.” In a very real way, rather than feeling open and flexible, Drupal’s code base increasingly feels like it’s erecting barriers and fighting with itself. When it’s necessary to write so much code for so many APIs to accomplish simple tasks, the framework is no longer developer-friendly. The irony is, the premise of that same Drupalcon talk was the ways APIs create “power and flexibility” - but that power has come at great cost to the developer experience.

I’m aware of all these APIs under the hood because I’ve seen them develop for a few years. But how is someone new to Drupal supposed to learn all this? (They could start with the Definitive Guide to Drupal 7, which sounds like a massive tome.) Greater abstraction and complexity lead to a steeper learning curve. Debugging Drupal - which requires “wrapping your head” around its architecture - has become a Herculean task. Good developer documentation is scarce because it takes so much time to explain something so complex.

There is a cycle: the code gets bigger and harder to understand; the bugs get more use-case-specific and harder to nail down; the issue queues get bloated; the developers have less time to devote to code quality improvement and big-picture architecture decisions. But someone wants all those use cases handled, so the code gets bigger and bigger and harder to understand... as of this writing, Drupal core has 9166 open issues, the Date module has 813, Rules has 494. Queues that big need a staff of dozens to manage effectively, and even if those resources existed, the business case for devoting them can’t be easy to make. The challenge here is not simply in maintaining our work; it’s in building projects from the get-go that aren’t so complicated as to need endless maintenance.

Some other examples of excessive complexity and abstraction in Drupal 7:

  • Field Tokens. This worked in Drupal 6 with contrib modules; to date with Drupal 7, this can’t be done. The APIs driving all these separate systems have gotten so complex, that either no one knows how to do this anymore, or the architecture doesn’t allow it.
  • The Media module was supposed to be an uber-abstracted API for handling audio, video, photos, etc. As of a few weeks ago, basic YouTube and Vimeo integration didn’t work. The parts of Media that did work (sponsored largely by Acquia) didn’t conform to long-standing Drupal standards. Fortunately there were workarounds for the site I was building, but their existence is a testament to the unrealistic ambition and excessive complexity of the master project.
  • The Render API, intended to increase flexibility, has compounded the old problem in Drupal of business logic being spread out all over the place. The point in the flow where structured data gets rendered into HTML strings isn’t standardized, so knowing how to modify one type of output doesn’t help with modifying another. (Recently I tried to modify a date_select field at the code level to show the date parts in a different order - as someone else tried to do a year ago - and gave up after hours. The solution ended up being in the UI - so the end-user was given code-free power at the expense of the development experience and overall flexibility.)

Drupal 8 has an “Initiatives” structure for prioritizing effort. I’d like to see a new initiative, Simplification: Drupal 8 should have fewer lines of code, fewer APIs, and fewer database tables than Drupal 7. Every component should be re-justified and eliminated if it duplicates an existing function. And the Drupal 8 contrib space should follow the same principles. I submit that this is more important than any single new feature that can be built, and that if the codebase becomes simpler, adding new features will be easier.

A few examples of places I think are ripe for simplifying:

  • The Form API has too much redundancy. #process handlers are a bear to work with (try altering the #process flow of a date field) and do much the same as #after_build.
  • The render API now has hook_page_build, hook_page_alter, hook_form_alter, hook_preprocess, hook_process, hook_node_views, hook_entity_view, (probably several more for field-level rendering), etc. This makes understanding even a well-architected site built by anyone else an enormous challenge. Somewhere in that mix there’s bound to be unnecessary redundancy.

Usable code isn’t a luxury, it’s critical to attracting and keeping developers in the project. I saw a presentation recently on Rapid Prototyping and it reminded me how far Drupal has come from being able to do anything like that. (I don’t mean the rapid prototype I did of a job listing site - I mean application development, building something new.) The demo included a massive data migration accomplished with 4 lines of javascript in the MongoDB terminal; by comparison, I recently tried to change a dropdown field to a text field (both identical strings in the database) and Drupal told me it couldn’t do that because “the field already had data.”

My own experience is that Drupal is becoming more frustrating and less rewarding to work with. Backend expertise is also harder to learn and find (at the last meetup in Boston, a very large Drupal community, only one other person did freelance custom development). Big firms like Acquia are hiring most of the rest, which is great for Acquia, but skews the product toward enterprise clients, and increases the cost of development for everyone else. If that’s the direction Drupal is headed - a project understood and maintained only by large enterprise vendors, for large enterprise users, giving the end-user enormous power but the developer a migraine - let’s at least make sure we go that way deliberately and with our eyes open. If we want the product to stay usable for newbie developers, or even people with years of experience - and ultimately, if we want the end-user experience to work - then the trend has to be reversed toward a better balance.

Aug 4 '11 2:58pm

Upgrading to OSX Lion: Some Gotchas

I upgraded my MacBookpro to OSX Lion today. It's $30 in the App Store (or copy the DMG from someone who paid and it's free). ArsTechnica has an extremely comprehensive review of Lion if you want the full details. Here are a few minor hiccups I ran into:

• Custom symlinks in /usr/bin were removed. So far I've noticed the symlink for Git missing (/usr/bin/git -> /usr/local/git/bin/git), I put it back. (There's also a StackExchange thread about this.)

• The AFP protocol on my ReadyNAS - which I was using, among other things, for Time Machine backups - is not compatible with Lion. Fortunately the Netgear folks are quick and have a new beta release with the new protocol; I installed that it seems to be working fine.

• I turned off the new mobile-inspired "natural" scrolling. It doesn't feel natural to me, and I don't want to get disoriented every time I use someone else's computer.

• The new (also mobile-inspired) bouncy scrolling is slightly annoying too, but I can't figure out how to disable that.

• The OS seems to handle memory much better. I'm running all the usual apps, but it's turning a lot more of the "active" RAM (yellow in Activity Monitor) into "free" (green) RAM. I'm not sure what the overall performance impact is yet, but it's nice to see the OS [apparently] cleaning up dead memory better than before.

• Some of the startup items in my user account were removed, I put them back.

Otherwise it's been pretty smooth. I like the new Spaces+Exposé hybrid called Mission Control. One of the main reasons I upgraded so quickly was the new full-disk encryption built in, which I'll set up as soon as I can reboot.

Aug 2 '11 10:51am

Drupal 7 / Drush tip: Find all field content using a text format

I'm working on a Drupal 7 site and decided one of the text formats ("input formats" in D6) was redundant. So I disabled it, and was warned that "any content stored with that format will not be displayed." How do I know what content is using that format? This little shell snippet told me:

drush sql-query "show tables like 'field_data_%'" | tail -n+2 | while read TABLE; do
  FIELD=`drush sql-query "show fields in $TABLE like '%format%';" | tail -n+2 | awk '{ print $1 }'`; 
  echo "$TABLE - $FIELD";
    if [[ "$FIELD" != "" ]]; then
     drush sql-query "select * from ${TABLE} where ${FIELD}='old_format''";

You'll need to run that in the terminal from your site's webroot and have Drush installed. Rename old_format to the code name of your text format. (drush sql-query "select * from {filter_format}" will show you that.) It'll work as a single command if you copy and paste it (as multiple lines or with line breaks stripped - the semi-colons indicate the end of each statement).

Breaking it down:

  1. Find all the tables used for content storage.
  2. Find all the 'format' fields in those tables. (They'll only exist if the field uses formats.)
  3. Find all the rows in those tables matching the format you want to delete. Alternatively, if you want everything to be in one format, you can see what does not use that format by changing the ${FIELD}=... condition to ${FIELD}<>'new_format'.

This won't fix anything for you, it'll just show you where to go - look at the entity_id columns (that's the nid if the content is nodes) and go edit that content.

Also note, this is checking the field_data_ tables, which (as far as I can tell) track the latest revision. If you are using content revisions you might want to change the first query to show tables like 'field_revision_%'. I'm not sure why D7 duplicates so much data, but that's for another post.

Update: I modified the title from Find all content to Find all field content because of the comment by David Rothstein below.

May 18 '11 11:48pm

Workaround to variables cache bug in Drupal 6

I run my Drupal crons with Drush and Jenkins, and have been running into a race condition frequently where it tells me, Attempting to re-run cron while it is already running, and fails to run.

That error is triggered when the cron_semaphore variable is found. It's set when cron starts, and is deleted when cron ends, so if it's still there, cron is presumably still running. Except it wasn't really - the logs show the previous crons ended successfully.

I dug into it a little further: drush vget cron_semaphore brought up the timestamp value of the last cron, like it was still set. But querying the `variables` table directly for cron_semaphore brought up nothing! That tipped me off to the problem - it was caching the variables array for too long.

Searching the issue brought up a bunch of posts acknowledging the issue, and suggesting that people clear their whole cache to fix it. I care about performance on the site in question, so clearing the whole cache every 15 minutes to run cron is not an option.

The underlying solution to the problem is very complex, and the subject of several ongoing threads:

Following Drupal core dev policy now (which is foolish IMHO), if this bug is resolved, it has to be resolved first in 8.x (which won't be released for another 2-3 years), then 7.x, then 6.x. So waiting for that to work for my D6 site in production isn't feasible.

As a stopgap, I have Jenkins clear only the 'variables' cache entry before running cron:
drush php-eval "cache_clear_all('variables', 'cache');"

That seems to fix the immediate problem of cron not running. It's not ideal, but at least it doesn't clear the entire site cache every 15 minutes.

May 11 '11 11:51am

A GitHub dev on the importance of side projects

GitHub developer Zach Holman wrote a great post a month ago, Why GitHub Hacks on Side Projects (discovered via Signal vs Noise). It's about having a culture that encourages quirky side projects, "automated inefficiencies," to give the mind breathing time between big challenges, to promote camaraderie, and to make people smile. I recommend anyone who does creative or technical work read it. Snippet:

You should build out a side project culture. A Campfire bot is natural for us, since we spend so much time in Campfire, but there’s plenty of other areas. Hack on your continuous integration server. An app that picks where you’re having lunch that day. A miniapp that collects and stores employee-created animated gifs. A continuous integration animated lunch machine. It doesn’t matter what it is; if it improves the lives of your coworkers or makes them laugh, it helps build a stronger company culture. And that’s cool.

May 8 '11 2:45pm

Three Quirks of Drupal Database Syntax

Database query syntax in Drupal can be finicky, but doing it right - following the coding standards as a matter of habit - is very important. Here are three "gotchas" I've run into or avoided recently:

1. Curly braces around tables: Unit testing with SimpleTest absolutely requires that table names in all your queries be wrapped in {curly braces}. SimpleTest runs in a sandbox with its own, clean database tables, so you can create nodes and users without messing up actual content. It does this by using the existing table prefix concept. If you write a query in a module like this,
$result = db_query("SELECT nid from node");
when that runs in test, it will load from the regular node table, not the sandboxed one (assuming you have no prefix on your regular database). Having tests write to actual database tables can make your tests break, or real content get lost. Instead, all queries (not just in tests) should be written like:

$result = db_query("SELECT nid from {node} node");
(The 2nd node being an optional alias to use later in the query, for example as node.nid JOINed to another table with a nid column.) When Drupal runs the query, it will prefix {node} by context as site_node, or simpletestXXnode, to keep the sandboxes separate. Make sure to always curly-brace your table names!

2. New string token syntax: Quotation marks around string tokens are different in Drupal 6 and 7. D7 uses the new "DBTNG" abstraction layer (backported to D6 as the DBTNG module). In Drupal 6, you'd write a query with a string token like this:
$result = db_query("SELECT nid from {node} where title='%s'", 'My Favorite Node');
Note the single quotation marks around the placeholder %s.

With D7 or DBTNG, however, the same static query would be written:
$result = db_query("SELECT nid from {node} WHERE title = :title", array(':title' => 'My Favorite Node'));
No more quotes around the :title token - DBTNG puts it in for you when it replaces the placeholder with the string value.

3. Uppercase SQL commands: Make sure to use UPPERCASE SQL commands (SELECT, FROM, ORDER BY, etc) in queries. Not doing so is valid syntax 99% of the time, but will occasionally trip you up. For example: the db_query_range function (in D6) does not like lowercase from. I was using it recently to paginate the results of a big query, like select * from {table}. The pagination was all messed up, and I didn't know why. Then I changed it to SELECT * FROM {table} and it worked. Using uppercase like that is a good habit, and in the few cases where it matters, I'll be glad I'm doing it from now on.

Apr 22 '11 1:41pm

Monitoring Drupal sites with Munin

One of the applications I've been working with recently is the Munin monitoring tool. Its homepage describes it simply:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

Munin graphGetting Munin set up on an Ubuntu server is very easy. (One caveat: a lot of new plugins require the latest version of Munin, which is only available in Ubuntu 10.) Munin works on a "master" and "node" structure, the basic idea being:

  1. On a cron, the master asks all its nodes for all their stats (usually via port 4949, so configure your firewall accordingly).
  2. Each node server asks all its plugins for their stats.
  3. Each plugin dumps out brief key:value pairs.
  4. The master collects all the data and compiles graphes as images on static HTML pages.

Its simplicity is admirable: Each plugin is its own script, written in any executable language. There are common environment variables and output syntax, but otherwise writing or modifying a plugin is very easy. The plugin directory is called Munin Exchange. (The latest version of each plugin isn't necessarily on there, though: in some cases searching for the plugin name brought up newer versions on Github.)

I set up Munin for two reasons: 1) get notifications of problems, 2) see historical graphs to spot trends and bottlenecks. I have Munin running on a dedicated monitoring server (also running Jenkins), since notifications coming from the web server wouldn't be much use if the web server went down. It's currently monitoring three nodes (including itself), giving me stats on memory (total and for specific processes), CPU, network traffic, apache, mysql, S3 buckets, memcached, varnish, and mongodb. Within a few days of it running, a memory leak on one server became apparent, and the "MySql slow query" spikes that coincide with cron (doing a bunch of stats/aggregation) are illuminating.

None of this is Drupal specific, but graphing patterns in Drupal simply requires a plugin, and McGo has fortunately given us a Munin module that provides just that. (The package includes two modules: Munin API to define stats and queries, and Munin Defaults with some basic node and user queries.) I asked for maintainer access and modified it a little - the 6.x-2.x branch now uses Drush for database queries rather than storing the credentials in the scripts, for example. The module generates the script code which you copy to files in your plugins directory.

Conclusions so far: getting Munin to show you graphs on all the major stats of a server takes a few hours (coming at it as a total beginner). Setting up useful notifications is more complicated, though, and will probably have to evolve over time through trial and error. For simple notifications on servers going down, for example, it's easier to set up a simple cron script (on another server) with curl and mail, or use the free version of CloudKick. Munin's notifications are more suited to spotting spikes and edge cases.

Apr 22 '11 10:33am

Yesterday's Cloud Collapse

Amazon's EC2 cloud hosting system went down for several hours yesterday. I first noticed the disruption because my dotCloud instances (which I've been playing with for Drupal feasibility) stopped responding. Then a server I'm running on a totally different hosting service,, went down at the same time. (It turns out that was just a coincidence; does not run on EC2 according to their support staff.) Anyway, the simultaneous outage made me think it was more than a coincidence, so I googled "cloud outage" and found a breaking CNN story.

Mashable explains the problem exposed by the outage: EC2 is supposed to be redundant across multiple "Availability Zones," but a cascading failure still managed to bring down the whole system. That article links to a more detailed explanation of what happened.

I expect there's going to be a knee-jerk reaction now among some [mostly old-school] sysadmins away from the cloud, back to co-locating physical servers in a data center. But I worked at a company that hosted dozens of sites that way, and when the data center had a fire and lost power, their sites all went down for days. The cloud is just an abstraction inside a physical machine. It's an abstraction that allows for tremendous efficiency, cost-savings, and redundancy. But physical failures (of power or connectivity) can still bring any infrastructure down.

One notable EC2-based service that was not disrupted (according to Mashable at least) was Netflix, because they built sufficient redundancy to handle an entire data center's failure. That's the obvious lesson for customers of any hosting service: if 24/7/365 uptime of your service is absolutely critical, then build in massive redundancy. That applies if you're hosting on physical servers or in the cloud. Redundancy is complicated, and expensive, and like an insurance policy, only seems worthwhile in a crisis. So it's probably not worth the cost for most applications.

I'm also somewhat fatalistic about infrastructure in general: it's all very fragile. And the more complex and interdependent our systems become, the more points of failure we introduce. Redundancy itself is kind of two steps forward, one step back, simply because it adds complexity.

Apr 19 '11 11:10am

Setting up Drupal on DotCloud's server automation platform

Managing a properly configured server stack is one of the pain points in developing small client sites. Shared hosting is usually sub-par, setting up a VPS from scratch is overkill, and automation/simplification of the server configuration and deployment is always welcome. So I've been very interested in seeing how DotCloud might work for Drupal sites.

DotCloud is in the same space as Heroku, an automated server/deployment platform for Rails applications. DotCloud is trying to cater to much more than Rails, however: they currently support PHP (for websites and "worker" daemons), Rails, Ruby, Python, MySql, Postgresql, and have Node.js, MongoDB and a whole bunch of other components on their roadmap.

The basic idea is to automate the creation of pre-configured EC2 instances using a shell-based API. So you create a web and database setup and push your code with four commands:

dotcloud create mysite
dotcloud deploy -t php mysite.www
dotcloud deploy -t mysql mysite.db
dotcloud push mysite.www ~/code

Each "deployment" is its own EC2 server instance, and you can SSH into each (but without root). The "push" command for deployment can use a Git repository, and files are deployed to the server Capistrano-style, with symlinked releases and rollbacks. (This feature alone, of pushing your code and having it automatically deploy, is invaluable.)

Getting it to work with Drupal is a little tricky. First of all, the PHP instances run on nginx, not Apache. So the usual .htaccess file in core doesn't apply. Drupal can be deployed on nginx with some contortions, and there is a drupal-for-nginx project on Github. However, I write this post after putting in several hours trying to adapt that code to work, and failing. (I've never used nginx before, which is probably the main problem.) I'll update it if I figure it out, but in the meantime, this is only a partial success.

The basic process is this:

  • Set up an account (currently needs a beta invitation which you can request)
  • Install the dotcloud client using python's easy_install app
  • Set up a web (nginx) instance with dotcloud deploy
  • Set up a database (mysql or postgres) instance
  • Set up a local Git repo, download Drupal, and configure settings.php (as shown with dotcloud info
  • Push the repository using dotcloud push
  • Navigate to your web instance's URL and install Drupal.
  • To use your own domain, set up a CNAME record and run dotcloud alias. ("Naked" domains, i.e. without a prefix like www, don't work, however, so you have to rely on DNS-level redirecting.)
  • For added utility, SSH in with dotcloud ssh and install Drush. (Easiest way I found was to put a symlink to the executable in ~/bin.)

The main outstanding issue is that friendly URL's don't work, because of the nginx configuration. I hope to figure this out soon.

Some other issues and considerations:

  • The platform is still in beta, so I experienced a number of API timeouts yesterday. I mentioned this on Twitter and they said they're working on it; today I had fewer timeouts.
  • The server instances don't give you root access. They come fully configured but you're locked into your home directory, like shared hosting. I understand the point here - if you changed the server stack, their API and scaling methodologies wouldn't work - but it means if something in the core server config is wrong, tough luck.
  • The shell (bash in Ubuntu 10.04 in my examples) is missing a Git client, vim, and nano, and some of its configuration (for vi for instance) is wonky out of the box.
  • The intended deployment direction is one-way, from a local dev environment to the servers, so if you change files on the server, you need to rsync them down. (You can SSH in with the dotcloud app and put on a normal SSH key for rsync.)
  • Because the webroot is a symlink, any uploaded files have to be outside the webroot (as a symlink as well). This is normal on Capistrano setups, but unusual for most Drupal sites (running on shared or VPS hosting).
  • It's free now, but only because it's in beta and they haven't announced pricing. It is yet to be seen if this will be cost-effective when it goes out of beta.
  • They promise automated scaling, but it's not clear how that actually works. (Nowhere in the process is there a choice of RAM capacity, for instance.) Does scaling always involve horizontally adding small instances, and if so, does that make sense for high-performance applications?

Conclusion so far

The promise of automated server creation and code deployment is very powerful, and this kind of platform could be perfect for static sites, daemons, or some custom apps. If/when I get it working in Drupal, it could be as simple as a shell script to create a whole live Drupal site from nothing in a few seconds.

Try it out! I'm very curious what others' experience or thoughts are on this approach. I'd especially love for someone to post a solution to the nginx-rewrite issue in the comments.

Update 4/22:

Their support staff recommended reducing the nginx.conf file to one line:

try_files $uri $uri/ /index.php?q=$uri;

And that worked. However, it leaves out all the other recommended rules for caching time, excluding private directories, etc. I asked about these and am still waiting for a reply.

Also, to get file uploads to work properly, you'll need to put your files directory outside of the webroot, and symlink sites/default/files (or equivalent) to that directory, using a postinstall script. Mine looks like this (after creating a ~/drupal-files directory):

chmod g+w /home/dotcloud/current/sites/default && \
ln -s /home/dotcloud/drupal-files /home/dotcloud/current/sites/default/files && \
echo "Symlink for files created."

That runs whenever you run dotcloud push, and is similar to sites deployed with Capistrano, where the same kind of symlink approach is generally used.

Apr 14 '11 9:29am

A showcase of red flags: How do web shops get away with this?

I recently had occasion to review the new website of a major bank's CRA/charity wing. As a web developer, I'm always curious how other sites are built. This one raised a number of red flags for me, so I'd like to write about it as a showcase. I have three questions on my mind:

  1. How do professional web shops get away with such poor quality work?
  2. How do clients know what to look for (and avoid)?
  3. With plenty of good web shops out there, why aren't big clients connecting with them?

I don't have the answers yet, but I do want to raise the questions. First, reviewing the site from my developer's perspective:

  • The page contents are loaded with Javascript. With Javascript turned off, there's a little bit of left nav, and the main area is essentially blank. This means the site is unreadable to screen readers (browsers for blind people), so the site is not 508 compliant. Maybe more importantly, it means the contents of the page are invisible to search engines. (See Google's cached copy of the homepage for example.)
  • The Javascript that pulls in the page contents is loading an XML file with AJAX (see line 72 of the homepage source). XML is meant for computers to talk to each other, not for human-readable websites, and AJAX is meant for interactive applications, not the main content area of every page. (I can only imagine the workflow for editing the content on the client side: does their CMS output XML? Do they manually edit XML? Or can the content never change without the original developers?)
  • The meta tags are all generic: The OpenGraph page title (used by Facebook) across the site is "ShareThis Homepage". (ShareThis has a "social" widget which I assume they copied the code from, but having those meta values is probably worse than having none at all.)
  • None of the titles are links, so even if Google could read the site, it would just see a lot of Read More's.
  • From a usability perspective, the 11px font size for most of the content is difficult to read.
  • The Initiatives by State map is built in Flash, which makes it unviewable on non-Android mobile devices. Flash is also unnecessary for maps now, given the slew of great HTML5-based mapping tools. Not to mention the odd usability quirks/bugs of the map's interface.

I could go on, but that's enough to make the point. So what's going on here? I've seen enough similar signs in other projects to feel confident in speculating about this one.

The vendor wasn't entirely incompetent - the hundreds of lines of Javascript code needed some technical proficiency to write - yet the site ignores so many core principles of good web development circa 2011. Whatever skills were applied here, were misplaced. The "web" has to accommodate our phones, TVs, even our cars, with "mobile" browsers (broadly defined) expected to eclipse the desktop in the not-too-distant future. That means progressive enhancement and basic HTML quality are critical. Web users also have an infinity of sites to visit, so to justify the investment in yet another site, you need some basic Search Engine Optimization for people to find you. Building a site that is readable only to a narrow subset of desktop browsers constitutes an unfinished product in my book.

On the client side, any site with more than one page, that needs to be updated more than once in a blue moon, needs a content management system. I don't see the tell-tales of any common CMS here, and the way the contents are populated with AJAX suggests the CMS under the hood is weak or non-existent. Reinventing the wheel with entirely custom code for a site makes it difficult to maintain in the long run: developers with expertise in common frameworks/CMSs won't want to touch it, and whoever does will need a long ramp-up/head-scratching period to understand it. It's also unnecessary with so many tested tools available. So clients need to insist on a CMS, and if a vendor tries to talk them out of one, or claim it will be 3x the price, they need to find a better vendor. I work with Drupal and think it's the best fit for many sites (and free of license fees), but there are many good options.

The site doesn't say who built it, and searching for relevant keywords doesn't bring up any clearly proud vendors. Was it a web shop at all, or an ad agency that added some token web services to their roster? (General rule: avoid those vendors.) Clients need to see their sites not as another piece of throwaway marketing material, but as a long-term, audience-building investment. Thinking of websites as advertisements that only need to be viewed on Windows running Internet Explorer is missing the point.

I wonder, given the client (with $10 billion in profit in 2010), how much this site cost. It's not a brochure site, but it's not particularly complex either. The only really custom piece is the map, and the same style could probably be implemented with OpenLayers (or Google Maps with some compromise from the client on color requirements). Whatever they paid, I suspect they could have paid one of the top Drupal shops the same price to build a maintainable, standards-based, truly impressive website, for visitors, internal staff, and reviewing developers alike.

Then again, being such a large client means the vendor likely had to deal with all kinds of red tape. Maybe the really good web shops don't connect with that class of client because it's not worth the hassle. But surely the U.S. House of Representatives, in the process of moving to Drupal, has its own brand of red tape, and the vendor has project managers who can handle it.

Websites are complex beasts and evaluating them from the client perspective is not the same as watching a proposed TV commercial. So how do client without core competencies in web development know what to avoid? Googling it will only get them so far. But the burden is ultimately on them: we all consume products about which we lack core expertise, and big corporations (as consumers and clients themselves) need to figure out the same heuristics as everyone else. Trusting reputable vendors is one approach, but it's a vicious cycle if they're locked into one vendor (as companies with existing B2B relationships often are).

Diversifying the advice you get is critical. Big projects should have RFPs and a bidding process. (That helps enforce a realistic division of labor: little shops like mine don't respond to RFPs, but big shops that can afford that investment are happy to manage the project and outsource some development so suit their own core competencies.)

The bidding process could even involve the vendors defending their proposals in front of their competitors. Then the top-tier CMS shop can eliminate the static-HTML or Cold Fusion shop from the running before it's too late. There are no silver bullets - there's a potential to be fleeced in any market - but in most of them, consumers have figured out ways to spot red flags and protect themselves. Website clients need to educate themselves and catch up.

Apr 13 '11 11:26pm

Web Development as a Spiritual Experience

I'm setting aside the code-heavy posting for a little while to share some abstract thoughts I've had recently. I've been enjoying a brief (intentional) lull in my work load, and the resulting cognitive surplus has made me remember how much I enjoy my work as a web developer. There are many good (and obvious) reasons for this:

  • Web development involves building things. The product may be virtual, but there's a pleasure in the construction (the kind Matthew Crawford talks about in Shop Class as Soulcraft), a fulfillment from creating something that didn't exist before (that often provides people a clear value).

  • Web development, insofar as it involves programming, involves a molding of things to one's will. (Coders at Work is a great read on this subject.) Code can be a pain in the ass sometimes, but it doesn't "disobey". Just as dealing with obstinate people can be stressful, dealing with machines can be comforting.

  • There's a tremendous amount of mastery involved in web development. Good web developers are familiar with the whole technology stack, from performance-tuning a tiny SQL query to scaling multiple servers. There's always something new to learn, and the learning contributes to a positive feedback loop. Web development is not a job for incurious people.

These reasons I've known for a while. But recently I've been pondering another reason, maybe the most important one:

The web is infinite. On the web anything can talk to anything else. Any knowable information can be discovered and connected to any other information. Physical boundaries are irrelevant. The promise of "open" (source, standards, data) is the most concrete manifestation of progress I can think of, maybe the most real example humanity has ever created. (Maybe this is why the web freaks out established powers.) If we expand our imagination a little more, the web could even have the potential to restructure the whole fabric of society.

So as a web developer, we deal with infinity every day. There's always something bigger and cooler to build, some new technology to master, some new innovation around the corner. Getting from "Wouldn't it be cool if..." to shipping a brand new creation is just a matter of putting in the time. (That makes time the only real enemy of the developer - but that post is for another time.)

I think there's a kind of spiritual experience in working with that kind of infinity. It's not the metaphorical infinity of religion; it's an infinity we can directly perceive through our work. I don't think developers of pre-web desktop software felt the same kind of potential. (That divide between proprietary lock-in and openness is the reason former titans of the tech industry have become irrelevant.) But I suspect the original pioneers of the web understood the power they were unleashing.

Long live the internet.

Apr 12 '11 1:00pm

How to render image fields with Drupal 7

Suppose you have a Drupal 7 site, with a node containing an image field called field_image, and you want to pull the URL of the image into your page template. (As opposed to a node template, where it's already rendered.) For bonus points, you want the image to be rendered through a "style" (aka Imagecache preset).

In Drupal 6, this involved theme('imagecache') and $node->field_image[0]['filepath']. In Drupal 7 it's a little different, because the Files API has been abstracted from the local file system (to handle other types of storage) and Imagecache is now (mostly) in the core Image module.

First, I'm separating this into the preprocessor logic where we check if the image field exists and get the right code, and the template output where we use the finished value. The page preprocessor would probably be in your theme's template.php and look like function THEME_preprocess_page(&$vars), and your page template would look like page.tpl.php.

Fortunately, the node object is already in the page preprocessor's variables, as $vars['node']. Let's break this down a little with the available API functions:

// filename relative to files directory
// e.g. 'masthead.jpg'
$filename = $vars['node']->field_image['und'][0]['filename'];
// relative path to raw image in 'scheme' format
// e.g. 'public://masthead.jpg'
$image_uri = file_build_uri($filename);
// relative path to 'styled' image (using an arbitrary 'banner' style) in 'scheme' format
// e.g. 'public://styles/banner/public/masthead.jpg'
image_style_path('banner', $filename);
// raw URL with an image style
// e.g. ''
// [passing a raw path here will return a very ungraceful fatal error, see]
$vars['masthead_raw'] = image_style_url('banner', $image_uri);
// html for a styled image
// e.g. '<img typeof="foaf:Image" src="" alt="" />'
$vars['masthead'] = theme('image_style', array('style_name' => 'banner', 'path' => $image_uri));

So to do something useful with this:

function THEME_preprocess_page(&$vars) {
  // pull the masthead image into the page template
  if (isset($vars['node']->field_image) && !empty($vars['node']->field_image['und'][0]['filename'])) {
    $filename = $vars['node']->field_image['und'][0]['filename'];
    $image_uri = file_build_uri($filename);
    // url
    $vars['masthead_raw'] = image_style_url('banner', $image_uri);
    // html
    $vars['masthead'] = theme('image_style', array('style_name' => 'banner', 'path' => $image_uri));

Now in your page template, you have $masthead (HTML) and $masthead_raw (URL) [with the $vars variables now being independently named] so you can do something like this in a PHP block:

<?php if ($masthead): ?>
  <div id="masthead"><?php echo $masthead; ?></div>
<?php endif; ?>

A quick-and-dirty alternative would be, directly in page.tpl.php:

if (!empty($node->field_image['und'][0]['filename']) {
 echo theme('image_style', array('style_name' => 'banner', 'path' => file_build_uri($node->field_image['und'][0]['filename'])));

(note $vars['node'] is now $node)

The shorthand version will work, but separating the logic from the output (with a preprocessor and template) is the "best practices" approach.

Apr 6 '11 9:48pm

Good Git GUIs for Mac

Today on Twitter, I saw two Git apps for OSX worth spreading:

Via @jayroh: Brotherbard has an "experimental fork" of GitX which has a nice GUI for branch trees, local and remote branches, staging, cherry-picking, and rebasing. (Some time I have to try the last two with the app in particular.) It's free.

Via @mortendk, if you're willing to spend $59 (and I generally don't mind paying for great Mac software -developers have to eat!) - check out Tower, which claims to be and likely is "the most powerful Git client for Mac."

Of course there's always the terminal, which I'll still use for most operations - git log --graph shows a rudimentary graph, for instance. Visualizing complex branch trees in a GUI is really nice, and GitK (the Java-based app which comes with the Mac Git package) doesn't have a lot going for it.

(If you're new to Git, check out the excellent (and free) Pro Git book, or (via @danigrrl), a Git tutorial for designers.

Apr 5 '11 10:25am

Lullabot post on configuring Varnish for Drupal

Lullabot has a beautifully comprehensive post on configuring Varnish for Drupal. This is a realm that has lacked good Drupal documentation in the past, so I haven't utilized Varnish as much as I should, but that will change now.

Added: Another great Varnish resource is schoefmax's Caching With Varnish presentation.

Apr 1 '11 7:34pm

Set up Hudson/Jenkins to notify via GTalk

I'm in the process of setting up a Jenkins (aka Hudson) Continuous Integration server. I wanted some of the jobs to notify me in real time, and what better way than IM (specifically GTalk), which I have running most of the time I'm online?

It took me a little while to get this working, and some of the documentation was lacking, so I thought I'd write quick notes on what worked for me.

This assumes you have Jenkins already set up. You'll need to install the Jabber Plugin (and its dependency Instant Messaging Plugin).

With that running, from the dashboard, go to Manage Jenkins -> Configure System. Scroll down to Jabber Notification. Enable it. Enter your credentials according to the official docs:

If you have a GMail or Google Mail account, specify the full e-mail address, such as If you are a Google App user, specify full e-mail address like
(If you have a european gmail account, you may need
Expand "Advanced settings" and put server address otherwise it won't work.

Set the port in the Advanced Settings to 5223.
Make sure your firewall allows TCP access to 5223! (On an Ubuntu server with UFW, do this with sudo ufw allow 5223.)

In the job configuration, enable Jabber Notification and put in the account your local IM client is logged in with. (In my case I set up a Google Apps user for Jenkins, which talks to my personal GTalk account, but that might not be necessary.)

Finally, add the GTalk account Jenkins is using to your buddy list (by inviting in GMail, adding in iChat, etc).

Now run the job and read the console output. If everything's set up right, you should get an IM notification.

Mar 30 '11 2:41pm

Quick tip: Extract all unique IP addresses from Apache logs

Say you wanted to count the number of unique IP addresses hitting your Apache server. It's very easy to do in a Linux (or compatible) shell.

First locate the log file for your site. The generic log is generally at /var/log/httpd/access_log or /var/log/apache2/access_log (depending on your distro). For virtualhost-specific logs, check the conf files or (if you have one active site and others in the background) run ls -alt /var/log/httpd to see which file is most recently updated.

Then spit out one line to see the format:
tail -n1 /var/log/httpd/access_log

Find the IP address in that line and count which part it is. In this example it's the 1st part (hence $1):

cat /var/log/httpd/access_log | awk '{print $1}' | sort | uniq > /var/log/httpd/unique-ips.log

You'll now have a list of sorted, unique IP addresses. To figure out the time range, run
head -n1 /var/log/httpd/access_log
to see the start point (and the tail) syntax above for the end point.)

To count the number of IPs:
cat  /var/log/httpd/unique-ips.log | wc -l

To paginate:
more  /var/log/httpd/unique-ips.log

Have fun.