{tech}.Ben Buckman.net

Varnish is a reverse-proxy cache that allows a site with a heavy backend (such as a Drupal site) and mostly consistent content to handle very high traffic load. The “cache” part refers to Varnish storing the entire output of a page in its memory, and the “reverse proxy” part means it functions as its own server, sitting in front of Apache and passing requests back to Apache only when necessary.

One of the challenges with implementing Varnish, however, is the complex “VCL” protocol it uses to process requests with custom logic. The syntax is unusual, the documentation relies heavily on complex examples, and there don’t seem to be any books or other comprehensive resources on the software. A recent link on the project site to Varnish Training is just a pitch for a paid course. Searching more specifically for Drupal + Varnish will bring up many good results - including Lullabot’s fantastic tutorial from April, and older examples for Mercury - but the latest stable release is now 3.x and many of the examples (written for 2.x) don’t work as written anymore. So it takes a lot of trial and error to get it all working.

I’ve been running Varnish on AntiquesNearMe.com, partly to keep our hosting costs down by getting more power out of less [virtual] hardware. A side benefit is the site’s ability to respond very nicely if the backend Apache server ever goes down. They’re on separate VPS's (connected via internal private networking), and if the Apache server completely explodes from memory overload, or I simply need to upgrade a server-related package, Varnish will display a themed “We’re down for a little while” message.

But it wasn’t until recently that I got Varnish’s primary function, caching, really tuned. I spent several days under the hood recently, and while I don’t want to rehash what’s already been well covered in Lullabot’s tutorial, here are some other things I learned:

Check syntax before restarting

After you update your VCL, you need to restart Varnish - using sudo /etc/init.d/varnish restart for instance - for the changes to take effect. If you have a syntax error, however, this will take down your site. So check the syntax first (change the path to your VCL as needed):
varnishd -C -f /etc/varnish/default.vcl > /dev/null

If there are errors, it will display them; if not, it shows nothing. Use that as a visual check before restarting. (Unfortunately the exit code of that command is always 0, so you can’t do check-then-restart as simply as check-varnish-syntax && /etc/init.d/varnish restart, but you could grep the output for the words “exit 1” to accomplish the same.)

Logging

The std.log function allows you to generate arbitrary messages about Varnish’s processing. Add import std; at the top of your VCL file, and then std.log("DEV: some useful message") anywhere you want. The “DEV” prefix is an arbitrary way of differentiating your logs from all the others. So you can then run in the shell, varnishlog | grep "DEV" and watch only the information you’ve chosen to see.

How I use this:
- At the top of vcl_recv() I put std.log("DEV: Request to URL: " + req.url);, to put all the other logs in context.
- When I pipe back to apache, I put std.log("DEV: piping " + req.url + " straight back to apache"); before the return (pipe);
- On blocked URLs (cron, install), the same
- On static files (images, JS, CSS), I put std.log("DEV: Always caching " + req.url);
- To understand all the regex madness going on with cookies, I log req.http.Cookie at every step to see what’s changed.

Plug some of these in, check the syntax, restart Varnish, run varnishlog|grep PREFIX as above, and watch as you hit a bunch of URLs in your browser. Varnish’s internal logic will quickly start making more sense.

Watch Varnish work with your browser

The Chrome/Safari Inspector and Firebug show the headers for every request made on a page. With Varnish running, look at the Response Headers for one of them: you’ll see “Via: Varnish” if the page was processed through Varnish, or “Server:Apache” if it went through Apache. (Using Chrome, for instance, login to your Drupal site and the page should load via Apache (assuming you see page elements not available to anonymous users), then open an Incognito window and it should run through Varnish.)

Add hit/miss headers

• When a page is supposed to be cached (not pipe'd immediately), Varnish checks if there is an existing hit or miss. To watch this in your Inspector, use this logic:

sub vcl_deliver {
  std.log("DEV: Hits on " + req.url + ": " + obj.hits);
 
  if (obj.hits > 0) {
    set resp.http.X-Varnish-Cache = "HIT";
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }
 
  return (deliver);
}

Then you can clear the caches, hit a page (using the browser technique above), see “via Varnish” and a MISS, hit it again, see a HIT (or not), and know if everything is working.

Clear Varnish when aggregated CSS+JS are rebuilt

If you have CSS/JS aggregation enabled (as recommended), your HTML source will reference long hash-string files. Varnish caches that HTML with the hash string. If you clear only those caches (“requisites” via Admin Menu or cc css+js via Drush), Varnish will still have the old references, but the files will have been deleted. Not good. You could simply never use that operation again, but that’s a little silly.

The heavy-handed solution I came up with (I welcome alternatives) is to wipe the Varnish cache when CSS+JS resets. That operation is not hook-able, however, so you have to patch core. In common.js, _drupal_flush_css_js(), add:

if (module_exists('varnish') && function_exists('varnish_purge_all_pages')) {
  varnish_purge_all_pages();
}

This still keeps Memcache and other in-Drupal caches intact, avoiding an unnecessary “clear all caches” operation, but makes sure Varnish doesn’t point to dead files. (You could take it a step further and purge only URLs that are Drupal-generated and not static; if you figure out the regex for that, please share.)

Per-page cookie logic

On AntiquesNearMe.com we have a cookie that remembers the last location you searched, which makes for a nicer UX. That cookie gets added to Varnish’s page “hash” and (correctly) bypasses the cache on pages that take that cookie into account. The cookie is not relevant to the rest of the site, however, so it should be ignored in those cases. How to handle this?

There are two ways to handle cookies in Varnish: strip cookies you know you don’t want, as in this old Pressflow example, or leave only the cookies you know you do want, as in Lullabot’s example. Each strategy has its pros and cons and works on its own, but it’s not advisable to combine them. I’m using Lullabot’s technique on this site, so to deal with the location cookie, I use if-else logic: if the cookie is available but not needed (determined by regex like req.url !~ "PATTERN" || ...), then strip it; otherwise keep it. If the cookie logic you need is more varied but still linear, you could create a series of elsif statements to handle all the use cases. (Just make sure to roast a huge pot of coffee first.)

Useful add-ons to varnish.module

• Added watchdog('varnish', ...) commands in varnish.module on cache-clearing operations, so I could look at the logs and spot problems.
• Added a block to varnish.module with a “Purge this page” button for each URL, shown only for admins. (I saw this in an Akamai module and it made a lot of sense to copy. I’d be happy to post a patch if others want this.)
• The Expire offers plug-n-play intelligence to selectively clear Varnish URLs only when necessary (clearing a landing page of blog posts only if a blog post is modified, for example.) Much better than the default behavior of aggressive clearing “just in case”.

I hope this helps people adopt Varnish. I am also available via my consulting business New Leaf Digital for paid implementation, strategic advice, or support for Varnish-aided sites.

The layout of Antiques Near Me (a startup I co-founded) has long been built using the sturdy 960.gs grid system (implemented in Drupal 6 using the Clean base theme). Grids are very helpful: They allow layouts to be created quickly; they allow elements to be fit into layouts easily; they keep dimensions consistent; they look clean. But they have a major drawback that always bothered me: the grid-X classes that determine an element's width are in the HTML. That mixes up markup/content and layout/style, which should ideally be completely separated between the HTML and CSS.

The rigidity of an in-markup grid becomes especially apparent when trying to implement "responsive" design principles. I'm not a designer, but the basic idea of responsive design for the web, as I understand it, is that a site's layout should adapt automagically to the device it's viewed in. For a nice mobile experience, for example, rather than create a separate mobile site - which I always thought was a poor use of resources, duplicating the content-generating backend - the same HTML can be used with @media queries in the CSS to make the layout look "native".

(I've put together some useful links on Responsive Design and @media queries using Delicious. The best implementation of a responsive layout that I've seen is on the site of FourKitchens.)

Besides the 960 grid, I was using LessCSS to generate my styles: it supports variables, mix-ins, nested styles, etc; it generally makes stylesheet coding much more intuitive. So for a while the thought simmered, why not move the static 960 grid into Less (using mixins), and apply the equivalent of grid-X classes directly in the CSS? Then I read this article in Smashing on The Semantic Grid System, which prescribed pretty much the same thing - using Less with a library called Semantic.gs - and I realized it was time to actually make it happen.

To make the transition, I forked semantic.gs and made some modifications: I added .alpha and .omega mixins (to cancel out side margins); for nested styles, I ditched semantic.gs's .row() approach (which seems to be buggy anyway) and created a .nested-column mixin instead. I added clear:both to the .clearfix mixin (seemed to make sense, though maybe there was a reason it wasn't already in).

To maintain the 960.gs dimensions and classes (as an intermediary step), I made a transitional-960gs.less stylesheet with these rules: @columns: 16; @column-width: 40; @gutter-width: 20;. Then I made equivalents of the .grid_X classes (as Clean's implementation had them) with an s_ prefix:

.s_container, .s_container_16 {
  margin-left: auto;
  margin-right: auto;
  width: @total-width;
  .clearfix();
}
.s_grid_1 {
  .column(1);
}
.s_grid_2 {
  .column(2);
}
...
.s_grid_16 {
  .column(16);
}

The s_grid_X classes were purely transitional: they allowed me to do a search-and-replace from grid_ to s_grid_ and remove the 960.gs stylesheet, before migrating all the styles into semantic equivalents. Once that was done, the s_grid_ classes could be removed.

960.gs and semantic.gs also implement their columns a little differently, one with padding and the other with margins, so what was actually a 1000px-wide layout with 960.gs became a 960px layout with semantic.gs. To compensate for this, I made a wrapper mixin applied to all the top-level wrappers:

.wide-wrapper {
  .s_container;
  padding-right: 20px;
  padding-left: 20px;
  .clearfix();
}

With the groundwork laid, I went through all the grid_/s_grid_ classes in use and replaced them with purely in-CSS semantic mixins. So if a block had a grid class before, now it only had a semantic ID or class, with the grid mixins applied to that selector.

Once the primary layout was replicated, I could make it "respond" to @media queries, using a responsive.less sheet. For example:

/* iPad in portrait, or any screen below 1000px */
@media only screen and (max-device-width: 1024px) and (orientation: portrait), screen and (max-width: 999px) {
  ...
}
 
/* very narrow browser, or iPhone -- note that <1000px styles above will apply here too! 
note: iPhone in portrait is 320px wide, in landscape is 480px wide */
@media only screen and (max-device-width: 480px), only screen and (-webkit-min-device-pixel-ratio: 2), screen and (max-width: 499px) {
  ...
}
 
/* iPhone - portrait */
@media only screen and (max-device-width: 480px) and (max-width: 320px) {
  ...
}

Some vitals tools for the process:

• Less.app (for Mac), or even better, the new CodeKit by the same author compiles and minifies the Less files instantly, so the HTML can refer to normal CSS files.
• The iOS Simulator (part of XCode) and Android Emulator (with the Android SDK), to simulate how your responsive styles work on different devices. (Getting these set up is a project in itself).
• To understand what various screen dimensions looked like, I added a simple viewport debugger to show the screen size in the corner of the page (written as a Drupal6/jQuery document-ready "behavior"; fills a #viewport-size element put separately in the template):
  

  Drupal.behaviors.viewportSize = function() {
    if (!$('#viewport-size').size()) return;
   
    Drupal.fillViewportSize = function() {
      $('#viewport-size').text( $(window).width() + 'x' + $(window).height() )
        .css('top', $('#admin-menu').height());
    };
    Drupal.fillViewportSize();
    $(window).bind('resize', function(event){
      Drupal.fillViewportSize();
    });  
  };

After three days of work, the layout is now entirely semantic, and the 960.gs stylesheet is gone. On a wide-screen monitor it looks exactly the same as before, but it now adapts to narrower screen sizes (you can see this by shrinking the window's width), and has special styles for iPad and iPhone (portrait and landscape), and was confirmed to work on a popular Android tablet. It'll be a continuing work in progress, but the experience is now much better on small devices, and the groundwork is laid for future tweaks or redesigns.

There are some downsides to this approach worth considering:

• Mobile devices still load the full CSS and HTML needed for the "desktop" layout, even if not all the elements are shown. This is a problem for performance.
• The stylesheets are enormous with all the mixins, compounding the previous issue. I haven't examined in depth how much of a problem this actually is, but I'll need to at some point.
• The contents of the page can only change as much as the stylesheets allow. The order of elements can't change (unless their visible order can be manipulated with CSS floats).

To mitigate these and modify the actual content on mobile devices - to reduce the performance overhead, load smaller images, or put less HTML on the page - would probably require backend modifications that detect the user agent (perhaps using Browscap). I've been avoiding that approach until now, but with most of the work done on the CSS side, a hybrid backend solution is probably the next logical step. (For the images, Responsive Images could also help on the client side.)

See the new layout at work, and my links on responsive design. I'm curious to hear what other people do to solve these issues.

Added: It appears the javascript analog to media queries is media query lists, which are event-able. And here's an approach with media queries and CSS transition events.

I'm working on a Drupal 7 site and decided one of the text formats ("input formats" in D6) was redundant. So I disabled it, and was warned that "any content stored with that format will not be displayed." How do I know what content is using that format? This little shell snippet told me:

drush sql-query "show tables like 'field_data_%'" | tail -n+2 | while read TABLE; do
  FIELD=`drush sql-query "show fields in $TABLE like '%format%';" | tail -n+2 | awk '{ print $1 }'`; 
  echo "$TABLE - $FIELD";
    if [[ "$FIELD" != "" ]]; then
     drush sql-query "select * from ${TABLE} where ${FIELD}='old_format''";
    fi
done

You'll need to run that in the terminal from your site's webroot and have Drush installed. Rename old_format to the code name of your text format. (drush sql-query "select * from {filter_format}" will show you that.) It'll work as a single command if you copy and paste it (as multiple lines or with line breaks stripped - the semi-colons indicate the end of each statement).

Breaking it down:

1. Find all the tables used for content storage.
2. Find all the 'format' fields in those tables. (They'll only exist if the field uses formats.)
3. Find all the rows in those tables matching the format you want to delete. Alternatively, if you want everything to be in one format, you can see what does not use that format by changing the ${FIELD}=... condition to ${FIELD}<>'new_format'.

This won't fix anything for you, it'll just show you where to go - look at the entity_id columns (that's the nid if the content is nodes) and go edit that content.

Also note, this is checking the field_data_ tables, which (as far as I can tell) track the latest revision. If you are using content revisions you might want to change the first query to show tables like 'field_revision_%'. I'm not sure why D7 duplicates so much data, but that's for another post.

Update: I modified the title from Find all content to Find all field content because of the comment by David Rothstein below.

Database query syntax in Drupal can be finicky, but doing it right - following the coding standards as a matter of habit - is very important. Here are three "gotchas" I've run into or avoided recently:

1. Curly braces around tables: Unit testing with SimpleTest absolutely requires that table names in all your queries be wrapped in {curly braces}. SimpleTest runs in a sandbox with its own, clean database tables, so you can create nodes and users without messing up actual content. It does this by using the existing table prefix concept. If you write a query in a module like this,
$result = db_query("SELECT nid from node");
when that runs in test, it will load from the regular node table, not the sandboxed one (assuming you have no prefix on your regular database). Having tests write to actual database tables can make your tests break, or real content get lost. Instead, all queries (not just in tests) should be written like:

$result = db_query("SELECT nid from {node} node");
(The 2nd node being an optional alias to use later in the query, for example as node.nid JOINed to another table with a nid column.) When Drupal runs the query, it will prefix {node} by context as site_node, or simpletestXXnode, to keep the sandboxes separate. Make sure to always curly-brace your table names!

2. New string token syntax: Quotation marks around string tokens are different in Drupal 6 and 7. D7 uses the new "DBTNG" abstraction layer (backported to D6 as the DBTNG module). In Drupal 6, you'd write a query with a string token like this:
$result = db_query("SELECT nid from {node} where title='%s'", 'My Favorite Node');
Note the single quotation marks around the placeholder %s.

With D7 or DBTNG, however, the same static query would be written:
$result = db_query("SELECT nid from {node} WHERE title = :title", array(':title' => 'My Favorite Node'));
No more quotes around the :title token - DBTNG puts it in for you when it replaces the placeholder with the string value.

3. Uppercase SQL commands: Make sure to use UPPERCASE SQL commands (SELECT, FROM, ORDER BY, etc) in queries. Not doing so is valid syntax 99% of the time, but will occasionally trip you up. For example: the db_query_range function (in D6) does not like lowercase from. I was using it recently to paginate the results of a big query, like select * from {table}. The pagination was all messed up, and I didn't know why. Then I changed it to SELECT * FROM {table} and it worked. Using uppercase like that is a good habit, and in the few cases where it matters, I'll be glad I'm doing it from now on.