Tech Blog :: Parse Drupal watchdog logs in syslog (using node.js script)


Nov 29 '11 1:06pm

Parse Drupal watchdog logs in syslog (using node.js script)

Drupal has the option of outputting its watchdog logs to syslog, the file-based core Unix logging mechanism. The log in most cases lives at /var/log/messages, and Drupal's logs get mixed in with all the others, so you need to cat /var/log/messages | grep drupal to filter.

But then you still have a big text file that's hard to parse. This is probably a "solved problem" many times over, but recently I had to parse the file specifically for 404'd URLs, and decided to do it (partly out of convenience but mostly to learn how) using Node.js (as a scripting language). Javascript is much easier than Bash at simple text parsing.

I put the code in a Gist, node.js script to parse Drupal logs in linux syslog (and find distinct 404'd URLs). The last few lines of URL filtering can be changed to any other specific use case you might have for reading the logs out of syslog. (This could also be used for reading non-Drupal syslogs, but the mapping applies keys like "URL" which wouldn't apply then.)

Note the comment at the top: to run it you'll need node.js and 2 NPM modules as dependencies. Then take your filtered log (using the greg method above) and pass it as a parameter, and read the output on screen or output with > to another file.

This is neat as an experiment and a reason to learn node.js, but

> Javascript is much easier than Bash at simple text parsing.

Web developer spotted. <3

I don't have a syslog'd Drupal instance handy to pull logs from, but here's my clumsy Bash one-liner to find 404'd urls in standard Apache logs:

grep -ir " 404 " /var/log/httpd/* | awk '{print $7}' | sort -nr | uniq -c | sort -nr | head -n 5

grep for the HTTP error code (surrounded by spaces), awk to print the seventh field in the log entry (usually the URL in standard apache logs), then sort, uniq, re-sort, and head to give us the top 5 most 404'd URLs from our logs.

Just one of probably several thousand different ways to skin this cat.