Use delegation, threading, and queues to speed up operations

I posted this to my personal site, but I wanted to mention it on this blog, as it's a performance optimization that I use quite often when programming for the web or for native applications: Don't Wait, Delegate! Proper use of threading and queueing.

There are hundreds of ways you can improve your app or website's performance, but few have the potential to improve your app or website's responsiveness as much as queueing or using background processes.

Don't Wait, Delegate! Proper use of threading and queueing

There are hundreds of ways you can improve your app or website's performance, but few have the potential to improve your app or website's responsiveness as much as queueing or using background processes. There are so many complex operations that can be improved by looking at them in a new way. For example:

Not-so-Instant Oil Change

I like changing the oil in my car, but it often takes a bit of time (definitely not an 'instant!'), and involves the following:

  • Drive to auto parts store, pick up oil and oil wrench, and drive back (30 minutes).
  • Jack up front of car on stands (5 minutes).
  • Drain oil into oil pan, remove old filter, prep new filter (5 minutes).
  • Install new filter, refill oil reservoir (2 minutes).
  • Remove jack stands, clean up mess (5 minutes).
  • Drive to oil disposal center with old oil in pan, and drive back (20 minutes).

Total elapsed time: Over 1 hour! Maybe it's worth spending a few extra bucks on an 'instant' oil change to get back 45 minutes of my life.

Getting XHProf working well on Mac OS X Lion

I was inspired today to get XHProf working on my Mac, using MAMP PRO 2.0.5/PHP 5.3.6, after reading @Dave Reid's tweet. Since I'm not leaving for DrupalCon until tomorrow, what else could I do today? There's an excellent article on Lullabot that will help you get 85% of the way towards having XHProf up and running on your Mac, working with your Drupal sites, but there are a few missing pieces and little tips that will help you get XHProf fully-armed and operational.

XHProf Callgraph example
Ooh, pretty visualizations!

Running a Windows XP VM in Parallels (Mac) from a USB Flash Drive

I thought I'd post my experience here, for the benefit of others, because I couldn't find a whole lot of information about this specific use of an external USB flash drive.

I have a MacBook Air with a dainty 128GB SSD drive, so I try to keep large files that I rarely use on external drives. I have plenty of external USB and FireWire storage (over 6 TB), and running VMs in either Parallels or VMWare Fusion works great (very highly performant) off any of these external drives.

However, there's no way I'm going to lug around an external hard drive and USB cable (and maybe power adapter) just so I can test things in Internet Explorer (basically, the only use I have for Windows).

Flash Drive to the Rescue!

I found a cheap 32GB USB flash drive that only sticks out of my MacBook Air half an inch, and copies at a consistent rate of 30MB/second (which is quite sufficient for most tasks). Also, the little drive should have very good read performance, since it's not a spinning platter. Write speed wouldn't be anything to brag about, but writing shouldn't happen all that often when simply opening up Internet Explorer—I hope!

Using apachebench (ab) with Drupal 7 to load test site with authenticated users

apachebench is an excellent performance and load-testing tool for any website, and Drupal-based sites are no exception. A lot of Drupal sites, though, need to be measured not only under heavy anonymous traffic load (users who aren't logged in), but also under heavy authenticated-user load. has some good tips for ab testing, but the details for using ab's '-C' option (notice the capital C... C is for Cookie) are lacking. Basically, if you pass the -C option with a valid session ID/cookie, Drupal will send ab the page as if ab were authenticated.

Instead of constantly going into the database and looking up session IDs and such nonsense, I have a simple script, which is quite revised from the 2008-era script originally from 2bits that worked with Drupal 5, which will give you the proper ab commands for stress-testing your Drupal site under authenticated user load. Simply copy the attached script (source pasted below) to your site's docroot, and run the command from the command line as follows:

Doing Some Benchmarks - Mac Processor Speed

I currently own or use a variety of Macs, and am approaching the end of a 'cycle' of Mac usage, where I need to decided what Mac I'd like to purchase next. Currently, I'm using a 27" iMac at work, an 11" MacBook Air (from work) for travel, and a 24" iMac at home. They're all great computers in their own right, and using Dropbox, MobileMe, and a couple other helper services, I can operate simultaneously on all three Macs, without any hiccups.

So, I'm thinking about getting a new Mac for hardcore development work (web and app), some graphic design, and possible portability. I have an iPad for lighter computing (reading, browsing, email, videos...), so even though the MacBook Air is probably the best thing to happen to a laptop in a very long time, I'm shying away from it as my primary personal computer.

Drupal 7 Front-End Performance - Shared Hosting Recommendations

Speedometer - BoostedI've spent a lot of time working on making sure my smaller Drupal sites (mostly run on shared hosts or very small VPSes) run lean and mean. This helps the pages load faster, users are happier, and my hosting providers don't have to shut down any of my sites, even when they're under pretty heavy load.

Here are my three recommendations for making your Drupal 7 website run great on a shared (or low-end VPS) host:

Drupal Performance Guide - Drupal and the LAMP/LEMP stack

LAMP Stack with Drupal - Druplicon, Linux, Apache, MySQL, PHP

Drupal is a scalable, flexible, and open source content management system that is built to run on a variety of server architectures. The only real requirement is that PHP runs on your system. You can run Linux, Microsoft, Mac OS X, etc., along with Apache, IIS, nginx, MariaDB, MySQL, PostgreSQL, etc. if you're willing to do a few extra things.

However, the overwhelming majority of Drupal websites use the most popular LAMP stack on the backend: Linux, Apache, MySQL and PHP, or the 'LEMP' variation, with Nginx instead of Apache. This white paper (which is a living document – I'll be updating it as time progresses) provides my thoughts on performance considerations for Drupal on a LAMP stack, but this information can be used for pretty much any system on any server, if you look at the basic principles.


Sending Recurring Emails to Thousands, using Simplenews as a Backend

I recently had a rather unique project requirement on one of my sites: I needed to send out a weekly email to hundreds (soon to be thousands) of site users, with the same template each week, but with the latest data from the website.

Basically, what I wanted to do was create a View on my site of the latest 10-12 items, and have that view be sent out to everyone (along with the views header/footer) in HTML (with a plain text alternative, of course), but I didn't want to have to create a new newsletter by hand each week to do this (this is something at which the Simplenews module excels... minus the automation).

After debating over whether I should write my own module to do the dirty work of sending out batches of emails, using modules like Views Send along with Rules and Views Bulk Operations, and some other crazy ideas, I finally found a potent combination for sending out automated weekly newsletters in a highly performant and optimal way, using the following modules:

  • Simplenews (for queuing/sending emails, managing subscriptions)
  • Rules (for scheduling the newsletters)
  • Elysia Cron (for easy cron scheduling and performance)
  • Views (to build the body of the newsletter)
  • Mime Mail (to send HTML emails, and to be able to easily have a custom email-friendly stylesheet)

Gzip/mod_deflate not Working? Check your Proxy Server

Recently, I was troubleshooting performance issues on a few different websites, and was stymied by the fact that YSlow repeatedly reported an F for "Compress components with gzip," even though online sites like GIDNetwork's Gzip test were reporting successful Gzipping of text components on the site.

Gzip Failed
Yslow results - not very happy.

After scratching my head for a while, I finally figured out the problem, hinted at by a comment on a question on Stack Overflow. Our work's proxy server was blocking the 'Accept-Encoding' http header that is sent along with every file request; this prevented a gzipped transfer of any file, thus Yslow gave an F.

I set up a secure tunnel (using SSH) from my computer to the web server directly, and then reloaded the page in FireFox, and re-ran YSlow: