Getting OSC's Drupal Install Optimized on Slicehost

Initially, when thinking about finally taking the plunge and purchasing a slice or two from Slicehost, I thought, "wow, this is going to be incredibly fast and awesome, compared to my Host Gator account!"

Slicehost + Drupal

But, after setting everything up and putting Open Source Catholic live on the fresh slice, running free -m, and looking at the results, reality set in: 256 MB of RAM is not much to work with if you're running a Drupal site on a LAMP stack! Drupal usually consumes 15-40 MB of RAM per page view for a logged-in user, and if you have a site with 10 or so logged in users at any moment... well, bad things can happen.

For anonymous users, using Boost will help your site fly no matter the amount of RAM you have. But even so, a bunch of requests to uncached pages will cause your site to load a heck of a lot slower, and will fill up your RAM faster than a fire hose fills up an 8 oz. glass!

Using default Apache, MySQL and PHP settings, free -m showed a full 250 MB of RAM used, along with 400-500 swap space used (swap should be kept to a minimum—if you have a lot of swap usage, that means the hard drive is being used instead of RAM, and the hard drive is inherently many times slower!). After performing a few quick modifications to Apache and MySQL, I was able to get this number down to 140 MB RAM / 40-60 MB swap, on average.

I modified the server configuration in two different places: Apache's httpd.conf, and MySQL's my.cnf:

Configuring Apache's httpd.conf

One quick way to control typical memory/resource usage if you're running Apache as your web server is to edit the defaults in your Apache configuration file, httpd.conf. In CentOS, this file is located at /etc/httpd/conf/httpd.conf. To edit the file, simply run $ sudo nano /etc/httpd/conf/httpd.conf from the command line (you need to edit it as root).

The pertinent areas of the file are listed here, with an explanation of what they mean below:

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#

KeepAlive On

KeepAlive allows you to keep a connection with a certain web browsing client over a period of time/requests that you choose. This is a good thing for you to turn on (it's off by default), as it will let your server keep an http connection open with a certain browser if they click through to another link, instead of opening a new connection each time someone goes somewhere on your site.

A caveat: make sure you keep the KeepAlive timeout pretty low, so that people who don't keep clicking through to more links on your site don't waste an Apache process on your server.

# 
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#

MaxKeepAliveRequests 100

MaxKeepAliveRequests is the number of http requests a browser can make before a new connection is made - each image, javascript file, css file, etc., is a new request, so make sure you have enough requests to cover at least two page loads.

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#

KeepAliveTimeout 4

KeepAliveTimeout: After four seconds, the http connection will be closed, and if the browser requests more information, a new connection will be made.

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c> StartServers 1 MinSpareServers 1 MaxSpareServers 5 ServerLimit 10 MaxClients 200 MaxRequestsPerChild 1000 </IfModule>
# worker MPM
# StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule worker.c> StartServers         2 MaxClients         200 MinSpareThreads     25 MaxSpareThreads     75  ThreadsPerChild     25 MaxRequestsPerChild  1000 </IfModule>

There are a lot of important directives in this section!

  • StartServers: In a low-memory environment, you want to keep this low, because each server instance takes up a few MB (at least) of RAM. While one of the server threads is serving up a Drupal, Joomla, or WordPress (or insert-your-CMS-here) page, the process can climb from 2-3 MB to a whopping 30-50MB!
  • MaxClients: How many clients you'd allow your server to serve at once. Try to keep it relatively low, in case your server gets spiked... but if you're running on a low-memory server, you'll probably not be able to reach this limit unless you're running some serious caching mechanisms.
  • ServerLimit: How many Apache server processes can run simultaneously. For a server running with 256 MB or RAM, you'd want to keep this pretty low—usually somewhere around 5-10—because estimating 30MB per process (~200 MB RAM total), you still need to factor in memory for MySQL and any other running processes!
  • MaxRequestsPerChild: This defines how often an Apache process will be quit, and a new one started. Typically this doesn't matter too much, but processes can start gobbling up and wasting memory, so you'd want them to reset every so often.

Configuring MySQL's my.cnf

The my.cnf file controls a lot of important MySQL server variables, and can help you tame server utilization and database access speed. Typically, this file is located at /etc/my.cnf. To edit the file, simply run $ sudo nano /etc/my.cnf from the command line (you need to edit it as root).

I've pasted my own my.cnf file here, and explained some of the more important sections below:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
max_connections = 75
max_user_connections = 75
key_buffer = 16M
myisam_sort_buffer-size = 32M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 286
interactive_timeout = 25
wait_timeout = 1000
connect_timeout = 10
max_allowed_packet = 1M
max_connect_errors = 1000
query_cache_limit = 1M
query_cache_size = 16M
query_cache_type = 1
tmp_table_size = 16M

Here are some of the more important variables:

  • Max connections / Max user connections: You don't want this too high, because it should only allow for about 6-8x the Apache ServerLimit, in order to allow each process to make 6-8 database connections at a time (thus keeping your server running happier during traffic spikes, and allowing more connections per thread when the server's not under load).
  • key_buffer / myisam_sort_buffer_size / query_cache_size / tmp_table_size: Keep these as high as you can, while taking into account the fact that, during traffic spikes, your Apache processes will take (if configured correctly) 80% of your RAM. Thus, the total of these caches should be around 15% of your RAM. (In this case, it equals 80 MB - a little more than 15%, but that's because I optimize the site for non-spike conditions... if there's a traffic spike, I don't expect my poor little slice to hold up).
  • Other values: The more RAM allocated, the merrier... however, you should refer to other sites for specific values; these values were pretty much copied from a few different recommended low-memory my.cnf configurations.

If you have more RAM, the values above will not be sufficient. Also, you should always check your logs/MySQL performance variables from time to time to see if any of your buffers need adjusting. Drupal has a page just for this (click on the MySQL version on your Status Report page)!

Another resource that is pretty helpful when tuning your MySQL server is JoomlaPerformance's guide (even though they're Joomla-specific, their general guidelines apply to any application using MySQL.

The moral of this story?

Buy as much RAM as you can afford, and make sure your Drupal site (or other PHP/MySQL-based site) is using all forms of caching to its fullest extent... in a way, hardware is cheap; however, shaving off half-seconds on your page load times (a goal achieved by following the advice above) is always a good thing, so do every bit of tuning you possibly can to make your site fly.

Comments

Memory and disk space is what made me move away from Slicehost and switch to Amazon EC2 instead. You pay quite a bit more for a small EC2 instance (per month about $80 USD for EC2 instead of $20 USD for Slicehost) but you get a lot more bang for your buck.

For one, the memory on EC2 is for a standard Linux instance 1.7Gb compared to the 256Mb on Slicehost. This makes a huge difference if you want to do something "useful" and not just play around. Next, on EC2 you get -out of the box- 160Gb of storage. On Slicehost you only 10Gb!!!

At the time, the application I was building didn't need CPU, just decent memory and lots of disk space. Even Slicehost's $800 USD monster package wouldn't fit. To me, EC2 is in the long run just a much better solution, especially if you reserve your instance.

The best way to save memory is to use a PHP opcode cache. These can drastically reduce the amount of memory used per Apache process. Check out eAccelerator and APC. These will also speed up the execution of your pages by keeping the cached opcodes of your PHP files in memory, instead of opening the files and parsing them each time a request comes in.

eAccelerator tends to save slightly more memory, but also be slightly slower than APC in page execution, but both save memory and have short executions that a stock mod_php installation.

For $20/mo at Linode, you get 360 MB of RAM. (No affiliate link here to try and get money, just a happy customer for about 4 years now.)

I can vouch for this as well. I had a 256 slice, added APC and saw big improvements on the number of request I could handle per second. Though, it wasn't until I upgraded my account to a 512MB slice before I was truely happy with SliceHost.

@ Anonymous: I forgot to mention, I am running APC on this slice, and that does help a bit in terms of responsiveness. Even so, there's little you can do to save precious RAM when you have so little!
I've been considering switching to Linode or Webbynode and playing around in a sandbox there, but the main concern right now is budget...
@ Luke: EC2 is a bit out of my price range right now, but it's definitely an option for expansion. If I needed true scalability, I would either go into the cloud, or go dedicated through SoftLayer or another good dedicated hosting company...

I would suggest considering switching from Apache to a more lightweight web server such as Lighttpd. That will definitely help you make the most of a limited RAM allotment, and it should be able to do everything you need it to.

@ Garrett - does Drupal play nice with Lighttpd? Also, Do you know about how much memory can be conserved in such a scenario?

geerlingguy, it plays very nice indeed, though I couldn't give you any hard numbers. I can tell you that the VPS account we're using is hosting about ten sites with XCache and still has plenty of free RAM…

There's a little bit of trickery required to get clean URLs working, but it's not impossible.

Check out this article entitled "Why Lighty instead of Apache?" at the Lighty focus group on G.D.o, and feel free to post your own thread if you have questions.

@ Garrett - awesome - will do. I'm glad I haven't moved all my sites over to my slice yet, because I don't feel so bad about messing only one of them up for a short period of time ;-)

My settings for low memory usage:

KeepAlive Off
MaxKeepAliveRequests 256
KeepAliveTimeout 2
MinSpareServers 2
MaxSpareServers 5
StartServers 3
MaxClients 100
MaxRequestsPerChild 150