Benchmarking PHP 7 vs HHVM - Drupal and Wordpress

[Multiple updates: I've added results for concurrencies of 1 and 10, results on bare metal vs. VMware instances, tested Drupal 8 vs Drupal 7 vs Wordpress 4.4, and I've also retested every single benchmark at least twice! Please make sure you're read through the entire post prior to contesting these benchmark results!]

tl;dr: Always test your own application, and trust, but verify every benchmark you see. PHP 7 is actually faster than HHVM in many cases, neck-in-neck in others, and slightly slower in others. Both PHP 7 and HHVM blow PHP ≤ 5.6 out of the water.

Skip to benchmark results:

Introduction and Methodology

As PHP 7 became a reality through this past year, there were scores of benchmarks pitting PHP 7 against 5.6 and HHVM using applications and frameworks like Drupal, Wordpress, Joomla, Laravel, October, etc.

One benchmark that really stood out to me (in that it seemed so wrong for Drupal, based on my experience) was The Definitive PHP 7.0 & HHVM Benchmark from Kinsta. Naming a benchmark that way certainly makes the general PHP populace take it seriously!

The results are pretty damning for PHP 7:

PHP 7 HHVM Definitive Benchmark screenshot by Kinsta

In the comments on that post, Thomas Svenson mentioned:

Standard installation for Drupal 8 has cache on as default. If you did not turn that off, then it is probably a reason to why the PHP 7 boost isn't bigger.

Would be interesting to see the result comparing the benchmark with/without caching enabled in Drupal 8. Should potentially reveal something interesting.

This was my main concern too, as there wasn't enough detail in the benchmarking article to determine what exactly was the system under test. Therefore, I'll submit my own PHP 7 vs HHVM benchmark here, using the following versions:

  • Ubuntu 14.04
  • Drupal 8.0.1
  • Nginx 1.4.6
  • MySQL 5.5.46
  • PHP 5.6.16, PHP 7.0.1, or HHVM 3.11.0

All tests were run using Drupal VM version 2.1.2 with VMware Fusion 8.1.0, on my mid-2013 MacBook Air 13" 1.7 GHz i7 with 8GB of RAM. Using the above notes, you can exactly replicate this benchmarking environment should you desire. All tests were run five times, the first two results were discarded (because they often reflect times when some caches are still warming), and the latter three were averaged.

After installing Drupal 8.0.1 with the standard installation profile (this is done automatically by Drupal VM), I logged in as the admin user (user 1), then grabbed the admin user's session cookie, and ran the following two commands:

# Benchmark Drupal 8 home page out of the box with default caching options enabled.
ab -n 750 -c 10 http://drupalvm.dev/

# Benchmark Drupal 8 /admin page logged in as user 1.
ab -n 750 -c 10 -C "SESSxyz=value" http://drupalvm.dev/admin

Drupal 8 results (concurrency 10)

Environment D8 Caching Requests/second Percent difference
PHP 5.6.16 Enabled (home, anonymous) 214.39 req/s ~
PHP 7.0.1 Enabled (home, anonymous) 407.10 req/s 62% faster than 5.6
HHVM 3.11.0 Enabled (home, anonymous) 260.19 req/s 19% faster than 5.6
PHP 5.6.16 Bypassed (/admin, user 1) 20.09 req/s ~
PHP 7.0.1 Bypassed (/admin, user 1) 39.26 req/s 65% faster than 5.6
HHVM 3.11.0 Bypassed (/admin, user 1) 34.41 req/s 53% faster than 5.6

...and some graphs of the above data:

PHP 5.6, PHP 7, and HHVM running Drupal 8.0.1, cached

PHP 5, PHP 7, HHVM benchmark cached Drupal 8 home page request

PHP 5.6, PHP 7, and HHVM running Drupal 8.0.1, uncached

PHP 5, PHP 7, HHVM benchmark uncached Drupal 8 admin request

Drupal 8 results (concurrency 1)

Sometimes, the use of concurrency (-c 10 in the above case)(to simulate concurrent users hitting the site at the same time, can cause benchmarks to be slightly inaccurate. The reason I usually use a level of concurrency is so the benchmark more closely mirrors real-world usage, and tests the full stack a little better (because PHP by itself is nice to benchmark, but very few sites are run on top of PHP alone!).

Anyways, I re-ran all the tests using -c 1, and am publishing the results below:

Environment D8 Caching Requests/second Percent difference
PHP 5.6.16 Enabled (home, anonymous) 171.34 req/s ~
PHP 7.0.1 Enabled (home, anonymous) 242.00 req/s 34% faster than 5.6
HHVM 3.11.0 Enabled (home, anonymous) 192.92 req/s 12% faster than 5.6
PHP 5.6.16 Bypassed (/admin, user 1) 19.89 req/s ~
PHP 7.0.1 Bypassed (/admin, user 1) 30.07 req/s 41% faster than 5.6
HHVM 3.11.0 Bypassed (/admin, user 1) 23.37 req/s 16% faster than 5.6

In all my benchmarking, I care more about deltas and reproducibility than measuring raw, clean-room-scenario performance, because unless a result is absolutely reproducible, it's of no value to me. Therefore if I can prove that there's no particular difference to testing with certain concurrency levels, I typically move the benchmark to a level that mirrors traffic patterns I actually see on my sites :)

Absolute numbers mean nothing to me—it's the comparison between test A and test B, and how reproducible that comparison is, that matters. That's why I enjoy benchmarking on the incredibly slow Raspberry Pi model 2 sometimes, because though it's much slower than my i7 laptop, it sometimes exposes surprising results!

Drupal 8 Results ('bare metal', concurrency 1)

Some people argue that running benchmarks in a VM is highly unreliable and leads to incorrect benchmarks, so I've also sacrificed a partition of a Lenovo T420 core i5 laptop (it has 3 SSDs inside, so I just formatted one, installed Ubuntu desktop 15.10, then installed PHP, MySQL, and Nginx exactly the same as with Drupal VM (same settings, same apt repos, etc.), and re-ran all the tests in that environment—so-called 'bare metal', where there's absolutely no overhead from shared filesystems, the hypervisor, etc.

Environment D8 Caching Requests/second Percent difference
PHP 5.6.16 Enabled (home, anonymous) 152.35 req/s ~
PHP 7.0.1 Enabled (home, anonymous) 230.67 req/s 41% faster than 5.6
HHVM 3.11.0 Enabled (home, anonymous) 142.50 req/s 7% slower than 5.6
PHP 5.6.16 Bypassed (/admin, user 1) 11.37 req/s ~
PHP 7.0.1 Bypassed (/admin, user 1) 13.13 req/s 14% faster than 5.6
HHVM 3.11.0 Bypassed (/admin, user 1) 11.40 req/s 0.3% faster than 5.6

After running these benchmarks with an identical environment on 'bare metal' (e.g. a laptop with a brand new/fresh install of Ubuntu 15.10 running the same software, with 8 GB of RAM and an SSD), it seems HHVM for some reason performed even worse than PHP 5.6.16 for Drupal 8.

Since this result is wildly different than the Kinsta post (basically the opposite of their results for Drupal 8), I decided to test Wordpress 4.4 as well.

Wordpress 4.4 Results ('bare metal', concurrency 1)

For Wordpress, I ran the test using the exact same Lenovo T420 environment as the test above, and tested an anonymous user (no cookie value) hitting the default home page, and an admin logged in (using a valid session cookie—actually all five of the cookies wordpress uses to track valid sessions) visiting the admin Dashboard page (/wp-admin/index.php).

Environment WP 4.4 Caching Requests/second Percent difference
PHP 5.6.16 Enabled (home, anonymous) 18.76 req/s ~
PHP 7.0.1 Enabled (home, anonymous) 40.45 req/s 73% faster than 5.6
HHVM 3.11.0 Enabled (home, anonymous) 40.14 req/s 73% faster than 5.6
PHP 5.6.16 Bypassed (/wp-admin/index.php, admin) 13.45 req/s ~
PHP 7.0.1 Bypassed (/wp-admin/index.php, admin) 28.10 req/s 71% faster than 5.6
HHVM 3.11.0 Bypassed (/wp-admin/index.php, admin) 35.43 req/s 90% faster than 5.6

...and some graphs of the above data:

PHP 5.6, PHP 7, and HHVM running Wordpress 4.4, anonymous home page

PHP 5 7 and HHVM benchmark comparison of Wordpress 4.4 home page anonymous

PHP 5.6, PHP 7, and HHVM running Wordpress 4.4, admin dashboard

PHP 5 7 and HHVM benchmark comparison of Wordpress 4.4 admin dashboard

These results highlight to me how much the particular project's architecture influences the benchmark. Wordpress still uses a traditional quasi-functional-style design, while Drupal 8 is heavily invested in OOP and a bit more formal data architecture. While I'm not as familiar with Wordpress's quirks as I am Drupal, I know that it's no speed demon, and also benefits from added caching layers in front of the site! It's interesting to see that PHP 7 and HHVM are practically neck-and neck for front-facing portions of Wordpress (and FAR faster than 5.6), while HHVM runs even a little faster than PHP 7 for administrative tasks.

Drupal 7 Results (concurrency 10)

I also benchmarked Drupal 7 on Drupal VM for another point of comparison (using -c 10):

Environment D7 Caching Requests/second Percent difference
PHP 5.6.16 Enabled (home, anonymous) 511.40 req/s ~
PHP 7.0.1 Enabled (home, anonymous) 736.90 req/s 36% faster than 5.6
HHVM 3.11.0 Enabled (home, anonymous) 585.71 req/s 14% faster than 5.6
PHP 5.6.16 Bypassed (/admin, user 1) 93.78 req/s ~
PHP 7.0.1 Bypassed (/admin, user 1) 169.95 req/s 57% faster than 5.6
HHVM 3.11.0 Bypassed (/admin, user 1) 143.25 req/s 42% faster than 5.6

For these tests, I went to the Performance configuration page prior to running the tests, and enabled anonymous page cache, block cache, and CSS and JS aggregation (to make D7 match up to D8 cached anonymous user results a little more evenly).

Some people point out benchmarks like these and say "Drupal 8 is slow"... and they're right, of course. But Drupal 8 trades performance for better architecture, much more pluggability, and the inclusion of many more essential 'out-of-the-box' features than Drupal 7, so there's that. Having built a few Drupal 8 sites, I don't ever want to go back to 7 again—but it's nice to know that PHP 7 can still accelerate all my existing D7 sites quite a bit!

Summary

tl;dr: For Drupal 7 and Drupal 8 at least, PHP 7 takes the performance crown—by a wide margin.

After running the benchmarks, I scratched my head, because almost every other benchmark I've seen either puts HHVM neck-and-neck with PHP 7 or makes it seem HHVM is still the clear victor. Maybe other people running these benchmarks didn't have PHP's opcache turned on? Maybe something else was missing? Not sure, but if you'd like to reproduce the SUT and find any results different than the above (in terms of percentages), please let me know!

I ran the HHVM benchmarks three times with fresh new VM instances just because I was surprised PHP 7 stepped out in front. PHP 5.6's performance is as expected... it's better than 5.3, but that's not saying much :)

The moral of the story: Trust, but verify... especially for benchmarks which compare a plethora of totally different applications, each result can tell a completely different story depending on the test process and system under test! Please run your own tests with your own application before definitively stating that one server is faster than another.

Installing HHVM in Drupal VM

Just for posterity, since I want people to be able to reproduce the steps exactly, here's the process I used after using Drupal VM's default config.yml (with Ubuntu 14.04) to build the VM:

  1. Log into Drupal VM with vagrant ssh
  2. $ sudo su
  3. # service php5-fpm stop
  4. # apt-get install -y python-software-properties
  5. # curl http://dl.hhvm.com/conf/hhvm.gpg.key | apt-key add -
  6. # add-apt-repository http://dl.hhvm.com/ubuntu
  7. # apt-get update && apt-get install -y hhvm
  8. # update-rc.d hhvm defaults
  9. # /usr/share/hhvm/install_fastcgi.sh
  10. # vi /etc/nginx/sites-enabled/drupalvm.dev.conf and inside the location ~ \.php$|^/update.php block:
    1. Clear out the contents of this configuration block.
    2. Replace with include hhvm.conf;
  11. # service hhvm restart
  12. # service nginx restart

Visit the /admin/reports/status/php page after logging in to confirm you're running HHVM instead of PHP.

Comments

Before you leave a comment arguing one of the following:

  • [HHVM|PHP 7] needs time to warm up its cache
  • Benchmarking inside a VM is unreliable
  • Your benchmarks are invalid because x, y, z

... please first read the blog post above—it outlines the exact, 100% reproducible system under test, and also mentions explicitly that you should always trust but verify. I don't argue my benchmarks are perfect... but there as close as I can get without a large government grant sponsoring my research ;)

You bring up an interesting point - would it be worthwhile rerunning the PHP 7 tests with opcode cache disabled, to see if that might produce similar stats to previous studies?

Please, to properly run a benchmark do not use concurrency. By using concurrency you cannot compare results between system.

I agree concurrency can cause some issues with test results in certain circumstances. I typically run benchmarks both with and without and compare the results. I didn't do it in this case because I didn't have the time when writing the post, but I will run the benchmarks again and update the post. More data is better data :)

[Update: I've added another table of all the results using -c 1; the results are interesting for their degrees of difference, but the general pattern is the same—for Drupal 8, HHVM is a little faster than PHP 5.6, but PHP 7 still takes the crown by a mile.]

Additionally, performance benchmarks are questionable when run from within a VM environment. Ideally, you'd run tests on bare metal, or you'd run enough tests in aggregate that you would be assured the whatever host effect there is would average out equally across multiple runs

I am aware of this fact—and to mitigate the issue, I ran every benchmark five times, discarding the first two results since they could trigger some cache warming. (That means every benchmark generated 3,750 requests over 5 runs in aggregate, so cache warming, disk issues, CPU scheduling, etc. should not have much affect). I also quit everything besides 3 tabs in Chrome and Mac OS X Mail during the benchmarking (plenty of free RAM, and CPU never went above 200% (out of 400% available).

Additionally, I've now reproduced every single result twice (both with -c 10 and -c 1), and also ran them on my other laptop (an 11" i7) with six new (destroy/rebuild) Drupal VM instances, and the every result has been within about a 3% margin of error.

[Update: I've added a whole slew of new benchmarks running on a Lenovo T420 core i5 laptop with a clean install Ubuntu 15.10, and literally nothing else running. I've also tested Wordpress in this environment for even further points of comparison. Please see the above results.]

I would think that running within a VM is the right thing to do - a lot of people using drupal 8/ hhvm/php7 will be running it inside a VM of some wort in real life. Very few people will be running this setup on bare metal.

You bring up a good point. Of the hundreds of Drupal sites I've encountered in the past 10 years, only *one* ran on a full dedicated 'bare metal' server. Every other one was running either on a VPS (Linode, Amazon, DigitalOcean, Rackspace, etc. etc.) or on shared hosting.

The funny thing is, running things on a local VM with VMware (e.g. This test) is much closer to a 'bare metal' test than running tests in Amazon's cloud, since I have no guarantee that my VM allocations on Amazon's cloud are on a physical machine with no noisy neighbors or strange little networking or CPU issues.

For the benefit of the naysayers, though, I will likely cave and run all the D8 tests once more on both an i5 Ubuntu 15.10 laptop ('bare metal' with no virtualization) and a Raspberry Pi model 2 B (if I can get HHVM compiled on it, that is—looks like ARM support is experimental).

Thanks Jeff,
Great information you compiled here. Very appreciated. As per your premise, "which one is more performant", it does not matter on what kind of machine virtual or bare metal you ran it (well...don't run it on a C64).

Hi Jeff,
We used a bare metal machine to do our benchmark test. You should try your's again using a bare metal machine to make the results authoritative. You will be surprised after the new test :)

Happy holidays!
Tom from Kinsta

I've added a whole slew of new benchmarks running on a Lenovo T420 core i5 laptop with a clean install Ubuntu 15.10, and literally nothing else running. I've also tested Wordpress in this environment for even further points of comparison. Please see the above results confirming that, at least in Drupal 8's case, PHP 7 still rules the roost for page load performance.

Hi,

I think that HHVM should be tested with & without the "repo authoritative mode", maybe this can explain some of the differences with other tests done by other people.

Hi Jeff, I like your approach of gathering the data and appreciate the effort! It looks like we can look forward to a slightly accelerated web.

Thanks for the extremely thorough test. I was a bit sceptical of that other benchmark for its lack of reproducibility.

"Some people point out benchmarks like these and say "Drupal 8 is slow"... and they're right, of course. But Drupal 8 trades performance for better architecture, much more pluggability, and the inclusion of many more essential 'out-of-the-box' features than Drupal 7"

It would be interesting to see a benchmark for the "minimal" install profile comparing D7 and D8. I suspect the difference would be smaller than for the "standard" install profile.

Hello Jeff,

Thanks for the accuracy and reliability of this study.
We are using HHVM for speeding up our clients' CMS for a while now. We are currently replaying all our benchmarks with PHP7. Our first target was our own wordpress website.

Here are the results : http://www.nxtweb.fr/en/2016/01/05/php7-versus-hhvm-premier-round-sur-n….

We came up with these conclusions :

=> HHVM (in our testing context) is still faster than Zend interpreter.
=> PHP7 uses less memory
=> Looking at how php code spends its time, it is rather easy to get more than bare HHVM or PHP7 (we wrote a little wordpress-specific HHVM extension as a proof of concept)
=> HHVM or PHP7 ? The most important is to switch to something better than PHP 5.6.

This seems a bit misleading.

I don't want my website to be able to handle tons requests super quickly my laptop. I want it to be able to handle it quickly on Digital Ocean or AWS which are going to have very different hardware with more server focused CPUs.

Just like I wouldn't benchmark a game performance on enterprise servers, I don't think we should be benchmarking backend server software on laptops.