drupal planet

Drupal News from the Midwest!

Some random bits of news from Midwestern Mac, LLC:

St. Louis-area Drupal Group

After taking a hiatus for the month of December, the St. Louis area Drupal Group will be meeting up (hopefully) on the third Thursday of the month as normal. We're hoping to have more structure to our meetups, and there are already some great ideas for meeting topics in 2013.

If you live in or around St. Louis and use or contribute to Drupal, please make an effort to join us and build up the Drupal community here in St. Louis!

As an aside, we still have a separate website for the St. Louis Drupal group—if anyone has ideas for how we can use that to spread the Drupal love in the center of the U.S., please let us know!

Server Check.in Launched

A couple weeks ago, we (Midwestern Mac, LLC) announced our newest service, Server Check.in, a website and server monitoring service that checks on your sites and servers every 10 minutes, notifying you of any problems. The service runs on Drupal, and integrates with services like Twilio and Stripe to handle SMS messaging and payments, respectively.

I (geerlingguy) wrote up a case study for Server Check.in and posted it to the Community showcase on drupal.org. This is the first application-type service built on by Midwestern Mac on Drupal, and we've already been hard at work improving the service.

If you have any questions about Server Check.in, or how it was built, please ask away; I had a great discussion with some other developers in this thread on Hacker News.

Hosted Apache Solr Search updated to 3.6.x

At the request of many people who wanted to do some neat new things with Solr on their Drupal sites, we've finally followed Acquia's lead and updated some of our Solr search servers to 3.6.x, meaning things like Location-based searching are now possible. And our servers are happier :)

Server Check.in - Website and Server uptime monitoring

Server Check.in is a simple and inexpensive server and website uptime monitoring service I've recently launched.

Server Check.in logo

If you have a website or online service you need to make sure is running, Server Check.in is a great way to get notified when there's a problem. Unlike most other monitoring solutions, Server Check.in offers free SMS (text) messages and email notifications, and it only costs $15/year (just $1.25/month!).

The Motivation

There are probably thousands of other uptime monitoring services on the web, and it's typically a good idea to use existing tools rather than build your own—if they're practical for your needs!

I had three main requirements for any service I wanted to use:

Preventing Security Holes

I was recently browsing a very popular review website, when I noticed the following warnings popping up:

Angie's List website errors

From simply loading their web page and seeing these error messages, I could conclude:

  1. The website is using Drupal.
  2. The website is using memcached.
  3. The website is running on Acquia's managed hosting cloud.
  4. The website has error reporting set to print all errors to the screen.

If I were trying to break into this review site, or cause them a bad day, the information presented in this simple error message would help me quickly tailor my attacks to become much more potent than if I started from a blank slate.

Security through obscurity

I will quickly point out that security through obscurity—thinking you're more secure simply because certain information about your website is kept secret—is no security at all. However, that doesn't mean that obscurity is not an important part of your site's security.

Line breaks instead of Paragraphs in TinyMCE (by default)

Most people who have grown up on the web, and have used Wysiwyg utilities online, or newer text editors/word processing applications are used to having a simple 'return' create a new paragraph, with (on average) one extra line of empty space between the new paragraph and the one before it.

However, a lot of people like having the 'return' key just go down one line. There are a few ways this is possible in most Wysiwygs:

  • You can change the block style from 'Paragraph' (which creates <p> tags around new lines of text) to 'div' (which creates <div> tags around new lines of text).
  • You can press Shift + Return when you want to just go down one line (using a <br /> tag instead of a <p> tag).

I use the second method when I'm using a Wysiwyg, as I like using paragraphs (which are semantic for text, and which allow for better CSS styling than a monolithic block of text with linebreaks). I also rarely use a Wysiwyg editor, so it's not really an issue for me anyways ;-)

APC Caching to Dramatically Reduce MySQL traffic

One Drupal site I manage has seen MySQL data throughput numbers rising constantly for the past year or so, and the site's page generation times have become progressively slower. After profiling the code with XHProf and monitoring query times on a staging server using Devel's query log, I found that there were a few queries that were running on pretty much every page load, grabbing data from cache tables with 5-10 MB in certain rows.

The two main culprits were cache_views and cache_field. These two tables alone contained more than 16MB of data, which was queried on almost every page request. There's an issue on drupal.org (_field_info_collate_fields() memory usage) to address the poor performance of field info caching for sites with more than a few fields, but I haven't found anything about better views caching strategies.

Knowing that these two tables, along with the system cache table, were queried on almost every page request, I decided I needed a way to cache the data so MySQL didn't have to spend so much time passing the cached data back to Drupal. Can you guess, in the following graph, when I started caching these things?

MySQL Throughput graph - munin

Reclaim Your Hard Drive - Saving Tons of Space with MySQL InnoDB tables

Drupal 7 uses InnoDB tables. InnoDB provides many benefits, but can cause some unexpected headaches. One headache for me is that, by default, MySQL tells InnoDB to create one file on your system, called ibdata1, to hold ALL the data from EVERY InnoDB table you have on your MySQL server. This file never shrinks in size; it only expands to contain new data. If you delete something from MySQL or drop a table, the space that table was using is reallocated for other new data. This isn't a bad thing, especially for those who have a lot of drive space, and not many databases that are altered or dropped quite frequently.

I develop a lot of sites on my little MacBook Air (with a 128GB SSD), so I often download database snapshots from live and testing environments, empty out the tables on my local environment, then import the database dumps. Can you spot the problem here?

Using Daisy Disk I just noticed that my ibdata1 file had grown to more than 10 GB, and my Air's drive only had about 5 GB free space!

So, after reading through MySQL's InnoDB Engine documentation and this answer on Stack Overflow, I found that it's not too hard to change MySQL to keep data tables in their own files, and delete the files after the tables are deleted (thus saving me a ton of space). It just takes a little time and annoyance.

Here's how to do it, roughly:

PSR-0 PSR-1 PSR-2, Drupal, and You!

For the past couple years, discussions about 'PSR-0', PHP standards, and some sort of framework standardizations have been popping up here and there. It wasn't until a bunch of 'PSR-0 Interoperability' patches started popping up in the Drupal core issue queues that I decided to take a closer look at PSR. (The latest? PSR-1 (Basic Coding Standard) and PSR-2 (Coding Style Guide) have been accepted).

Filter/Search on multiple fields with Views 3.x

[Update: As of Views 7.x-3.4, you can now use the new "Global: Combine fields filter" to combine fields for an exposed search. Just add the fields you want to search to the view's Fields section, then add a 'Global: Combine fields filter' and select all the fields you want to search. Simple as that!]


A common need I run into with a ton of Drupal sites and Views is searching/filtering content based on multiple fields. For example, a lot of people would like to search for content using either the Title or the Body for a particular content type.

There are two primary solutions offered for this situation, but they both have downsides or are overly complex, in my opinion:

  • Use the Computed Field module to create yet another field stored in the database, combining the two (or more) fields you want to search, then expose a filter for that field instead of both of the individual fields. (I don't like this because it duplicates content/storage, and involves an extra module to do so).
  • Use the Views Filters Populate to invisibly populate a second field that you've added to a views OR group (using Views OR in Views 2.x, or the built-in AND/OR functionality in Views 3.x). (This module is slightly limited in that you can only work with strings, and again, it involves an extra module).

Instead of using an extra module, I simply do the following to achieve a multi-field search:

Getting XHProf working well on Mac OS X Lion

I was inspired today to get XHProf working on my Mac, using MAMP PRO 2.0.5/PHP 5.3.6, after reading @Dave Reid's tweet. Since I'm not leaving for DrupalCon until tomorrow, what else could I do today? There's an excellent article on Lullabot that will help you get 85% of the way towards having XHProf up and running on your Mac, working with your Drupal sites, but there are a few missing pieces and little tips that will help you get XHProf fully-armed and operational.

XHProf Callgraph example
Ooh, pretty visualizations!

Using Batch API to build huge CSV files for custom exports

Flocknote is a large web application that lets churches easily manage communications with their members via email, text message, and phone calls. Many of the core features of email marketing services like MailChimp and Constant Contact are implemented in flocknote similarly, such as list management and mass emailing (and many features like shared list/member information management, text messaging, etc. are unique to flocknote).

Until recently, few groups using flocknote didn't have subscription lists that were big enough to hit our relatively high PHP max_time_limit setting when importing and exporting subscriber data. Since we're getting bigger, though, I've started implementing Batch API all over the place so user-facing bulk operations could not only complete without resulting in a half-finished operation, but could also show the end user exactly how much has been done, and how much is left:

Exporting List Subscribers - Batch API CSV Export

I've seen many tutorials, blog posts, and examples for using Drupal's Batch API for importing tons of data, but very few (actually, none) for exporting tons of data—and specifically, in my case, building a CSV file with tons of data for download. The closest thing I've seen is a feature request in the Webform issue queue: Use BatchAPI to Export very large data sets to CSV/Excel.

Before I get started, I want to mention that, for many people, something like Views Data Export (for getting a ton of data out of a View) or Node Export (specifically for exporting nodes) might be exactly what you need, and save you a few hours' time working with Batch API. However, since my particular circumstance ruled out Views, and since I was exporting a bit more customized data than just nodes or users, I needed to write my own batch export functionality.

Pages

Subscribe to RSS - drupal planet