drupal

Using Batch API to build huge CSV files for custom exports

Flocknote is a large web application that lets churches easily manage communications with their members via email, text message, and phone calls. Many of the core features of email marketing services like MailChimp and Constant Contact are implemented in flocknote similarly, such as list management and mass emailing (and many features like shared list/member information management, text messaging, etc. are unique to flocknote).

Until recently, few groups using flocknote didn't have subscription lists that were big enough to hit our relatively high PHP max_time_limit setting when importing and exporting subscriber data. Since we're getting bigger, though, I've started implementing Batch API all over the place so user-facing bulk operations could not only complete without resulting in a half-finished operation, but could also show the end user exactly how much has been done, and how much is left:

Exporting List Subscribers - Batch API CSV Export

I've seen many tutorials, blog posts, and examples for using Drupal's Batch API for importing tons of data, but very few (actually, none) for exporting tons of data—and specifically, in my case, building a CSV file with tons of data for download. The closest thing I've seen is a feature request in the Webform issue queue: Use BatchAPI to Export very large data sets to CSV/Excel.

Before I get started, I want to mention that, for many people, something like Views Data Export (for getting a ton of data out of a View) or Node Export (specifically for exporting nodes) might be exactly what you need, and save you a few hours' time working with Batch API. However, since my particular circumstance ruled out Views, and since I was exporting a bit more customized data than just nodes or users, I needed to write my own batch export functionality.

A First Timer's Guide to DrupalCon

Preparing for your first DrupalCon? Even if this isn't your first, here are a few tips and tidbits I've learned from my first DrupalCon last year, and would like to pass on to you. (I'm posting this now so you have time to order the things you need to make your conference experience better and get it shipped!).

Keep things you need handy

I expected to have some downtime every now and then to run back to my hotel room and grab something I needed for later in the day (like a power cord), but quickly realized that I wouldn't have downtime. Instead, I ended up attending many awesomesauce presentations, BoFs (Birds of a Feather gatherings), core conversations, and informal meetings continuously, from the time I got into the convention floors until about 8 p.m. (and later!).

Bring a bag large enough to hold your laptop or iPad, a charger, a few snacks (granola bars are great!), and any other little devices or chargers you'll need during the day.

Power to the People!

Monster Outlets to GoHotels and convention centers have a very low AC outlet / conference attendee ratio. Usually something like 1:100. Most laptops' batteries last 3-5 hours. You're going to have your laptop on and with you all day, and the battery will die if you don't charge up every now and then.

One of the best things you can do, especially if you want people to not hate you for hogging an entire outlet for one laptop charger, is buy a travel power strip, like the one I bought for this year's DrupalCon—Monster's Outlets to Go Powerstrip*. There are a few other options out there, but I like this one the most due to its compactness. Some adapters even include or two USB plugs (though not all are created equal—check to make sure the USB plugs provide enough power to charge your device!).

Instead of hogging a wall jack all to yourself, you can now power one or two of your own devices, and let one or two other people charge their devices.

For non-US residents, be sure you have the proper power adapters for your devices!

Don't only go to sessions

I made the mistake of trying to attend every session that piqued my interest last year. It wasn't until the last day of the conference that I hopped out of a session that had lost my interest and found that I was missing some of the best parts of DrupalCon:

  • Birds of a Feather gatherings (people basically come together and talk about/work through things things they have in common, like newspaper websites, Church sites, or a passion for DevOps!).
  • Core Conversations (people who want to make Drupal and Drupal.org better come together and, well, make Drupal and Drupal.org better).
  • The Expo area (talking to some of the people in Drupal consultancies, or people from hosting providers, or anyone else on the expo floor, is pretty enriching).
  • The community (getting to meet people I converse with every week on drupal.org, in IRC, etc. is awesome).

Creating an Image Effect to put a play button on Video thumbnails

I had a rather interesting feature to implement on flocknote lately (after doing a pretty vast redesign of the UX/UI on the site over the past month... it was refreshing to dig into PHP again!):

We want to allow insertion of YouTube and Vimeo (and potentially other) videos into 'Notes' on the site, and there are a few moving parts in this equation:

  • I had to create a text format filter similar to the 'Embedded media inline' module in Drupal 6 so people could simply put a 'merge tag' in their Note (like [video=URL]) where they want the video to appear.
  • When a user views the embedded video on the site, the video should show at a uniform width/height, and be able to play the video (basically, a merge tag the user enters should be converted to the proper embed code for the provider (in this case, an <iframe> with the proper formatting).
  • When a user sees the video in the note email, the video can't actually play since very few email clients support any kind of video embedded in an email. So, instead, the video shows as a frame with a play button on top (this is the trickiest part), and links to the video on YouTube, Vimeo, etc.

Creating my own Image Effect for a Video Play Button

What I wanted to end up with was an image that had a custom-made iOS-style play button (play icon in a circle with a translucent grey background) right in the middle (I like the simple look of videos on my iPad...):

Video Play Button Example

So, I decided to work with Drupal's Image Effect API and expose a new image effect, aptly named 'Video Play Button', to Drupal's simple set of 'Resize, Scale, etc.' image effects. This is a pretty simple process:

What happened to Life is a Prayer.com?

[Update: I've finished building a new theme for Life is a Prayer.com; it's a bit of the old, mixed in with a bit of the new. What do you think?]

I'm in the middle of upgrading Life is a Prayer.com to Drupal 7 (to the non-geeks out there: I'm changing up the structure of the site a little bit, and giving it a new coat of varnish).

So... for a while you'll probably see a few things here and there that are out of place (especially in sections like the Pictures area, where photo galleries are displayed). Please report any problems you have in the comments below, and wish me luck as I start redesigning the site to make it look even better than ever!

(This current look is only temporary - Things will be back to normal soon!).

Interview on Make Web Not War

A recent interview focusing on my involvement in open source development (mostly centering around my work with Make Web Not War website:

Make Web Not War - Interview with Jeff Geerling
Interview - Jeff Geerling - Open Source Catholic

In the interview, I speak about my involvement in Drupal, and my appreciation for a variety of different open source projects. I'm glad Microsoft is putting some resources behind sites like 'Make Web Not War', and I hope they continue to reach out into different developer communities.

Moving Comments into a Block - Drupal 7

[Note: It looks like there's a new module, as of January 2013, Node Comment Block, which uses the technique outlined below to move comments into a block.]

Most of the time, Drupal's convention of printing comments and the comment form inside the node template (node.tpl.php) is desirable, and doesn't cause any headaches.

However, I've had a few cases where I wanted to either put comments and the comment form in another place on the page, and in the most recent case, I asked around to see what people recommended for moving comments out of the normal rendering method. I found a few mentions of using Panels, and also noticed the Commentsblock module that does something like this using Views.

However, I just wanted to grab the normal comment information, and stick it directly into a block, and put that block somewhere else. I didn't want Views' overhead, or to have to re-theme and tweak things in Views, since I already have a firm grasp of comment rendering and form theming with the core comment display.

So, I set out to do something similar to this comment on drupal.org (which was also suggested by Jimajamma on Drupal Answers).

Using apachebench (ab) with Drupal 7 to load test site with authenticated users

apachebench is an excellent performance and load-testing tool for any website, and Drupal-based sites are no exception. A lot of Drupal sites, though, need to be measured not only under heavy anonymous traffic load (users who aren't logged in), but also under heavy authenticated-user load.

Drupal.org has some good tips for ab testing, but the details for using ab's '-C' option (notice the capital C... C is for Cookie) are lacking. Basically, if you pass the -C option with a valid session ID/cookie, Drupal will send ab the page as if ab were authenticated.

Instead of constantly going into the database and looking up session IDs and such nonsense, I have a simple script, which is quite revised from the 2008-era script originally from 2bits that worked with Drupal 5, which will give you the proper ab commands for stress-testing your Drupal site under authenticated user load. Simply copy the attached script (source pasted below) to your site's docroot, and run the command from the command line as follows:

Changing RSS Feed item links (and other data) in Drupal 7

You can do a lot of great things with field display in Drupal 7's 'manage display' tab for a content type. You can control the order and label position of each field attached to a node type in that tab for Full node displays, Teasers, and RSS displays (or other displays you set up).

However, there's no way to change certain aspects of a node's display inside an RSS Feed, such as the 'creator' tag, the 'link' tag, or the 'title' tag. For a news aggregation site I run, I wanted to modify the <link> tag when displaying 'story' nodes, and make the link tag give an absolute URL to the original source instead of to my drupal site (so, instead of http://www.mysite.com/node/12, it would go to http://www.example.com/original-story-url).

A lot of blogs also use this kind of format for reposted blog items (such as Daring Fireball), so users go straight to the source when they click on the title of an item in their RSS reader of choice. My method below can be modified to conditionally change a link if a field has a value (say, a 'RSS absolute URL' field or something like that).

Simple Git feature branch workflow

After reading A successful Git branching model [nvie.com], which I consider one of the best graphical/textual depictions of the ideal Git model for development teams (and most large projects), I simply wanted to adapt a similar (but way less complex) model for some of my smaller sites and multisite Drupal installs.

Since I'm (almost always) the only developer, and I develop locally, I don't want the complexity of working on many branches at once (master, hotfixes, develop, release, staging, etc...), but I do want to have a clean separation between what I'm working on and the actual live master branch that I deploy to the server.

So, I've adopted a simple 'feature branch model' for my smaller projects:

  • master - the live/production code. Only touch when merging in a feature or simply fixing little bugs or really pressing problems.
  • [issue-number]-feature-branches - Where I work on stuff.

Graphically:

Calling form validate functions in include files from other modules

Update: See comments below, and completely ignore this post. Nothing to see here...

module_load_include() is a great way to add code from other module's include files, but it doesn't always work as you'd expect. Recently, I was building a form in one module that pulled up a validation function from another module when a particular submit button was pressed: