Composer and Drupal are still strange bedfellows

More and more sites are being built in Drupal 8 (over 160,000 as of DrupalCon Baltimore 2017!). As developers determine best practices for Drupal 8 site builds and deployment, they need to come to terms with Composer. In one of the most visible signs that Drupal is 'off the island', many modules are now requiring developers to have at least a fundamental grasp of Composer and dependency management.

But even more than that, many developers now use Composer in place of manual dependency management or a simpler tools like Drush Make files.

With these major changes comes some growing pains. Seeing these pains on a daily basis, I wrote Tips for Managing Drupal 8 projects with Composer to highlight some best practices and tricks for making Composer more powerful and helpful.

But many developers still wrestle with Composer, and mourn the fact that deployments aren't as simple as dragging zip files and tarballs around between servers, or checking everything into a Git repository and doing a git push. For example:

  • If I manage my codebase with Composer and follow Composer's own recommendation—don't commit dependencies in my vendor directory, what's the best way to actually deploy my codebase? Should I run composer install on my production web server? What about shared hosting where I might not have command line access at all?
  • Many modules (like Webform) require dependencies to be installed in a libraries folder in the docroot. How can I add front end dependencies via Composer in custom locations outside of the vendor directory?

And on and on.

DrupalCon Baltimore 2017 - participants sitting and waiting to see the opening Keynote
Over 3,000 community members attended DrupalCon Baltimore 2017.
(Photo by Michael Cannon)

During a BoF I led at DrupalCon Baltimore 2017 (Managing Drupal sites with Composer), we identified over 20 common pain points people are having with Composer, and for many of them, we discussed ways to overcome the problems. However, there are still a few open questions, or problems which could be solved in a number of different ways (some better than others).

I've taken all my notes from the BoF, and organized them into a series of problems (questions) and answers below. Please leave follow-up comments below this post if you have any other thoughts or ideas, or if something is not clear yet!

Note: I want to make it clear I'm not against using Composer—quite the contrary, I've been using Composer for other PHP projects for years, and I'm glad Drupal's finally on board! But Drupal, with all it's legacy and complexity, does require some special treatment for edge cases, as evidenced by the breadth of problems listed below!

Common Problems and Solutions with Drupal + Composer

Contents:

How do I deploy a Drupal codebase if the vendor directory isn't in the codebase?

This was one of the most discussed issues. Basically: If it's best to not commit the vendor directory to my Git repository, and I deploy my Git repository to my production server... how do I run my site in production?

In the old days, we used to commit every file for every module, theme, and library—custom or contributed—to our website's codebase. Then, we could just git push the codebase to the production web server, and be done with it. But nowadays, there are two different ways you can not commit everything and still run a codebase on production:

  1. Using Deployment Artifacts: Acquia's BLT project is the best example of this—instead of cloning your source code repository to the production server, BLT uses an intermediary (either a developer locally, or in a CI environment like Travis CI or Jenkins) to 'build' the codebase, and then commit the 'build' (the artifact to be deployed, containing all code, including core, contrib modules, libraries, etc.) to a special branch or a separate repository entirely. Then you deploy this artifact to production.
  2. Run composer install on Prod: This is often a simpler solution, as you can still use the tried-and-true 'git push to prod' method, then run composer install in place, and all the new dependencies will be installed.

The second method may seem simpler and more efficient at first, but there be dragons. Not only is it likely that Composer (which is quite the RAM-hungry monster) will run out of memory, leaving a half-built copy of your codebase, it's also difficult to manage the more complicated deployment steps and synchronization required to make this work well.

The first method is the most stable production deployment option, but it requires either two separate Git repositories, or some other mechanism of storing the deployment artifact, and it requires an additional manual step or a CI system to actually build the deployment artifact.

Some hosting providers like Acquia are building new systems like Acquia Cloud CD to make the deployment artifact process more streamlined. But in either case (composer install on prod or deployment artifacts), there are tradeoffs.

I need to put a library (e.g. CKEditor for Webform) in the libraries directory, and not in vendor. Is this even possible with Composer?

Using the composer/installers package, you can set specific paths for certain types of packages, and you can even set specific directories per-package as long as the package has a type and requires composer/installers.

The last bit is the part that makes this a little difficult. Let's take two examples:

First, if you want to install all the Drupal modules (type of drupal-module) into your codebase in docroot/modules/contrib. Tell Composer by setting paths in the extra section of composer.json:

"extra": {
    "installer-paths": {
        "docroot/core": [
            "type:drupal-core"
        ],
        "docroot/modules/contrib/{$name}": [
            "type:drupal-module"
        ],
        "docroot/themes/contrib/{$name}": [
            "type:drupal-theme"
        ],
        "docroot/libraries/{$name}": [
            "type:drupal-library"
        ],
    },
}

Now, lets say you've installed the Webform module, which requires the geocomplete library be placed inside your site's libraries directory.

Since geocomplete isn't available through Packagist (and doesn't have type: 'drupal-library'), we can't easily tell Composer to put that particular library somewhere outside of vendor.

Currently, there are three ways to work around this issue, and all of them are slightly hacky (in my opinion):

  1. Add the Composer Installers Extender plugin, which allows you to set a custom install path per-dependency.
  2. Add the library as a custom repository in your composer.json file's repositories section:

    "repositories": { "drupal": { "type": "composer", "url": "https://packages.drupal.org/8" }, "fontawesome-iconpicker": { "type": "package", "package": { "name": "itsjavi/fontawesome-iconpicker", "version": "v1.3.0", "type": "drupal-library", "extra": { "installer-name": "fontawesome-iconpicker" }, "dist": { "url": "https://github.com/itsjavi/fontawesome-iconpicker/archive/1.3.0.zip", "type": "zip" }, "require": { "composer/installers": "~1.0" } } } }

  3. Attach a script to one of Composer's 'hooks' to move the library at the end of the composer install process. See the Composer Scripts documentation for more info and examples.

After the BoF, I opened a core issue, Drupal core should help make 3rd party library management not torturous, and basically asked if it might be possible for Drupal core to solve the problem using something like the third option, but using a class or library that was provided by core and available to all contributed and custom Drupal projects (instead of each codebase solving the problem in some one-off way).

The workarounds are all a bit burdensome, even after you have a few site builds under your belt. Hopefully this situation improves over time (note that a similar issue, Best practices for handling external libraries in Drupal 8, has been open since late 2015).

One frontend library I need to add to my site (for a module, not a theme) is not on Packagist. How can I composer require it without it being on Packagist?

There are two ways you can add packages that might exist on frontend packaging sites, but not Packagist:

  1. Use Asset Packagist to require Bower or NPM packages (even if they're not on Packagist).
  2. Either get the upstream package maintainer to add a composer.json file to the package repo and submit it to Packagist, or fork the repository and add your fork to Packagist.

The former option is my preferred method, though I've had to do the second option in a pinch a couple times. The best long-term solution is to try to get the upstream library added to Packagist, so you don't have any extra maintenance burden.

What's the best way to start building the codebase for a new Drupal 8 website?

The community seems to have settled on using a starter template like the Composer template for Drupal projects. In the beginning, some people did a require drupal/core to kick off a new Drupal codebase... but doing that can lead to some pain further down the road (for various reasons).

When you start with the Composer template, or something like Acquia's BLT, you have more flexibility in deploying your site, storing things in different places in your repository, and a more stable upgrade path.

See the documentation on Drupal.org for more info: Using Composer to manage Drupal site dependencies.

If I have a multisite installation, how can I put certain modules in one sites/site-a/modules directory, and others in another sites/site-b/modules directory?

You're in luck! Just like we can do with libraries (see the earlier question above), we can use the composer/installers package to define a specific directory per-dependency (as long as the dependencies—e.g. modules or themes—have a type assigned):

"extra": {
    "installer-paths": {
        "sites/example.com/modules/{$name}": ["drupal/module"]
    }
}

Using the command line to install modules is a lot to ask. Isn't there some sort of UI I could use?

You're in luck again! Matt Glaman built a fancy UI, Conductor, to use with Composer (he's asked for feedback on GitHub of how he can make it even better). Additionally, some IDEs have support for basic functionality (e.g. Composer PHP Support for Eclipse, or the Sublime Text Composer plugin).

And finally, There's a core issue for that™: Do not leave non-experts behind: do not require Composer unless a GUI is also included.

Just like with Git, though, it pays dividends to know how to use Composer on the command line.

What happens if there are dependencies required by two modules or core and a module, and there's a conflict with version requirements?

This... is a hard problem to solve. As Drupal has moved to a more semver-style release cadence (8.x.0 releases can include major new features, and could drop experimental modules), there are times when a core dependency and a contributed module or install profile dependency might conflict (e.g. core requires version 1.3.2 or later, and an install profile you're using requires version 1.2.9 but nothing later than 1.2).

In these cases, the best option is likely to get the non-core packages (e.g. contrib modules, profiles, themes, etc.) to align with Drupal core. When two different modules or other contrib libraries conflict, the best option is to work it out between the two modules in their issue queues, or to manually override one of the dependency definitions.

This is a hard problem to solve, and best practices here are still being discussed.

How can I include libraries and shared modules that I made for my own sites (they're not on Packagist)?

Many people have private modules or themes they use on multiple Drupal codebases, where they want to maintain the module or theme separately from the individual sites. In these situations, there are two options:

  1. You can add additional repositories in your composer.json file, one for each of these custom libraries, then you can composer require them just like any other dependency.
  2. If you have many of these dependencies, you could set up a private Packagist. This option is probably best for large organizations that have specific security requirements and a number of custom libraries and packages.

See related documentation in the Composer docs: Handling private packages.

I have a Drush make file for my site. Can I easily convert this to a composer.json file?

Well, there's the Drush makefile to composer.json converter tool, but other than that, it might be easiest to regenerate your site from scratch using something like the Composer template for Drupal projects.

Note that if you built your site by hand using .zip or .tar.gz downloads, you can use Drush to at least generate a makefile (see drush make-generate).

I set up my Drupal 8 site by downloading archives from Drupal.org and copying them into the codebase. How can I convert my codebase to use a composer.json file?

See the above question—you can't easily go straight from custom codebase to Composer-managed, but you can use drush make-generate to generate a Drush Make file; and from there you can convert to a Composer-based project.

What is the difference between ^ and ~ in version requirements, and should I use one over the other?

Put simply:

  • ~: ~8.3 won't go to 8.4.0
  • ^: ^8.3 will go to 8.4.x, 8.5.x, etc. to 8.9xxx

Both operators tell Composer 'use at least this version or higher', but the ~ says "stay within the same minor release (8.3.x)", while the ^ says "stay within the same major release (all releases up to, but not including, 9.0.0)".

There's some good discussion about standards in core here: Prefer carat over tilde in composer.json, and as with most other things, you should also read through the official Composer documentation for a more canonical answer.

As long as you trust a module or library's maintainers to follow semver standards and not break backwards compatibility within a major release, it's best to use the ^ to indicate required minimum versions (e.g. composer require drupal/honeypot ^1.0.0.

When I run composer update a lot of weird things happen and everything is updated, even if I don't want it to be updated. Is there a way I can just update one module at a time?

Yes! In fact, in real world usage, you'd rarely run composer update alone.

To just update one module or library at a time, you should run the command (e.g. for Honeypot):

composer update drupal/honeypot --with-dependencies

When you add those arguments, you are telling Composer:

  • Only update Honeypot
  • Check if Honeypot has any dependencies that also need updating
  • Don't update anything else!

This is generally safer than running composer update alone, which will look at all the modules, themes, etc. on your site and update them. Technically speaking, if we lived in an ideal world where patch and minor releases never broke anything, composer update would be perfectly safe, even if it updated every module on your site.

But we live in the real world, and despite the best efforts of module maintainers, many bugfix and patch releases end up breaking something. So better safe updating one thing at a time, then committing that update (so you can verify exactly where something went wrong in the case of failure).

When I run composer install on my production server, it runs out of memory. How can I avoid this issue?

Short answer: don't use Composer on your production server. Build a deployment artifact instead (on a non-production server), then deploy the artifact to production.

More realistic answer: If you have to use Composer in production (e.g. you deploy your codebase then run composer install as part of your deployment process), you might need to bump up your command line PHP memory_limit. See Composer's troubleshooting guide for memory limit errors.

Another realistic answer if you don't want to use CI or some other means of building your production codebase: commit vendor directories and all depdendencies to your codebase. This isn't necessarily the most technically pure way of doing things. But it's definitely not the worst thing you could do—it's akin to how must people managed Drupal sites before Drupal 8.

For security reasons, I need to verify that the code I'm running in production wasn't hacked or modified. How can I do this if I'm using Composer to manage my dependencies?

If you commit all your code to a Git repository (including dependencies), you can easily do a git diff on the production codebase to see if any files have been hacked. But if you're constantly using composer install to install dependencies, you need to be able to verify that you're getting the right code in two places:

  1. When you or anyone else runs composer install.
  2. When the code is deployed to production.

In the first case, you should always make sure you're using https endpoints; Composer will validate certificates when communicating with secure channels. If you use http repositories, there's no protection against man-in-the-middle attacks.

To verify your production code, the best solution is to not run composer commands on production. Instead, use a CI tool (like Travis CI, Jenkins, CircleCI, etc.) to build your codebase, then commit your codebase to an artifact repository (or a separate branch on your main repository), then deploy that code to production. That way, you preserve the ability to do a git diff on your production codebase.

I totally screwed up some of the dependencies on my site while I was tinkering with them. Is there any way to 'nuke' all the things in my codebase that Composer manages so I can do a fresh composer install and get them back into the correct state (short of deleting my entire codebase and re-cloning my site locally)?

This can be tricky—with a Drupal site, Composer might be managing code inside vendor/, modules/, libraries/, profiles/, themes/, and possibly other places too! The best way to 'nuke' all the dependencies Composer downloaded and start fresh with a composer install is to run:

git clean -fdx .

This command tells Git: force delete all files and directories in the current directory that aren't being tracked by Git, including files not tracked because of .gitignore (x).

Then run composer install to get back all the dependencies in their correct state.

My team constantly battles with composer.lock file merge conflicts. How can we avoid this painful experience?

The two strategies that were discussed at the BoF for avoiding merge conflicts are:

  • Have one dedicated dependency manager on the team—e.g. only one person/role who runs composer require *, composer remove *, composer update * etc. Typically this would be a senior or lead developer who is communicating with the other members of the team and can unblock them.
  • Have one dedicated 'sweeper'—a person whose job it is to deal with merge conflicts.

If push comes to shove, and you have to deal with a merge conflict (or if you're the dedicated conflict resolver), one tip for successfully merging is to accept all the composer.json changes, but clear out all the composer.lock changes that were conflicting, then run composer update none (or composer update lock... it's not obvious if there's a difference). This will update the lock file based on your current composer.json and make sure the file hashes and timestamps are all in sync.

Do you have any other tips for making Composer use easier and more delightful?

I'm glad you asked! I was thinking you'd keep asking the hard questions. Here are some of the other miscellaneous Composer tips that were mentioned during the DrupalCon BoF:

  • Don't run Composer in prod! (I think this was mentioned at least five times in the BoF). Instead, either commit vendor to codebase, or use CI to commit vendor and create a 'deployment artifact' that is deployed to production.
  • Use the Composer Merge Plugin to mix custom module composer.json files into one root repository.
  • Don't use composer create-project drupal/drupal when setting up Drupal. Use drupal-composer/drupal-project instead, as it follows more of the Composer best practices for managing a Drupal 8 codebase.
  • Don't run composer update unless you really know what you're doing. Use composer update [package] --with-dependencies instead.
  • Commit the composer.lock file for individual sites and deployed projects. But don't commit the .lock file to contributed modules or modules shared among multiple sites.

Conclusion

As I alluded to in this post's title, there are still some growing pains as the Drupal community more comprehensively adopts Composer and structured dependency management in general.

One thing that's clear: developers who need to include external libraries and code have a much easier time in Drupal 8 than they did in past versions.

But with that benefit comes some downside: site builders and sysadmins tasked with deploying Drupal 8 securely and efficiently will have to adapt to some new tools and techniques if they want to avoid dependency hell.

Comments

Thanks for such a wonderfully thorough and detailed write up! I failed to make it to the BOF, and this is incredibly helpful and fully sums up the state of affairs perfectly. I've got some internal meetings upcoming to talk about a potential tool that may help alleviate some of the complexities with constructing a site, particularly for the audience that considers itself "non-expert". (something like a gui for BLT, that abstracts away the build complexities with a built in set of 'best practices').

One thing that Im *really* curious about is the discussion of multisite, and how we can continue to support running multiple 'applications' off of one codebase. Its obvious that it's a fairly critical feature for many (https://www.drupal.org/node/2306013), but was perhaps an afterthought as to the impact our "composer off the island" was going to have.

As an aside, I met with Mattias Schrieber from the Typo3 project at Drupalcon, and they too are simultaneously experiencing much of the same composer growing pains that we are. Im really hoping we can collaborate on this to make a solution that can help us all.

Great post, thank you for this, it's a great overview.
I would add one tip, install prestissimo to speed up your installations since it brings concurrency when downloading the packages. Very handy since Composer tends to be slow.
https://github.com/hirak/prestissimo

Thank you so much for writing this article Jeff! It really outlines just how complex and difficult Composer can be to wrestle with. This is a big reason why some site builders are quite anxious about the proposed move towards making Composer a requirement for building Drupal 8 web sites.

Like you write, there is a discussion about it in "Do not leave non-experts behind: do not require Composer unless a GUI is also included" -- https://www.drupal.org/node/2845379

I really hope that the issue will be resolved, and that a more user friendly solution will be made available for non-experts, so that site builders can download a module, and that required libraries defined via Composer can be solved by Drupal, "behind the scene".

The last paragraph in this post by Mixologic gives me hope, as long as it will not be dependent on Acquias BLT project... https://www.drupal.org/node/2845379#comment-11983008

"Thats why Im advocating for something that is not just a gui version of composer, but something more specific to Drupal, the product, that hides all the work composer is doing from the end user. I.e. this tool shouldn't even have the word "composer" in its interface. We want to give the users a button that says "add this module to my site" and it would do everything necessary to ensure that they end up with all of the php, javascript, css, fonts, and any other assets required to start using the new feature they want. Sitebuilders do not really need to understand the differences between WHERE/GROUP BY/HAVING in sql because they have views and the field api that abstracts it away for them, so by the same token they shouldn't need to understand the differences between carets and tildes, the nuances between 'prefer stable' and 'minimum stability', or have to understand 'conflict' vs 'replace' vs 'provide'."

This is a really great and in-depth article! Don't you think it's ironic how composer has brought on some dependency issues where it is supposed to solve dependencies issues. Plus I always sometimes get confused in some situations about here I should use composer and where should I use Drush!

No, it's not ironic. Thing is that pretty much every framework already used a dependency manager, except for Drupal. So everything was done by hand. Or we used drush, which isn't really a dependency manager, because it didn't take versioning properly into account. Drush make made sure you pinned to an exact version (not a great way of doing things IMO), drush up just looked to download the latest version or a specific mentioned version.
Proper dependency management is hard, and as long semantic versioning isn't followed properly, library maintainers don't use it perfectly and update their libraries when new major versions come out, it will remain hard.

I think the arguments against running composer on production are compelling. What if you have a connection glitch and one of your libraries cannot be downloaded? Friends don't let friends run composer on production!

Let's give Acquia's competition fair credit. Platform.sh may have been the first to show how you can automate the build process, and Pantheon just released tools for an automated build/deploy process with GitHub and Circle CI: https://pantheon.io/docs/guides/github-pull-requests/#update-your-project

---

As we discussed at the BoF, "git clean -fd ." will not remove anything listed
in .gitignore. You need "git clean -fdx .".

In fact, you need to add an extra `f` so that git will remove .git directories as well - depending on your settings (prefer source vs. prefer dist) or the availability of the package, composer might install the dependency by checking out its source git repo. So the best bet would be `git clean -ffdx .`, but always be careful! You may be ignoring a file that is important, and, like `rm -rf`, there is no going back after running that command.

This write-up is great, and will be a great help to many people.
But I think the explanation about ~ in the article is a bit misleading. It doesn't let you stay on the same minor version, it let's you stay on the level above the one you define. If you use ~8.3 you will stay on the 8.x major version. If you use ~8.3.0 you stay on the 8.3 minor version. Maybe a link to the official documentation can also be handy in that place? (https://getcomposer.org/doc/articles/versions.md#tilde)

Ah, thanks! I'll update that section in a bit.

to work around the libraries issue (with modules requiring libraries being in /libraries vs /vendor) can't you use symlinks?

provided you are not using windows, git will version symlinks just fine

Agreed, a very useful resource, thank you.

I am still skeptical when I read core maintiners deny that D8 has abandoned small users such as clubs and churches. It is good to read of the Conductor GUI which apparently allows ordinary (i.e. non-CLI) users to install modules just by clicking around. I wonder whether it really is robust for handling installations, updates, and restoring backups, and whether it is worth including in Drupal VM.

Thanks very much for this write up. Could you consider putting anchors in at each headline to make linking to specific sections easier?

Good idea, will do in a little bit!

Just some clarification about this line:

Using the composer/installers package, you can set specific paths for certain types of packages, and you can even set specific directories per-package as long as the package has a type and requires composer/installers.

  • Technically all packages have a type; if it is not explicitly defined in the package's composer.json file then it defaults to the generic type "library"
  • In order to use composer/installers, the package must be a type that is supported by that installer (one of the types listed in the table at https://github.com/composer/installers#current-supported-package-types)
  • The package itself doesn't technically need require composer/installers (though if you are a publishing a package it's a good idea); if the root package (your project) requires composer/installers, it will still work (just noting that you're not out of luck if the package author neglects to require composer/installers but still defines a type that is supported by it)

Composer Installers Extender plugin and Asset Packagist are mentioned a few times. It should be noted that this approach is not without issues of it is own (https://github.com/oomphinc/composer-installers-extender/issues/6).

I hope these can be resolved. I think this combination has great promise regarding handling of frontend libraries for Drupal with Composer.

Thanks, Jeff. Here is another item to be addressed: Two days before DrupalCon the Superfish project made a commit that broke running composer require. Read about it here: https://www.drupal.org/node/2872222 I lost a full day at DrupalCon 2017 trying to learn how to work around this problem. And I went to some very knowledgable people. No one could help. While the Superfish problem was eventually fixed (after DrupalCon ended), the bottom line is that if you have installed a module with a broken packagist and you need to apply security updates, you're out of luck. That is the technical issue. But there is a community issue here as well: look at the workaround #21 on this issue, because it was written by a guy just like me who wasn't a composer guru but needed a fix. Why is is that he came up with the solution and none of the experts chimed in with a fix? Where do we go when we can't apply security updates because a project has introduced an error?

Users who have been using drush pm-updatecode need to know the effects of continuing to use it after beginning to use composer. For example, project address says we have to use composer: https://www.drupal.org/project/address But it doesn't say that if we subsequently use drush pm-updatecode then our project can no longer be updated with drush pm-updatecode. I understand that a project lead has every right to insist on using composer, but should they put out instructions that will knowingly break someone's installation if they are using an established, accepted process like drush pm-updatecode? Isn't it reasonable that users be warned about the ramifications? Here is an example of the frustration that stuff does: https://www.drupal.org/node/2730825 I've updated the Drupal documentation on this: https://www.drupal.org/docs/develop/using-composer/troubleshooting-composer - We need to make it clear that once one begins to use composer, they must commit to it.

Hi Mark, Feel free to share your thoughts in this issue, where the topic is discussed: "Do not leave non-experts behind: do not require Composer unless a GUI is also included" -- https://www.drupal.org/node/2845379

This amazing blog post should be migrated to Drupal.org Documentation! I mean that. Thanks for writing this Jeff.

Are you sure you're correct with tilde and caret operator? The docs say:
'The ~ operator is best explained by example: ~1.2 is equivalent to >=1.2 <2.0.0, while ~1.2.3 is equivalent to >=1.2.3 <1.3.0"

So ~8.3 will go to 8.4

I've been delving into composer's source code (to track down this issue: https://www.drupal.org/node/2893017 ), and checked out the different between composer update none, composer update nothing and composer update --lock:

  1. composer update --lock is explicitly supported & documented: https://getcomposer.org/doc/03-cli.md#update
  2. composer update nothing works because 'nothing' doesn't match any packages, and is even explicitly recognised, to avoid debug messages about that: https://github.com/composer/composer/blob/a3f2b7bb92a54f4e25b2d0998f00936ecac6f933/src/Composer/Installer.php#L1332
  3. composer update none also works because 'none' doesn't match any packages, but will probably still show a debug message in verbose output about there not being any matching packages.

So, basically, use composer update --lock :-)