Drupal 8 successes and failures

Thoughts about Drupal 8, Drupal 7, Backdrop, the Drupal Community, DrupalCon's meteoric price increases, DrupalCamps, and the future of the framework/CMS/enterprise experience engine that is Drupal have been bubbling up in the back of my mind for, well, years now.

I am almost always an optimist about the future, and Drupal 8 promised (and usually delivered) on many things:

  • Vastly improved content administration
  • Views in core, and even better than ever
  • Media in core
  • Layouts in core
  • Modern programming paradigms (fewer #DrupalWTFs)
  • 'Getting off the island' and becoming more of a normal PHP application (kinda the opposite of something like Wordpress)

But one thing that has always been annoying, and now is probably to the state of alarming, for some, is the fact that Drupal 8 adoption has still not hit a level of growth which will put it ahead of Drupal 7 adoption any time soon.

This fact has been beaten to death, so you can read more about that elsewhere, or see the current usage graph here and yes, not every site reports back to Drupal.org, so those numbers are not perfect... but it's the best data we have to work with.

So people are asking questions:

  • Should I stick with D7 LTS support for years and years, and re-platform on something else if I ever get the budget?
  • Should I upgrade to Drupal 8 now (or soon) since Drupal 8 to 9 will supposedly be a painless upgrade?
  • When will [insert critical module like Rules here] be ready for Drupal 8?
  • Is Drupal dead?
  • With 'ambitious digital experiences' being the new market Drupal targets, should I still build [insert any kind of non-enterprise type of website here] on it?

And the main driver for this post, in particular, was the following tweet by @webchick (Angie Byron), who coincidentally is probably the main reason I dove headfirst into the Drupal community many years ago after she mentored me through my first core patch (hi Angie!):

I won't answer all the questions above—there are a lot of nuances to each that I could not possibly answer in a blog post—but I do want to jot down a number of areas where I have seen pain (and usually experienced on my own) and which are still holding back widespread adoption of Drupal 8 by those who used to default to Drupal for 'all the things'.

Drupal 8 Failures

Caveat to those who read on—you may think I'm trying to disparage Drupal through the rest of this post. I'm not. I'm exposing the dark side of a major open source project's decision to radically re-architect it's core software on an entirely new foundation. It's helpful to know these things so we can figure out ways to avoid hitting all the same pain points in the future, and also as a sort of 'call to action' in case anyone reading this thinks they can push some initiative forward in one area or another (it's no coincidence I'm finishing this post on the flight to DrupalCon Seattle!).

The web has changed

So... a lot of people mention that because more people build custom Node.js-based single page apps using the MEAN stack, or now do hip and trendy 'full stack development', and Drupal is some old monolith, Drupal has been left in the dust. I don't buy that argument, because otherwise we'd see similar attrition in pretty much all the other PHP CMS communities... and we don't.

Sure, there are use cases where someone would consider either Drupal or a hip trendy decoupled web framework backend. But Drupal 8 is actually a really good choice for those who build decoupled architectures using JSON:API, or GraphQL, or whatever other fancy decoupled framework and need a reliable content backend. To be honest, though, it seems that those who do 'decoupled' with Drupal are often people who started with Drupal (or something like it, and then get into the decoupled game. And that's a pretty small slice of the market. Drupal is a hard sell if you have a team of non-PHP developers (whether they do Node, Ruby, Python, Go, or whatever) and are looking into decoupled or otherwise buzzwordy architectures.

Migrations instead of Updates for modules

With Drupal 4, 5, 6, and 7, modules could define upgrade paths from one major version to the next through Drupal's normal update.php mechanism, and while the entire update mechanism was a very Drupal-centric oddity, it worked. And most modules would, as part of the general upgrade process, write an update path so those using the Drupal 6 version would have all that module's configuration make its way to Drupal 7, as long as they were running the latest and greatest Drupal 6 module version when they upgraded.

Things became a bit harder in Drupal 8, because of two things:

  1. New API architecture often required full module rewrites.
  2. The traditional update.php process was abandoned for major version upgrades (see: Drupal 7 sites can no longer be upgraded to Drupal 8 with update.php).

And these two problems kind of fed into each other—not only did module authors had to often rewrite (or at least radically alter) large swaths of code to support the new Drupal 8 APIs, but they also had to scrap any hook_update() upgrade implementation they may have worked on once that change record was published. I'm not speaking in the hypothetical here; this is exactly what happened with the Honeypot module. In fact, I still have not had time to work on writing a module migration for Honeypot for Drupal 8, even though I had a fully working and tested upgrade path using the old update.php method years ago.

For a simpler module like Honeypot, this isn't a major issue, because site builders can reconfigure Honeypot pretty quickly. But for more complex modules that are not in core, like Rules, XML Sitemaps, Google Analytics, etc., there are a ton of configuration options, and without a reliable configuration upgrade path, site owners literally have to re-do all the original configuration work they did when they built out their current Drupal site. Even if the module has the exact same features and functionality. Sometimes one little checkbox buried in the third page of a module's settings is the difference between some site feature working correctly or not—and when you have hundreds or thousands of said checkboxes to check on the new site... the upgrade becomes a much more risky proposition.

There is an open issue in the Drupal project issue queue to help improve this situation, but we're far from the end game here: [Meta] Better support for D7 -> D8 contrib migrate.

Composer is still not a first-class citizen... but is often necessary

One of the largest 'get off the island' tasks the Drupal community checked off early was 'start using Composer for dependency management'. However, Drupal is... weird. You can't resolve two decades worth of architectural assumptions and dependency cruft in one major release.

Especially since until Drupal 8, it was not even really possible (except if you were willing to do some really wacky stuff) to manage a Drupal codebase using Composer.

I've spilled enough ink on these pages over the years over Composer and Drupal 8 (2019, 2018, 2018-2, 2017, 2017-2, 2017-3, etc.), and most of my older postings still highlight current problems in using Drupal and Composer together.

There's a massive initiative to make things better: [META] Improve Drupal's use of Composer. It will still take time, and maybe even cause a little more strife in the end, as some more old Drupalisms may need to be put to rest. But having one standard way to build and maintain Drupal codebases will be better in the end, because right now it can be quite messy, especially for those who downloaded Drupal as a tarball and use no CI system.

OOP and Symfony are the future

Along the same theme as the previous topic, Drupal rearchitected most of the foundational bits of code (the menu routing system, the HTTP request system, the Block system, the Entity system—pretty much everything except maybe Forms API) on top of Symfony, a very robust and widely-used PHP Framework.

But this meant that large swaths of Drupal experience were thrown out the window. A 'Block' was still a 'Block'... but the way they are built changed from weird-but-conventional Drupal hooks to using 'Plugins'. And menus used to be defined in a hook, but now there are sometimes multiple new YAML files you have to add to a module to get Drupal's menu system to pick up a new menu item—and you have to know how to wire up a menu item to a route, and what a route is, etc. In the past many of these things were kind of papered over by Drupal's simple-but-good-enough menu system, but now you have to be more formal about everything.

Speaking of which, a solid understanding of OOP-style programming is basically required in Drupal 8, whereas weekend hackers could kind of cobble together things in Drupal <= 7 using some hooks and copied-and-pasted code from Stack Exchange. Debugging—for those not used to a full-fledged debugger—is also a lot different. Simple print() statements or dsm() don't always cut it anymore. And debugging things on the frontend—well, I'll get to that soon.

The overarching issue is that the Drupal community was sold on the idea that moving Drupal to Symfony would pull in thousands of other PHP developers who would flock to Drupal once it started using a more modern code architecture. That promise really never panned out, as if anything, it seems harder to find solid senior-level Drupal engineers nowadays (at least in my experience—am I wrong here?).

Some Drupal developers who are not classically-trained (like me! I never took a comp sci class in my life) chose to expand their knowledge and grow with Drupal 8's new architecture. Others chose more familiar pastures and either moved on to some other PHP-based CMS or switched to some other ecosystem. I don't blame them, not at all; it's a tough decision you have to make to balance your career desires and opportunities, and everyone has to make their own decision.

In any case, the new architecture has more complexity than the old; and because of this, it's almost a necessity to adopt the following:

  • Use an IDE like PHPStorm, or lots of plugins with other editors, to be able to code efficiently across sometimes multiple files.
  • Integrate some sort of linting framework lest you hit deprecated code and weird syntax issues.
  • Have a CI/build process because a modern Drupal site can't usually be managed and run in one Git codebase and branch, checked out on a production server.

Themes have to be rebuilt

Along with all the other changes, Drupal's theme system was completely swapped out—it went from using the unholy monster that was PHPTemplate to a clean, new, standard system from Symfony, Twig. I'm one of the first to admit that this was probably one of the best and most necessary architecture changes in Drupal. The theme system was dangerous, messy, and difficult to work with on the best days.

But this is in many cases the straw that breaks the camel's back. In addition to the revamped architecture, new required build processes, and upgrade difficulties, almost every Drupal site has to completely rewrite its theme. And for many of the Drupal 7 sites I've built and worked on, this is probably where the majority of the effort would need to happen.

Try as we might (as a general web development community), the number of sites using a strict frontend design system where the design is decoupled from the theme itself, and can evolve and be migrated from one system to another, is vanishingly small. In 99% of the sites I've seen, very little of the frontend code from Drupal 7 could be quickly moved to Drupal 8. Maybe a few theme and form hooks, and a few CSS files, but the theme is usually very deep and complex, and most organizations use an upgrade as an opportunity to sink another chunk of money into refreshing their site's themes anyways.

But at least with Drupal 5 to 6 or 6 to 7, that was a choice, and you could upgrade the underlying system without also upgrading the theme. In Drupal 8, you kind of have to rebuild your theme or build an entirely new theme.

Custom code requires a comprehensive rewrite

I admit that I am guilty of running two Drupal 7 sites with a very large amount of custom code. Hosted Apache Solr and Server Check.in are both currently running on Drupal 7 (well, the frontend parts at least), and I have tens of thousands of lines of custom code which integrates with backend APIs (using things like Drupal's Entity API, Form API, Block API, Queue API, etc.).

Some of this code will not need to be rewritten completely (thankfully!), but there is enough that I will have to schedule a substantial chunk of time—which I could devote to features, bugfixes, or improving the platform in other ways—to upgrade to Drupal 8 (or 9) when the time comes.

There is always technical debt associated with custom code. And I always try to manage that by adding separate Behat tests which test the frontend functionality in as generic a way as possible (that way I can at least upgrade against a set of critical feature tests). But I'm not alone in facing this problem. Thousands of project managers (some of whom have precious little budget to work with) have to decide whether to allocate time and money to a Drupal site rebuild. The more custom code, the more difficult the decision.

Multisite is... interesting

One of the few huge differentiators between Drupal and most other CMSes has always been the ability to run 'multisite' installations. That is, you have one codebase, maybe even on one server, and you can run many Drupal websites (each with its own database, set of modules, unique files directory, theme, etc.).

Many multisite detractors are quick to point out that this is kind of an abomination and is architecturally impure. However, the site you're reading right now (assuming I haven't yet upgraded it to something else) is actually a multisite—I run six different Drupal 7 sites off one codebase, and there's no way I could've justified building each of these sites in Drupal at all if I wasn't able to build one build pipeline, one production server, and one development workflow that literally does all six sites. The dollar cost alone from running 1 Drupal production server to 6 prevents me from even considering it (most of these sites are maintained by me gratis).

There are a lot of massive Drupal multisite installations, especially in education and non-profits, where the cost benefit of not having to manage tens or hundreds (or in some cases thousands) of Drupal codebases, CI workflows, and many more production servers (since you can no longer share PHP's opcache between sites, besides some other things) necessitates multisite installation.

Here's the rub: Multisite architecture is kind of in conflict with some of the core ways Composer works. So trying to manage a modern Drupal 8 codebase with Composer and having the ability to have different copies/versions of different modules inside the codebase is... not quite impossible, but can be very close to that. Especially if you are not a Composer whiz.

Yes, yes, there are a thousand other arguments against multisite... but the fact is, there are a number of organizations—usually some of the orgs with hundreds or thousands of the sites that show up in the Drupal project usage statistics—who are holding off upgrading to Drupal 8 because multisite is harder, and the future of multisite is still fuzzy.

Conclusion

Drupal 8 was a radical re-architecture of a widely-used CMS platform. Many developers made their careers through the Drupal 6 and 7 development lifecycle, and were sideswiped by what happened when Drupal 8 was released. There's no doubt Drupal 8 has a great feature set, a thoroughly-tested core codebase, is excellent as a general site-building tool, and is primed for the building great (and 'ambitious') digital experiences.

But do I recommend Drupal 8 in all the same kinds of situations where I used to recommend Drupal 7 in the past? Definitely not.

Drupal 8 is a very different framework and platform than Drupal 6 or 7 was. There are some massive benefits, like the fact that it is easier to use modern programming paradigms, dependency management tools, and site architecture. And these benefits are massive for new site builds or migrations from outside the Drupal ecosystem into Drupal. But there are many tradeoffs for older Drupal sites; many users (and developers) have been left with a dilemma as they face re-building an entire site, in light of the fact that upgrades are more time-consuming and difficult than they had been in the past.

It may be noted that many of the more 'ambitious' Drupal 6 sites also needed a full migration to Drupal 7 and couldn't be directly upgraded—but for the long tail of smaller sites which usually used core modules and a smattering of contrib modules, and had little if any custom code, the upgrade.php process worked quite well, and resulted in hundreds of thousands of site upgrades that I don't believe we will see with Drupal 7 to Drupal 8.

Architecturally, almost every major change that resulted in the Drupal 8 we know and love (and sometimes shake our fists at!) is sound. But when taken as a whole, I do not begrudge the project managers who have to decide if and when to upgrade to Drupal 8—or sit tight on Drupal 7 LTS, or move to Backdrop, or re-platform to some other system.

I am still optimistic about Drupal's future, especially as the plan seems to be to not make such a massive set of architecture changes in a major version again, but instead to upgrade subsystems here and there through point releases. But I think the usage pattern and value proposition for Drupal has changed. I definitely think there are classes of websites that are more ideally situated on some other platform now, and I also think there will be a large set of organizations willing to stick it out on Drupal 7 LTS for as long as there is some form of commercial support available.

But I think the moral of Drupal's saga is if you revamp many major portions of an ecosystem's architecture in one release, you have to accept the attrition that comes with such a refactoring. The radical alternative is to kind of stick your head in the sand like Wordpress seems to be doing (with regard to modern best practices and the PHP community), but I'm not sure if I like that solution much, either 😋.

Comments

Try to think a bit more out of the box. The world moved on since Drupal 7. You can have a business page with Livestream and media on Facebook for free. Why pay for hosting, setup and development when you get 89% of the use case with social platforms?

You still need to own your digital identity; even if you have an identity on some other social services, you need a solid web presence that you own and maintain on your own domain to earn trust in the digital economy.

What are you talking about? Facebook is the place for identity, not a random url. :)

PS Look, this is not a contest of who can say smarter things. You dont need to convince me about digital identity. I am on Facebook and Linkedin. It never crossed my mind to put up a page with my CV or family pics.

I agree with 99% of this article. I really enjoy developing with Drupal - as we do a lot of enterprise type sites and Drupal let's us do whatever we want - but these painpoints are PAINful. We're doing decoupled work now precisely because the deeply nested cross-wiring between data management and front-end consumption is often a tangled mess and you often have to sacrifice what you want on the front-end to get what you need out of the back, or vice-versa. Decoupled keeps Drupal's job cleaner while still letting us leverage things it does much better than something like Wordpress (roles & permissions, elegant caching, integrations, etc). But as soon as Drupal is in charge of the whole thing, then everything slows down, everything relies on senior Drupal developers, and everything has to be rebuild in a few years. I'm very curious where it all goes. Thanks for your reflections!

I agree with most of what I read but it is true that drupal 8 lacks more documentation and many large modules such as tbmegamenu and others could not port to drupal 8 which is a pity

Interesting post and I agree with many of your points.

On the subject of comparing Drupal 7 and Drupal 8:

"But one thing that has always been annoying, and now is probably to the state of alarming, for some, is the fact that Drupal 8 adoption has still not hit a level of growth which will put it ahead of Drupal 7 adoption any time soon.

Some Drupal developers who are not classically-trained (like me! I never took a comp sci class in my life) chose to expand their knowledge and grow with Drupal 8's new architecture. Others chose more familiar pastures and either moved on to some other PHP-based CMS or switched to some other ecosystem.

But I think the usage pattern and value proposition for Drupal has changed."

I think that the last of these statements is very much a deliberate effort on the part of the Drupal head honchos, and perhaps necessary to better secure Drupal's long-term future. And the first two statements are, perhaps, necessary consequences of that change.

To elaborate: Drupal 7 felt somewhat similar to Wordpress in it's technology and value proposition. There were obviously big differences between them but none-the-less it felt to me like they were both competing for the small business website / brochure-ware market. Except that Drupal 7 would inevitably come out that competition worse off.

Drupal 8 has made a big leap in terms of the amount of knowledge a developer needs to work with it. But it's also effectively given up on competing for that small biz / brochure-ware / blogger website market in favour of gearing towards the requirements of larger and more complex websites.

So it's not surprising that the number of sites adopting Drupal 8 is not got to the level of Drupal 7, simply because Drupal 8 is not really the best solution to many of those small biz sites currently using Drupal 7. Truth be told, many of them would be best advised to migrate to Wordpress as it's less expensive to develop and maintain and provides a simpler admin experience out of the box.

Now that may sound like a bad thing but I think it's the right direction for Drupal to go in. It needs to play to it's strengths which means competing for the specific market which it can excel at rather than trying to be a "jack of all trades".

At the end of your post you mention Wordpress again in terms of "sticking it's head in the sand" with regards to modern best practices. Now, Wordpress is often cited for it's market dominance but like you I wander if there's a day of reckoning approaching WP. It's not really a great choice for an advanced complicated website (Drupal definitely wins out there). But at the other end of the scale it's increasingly having to compete with the likes of Wix and Squarespace. I think the raw numbers of websites using Wordpress may hide that difficult fact. And the relative lack of Drupal 8 websites may hide the positive fact that it's a small but more committed user-base.

Thank you for your thoughts, I fully agree on you. Few comments:
- If you compare D8 usage statistics on d.o. you find that total number of dev-versions used is almost 70k! I would assume these are CI machines or something similar. So adoption is even worse than it might seem from graph. See Google Trends to notice the drastic drop of interest over time.
- Lack of interest on D8 causes snowball effect. Documentation is sparse and outdated (e.g. Stackoverflow answers works with 8.0, but not with 8.6). Companies that provide commercial (micro)services do not bother to write integrations/tutorials/SDK's to Drupal themselves, but offer them for other languages/frameworks. For example many payment gateways in Drupal Commerce are outdated.
- Took me over 100 commits until I had managed to set-up and get e2e-tests to pass on my (not so huge/complex) Drupal 8 project on CircleCI. With recent node.js project I managed to accomplish same in first commit.
I really wish Drupal project to succeed.

Just a quick overview of multisite. In fact, I would argue that the use of Composer made that even better.

On D7 multisite mostly understand the same codebase, different DB and all hosted on the same server.

With D8 and Composer, you can write modules in separate git repos and pulling them with a composer. So you could built 10 same websites which share modules, but also option to host them either on the same server, or different. But codebase is the same.

On the current project, we are doing that. All custom modules have each own git repo, and we are using Satis (much faster than adding git repos to composer.json) to manage them as composer packages separately.
With that, we could share a codebase, but have multiple projects built. Projects can then be hosted on the same server (etc. different folders), or in my case HA applications which are separately hosted.

In fact, this is multisite without Drupal multisite feature as we know it from D7.

Jeff, thank you for taking the time to write this blog post! For what it's worth, my biggest hindrance to Drupal 8 adoption has been the "C" word (composer). Personally, I've never really run into a major dependency conflicts so I would have really liked to see more development happen with the Libraries API, which is still in -alpha1. https://www.drupal.org/project/libraries. Anyway, thank you so much for your continued contribution to the Drupal community.
All the best,

But thus far Drupal 8 is a nightmare to manage. Until Core is updated automatically via "composer update" it takes many wasted hours to update sites, especially when plugins and the core itself have bugs that give the unhelpful "website has encountered a problem" message. There were 4 core updates in the past month. Many of these were caused because modules were put into Core (like Views), and when they had a security issue, Core had to be updated. Symfony has a similar issue. Since my update procedure consists of replacing the Core and Vendor directories and running composer update, I do not see why the replacement steps cannot be put into the composer workflow.

Drupal 8 adoption has tanked because those pushing the things we're seeing have forced everyone to do things their way without making their ways, optional. For example, this whole Composer thing: are we really going to pretend that everyone is okay with being forced to use that just to install something like Drupal Commerce!? They shoehorned everyone into it, completely disregarding how it might be beneficial (or merely convenient and easier) to allow it as an optional component. I keep reading how Composer is the future this, how GIT is the future that, etc., etc. and yet, for every leap forward these things supposedly provide, the massive user experience regression they create is synonymous to taking a dozen leaps backwards especially when you experience that from just installing or upgrading something. These things being involved in the bigger maintenance equation should be trivial point-and-click operations, or better yet, be completely transparent to the entire set of processes involving how Drupal is used or maintained, and yet, despite being in a technologically advanced society where we can make cars drive themselves, allow refrigerators to access the internet, or make little speakers order us pizza from our favorite restaurants by speaking to it, we're consigning to this idea that the only way to use something like Drupal Commerce (or other Drupal modules) is only by issuing commands at a command line!? And we're going to chalk all that up to something we're told we should embrace!? What a joke!

Here's a memo to anyone in favor of those things: the whole point of a system like Drupal is to make doing things, easier. Between GIT, Composer, Drupal Console, and Drush; all these things should be optional. Period.

Look, Drupal's current trajectory isn't looking good and the Dries-itis it's suffering from, needs excised:

1.) Keep politics out of the overall equation. I could care less about the dyes someone used to paint their stupid hair with, who you're dating and whether that person is of the same sex... I don't care about someone's personal fetishes, the color of their skin, or bullshit gender identification. Just make Drupal work and with the fewest number of moronic inconveniences possible.

2.) Decouple Drupal usage from those encroaching things I mentioned above (i.e. - GIT, Composer, Drupal Console, and Drush). Drupal should work without all of them, right out of the box, no fuss, no muss.

Those 2 things alone will usher in the buy-in we enjoyed back in Drupal 7. It's baffling that something like this keeps being ignored but says everything about why Drupal is in the dire straights it's in.

I'm with you on all points. Too much dogma from the top. Dries got aquified.

It is not possible to keep all happy, small, medium and big consumers of Drupal. Also applicable for all roles - site builders, developers, etc..

If you remove Git, Composer, Drupal console, Drush, what you get then?

1. GIT in 2019 should not be a problem. Not for developers, but for anyone who is doing something on the web. This is not a Drupal thing, it is basics for everyone.

2. Composer - we could argue about issues, but again comparing with D7, biggest progress made. It is a problem for site builders, but why not learn something new? Composer is again not made up thing from Drupal but widely adopted tool in PHP community.

3. Drush - it is a Drupal thing from ages (Console is just making more tied to Drupal core). Really don't see an issue here? It is used on D7 also? No complaints there?

We can argue that a lot of new things are introduced in Drupal 8. Yes, new for Drupal community, not for PHP community. Drupal is a PHP based CMS. We should have much earlier start with the adoption of new things. We are pretty late here btw.

If you remove all of that stuff, you get CMS which will use masses for small sites, easy install, make few pages and that is that? Don't forget that bigger companies support Drupal with people and contribution in core and contrib modules/themes which would not exist if Drupal is made on way as you suggest while then non of this companies would use Drupal.

Actually, the one you answer to is perfectly right.

1. Git is and should still be a problem for MANY people.
It's a great tool, no doubt about that.
But it's also a tool that requires you to...
- Understand a whole lot of concepts with very unintuitive vocabulary if you want it to be any useful
- Have a hosting service that allows you to connect in ssh for git transactions.

2. Composer is even "worse".
Yeah, it's a very nice tool to manage dependencies. But it's another whole group of concepts to grasp, it's another tool that needs to be installed (or at least installable) on your host, and it's even more "technical" than Git because of the required formalism.

3. Drush
Well yet another thing that requires some flexibility from the provider.

Reality check for you: MANY MANY content creators over the world are NOT developers, NEITHER webdesigner. They are people that just want an easy way to structurate and publish data.
Drupal 7 was great for that because you could install and update with minor understanding of technical operations.
You could also drop the whole codebase everywhere and install it.
For sites that didn't have more than a dozen modules, this was perfect.

In other words, Drupal, in 7 era, is still in the camp of "something that mostly anyone can install and use" (even if obviously such lightly-savvy people will make everything with contrib themes and modules, and probably struggle for big updates). You could use it in a clean way, or taking some shortcuts, or in unclean way.
Drupal 8 slaps people in the face: "learn how to do it THE PROPER (read: "MY SINGLE") way, or scram".

For people that create project for a living, this is actually rather a good thing. For those just looking for a tool to facilitate the expression of their thoughts, it's a big middle-finger.

Worse, although that's a different topic, Drupal 8 is not even yet capable of the basic media management every other CMS out there can (hint: basic is not just "adding a file into a website and display it": it's "providing a clean way for an end-user to upload, edit, display and reuse images and videos"). IN 2019. That's another big reason to push away lots of users.

Thank you for taking time to share your thoughts on the moderate adoption of Drupal 8. The reality scares me...just thinking about abandoning Drupal >8 gives me knots in my stomach.

I am an old geezer with no formal code or Drupal training who cut his teeth on Drupal 5, 6 and 7. However, after building three sites in Drupal 8, I don't want to look back.

Although I could muddle my way through on earlier version of Drupal, they never felt quite like a complete CMS. After I finally got the partial hang of using Composer and Drush, I finally feel comfortable, confident and in control with Drupal 8.

I understand the pain of upgrading an existing site from >7 and would be at the same crossroads as you point out. I also understand the pain of waiting on modules to transition to 8.

With that said, I would definitely take the bitter pill and move as quickly as possible to Drupal 8. For me, it was well worth the challenge. Those who fail to move forward and fail take a step into the future with Drupal >8 are missing more than just being stuck in old technology. --just my humble untrained opinion.

This is a most helpful article that should be widely shared by the Drupal Community. You can only make things right by ackowledging what needs to be made right.

In particular I was drawn to the section "Themes have to be rebuilt" and your insight

"But this is in many cases the straw that breaks the camel's back. In addition to the revamped architecture, new required build processes, and upgrade difficulties, almost every Drupal site has to completely rewrite its theme. And for many of the Drupal 7 sites I've built and worked on, this is probably where the majority of the effort would need to happen."

Just last night (At the DrupalCon agency leaders dinner) this came up and specifically how do we move the 800k websites when the theming costs have risen so much on D8 v D7. We're now started working with various players in the ecosystem to launch a solution - A collaboration of various agencies, hosting companies and our site builder (www.cohesiondx.com which cuts the effort in theming Drupal sites by 80%)

We hope this will be one of a number of initiatives to fix this issue and keep those 800k businesses on Drupal, which in turn will keep the platform and its community healthy and growing.

Most of the comments say much of what I want to say, so I'll keep mine brief. Drupal 7 suited my needs for ambitious but smallish community and business websites, primarily because of a large stable of mature modules, an awesome way to model content and create custom content types, and of course Views. Drupal 8 has all of this, and could with some imagination be packaged in a way that would let me continue using it for the kind of sites I regularly build. But the obsession with The Enterprise is taking my Drupal away from me. The architecture of D8 is not in question, at least not for me. I view every significant architectural change in D8 as necessary, and in the long run, advantageous. I believe we could hide some of the complexity and have a Drupal 8 that still works for the likes of me. Right now however, I'm being forced to use the abomination that is Wordpress to build my latest project, and I hate it. I hate Wordpress. I've tried others such as October CMS and Grav CMS. I'm building my own site on Grav because it's sort of like a mini Drupal. I don't find October compelling at all. I think WP is headed for a reckoning because of it's antiquated code base, so if we could find a way to fit Drupal back into the small-but-ambitious market, it would be a win for all of us. Thanks for writing this article.

Jeff,
I've been a fan of you (and your view on Drupal) for several years. I started out as a complete newbie on Drupal. My sites I always build with Joomla. Drupal 8 was a complete new experience for me. Loved the structured way the CMS was build. Greatest problems? Simple things as how to publish my content on social media, the update process (love Joomla) and a few more. Drupal 7 had it all but. A complete chaos, works great if you are used to it. I worked for a renowned company for years. If I got ill, send someone else and the work continues.
My point? Drupal 7 was too nerdish. Drupal 8 is way better structured. And yes, the legacy ,knowledge and community should be considered. But building a site with a huge part of custom code? Not worthy an experienced programmer, I'm sorry.

It is a brave step. Every software supplier should have the balls to look critically at his/her product and pose the question: shall we keep on adding stuff to solve (design) errors or shall we do a lessons learned approach and start from scratch?
If 7 is stable, what is the issue? If you start new, start with 8. Didn't Microsoft (finally) had the courage to cut backwards support some years ago for (I think) Office? Good work Drupalteam!

Here are my two cents:

Take a look at the Stack Overflow Developer Survey Results 2019, more specifically "Most Loved, Dreaded, and Wanted Web Frameworks", you'll see that Drupal tops the most dreaded category. Ouch.
https://insights.stackoverflow.com/survey/2019#most-loved-dreaded-and-wa...

This should worry people in the Drupal community a lot. I don't work on a Drupal project at the moment - or perhaps even any more - all (bar one) the sites I was responsible for in my current company have been migrated over to other platforms. The last instance was handed over to another company, it's still alive, and it caused me so much grey hair that I don't want to think about it.

The weirdest thing for me is that Drupal is still based on patches instead of pull requests. This might be fixed this year, 10 years or so too late. It would be a fun and devastating exercise to count the amount of time that has been wasted because of the patch based workflow.

Here are a few examples that also explain some of my own reasons for frustration, some issues just don't get fixed.

#1349080 node_access filters out accessible nodes when node is left joined
Reported in December 2011, first patch was submitted in July 2012, still not fixed. At least it got the RTBC stamp last week!

#1064882 Remove syscalls from calling date_default_timezone_set(), date_default_timezone_get()
Reported in February 2011, first patch came on the same day, still not fixed

#1679344 Race condition in node_save() when not using DB for cache_field
Reported in August 2012, first patch was submitted in November 2012, still not fixed

#1329742 Autocomplete with tagging silently discards invalid input
Reported in November 2011, first patch was submitted in January 2014, still not fixed

#2564307 [meta] Remaining Drupal 8 PostgreSQL issues
Well I do feel sorry for people trying to use Drupal together with PostgreSQL.

I, too, am still optimistic about Drupal's future. However, right now the increase in knowledge required to make a modern Drupal 8 site is huge. Part of the new knowledge is simply that making websites is getting more complicated and uses more and more varied technologies. Part of the new knowledge is that Drupal is "growing up" after a long period of adolescence. To be honest, Drupal 7 needed the overhaul that became Drupal 8.

Given that the Drupal community wasn't really keeping up to date with the rest of the industry, we should expect some frustration now as people are "forced" to learn a lot and in a relatively short about of time i.e. to make a new Drupal 8 site that the client expects within a few months. That said, I also know people who left Drupal because they didn't want to learn all this new knowledge.

Can Drupal be made easier? For sure. We've got complex OSs on our phones that get upgraded usually smoothly. True, our phones don't have the same issues with templates and such but it does indicate that these general upgrade problems are being resolved.

Profiles/distributions are one way to lower the amount of time and effort required to put up a site. I'm using one currently for a project and there is no way I would have the budget to put in the customization the profile gives me.

So, for me, the pain has moved from the site itself to the deployment workflow. Long gone are the days of downloading a module and FTPing it to the site. Even adding a module to the composer.json file and running `composer update` isn't enough in some cases because a build process is often required for some front end theming technologies. Going one step further, does one want to store just the custom code in the repo and use a build process to build out the artifact? (Build out an artifact == a build process in which Composer gets Drupal core, contributed modules, other libraries and custom code/theme and writes the full web site to a release branch in the repo.) If so, get ready for one team member to be the designated DevOps person because now complexity goes up a fair amount. Want to deploy Docker containers? Add yet another level of complexity.

To mitigate this, it will be a combination of the initiative that is teaching composer to update Drupal more easily plus hosting companies continuing to offer "out-of-the-box" configurations that help developers with the above. We are already starting to see "standard" deployment configurations in the bigger hosting companies and I'm sure we will see more.

It's 2019 as I write this...I don't see the deployment complexity being significantly resolved for at least another three years. There are just so many pieces to organize. And developers and site builders will still need to know these concepts and technologies for when things break.

My approach is to embrace the change and have fun learning all the new things there are to learn.

I could have predicted these struggles for Drupal 8 since computing history has shown that you loose anytime you take a programming platform popular in large part because it has a relatively low barrier to entry, and then evolve it to require more skill to use.

Oh, actually, I did predict it. Five (5) years ago, back in 2014:

https://mikeschinkel.com/2014/the-decline-of-drupal-or-how-to-fix-drupal-8/

tldr;?

First you lost people who liked Drupal because it was simple and did not want to have to learn how to become more advanced programmers, and secondly you'll never gain the people who are advanced because they already have better options in the advanced arena that have none of the baggage of negative opinion about prior versions of the platform.

Drupal 8 should have become a new project with a new name and Drupal per se should have stayed it's course. That would have given the the Drupal 8 cum Whatever a better chance for success. As it is, Drupal 8 down as is "too" good for most existing Drupal users, and not good enough for more advanced developers because the Drupal name drags its does for those who would never consider using Drupal.

#jmtcw

Jeff's excellent post and many excellent comments here (and elsewhere) tells us that most of us Drupal 'old hands' have mixed, even confused, feelings.

Composer, like Drupal 8, just works. Except when it just doesn't. Like Drupal 8, when it is off-colour, it outputs often opaque error messages which can be found in many questions and few answers on Stack Exchange.

A few years ago frontend developers realized that in order to write CSS in a *professional* way they needed tools with names like Susy, Sass, Less, Compass, Node, Bower, Gulp, Grunt, Gurgle, Swallow and Git (to exaggerate only slightly), proving that themers are real developers too. Then one leading Drupal frontender had the nerve to stand up and say it might be quicker just to write CSS. For backenders, there is no way back.

Does D8 server my clients better and give better value than D7? No. Views, Media and Page Builder could have been added to D7 core. At a certain very high budget level D8 probably does give better value than D8. Cui bono?

From Drupal Business Survey 2018:

‘We have been working with Drupal 8 since beginning of 2016. Since our clients mostly fit in the small business category, we have struggled to push our project budgets high enough to be profitable on Drupal 8 projects, as we were on Drupal 7 projects. It's not easy to say what all the reasons are, but Composer is finicky..security updates are more hassle because of more changes..’
(source: https://www.drupal.org/association/blog/drupal-business-survey-2018)

SOLUTIONS?

Enough complaining about the uncertainties I, like others, feel. What about some solid proposals to ameliorate the problems? Here are seven heresies which could lead to Drupal nirvana, if anyone dares whisper them:
1. Make composer optional everywhere. Really! Encourage module developers to provide instructions for manually obtaining libraries. Reinstate drush dl [project] and drush pm-update.
2. Reinstate drush pm-disable and drush pm-enable by automatically exporting and importing just the relevant config for a given module or theme.
3. Improved debugging. Create a tool--not sure how--so that var_dump() and similar simple debugging works at least as reliably as it does in D7 anywhere in code. And get rid of the idiotic 'the website encountered an unexpected error' message for something more useful.
4. Modular core, so that users who do not need the JSON API, or media library, or WYSIWYG integration for everything, do not need to download all those tools, and do not need to apply updates every time a security hole is found in a sub-system they do not use.
5. In the context of modular core, only include in vendor/ folder the required items, and anyway thin out vendor/ as far as possible.
6. I have never seen anyone evaluate how much unit and simpletest coverage is contributing to Drupal. It is doing some good and plenty harm. Test coverage is a theological doctrine imbibed by people who learned C++ or Java at university. Especially in contrib, the expectation for tests should be relaxed. In the current culture many module maintainers decline to commit very necessary patches for want of test coverage, which is preventing essential features and modules getting stable releases.
7. Prioritize improving documentation. For once, please, let us not hear a Driesnote with (self-)congratulations about the wonderful new features. Documentation is missed because it is less necessary for the big Drupal shops whose devs built this stuff, and can find the comments in the code if they need to. They underestimate how far its neglect is harming the wider ecosystem. A Driesnote which states that in the last year there was no single new feature in the past 12 months, and massively improved documentation, would do more for Drupal than the world's snazziest page builder.

Great article.
I switched to Larvel and don't regret it. I rebuild websites faster with Laravel than D8.