Fork Yeah! Examining open source history after Red Hat's move

We're at the stage in the Red Hat drama where everyone is consulting history, trying to figure out what parts are being repeated in 2023 after Red Hat effectively locked down the sources used to build RHEL clones.

One talk linked quite often was Fork Yeah! The Rise and Development of illumos, by Bryan Cantrill over a decade ago. Bryan was a software engineer at Sun, who went over to Oracle after the buyout, then left to join Joyent, and now resides as CTO of Oxide.

The talk focuses on Sun Microsystem's handling of Solaris and OpenSolaris, both before and after their Oracle acquisition, and the whole talk is worth a listen—so much context about the history of ZFS, Solaris, Illumos, dtrace, and even UNIX and Linux history are contained within.

But there was one section (around the 32:00 mark) where if you substitute "Red Hat" for "Sun," rhymes with this year's "open source company" drama:

I went back and looked at some of the mail trails about this and like, "oh, my God!"

It's like you look at the number of words spent on this—words from bright people that can actually write code instead of writing this, and they're writing just this—huge screens and so on, and it was just sad. Very, very sad, and a huge waste of time.

And it was entirely avoidable on the one hand, understandable on the other, because those strings were just too tempting for Sun to pull.

And sadly, what this ultimately ended up doing is really deflating the community because it was clear Sun couldn't tolerate an independent OGB. And I don't think this is necessarily a problem with Sun.

I actually think that Sun's intentions were as good as a company's could possibly be. But it's just a company has it's own interests.

And it's very hard, when that company has created this open-source community from scratch, populated it effectively from scratch, it's very hard for it to not think of that community as belonging to it.

And that's not the way the community thinks. So this is really a challenge you have when you're leaving this proprietary chasm.

But the community was very deflated by this.

I was especially saddened the way Red Hat leadership—and even in some interpretations, long-time veterans of Linux and the open source movement—has positioned the downstream "community" of those who build and run distributions like AlmaLinux and Rocky Linux. I won't deal with Oracle here because they are such a different entity and—well, listen to the video linked earlier starting around 33 minutes for a good screed against Oracle!

But when I read the following line in Red Hat's blog post:

Simply rebuilding code, without adding value or changing it in any way, represents a real threat to open source companies everywhere. This is a real threat to open source, and one that has the potential to revert open source back into a hobbyist- and hackers-only activity.

I was especially crushed. Bryan's speech especially highlighted the importance of hobbyists and hackers to a healthy OS distribution. Projects like ZFS, dtrace, vim, et all are often the brainchild of one or two developers, and would never happen based solely on bureaucratic development practices of a giant software corporation.

Many said Red Hat's communication is the problem, not the decisions themselves. I'd agree to a small extent, but with the caveat that the timing has been excruciatingly poor, considering the RHEL release cycles and community expectations.

Mike McGrath pointed out Red Hat never promised 10 years of support for CentOS 8 in a recent LinkedIn post:

Red Hat never said CentOS8 would be supported for 10 years and were careful not to. We should have made it clear that we were in negotiations with the board at that time about its lifecycle and that was the major folly. A community member, not knowing better updated the wiki to say 8 would last for 10 years because he simply thought that's what it would be (which was an entirely reasonable thing to do).

In this case, it sounds like the intention was to drop CentOS 8 but not all internally were willing, so it got released, then nerfed a couple years in.

Then, after promising sanitized RHEL git sources on, that was revoked a couple years into the RHEL 9 release cycle—and this time, the change was implemented behind the scenes a couple weeks before it was announced publicly.

Regardless of the worth of the downstream 'rebuilder' communities, the fact that an open source project's community has expectations that are continually unmet—whether or not they pay Red Hat—has a chilling effect on non-paying users of any software projects touched by Red Hat.

That's what saddens me. Just like the downfall of Sun, and the subsequent lawn-mowering of their open source assets by Oracle, I can't help but draw some parallels with the viewpoint that GPL licensed projects at Red Hat are viewed as being compliant if they upstream development, but chill the sharing of downstream source code with restrictive service agreements.

It is only RHEL today, but this pernicious behavior towards downstream community doesn't leave a good taste in the mouth. And the response from many Red Hat employees was not one of compassion and reconciliation, but one of vitriol and defensiveness.

Community projects like Alma and Rocky Linux have gone their different ways, but in hindsight, both downstream communities have proven (at least to me) the value of the individual contributors within—from submitting patches to Stream, to maintaining EPEL repositories, to promoting and contributing to Fedora, there is a lot of value pointed directly into Red Hat's own projects from downstream rebuilder communities.

It saddens me that the first instinct within Red Hat is to defend the business concerns, while barely listening to community concerns. Red Hat must make a profit—and indeed, they just increased revenue 11% quarter-over-quarter (one of the few bright spots on IBM's balance sheet this year).

But it's especially saddening to see the extreme defense of the business after that 11% increase, the quarter after the first mass layoff at Red Hat in over a decade, when 4% of staff were fired.


Seems like IBM is going strangle RHEL until it is a small hallowed out shell with diminishing returns, they will probably then sell it all to Oracle. Funny how the SCO vs IBM lawsuit seems to be left behind in favor of profits. Oh well - I guess there is still Debian and it's derivatives.

Good. Let it die and let the talented community friendly employees at Red Hat be picked up by other companies and projects where they can continue to do great things. And by the same token, hopefully the not-so-community-friendly employees at Red Hat will wake up to the fact that without the Linux community, they wouldn't have a job. I'm not holding my breath on that latter part though.

> Projects like ZFS, dtrace, vim, et all are often the brainchild of one or two developers, and would never happen based solely on bureaucratic development practices of a giant software corporation.

But... ZFS and dtrace were created by Sun and proprietary for years before being open sourced with dedicated teams focused on building them both before and after publication. So it seems to me that in fact it could and did happen based "solely on bureaucratic development practices of a giant software corporation." And by all accounts they accomplished more in those few years than I think any purely OSS filesystem over their whole lifetimes. Some of the most popular and advanced OSS filesystems started commercial or at least developed by companies. XFS, JFS, ReiserFS.

vim was nothing like that.

My point there was the nature of the project, being the brainchild of typically one, or two developers. Corporations like Red Hat and Sun believe the myth that somehow Red Hat or Sun are the reason projects like that come to fruition, but fail to realize the talent that has to be fostered to get those projects to happen.

Organizations can be set up to build these things, or not, but it is not the organization that builds them, it's the individuals within. And once you start a brain drain, innovation dies a slow death. And for companies like Red Hat, where a lot of their momentum is built on the open source movement, community is key to innovation. Starve portions of the community, and innovation suffers. That's where I'm most upset—even if it's the best business move, it doesn't seem the community fallout was considered—or worse, it was not deemed important in the case of RHEL.

Red Hat has had many rounds of layoffs over its 30 year existence. Perhaps you can correct that in your article.

Regarding mass layoffs (e.g. a significant percentage of the entire company), from my research, I hadn't found one in at least the past decade. I have only scanned two news archives and may have missed any other major layoff event, though.

Do you have any evidence of other mass layoffs in Red Hat's history?

I'm only asking because a (now former) employee who was at Red Hat for much of the past decade said this was the first mass layoff—maybe they mean in the time they were there, or maybe since IBM bought Red Hat?

But as someone who consulted for Red Hat for a couple years, and has followed the company fairly closely since Ansible's acquisition in 2015, I don't recall any layoffs affecting more than a team or two in that time period, at least...

I've updated the post—apparently there was a mass layoff back in 2001 after the dotcom bubble burst; I was only searching back to 2005!

Jeff - you need to ease up on the whining please. Somebody has to pay for their labor. You don't have to like it. Just vote with your empty wallet and shop elsewhere for free totally free meet my unpaid expectations immediately.

Somebody has to pay for my labor making these cracked Windows 11 Enterprise installers too; my customers love my work on blocking telemetry (although I keep telling them "bad data is much worse than no data").

I love being able to use the term "begging the question" correctly.