Removing official support for Red Hat Enterprise Linux

For all of my open source projects, effective immediately, I am no longer going to maintain 'official' support for Red Hat Enterprise Linux.

I will still support users of CentOS Stream, Rocky Linux, and Alma Linux, as I am able to test against those targets.

Support will be 'best effort', and if you mention you are using my work on Red Hat Enterprise Linux, I will close your bug/feature/support request as 'not reproducible', since doing so would require I jump through artificial barriers Red Hat has erected to prevent the use of their Linux distribution by the wider community.

For more of my reasoning, see my previous blog post: Dear Red Hat: Are you dumb?.

This decision will not change until and unless I see evidence Red Hat cares about giving free and open access to the sources required to build and test against their Linux distribution.

Process

The timeline for this transition to not supporting RHEL is as follows:

Today: Removing official support for RHEL in Ansible role metadata, so users searching for roles to work with RHEL will not find my roles (I would rather they find roles that are actually tested against RHEL, because I cannot guarantee my roles will work for these users).

Ongoing: As issues crop up on various roles against RHEL, I will decide on a case-by-case basis whether to strip all RHEL-like support (effectively making the project only run on Fedora, Arch, Ubuntu, Debian, or other distros), or attempting to fix the problem so existing users of Rocky Linux, AlmaLinux, and CentOS Stream may still benefit from the fix.

Further reading

red hat
linux
open source

Comments

but they’ve already had you jump through barriers since the last announcement, no?

Yes, but here's the problem:

After the CentOS Stream announcement, anyone using CentOS kinda scrambled for a while until Rocky Linux and AlmaLinux created a new status quo. This was mid-CentOS 8 transition, so half of us abandoned ship and went to BSD/Debian/etc., the other half waited then adopted Rocky/Alma.

Then we were like "wow that was insane, Red Hat wouldn't be crazy enough to burn the rest of the community, would they?"

Further, "if Red Hat is dumb enough to do that... surely they wouldn't do it mid-release again, no?"

And now they have. I'm done with their shenanigans. I'm using Debian on all but one of my servers now, and I will not be adding RHEL support to any new projects moving forward. I will still be supporting Fedora, which means some things might work on RHEL, but I'm not going to waste an ounce of time verifying it or validating bug reports from RHEL users.

Yep, we were in the second group and decided to adopt Alma Linux on the systems shipped to our customers. We started shipping out Alma Linux 9-based systems for new installations a couple months ago, but we still have a LOT of older installations running CentOS 7 (that are still in production) and we wanted see how will this whole CentOS Stream thing unfold until support for CentOS 7 ends. If the Alma and Rocky Linux projects were to fold because of this new move then we'll have to move to Debian or Ubuntu, but I feel like they're a much bigger PITA maintenance-wise (and their package-building process is also a lot more unpleasant to use than rpmbuild TBH). Obviously upstream distros like Fedora, CentOS Stream et al. are not an option for server installations.

openSuSE is comparable to Fedora, not Red Hat, Alma, or CentOS. If you want to use their enterprise distribution, it would be SuSE Linux Enterprise Server which as far as I'm aware has been always been as locked down as RHEL is apparently going to be.

It's not exactly that way:
- openSUSE Leap is 1:1 binary compatible with SUSE Linux Enterprise Server (SLES) since the release of version 15.
- openSUSE Tumbleweed is the upstream of both Leap and SLES and, yes, in that regard, it's similar to Fedora.

In the SUSE ecosystem there's nothing like CentOS Stream as openSUSE Leap and SLES do work like the old CentOS model (1:1 binary clone). You don't need a CentOS. If you want to clone SLES you just pick Leap and you are good to go (and, in fact SUSE provides the Open Build Services so they not just provide the source but all the build facilities for you to build your very own clon or derivated distro).

Hope it helps.

That's a common misconception. openSUSE *Tumbleweed* is sort-of comparable to Fedora. The stable openSUSE *Leap* releases are 1-to-1 binary compatible with SUSE Linux Enterprise Server (for example, openSUSE Leap 15.5 == SUSE Linux Enterprise 15sp5). SUSE has gone the opposite direction on this as RH. SLES sources are not locked down in any way.

Oracle is famous for throwing their lawyers at you, costing you a ton of money for all sorts of things you were not even remotely aware you had to pay for. That's why many users try to avoid Oracle Linux.

"support" means different things to many people. In this case, if OS=rhel do: close ticket. This is fully understandable, as the issue might be RHEL specific, with no ability to test/debug the issue.

Rocky and Alma, will in their words basically: 'strive' to be 1:1 RHEL clones, but now, that is essentially impossible to do, or verify. So if OS=rocky, ok, cool, spin up rocky, and see if the bug is real, and try to fix it. Simple.

Maybe some smart a is using RHEL, and it happens that at that point Rocky is 1:1 compatible, they might submit it and pretend its on rocky, and if they're lucky, it will also fix an issue in RHEL. But no way to know for sure as someone that isn't going to sign some creepy EULA/NDA thing about how source code is to be used, and expose yourself to RHEL's group of corrupt lawyers and decision makers.

I've put a hold on our pending POs for RHEL licenses and have reached out to Canonical and SUSE for quotes. It's not going to be easy migrating but I can't just continue to give them money after pulling crap like this.

Red Hat's isn't any better. I know, as I used to work for them. Clients would spend weeks engaging with support and finally get frustrated enough to complain to their sales team. Who would then get me on a call with them and I would fix it in 15 minutes. Happened way too often for it to be a fluke.

SUSE does exactly the same thing for sources. In fact, AFAICR in order to get the sources you had to send an e-mail and they would send you a DVD and charge you for it.

It's not backported. Leap and SLES used to share the same source packages (but here's the key: Leap carried a few more, and enabled some different build flags) and since a couple of years ago, Leap and SLES use the same binary packages (built by SLE on the IBS -internal-, not OBS -public-). But source with all the flags is still hard to get for SLES, unless you pull it from Leap, which would be exactly the same thing Red Hat just did:
- if you want the exact source packages used to build RHEL/SLES, only customers can get them
- otherwise, you get the source packages from the upstream of SLES (openSUSE Leap) or the upstream of RHEL (CentOS Stream)

If you pull the packages from openSUSE LEAP you get the same exact binary packages and source packages as you may get from SLES. LEAP and SLES are like what CentOS and RHEL used to be, not what CentOS Stream and RHEL are now.
That model was introduced back in 2018 and one of the objectives was to remove any barrier to access the SLES source code with all its HW, SW and security hardening/certifications and offer a 1:1 clone of SLES that the community can easily use. Not only all the code is there in the Open Build Service (OBS), the community can also use OBS' free build infrastructure to create their very own 1:1 clones of SLES or LEAP or build custom derived distributions.

My thanks also for taking a stand. I doubt this is "Dumb" as noted in your previous article, it feels more like blind greed on RedHat's part.

What about using the developer subscription they offer for these use cases?

The developer account terms could change in the future. Offering RH subscriptions for access to individual products would provide revenue.

I wonder if Red hat being an IBM company has something to do with this debacle?

Sorry Jeff but you are wrong this time.

You can reproduce with a free RHEL developer subscription. If you don't want to support RHEL because of some tantrum, just state that clearly.
https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-…

You do not need the RHEL sources to support an Ansible role but in case you want them, you can still get them: just download the sources from the Red Hat Customer Portal, which you can access with your free developer subscription.

That being said, I honestly think Red Hat screwed big time by killing CentOS as a free RHEL clone. Red Hat had something (CentOS) which was fully under their control, something that was slightly worse than RHEL (eg no errata metadata, always some delay vs RHEL, etc). By killing it and at the same time still offering the RHEL sources to anyone at git.centos.org, Red Hat opened the gates for a myriad of RHEL clones that are actually making quite some money in a very explicit, even obscene, free-loading way. Red Hat should have either kept CentOS as the free RHEL clone under their control, or killed it and closed the sources at the same time so that new clones (Alma, Rocky, etc) were never started.

By the way, this has nothing to do with IBM.

I used to work at a corporation too. And unless you were the one who has made this decision (in which case Hell has a special place for people like you) I highly doubt that they told you (or anyone really) what was behind this decision. They never EVER tell that to anyone at a corporation.

In my humble opinion, this situation has everything to do with IBM. "JD" simply lacks the importance required to be well-informed.

After the disastrous experiment with the "Partner Led" Pods (also known as Project "Assign all the top Red Hat accounts to IBM Account Managers with RH Technical Sales providing them support"), it has been discovered that unlike Red Hat's Account Executives who are currently being forced out, IBM Account Managers have no idea how to sell Red Hat's products or how to "compete with free." These past two quarters have made that abundantly clear. Since RH wants to transition the entire NA Sales Force to this new model in the coming quarters, they are scrambling to find ways for IBM's incompetent Sales Force to more easily make sales and achieve some quick victories out of the gate. The simplest option was to eliminate or at least cast doubt on the competition (Oracle and "free").

The global slump isn't helping matters either. Companies are significantly reducing IT spending and delaying migrations or expansions. Meanwhile, just as Red Hat has rushed head first to fully embrace the cloud (I have already obtained my meaningless mandatory certs!), companies are retracting from the cloud as they realize its high costs. This disrupts all of Red Hat's plans and leads them (like most companies) to lay off excellent personnel, greatly reduce spending on everything, stop providing deep discounts, and search for quick revenue streams in the short term to satisfy the IBM overlords. As of now, our leaders have only one strategy left: cost-cutting to become more like IBM. We no longer innovate, and we haven't since Whiteside left.

The Red Hat I joined over twelve years ago bears no resemblance to the shell it has become today (farewell memo list!). I now find myself attending meeting after meeting, listening to our leaders drone on about the importance of the "Rule of 40". Everything we do is now driven by whether it will increase revenue or not and nothing else. Will this fix help a specific customer? Yes! Will it drive more revenue? No. Then scrap it.

We have become nothing but a profit-driven machine for IBM. This cannot continue indefinitely, as we can't keep squeezing the same stones forever, as we are also squeezing our own people as we do it. We have been squeezing too hard already as I receive email after email every day of fellow Hatters, ones with massive amounts of experience and technical knowledge whom we can't afford to lose, who are heading off to ships that aren't sinking. In truth, I would have probably have followed them already if it wasn't for the need for excellent health insurance (Cigna isn't great but having multiple kids means you have to choose your next job with care).

So yes, in rebuttal to "JD", this has everything to do with IBM.

Great conspiracy theory. But reality is a lot simpler than that. Someone at RH made a huge mistake* by killing CentOS-as-a-RHEL-clone, then a huge market for out-of-RH's-control clones started, and that's making RH uneasy because it's blatant freeloading.

Yeah JD, keep calling your own customers and those who literally sell servers with RHEL/CentOS pre-loaded on them "freeloaders", what could possibly go wrong /s

Consider the effect of the developer subscription license conditions for something as basic as a continuous integration and testing. That CI system has to be engineered to build no more than 16 simultaneous docker-enclosed tests for the RHEL developer subscription. Then once a year that CI system will throw its hands up and all RHEL tests will fail, until the license is renewed. Easier that the CI system run, say, Rocky Linux.

The idea that RHEL users should get the same level of support as other distributions is undermined by Red Hat's developer subscription terms which make Red Hat harder to support than other distributions. Consider the ease of offering support for Debian versus RHEL if you discover a fault with the operating system: Debian will take that bug report, but Red Hat developer subscription "does not include support on any operating system-related issue."

Claiming "free loading" isn't helpful, since free software is basically a virtuous circle of freeloaders. Red Hat Enterprise Linux includes vast amounts of software it obtains by "free loading" off the efforts of others. In most cases Red Hat didn't even ask those packages if they'd like to be included in RHEL, Red Hat just took a legalistic view of the copyright license and included the application. Red Hat can hardly make a moral argument when people do the same to its efforts.

There's the ROSI - Red Hat Enterprise Linux for Open Source Infrastructure subscription for that (google it, link is not allowed by the blog).

Enhancements are being made to that too.

Great! Set up the subscription for all of us and write the pipeline integrations. When you're done submit a PR and I'm sure Jeff will take a look at reintegration. Until then the extra effort isn't worth it because it was already made and then invalidated with this change.

Red Hat has explicitly stated that this license does not apply to individual developers of open source tooling.

From Red Hat's blog post:

> RHEL for Open Source Infrastructure is not intended for individual developers, current Red Hat customers/partners, governmental organizations, healthcare organizations, academic institutions or non-profits that want to use RHEL outside of independent open source project infrastructure.

Oh so should I wait around for that and pray they don't alter the deal further or should I get off my and abandon their problems they want to inject into my days?

This isn't a viable solution, though. To get no-cost access to RHEL, you still must agree to Redhat's enterprise agreement, which has the clauses trying to negate one's rights under the GPL and LGPL. I wouldn't agree to it and you shouldn't either.

While I hope you choose to continue to support RHEL-like distros I respect the decision and I hope it gives this fight teeth. Thanks for your incredible support of the Ansible community. If IBM gets any bolder you might have to switch to Salt or Puppet…

You caused me a scare there, but I just checked and Ansible is GPL 3+ licensed so it will be hard to curtail that one. They may hold back sources to some non-GPL modules but they would not be able to EULA a paying customer to not distribute changes to the GPL code further.
The GPL was the best gift to the free software community ever.

The GPL v2 contains the same clause as GPL v3 about further restrictions on the source code. Yet the Red Hat subscription license still says that they will cancel your subscription for distributing the source. So you may get the sources once but you won't get it after that. And there is a chance they take you to court.

You should switch to FreeBSD. This would and will never happen there.

I think this is a good idea. I have stopped myself using red hat, and mostly anything they touch since my trust level is pretty low with them, and i feel by using the fedoras etc, you contribute to Redhat. It's annoying, because I know redhat have contributed well to open source, so I wish my feelings were different. The developer subscription feels like a dangling carrot, ready to be pulled away like everything else, and comes with terms.

Our organization are using several of your roles, it is really bad to stop support for redhat overnight. Finally it was not a good idea to trust you and use them. Let's move to more serious tools.

One could say the same about using Red Hat Enterprise Linux... ;)

Luckily, there are alternatives.

Also, lucky for you, I am not a jerk so I keep every last line of source code I've ever written—including the build tools to build and test them—available perpetually, free to all users. A bit unlike Red Hat Enterprise Linux.

You're free for fork it, that's the wonderful thing about working with people and corporations who are good stewards of open source code.

Touche ... Irony like Recursion is beyond some folk. CentOS was analogous the Canary in the RedHat coal mine.

Many months ago, before #IBM took over, it was alive, it's clearly now Dead.

What has taken it's place, is a facsimile of a Canary. Likely, it is now an Albatross, soon to roost, around RedHat's neck, mariner style, but with no #redemption.

Wow. I can't imagine the entitlement factor to not only feel this way, but to also publicly announce your selfish opinion and shame yourself.

Did you pay for this software? did you pay for support? did you get a contract that said "free forever, on whatever OS you choose, and i'll support you till the day i die"? didn't think so. Keep this trash to yourself, and thank Jeff for doing your job for you while you're at it.

Thank you Jeff for all your good work. It is perfectly understandable that you cannot in good faith claim to support RHEL if you are not able to test on a 100% compatible environment. The "no-cost" subscription is not a good substitute as there are too many hoops to jump through, and hard to trust while RedHat is in full "I am altering the deal. Pray I do not alter it any further." mode.

I will probably be crucified for this, but I have to ask, because I am not at all familiar with all that's been going on with RHEL and Alma/Rocky Linux.

I have read/heard some arguments that RH is doing this to stop freeloading of RHEL's sources. Now, if that is true, then can Rocky/Alma Linux (and perhaps, Oracle) be considered RHEL freeloaders? Do they take RHEL's sources (which are used to package a paid product) and release their respective distributions, without contributing back to RHEL?

If so, is this move by RH a way to stop such freeloading? I think it is.

While I don't agree with RH's move, if I put myself in the shoes of RH, I can see why they decided to do this: Apart from official support from RH, what does RHEL offer that Alma/Rocky Linux don't?

So, if most enterprises drop RHEL for Rocky/Alma Linux, which aim to be carbon copies (which are also probably free, and with "good enough" support from the community), then RH stands to lose a sizeable portion of their revenue... and their solution was to stop them from being able to (at least easily) build RHEL from source... even if it means giving the middle finger to everyone else.

Do they take RHEL's sources (which are used to package a paid product) and release their respective distributions, without contributing back to RHEL?

One interpretation could be that yes, this is what they do. But both communities had finally gotten to a point where their contributors were also filing more Bugzilla reports and helping upstream... but now they will be spending more time re-working their build processes to work in the new scheme.

Another interpretation could be that by giving end users the ability to use Rocky Linux and AlmaLinux instead of RHEL, they were casting a wider net into the sea of potential contributors.

And not all contributions are code. People blogging about how to do X with Linux might use a RHEL-based distro as the base. Beginners would see that, get familiar with rpm-based tools, and eventually be part of the RHEL ecosystem. Or, a developer might try running Rocky because it works well on a certain platform, and eventually start contributing Linux kernel patches, or get involved in the Fedora community for more desktop-oriented use.

You lose that funnel when you cut out the community of... "freeloaders." I would never have gotten involved in Drupal had it not been for my ability to download the software and freeload for a few years. After I was comfortable in that ecosystem (and PHP more broadly), I started contributing source, running a local user group, even donating thousands of dollars over the years to the project.

I took some time to let the dust settle, and in short, I agree with you (and with all the people who replied to my comment), from a user and open source contributor's perspective.

When RH says that the clones "lower the value", what they really mean is "we lose money because of them".

RH is betting on all the enterprises that have to use RHEL due to commercial software that's only supported or packaged for RHEL (and RHEL-like distros). My previous employer only used CentOS on production machines, simply because they didn't want to pay for RHEL. Furthermore, I was taught Linux administration with CentOS, like so many university students. They were largely formatted for that ecosystem, and are, thus, more likely to recommend RHEL at their job.

While this will be financially good for RH in the short term, what this will result in RHEL's demise, as people will move away from it, not just in the enterprise, but also in education, and for personal projects.

In the end, it has nothing to do with open source, only revenue source(s), and is short sighted.

Worse yet, because RH is one big contributor to Linux-related projects, it will ultimately harm the ecosystem as well, as revenue from RHEL starts to diminish, and thus, less RH engineers are allocated to contribute.

That's exactly what OSS (and specifically the GPL) *IS*. That's the entire point, that one company can't take all those contributions and close it up to prevent others from using it as they see fit. But that's what RedHat (well, IBM really) is trying to do, in spite of the fact that RH themselves were built from the code created by others and released under the GPL.
If IBM didn't want to share their OS code, they shouldn't have bought RedHat, they could have kept doing that with their own AIX, or any other closed-source OS.

The term "freeloaders" should not be used lightly. Keep in mind that there are hundreds of open source packages that Redhat uses in RHEL without compensating the authors of those packages. If you say that Alma and Rocky "freeload" off of RHEL, then you must also necessarily say that RHEL "freeloads" off of other open source projects. IMHO the morally/ethically "correct" stance is that no freeloading takes place in either case.

The article author, Liam Proven, is a Douglas Adams fan. The term refers to the Douglas Adams quote "I worked hard to get where I am today, and I didn’t become captain of a Vogon constructor ship simply so I could turn it into a taxi service for a load of degenerate freeloaders. "

In this context, Red Hat is definitely acting like a Vogon and his paraphrasing fits.

Do you think Ansible, Molecule, and AWX are in jeopardy of being moved behind a pay wall? I'm extremely nervous because I use the tools literally every single day.

So far... no. I'm meeting with some Ansible folks Friday and hope to make it abundantly clear why people are nervous, and I hope that they will be able to re-assure the community of their commitment to not pull a CentOS.

It's obvious nothing's going to happen to Ansible, Molecule and AWS because those are UPSTREAM projects. This whole discussion is about DOWNSTREAM.

RHEL is a downstream product, where Red Hat does a lot of work by curating, testing, documenting, etc collections of upstreams. Red Hat contributed all their code to upstream. In fact, Red Hat is the main contributor to many key open source project, lightyears ahead of SUSE, Canonical and of course Rocky or Alma, which contribute nothing.

Ansible (what you get when you pip install ansible) is a downstream project that is entirely community driven (it is called "Ansible community distribution").

It was made that way during the great collections migration a few years back, and my fear is that since Red Hat Ansible Automation Platform is the revenue-generating side, when the next round of layoffs happen, the Ansible community team could be in the crosshairs, since their work does not directly increase revenue for Red Hat.

Is this not a logical conclusion based on the fact Red Hat is actively hostile towards "rebuilders"? What is Ansible besides a rebuild of the upstream ansible-core package plus a bunch of upstream collections?

RHEL is downstream Because They Put It Downstream. Remember when it was upstream and Cent was down? And that changed? Now remember what you just said about how nothing will happen to those things because they are upstream?

This seems odd to me. Red Hat Enterprise Linux Is Red Hat’s commercial side. There open source side is Fedora. It is like beta to Enterprise. Red Hat has always been structured this way. I’ve used Fedora since the beginning. Red Hat sells support with the Enterprise version.

Red Hat has been the Microsoft of Linux for many years. Companies like it because they can pay for 'support', but I digress. I ran across Alpine Linux about a year ago, and have been impressed with everything they do. From their package manager which is dirt simple to use, to their aport (their package builder), to the way their packages are short, simple, and to the point. I like the fact that they started out with security as a primary goal and still have that mentality. The musl C library cleans up a lot of the bloat that glibc has become because of Red Hat, in many cases. You can be free from systemd (or not). And it is one of the most tested distributions out there because it is usually the base for docker containers which are starting to explode. I've developed on Red Hat, Debian, Alpine, and many others, but Alpine's simple, secure philosophy is a breath of fresh air.