Red Hat Enterprise Linux 8 is supported until 2029, and that distribution includes Python 3.6 for system python. Ansible's long been stuck between a rock and a hard place supporting certain modules (especially packaging modules like dnf/yum on RHEL and its derivatives, because the Python bindings for the packaging modules are stuck supporting system Python.
Users are getting errors like:
/bin/sh: /usr/bin/python3: No such file or directory
The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error.
...or...
SyntaxError: future feature annotations is not defined
As ansible-core evolves, they don't want to support old insecure versions of Python forever—Python 3.6 was out of security support back in 2021!.
But that creates a conundrum; if you're a Red Hat customer using Ansible to automate your RHEL 8 infrastructure—or if you're using one of the many derivatives, like AlmaLinux, Rocky Linux, Oracle Linux, etc.—then you will start running into issues automating these older servers if you install the newest version of Ansible.
Luckily, Ansible core 2.16 is slated to be something of an 'LTS' release, according to one of the core maintainers—so if you lock into that version of Ansible core anywhere you run code against RHEL 8 servers, you should be good to go:
pip3 install ansible-core<2.17
Ideally, any security issues in Ansible itself will be fixed and backported in the 2.16 release branch—but it won't get any new features or any other Ansible 2.17+ goodness.
NOTE: You can install a newer version of Python on RHEL 8 systems and get at least much of Ansible to work, but many system-level components—most notably
dnf—won't work because of the older system Python bindings. That makes it functionally useless for me, becausednf/packageis probably the module that appears most frequently in my playbooks!
Where this really throws in a wrench is for 'community edition' users (I still don't know what the main ansible bundle package should be technically called—I still just call it Ansible hehe). There are two options, one probably better than the other:
- Lock in version 9.x:
pip install 'ansible<10.0' - Switch to installing
ansible-coreand a set of collections you need (which is similar to what the mainansiblepackage does, just you do it through a requirements file and lose a few packaging conveniences.
With the first option, you wind up on an already-unsupported release of Ansible 9.x:
Starting with version 2.10, the Ansible community team guarantees maintenance for only one major community package release at a time. For example, when Ansible 4.0.0 gets released, the team will stop making new 3.x releases. Community members may maintain older versions if desired.
With the second option, you'll lock in Ansible 2.16, but if you don't also start locking in other collection versions for modules you use, you may run into incompatibilities in the future, as other collections use features only present in Ansible 2.17 or later, or newer versions of Python.
In any case, I've moved my personal infrastructure over to Debian, so these things aren't an issue as I've started upgrading servers on a 5 year cycle instead of 10 years—but even there, Ubuntu 18.04 stragglers (to be fair, it's LTS support ended last year) also have issues since system python is too old for Ansible.
The main takeaway: Ansible's core developers have consistently chosen newer Python version compatibility at the expense of supporting old, unsupported versions. So plan your infrastructure lifecycle accordingly. IMO, if you are an organization that chooses to run servers for 10 years without migrating to newer OS releases... you should figure out how you can use Ansible to make those OS upgrades and system migrations easier :P
Comments
Important follow-up from felixfontein over on GitHub:
It's good to know! Still, the same caveats apply, but at least Ansible (community) 9.x should be somewhat supported for another 6 months.
This goes beyond the issue about what versions you can install on EL8.
What this is really going to break for most companies is patching EL8 from a centralized location. Since you don't typically install Ansible on all your boxes, you are running Ansible against everything from one place. Be it your laptop, a central Ansible controller, or something like AWX/Ascender. When Ansible connects in, it is copying the module from the Ansible install on the controller to the remote host and executing it. If you don't pin your controller to the old version of Ansible, use a different venv, or build a separate Execution Environment pinned to the old version; it will fail on the EL8 boxes.
As newer versions of modules start to rely on functionality built into newer Ansible versions, the disparity on what you can do against EL8 vs EL9+ will drift further and further apart so that you are maintaining different versions of playbooks or at least having to account for only using older modules on EL8 via includes like you typically do now with different families of distros.
So now you have to move to either sticking with the old version for everything until 2029, or automate/patch EL8 separately. As they can't be patched in the same job if running the newer Ansible.
If I'm going to run a 10-year LTS-ish os, it is reasonable to assume that I'll take similar stability goals into mind and freeze my ansible playbooks and the like. Alternately I can always compile my own python for /usr/local/bin or the like, and potentially mirror any upstream sites I'm relying on internally just.in.case. upstream moves or disappears. I took both approaches in former $jobs.
Developer convenience over user convenience.
I have seen that mistake been made over and over again.
Most likely Red Hat will fix this self-inflicted problem if you are willing to pay them for it.
Same here with supported Ubuntu and Raspbian versions.
It looks like the Ansible community team is not looking after heterogenous environments any longer. In our Ubuntu/Raspbian environment Ansible worked as a charm up to now. Installing the first server with Ubuntu 24.04 LTS we mentioned that an Ansible run from a 22.04 LTS-server breaks with incompatible and not available Python modules. Also the other way around fails. Upgrading the Ansible server to 24.04 LTS, the Ansible run on managed Systems with Python Version <= 3.10 also gonna fail with Python library errors.
> As ansible-core evolves, they don't want to support old insecure versions of Python forever—Python 3.6 was out of security support back in 2021!.
True, and I agree with your other points about ongoing issues and using newer versions of core and its modules. However, if the reader quits after the first paragraph or two it could be interpreted as implying RHEL runs insecure versions. You of course mention this later in the article that bug and security fix commits from newer versions get back ported and at some point, not new features (which EL8 no longer does) This is meant to keep a API stable, modules, etc. for production users at the cost of the latest and greatest features, at least once EL leaves its "enhancement phase" (e.g. now called the Full Support Phase) of the LTS lifecycle.
I generally suggest when using Enterprise Linux distributions to migrate to RHEL 9 (or RockyLinux 9 etc.) to get versions newer than EL8 offers. RHEL/Rocky etc. are now at 9.4 and 2 years into the 10 year LTS cycle, so it's a great time to consider the migration as there is still 3 years left in the Full Support Phase which could also result in new versions/features for Ansible and other components. However, once 9.x makes it to Maintenance Support (around 2027) newer versions wont be added or software enhancements back ported, its only bug and security fixes, again avoiding new software functionality which could lead to breakage in the API during Maintenance Support Phase.
Also, an often overlooked issue with superseding the supported versions (Ansible, Python, etc.) as they do not come from AppStream, or DNF Modules, can be the headaches this can cause during major version upgrades when the target OS (presumably RHEL 9.x) may consider its installation a downgrade of currently installed versions. Not to mention using pip directly can occasionaly lead to some unexpected results and errors for related tools. This of course comes from a "drink the Cool Aid" perspective and tries to stay as close to the intended design of Enterprise Linux as possible to reduce these unexpected consequences.
If the latest and greatest versions are a concern there is always RHCOS (Red Hat Enterprise Linux CoreOS) or Fedora CoreOS.
Not that your suggestion isn't valid of course to install via pip. For a hobbyist this might be perfectly acceptable, however for extremely large production environments with hundreds, or thousands, of systems it might be best to avoid this unless the user is intimately familiar with the distro and EL ecosystem in general. Staying with a bundled version of most tools reduces the potential issues from other installed tools as well as when attempting major version upgrades
**Disclaimer**
I am not a Red Hat employee, but I have been a user of Red Hat since the Halloween release. My daily drivers and most systems I have used for a few decades have been Red Hat or part of the Enterprise Linux ecosystem. So I suppose you could say I drink a lot of Cool Aid.
"for extremely large production environments with hundreds, or thousands, of systems it might be best to avoid this unless the user is intimately familiar with the distro and EL ecosystem in general"
- if you have hundreds or thousands of systems you BETTER be intimately familiar with the distro and ecosystem...
Short answer is if you want to be stable, you don't 'want' to be current. You want to be stable with all its feature set (or lack therof vs. bleeding edge current).
For running Ansible _on_ an EL8 host, EPEL ships an "ansible" package that installs a "curated set of Ansible collections" on top of ansible-core 2.16 and python3.12 packages from appstream. The dnf module seems to work fine with that, and looking at its dnf.py it looks like it's re-spawning itself with the system python to get the imports to work.
Running Ansible _against_ EL8 hosts in the inventory is a different story and you'll probably need to some extra fact-less play to bootstrap a supported python version (possibly with command or even raw tasks) and then muck around with `ansible_python_interpreter` and the like for the rest.
On the bright side, you don't have to worry about EL7 anymore :-)
Due to the issue 2.16 will be supported till end of RHEL 8.
https://github.com/ansible/ansible/issues/83357#issuecomment-2150254754
Thanks Jeff. Hit this today. Found your article right away.
Heh, I hit my old articles far too often searching for the same problem over and over again.