On new Debian 12 Bookworm installs, when I try running pip3 install [something] (whether that's Ansible or some other Python tool), I get the following error message:
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
The error message says you can pass in the flag --break-system-packages but that sounds terrifying. I just want pip to stop nagging me, but let me manage my system dependencies like I have for many years.
I think some Python developers really want people like me to use virtual environments, but that's way too much effort when I don't really care to do that, thankyouverymuch. If you want to use venv more power to you. I just like getting stuff done on my little servers.
So the solution for Debian 12, at least, is to delete the EXTERNALLY-MANAGED file in your system Python installation:
sudo rm -rf /usr/lib/python3.11/EXTERNALLY-MANAGED
Note that the python3.11 version number should match whatever you have installed—it was 3.11 at the time of this blog post's writing.
See this answer on Stack Overflow for more. Another interesting option is to install and use pipx, which does the grunt work of managing the venvs for you.
Comments
This merges your development environment with the OS's working environment and its tools, some of which are dependent on python. This can break things in ways that can be difficult to recover. See the Motivation section of the PEP that implemented the check for a more thorough explanation. https://peps.python.org/pep-0668/
Venvs aren't hard, but you do you.
They're not hard, just annoying.
And 99% of my work is inside containers or VMs, where rebuilding is done at the drop of a hat. Having to add on any extra stuff to manage virtual Python environments is just annoying and takes more space + time in the build.
Just like with PHP installs, Java, etc., I will rely on the OS install and if I absolutely need more separation, I throw the extra environment inside Docker. Some people also like to install and manage multiple versions of PHP, Node.js, or Java, but with modern tooling, it feels redundant to try to manage environments using separate tools for each programming language.
It's a pain if you want to install some command line utility via pip into your $PATH for general use. Think awscli for example (although their v2 has its own installer), or ansible as you mention. I feel like this case got overlooked in the PEP.
Yeah, I think the PEP authors are considering mostly the deployment of big Python applications to different systems.
But for most sysadmin types, we just want to throw a few CLI tools on the system (ansible, awscli, etc.), and throwing another layer of abstraction on top of system Python is just annoying (and requires extra automation steps to manage).
If it works for you, great. As a warning, you can end up with some interesting problems when you try to update the system. There's a reason the file you deleted has the same effect as using `--break-system-packages`. (Of course you also lose the ability to use multiple versions of a dependency too).
As another alternatives, for a one-off thing I just use `python3 -m venv venv` to create a local `./venv` directory in the project, and `. ./venv/bin/activate` to use it (or just do `./venv/bin/{pip,python}` commands directly). If you want still use those system packages you installed, make it `python -m venv --system-site-packages venv` instead.