On Wed, 24 Feb 2021 13:47:40 +0100 Stéfane Fermigier <sf@fermigier.com> wrote:
On Wed, Feb 24, 2021 at 12:42 PM Paul Moore <p.f.moore-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier <sf-xq1QBjP4wZx54TAoqtyWWQ@public.gmane.org> wrote:
There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.
Ultimately the problem is that a general tool can't deal with conflicts (except by raising an error). If application A depends on lib==1.0 and application B depends on lib==2.0, you simply can't have a (consistent) environment that supports both A and B. But that's the rare case - 99% of the time, there are no conflicts. One env per app is a safe, but heavy handed, approach. Managing environments manually isn't exactly *hard*, but it's annoying manual work that pipx does an excellent job of automating, so it's a disk space vs admin time trade-off.
There are three ways to approach the question:
1) Fully isolated envs. The safest option but uses the most space.
2) Try to minimise the number of dependencies installed by interpreting the requirements specification in the looser way possible. This is both algorithmically hard (see https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and risky, as you've noted.
3) But the best way IMHO is to compute dependencies for each virtualenv independently from the others, but still share the packages, using some indirection mechanisms (hard links, symlinks or some Python-specific constructs) when the versions match exactly.
The 3rd solution is probably the best of the 3, but the sharing mechanism still needs to be specified (and, if needed, implemented) properly.
I wouldn't want to repeat myself too often, but conda and conda-based distributions already have sharing through hardlinks (or, on Windows, whatever is available) baked-in, assuming you install your software from conda packages. That also applies to non-Python packages, and to python itself (which is just a package like any other). Regards Antoine.