
Hi all, First post here. I have a cluster where the common software is NFS shared from the file server to other nodes. All the python packages are kept in a directory which is referenced by PYTHONPATH. The good part of that is that there is just one copy of each package-version. The bad part, as you have all no doubt guessed, is that python by itself is really bad at specifying and loading a set of particular library versions (see below), so upgrading one program will break another due to conflicting installed versions. Hence the common use of virtualenv's. But as far as I can tell each virtualenv installs a copy of each package-version it needs, resulting in multiple copies of the same package-version for common packages on the same disk. What I am after is some method of keeping exactly one copy of each package-version in the common area (ie, one might find foo-1.2, foo-1.7, and foo-2.3 there), while also presenting only the one version of each (let's say foo-1.7) to a particular installed program. On linux it might do that by making soft links to the common PYTHONPATH area from another directory for which it sets PYTHONPATH for the application. Finally, this has to be usable by any account which has read execute access to the main directory. Does such a beast exist? If so, please point me to it! The limitations of python version handling to which I refer above can be illustrated for "scanpy-scripts"'s dependencies. Given all the needed libraries in one place (plus incompatible versions) the right set can be loaded (and verified) like this: export PYTHONPATH=/path/to_common_area python3 __requires__ = ['scipy <1.3.0,>=1.2.0', 'anndata <0.6.20', 'loompy <3.0.0,>=2.00', 'h5py <2.10'] import pkg_resources import scipy import anndata import loompy import h5py import scanpy print(scipy.__version__) print(anndata.__version__) print(loompy.__version__) print(h5py.__version__) print(scanpy.__version__) quit() which emits exactly the versions scanpy-scripts needs: 1.2.3 0.6.19 2.0.17 2.9.0 1.4.3 However, adding , 'scanpy <1.4.4,>=1.4.2' at the end of __requires__ makes the whole thing fail at import pkg_resources with (many lines deleted) 792, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (scipy 1.2.3 (/home/common/lib/python3.6/site-packages/scipy-1.2.3-py3.6-linux-x86_64.egg), Requirement.parse('scipy>=1.3.1'), {'umap-learn'}) even though the scanpy it loaded in the first case was within the desired range. Moreover, specifying the desired versions as parameters to import pkg_resources does not work at all since pkg_resources only keeps the highest version of each package it finds when imported. (A limitation that never made the least bit of sense to me.) The test system is CentOS 8 with python 3.6.8. Thanks, David Mathog