Thanks for the link. Unfortunately there was not a reference to a completed package that actually did this. As in, I really do not want to reinvent the wheel. Ugh, sorry, that's a pun in this context. Here is a first shot at this, just installing a moderately complicated package in a virtualenv and then reinstalling it in another virtualenv. Extract and execinput are my own programs (from drm_tools on sourceforge) but it is obvious from the context what they are doing. The links had to be soft because linux does not actually allow a normal user (or maybe even root) to make a hard link to a directory. cd /usr/common/lib/python3.6/Envs rm -rf ~/.cache/pip #make download clearer python3 -m venv scanpy source scanpy/bin/activate python -m pip install -U pip #update 9.0.3 to 20.1.1 which python3 #using the one in scanpy pip3 install scanpy scanpy -h #seems to start deactivate rm -rf ~/.cache/pip #make download clearer python3 -m venv scanpy2 source scanpy2/bin/activate python -m pip install -U pip #update 9.0.3 to 20.1.1 export DST=/usr/common/lib/python3.6/Envs/scanpy/lib/python3.6/site-packages export SRC=/usr/common/lib/python3.6/Envs/scanpy2/lib/python3.6/site-packages ls -1 $DST \ | grep -v __pycache__ \ | grep -v scanpy \ | grep -v easy_install.py \ | extract -fmt "ln -s $DST/[1,] $SRC/[1,]" \ | execinput pip3 install scanpy #downloaded scanpy, "Requirement already satisfied" for all the others #Installing collected packages: scanpy # Successfully installed scanpy-1.5.1 scanpy -h #seems to start deactivate source scanpy/bin/activate scanpy -h #seems to start (still) deactivate So that method seems to have some promise. It saved a considerable amount of space too: du -k scanpy | tail -1 457408 scanpy du -k scanpy2 | tail -1 24900 scanpy2 However, two potential problems are evident on inspection. The first is that when the 2nd scanpy installation was performed it updated the dates on all the directories in $DST. A workaround would be to copy all of those directories into the virtualenv temporarily, just for the installation, and then remove them and put the links in afterwards. That strikes me as awfully cludgy. Setting them read only would likely break the install. The second issue is that each package install creates two directories like: llvmlite llvmlite-0.33.0.dist-info where the latter contains top_level.txt which in turn contains one line: llvmlite pointing to the first directory. If another version must cohabit with it the "llvmlite" directories will conflict. For this sort of approach to work easily the llvmlite directory should be named "llvmlite-0.33.0" and top_level.txt should reference that too. It would be possible (probably) to work around it though by having llvmlite-0.33.0 only in the common area and use: ln -s $COMMON/llvmlite-0.33.0 $VENVAREA/llvmlite The top_level.txt in each could then reference the unversioned name. Unknown if this soft link approach will work on Windows. Regards, David Mathog On Wed, Jun 24, 2020 at 1:26 PM Steve Dower <steve.dower@python.org> wrote:
On 24Jun2020 1923, David Mathog wrote:
I think I will experiment a little with pipenv and if necessary after each package install use a script to remove the installed libraries and replace them with a hard link to the one in the common area. Maybe it will be possible to put in those links before installing the package of interest (like for scanpy, see first post), which will hopefully keep it from having to rebuild all those packages too.
Here's a recent discussion about this exact idea (with a link to an earlier discussion on this list): https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-g...
It's totally possible, though it's always a balance of trade-offs. Some of the people on that post may be interested in developing a tool to automate parts of the process.
Cheers, Steve