[Distutils] pip upgrade woes

[Paul Moore]

On 24 March 2015 at 18:10, Ionel Cristian Mărieș <contact at ionelmc.ro> wrote:
> There's one issue with pip upgrade that annoys me occasionally, and when it
> does it's very annoying.
>
> Every so often me or some customer has to upgrade some core packages like
> pip, setuptools or virtualenv on some machine. Now this becomes very
> annoying because said packages were installed there with either easy_install
> or just `setup.py install`. Several upgrades like that and now the machine
> has a hadful of eggs there. Lots of mistakes were made but what's done is
> done.
>
> Now, if `pip upgrade pip setuptools virtualenv` it will run around a bit,
> flap its wings and in the end it's not gonna fly like an eagle, and won't be
> able to go beyond it's cursed fences. An so, I feel like chicken farmer when
> I try to upgrade packages and pip can't upgrade them. Cause those old eggs
> are still going to be first on sys.path. And when I try to run pip it's
> still that old one.
>
> Sometimes few `pip uninstall` solve the issue, but most of the time I have
> to manually remove files because pip can't figure out what files to remove.
>
> One big issue is that pip uninstall only uninstalls the first package it
> finds, and similarly, pip install will only uninstall the first package it
> finds before coping the new files.
>
> This whole process becomes a whole lot more annoying when you have to
> explain someone how to cleanup this mess and get latest pip and setuptools.
>
> So I'm wondering if there's a better way to cleanup machines like that. Any
> ideas?

If I understand your problem correctly, the issue is that these
machines have older installs of packages, added by tools that don't
have uninstall capabilities, using formats that are not used by pip.
You're not able to get pip uninstall to work either because the
uninstall data isn't in a form pip can handle, or because there is no
uninstall data. I suspect the "first on sys.path" issue is caused by
setuptools' .pth files, which are even messier to tidy up, from what I
recall of my infrequent dealings with them.

It should be possible to write some sort of "purge" command that
searches out and removes all traces of a package like that. I wouldn't
want pip to do anything like that automatically, for obvious reasons.
I'm not even that keen on it being a new pip subcommand, because
there's a lot of risk of breakage (I'd imagine the code would need a
certain amount of guesswork to identify non-standard files and
directories that "belong" to a package).

As a start, I'd suggest looking at writing some sort of independent
purge-package command that you could use when you hit problems (pip
install -U setuptools... weirdness happens, so purge-package
setuptools; pip install setuptools). If all the scenarios that tool
handles end up being clearly defined and low-risk, then it might be
that there's scope for a "pip purge" command of some sort based on it,
that removes all trace of a package even when the standard uninstall
metadata isn't available.

Maybe someone else here with more understanding of the history of
easy_install and how eggs used to work (maybe still do, for all I
know...) can offer something more specific. Sorry that I can't.
Paul