[Python-Dev] PEP 376 - Open questions

[glyph at divmod.com]

On 02:55 am, eric at trueblade.com wrote:
>I really don't get this use case of multiple installers trying to 
>install the same package. There's just no way that running "yum install 
>twisted" and "apt-get install twisted" and "pip install twisted" are 
>going to coexist with each other. The best they can do is say "a file 
>I'm trying to install already exists". Why try for anything else?

Here are some use-cases.

You do "apt-get install twisted", or, rather, Twisted comes preinstalled 
on your OS.  Later, you do "pip install 
py_awesome_network_thing_system", which depends on twisted.

You've configured pip to install files into ~/.local, though, and it's 
not sure if Twisted is installed or not.  So thanks to automatic 
dependency resolution, it downloads and installs another copy.  But 
that's not what you want: you want it to use the system-installed 
Twisted (and zope.interface, and pycrypto, and pytz, etc etc) and just 
install PANTS into your .local environment.  No files will conflict, but 
it would be helpful for the package installation tool to have an API to 
inspect both the system package installation and any other places where 
packages have been installed in order to make a decision about how it 
needs to satisfy its dependencies.

If you're not as interested as I am in per-user installation of 
packages, then consider the fact that Ubuntu stores Python packages in 
/usr/lib/pythonX.Y/dist-packages now.  Python package installers should 
be able to query that before putting things into site-packages. (which I 
*believe* is now reserved for non-OS-installed packages, but it's very 
hard to find documentation on the motivation behind debian's 
manipulations of distutils.  In any event, I digress.)

You're correct that worrying about 'yum install twisted' and 'apt-get 
install twisted' is way out of scope for any Python installation system; 
unfortunately, yum and apt are likely as not to have giant, opaque 
makefiles or shell scripts attached to these packages for obscure 
linux-y reasons that (hopefully) we don't need to care about, but would 
fight with each other if you tried to combine them, totally out of scope 
with distutils.

One hopes that regardless of whatever else they're doing, these packages 
*are* providing correct installation metadata.  (Unfortunately, I can 
bet you a nickel they will screw it up in the first few python packages 
that include it; see, for example, 
<https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/361865>.  This 
spec should try to gracefully address what to do if system packagers 
have done something unusual or wrong.)

So, the filesystem and the OS package database and the Python package 
"database" (such as it is) are definitely three distinct things, but 
they have some overlap, and there should be APIs for querying each.  If 
for no other reason so you know when Python packages shouldn't stomp on 
OS packages, and vice versa.