[Python-Dev] Status of packaging in 3.3

[Chris McDonough]

On 06/21/2012 04:45 AM, Nick Coghlan wrote:
> On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough<chrism at plope.com>  wrote:
>> All of these are really pretty minor issues compared with the main benefit
>> of not needing to ship everything with everything else. The killer feature
>> is that developers can specify dependencies and users can have those
>> dependencies installed automatically in a cross-platform way.  Everything
>> else is complete noise if this use case is not served.
>
> Cool. This is the kind of thing we need recorded in a PEP - there's a
> lot of domain knowledge floating around in the heads of packaging
> folks that needs to be captured so we can know *what the addition of
> packaging to the standard library is intended to fix*.
>
> And, like it or not, setuptools has a serious PR problem due to the
> fact it monkeypatches the standard library, uses *.pth files to alter
> sys.path for every installed application by default, actually *uses*
> the ability to run code in *.pth files and has hard to follow
> documentation to boot. I *don't* trust that I fully understand the
> import system on any machine with setuptools installed, because it is
> demonstrably happy to install state to the file system that will
> affect *all* Python programs running on the machine.

I don't know about Red Hat but both Ubuntu and Apple put all kinds of 
stuff on the default sys.path of the system Python of the box that's 
related to their software's concerns only.  I don't understand why 
people accept this but get crazy about the fact that installing a 
setuptools distribution using easy_install changes the default sys.path.

Installing a distribution will change behavior whether or not sys.path 
is changed as a result.  That's its purpose.  The code that runs in the 
.pth *file* (there's only one that matters: easy_install.pth) just 
mutates sys.path.  The end result is this: if you understand how 
sys.path works, you understand how eggs work.  Each egg is addded to 
sys.path.  That's all there is to it.  It's the same as manually 
mutating a global PYTHONPATH, except you don't need to do it.

And note that this is not "setuptools" in general.  It's easy_install in 
particular.  Everything you've brought up so far I think is limited to 
easy_install.  It doesn't happen when you use pip.  I think it's a 
mistake that pip doesn't do it, but I think you have to make more 
accurate distinctions.

> A packaging PEP needs to explain:
> - what needs to be done to eliminate any need for monkeypatching
> - what's involved in making sure that *.pth are *not* needed by default
> - making sure that executable code in implicitly loaded *.pth files
> isn't used *at all*

I'll note that these goals are completely sideways to any actual 
functional goal.  It'd be a shame to have monkeypatching going on, but 
the other stuff I don't think are reasonable goals.  Instead they 
represent fears, and those fears just need to be managed.

> I *think* trying to achieve this is actually the genesis of the
> original distribute fork, that subsequently became distutils2 as Tarek
> discovered how much of the complexity in setuptools was actually due
> to the desire to *not* officially fork distutils (and instead
> monkeypatch it, effectively creating a runtime fork).
>
> However, for those of us that weren't directly involved, this is all
> still a strange mystery dealt with by other people. I've cribbed
> together bits and pieces just from following the fragments of the
> discussions that have happened on python-dev and at PyCon US, but if
> we want the madness to ever stop, then *the problems with the status
> quo* need to be written down so that other core developers can
> understand them.

It'd also be useful if other core developers actually tried to use 
setuptools in anger.  That'd be a good start towards understanding some 
of its tradeoffs.  People can write this stuff down til they're blue in 
the face, but if core devs  don't try the stuff, they'll always fear it.

> In fact, I just remembered that Tarek *has* written a lot of this
> down, just not in PEP form: http://www.aosabook.org/en/packaging.html


Cool.

- C