[Python-Dev] Status of packaging in 3.3

[Chris McDonough]

On 06/21/2012 08:21 AM, Nick Coghlan wrote:
>
>> Installing a distribution will change behavior whether or not sys.path is
>> changed as a result.  That's its purpose.
>
> No it won't. An ordinary package will only change the behaviour of
> Python applications that import a package by that name. Other Python
> applications will be completely unaffected (as it should be).

If a Python application is effected by a change to sys.path which 
doesn't impact modules it uses, then that Python application is plain 
broken, because the developer of that application cannot make 
assumptions about what a user does to sys.path unrelated to the modules 
it requires.  This is completely independent of easy_install.

Any Python application is going to be effected by the installation of a 
distribution that does impact modules it imports, whether sys.path is 
used to change the working set of modules or not.

So what concrete situation are we actually talking about here?

>>   The code that runs in the .pth
>> *file* (there's only one that matters: easy_install.pth) just mutates
>> sys.path.  The end result is this: if you understand how sys.path works, you
>> understand how eggs work.  Each egg is addded to sys.path.  That's all there
>> is to it.  It's the same as manually mutating a global PYTHONPATH, except
>> you don't need to do it.
>
> Yes, it's the same as mutating PYTHONPATH. That's a similarly bad
> system global change. Individual libraries do not have the right to
> change the sys.path seen on initialisation by every other Python
> application on that system.

Is it reasonable to even assume there is only 
one-sys.path-to-rule-them-all? And that users install "the set of 
libraries they need" into a common place?  This quickly turns into 
failure, because Python is used for many, many tasks, and those tasks 
sometimes *require conflicting versions of libraries*.  This is the root 
cause of why virtualenv exists and is popular.

The reason it's disappointing to see OS vendors mutating the default 
sys.path is because they put *very old versions of very common 
non-stdlib packages* (e.g. zope.interface, lxml) on sys.path by default. 
  The path is tainted out of the box for anyone who wants to use the 
system Python for development of newer software.  So at some point they 
invariably punt to virtualenv or a virtualenv-like system where the 
OS-vendor-provided path is not present.

If Python supported the installation of multiple versions of the same 
module and versioned imports, both PYTHONPATH and virtualenv would be 
much less important.  But given lack of enthusiasm for that, I don't 
think it's reasonable to assume there is only one sys.path on every system.

I sympathize, however, with Oscar's report that PYTHONPATH can't the 
setuptools-derived path.  That's indeed a mistake that a future tool 
should not make.

>> And note that this is not "setuptools" in general.  It's easy_install in
>> particular.  Everything you've brought up so far I think is limited to
>> easy_install.  It doesn't happen when you use pip.  I think it's a mistake
>> that pip doesn't do it, but I think you have to make more accurate
>> distinctions.
>
> What part of "PR problem" was unclear? setuptools and easy_install are
> inextricably linked in everyone's minds, just like pip and distribute.

Hopefully for the purposes of the discussion, folks here can make the 
mental separation between setuptools and easy_install.  We can't help 
what other folks think in the meantime, certainly not solely by making 
technological compromises anyway.

>>> A packaging PEP needs to explain:
>>> - what needs to be done to eliminate any need for monkeypatching
>>> - what's involved in making sure that *.pth are *not* needed by default
>>> - making sure that executable code in implicitly loaded *.pth files
>>> isn't used *at all*
>>
>> I'll note that these goals are completely sideways to any actual functional
>> goal.  It'd be a shame to have monkeypatching going on, but the other stuff
>> I don't think are reasonable goals.  Instead they represent fears, and those
>> fears just need to be managed.
>
> No, they reflect the mindset of someone with configuration management
> and auditing responsibilities for shared systems with multiple
> applications installed which may be written in a variety of languages,
> not just Python. You may not care about those people, but I do.

I care about deploying Python-based applications to many platforms.  You 
care about deploying multilanguage-based applications to a single 
platform.  There's going to be conflict there.

My only comment on that is this: Since this is a problem related to the 
installation of Python distributions, it should deal with the problems 
that Python developers have more forcefully than non-Python developers 
and non-programmers.

>> It'd also be useful if other core developers actually tried to use
>> setuptools in anger.  That'd be a good start towards understanding some of
>> its tradeoffs.  People can write this stuff down til they're blue in the
>> face, but if core devs  don't try the stuff, they'll always fear it.
>
> setuptools (or, perhaps, easy_install, although I've seen enough posts
> about eggs being uploaded to PyPI to suspect otherwise), encourages
> the deployment of system configuration changes that alter the runtime
> environment of every single Python application executed on the system.
> That's simply not cool.

Again, it would help if you tried it in anger.  What's the worst that 
could happen?  You might like it! ;-)

- C