[Python-Dev] Status of packaging in 3.3

[Nick Coghlan]

On Thu, Jun 21, 2012 at 10:51 PM, Chris McDonough <chrism at plope.com> wrote:
> Is it reasonable to even assume there is only one-sys.path-to-rule-them-all?
> And that users install "the set of libraries they need" into a common place?
>  This quickly turns into failure, because Python is used for many, many
> tasks, and those tasks sometimes *require conflicting versions of
> libraries*.  This is the root cause of why virtualenv exists and is popular.

And why I'm very happy to see pyvenv make it's way into the standard library :)

> I care about deploying Python-based applications to many platforms.  You
> care about deploying multilanguage-based applications to a single platform.
>  There's going to be conflict there.
>
> My only comment on that is this: Since this is a problem related to the
> installation of Python distributions, it should deal with the problems that
> Python developers have more forcefully than non-Python developers and
> non-programmers.

Thanks to venv, there's an alternative available that may be able to
keep both of us happy: split the defaults. For system installs, adopt
a vendor-centric, multi-language,
easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding
*.pth files by unpacking eggs to the file system). For venv installs,
do whatever is most convenient for pure Python developers (e.g.
leaving eggs packed and using *.pth files to extend sys.path within
the venv).

One of Python's great virtues is its role as a glue language, and part
of being an effective glue language is playing well with others. That
should apply to packaging & distribution as well, not just to runtime
bindings to tools written in other languages.

When we add the scientific users into the mix, we're actually getting
to a *third* audience: multi-language developers that want to use
*Python's* packaging utilities for their source and binary
distribution formats.

The Python community covers a broad spectrum of use cases, and I
suspect that's one of the big reasons packaging can get so contentious
- the goals end up being in direct conflict. Currently, I've
identified at least half a dozen significant communities with very
different needs (the names aren't meant to be all encompassing, just
good representatives of each category, and many individuals will span
multiple categories depending on which hat they're wearing at the
time):

Library authors: just want to quickly and easily publish their work on
the Python package index in a way that is discoverable by others and
allows feedback to reach them at their development site

Web developers: creators of Python applications, relying primarily on
other Python software and underlying OS provided functionality,
potentially with some native extensions, that may need to run on
multiple platforms, but can require installation using a language
specific mechanism by technical staff

Rich client developers: creators of Python applications relying
primarily on other Python software and underlying OS provided
functionality, potentially with native extensions, that need to run on
multiple platforms, but must be installed using standard system
utilities for the benefit of non-technical end users

Enterprise developers: creators of Python or mixed language
applications that need to integrate with corporate system
administration policies (including packaging, auditing and
configuration management)

Scientists: creators of Python data analysis and modelling
applications, with complex dependencies on software written in a
variety of other languages and using various build systems

Python embedders: developers that embed a Python runtime inside a
larger application

>> setuptools (or, perhaps, easy_install, although I've seen enough posts
>> about eggs being uploaded to PyPI to suspect otherwise), encourages
>> the deployment of system configuration changes that alter the runtime
>> environment of every single Python application executed on the system.
>> That's simply not cool.
>
> Again, it would help if you tried it in anger.  What's the worst that could
> happen?  You might like it! ;-)

Oh, believe me, if I ever had distribution needs that required the
power and flexibility of setuptools, I would reach for it in a
heartbeat (in fact, I already use it today, albeit for tasks that
ordinary distutils could probably handle). That said, I do get to
cheat though - since I don't need to worry about cross-platform
deployment, I can just use the relevant RPM hooks directly :)

You're right that most of my ire should be directed at the default
behaviour of easy_install rather than at setuptools itself, though. I
shall moderate my expressed opinions accordingly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia