[Python-Dev] Status of packaging in 3.3

[Chris McDonough]

On 06/21/2012 09:29 AM, Nick Coghlan wrote:
>> My only comment on that is this: Since this is a problem related to the
>> installation of Python distributions, it should deal with the problems that
>> Python developers have more forcefully than non-Python developers and
>> non-programmers.
>
> Thanks to venv, there's an alternative available that may be able to
> keep both of us happy: split the defaults. For system installs, adopt
> a vendor-centric, multi-language,
> easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding
> *.pth files by unpacking eggs to the file system). For venv installs,
> do whatever is most convenient for pure Python developers (e.g.
> leaving eggs packed and using *.pth files to extend sys.path within
> the venv).

I'd like to agree with this, but I think there's a distinction that 
needs to be made here that's maybe not obvious to everyone.

A tool to generate an OS-specific system package from a Python library 
project should be unrelated to a Python distribution *installer*. 
Instead, you'd use related tools that understood how to unpack the 
distribution packaging format to build one or more package structures. 
The resulting structures will be processed and then eventually installed 
by native OS install tools.  But the Python distribution installer (e.g 
easy_install, pip, or some future similar tool) would just never come 
into play to create those structures.  The Python distribution installer 
and the OS-specific build tool might share code to introspect and unpack 
files from the packaging format, but they'd otherwise have nothing to do 
with one another.

This seems like the most reasonable separation of concerns to me anyway, 
and I'd be willing to work on the code that would be shared by both the 
Python-level installer and by OS-level packaging tools.

> One of Python's great virtues is its role as a glue language, and part
> of being an effective glue language is playing well with others. That
> should apply to packaging&  distribution as well, not just to runtime
> bindings to tools written in other languages.
>
> When we add the scientific users into the mix, we're actually getting
> to a *third* audience: multi-language developers that want to use
> *Python's* packaging utilities for their source and binary
> distribution formats.
>
> The Python community covers a broad spectrum of use cases, and I
> suspect that's one of the big reasons packaging can get so contentious
> - the goals end up being in direct conflict. Currently, I've
> identified at least half a dozen significant communities with very
> different needs (the names aren't meant to be all encompassing, just
> good representatives of each category, and many individuals will span
> multiple categories depending on which hat they're wearing at the
> time):
>
> Library authors: just want to quickly and easily publish their work on
> the Python package index in a way that is discoverable by others and
> allows feedback to reach them at their development site
>
> Web developers: creators of Python applications, relying primarily on
> other Python software and underlying OS provided functionality,
> potentially with some native extensions, that may need to run on
> multiple platforms, but can require installation using a language
> specific mechanism by technical staff
>
> Rich client developers: creators of Python applications relying
> primarily on other Python software and underlying OS provided
> functionality, potentially with native extensions, that need to run on
> multiple platforms, but must be installed using standard system
> utilities for the benefit of non-technical end users
>
> Enterprise developers: creators of Python or mixed language
> applications that need to integrate with corporate system
> administration policies (including packaging, auditing and
> configuration management)
>
> Scientists: creators of Python data analysis and modelling
> applications, with complex dependencies on software written in a
> variety of other languages and using various build systems
>
> Python embedders: developers that embed a Python runtime inside a
> larger application

I think we'll also need to put some limits on the goal independent of 
the union of everything all the audiences require.

Here's some scope suggestions that I believe could be shared by all of 
the audiences you list above except for embedders; I think that use case 
is pretty much separate.  It might also leave "rich client developers" 
wanting, but no more than they're already wanting.

- Install code that can *later be imported*.  This could be pure Python
   code or C code which requires compilation.  But it's not for the
   purpose of compiling and installing completely arbitrary C code to
   arbitrary locations, it's ust written for the purpose of compiling C
   code which then *lives in the installed distribution* to
   provide an importable Python module that lives in the same
   distribution with logic.

- Install "console scripts" which are shell-scripts/batch-files
   that cause some logic written in Python to get run as a result.
   These console scripts are written to sys.prefix + '/{bin/Scripts}'
   depending on the platform.

- Install "package resources", which are non-Python source files that
   happen to live in package directories.

IOW, an installer should be about installing Python libraries and 
supporting files to a well-known location defined by the interpreter or 
venv that runs it, not full applications-that-require-persistent-state 
which just happen to be written in Python and which require deployment 
to arbitrary locations.

You shouldn't expect the Python packaging tools to install an instance 
of an application on a system, you should expect them to install enough 
code that would allow you to *generate* an instance of such an 
application.  Most tools make that possible by installing a console 
script which can generate a sandbox that can be used to keep application 
state.

Hopefully this is preaching to the choir.

>>> setuptools (or, perhaps, easy_install, although I've seen enough posts
>>> about eggs being uploaded to PyPI to suspect otherwise), encourages
>>> the deployment of system configuration changes that alter the runtime
>>> environment of every single Python application executed on the system.
>>> That's simply not cool.
>>
>> Again, it would help if you tried it in anger.  What's the worst that could
>> happen?  You might like it! ;-)
>
> Oh, believe me, if I ever had distribution needs that required the
> power and flexibility of setuptools, I would reach for it in a
> heartbeat (in fact, I already use it today, albeit for tasks that
> ordinary distutils could probably handle). That said, I do get to
> cheat though - since I don't need to worry about cross-platform
> deployment, I can just use the relevant RPM hooks directly :)

Ideally this is all you'd ever need to care deeply about in an ideal 
world, too, given the separation of installer vs. 
system-packaging-support-tools outlined above.

> You're right that most of my ire should be directed at the default
> behaviour of easy_install rather than at setuptools itself, though. I
> shall moderate my expressed opinions accordingly.

Woot!

- C