[Distutils] RFC: Binary Distribution Format for distutils2/packaging

[Jim Fulton]

On Wed, Mar 14, 2012 at 6:52 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 14 March 2012 02:50, Carl Meyer <carl at oddbird.net> wrote:
>> In terms of distribution format, this sounds great; I think a clean
>> break and new format that existing systems will just naturally ignore is
>> the way to go.
>
> Agreed. I created a patch a while ago for a "bdist_simple" format,
> which has languished in the tracker for a while. It's a very basic
> patch, and the format is essentially just a stripped down version of
> the wininst format (a zip file with distinguished placeholder
> directories at the top level). I'm not sure I understand Jim's issues
> with the wininst format, so I can't respond as to why this wouldn't
> address them.

My main issue is that I think we need a standard binary format that
isn't system dependent.

I see the wininst format as sort of a two-edged sword for libraries.
On the one hand, it makes installing libraries natural for Windows
users, on the other, it encourages poor hygiene.

...

>> Your message doesn't address the on-disk installed format. I hope that
>> the installed format (at least the d2/p default) will be consistent with
>> PEP 376 (unzipped and "flat"), not an egg-style importable zip file
>> relying on its own dedicated entry in a pth file.
>
> I agree that a flat on-disk format is essential. It's what pip and the
> standard tools use.

Um....nevermind.

> The .pth based egg format appears to only have
> benefits if you want to manage multiple versions, and as far as I can
> see that requirement is now better satisfied via virtualenv/pip.

I don't see how virtualenv/pip address that.  I assume you mean
that if 2 applications/scripts require two different versions of a package,
they should use two separate virtualenvs.  That feels kinda heavy to me,
but to each their own. :)

FTR, buildout uses eggs to support multiple package versions. It doesn't
use .pth files, but bakes necessary paths into generated scripts.

Buildout on d2/p will continue creating egg-like structures,
both to support multiple versions and to support caching and
sharing installed packages.

> One other point - it's actually relatively easy to write a tool to
> introspect egg and wininst format files, and unpack them manually into
> the on-disk layout needed. I have the bones of such a thing for pip
> (needs completing before I can release it).
>
> I would suggest that for backward compatibility and transition
> purposes, the following things need to be considered:
>
> - There should be converters from egg and wininst formats to the new
> format. Not everyone will start producing the new format immediately,
> and we should offer users a migration path where packagers haven't
> moved over. These should be trivial to write.

+1

...

> - External tools like pip will need to handle the new format, ideally.
> Or at least keep out of the way when pysetup install is used. I'm
> assuming that pysetup install works in a virtualenv.

My hope is that pip will eventually use d2/p.

> Is the intention to promote the new format for older Python versions
> as well, via distutils2,

Yes

> or will this be a purely Python 3.3 option?

No.

> If the former, conversion and compatibility is crucial,

+1

...

> The key here is that the new format needs to be a unifying effort - we
> already have 3 binary formats, if this new format merely adds a
> fourth, then in my view it's a failure. It needs to be the "one
> obvious way" if it's to work.
>
> Paul.
>
> PS A couple of relevant python-dev threads, for people who haven't seen them:
>
> http://mail.python.org/pipermail/python-dev/2011-October/114285.html
> http://mail.python.org/pipermail/python-dev/2011-October/113956.html
> http://mail.python.org/pipermail/python-dev/2011-October/114243.html

Thanks for sharing these.

You raise an interesting point in the last link.

It's reasonable to argue that this is only a windows problem.

It could be argued that, on other platforms:

- Consumers of applications should get application installers,
  possibly with embedded copies of Python.

- Consumers of libraries are developers who should be able
  to install development tools.

I find the notion that we only need a binary format for Windows
unsatisfying and unsettling, but if everyone else feels that way,
I can live with it.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton