[Distutils] RFC: Binary Distribution Format for distutils2/packaging
jim at zope.com
Wed Mar 14 16:43:35 CET 2012
On Wed, Mar 14, 2012 at 6:52 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 14 March 2012 02:50, Carl Meyer <carl at oddbird.net> wrote:
>> In terms of distribution format, this sounds great; I think a clean
>> break and new format that existing systems will just naturally ignore is
>> the way to go.
> Agreed. I created a patch a while ago for a "bdist_simple" format,
> which has languished in the tracker for a while. It's a very basic
> patch, and the format is essentially just a stripped down version of
> the wininst format (a zip file with distinguished placeholder
> directories at the top level). I'm not sure I understand Jim's issues
> with the wininst format, so I can't respond as to why this wouldn't
> address them.
My main issue is that I think we need a standard binary format that
isn't system dependent.
I see the wininst format as sort of a two-edged sword for libraries.
On the one hand, it makes installing libraries natural for Windows
users, on the other, it encourages poor hygiene.
>> Your message doesn't address the on-disk installed format. I hope that
>> the installed format (at least the d2/p default) will be consistent with
>> PEP 376 (unzipped and "flat"), not an egg-style importable zip file
>> relying on its own dedicated entry in a pth file.
> I agree that a flat on-disk format is essential. It's what pip and the
> standard tools use.
> The .pth based egg format appears to only have
> benefits if you want to manage multiple versions, and as far as I can
> see that requirement is now better satisfied via virtualenv/pip.
I don't see how virtualenv/pip address that. I assume you mean
that if 2 applications/scripts require two different versions of a package,
they should use two separate virtualenvs. That feels kinda heavy to me,
but to each their own. :)
FTR, buildout uses eggs to support multiple package versions. It doesn't
use .pth files, but bakes necessary paths into generated scripts.
Buildout on d2/p will continue creating egg-like structures,
both to support multiple versions and to support caching and
sharing installed packages.
> One other point - it's actually relatively easy to write a tool to
> introspect egg and wininst format files, and unpack them manually into
> the on-disk layout needed. I have the bones of such a thing for pip
> (needs completing before I can release it).
> I would suggest that for backward compatibility and transition
> purposes, the following things need to be considered:
> - There should be converters from egg and wininst formats to the new
> format. Not everyone will start producing the new format immediately,
> and we should offer users a migration path where packagers haven't
> moved over. These should be trivial to write.
> - External tools like pip will need to handle the new format, ideally.
> Or at least keep out of the way when pysetup install is used. I'm
> assuming that pysetup install works in a virtualenv.
My hope is that pip will eventually use d2/p.
> Is the intention to promote the new format for older Python versions
> as well, via distutils2,
> or will this be a purely Python 3.3 option?
> If the former, conversion and compatibility is crucial,
> The key here is that the new format needs to be a unifying effort - we
> already have 3 binary formats, if this new format merely adds a
> fourth, then in my view it's a failure. It needs to be the "one
> obvious way" if it's to work.
> PS A couple of relevant python-dev threads, for people who haven't seen them:
Thanks for sharing these.
You raise an interesting point in the last link.
It's reasonable to argue that this is only a windows problem.
It could be argued that, on other platforms:
- Consumers of applications should get application installers,
possibly with embedded copies of Python.
- Consumers of libraries are developers who should be able
to install development tools.
I find the notion that we only need a binary format for Windows
unsatisfying and unsettling, but if everyone else feels that way,
I can live with it.
More information about the Distutils-SIG