On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum <guido@python.org> wrote:

I don't know or care much about PyPI metadata, so do what you feel is right. If you are uncomfortable being PEP-uncle *and* -author, find another author or another uncle. But since it doesn't affect the language or library, it's fine with me if you are both. :-)

To clarify what will happen if I handle both jobs: I'll bug a few specific people like Vinay Sajip (distlib), and MvL or Richard Jones (PyPI) to at least glance over it, as well as giving distutils-sig, catalog-sig and python-dev one last chance to comment on my revised draft. Then, wearing my BDFL-Delegate hat, I'd decide what feedback I considered worth addressing and what could be safely ignored (or postponed to the next time the metadata gets updated). I don't expect anything I want to do to be particularly controversial, but I think it's worth trying to get it right (even if it delays wheel support in pip for a few more weeks). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 3 February 2013 11:27, Antoine Pitrou <solipsis@pitrou.net> wrote:

...

I don't expect anything I want to do to be particularly controversial, but I think it's worth trying to get it right (even if it delays wheel support in pip for a few more weeks).

Will wheel be implemented in distutils?

There are no plans to do so (and I don't think it's necessary, particularly given the general "distutils is frozen" principle). To clarify: - The wheel project provides a bdist_wheel command to build wheels, but this depends on setuptools/distribute. I don't know how hard it would be to write a version that didn't depend on them. - The wheel project provides standalone commands to install wheels, and to convert eggs and wininst installers to wheels. The pip integration is basically to allow pip to find wheels on PyPI or any local indexes you have, and to install them via the "pip install" command. It also incidentally includes pip support for the new dist-info metadata format (I can't recall which PEP that is, sorry) simply because that's the format that wheel uses rather than the legacy egg-info format. So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted. Paul

On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Sun, 3 Feb 2013 12:34:36 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted.

But it's an "accepted format" by whom? If we accept the PEP but refuse to integrate support in the stdlib, it doesn't make much sense.

I vote for removing the "distutils is frozen" principle. It was already destructive enough when packaging was still alive - bug fixes were actively turned down for the dubious reason that fixing bugs would break compatibility. Its rationale has become completely obsolete now that packaging has failed.

The rationale for the distutils freeze is "don't break setuptools". That rationale still holds. The most likely current candidate for integration into Python 3.4 is distlib, which, as was proposed when packaging was dropped, is built incrementally as components are standardised, rather than being a mish-mash of updated code and flawed design inherited from distutils. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/03/2013 08:09 AM, Antoine Pitrou wrote:

I'm sure it is perfectly possible to evolve and bugfix distutils without breaking distribute. What's more, distribute is actually maintained and can evolve to accomodate the fixes.

I wouldn't be on that proposition: there are a lot of burnouts down that rabbit hole.

As for setuptools (as opposed to distribute), I don't think we should care anymore.

Yes, you need to care. It is *still* true today that distribute and setuptools remain largely interchangeable, which is the only thing that makes distribute viable, in the ecosystem sense. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEOZakACgkQ+gerLs4ltQ5N7ACgv/v0GTs7xyQ7I/jvrcu/s2i5 kHcAoJUKigWKZTiVr1ARMLt2xtQF+TAL =mjAm -----END PGP SIGNATURE-----

On Sat, Feb 23, 2013 at 5:29 AM, Chris Withers <chris@simplistix.co.uk> wrote:

...but let's make sure we keep caring about the tools that people really use, which includes both setuptools and distribute.

The lack of a meaningful transition plan is where I think we fell down with PEP 345 & 386, and is also the main reason I've postponed accepting PEP 426 (PJE and others pointed out a couple of major gaps that either made it difficult to update pkg_resources to read the new format, or else made it difficult to generate metadata that existing releases of pkg_resources would understand). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

The neat thing about wheel is that you can install them without having the software used to build them. So we might try to provide a very simple wheel installer script with Python that did not even depend on DistUtils. You would be able to install pip etc with that tool. There is no need to put wheel directly into DistUtils. On Feb 3, 2013 7:54 AM, "Antoine Pitrou" <solipsis@pitrou.net> wrote:

On Sun, 3 Feb 2013 12:34:36 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted.

But it's an "accepted format" by whom? If we accept the PEP but refuse to integrate support in the stdlib, it doesn't make much sense.

I vote for removing the "distutils is frozen" principle. It was already destructive enough when packaging was still alive - bug fixes were actively turned down for the dubious reason that fixing bugs would break compatibility. Its rationale has become completely obsolete now that packaging has failed.

Regards

Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

My position is that these days distutils doesn't belong in the standard library any more than Django does. So I am mildly opposed to supporting it when you should be using better designed third party tools like Bento or setuptools. Wheel makes it possible for Python to get out of the build tool business. Just install your preferred tools with a concise bootstrap installer. On Sun, 3 Feb 2013 09:41:29 -0500 Daniel Holth <dholth@gmail.com> wrote:

The neat thing about wheel is that you can install them without having the software used to build them. So we might try to provide a very simple wheel installer script with Python that did not even depend on DistUtils. You would be able to install pip etc with that tool. There is no need to put wheel directly into DistUtils.

But you still need to *produce* wheels, don't you? It would be more useful for everyone if that facility was bundled with Python. Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

Éric Araujo <merwok <at> netwok.org> writes:

Looks like we agree that a basic tool able to bootstrap the packaging story is needed :)

Agreed. Just because distutils can't easily/reliably build things that are better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have the ability to build pure-Python distributions and distributions including C libs and extensions, with the ability to extend easily by third-party tools. It just needs to be done in a way which is easy to build on, so the included battery stays small and simple. Easier said than done, I know :-) Regards, Vinay Sajip

On Fri, Feb 15, 2013 at 12:27 PM, Erik Bray <erik.m.bray@gmail.com> wrote:

On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

...

Éric Araujo <merwok <at> netwok.org> writes:

...

Looks like we agree that a basic tool able to bootstrap the packaging story is needed :)

Agreed. Just because distutils can't easily/reliably build things that are better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have the ability to build pure-Python distributions and distributions including C libs and extensions, with the ability to extend easily by third-party tools. It just needs to be done in a way which is easy to build on, so the included battery stays small and simple. Easier said than done, I know :-)

Regards,

Vinay Sajip

Sorry to revive an old-ish discussion--I'm just catching up on things. But I just wanted to add that distutils is still pretty okay for building reasonably complex projects. Although it does not rise to the level of complexity of Numpy or SciPy, the Astropy project (https://github.com/astropy/astropy) has managed to put together a pretty nice build system on top of mostly-plain distutils (it does use distribute but primarily just for 2to3 support).

This has necessitated a number of hacks to overcome shortcomings and bugs in distutils, but most of those shortcomings could probably be fixed in distutils within the framework of a slightly lifted freeze. But in any case I haven't found it worthwhile to switch to something like bento when the batteries included in the stdlib have been mostly Good Enough. Having fewer installation dependencies has also made it significantly easier for non-advanced users to install. Even the distribute requirement doesn't add too much overhead, as most users have it on their systems by default now, and for those who don't distribute_setup.py works okay.

TL;DR, strong -1 on the stdlib "getting out of the build business". Also as I think Nick already mentioned one of the wins of Setup-Requires-Dist is to have a standard way to bring in extra build requirements (such as bento) so if we have better support for a feature like that it's not necessary to "bless" any preferred tool.

Distutils is not really going away. We need it to build the existing 28,000 packages. However empirically it seems if you try to write a significant extension to or improvement of distutils then you are likely to get burnt out and switch careers. Instead of literally killing distutils we hope to make it very easy to use other build tools when you need them and not use any build tools at all when you don't. As a thought experiment: what if one of those third party build tools hosted on pypi was distutils itself? What would you need to do to make that happen? The packaging peps PEP-376 and so on are brilliant because they are simple enough to be implemented twice. If we had better ways to separate interface from implementation in Python I'd like to see two of whatever else we come up with for packaging.

Daniel Holth <dholth@gmail.com> writes:

Wheel makes it possible for Python to get out of the build tool business. Just install your preferred tools with a concise bootstrap installer.

If this is true, it would also have been possible with eggs, yet it didn't happen. Why do you think it will happen now or am I missing something?

On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth <dholth@gmail.com> wrote:

Bento is the only available packaging tool to heap praise onto and it is impressive.

If Bento is cool, is there some way we can help it gain more traction in the Python ecosystem? Not necessarily by incorporating it into stdlib, but perhaps by officially sanctioning it in other ways (documentation, PyPI, perhaps getting some helpful hooks / tweaks to Python itself)? I don't know the answer to these questions, but if there is a good solution out there, it would be cool to throw our community weight behind it. Schiavo Simon

Daniel Holth <dholth <at> gmail.com> writes:

Bento is the only available packaging tool to heap praise onto and it is impressive. I am reacting to all the hate heaped on setup tools when I think the underlying DistUtils design is a big part of the problem. My feeling is that stdlib packaging tools should be for bootstrapping and reference, more like wsgiref than django.

Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the somewhat idiosyncratic source style, David Cournapeau himself points to what he regards as weaknesses in it[1]. It is no doubt a better build tool than distutils for NumPy/SciPy users, and that's great. However, from what I could see, the documentation is not very detailed, making it hard to evaluate, and much of the build internals seem to be in some other tool called yaku, for which I couldn't find much documentation. When I see a sentiment suggesting that "Python should get out of the build tools business" (paraphrasing what Daniel said, not quoting him), I can certainly see why the distutils compiler infrastructure would make one think that, if it was impossible to come up with something better. However, I do feel there is room for a battery included in this area, as long as it is simple and easier to build on than distutils has been. Although I have not started to add anything in this area to distlib, I have spent some time thinking about the problems, and I have some experimental work in mind. I don't believe that "hate" is the right word when talking about why setuptools isn't loved more by python-dev. Since it was extending the flawed distutils design, it doesn't make sense to have it getting any closer to the stdlib, and it could perhaps be argued that distutils2/packaging also failed to make sufficient progress for the same reason. Certainly, with distlib I've tried to incorporate a lot of the setuptools functionality which developers find useful - in-package data, package exports (entry points), wrapping callables with scripts and so on. Regards, Vinay Sajip [1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento

Simon Cross <hodgestar+pythondev <at> gmail.com> writes:

For the record, all the reasons listed at [1] appear trivial.

In Bento's author's own words - "Weak documentation", "Mediocre code quality", "at a lower level, a lot of code leaves to be desired" may be trivial if David is just being self-deprecating, but what if he isn't? Or perhaps that part of the page is out of date, and needs updating? I can certainly agree with the "Weak documentation" part of the assessment, but this makes it hard to assess the package as a whole. Note that I'm not sniping - writing good documentation is hard work. Regards, Vinay Sajip

On Sun, Feb 3, 2013 at 10:34 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Simon Cross <hodgestar+pythondev <at> gmail.com> writes:

...

For the record, all the reasons listed at [1] appear trivial.

In Bento's author's own words - "Weak documentation", "Mediocre code quality", "at a lower level, a lot of code leaves to be desired" may be trivial if David is just being self-deprecating, but what if he isn't? Or perhaps that part of the page is out of date, and needs updating? I can certainly agree with the "Weak documentation" part of the assessment, but this makes it hard to assess the package as a whole. Note that I'm not sniping - writing good documentation is hard work.

You are putting the words out of the context in which those were written: it is stated that the focus is on the general architecture and low-coupling which are the main issues I saw with distutils. Bento is designed to use multiple build backends (it can use distutils to build C extensions, or waf, the latter being how numpy/scipy is being built with bento). FWIW, I am not in favor of having bento blessed (or any other tool for that matter). The fundamental mistake of the previous attempts at packaging has been to formalize too early, or impose de-facto standards without much specification. That's why wheel and similar efforts are the way forward: they tackle a narrow but well defined sub-problem of packaging. Thus, they can be reused by other libraries to build higher abstractions. They are also less prone to the 'fatigue' that often arise in packaging efforts. David

David Cournapeau <cournape <at> gmail.com> writes:

You are putting the words out of the context in which those were written: it is stated that the focus is on the general architecture

OK, no offence was meant. Thanks for the clarification. Regards, Vinay Sajip

On 3 February 2013 14:41, Daniel Holth <dholth@gmail.com> wrote:

The neat thing about wheel is that you can install them without having the software used to build them. So we might try to provide a very simple wheel installer script with Python that did not even depend on DistUtils. You would be able to install pip etc with that tool. There is no need to put wheel directly into DistUtils.

The other side of the equation, adding a bdist_wheel command to distutils should *also* be simple. As command classes are essentially independent code, it's not even really going to violate whatever remains of the "distutils is frozen" philosophy. Is there any mileage to submitting a patch to the Python stdlib that implements a bdist_wheel distutils subcommand and a wheel installer in the stdlib? I'm willing to write that patch based on Daniel's existing wheel code if no-one else is willing to, but only if there's likely to be some level of support from python-dev (for example, if Antoine and Nick are in general agreement on the principle, I'd be willing to write the code and we could see where the subsequent debate leads). Paul

On Sun, 3 Feb 2013 12:34:36 +0000 Paul Moore <p.f.moore@gmail.com> wrote:

...

So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted.

But it's an "accepted format" by whom? If we accept the PEP but refuse to integrate support in the stdlib, it doesn't make much sense. There’s a chicken and egg problem: python-dev could have waited a year or two to see the wheel format being used and fixed/improved before deciding to do something in the stdlib, but the pip maintainers wanted an official blessing first. Accepting the wheel PEP would mean that

I vote for removing the "distutils is frozen" principle. I’ve also been thinking about that. There have been two exceptions to

Hi, I don’t have a problem with Nick improving the PEP and deciding on its acceptance. Le 03/02/2013 07:48, Antoine Pitrou a écrit : python-dev and the community think wheels are an interesting way forward, and after some time needed to find out what fixes and improvements are needed, python-dev is ready to add stable, official support in the stdlib. (I know Daniel Holth is frustrated by the wait for what he sees as a simple format that’s been stable since last year, but it’s really not long for stdlib inclusion, and there *are* things to sort out.) the freeze, for ABI flags in extension module names and for pycache directories. When the stable ABI was added and MvL wanted to change distutils (I don’t know to do what exactly), Tarek stood firm on the freeze and asked for any improvement to go into distutils2, and after MvL said that he would not contibute to an outside project, we merged d2 into the stdlib. Namespace packages did not impact distutils either. Now that we’ve removed packaging from the stdlib, we have two Python features that are not supported in the standard packaging system, and I agree that it is a bad thing for our users. I’d like to propose a reformulation of the freeze: - refactorings for the sake of cleanup are still shunned - fixes to really old bugs that have become the expected behavior are still avoided - fixes to follow OS changes are still allowed (we’ve had a number for Debian multiarch, Apple moving stuff around, Windows manifest options changes) - support for Python evolutions that involve totally new code, commands or setup parameters are now possible (this enables stable API support as well as a new bdist format) - behavior changes to track Python behavior changes are now possible (this enables recognizing namespace packages, unless we decide they need a new setup parameter) We’ll probably need to talk this over at PyCon (FYI I won’t be at the language summit but I’ll take part in the packaging mini-summit planned thanks to Nick). Regards

I did think that updating distutils to have basic support for the packaging PEPs was a decent idea, but then it wound up being more or less rewritten entirely like I've been cajoled into doing with PEP 426 (Metadata). I don't know whether distutils(1) can survive the minimum changes required for wheel (.dist-info folders and a bdist_wheel command) without breaking a significant number of its clients or not. History suggests no. It may still be worth a shot. bdist_wheel started out as a distutils2 plugin but it was inconvenient to develop it inside the CPython repository and it was not practical to convert projects over from setuptools to distutils2. The pypi metadata has had a standardlib generator for a long time without having anything in the standard library to make use of the metadata. Wheels are just .zip files of one package's share of the site-packages directory. Why not support them asymmetrically. I value the PEP and consensus for the format more than the implementation. One idea that's sure to be controversial would be to include a bootstrap installer as a script without committing to a Python API (just a command-line API). Since Python 2.6, wheel's own wheel archive can be used to install wheels: $ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ... On Sun, Feb 3, 2013 at 2:40 PM, Chris Jerdonek <chris.jerdonek@gmail.com>wrote:

On Sun, Feb 3, 2013 at 10:33 AM, Éric Araujo <merwok@netwok.org> wrote:

...

...

I vote for removing the "distutils is frozen" principle. I’ve also been thinking about that. There have been two exceptions to

Le 03/02/2013 07:48, Antoine Pitrou a écrit : the freeze, for ABI flags in extension module names and for pycache directories. When the stable ABI was added and MvL wanted to change distutils (I don’t know to do what exactly), Tarek stood firm on the freeze and asked for any improvement to go into distutils2, and after MvL said that he would not contibute to an outside project, we merged d2 into the stdlib. Namespace packages did not impact distutils either. Now that we’ve removed packaging from the stdlib, we have two Python features that are not supported in the standard packaging system, and I agree that it is a bad thing for our users.

I’d like to propose a reformulation of the freeze:

This could be common knowledge, but is the current formulation of the freeze spelled out somewhere?

--Chris _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com

Le 06/02/2013 05:03, Chris Jerdonek a écrit :

I asked this earlier, but didn't see a response. Is the freeze stated somewhere like in a PEP?

It’s part in notes from the PyCon 2010 Language Summit, part in unwritten policy in the heads of people involved in distutils bugs these last years.

If not, can someone state it precisely (e.g. what's allowed to change and what's not)?

Bug fixes are possible; changes to keep working with Python (i.e. pyc files are generated in pycache directories after 3.2); changes to be able to build on current OSes (e.g. following Mac dev tools location change, introduction of Debian multiarch, etc.). Some bugs have been here for so long that everybody depends on them or work around them, or they would be extremely painful to fix (e.g. in the option parsing code) for little benefit, so they are wontfix. Cleanups, refactorings and improvements were banned by the feature freeze. Regards

On Sun, Feb 3, 2013 at 10:34 PM, Paul Moore <p.f.moore@gmail.com> wrote:

So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted.

And they're right to be concerned - I've just made it clear to Daniel that before PEP 427 will be accepted, it must either switch to using S/MIME for signatures and drop support for Java Web Signatures completely, or else it must contain a compelling rationale for why we should even be considering a signature scheme that isn't yet an IETF standard. I take the disclaimer the IETF put on their drafts seriously: "It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." MvL raised this concern last time the wheel format was discussed, and, to date, nothing has happened to address it. JWS *does* look like a neat piece of technology, but it's just too young to be basing our binary distribution infrastructure on it (especially as new crypto is, by default, bad crypto - that's why NIST/NSA hold their multi-year competitions when they need to come up with new crypto related algorithms). The other two PEPs (the new metadata and the version compatibility tags) are in a much better place. Most of the issues with PEP 426 have been inherited from the previous version of the metadata, rather than being related to the changes Daniel needed for the wheel format, and I've just completed a new draft that should address most of those problems. It's been a while since I looked closely at the compatibility tag PEP, but I don't recall their being any significant problems with it last time it was discussed. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

The pip integration is basically to allow pip to find wheels on PyPI or any local indexes you have, and to install them via the "pip install" command.

it also offers "pip wheel" for building wheels (using bdist_wheel) locally for your requirements, since wheels wouldn't be pervasive on PyPI for a while so, let's say you have a large app that has a requirements file, that ultimately installs 90 packages. run: pip wheel --wheel-dir=/my_wheels -r requirements.txt this will produce 90 wheel archives in a local directory, that you can then install against. Marcus

They can be signed with pypi detached signatures already. It works now exactly as for sdist. The innovation was supposed to be in convenience for the signer, in allowing keys to be trusted per package and for a list of dependencies and the expected signing keys to be shared easily. Does anyone have a concise pure py3 s/mime implementation? On Feb 3, 2013 4:52 PM, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:

Paul Moore <p.f.moore <at> gmail.com> writes:

...

So it's perfectly possible to use wheels right now, without the pip integration. But the pip developers don't want to integrate the wheel format just because it exists - they want the assurance that it's an accepted format supported by PEPs, hence the interest in getting the 3 wheel PEPs (of which the metadata PEP is the first) accepted.

Likewise, I will look at the possibility of providing wheel support in distlib, once it has been accepted as a standard and the open issues (such as signature scheme) have been resolved.

Regards,

Vinay Sajip

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com