Mailman 3 pip merges wheel - Distutils-SIG

pip merges wheel

Daniel Holth

17 Mar 2013 17 Mar '13

12:06 a.m.

Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need. "pip wheel ..." uses the wheel reference implementation ("pip install wheel") to compile a dependency tree as .whl archives. Used together with "pip install --use-wheel ..." it provides a powerful way to speed up repeated installs and reap other good packaging benefits. I've been using this code in production for months and it works well. I am now a pip maintainer. We are committed to offering excellent wheel support in pip including a good way to produce and consume the format. In the future we will likely refactor the code to offer the same features with more distlib and less setuptools but this change will be mostly transparent to the end user. Once everyone is comfortable with the format we will move towards installing wheels by default when they are available instead of requiring the --use-wheel flag. Enjoy! Please share your experiences with the new feature. The most common issue is that you must install distribute >= 0.6.34 for everything to work well. Daniel Holth

Show replies by thread

Marcus Smith

18 Mar 18 Mar

3:38 a.m.

the pip docs have a cookbook entry now for the wheel support http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing... the usage reference is up to date as well http://www.pip-installer.org/en/latest/usage.html Marcus On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:

...

Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

"pip wheel ..." uses the wheel reference implementation ("pip install wheel") to compile a dependency tree as .whl archives. Used together with "pip install --use-wheel ..." it provides a powerful way to speed up repeated installs and reap other good packaging benefits. I've been using this code in production for months and it works well.

I am now a pip maintainer. We are committed to offering excellent wheel support in pip including a good way to produce and consume the format. In the future we will likely refactor the code to offer the same features with more distlib and less setuptools but this change will be mostly transparent to the end user. Once everyone is comfortable with the format we will move towards installing wheels by default when they are available instead of requiring the --use-wheel flag.

Enjoy! Please share your experiences with the new feature. The most common issue is that you must install distribute >= 0.6.34 for everything to work well.

Daniel Holth _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Glyph

9:08 a.m.

On Mar 16, 2013, at 5:06 PM, Daniel Holth wrote:

...

Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

Hi! I don't really understand all the issues that lead to the creation of the Wheel format, and I don't care to. But, I _very_ much would like some of the advertised benefits, such as being able to install binary packages on Windows. Well, _I_ don't care about installing binary packages on Windows. But I want users of Twisted on Windows to be able to 'pip install twisted' and just get a working install without having to learn about MSVCRT versions and the various miseries thereof. I am quite excited about the possibility that such a situation might be near our grasp. My understanding is that in order to achieve this nirvana, what we must do is: A twisted developer, on each supported Windows configuration, must 'pip install wheel; pip wheel ./Twisted' and place that build artifact on PyPI. Make our dependencies do the same thing. Tell our users to do 'pip install --use-wheel twisted' I have two questions: first, is this sequence of steps accurate, and if it is, why is '--use-wheel' not just the default? Does this option just mean '--please-work --no-really' or is there some functional change in behavior in using wheels that might cause problems? Thanks a lot! -glyph

Paul Moore

10:22 a.m.

On 18 March 2013 09:08, Glyph wrote:

...

My understanding is that in order to achieve this nirvana, what we must do is:

(Daniel may wish to chime in with more details)

...

A twisted developer, on each supported Windows configuration, must 'pip install wheel; pip wheel ./Twisted' and place that build artifact on PyPI.

To create the wheel, just run pip wheel <a twisted sdist>, as you say on each supported Windows configuration. And it doesn't have to just be Windows - if you care to, builds for other platforms can also be created the same way. This may save on the "you need the following dev packages installed" type of FAQs.

...

Make our dependencies do the same thing.

If the dependencies don't, install from source will still work, so it's probably not crucial for pure Python dependencies.

...

Tell our users to do 'pip install --use-wheel twisted'

I have two questions: first, is this sequence of steps accurate,

See above, but basically yes.

...

and if it is, why is '--use-wheel' not just the default? Does this option just mean '--please-work --no-really' or is there some functional change in behavior in using wheels that might cause problems?

It simply reflects that the wheel format and pip's support of it is relatively new, and so somewhat experimental. There's no functional change other than the fact that you get a wheel install if one's available - or at least there shouldn't be :-) In due course, I expect --use-wheel to become the default. We may offer a --no-wheel option at that stage, if it seems useful. Paul

Nick Coghlan

2:46 p.m.

On Sun, Mar 17, 2013 at 8:38 PM, Marcus Smith wrote:

...

the pip docs have a cookbook entry now for the wheel support http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing...

the usage reference is up to date as well http://www.pip-installer.org/en/latest/usage.html

Great news! Given Glyph's questions, It may be a good idea to add a "publishing wheels" entry to the cookbook, explaining: 1. After building the wheel, publishing it to PyPI means users on a matching system can download and install it without building it 2. The current caveats on handling C extensions that are *wrappers* around shared libraries/DLLs that are expected to be installed on the target system, rather than CPython-only C extensions. (Specifically, bundled DLLs on Windows should work just fine, but depending on system binaries on *nix systems can cause problems if other users have the right SO version installed, and bundling on *nix systems may require tweaking of the environment to find the bundled SO instead of only looking in the system paths) 3. Users currently need to add a "--use-wheel" flag to enable the wheel support. It is expected that using wheels will become the default behaviour in the feature, but we want it to be opt in until it has seen some more widespread real word usage. Cheers, Nick.

...

Marcus

On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:

...
Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

"pip wheel ..." uses the wheel reference implementation ("pip install wheel") to compile a dependency tree as .whl archives. Used together with "pip install --use-wheel ..." it provides a powerful way to speed up repeated installs and reap other good packaging benefits. I've been using this code in production for months and it works well.

I am now a pip maintainer. We are committed to offering excellent wheel support in pip including a good way to produce and consume the format. In the future we will likely refactor the code to offer the same features with more distlib and less setuptools but this change will be mostly transparent to the end user. Once everyone is comfortable with the format we will move towards installing wheels by default when they are available instead of requiring the --use-wheel flag.

Enjoy! Please share your experiences with the new feature. The most common issue is that you must install distribute >= 0.6.34 for everything to work well.

Daniel Holth _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Lennart Regebro

9:16 p.m.

On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:

...

Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate. The other option is of course that we start adding all sorts of development commands to pip, such as build, test, sdist etc. But I do think it's the wrong place. //Lennart

Marcus Smith

10:13 p.m.

...

I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate.

I understand where you're coming from, but a few thoughts: 1) pip is *currently* very much a build tool in that it build/installs from source archives, but I understand the new model is for pip to eventually be working with pre-built wheels much of the time, with no build system required. 2) the motivation for "pip wheel" is *not* really for building single wheels for the project you're developing. For that use case, I agree it makes more sense conceptually to install the "wheel" package and use it's "bdist_wheel" setuptools extension (i.e. "python setup.py bdist_wheel" ) 3) the real motivation for "pip wheel" (which is a builder convenience tool) is to help people *install* from wheels *now* given that pypi won't be full of wheels for a time to come. This allows people to quickly and easily get all their source archive dependencies converted with a single command (if you're using requirements files) and start gaining the benefits of wheel with very little fiddling. 4) even when pypi is full of wheels, I can imagine people wanting to build certain dependencies with different build options, and "pip wheel" could help with that. 5) I can imagine "pip wheel" disappearing at some point down the road. We'll have to see, but until then, "pip wheel" is going to very critical IMO in getting people familiar with and actually using wheel-based installs.

...

The other option is of course that we start adding all sorts of development commands to pip, such as build, test, sdist etc. But I do think it's the wrong place.

I've thought of that too, but that's a discussion for another day or parallel universe. Marcus

Nick Coghlan

10:31 p.m.

On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote:

...

...
The other option is of course that we start adding all sorts of development commands to pip, such as build, test, sdist etc. But I do think it's the wrong place.

I've thought of that too, but that's a discussion for another day or parallel universe.

The meta-build hooks are definitely a topic for post-metadata-2.0 :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Eric V. Smith

10:51 p.m.

On 3/18/2013 5:16 PM, Lennart Regebro wrote:

...

On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:

...
Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate.

I completely agree. And those tools to build distributions could be downloaded separately. -- Eric.

Nick Coghlan

11:13 p.m.

On Mon, Mar 18, 2013 at 3:51 PM, Eric V. Smith wrote:

...

On 3/18/2013 5:16 PM, Lennart Regebro wrote:

...
On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:

...
Earlier today we merged the existing wheel branch into mainline pip. This adds opt-in wheel install support (built into pip, "pip install --use-wheel ...") and the convenient "pip wheel ..." command for creating the wheels you need.

I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate.

I completely agree. And those tools to build distributions could be downloaded separately.

As Marcus already noted, we cannot achieve this nirvana until the primary distribution format for Python software is something other than source archives. pip's support of the wheel format is a necessary step on that path - it's already a build system *because* our current "installation" command is "./setup.py install", and handling that command invokes the build system. Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model). Cheers, Nick.

...

-- Eric. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Barry Warsaw

11:37 p.m.

On Mar 18, 2013, at 02:16 PM, Lennart Regebro wrote:

...

I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate.

+1. Doesn't this violate Nick's mission for killing `setup.py install` which IIUC is motivated by wanting to separate building and installing? I really want pip to just be about installing and use other tools to build source and binary distributions. -Barry

Barry Warsaw

11:39 p.m.

On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

...

Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model).

If that's the end goal, then it should be the default now. -Barry

Donald Stufft

11:41 p.m.

On Mar 18, 2013, at 7:39 PM, Barry Warsaw wrote:

...

On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

...
Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model).

If that's the end goal, then it should be the default now.

-Barry _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

--wheel-only as the default now would make approximately 3 things installable from PyPI, one of which is wheel itself. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Lennart Regebro

11:48 p.m.

On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote:

...

1) pip is *currently* very much a build tool in that it build/installs from source archives, but I understand the new model is for pip to eventually be working with pre-built wheels much of the time, with no build system required.

I used the word "build tool" in the packaging summit, that was the wrong word. My point is that is currently a tool to *install* distributions as opposed to *create* distributions. I think it's best if it is kept that way. I'd rather see the wheel command go into another tool used to build distributions (and perhaps upload them to the cheeseshop). [ I overheard the proposal to call such a tool "pup". ;-) ] //Lennart

Nick Coghlan

11:52 p.m.

On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote:

...

On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

...
Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model).

If that's the end goal, then it should be the default now.

No, user experience is king. Right now, defaulting to wheel-only would be an awful user experience (because you wouldn't be able to install anything), as well as being completely backwards incompatible with the current behaviour (because everything would break). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

11:59 p.m.

On Mon, Mar 18, 2013 at 4:48 PM, Lennart Regebro wrote:

...

On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote:

...
1) pip is *currently* very much a build tool in that it build/installs from source archives, but I understand the new model is for pip to eventually be working with pre-built wheels much of the time, with no build system required.

I used the word "build tool" in the packaging summit, that was the wrong word. My point is that is currently a tool to *install* distributions as opposed to *create* distributions. I think it's best if it is kept that way. I'd rather see the wheel command go into another tool used to build distributions (and perhaps upload them to the cheeseshop).

No, that's not the intended use case of "pip wheel". It's intended for use as a multi-stage installation tool. Stage 1: install from sdist on a staging server which has development dependencies installed Stage 2: upload to a private PyPI index you control (perhaps just a directory published internally over HTTP) Stage 3: install from wheel on production servers without development dependencies For actual development use, it makes a lot more sense to install the wheel project and just use ./setup.py bdist_wheel. Cheers, Nick.

...

[ I overheard the proposal to call such a tool "pup". ;-) ]

//Lennart _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Daniel Holth

19 Mar 19 Mar

12:28 a.m.

I do understand the confusion. Binary package formats have more than one use. Coincidentally we have implemented the slightly different "cache compiles" and "distribute software" features using the same format. It might help if you can imagine that "pip wheel" produces a different format than "python setup.py bdist_wheel upload". Daniel

Nick Coghlan

1:15 a.m.

On Mon, Mar 18, 2013 at 5:28 PM, Daniel Holth wrote:

...

I do understand the confusion. Binary package formats have more than one use. Coincidentally we have implemented the slightly different "cache compiles" and "distribute software" features using the same format. It might help if you can imagine that "pip wheel" produces a different format than "python setup.py bdist_wheel upload".

I think it helps more to just imagine that it produces the same format, but for different reasons. That particular imagining has the virtue of being accurate :) We do need to do some work on better explaining that wheel has two major use cases, and that these two use cases correspond to "pip wheel" (for caching your own local builds of an sdist) and "./setup.py bdist_wheel upload" for publication of pre-built binaries via PyPI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Glyph

11:40 p.m.

On Mar 18, 2013, at 4:52 PM, Nick Coghlan wrote:

...

On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote:

...
On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

...
Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model).

If that's the end goal, then it should be the default now.

No, user experience is king. Right now, defaulting to wheel-only would be an awful user experience (because you wouldn't be able to install anything), as well as being completely backwards incompatible with the current behaviour (because everything would break).

But it could default to wheels-and-other-things right now without breaking anything, right? What's the rationale for not just preferring wheels if they're available? -glyph

Daniel Holth

20 Mar 20 Mar

midnight

It's possible to upload broken wheels. I don't want "I had to find the disable flag" to be anyone's first impression. On Mar 19, 2013 7:40 PM, "Glyph" wrote:

...

On Mar 18, 2013, at 4:52 PM, Nick Coghlan wrote:

...
On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote:

...
On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote:

...
Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model).

If that's the end goal, then it should be the default now.

No, user experience is king. Right now, defaulting to wheel-only would be an awful user experience (because you wouldn't be able to install anything), as well as being completely backwards incompatible with the current behaviour (because everything would break).

But it could default to wheels-and-other-things right now without breaking anything, right? What's the rationale for not just preferring wheels if they're available?

-glyph _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Glyph

12:33 a.m.

On Mar 19, 2013, at 5:00 PM, Daniel Holth wrote:

...

It's possible to upload broken wheels. I don't want "I had to find the disable flag" to be anyone's first impression.

It's possible to upload broken sdists, too. Trust me, Windows' users (who do not have C compilers) impression of pip _could not_ get any worse. -glyph

Daniel Holth

12:43 a.m.

we might do different defaults for each platform On Mar 19, 2013 8:33 PM, "Glyph" wrote:

...

On Mar 19, 2013, at 5:00 PM, Daniel Holth wrote:

It's possible to upload broken wheels. I don't want "I had to find the disable flag" to be anyone's first impression.

It's possible to upload broken sdists, too.

Trust me, Windows' users (who do not have C compilers) impression of pip _could not_ get any worse.

-glyph

4054

Age (days ago)

4057

Last active (days ago)

List overview

Download

21 comments

9 participants

participants (9)

Barry Warsaw
Daniel Holth
Donald Stufft
Eric V. Smith
Glyph
Lennart Regebro
Marcus Smith
Nick Coghlan
Paul Moore

pip merges wheel

Marcus Smith

Marcus Smith

tags

participants (9)