If you want wheel to be successful, provide a build server.
If you want wheel to be successful, **provide a build server**.
Quoting the author of psutil:
https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
{{{ On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry). What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip. }}}
What do you think?
Regards, Thomas Güttler
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
25.05.2016, 10:13, Thomas Güttler kirjoitti:
If you want wheel to be successful, **provide a build server**.
Quoting the author of psutil:
https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
{{{ On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry). What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip. }}}
What do you think?
Regards, Thomas Güttler
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
Regards, Thomas Güttler
On 25 May 2016 at 14:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
The other aspect of this is who has sufficient time/expertise to set something like this up? Are you volunteering to do this? Paul
Am 25.05.2016 um 15:55 schrieb Paul Moore:
On 25 May 2016 at 14:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
The other aspect of this is who has sufficient time/expertise to set something like this up? Are you volunteering to do this?
I am volunteering for doing coordination work: - communication - layout of datastructures - interchange of datastructures. - no coding
But we need at least ten people how say "I'm willing to help"
Regards, Thomas Güttler
On Wed, May 25, 2016 at 11:22 AM, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 15:55 schrieb Paul Moore:
On 25 May 2016 at 14:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
The other aspect of this is who has sufficient time/expertise to set something like this up? Are you volunteering to do this?
I am volunteering for doing coordination work:
- communication
- layout of datastructures
- interchange of datastructures.
- no coding
But we need at least ten people how say "I'm willing to help"
We (scientific Python folks) have been thinking about this too [1].
We're getting to the stage where the informal methods we have been using are difficult to coordinate. As more people provide wheels, it gets more common for packages to release source before wheels, and cause a cry of pain from users whose installation suddenly fails or changes.
The situation got particularly bad when Python 3.5 was released, because none or very few of us had Python 3.5 packages ready, and so nearly all new installs were suddenly not getting wheels, and suffering.
We all really need some system that can, from some simple trigger, like a push of a git tag, build wheels for Windows 32 + 64, OSX, and manylinux1_x86_64, for Pythons 2.7, 3.4 and 3.5 (and 3.6?), test these installs, and, if the tests pass, and push these either to an accessible spot (we have been using donated rackspace hosting) or to pypi directly.
We're fairly close to that at the moment. Many of us are building and testing binaries on Appveyor, Travis by default. We have lots of projects set up with repos MacPython github org, where the repositories only exist to build, test OSX wheels on travis-ci OSX VMs, and push the wheels to rackspace - e.g. [2]. We have been setting up similar systems for manylinux - e.g. [3, 4].
The problem is that these systems are largely manual, in that a release for e.g. numpy involves:
* Trigger build / test on separate Appveyor repo; * Trigger build / test on MacPython/numpy-wheels; * Trigger build on manylinux-builds / test on manylinux testing; * When all these are done and tests passing, locate generated binaries on rackspace / Appveyor, and upload to pypi.
This is complicated, and it's relatively hard for a given package to set this up for themselves.
When Python 3.6 comes out, we'll all have to do this release procedure at more or less the same time.
So, I would love to have a system that could either collate these different services (Appveyor, Travis, Circle-CI, Rackspace, AWS) into something coherent, or generate something new that is more streamlined. See conda-forge [5] for an example of collating build services.
I think it's fine for each package to specify its own build and test recipes as long as they can do it in a way that is well defined, with examples to work from. The success of travis-ci is a testament to the ingenuity of packagers in getting their packages built and tested.
Maybe this could be a PEP of its own. It would certainly help to have some idea of what kind of support the PSF can give - the spec would look different for a new custom system and a system collating Appveyor / Travis etc.
I'm certainly happy to devote time to this (in the hope of saving a lot of time later).
Best,
Matthew
[1] https://github.com/scipy/scipy/issues/6157#issuecomment-219314029 [2] https://github.com/MacPython/numpy-wheels [3] https://github.com/matthew-brett/manylinux-builds [4] https://github.com/matthew-brett/manylinux-testing [5] https://conda-forge.github.io/
On May 25, 2016, at 8:22 AM, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 15:55 schrieb Paul Moore:
On 25 May 2016 at 14:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
The other aspect of this is who has sufficient time/expertise to set something like this up? Are you volunteering to do this?
I am volunteering for doing coordination work:
- communication
- layout of datastructures
- interchange of datastructures.
- no coding
But we need at least ten people how say "I'm willing to help"
Short answer: this is not how anything works.
Long answer: This is not a question of getting some number of people to help. If you can clone us a small army of Donalds, Nicks, and Richards then we _might_ be able to pull this off. The money isn't the problem per se, it's the human cost in upkeep for a system designed explicitly to run hostile code safely.
--Noah
On 25 May 2016 at 23:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
No, money for a project of this scale is not available (the person you spoke to may have been thinking of PSF development grants, which typically only cover 4-12 weeks of dedicated development work by a single developer on a project with specific near term objectives, not running on ongoing service that would dwarf PyPI itself in complexity)
Regards, Nick.
Am 25.05.2016 um 15:56 schrieb Nick Coghlan:
On 25 May 2016 at 23:42, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
Amen to that, but who will pay for it? I imagine a great deal of processing power would be required for this. How do implementors of other languages handle this?
I talked with someone who is member of the python software foundation, and he said that money for projects like this is available. Of course this was no official statement.
No, money for a project of this scale is not available (the person you spoke to may have been thinking of PSF development grants, which typically only cover 4-12 weeks of dedicated development work by a single developer on a project with specific near term objectives, not running on ongoing service that would dwarf PyPI itself in complexity)
Yes, running a permanent build-server might be too high regular expenses.
I think providing a self hostable build server which can be started with one command would be such a project.
Regards, Thomas Güttler
On Wed, May 25, 2016 at 8:27 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I think providing a self hostable build server which can be started with one command would be such a project.
The manylinux Docker container is heading in that direction already. I suggest you consider helping out with that effort.
There is also the conda ecosystem, with the conda-forge project using public CI systems to build packages:
https://conda-forge.github.io/
I imagine a similar system could be adopted for wheels....
-CHB
Regards, Thomas Güttler
-- Thomas Guettler http://www.thomas-guettler.de/ _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
For the build, one could use continuous integration providers like appveyor, circleci and travisci to perform the builds. This is what has been done by conda folks with conda-forge.
On Wed, May 25, 2016 at 11:51 AM, Chris Barker chris.barker@noaa.gov wrote:
On Wed, May 25, 2016 at 8:27 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I think providing a self hostable build server which can be started with one command would be such a project.
The manylinux Docker container is heading in that direction already. I suggest you consider helping out with that effort.
There is also the conda ecosystem, with the conda-forge project using public CI systems to build packages:
https://conda-forge.github.io/
I imagine a similar system could be adopted for wheels....
-CHB
Regards, Thomas Güttler
-- Thomas Guettler http://www.thomas-guettler.de/ _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Wed, May 25, 2016 at 10:13 AM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry).
He may accept a PR with Travis configuration to build the wheels though ... Using that manylinux docker image should be easy enough on Travis.
On the other hand, there might be weird dependencies and the wheel don't build well on the manylinux container - so there's still some extra effort for the maintainer.
A simpler package would probably make a better discussion.
PS. To build all the manylinux wheels for a package you only need to run something like: docker run -itv $(pwd):/code quay.io/pypa/manylinux1_x86_64 bash -c 'set -eux; cd code; rm -rf wheelhouse; for variant in /opt/python/*; do rm -rf dist build *.egg-info && $variant/bin/python setup.py clean --all bdist_wheel; auditwheel repair dist/*.whl; done; rm -rf dist build *.egg-info'
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 25 May 2016 at 08:57, Ionel Cristian Mărieș distutils-sig@python.org wrote:
He may accept a PR with Travis configuration to build the wheels though ... Using that manylinux docker image should be easy enough on Travis.
A PR to the packaging user Guide explaining how to build Linux (manylinux) wheels using Travis would be a worthwhile addition (it would sit nicely alongside https://packaging.python.org/en/latest/appveyor/ which explains how to do the same for Windows using Appveyor).
Paul
On May 25, 2016, at 12:13 AM, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Quoting the author of psutil:
https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
{{{ On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry). What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip. }}}
What do you think?
The problems haven't really changed every time someone brings this up. Running untrusted code from the internet isn't impossible (eg. Travis, Heroku, Lambda) but it requires serious care and feeding at a scale we don't currently have the resources for. Until something in that equation changes, the best we can do it try to piggyback on an existing sandbox environment like Travis.
--Noah
On 25/05/16 09:13, Thomas Güttler wrote:
If you want wheel to be successful, **provide a build server**.
Quoting the author of psutil:
https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
{{{ On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that front (sorry). What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch version you have on the server, upload it on the server and install it with pip. }}}
What do you think?
This is something best left to operating systems/package managers, e.g. Debian, Ubuntu, Fedora, RedHat, Guix(SD), Nix(OS). Contrary to the quote above, I believe psutil is available from all of the projects above (in a binary and source format).
These projects have the expressiveness to reason about packages that are not just pure Python (e.g. that depend on GCC), and the software and infrastructure (e.g. build servers) to manage this.
In my mind providing a build server is just not in scope for Python, or PyPI.
On 25 May 2016 at 17:13, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Thomas, aside from that statement being demonstrably untrue (since the wheel format has already proven to be wildly successful, even with developers coping with Linux ABI fragmentation), an attitude of "give me more free stuff, or your project will fail" is not an acceptable tone to adopt on this list.
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
Regards, Nick.
Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
On 25 May 2016 at 17:13, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Thomas, aside from that statement being demonstrably untrue (since the wheel format has already proven to be wildly successful, even with developers coping with Linux ABI fragmentation), an attitude of "give me more free stuff, or your project will fail" is not an acceptable tone to adopt on this list.
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
You seem to be angry. Why?
I am just bringing information from A to B (from github issue to this list).
I guess the author of psutil is just a volunteer like you are.
What is the problem? I think the problem is that creating wheels for psutil is difficult.
I think we all agree on this.
If not, if you think it is easy, then please tell us.
My intention was, to find a solution.
Regards, Thomas Güttler
On 25 May 2016 at 16:11, Thomas Güttler guettliml@thomas-guettler.de wrote:
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
You seem to be angry. Why?
I don't know about Nick, but your tone is certainly annoying to me. I do this work in my (very limited!) spare time, and in most cases, what I do is of no benefit to me personally, but is purely to help others who raise issues.
Your comment "If you want wheel to be successful, **provide a build server**" is both patronising (do you think no-one here is aware that a build server would be beneficial?), inaccurate (wheels *are* successful - ask anyone who's got benefit from them, it's not hard to find such people) and demanding (you suggest others need to do work, but offer none of your own time to help).
Many of your postings here have had a similar tone - insisting that things "have to be" done a certain way, or that there "must" be a solution in the form you prefer. I've tended to assume that this has simply been unintentional, and maybe getting exaggerated by the lack of non-verbal cues available in email, but you need to consider your words more carefully if you don't want to start seriously offending people. And that's not behaviour that's acceptable on this list (or indeed, any of the other Python lists).
I am just bringing information from A to B (from github issue to this list).
Your information was phrased as a demand. If you don't think that's the case, I encourage you to re-read your own post.
I guess the author of psutil is just a volunteer like you are.
Yes, and (on the issue you linked to) he politely explained why he didn't provide Linux binaries and offered some approaches you could take. You seemed to take that as somehow implying that he was dissatisfied and posted "information" here implying that. I hope you didn't mean it that way, but in doing so, as far as I can tell you misrepresented him.
What is the problem? I think the problem is that creating wheels for psutil is difficult.
Hardly. The problem is that Unix systems are very diverse, and trying to maintain compatible builds for all the variations is more work than a single volunteer can manage. The people working on manylinux on this mailing list have put a *lot* of work into trying to alleviate that issue. Maybe the psutil author hasn't heard of this, or maybe he's not had the time to investigate or take advantage of it. Or maybe he doesn't feel (or indeed, he's sure - after all, psutil is a very low-level package) that manylinux isn't the right answer for his project.
But just because you personally weren't willing to set up the prerequisites to build a copy of psutil for yourself, doesn't mean that "creating wheels for psutil is difficult".
I think we all agree on this.
No.
If not, if you think it is easy, then please tell us.
Building and publishing wheels for every platform variation your user base might be interested *is* difficult, simply because of the scale of the problem and the fact that most volunteer developers don't have access to all those platforms. And yes, a build farm would help those people. But it's not anyone's *responsibility* to provide that. Indeed, the whole open source ecosystem is built on people seeing a need and *putting their own effort into making it happen, and then sharing the results*. You seem to think that your contribution should be to nag people who are already providing freely of their time, to do more, so that you don't have to.
My intention was, to find a solution.
Well, it didn't come across that way.
From another post: I am volunteering for doing coordination work:
- communication
- layout of datastructures
- interchange of datastructures.
- no coding
But we need at least ten people how say "I'm willing to help"
We don't need project managers for this task. We need people willing to code, build servers, etc. If (and it's a big if) the management of the work becomes too much of an overhead for those people, then (and *only* then) they might appreciate some help with the administrative tasks. But volunteering to co-ordinate resources that don't exist is not useful. Nor is trying to find such resources from the pool of people who are *already* doing plenty. If you want to find resources, I suggest you look among people who are not yet contributing to the packaging ecosystem.
Paul
On 05/25/2016 11:48 AM, Paul Moore wrote:
I don't know about Nick, but your tone is certainly annoying to me. I do this work in my (very limited!) spare time, and in most cases, what I do is of no benefit to me personally, but is purely to help others who raise issues.
I wonder if this is an issue of English being Thomas' second language. By his email address, he appears to reside in Germany. So maybe something has been lost in translation?
*Randy Syring* Husband | Father | Redeemed Sinner
/"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/
On 25 May 2016 at 19:03, Randy Syring randy@thesyrings.us wrote:
On 05/25/2016 11:48 AM, Paul Moore wrote:
I don't know about Nick, but your tone is certainly annoying to me. I do this work in my (very limited!) spare time, and in most cases, what I do is of no benefit to me personally, but is purely to help others who raise issues.
I wonder if this is an issue of English being Thomas' second language. By his email address, he appears to reside in Germany. So maybe something has been lost in translation?
I've certainly been happy to assume that it is, for the most part. However, I do believe that there comes a point where even if there are language issues, it's important to let people know how their comments are coming across. I hope Thomas took my post in that vein.
Paul
On 26 May 2016 at 01:11, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
On 25 May 2016 at 17:13, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Thomas, aside from that statement being demonstrably untrue (since the wheel format has already proven to be wildly successful, even with developers coping with Linux ABI fragmentation), an attitude of "give me more free stuff, or your project will fail" is not an acceptable tone to adopt on this list.
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
You seem to be angry. Why?
Thomas, I am pointing out that your current exhibition of entitled behaviour across multiple venues (as represented by the specific sentence I quoted) is problematic. Please stop trying to crack the whip and generate an artificial sense of urgency - software publication and distribution is a complex problem, and most of the current challenges in the PyPI ecosystem stem from an ongoing lack of funding which requires organisational change moreso than technical change. While various folks are working on that, it's mostly not a distutils-sig level problem, but rather a question for the PSF and for commercial Python redistributors.
If you're looking to provide information, or ask if a particular solution that seems obvious to you would be feasible in practice, then do that. The one thing I am asking you to STOP doing is combining your questions with exaggerated hyperbole that's designed to make volunteers feel bad.
Regards, Nick.
Hi,
On Thu, May 26, 2016 at 1:57 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 26 May 2016 at 01:11, Thomas Güttler guettliml@thomas-guettler.de wrote:
Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
On 25 May 2016 at 17:13, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Thomas, aside from that statement being demonstrably untrue (since the wheel format has already proven to be wildly successful, even with developers coping with Linux ABI fragmentation), an attitude of "give me more free stuff, or your project will fail" is not an acceptable tone to adopt on this list.
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
You seem to be angry. Why?
Thomas, I am pointing out that your current exhibition of entitled behaviour across multiple venues (as represented by the specific sentence I quoted) is problematic. Please stop trying to crack the whip and generate an artificial sense of urgency - software publication and distribution is a complex problem, and most of the current challenges in the PyPI ecosystem stem from an ongoing lack of funding which requires organisational change moreso than technical change. While various folks are working on that, it's mostly not a distutils-sig level problem, but rather a question for the PSF and for commercial Python redistributors.
If you're looking to provide information, or ask if a particular solution that seems obvious to you would be feasible in practice, then do that. The one thing I am asking you to STOP doing is combining your questions with exaggerated hyperbole that's designed to make volunteers feel bad.
I just wanted to make sure that we didn't lose out on starting a discussion of this problem.
The problem is of course caused by the runaway success of the wheel format, and I'm sure it can be solved in a sensible way, but however expressed, it's true that wheels have become so standard that we do need to think about automation for build and release, if we aren't going run into trouble. By trouble, I mean that users will often hit the situation where they don't get wheels when they expect to, and get turned off pypi / wheels as a result. I have personally put a great deal of work into building and releasing wheels, so that is something I'd really like to avoid.
So - can I humbly ask - what is the best way to get that discussion going?
Cheers,
Matthew
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
On Thu, May 26, 2016 at 2:21 PM Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, May 26, 2016 at 1:57 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 26 May 2016 at 01:11, Thomas Güttler guettliml@thomas-guettler.de
wrote:
Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
On 25 May 2016 at 17:13, Thomas Güttler guettliml@thomas-guettler.de wrote:
If you want wheel to be successful, **provide a build server**.
Thomas, aside from that statement being demonstrably untrue (since the wheel format has already proven to be wildly successful, even with developers coping with Linux ABI fragmentation), an attitude of "give me more free stuff, or your project will fail" is not an acceptable tone to adopt on this list.
The contributors here are (mainly) volunteers working on infrastructure provided by a public interest charity, not your personal servants to be ordered about as you feel inclined.
You seem to be angry. Why?
Thomas, I am pointing out that your current exhibition of entitled behaviour across multiple venues (as represented by the specific sentence I quoted) is problematic. Please stop trying to crack the whip and generate an artificial sense of urgency - software publication and distribution is a complex problem, and most of the current challenges in the PyPI ecosystem stem from an ongoing lack of funding which requires organisational change moreso than technical change. While various folks are working on that, it's mostly not a distutils-sig level problem, but rather a question for the PSF and for commercial Python redistributors.
If you're looking to provide information, or ask if a particular solution that seems obvious to you would be feasible in practice, then do that. The one thing I am asking you to STOP doing is combining your questions with exaggerated hyperbole that's designed to make volunteers feel bad.
I just wanted to make sure that we didn't lose out on starting a discussion of this problem.
The problem is of course caused by the runaway success of the wheel format, and I'm sure it can be solved in a sensible way, but however expressed, it's true that wheels have become so standard that we do need to think about automation for build and release, if we aren't going run into trouble. By trouble, I mean that users will often hit the situation where they don't get wheels when they expect to, and get turned off pypi / wheels as a result. I have personally put a great deal of work into building and releasing wheels, so that is something I'd really like to avoid.
So - can I humbly ask - what is the best way to get that discussion going?
Cheers,
Matthew _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On May 26, 2016, at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
pip install —binary-only :all: ...
— Donald Stufft
On May 26, 2016, at 2:36 PM, Donald Stufft donald@stufft.io wrote:
On May 26, 2016, at 2:28 PM, Daniel Holth <dholth@gmail.com mailto:dholth@gmail.com> wrote:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
pip install —binary-only :all: ...
Sorry, it’s —only-binary.
— Donald Stufft
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
Cheers,
Matthew
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
I believe it would find the latest version that has a wheel available, I could be misremembering though.
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
I want to get something setup that would allow people to only need to upload a source release to PyPI and then have wheels automatically built for them (but not mandate that- Projects that wish it should always be able to control their wheel generation). I don’t know what that would specifically look like, if someone is motivated to work on it I’m happy to help figure out what it should look like and provide guidance where I can, otherwise it’ll wait until I get around to it.
— Donald Stufft
I want to get something setup that would allow people to only need to
upload
a source release to PyPI and then have wheels automatically built for them (but not mandate that- Projects that wish it should always be able to
control
their wheel generation). I don’t know what that would specifically look like, if someone is motivated to work on it I’m happy to help figure out
what
it should look like and provide guidance where I can, otherwise it’ll wait until I get around to it.
One first step towards this that's a natural follow-on to the manylinux work might be to define a overall build configuration file / format and process for automating the whole wheel build cycle (i'm thinking of something modeled after conda-build) that would, among other things
for potentially multiple versions of python: - run `pip wheel` (or setu.py bdist_wheel) to compile the wheel - run `auditwheel` (linux) or `delocate` (osx) to bundle any external libraries
-Robert
On Thu, May 26, 2016 at 2:47 PM, Donald Stufft donald@stufft.io wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com
wrote:
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a
wheel
for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
I believe it would find the latest version that has a wheel available, I could be misremembering though.
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
I want to get something setup that would allow people to only need to upload a source release to PyPI and then have wheels automatically built for them (but not mandate that- Projects that wish it should always be able to control their wheel generation). I don’t know what that would specifically look like, if someone is motivated to work on it I’m happy to help figure out what it should look like and provide guidance where I can, otherwise it’ll wait until I get around to it.
— Donald Stufft
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
conda-forge/conda-smithy
| Src: https://github.com/conda-forge/conda-smithy | Homepage: https://conda-forge.github.io/ :
"conda-forge https://github.com/conda-forge is a github organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX"
https://conda-forge.github.io/feedstocks.html
so, for psutil-feedstock:
https://github.com/conda-forge/psutil-feedstock
- https://github.com/conda-forge/psutil-feedstock/blob/master/recipe/meta.yaml - https://github.com/conda-forge/psutil-feedstock/blob/master/appveyor.yml - https://github.com/conda-forge/psutil-feedstock/blob/master/ci_support/run_d... - https://github.com/conda-forge/psutil-feedstock/blob/master/.travis.yml
… https://github.com/conda-forge/conda-smithy/tree/master/conda_smithy/templat...
On Thursday, May 26, 2016, Robert T. McGibbon rmcgibbo@gmail.com wrote:
I want to get something setup that would allow people to only need to
upload
a source release to PyPI and then have wheels automatically built for
them
(but not mandate that- Projects that wish it should always be able to
control
their wheel generation). I don’t know what that would specifically look like, if someone is motivated to work on it I’m happy to help figure out
what
it should look like and provide guidance where I can, otherwise it’ll
wait
until I get around to it.
One first step towards this that's a natural follow-on to the manylinux work might be to define a overall build configuration file / format and process for automating the whole wheel build cycle (i'm thinking of something modeled after conda-build) that would, among other things
for potentially multiple versions of python:
- run `pip wheel` (or setu.py bdist_wheel) to compile the wheel
- run `auditwheel` (linux) or `delocate` (osx) to bundle any external
libraries
-Robert
On Thu, May 26, 2016 at 2:47 PM, Donald Stufft <donald@stufft.io javascript:_e(%7B%7D,'cvml','donald@stufft.io');> wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett <matthew.brett@gmail.com
javascript:_e(%7B%7D,'cvml','matthew.brett@gmail.com');> wrote:
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth <dholth@gmail.com
javascript:_e(%7B%7D,'cvml','dholth@gmail.com');> wrote:
Maybe there could be a way to say "the most recent release that has a
wheel
for my platform". That would help with the problem of binaries not
being
available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
I believe it would find the latest version that has a wheel available, I could be misremembering though.
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
I want to get something setup that would allow people to only need to upload a source release to PyPI and then have wheels automatically built for them (but not mandate that- Projects that wish it should always be able to control their wheel generation). I don’t know what that would specifically look like, if someone is motivated to work on it I’m happy to help figure out what it should look like and provide guidance where I can, otherwise it’ll wait until I get around to it.
— Donald Stufft
Distutils-SIG maillist - Distutils-SIG@python.org javascript:_e(%7B%7D,'cvml','Distutils-SIG@python.org'); https://mail.python.org/mailman/listinfo/distutils-sig
-- -Robert
On Thu, May 26, 2016 at 11:22 PM, Wes Turner wes.turner@gmail.com wrote:
"conda-forge https://github.com/conda-forge is a github organization containing repositories of conda recipes.
Just to be clear -- while conda-forge is about conda packages, a good deal of the CI integration stuff would apply just as well to building wheels.
It would make a lot of sense to build off of what they've done.
-CHB
On 27 May 2016 04:48, "Donald Stufft" donald@stufft.io wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com
wrote:
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
I want to get something setup that would allow people to only need to
upload
a source release to PyPI and then have wheels automatically built for them (but not mandate that- Projects that wish it should always be able to
control
their wheel generation).
A possible preceding step for that might be to create a service that reports per-project information on clients downloading the sdist versions of a project. With the Big Query data publicly available, that shouldn't need to be part of PyPI itself (at least in the near term), so it should just need an interested volunteer, rather than being gated on Warehouse.
The intent there would be to allow projects that decide to build their own wheels to prioritise their wheel creation, and be able to quantify the impact of each addition to their build matrix, rather than necessarily providing pre-built binaries for all supported platforms supported by the source release.
The other thing that data could be used for is to start quantifying the throughput requirements for an "all of PyPI" build service by looking at release publication rates (rather than download rates). Again, likely pursuable by volunteers using the free tier of a public PaaS, rather than requiring ongoing investment.
Cheers, Nick.
On 27 May 2016 10:40, "Nick Coghlan" ncoghlan@gmail.com wrote:
On 27 May 2016 04:48, "Donald Stufft" donald@stufft.io wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com
wrote:
It would be very good to work out a plan for new Python releases as well. We really need to get wheels up to pypi a fair while before the release date, and it's easy to forget to do that, because, at the moment, we don't have much testing infrastructure to make sure that a range of wheel installs are working OK.
I want to get something setup that would allow people to only need to
upload
a source release to PyPI and then have wheels automatically built for
them
(but not mandate that- Projects that wish it should always be able to
control
their wheel generation).
A possible preceding step for that might be to create a service that
reports per-project information on clients downloading the sdist versions of a project. With the Big Query data publicly available, that shouldn't need to be part of PyPI itself (at least in the near term), so it should just need an interested volunteer, rather than being gated on Warehouse.
It belatedly occurs to me that experimenting with that myself would align pretty well with some ideas I'm exploring for $(day job), so if nobody else pursues this, I'll likely take a look myself some time in the next few weeks (and if they do take a look, I should be able to find the time to help).
Cheers, Nick.
Hi,
On Thu, May 26, 2016 at 11:47 AM, Donald Stufft donald@stufft.io wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com wrote:
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
I believe it would find the latest version that has a wheel available, I could be misremembering though.
Reflecting a bit more on this - how easy would be be to add a flag ``--prefer-binary`` that would have the effect of:
* Installing a binary wheel for current release if available, otherwise; * Checking previous release for binary wheel, installing if present, otherwise; * Install from sdist
I think that would help a great deal in reducing surprise in wheel installs while we get better at putting up wheels at the same time as sdists.
Cheers,
Matthew
(this is my first mail to the list, hopefully, this goes through)
Hey Matthew,
FYI, the --prefer-binary flag that you mention has come up in earlier discussions and already has an issue over at pip’s github repo ( https://github.com/pypa/pip/issues/3785).
Regards, Pradyun
On Tue, 14 Jun 2016 at 00:21 Matthew Brett matthew.brett@gmail.com wrote:
Hi,
On Thu, May 26, 2016 at 11:47 AM, Donald Stufft donald@stufft.io wrote:
On May 26, 2016, at 2:41 PM, Matthew Brett matthew.brett@gmail.com
wrote:
On Thu, May 26, 2016 at 2:28 PM, Daniel Holth dholth@gmail.com wrote:
Maybe there could be a way to say "the most recent release that has a
wheel
for my platform". That would help with the problem of binaries not
being
available concurrently with a new source distribution.
Yes, that would certainly help get over some of the immediate problems.
Sorry for my ignorance - but does ``--only-binary`` search for an earlier release with a binary or just bomb out if the latest release does not have a binary? It would also be good to have a flag to say "if this is pure Python go ahead and use the source, otherwise error". Otherwise I guess we'd have to rely on everyone with a pure Python package generating wheels.
I believe it would find the latest version that has a wheel available, I could be misremembering though.
Reflecting a bit more on this - how easy would be be to add a flag ``--prefer-binary`` that would have the effect of:
- Installing a binary wheel for current release if available, otherwise;
- Checking previous release for binary wheel, installing if present,
otherwise;
- Install from sdist
I think that would help a great deal in reducing surprise in wheel installs while we get better at putting up wheels at the same time as sdists.
Cheers,
Matthew _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Am 26.05.2016 um 20:28 schrieb Daniel Holth:
Maybe there could be a way to say "the most recent release that has a wheel for my platform". That would help with the problem of binaries not being available concurrently with a new source distribution.
I don't get what you want to say.
If you are maintainer, then there is no "my platform". There is matrix:
python_versions x supported_platforms
If you have optional dependencies this matrix gets a third dimension :-)
Tell me if I speak none sense, but AFAIK there are two ways to to build for other platforms:
1. Cross compiling: you create code for platform B an platform A. 2. VM: create code for platform B on platform A by running a VM of platform B.
I guess taking the second way is easier - but slower.
Before coding a single line I would check what already exists (outside the "python world")
What do you think?
Please tell me what's wrong with above text.
Regards, Thomas Güttler
On Fri, May 27, 2016 at 2:25 PM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I don't get what you want to say.
If you are maintainer, then there is no "my platform". There is matrix:
Missing the context, but didn't he write that from user perspective?
One could argue that getting old versions just because there's a wheel ain't the best idea. Plus users can always pin the dependency to the "right" version (the one that has wheel for their platform). Functionally nothing is missing, it's an argument about default behavior.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
So this was a problem with eggs too. Let's say ZODB 3.0.1 was just released. You are happily using 3.0.0, the next version is a minor upgrade, but there are no precompiled packages for 3.0.1, so your build breaks on Friday morning when you are trying to deploy. Compiling it yourself is not an option because you are on Windows XP, and the compiler is so hard to install that you would need to hire a consultant to get it done; only to bridge the 1-week gap between a new ZODB release and the precompiled eggs for your platform.
If you find yourself in that situation then you would appreciate an easy way to use the newest available binaries rather than the newest available source, or perhaps to install directly from your cached binaries without going through the package index. Manually pinning the older version just to get the binary is going to ruin your day.
With wheel we can at least approximate this by doing a two-phase install, first dumping all your project's dependencies as wheels to a wheel directory, and then only pointing the installer at the wheel directory for the deployment.
Daniel
On Fri, May 27, 2016 at 7:36 AM Ionel Cristian Mărieș < distutils-sig@python.org> wrote:
On Fri, May 27, 2016 at 2:25 PM, Thomas Güttler < guettliml@thomas-guettler.de> wrote:
I don't get what you want to say.
If you are maintainer, then there is no "my platform". There is matrix:
Missing the context, but didn't he write that from user perspective?
One could argue that getting old versions just because there's a wheel ain't the best idea. Plus users can always pin the dependency to the "right" version (the one that has wheel for their platform). Functionally nothing is missing, it's an argument about default behavior.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Not that this isn't an issue, but:
On Fri, May 27, 2016 at 5:34 AM, Daniel Holth dholth@gmail.com wrote:
So this was a problem with eggs too. Let's say ZODB 3.0.1 was just released. You are happily using 3.0.0, the next version is a minor upgrade, but there are no precompiled packages for 3.0.1, so your build breaks on Friday morning when you are trying to deploy.
If you are using pip, etc. to DEPLOY, they you'd better darn be using explicit versions in your requirements.txt.
Just sayin'
Of course, you may not want to use explicit versions in development for CI testing, and then you run into the same issue. But having a test fail on commit is not nearly as big a deal....
If you find yourself in that situation then you would appreciate an easy
way to use the newest available binaries rather than the newest available source,
I think the trick here is that you need "binaries" for packages with C extensions, but source is just fine for pure python.
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
-CHB
On 27 May 2016 at 17:28, Chris Barker chris.barker@noaa.gov wrote:
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
It would be *really* nice to have some sort of metadata/flag that said "this project is pure Python". Normally, what I want is not *quite* --only-binary, but rather "only binary except for pure Python where I'm happy to take a source distribution". But AFAIK, there's no way for pip to know that :-(
Paul
On May 27, 2016, at 12:37 PM, Paul Moore p.f.moore@gmail.com wrote:
On 27 May 2016 at 17:28, Chris Barker chris.barker@noaa.gov wrote:
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
It would be *really* nice to have some sort of metadata/flag that said "this project is pure Python". Normally, what I want is not *quite* --only-binary, but rather "only binary except for pure Python where I'm happy to take a source distribution". But AFAIK, there's no way for pip to know that :-(
The flip side is it should be trivial for pure Python projects to release wheels, often requiring them to do nothing different except ensuring `wheel` is installed and running ``setup.py sdist bdist_wheel` instead of just `setup.py sdist`.
— Donald Stufft
This reminds me of something I wanted to ask – how come the "virtualenv" tool installs wheel but the built-in "venv" tool does not? The latter is the currently recommended method of creating a virtualenv, isn't it?
27.05.2016, 19:40, Donald Stufft kirjoitti:
On May 27, 2016, at 12:37 PM, Paul Moore p.f.moore@gmail.com wrote:
On 27 May 2016 at 17:28, Chris Barker chris.barker@noaa.gov wrote:
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
It would be *really* nice to have some sort of metadata/flag that said "this project is pure Python". Normally, what I want is not *quite* --only-binary, but rather "only binary except for pure Python where I'm happy to take a source distribution". But AFAIK, there's no way for pip to know that :-(
The flip side is it should be trivial for pure Python projects to release wheels, often requiring them to do nothing different except ensuring `wheel` is installed and running ``setup.py sdist bdist_wheel` instead of just `setup.py sdist`.
— Donald Stufft
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On May 27, 2016, at 12:43 PM, Alex Grönholm alex.gronholm@nextday.fi wrote:
This reminds me of something I wanted to ask – how come the "virtualenv" tool installs wheel but the built-in "venv" tool does not? The latter is the currently recommended method of creating a virtualenv, isn't it?
virtualenv, by nature of not being in the standard library, can be a bit more liberal. Once pyproject.toml support is in pip and we can automatically install wheel for wheel caching as part of the build process, I intend to remove both setuptools and wheel from virtualenv (and venv).
— Donald Stufft
On 27 May 2016 at 17:40, Donald Stufft donald@stufft.io wrote:
On May 27, 2016, at 12:37 PM, Paul Moore p.f.moore@gmail.com wrote:
On 27 May 2016 at 17:28, Chris Barker chris.barker@noaa.gov wrote:
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
It would be *really* nice to have some sort of metadata/flag that said "this project is pure Python". Normally, what I want is not *quite* --only-binary, but rather "only binary except for pure Python where I'm happy to take a source distribution". But AFAIK, there's no way for pip to know that :-(
The flip side is it should be trivial for pure Python projects to release wheels, often requiring them to do nothing different except ensuring `wheel` is installed and running ``setup.py sdist bdist_wheel` instead of just `setup.py sdist`.
Well, yes. But it pretty much is trivial already, but there are still a reasonable number of projects that don't do so. My theories as to why:
- The project hasn't done a new release yet, and doesn't want to upload a wheel *except* as part of a new release. - The project has tooling to do a release, and hasn't got round to changing it. - The project isn't aware of wheels. - The project doesn't see the value of wheels for pure-python code.
Paul
On Fri, May 27, 2016 at 12:28 PM Chris Barker chris.barker@noaa.gov wrote:
Not that this isn't an issue, but:
On Fri, May 27, 2016 at 5:34 AM, Daniel Holth dholth@gmail.com wrote:
So this was a problem with eggs too. Let's say ZODB 3.0.1 was just released. You are happily using 3.0.0, the next version is a minor upgrade, but there are no precompiled packages for 3.0.1, so your build breaks on Friday morning when you are trying to deploy.
If you are using pip, etc. to DEPLOY, they you'd better darn be using explicit versions in your requirements.txt.
Just sayin'
Yes, it goes without saying that you can avoid these problems by some combination of best practices such as running your own devpi. But maybe we could figure out how to provide best practices by default.
Of course, you may not want to use explicit versions in development for CI
testing, and then you run into the same issue. But having a test fail on commit is not nearly as big a deal....
If you find yourself in that situation then you would appreciate an easy
way to use the newest available binaries rather than the newest available source,
I think the trick here is that you need "binaries" for packages with C extensions, but source is just fine for pure python.
So I kind of like the idea of making wheels the default for distributing on PyPi always -- even for pure python modules. And wheels are trivial to build from pure python packages -- so why not?
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
On 26 May 2016 at 19:20, Matthew Brett matthew.brett@gmail.com wrote:
I just wanted to make sure that we didn't lose out on starting a discussion of this problem.
The problem is of course caused by the runaway success of the wheel format, and I'm sure it can be solved in a sensible way, but however expressed, it's true that wheels have become so standard that we do need to think about automation for build and release, if we aren't going run into trouble. By trouble, I mean that users will often hit the situation where they don't get wheels when they expect to, and get turned off pypi / wheels as a result. I have personally put a great deal of work into building and releasing wheels, so that is something I'd really like to avoid.
So - can I humbly ask - what is the best way to get that discussion going?
I would certainly like to promote projects using existing CI systems for building wheels. It seems like a pretty simple approach that would cover a decent proportion of the problem.
There's already a section in the Packaging User Guide explaining how to use Appveyor to build Windows binary wheels. If you don't have dependencies to deal with (e.g., you're simply providing a C speedup module) it's pretty simple. If you *do* have dependencies to deal with, I suspect you could arrange for them to be uploaded when you build somehow without too much issue - but I don't have projects like that so I can't really add that to the document. The biggest problem with Appveyor is that it's less well known than Travis. However, I don't know of many (frankly, any!) projects using this approach - whether that's because of a lack of awareness, or because it's not as useful as it seemed to me when I wrote it, or some other reason, I don't honestly know. Maybe we could get some feedback on what can be done to improve that section? Is there going to be any sort of packaging discussions at PyCon? Maybe it's a question that could be asked? (I'm not going to be there myself, unfortunately).
For Linux/Unix, the problem was always the plethora of binary formats. If we can assume that manylinux builds will constitute a 90% solution here, maybe someone could contribute a section explaining how to add the building of manylinux wheels into a project's Travis config. I know almost nothing about OSX, but doesn't Travis offer OSX builders? Could they be used to build extensions in the same way as the above?
Going beyond this level (which is basically making sure projects know they have easy access to build environments for the key platforms, and encouraging them to use them) is when it gets harder. Integrating artifacts from travis/appveyor into a release process would need some work, for example. Maybe a way forward here would be for some projects to try doing so, and writing up a set of "how we did it" notes that could over time turn into a set of recommended practices in the PUG. Or an interested 3rd party could probably do most of the experimentation and development, simply by forking a few typical projects and seeing what was needed.
With all of the above, we'd end up with some (hopefully!) pretty good documentation of how to (relatively) painlessly add building of wheels for all major platforms into *any* project (that didn't have specialised issues to deal with).
I'm not sure where we'd go from there. The next step seems to be to actually provide some sort of build service, or curated set of wheels for projects that don't/won't/can't produce their own. But that's a lot of work - the conda folks, and people like Christoph Gohlke, know how much. Maybe by making the process simple enough (as described above) might encourage someone to put together a wheel service along the lines of Christoph's (but for all platforms, and in a format that pip can access directly). Once something like that starts up, it may attract additional volunteers, and grow from there. But it's still quite a commitment from someone.
Paul
As a lurker on this list and long time user of everything python I feel this is becoming appropriate
participants (17)
-
Alex Grönholm -
Chris Barker -
Christopher Baines -
Daniel Holth -
Donald Stufft -
Ionel Cristian Mărieș -
Matthew Brett -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Pradyun Gedam -
Randy Syring -
Robert T. McGibbon -
Robin Becker -
Sylvain Corlay -
Thomas Güttler -
Wes Turner