Hi folks, currently, I came across http://pythonwheels.com/ during researching how to make a proper Python distribution for PyPI. I thought it would be great idea to tell other maintainers to upload their content as wheels so I approached a couple of them. Some of them already provided wheels. Happy being able to have built my own distribution, I discussed the issue at hand with some people and I would like to share my findings and propose some ideas: 1) documentation is weirdly split up/distributed and references old material 2) once up and running (setup.cfg, setup.py etc. etc.) it works but everybody needs to do it on their own 3) more than one way to do (upload, wheel, source/binary etc.) it (sigh) 4) making contact to propose wheels on github or per email is easy otherwise almost impossible or very tedious 5) reactions went evenly split from "none", "yes", "when ready" to "nope" None: well, okay yes: that's good when ready: well, okay nope: what a pity for wheels; example: https://github.com/simplejson/simplejson/issues/122 I personally find the situation not satisfying. Someone proposes the following solution in form of a question: Why do developers need to build their distribution themselves? I had not real answer to him, but pondering a while over it, I found it really insightful. Viewing this from a different angle, packaging your own distribution is actually a waste of time. It is a tedious, error-prone task involving no creativity whatsoever. Developers on the other hand are actually people with very little time and a lot of creativity at hand which should spend better. The logical conclusion would be that PyPI should build wheels for the developers for every python/platform combination necessary. With this post, I would like raise awareness of the people in charge of the Python infrastructure. Best, Sven
Hi Sven, Just adding a few comments inline: On Sun, Sep 6, 2015 at 7:33 PM, Sven R. Kunze <srkunze@mail.de> wrote:
3) more than one way to do (upload, wheel, source/binary etc.) it (sigh)
And most are uploading/registering over http (sight)
nope: what a pity for wheels; example: https://github.com/simplejson/simplejson/issues/122
But that's for non pure-python wheels, wheel can be universal, in which case they are easy to build.
Why do developers need to build their distribution themselves?
Historical reason. On GitHub, at least it is pretty easy to make Travis-CI build your wheels, some scientific packages (which are not the easier to build) have done that, so automation is possible. And these case need really particular environements where all aspects of the builds are controlled.
I think that some of that could be done by warehouse at some point: https://github.com/pypa/warehouse But you will never be able to cover all. I'm sure people will ask PyPI to build for windows 98 server version otherwise. Personally for pure python packages I know use https://pypi.python.org/pypi/flit which is one of the only packaging tools for which I can remember all the step to get a package on PyPI without reading the docs. -- M [Sven, sorry for duplicate :-) ]
On 06Sep2015 1033, Sven R. Kunze wrote:
This would be a wonderful situation to end up in, but the problem is that many wheels have difficult source dependencies to configure. It is much easier for the developers who should already have working systems to build the wheel themselves then it would be either for them to provide/configure a remote system to do it, or for the end-user to configure their own system. (And if it can't be tested on a particular system, then the developer probably shouldn't release wheels for that system anyway.) What I would rather see is a way to delegate building to other people by explicitly allowing someone to add wheels to PyPI for existing releases without necessarily being able to make a new release or delete old ones. There is some trust involved, but it could also enable more ad-hoc systems of building wheels through Travis/Jenkins/VSO/etc. automation without needing to reveal login information through your repo (i.e. you give the "Jenkins-wheel" user permission to publish wheels for your package). Not sure how feasible this is, but I'd guess it's easier than trying to run our own build servers. Cheers, Steve
On September 6, 2015 12:33:29 PM CDT, "Sven R. Kunze" <srkunze@mail.de> wrote:
You can already do this with CI services. I wrote a post about doing that with AppVeyor: http://kirbyfan64.github.io/posts/using-appveyor-to-distribute-python-wheels... but the idea behind it should apply easily to Travis and others. In reality, you're probably using a CI service to run your tests anyway, so it might as well build your wheels, too!
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
On 7 September 2015 at 11:22, Ryan Gonzalez <rymg19@gmail.com> wrote:
Right, Appveyor also has the most well-defined CI instructions on packaging.python.org: https://packaging.python.org/en/latest/appveyor.html It doesn't do auto-upload, as many projects only release occasionally rather than for every commit. However, it may be desirable to go into more detail about how to do that, if you'd be interested in sending a PR based on your post. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Sep 06, 2015 at 07:33:29PM +0200, Sven R. Kunze wrote:
Over on the python-list mailing list, Ned Batchelder asked a question. I haven't seen an answer there, and as far as I know he isn't subscribed here, so I'll take the liberty of copying his question here: Ned says: "As a developer of a Python package, I don't see how this would be better. The developer would still have to get their software into some kind of uniform configuration, so the central authority could package it. You've moved the problem from, "everyone has to make wheels" to "everyone has to make a tree that's structured properly." But if we can do the second thing, the first thing is really easy. Maybe I've misunderstood?" -- Steve
On September 6, 2015 at 9:27:32 PM, Steven D'Aprano (steve@pearwood.info) wrote:
A PyPI build farm for authors is something I plan on getting too if someone doesn't beat me to it. It won't be mandatory to use it, and it's not going to cover every corner case where you need some crazy obscure library installed, but it will ideally try to make things better. As far why it's better, it's actually pretty simple. Let's take lxml for example which binds against libxml2. It needs built on Windows, it needs built on OSX, it needs built on various Linux distributions in order to cover the spread of just the common cases. If we want to start to get into uncommon platforms we're looking at various BSDs, Solaris, etc. It's a lot of infrastructure to maintain for the common cases much less the uncommon cases that we can centralize the maintenance into one location. In addition to all of that, it turns out you pretty much need to get most of the way to defining that configuration in a central location anyways since pip already needs to know how to build your project, the only things it doesn't know is what platforms you support and what, if any, extra libraries you require to be installed. There are some PEPs in the works that may make that second part known ahead of time and for the first one, we can simply ask when a project enable the build farm. Even if we only support things that don't require additional libraries to be installed, that's still a pretty big win by default that will allow a wide number of projects to be installed from a binary distribution and not require a compiler toolchain that previously couldn't be due to authors not willing or not able to manage the overhead of the infrastructure around building on all of those platforms. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sep 6, 2015, at 21:20, Donald Stufft <donald@stufft.io> wrote:
IIRC, Apple included ancient versions (even at the time) of libxml2 up to around 10.7, and at one point they even included one of the broken 2.7.x versions. So a build farm building for 10.6+ (which I think is what python.org builds still target?) is going to build against an ancient libxml2, meaning some features of lxml2 will be disabled, and others may even be broken. Even if I'm remembering wrong about Apple, I'm sure there are linux distros with similar issues. Fortunately, lxml has a built-in option (triggered by an env variable) for dealing with this, by downloading the source, building a local copy of the libs, and statically linking them into lxml, but that means you need some way for a package to specify env variables to be set on the build server. And can you expect most libraries with similar issues to do the same? (I don't know how many packages actually have similar problems, but since you specifically mentioned lxml as your example, and I had headaches building it for a binary-distributed app supporting 10.6-10.9 a few years ago, I happened to remember this problem.)
On Sep 6, 2015 10:28 PM, "Andrew Barnert via Python-ideas" < python-ideas@python.org> wrote:
on OSX, it needs built on various Linux distributions in order to cover
needs built the python.org builds still target?) is going to build against an ancient libxml2, meaning some features of lxml2 will be disabled, and others may even be broken. Even if I'm remembering wrong about Apple, I'm sure there are linux distros with similar issues.
Fortunately, lxml has a built-in option (triggered by an env variable)
for dealing with this, by downloading the source, building a local copy of the libs, and statically linking them into lxml, but that means you need some way for a package to specify env variables to be set on the build server. And can you expect most libraries with similar issues to do the same? Yes, you can! :-) I mean, not everyone will necessarily use it, but adding code like if "PYPI_BUILD_SERVER" in os.environ: do_static_link = True to your setup.py is *wayyyy* easier than buying an OS X machine and maintaining it and doing manual builds at every release. Or finding a volunteer who has an OS X box and nagging them at every release and dealing with trust hassles. And there are a lot of packages out there that just have some cython files in them for speedups with no external dependencies, or whatever. A build farm wouldn't have to be perfect to be extremely useful. -n
On 07.09.2015 07:39, Nathaniel Smith wrote:
You bet what I just needed to do. Depending on somebody else machine is really frustrating.
I agree. Just good enough suffices for 80% of all the packages to be in good shape. Nick mentioned some improvements that are necessary before we can indulge such a building farm (except the farm itself). Best, Sven
Another example for the sake of documentation: https://github.com/tornadoweb/tornado/issues/1383#issuecomment-84098055 On 07.09.2015 07:39, Nathaniel Smith wrote:
Hi Sven, Just adding a few comments inline: On Sun, Sep 6, 2015 at 7:33 PM, Sven R. Kunze <srkunze@mail.de> wrote:
3) more than one way to do (upload, wheel, source/binary etc.) it (sigh)
And most are uploading/registering over http (sight)
nope: what a pity for wheels; example: https://github.com/simplejson/simplejson/issues/122
But that's for non pure-python wheels, wheel can be universal, in which case they are easy to build.
Why do developers need to build their distribution themselves?
Historical reason. On GitHub, at least it is pretty easy to make Travis-CI build your wheels, some scientific packages (which are not the easier to build) have done that, so automation is possible. And these case need really particular environements where all aspects of the builds are controlled.
I think that some of that could be done by warehouse at some point: https://github.com/pypa/warehouse But you will never be able to cover all. I'm sure people will ask PyPI to build for windows 98 server version otherwise. Personally for pure python packages I know use https://pypi.python.org/pypi/flit which is one of the only packaging tools for which I can remember all the step to get a package on PyPI without reading the docs. -- M [Sven, sorry for duplicate :-) ]
On 06Sep2015 1033, Sven R. Kunze wrote:
This would be a wonderful situation to end up in, but the problem is that many wheels have difficult source dependencies to configure. It is much easier for the developers who should already have working systems to build the wheel themselves then it would be either for them to provide/configure a remote system to do it, or for the end-user to configure their own system. (And if it can't be tested on a particular system, then the developer probably shouldn't release wheels for that system anyway.) What I would rather see is a way to delegate building to other people by explicitly allowing someone to add wheels to PyPI for existing releases without necessarily being able to make a new release or delete old ones. There is some trust involved, but it could also enable more ad-hoc systems of building wheels through Travis/Jenkins/VSO/etc. automation without needing to reveal login information through your repo (i.e. you give the "Jenkins-wheel" user permission to publish wheels for your package). Not sure how feasible this is, but I'd guess it's easier than trying to run our own build servers. Cheers, Steve
On September 6, 2015 12:33:29 PM CDT, "Sven R. Kunze" <srkunze@mail.de> wrote:
You can already do this with CI services. I wrote a post about doing that with AppVeyor: http://kirbyfan64.github.io/posts/using-appveyor-to-distribute-python-wheels... but the idea behind it should apply easily to Travis and others. In reality, you're probably using a CI service to run your tests anyway, so it might as well build your wheels, too!
-- Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
On 7 September 2015 at 11:22, Ryan Gonzalez <rymg19@gmail.com> wrote:
Right, Appveyor also has the most well-defined CI instructions on packaging.python.org: https://packaging.python.org/en/latest/appveyor.html It doesn't do auto-upload, as many projects only release occasionally rather than for every commit. However, it may be desirable to go into more detail about how to do that, if you'd be interested in sending a PR based on your post. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Sep 06, 2015 at 07:33:29PM +0200, Sven R. Kunze wrote:
Over on the python-list mailing list, Ned Batchelder asked a question. I haven't seen an answer there, and as far as I know he isn't subscribed here, so I'll take the liberty of copying his question here: Ned says: "As a developer of a Python package, I don't see how this would be better. The developer would still have to get their software into some kind of uniform configuration, so the central authority could package it. You've moved the problem from, "everyone has to make wheels" to "everyone has to make a tree that's structured properly." But if we can do the second thing, the first thing is really easy. Maybe I've misunderstood?" -- Steve
On September 6, 2015 at 9:27:32 PM, Steven D'Aprano (steve@pearwood.info) wrote:
A PyPI build farm for authors is something I plan on getting too if someone doesn't beat me to it. It won't be mandatory to use it, and it's not going to cover every corner case where you need some crazy obscure library installed, but it will ideally try to make things better. As far why it's better, it's actually pretty simple. Let's take lxml for example which binds against libxml2. It needs built on Windows, it needs built on OSX, it needs built on various Linux distributions in order to cover the spread of just the common cases. If we want to start to get into uncommon platforms we're looking at various BSDs, Solaris, etc. It's a lot of infrastructure to maintain for the common cases much less the uncommon cases that we can centralize the maintenance into one location. In addition to all of that, it turns out you pretty much need to get most of the way to defining that configuration in a central location anyways since pip already needs to know how to build your project, the only things it doesn't know is what platforms you support and what, if any, extra libraries you require to be installed. There are some PEPs in the works that may make that second part known ahead of time and for the first one, we can simply ask when a project enable the build farm. Even if we only support things that don't require additional libraries to be installed, that's still a pretty big win by default that will allow a wide number of projects to be installed from a binary distribution and not require a compiler toolchain that previously couldn't be due to authors not willing or not able to manage the overhead of the infrastructure around building on all of those platforms. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Sep 6, 2015, at 21:20, Donald Stufft <donald@stufft.io> wrote:
IIRC, Apple included ancient versions (even at the time) of libxml2 up to around 10.7, and at one point they even included one of the broken 2.7.x versions. So a build farm building for 10.6+ (which I think is what python.org builds still target?) is going to build against an ancient libxml2, meaning some features of lxml2 will be disabled, and others may even be broken. Even if I'm remembering wrong about Apple, I'm sure there are linux distros with similar issues. Fortunately, lxml has a built-in option (triggered by an env variable) for dealing with this, by downloading the source, building a local copy of the libs, and statically linking them into lxml, but that means you need some way for a package to specify env variables to be set on the build server. And can you expect most libraries with similar issues to do the same? (I don't know how many packages actually have similar problems, but since you specifically mentioned lxml as your example, and I had headaches building it for a binary-distributed app supporting 10.6-10.9 a few years ago, I happened to remember this problem.)
On Sep 6, 2015 10:28 PM, "Andrew Barnert via Python-ideas" < python-ideas@python.org> wrote:
on OSX, it needs built on various Linux distributions in order to cover
needs built the python.org builds still target?) is going to build against an ancient libxml2, meaning some features of lxml2 will be disabled, and others may even be broken. Even if I'm remembering wrong about Apple, I'm sure there are linux distros with similar issues.
Fortunately, lxml has a built-in option (triggered by an env variable)
for dealing with this, by downloading the source, building a local copy of the libs, and statically linking them into lxml, but that means you need some way for a package to specify env variables to be set on the build server. And can you expect most libraries with similar issues to do the same? Yes, you can! :-) I mean, not everyone will necessarily use it, but adding code like if "PYPI_BUILD_SERVER" in os.environ: do_static_link = True to your setup.py is *wayyyy* easier than buying an OS X machine and maintaining it and doing manual builds at every release. Or finding a volunteer who has an OS X box and nagging them at every release and dealing with trust hassles. And there are a lot of packages out there that just have some cython files in them for speedups with no external dependencies, or whatever. A build farm wouldn't have to be perfect to be extremely useful. -n
On 07.09.2015 07:39, Nathaniel Smith wrote:
You bet what I just needed to do. Depending on somebody else machine is really frustrating.
I agree. Just good enough suffices for 80% of all the packages to be in good shape. Nick mentioned some improvements that are necessary before we can indulge such a building farm (except the farm itself). Best, Sven
Another example for the sake of documentation: https://github.com/tornadoweb/tornado/issues/1383#issuecomment-84098055 On 07.09.2015 07:39, Nathaniel Smith wrote:
participants (9)
-
Andrew Barnert -
Donald Stufft -
Matthias Bussonnier -
Nathaniel Smith -
Nick Coghlan -
Ryan Gonzalez -
Steve Dower -
Steven D'Aprano -
Sven R. Kunze