[Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
![](https://secure.gravatar.com/avatar/d3dd3ba5cbd197a5927366c6c9a5003d.jpg?s=120&d=mm&r=g)
Hi All, I am working on ticket #9217 / PR #1051 to add lots more wheel generation to the Twisted CI. I decided to give the cibuildwheel package a try and it made this process almost too easy (well... sort of :] ). I've got AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of wheels for the supported Python versions and bit depths. Travis doesn't save artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, though it's having some Docker issues at the moment and hasn't been successful yet. For some reason in this one case the project directory isn't getting mounted into the container as expected. wheel links: https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 (and next comment) Now that I've got the wheel builds happening I figured it'd be good to try them out on 'real' machines. Turns out we get a failure on twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least the two checks I've done so far (Windows and OSX). I haven't done more than a cursory look at that yet, but it's on the list to understand and resolve. More testing would of course be welcome. Real world, just trial Twisted's own tests, whatever would be appreciated if you are interested. failures with wheels: https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 Overall, it's a bit unclear what the intended use of the various CI hosts are for Twisted. I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added). There wasn't any artifact storage being used on Circle either. So, I'm not sure if there was a reason to use Circle that went away, or... But, not having to hook Travis up to S3 or somesuch for storage is quite nice so Circle wins at least in that category. Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not), or just artifacts on the build server to grab manually (would need some S3 or such for Travis, or Circle for Linux builds as well). Anything else? Do we want automated tests against the wheels? cibuildwheel does have a feature for that though I haven't done anything with it yet. Anyways... hello, thanks for Twisted, and I hope this work ends up saving some people some time. Cheers, -kyle
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Aug 29, 2018, at 10:13 AM, Kyle Altendorf <sda@fstab.net <mailto:sda@fstab.net>> wrote:
Hi All,
Hi Kyle!
I am working on ticket #9217 / PR #1051 to add lots more wheel generation to the Twisted CI. I decided to give the cibuildwheel package a try and it made this process almost too easy (well... sort of :] ). I've got AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of wheels for the supported Python versions and bit depths. Travis doesn't save artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, though it's having some Docker issues at the moment and hasn't been successful yet. For some reason in this one case the project directory isn't getting mounted into the container as expected.
Thanks for doing this!
It's not clear to me that we have any test environments that accurately test what happens after you install from PyPI. If we could build these wheels and then actually run tests on installing them that would be a big step forward in test fidelity! But given the current state of the art, I'd be happy if they just got built from the same code that passed our tests in existing configurations.
Overall, it's a bit unclear what the intended use of the various CI hosts are for Twisted.
Ideally, Travis would handle it all. But Travis can't run multiple kernels, so we have buildbots for lots of alternate kernels, just to make sure. Including BSD kernels, since those are actually substantially different where sockets are concerned. And Travis macOS support is so slow that it was a serious blocker for a while, so the buildbot is filling in until Circle can take over. And Travis can't do Windows, so that's where appveyor comes in. There might still be various non-special-kernel testing things that buildbot does, but those are probably holdovers that, if they still have a reason to exist, it's exactly because Travis doesn't have artifact hosting. Hopefully that's comprehensive?
I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added).
The issue for addressing this is https://twistedmatrix.com/trac/ticket/9445 <https://twistedmatrix.com/trac/ticket/9445> if you'd like to pick up where Adi left off. It's probably just resolving some conflicts, resubmitting for review, and bugging people to get it through the process :).
There wasn't any artifact storage being used on Circle either. So, I'm not sure if there was a reason to use Circle that went away, or... But, not having to hook Travis up to S3 or somesuch for storage is quite nice so Circle wins at least in that category.
I think the big problem on Circle is capacity limits, but I don't know if that's just a macOS thing, or what.
Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not)
I definitely want this. This is really the whole point, what we're trying to get to :). Why don't you think we want this?
or just artifacts on the build server to grab manually (would need some S3 or such for Travis, or Circle for Linux builds as well). Anything else? Do we want automated tests against the wheels? cibuildwheel does have a feature for that though I haven't done anything with it yet.
Ideally we'd push the wheels on tag once the tests passed, but I'd be OK with just unconditionally just building / pushing on tag for starters.
Anyways... hello, thanks for Twisted, and I hope this work ends up saving some people some time.
I can't wait for fully automated releases :).
![](https://secure.gravatar.com/avatar/d3dd3ba5cbd197a5927366c6c9a5003d.jpg?s=120&d=mm&r=g)
On 2018-09-05 03:25, Glyph wrote:
cibuildwheel does have an option to provide a test command. If we did that would we replace the existing tests or supplement? If supplementing that will presumably be a significant increase in build times. Especially if we did it for all wheels. So, how much of a build time hit do we want to take?
https://github.com/twisted/twisted/pull/1056/files It looks like the builds I added in my wheel PR, for whatever that is worth. Well, other than picking xcode v9 vs me picking v10 which neither of the PRs use anyways. I added the requires over in the wheel PR as well in case it ends up saving a useful amount of our OS X quota.
Uhm... hmm... *shrug* I figured someone might want to do a final spot check or have manual intervention for a real release. You can't replace a file on PyPI so... but sure, I didn't mean to argue against automation. Here's one spot I did automatic deployment off of Travis to PyPI. https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4... Pretty straightforward though I think it does the wheel build which isn't what we'd really want. I'd have to check on how to deploy existing wheels as well as those from OS X and Windows. I'd suggest a separate PR? I mean maybe it'd be easier to get one PR reviewed than two but that doesn't seem like a great reason to mash things together. Cheers, -kyle
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Aug 29, 2018, at 10:13 AM, Kyle Altendorf <sda@fstab.net <mailto:sda@fstab.net>> wrote:
Hi All,
Hi Kyle!
I am working on ticket #9217 / PR #1051 to add lots more wheel generation to the Twisted CI. I decided to give the cibuildwheel package a try and it made this process almost too easy (well... sort of :] ). I've got AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of wheels for the supported Python versions and bit depths. Travis doesn't save artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, though it's having some Docker issues at the moment and hasn't been successful yet. For some reason in this one case the project directory isn't getting mounted into the container as expected.
Thanks for doing this!
It's not clear to me that we have any test environments that accurately test what happens after you install from PyPI. If we could build these wheels and then actually run tests on installing them that would be a big step forward in test fidelity! But given the current state of the art, I'd be happy if they just got built from the same code that passed our tests in existing configurations.
Overall, it's a bit unclear what the intended use of the various CI hosts are for Twisted.
Ideally, Travis would handle it all. But Travis can't run multiple kernels, so we have buildbots for lots of alternate kernels, just to make sure. Including BSD kernels, since those are actually substantially different where sockets are concerned. And Travis macOS support is so slow that it was a serious blocker for a while, so the buildbot is filling in until Circle can take over. And Travis can't do Windows, so that's where appveyor comes in. There might still be various non-special-kernel testing things that buildbot does, but those are probably holdovers that, if they still have a reason to exist, it's exactly because Travis doesn't have artifact hosting. Hopefully that's comprehensive?
I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added).
The issue for addressing this is https://twistedmatrix.com/trac/ticket/9445 <https://twistedmatrix.com/trac/ticket/9445> if you'd like to pick up where Adi left off. It's probably just resolving some conflicts, resubmitting for review, and bugging people to get it through the process :).
There wasn't any artifact storage being used on Circle either. So, I'm not sure if there was a reason to use Circle that went away, or... But, not having to hook Travis up to S3 or somesuch for storage is quite nice so Circle wins at least in that category.
I think the big problem on Circle is capacity limits, but I don't know if that's just a macOS thing, or what.
Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not)
I definitely want this. This is really the whole point, what we're trying to get to :). Why don't you think we want this?
or just artifacts on the build server to grab manually (would need some S3 or such for Travis, or Circle for Linux builds as well). Anything else? Do we want automated tests against the wheels? cibuildwheel does have a feature for that though I haven't done anything with it yet.
Ideally we'd push the wheels on tag once the tests passed, but I'd be OK with just unconditionally just building / pushing on tag for starters.
Anyways... hello, thanks for Twisted, and I hope this work ends up saving some people some time.
I can't wait for fully automated releases :).
![](https://secure.gravatar.com/avatar/d3dd3ba5cbd197a5927366c6c9a5003d.jpg?s=120&d=mm&r=g)
On 2018-09-05 03:25, Glyph wrote:
cibuildwheel does have an option to provide a test command. If we did that would we replace the existing tests or supplement? If supplementing that will presumably be a significant increase in build times. Especially if we did it for all wheels. So, how much of a build time hit do we want to take?
https://github.com/twisted/twisted/pull/1056/files It looks like the builds I added in my wheel PR, for whatever that is worth. Well, other than picking xcode v9 vs me picking v10 which neither of the PRs use anyways. I added the requires over in the wheel PR as well in case it ends up saving a useful amount of our OS X quota.
Uhm... hmm... *shrug* I figured someone might want to do a final spot check or have manual intervention for a real release. You can't replace a file on PyPI so... but sure, I didn't mean to argue against automation. Here's one spot I did automatic deployment off of Travis to PyPI. https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4... Pretty straightforward though I think it does the wheel build which isn't what we'd really want. I'd have to check on how to deploy existing wheels as well as those from OS X and Windows. I'd suggest a separate PR? I mean maybe it'd be easier to get one PR reviewed than two but that doesn't seem like a great reason to mash things together. Cheers, -kyle
participants (2)
-
Glyph
-
Kyle Altendorf