Moving Python.org to Buildbot 9 + Python 3
In the past two years, I've put a lot of effort into porting the following to Python 3:
Twisted: https://www.slideshare.net/CraigRodrigues1/the-onward- journey-porting-twisted-to-python-3
Buildbot: https://github.com/buildbot/buildbot/pulls?utf8=%E2%9C%93& q=is%3Aclosed%20is%3Apr%20author%3Arodrigc%20
I would like to help Python.org move to Buildbot 9 + Python 3 for the following reasons:
(1) Python.org currently uses Buildbot 0.8.14. This branch of buildbot is not under active development and is not being ported to Python 3.
(2) The current version of Buildbot (0.9.11) is on the main branch which is actively maintained. This branch has been ported to Python 3.
(3) Buildbot 0.8.14 is not actively being maintained.
(4) Buildbot 0.9.x is actively receiving updates for new features such as Docker support, and support for Hyper.sh.
A while back, I worked with Zachary Ware, who set up a proof of concept of running Buildbot 9 on Python.org.
Zachary set this up:
https://buildbot.python.org/test/
That is an instance running Buildbot 9, using Python 3.4.3 and Twisted 17.5.0.
This is only a proof of concept, and is not actively being used or worked on.
What are the next steps to completely convert Python.org to use Buildbot 9? Who can help with this?
-- Craig
The question most relevant to this list is, what are the consequences on the slave maintainers? Assuming we convert the master, will the slaves be required to run python3/0.9? (That would certainly be desirable, but would it be required?)
Are there advantages for the slave maintainers in the new version?
Getting python3 on some of the platforms may be trickier than on others.
In response to your posting on core-mentorship Victor mentioned there were issues with the proof-of-concept, so figuring out those issues is presumably the first step.
--David
On Wed, Sep 20, 2017 at 8:02 AM, R. David Murray <rdmurray@bitdance.com> wrote:
In response to your posting on core-mentorship Victor mentioned there were issues with the proof-of-concept, so figuring out those issues is presumably the first step.
Zach reported his progress of setting up http://buildbot.python.org/test/ in this thread:
https://github.com/python/buildmaster-config/pull/12
In June, he reported several problems. These problems have all been fixed. In September he updated http://buildbot.python.org/test/ and got it working on Python 3. He was able to do some simple builds. He shut down the worker, because he was performing this test on his own VM which has limited resources.
If there are additional problems, Zach will have to provide details.
I'm happy to help look into any problems, but you'll have to tell me what those problems are for me to investigate.
Craig
Hi Craig,
Sorry I've been largely unresponsive; time has been hard to come by lately.
On Thu, Sep 21, 2017 at 12:41 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I'm happy to help look into any problems, but you'll have to tell me what those problems are for me to investigate.
For starters, there is still a worker connected to the test instance, which seems to have been failing the daily refleak builds with an exception since I connected it. However, there's no way to see what that exception is from the web UI; it looks like it just doesn't load. I don't think there's anything particularly exotic in the config to trip it up, so it looks to me like a bug in buildbot (possibly related to running on Python 3).
That seems to be a theme with this test instance; pretty much everything I click on looks incomplete whether logged in or not, and no errors appear in the logs. Is there something missing/broken in the configuration?
-- Zach
On Thu, Sep 21, 2017 at 11:12 AM, Zachary Ware <zachary.ware+pydev@gmail.com
wrote:
That seems to be a theme with this test instance; pretty much everything I click on looks incomplete whether logged in or not, and no errors appear in the logs. Is there something missing/broken in the configuration?
I'm not sure. If you can make the twistd.log files on the worker and master available, there might be some error messages in there that will help shed light on the problem.
-- Craig
On 9/22/17 4:12 AM, Zachary Ware wrote:
Hi Craig,
Sorry I've been largely unresponsive; time has been hard to come by lately.
On Thu, Sep 21, 2017 at 12:41 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
I'm happy to help look into any problems, but you'll have to tell me what those problems are for me to investigate.
For starters, there is still a worker connected to the test instance, which seems to have been failing the daily refleak builds with an exception since I connected it. However, there's no way to see what that exception is from the web UI; it looks like it just doesn't load. I don't think there's anything particularly exotic in the config to trip it up, so it looks to me like a bug in buildbot (possibly related to running on Python 3).
That seems to be a theme with this test instance; pretty much everything I click on looks incomplete whether logged in or not, and no errors appear in the logs. Is there something missing/broken in the configuration?
I can provision a Python 3 buildbot 0.9 worker upon request if that would help ease/verify/test the migration.
The FreeBSD buildbot port/package was updated to 0.9 a while ago, and given the lack of backward compatibility (with a 0.8 master), I've had to hold off on upgrades. Getting the 0.9 migration done sooner would for me result in less management and support overhead than keeping it as is, effectively an unsupported state downstream.
I would also be happy to switch the existing 2 'stable' FreeBSD (0.8) workers over to 0.9 and a 0.9 server at any time, but would want to ensure that doing so wouldn't:
- reduce branch/build-type coverage the 0.8 server config provides
- reduce visibility of their statuses to developers. (see them go and stay red)
-- Regards,
./koobs
On Wed, Sep 20, 2017 at 8:02 AM, R. David Murray <rdmurray@bitdance.com> wrote:
The question most relevant to this list is, what are the consequences on the slave maintainers? Assuming we convert the master, will the slaves be required to run python3/0.9? (That would certainly be desirable, but would it be required?)
Are there advantages for the slave maintainers in the new version?
In Buildbot 9, the terminology has changed from "slave" to "worker", so I will use that term. :)
I would like to propose the following strategy for Python.org's buildbot upgrade strategy.
(1) The master will be upgraded to Buildbot Nine, running under Python 3.
(2) The existing buildbot workers will remain unchanged running at their current 0.8.x versions under Python 2.
(3) Over time, the buildbot workers can be upgraded to 0.9.x under Python 2 or 3, as is convenient for the maintainers of those workers.
There is no rush on this, and I am willing to put the effort to push fixes into Buildbot and Twisted in order to get this to work.
-- Craig
On Wed, Sep 20, 2017 at 8:02 AM, R. David Murray <rdmurray@bitdance.com> wrote:
In response to your posting on core-mentorship Victor mentioned there were issues with the proof-of-concept, so figuring out those issues is presumably the first step.
I've been working with Zach quite a lot, and I believe we have solved the issues that Victor previously saw.
If you look at the proof of concept at:
http://buildbot.python.org/test/
This has been building things from https://github.com/zware/cpython
which is a test fork of cpython.
-- Craig
2017-10-06 21:35 GMT+02:00 Craig Rodrigues <rodrigc@crodrigues.org>:
The new UI LGTM. It seems like email notifications are also working. So everything seems to be ready from my point of view.
I propose right now to start moving (slowly) buildbot slaves from the old buildbot 0.8/Python 2 instance to the new buildbot 0.9/Python 3 instance. We can stop the migration if things go bad, or even rollback in the worst case.
I'm fully in favor of moving to buildbot 0.9 since previously we used a fork of buildbot, and I hate to maintain a fork. We have limited resources working on buildbots: mostly Zachary Ware, R. David Murray and me. I'm taking any change which gives us less work to do for the long-term :-)
We now have a major buildbot contributor, Craig Rodrigues, who offers his strong support for the migration. So if something goes wrong, we will harass him until he fixes buildbot :-D
By the way, it like the idea of eating our own dog food: migrate from Python 2 to Python 3 :-)
Victor
On Fri, Oct 6, 2017 at 1:20 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
By the way, it like the idea of eating our own dog food: migrate from Python 2 to Python 3 :-)
I performed some interop testing between Python 2 buildbot worker and Python 3 buildbot master:
https://lists.buildbot.net/pipermail/devel/2017-October/012433.html
Based on my experiments, I think it is safe to perform the migration as follows:
Update the buildbot master to buildbot 9 + Python 3. Leave the buildbot workers/slaves alone. They should be able to successfully connect to the new buildbot master and work.
For the buildbot worker/slave maintainers, give them the following options:
a. Upgrade the buildbot worker to buildbot 9 + Python 3 (MOST PREFERRED) b. Upgrade the buildbot worker to buildbot 9 + Python 2 (LESS PREFERRED, BUT WILL WORK) c. Leave the buildbot slave alone at buildbot 8 + Python 2 (LEAST PREFERRED, BUT WILL WORK)
I did all my experiments on Linux + OS X, so we might want to double check with a Windows buildbot worker. I don't have Windows environments readily available for this type of interop testing.
-- Craig
On Fri, Oct 6, 2017 at 4:33 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Based on my experiments, I think it is safe to perform the migration as follows:
Update the buildbot master to buildbot 9 + Python 3. Leave the buildbot workers/slaves alone. They should be able to successfully connect to the new buildbot master and work.
For the buildbot worker/slave maintainers, give them the following options:
a. Upgrade the buildbot worker to buildbot 9 + Python 3 (MOST PREFERRED) b. Upgrade the buildbot worker to buildbot 9 + Python 2 (LESS PREFERRED, BUT WILL WORK) c. Leave the buildbot slave alone at buildbot 8 + Python 2 (LEAST PREFERRED, BUT WILL WORK)
Yes, my own preference would be to do workers last. It sounds to me like most of the benefits of this change accrue on the master side (eliminating our custom version) and at least for my part, I'd prefer not to be critical path to allowing that upgrade to proceed.
Over a longer period, I'm curious about the preferences - are there concrete benefits in our environment of 0.9 over 0.8 on the worker side? E.g., what does upgrading bring on the worker side? In some cases an upgrade is trivial, but I've still got a few local tweaks on the Windows side that, while probably not too difficult to port, incur cost only worth doing if there is a concrete requirement or benefit.
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
-- David
On Fri, Oct 6, 2017 at 1:46 PM, David Bolen <db3l.net@gmail.com> wrote:
Over a longer period, I'm curious about the preferences - are there concrete benefits in our environment of 0.9 over 0.8 on the worker side? E.g., what does upgrading bring on the worker side?
On the worker side, the are no performance improvements migrating a buildbot-worker from 8 to 9. The only things that are there are: -> buildbot 8 worker will not run under Python 3, and never will -> I performed some minor cleanups in the buildbot-worker code regarding bytes vs. unicode issues. Not a major deal with a setup where mostly ASCII is involved, but becomes an issue when dealing with non-ASCII output from commands, and when the locale is set to something which is not UTF-8. For example, I'm working with someone in China who is trying to set up buildbot 9 + Python 3 + cp936 locale. I haven't fully fixed those issues in the buildbot-worker because I don't have access to Windows set up with that locale. :) But I'm slowly working through the issues and fixing things.
For python.org, I don't think these issues are really a concern right now, because the python.org jobs and builders are doing mostly ASCII and UTF-8 compatible stuff, and not using any "exotic" locales on the worker.
buildbot was written quite a while ago, and the assumptions about bytes/unicode were done in the Python 2 days. Obviously this has to be fixed properly for Python 3 :)
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
Thank you for the offer. I will contact you off the list to coordinate setting up some test environment. I'm a bit busy in the next few days, but maybe next week.
-- Craig
On Fri, Oct 6, 2017 at 1:46 PM, David Bolen <db3l.net@gmail.com> wrote:
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
I don't think I need this right now. If I look at: http://buildbot.python.org/all/#/workers
Your windows buildbot workers seem to be working fine with the new buildbot master.
This is one buildbot worker kloth-win64 which is not working. I don't know what the problem with that one is.
-- Craig
Yes they look to be ok now. Although for future reference, both Windows XP and 7 workers did initially break - apparently due to an incompatibility when using Python < 2.7 on the worker (I have 2.6 on those machines). The master is sending remote methods down using unicode for method names. So 2.7+ appears to be a worker requirement in this mixed-mode setup (though I doubt any new worker would try to use 2.6).
For now, I've locally modified my version of twisted on each buildbot to deal with that, pending time to have them upgraded to 2.7 (or possibly in the Win7 case, just go to 3.x).
I'm not sure if that might be the same issue with kloth-win64.
-- David
On Fri, Oct 13, 2017 at 3:59 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
On Fri, Oct 6, 2017 at 1:46 PM, David Bolen <db3l.net@gmail.com> wrote:
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
I don't think I need this right now. If I look at: http://buildbot.python.org/all/#/workers
Your windows buildbot workers seem to be working fine with the new buildbot master.
This is one buildbot worker kloth-win64 which is not working. I don't know what the problem with that one is.
-- Craig
Did you try leaving things on Python 2.6, but upgrading the worker to buidlbot-worker 0.9.12 ? buildbot-worker is still tested on Python 2.6.
-- Craig
On Fri, Oct 13, 2017 at 1:07 PM, David Bolen <db3l.net@gmail.com> wrote:
Yes they look to be ok now. Although for future reference, both Windows XP and 7 workers did initially break - apparently due to an incompatibility when using Python < 2.7 on the worker (I have 2.6 on those machines). The master is sending remote methods down using unicode for method names. So 2.7+ appears to be a worker requirement in this mixed-mode setup (though I doubt any new worker would try to use 2.6).
For now, I've locally modified my version of twisted on each buildbot to deal with that, pending time to have them upgraded to 2.7 (or possibly in the Win7 case, just go to 3.x).
I'm not sure if that might be the same issue with kloth-win64.
-- David
On Fri, Oct 13, 2017 at 3:59 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
On Fri, Oct 6, 2017 at 1:46 PM, David Bolen <db3l.net@gmail.com> wrote:
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
I don't think I need this right now. If I look at: http://buildbot.python.org/all/#/workers
Your windows buildbot workers seem to be working fine with the new buildbot master.
This is one buildbot worker kloth-win64 which is not working. I don't know what the problem with that one is.
-- Craig
Does testing include a mixture with a master under 3.x? I only ask since I'd guess that a master under 2.x might be sending parameters down as strings rather than unicode.
Anyway, no, I hadn't tried a new worker, because both the same and an earlier buildbot version was working fine on my Win8/10 workers, so I focused more on other environmental differences, and since time was tight, once I identified a workaround, I stopped for the time being.
I did misspeak in the last note - the issue was a unicode name for a keyword parameter remote function call, not the function name itself. The actual failure occurs inside twisted while dispatching the remote call from the master. It takes a PB-transmitted dictionary for the arguments, using either apply() or **kwargs against them directly. That fails with a "TypeError: keywords must be strings" under 2.6 but 2.7 is fine.
Since the dispatch failure occurs during the dispatch to buildbot-worker, but before it gets any chance to run, I'm not sure that it could change things. Unless the master changes something in how it transmits request arguments that needed a newer worker on the client to identify.
Oh, I did also peek at the latest twisted source but didn't see any differences in that path at a quick glance (aside from switching from apply() to **kw) that would seem to be able to resolve this, but that's something else I could test.
Or I could just move off of 2.6 which I last updated to in 2009 :-)
-- David
On Fri, Oct 13, 2017 at 4:17 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
Did you try leaving things on Python 2.6, but upgrading the worker to buidlbot-worker 0.9.12 ? buildbot-worker is still tested on Python 2.6.
-- Craig
On Fri, Oct 13, 2017 at 1:07 PM, David Bolen <db3l.net@gmail.com> wrote:
Yes they look to be ok now. Although for future reference, both Windows XP and 7 workers did initially break - apparently due to an incompatibility when using Python < 2.7 on the worker (I have 2.6 on those machines). The master is sending remote methods down using unicode for method names. So 2.7+ appears to be a worker requirement in this mixed-mode setup (though I doubt any new worker would try to use 2.6).
For now, I've locally modified my version of twisted on each buildbot to deal with that, pending time to have them upgraded to 2.7 (or possibly in the Win7 case, just go to 3.x).
I'm not sure if that might be the same issue with kloth-win64.
-- David
On Fri, Oct 13, 2017 at 3:59 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
On Fri, Oct 6, 2017 at 1:46 PM, David Bolen <db3l.net@gmail.com> wrote:
Oh, and I can offer any of my Windows workers for interoperability testing if you'd like me to temporarily configure them to an additional master. At least one of them (XP) is probably not worth the effort of getting to (a) vs. (b).
I don't think I need this right now. If I look at: http://buildbot.python.org/all/#/workers
Your windows buildbot workers seem to be working fine with the new buildbot master.
This is one buildbot worker kloth-win64 which is not working. I don't know what the problem with that one is.
-- Craig
On Fri, Oct 13, 2017 at 1:36 PM, David Bolen <db3l.net@gmail.com> wrote:
Does testing include a mixture with a master under 3.x? I only ask since I'd guess that a master under 2.x might be sending parameters down as strings rather than unicode.
I've tested the following combinations of master 3.x <==> worker 2.x, master 2.x <==> worker 3.x:
https://lists.buildbot.net/pipermail/devel/2017-October/012432.html
Oh, I did also peek at the latest twisted source but didn't see any differences in that path at a quick glance (aside from switching from apply() to **kw) that would seem to be able to resolve this, but that's something else I could test.
Yes, that was something in Twisted that was fixed in the past year.
Or I could just move off of 2.6 which I last updated to in 2009 :-)
It looks like you have done more than your fair share of investigation to get things working. Believe me, testing all the permutations and combinations of things is a major pain in the neck.
You have identified the problem, applied a workaround to get things working now, and have identified a path to success moving forward. If you are OK with that, then that is probably good enough.
-- Craig
I talked with Zach on IRC. He wants to move everything to buildbot nine right now. Well, if he takes the responsability, I'm totally fine with that. Let me elaborate.
If a few workers are broken, we can skip them to give time to the buildbot owner to fix them, or give us time to fix their config. It's not a blocker issue.
A blocker issue would be to break *everything*. This option seems unlikely since I'm seeing a working buildbot nine instance. Moreover, as Zach wrote me on IRC, we *can* revert to buildbot 0.8.
So ok, let's move "everything" (the server) to buildbot nine! Yahoo!
2017-10-06 22:33 GMT+02:00 Craig Rodrigues <rodrigc@crodrigues.org>:
Based on my experiments, I think it is safe to perform the migration as follows:
- Update the buildbot master to buildbot 9 + Python 3. Leave the buildbot workers/slaves alone. They should be able to successfully connect to the new buildbot master and work.
Oh great, you confirmed that we can upgrade the server right now, and only upgrade workers later. Nice.
Victor
When another project tried to convert from Buildbot 0.8 to 0.9 earlier this year, it was a total disaster and had to be reverted. I don't know if 0.9 has improved substantially since then.
My experience is that 0.9 invested a lot of effort to make the UI pretty, but it was much less usable. It also placed an order of magnitude higher computational burden on the master. An under-powered master could not be effectively queried and builds started to fall behind.
Thanks, David
On Wed, Sep 20, 2017 at 6:27 AM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
In the past two years, I've put a lot of effort into porting the following to Python 3:
Twisted: https://www.slideshare.net/CraigRodrigues1/the-onward- journey-porting-twisted-to-python-3
Buildbot: https://github.com/buildbot/buildbot/pulls?utf8=%E2%9C%93& q=is%3Aclosed%20is%3Apr%20author%3Arodrigc%20
I would like to help Python.org move to Buildbot 9 + Python 3 for the following reasons:
(1) Python.org currently uses Buildbot 0.8.14. This branch of buildbot is not under active development and is not being ported to Python 3.
(2) The current version of Buildbot (0.9.11) is on the main branch which is actively maintained. This branch has been ported to Python 3.
(3) Buildbot 0.8.14 is not actively being maintained.
(4) Buildbot 0.9.x is actively receiving updates for new features such as Docker support, and support for Hyper.sh.
A while back, I worked with Zachary Ware, who set up a proof of concept of running Buildbot 9 on Python.org.
Zachary set this up:
https://buildbot.python.org/test/
That is an instance running Buildbot 9, using Python 3.4.3 and Twisted 17.5.0.
This is only a proof of concept, and is not actively being used or worked on.
What are the next steps to completely convert Python.org to use Buildbot 9? Who can help with this?
-- Craig
Python-Buildbots mailing list Python-Buildbots@python.org https://mail.python.org/mailman/listinfo/python-buildbots
I got involved with the Buildbot project only this year (focusing on Python 3 porting).
I am aware of previous reports of performance problems with buildbot nine, but that is a bit before my time.
I believe that the earlier performance problems have been addressed. On the buildbot-users mailing list, there are some people in companies who have set up buildbot nine and have a master handling hundreds of builds.
The references implementation of buildbot is being actively used to build buildbot itself, and highlights integration with GitHub Pull Requests:
https://nine.buildbot.net/#/console
Craig
On Wed, Sep 20, 2017 at 8:16 AM, David Edelsohn <dje.gcc@gmail.com> wrote:
When another project tried to convert from Buildbot 0.8 to 0.9 earlier this year, it was a total disaster and had to be reverted. I don't know if 0.9 has improved substantially since then.
My experience is that 0.9 invested a lot of effort to make the UI pretty, but it was much less usable. It also placed an order of magnitude higher computational burden on the master. An under-powered master could not be effectively queried and builds started to fall behind.
Thanks, David
On Wed, Sep 20, 2017 at 6:27 AM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
In the past two years, I've put a lot of effort into porting the following to Python 3:
Twisted: https://www.slideshare.net/CraigRodrigues1/the-onward- journey-porting-twisted-to-python-3
Buildbot: https://github.com/buildbot/buildbot/pulls?utf8=%E2%9C%93& q=is%3Aclosed%20is%3Apr%20author%3Arodrigc%20
I would like to help Python.org move to Buildbot 9 + Python 3 for the following reasons:
(1) Python.org currently uses Buildbot 0.8.14. This branch of buildbot is not under active development and is not being ported to Python 3.
(2) The current version of Buildbot (0.9.11) is on the main branch which is actively maintained. This branch has been ported to Python 3.
(3) Buildbot 0.8.14 is not actively being maintained.
(4) Buildbot 0.9.x is actively receiving updates for new features such as Docker support, and support for Hyper.sh.
A while back, I worked with Zachary Ware, who set up a proof of concept of running Buildbot 9 on Python.org.
Zachary set this up:
https://buildbot.python.org/test/
That is an instance running Buildbot 9, using Python 3.4.3 and Twisted 17.5.0.
This is only a proof of concept, and is not actively being used or worked on.
What are the next steps to completely convert Python.org to use Buildbot 9? Who can help with this?
-- Craig
Python-Buildbots mailing list Python-Buildbots@python.org https://mail.python.org/mailman/listinfo/python-buildbots
On 21 September 2017 at 03:49, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
The references implementation of buildbot is being actively used to build buildbot itself, and highlights integration with GitHub Pull Requests:
Just noting that while the PR support in Buildbot 0.9 is definitely a nice feature, we can't use it directly in its current form, as we promise folks contributing workers to the BuildBot fleet that we'll only run approved code on their systems. Since PRs contain *un*approved code, we'd need the Buildbot trigger to be a "test this on the Buildbot fleet" comment from a reviewer with merge privileges, rather than something that happens automatically for every PR.
The other reason an approach like that would be better for us is that for the majority of changes, the risk of cross-platform compatibility issues is genuinely low, so it makes more sense to go with pre-merge CI on Linux and Windows, and only run post-merge testing across the entire fleet)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Craig Rodrigues -
David Bolen -
David Edelsohn -
Kubilay Kocak -
Nick Coghlan -
R. David Murray -
Victor Stinner -
Zachary Ware