Adding Emscripten/NodeJS buildbot
Hi!
I'm reposting this because it seems to have been eaten by the moderation filter because I subscribed incorrectly and Mailman thought I wasn't a member of the list (sorry mailing list admins!).
I am working with a small group of people to port CPython to Emscripten (and maybe eventually WASI). I currently have a repo here where we run a build against CPython main branch nightly https://github.com/ethanhs/python-wasm, but I think it is time to set up a buildbot for Emscripten. I am able to run the CPython test suite under nodejs with the proper environment set up, however several tests do fail due to issues with musl or in Emscripten (or in Python too, of course). https://gist.github.com/tiran/8d6b1e038206c2431daceec0768156d4
I have set up a VM on a server I run with 4 cores and 4GB of memory (the docs note the large memory tests require more than 512MB, but don't say how much the large memory tests take).
Just to make sure I understand the process, since Emscripten is unlike the other builders in that it uses a custom toolchain, it looks like I will need to modify the buildbot code to add an EmscriptenBuild? How does development on the buildmaster work? Is there any way to easily test it locally? Also any suggestions on how to handle cross compiling in the configuration? We need a second build for freezing modules.
Thanks for any guidance you can provide!
Best, Ethan Smith
Hi Ethan,
On Tue, Jan 18, 2022 at 8:35 AM Ethan Smith <ethan@ethanhs.me> wrote:
I'm reposting this because it seems to have been eaten by the moderation filter because I subscribed incorrectly and Mailman thought I wasn't a member of the list (sorry mailing list admins!).
It's a low-volume list with "guilty until proven innocent" moderation :). Your first message made it through, but whoever approved it didn't have time to properly respond (and I haven't had such a chance either).
I am working with a small group of people to port CPython to Emscripten (and maybe eventually WASI). I currently have a repo here where we run a build against CPython main branch nightly https://github.com/ethanhs/python-wasm, but I think it is time to set up a buildbot for Emscripten.
Great!
I have set up a VM on a server I run with 4 cores and 4GB of memory (the docs note the large memory tests require more than 512MB, but don't say how much the large memory tests take).
Our resource suggestions are probably rather out of date, but mostly
just from a speed perspective. For example, my ware-gentoo-x86
worker has only 2 GB of RAM. It doesn't run all of the bigmem tests,
but it's still effective if rather slow.
Just to make sure I understand the process, since Emscripten is unlike the other builders in that it uses a custom toolchain, it looks like I will need to modify the buildbot code to add an EmscriptenBuild? How does development on the buildmaster work? Is there any way to easily test it locally? Also any suggestions on how to handle cross compiling in the configuration? We need a second build for freezing modules.
You're blazing new trails here, so I'm afraid you're largely going to be on your own. Our build master configuration can be found at https://github.com/python/buildmaster-config/ and the bits you'll most likely need to add to are in https://github.com/python/buildmaster-config/blob/main/master/custom/factori... and https://github.com/python/buildmaster-config/blob/main/master/custom/builder.... It should (hopefully, it's been a while since I tried :)) be relatively straightforward to get a copy of the master running locally to test against.
Feel free to make whatever PR(s) you need, and hopefully one of us will get a chance to review quickly. I can't make any guarantees about my own availability to review, but I will do my best to answer any other questions you have.
Thanks for working on this! Zach
Hi Zach,
I appreciate your response!
On Tue, Jan 18, 2022 at 7:49 AM Zachary Ware <zach@python.org> wrote:
Hi Ethan,
On Tue, Jan 18, 2022 at 8:35 AM Ethan Smith <ethan@ethanhs.me> wrote:
I'm reposting this because it seems to have been eaten by the moderation filter because I subscribed incorrectly and Mailman thought I wasn't a member of the list (sorry mailing list admins!).
It's a low-volume list with "guilty until proven innocent" moderation :). Your first message made it through, but whoever approved it didn't have time to properly respond (and I haven't had such a chance either).
Yeah I was mostly concerned because I didn't see my own posts showing up on the archive yesterday. I seem to have found a bug in mailman where I can simultaneously be a member and not a member of the list, so... trying to sort that one out :)
I am working with a small group of people to port CPython to Emscripten (and maybe eventually WASI). I currently have a repo here where we run a build against CPython main branch nightly https://github.com/ethanhs/python-wasm, but I think it is time to set up a buildbot for Emscripten.
Great!
I have set up a VM on a server I run with 4 cores and 4GB of memory (the docs note the large memory tests require more than 512MB, but don't say how much the large memory tests take).
Our resource suggestions are probably rather out of date, but mostly just from a speed perspective. For example, my
ware-gentoo-x86
worker has only 2 GB of RAM. It doesn't run all of the bigmem tests, but it's still effective if rather slow.
Ok, that's good to know. Since there are two builds to do I expect this may be a slower build bot unfortunately :/
Just to make sure I understand the process, since Emscripten is unlike the other builders in that it uses a custom toolchain, it looks like I will need to modify the buildbot code to add an EmscriptenBuild? How does development on the buildmaster work? Is there any way to easily test it locally? Also any suggestions on how to handle cross compiling in the configuration? We need a second build for freezing modules.
You're blazing new trails here, so I'm afraid you're largely going to be on your own. Our build master configuration can be found at https://github.com/python/buildmaster-config/ and the bits you'll most likely need to add to are in
https://github.com/python/buildmaster-config/blob/main/master/custom/factori... and https://github.com/python/buildmaster-config/blob/main/master/custom/builder... . It should (hopefully, it's been a while since I tried :)) be relatively straightforward to get a copy of the master running locally to test against.
Ok that's great to hear! I'll play around with running a master and try working on a useful worker config then :)
Feel free to make whatever PR(s) you need, and hopefully one of us will get a chance to review quickly. I can't make any guarantees about my own availability to review, but I will do my best to answer any other questions you have.
Thank you, I really appreciate it! I'll probably poke at things a bit and come back with questions if I have any, then hopefully make a PR.
Ethan
Thanks for working on this! Zach
participants (2)
-
Ethan Smith
-
Zachary Ware