[issue13124] Add "Running a Build Slave" page to the devguide
New submission from Eric Snow <ericsnowcurrently@gmail.com>: Here's a patch to add a page to the devguide on running a build slave. I copied pretty liberally from the wiki (http://wiki.python.org/moin/BuildBot). That page may or may not be up to date (I don't know), so the bulk of this new page may also be out of date... The current devguide buildbots page is oriented toward the use of buildbots in the dev process. In contrast, this new page is all about running one. I figured it is yet another easy way that people can contribute to Python. I didn't stick in the windows-specific buildbot info (http://wiki.python.org/moin/BuildbotOnWindows), but did link to the wiki page. I did include the info, as far as I understood it, from a recent python-dev thread (http://mail.python.org/pipermail/python-dev/2011-October/113935.html). I added a link to the devguide index page, but it may need to be shifted to the right spot (maybe right after "Running & Writing Tests"). I also renamed buildbots.rst to continuous_integration.rst (and updated reference accordingly). I want to be sure attribution is appropriate, regarding the text pulled from the wiki page and from the python-dev thread. Is linking to the thread sufficient? This is a rough stab (especially with the inclusion of the still-ongoing python-dev thread); but hopefully this gets a ball rolling and doesn't take much work to polish. :) ---------- assignee: docs@python components: Devguide, Documentation files: buildslaves.diff keywords: patch messages: 145120 nosy: brett.cannon, docs@python, eric.snow priority: normal severity: normal status: open title: Add "Running a Build Slave" page to the devguide type: feature request versions: Python 3.3 Added file: http://bugs.python.org/file23338/buildslaves.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Ezio Melotti <ezio.melotti@gmail.com> added the comment: A few comments: * are you following some convention about the spelling of buildbot/Buildbot/BuildBot? * "+3. the standard development toolchain." Which is? I think a compiler is the only missing bit. * "(in which case you'll have to install Python and Twisted first)." I would say "you'll need" and not assume that they are not installed. * "a couple of third-party libraries (such as zlib and OpenSSL)." Actually these are not required, and it might even be good to have some buildbot without e.g. zlib. * "+ % su - buildbot" I think we use $ elsewhere in the doc. You can also set the highlightlang to sh (iirc). * "/usr/local/bin", these should get some markup: either ``..`` or :file:`...` (is :file: ok for dirs too?). * "You'll need to get someone to create the slavename/slavepasswd on dinsdale.python.org before doing this." So maybe put it before? * "monitor the build results and solve any setup issues causing test failures" If running a buildbot implies some responsibilities, maybe they should be mentioned earlier. * "executing arbitrarily horrible and/or malicious code" This sounds bad and misleading. At worst the test can use resources extensively and/or cause crashes, but that doesn't mean that the code is arbitrary, horrible, or malicious. * "for cutting of the builder" s/of/off/ * "Therefore builders should be set up in both configurations" This might lead build masters to think they have to set up both. * "Some tests, as the start of `this thread " You already have a link to this at the end of the section * "+Running a build slave requires some network access." Isn't 9020 the only *required* one? If the others are not available, test will (should) be skipped. ---------- nosy: +ezio.melotti stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Eric Snow <ericsnowcurrently@gmail.com> added the comment: Great feedback! Keep in mind that nearly all the content in my patch is pulled unedited from either the wiki page and the python-dev thread I mentioned. I'll work on cleaning it up when I get a chance (probably after Wednesday's PyCon CFP closes :). If anyone wants to do it before I get a chance, they are welcome to it. Either way I think such a page will be helpful to have in the devguide. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Stefan Krah <stefan-usenet@bytereef.org> added the comment: I think the whole security paragraph should be deleted. The discussion on python-dev has overstated the risks. Anyone who is *that* security conscious and in effect does not trust the Python committers should also audit the thousands of lines of ./configure and any code that is executed by `make install`, to name a few "security risks". What I would like to see in the devguide is a cookbook shell script with minimal comments that interested persons can simply run. The easiest way for me (I'm running two bots) was to install Python-2.7 from source, then pip, then `pip install buildslave`. (Yes, as root :). That automatically gets everything right and you don't depend on stale packages from the distro. ---------- nosy: +skrah _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Eric Snow <ericsnowcurrently@gmail.com> added the comment: Per Ezio's and Stefan's feedback I cleaned things up a little: * consolidated on "Buildbot" * clarified the meaning of the "standard development toolchain" * % changed to $ * cleaned up the network ports section * removed a bunch of outdated/unnecessary info I did not do anything about syntax highlighting for lack of know-how. I also did not add anything about the responsibilities of running a build slave, mostly because I have no idea what those are. :) Much of the section on installation needs double-checking as I am not sure how much of it is necessary and how much not. This is about as far as I think I will be able to take this tracker issue. Unfortunately I don't have a lot of time to get up to speed on how Buildbot works, and I know nearly nothing about it currently. I'm hoping that finishing this won't be much work for someone with a working knowledge of our buildbots. I've tried to get it close. It's a gap in the devguide that I think would be worth filling. Incidently, I pulled the section on security (which motivated me here in the first place) and stuck it into the wiki page. If any of that is worth pulling back in, there it is. Thanks! ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Changes by Eric Snow <ericsnowcurrently@gmail.com>: Added file: http://bugs.python.org/file23479/devguide-buildbots-2.diff _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Changes by Tshepang Lekhonkhobe <tshepang@gmail.com>: ---------- nosy: +tshepang _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Stefan Krah <stefan-usenet@bytereef.org> added the comment: Here's a terse shell script that IMO even moderately experienced admins will prefer to the current version. I'm not sure if the devguide is the right place for this, since non-devs are very welcome to set up buildbots. ---------- nosy: +pitrou Added file: http://bugs.python.org/file24398/buildslave_install.sh _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment:
Here's a terse shell script that IMO even moderately experienced admins will prefer to the current version.
I'm sure some admins will prefer using their system's packages (I think buildbot is packaged for Debian/Ubuntu, I see it in Mageia's packages, not sure about Fedora). Anyway, the current instructions are on the wiki: http://wiki.python.org/moin/BuildBot You could add your script or link to it there. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Stefan Krah <stefan-usenet@bytereef.org> added the comment: Antoine Pitrou <report@bugs.python.org> wrote:
I'm sure some admins will prefer using their system's packages (I think buildbot is packaged for Debian/Ubuntu, I see it in Mageia's packages, not sure about Fedora).
Yes, certainly. I had a bad experience using the packaged buildbot-slave, but I don't remember which distro it was. Generally speaking, I've had so much trouble with rarely used packages on multiple distros that I *always* compile or install that kind of software directly from upstream.
Anyway, the current instructions are on the wiki: http://wiki.python.org/moin/BuildBot
That's probably the best place. Eric: I'm afraid the excerpts from the mailing list discussion you put there are a bit confusing and don't reflect any kind of majority opinion. For example I'm running two bots and I don't even know what this means: "Some tests, as the start of this thread [...] indicates, must have some special logic to make sure they do or do not run, or run differently, in privileged vs. unprivileged configurations, but generally speaking most things should work in both places." That does not concern a buildbot owner, since he has no influence on that. Also, you included that nasty quote "executing arbitrarily horrible and/or malicious code". I re-read the thread, and that quote is a half-sarcastic concession from Glyph (who actually had a much lighter view on the whole situation initially!) to the concerns of the people who were almost accusing buildbot owners of being clueless. The other side of the discussion (Martin's view) is missing completely, despite the fact that most core developers *probably* share that view. I'm sure you meant well, but I think we should give buildbot admins some credit. Experienced admins don't need any advice, inexperienced admins would be served better by a cookbook approach. The script I posted is a start: All buildbot software is in /home/buildbot and runs under the password-less user buildbot. In combination with a VM, this is enough for me. If anyone thinks this is not secure enough and wants to add chroot, jails, whatever, the best way forward is to post an improved version. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Changes by Stefan Krah <stefan-usenet@bytereef.org>: Added file: http://bugs.python.org/file24399/buildslave_install.txt _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Changes by Stefan Krah <stefan-usenet@bytereef.org>: Removed file: http://bugs.python.org/file24398/buildslave_install.txt _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Eric Snow <ericsnowcurrently@gmail.com> added the comment: No problem, Stefan. :) The main motivation was to capture the discussion at the time so that something actually came of it. Adding info to the devguide, essentially the catalog/how-to of core dev activities, was meant to simply provide another clear avenue for those that wanted to help with Python's core development. If not a dedicated page, we should at least have a link in the "Contributing" section to the (single, authoritative) resource we have on setting-up/running a build slave, regardless of where that resource resides. That said, I gladly defer to those that are more involved with the buildbots. Incidentally, the second patch is a bit cleaner than the first. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Stefan Krah <stefan-usenet@bytereef.org> added the comment: Eric Snow <report@bugs.python.org> wrote:
Incidentally, the second patch is a bit cleaner than the first.
Yes, indeed it is. :) To prevent a misunderstanding: In my last mail I was talking about the "Build Slaves and Security" additions to: http://wiki.python.org/moin/BuildBot ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Brett Cannon added the comment: Did you want to commit this patch yourself, Eric, since you have commit privs now? ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Eric Snow added the comment: Yeah, at some point I will. :) I've been pretty focused on implementing a C-OrderedDict, which I'm (hopefully) wrapping up pretty soon. Once that happens I'm going to revisit a bunch of the open tickets I have. If you want to push this sooner than that, feel free. :) ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Eric Snow added the comment: Looking this over, it seems like there were outstanding objections to adding this to the devguide and to the content. I still think the devguide is the right place and that something like my second patch would be appropriate. Before I take any time to update the patch, does anyone object to the location or intent of the changes? Also, I'm fine with adapting/incorporating or linking to Stefan's script. The whole point is to make it as easy as possible to spin up and run a build slave. :) ---------- versions: +Python 3.4 -Python 3.3 _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Stefan Krah added the comment: Eric Snow <report@bugs.python.org> wrote:
Looking this over, it seems like there were outstanding objections to adding this to the devguide and to the content.
I have no issues with the content of the second patch. However, snakebite has changed the situation a little: We don't need more buildbots, we need experts in AIX and HP-UX. For DragonFlyBSD, NetBSD and OpenBSD we'd ideally need someone who reports the OS bugs (there are a couple) upstream so that the bots finally turn green and become useful. So on fact the first paragraph of the patch is now a little outdated. Given the current situation I think it would be more useful to add http://mail.python.org/pipermail/python-dev/2012-September/121651.html to the devguide. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Ezio Melotti added the comment:
Before I take any time to update the patch, does anyone object to the location or intent of the changes?
Adding a new page to the devguide seems OK to me. It makes the devguide bigger, but it can easily be ignored by developers/contributors if they are not interested. ---------- _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Changes by Stefan Krah <stefan-usenet@bytereef.org>: ---------- nosy: -skrah _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
Carol Willing added the comment: Closing this issue since a newer issue #24440 addresses devguide documentation of installing and running a buildbot. Thanks to R. David Murray for submitting the patch to #24440. ---------- nosy: +willingc resolution: -> duplicate stage: patch review -> resolved status: open -> closed superseder: -> Move the buildslave setup information from the wiki to the devguide _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
R. David Murray added the comment: Hmm. You'd think I'd remember to search the bug tracker before opening a new issue :(. I'll review this and make sure I've incorporated anything relevant from here into my patch when I've updated it. I think the security information is useful, but based on Stefan's feedback I'll fix the language to be informative rather than cautionary. I'll be updating the patch in the linked issue in the near future. ---------- nosy: +r.david.murray _______________________________________ Python tracker <report@bugs.python.org> <http://bugs.python.org/issue13124> _______________________________________
participants (8)
-
Antoine Pitrou -
Brett Cannon
-
Carol Willing
-
Eric Snow
-
Ezio Melotti
-
R. David Murray
-
Stefan Krah
-
Tshepang Lekhonkhobe