Hi all, Short version ============= We now have a numpy buildbot running at http://buildbot.scipy.org Long version ============ Albert Strasheim and I set up a buildbot for numpy this week. For those of you unfamiliar with The Buildbot, it is """ ...a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve. The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin. """ While we are still working on automatic e-mail notifications, the system already provides valuable feedback -- take a look at the waterfall display: http://buildbot.scipy.org If your platform is not currently on the list, please consider volunteering a machine as a build slave. This machine will be required to run the buildbot client, and to build a new version of numpy whenever changes are made to the repository. (The machine does not have to be dedicated to this task, and can be your own workstation.) We'd like to thank Robert Kern, Jeff Strunk and Gert-Jan van Rooyen who helped us to get the ball rolling, as well as Neilen Marais for offering his workstation as a build slave. Regards Stéfan
Stefan van der Walt wrote:
Hi all,
Short version =============
We now have a numpy buildbot running at
Long version ============
Albert Strasheim and I set up a buildbot for numpy this week. For those of you unfamiliar with The Buildbot, it is
""" ...a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.
The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin. """
While we are still working on automatic e-mail notifications, the system already provides valuable feedback -- take a look at the waterfall display:
If your platform is not currently on the list, please consider volunteering a machine as a build slave. This machine will be required to run the buildbot client, and to build a new version of numpy whenever changes are made to the repository. (The machine does not have to be dedicated to this task, and can be your own workstation.)
We'd like to thank Robert Kern, Jeff Strunk and Gert-Jan van Rooyen who helped us to get the ball rolling, as well as Neilen Marais for offering his workstation as a build slave.
This is really great new :) Thanks for the hard work, David
On Sat, Jun 16, 2007 at 10:11:55AM +0200, Stefan van der Walt wrote:
Hi all,
Short version =============
We now have a numpy buildbot running at
While we are still working on automatic e-mail notifications, the system already provides valuable feedback -- take a look at the waterfall display:
If your platform is not currently on the list, please consider volunteering a machine as a build slave. This machine will be required to run the buildbot client, and to build a new version of numpy whenever changes are made to the repository. (The machine does not have to be dedicated to this task, and can be your own workstation.)
Awesome. I've got a iBook (PPC G4) running OS X that can be used as a slave (it's just being a server right now). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
David M. Cooke wrote:
Awesome. I've got a iBook (PPC G4) running OS X that can be used as a slave (it's just being a server right now).
It looks like they already have a PPC OS-X box. Anyone have an Intel machine to offer up? (mine's a PPC Dual G5) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Jun 18, 2007 at 09:16:06AM -0700, Christopher Barker wrote:
David M. Cooke wrote:
Awesome. I've got a iBook (PPC G4) running OS X that can be used as a slave (it's just being a server right now).
It looks like they already have a PPC OS-X box. Anyone have an Intel machine to offer up? (mine's a PPC Dual G5)
That one's mine. You can tell because it's slow ;-) A 64-bit G5 build would be good too, as its longdouble semantics are different IIRC. And I don't think there's a Python 2.3 builder yet? -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
David M. Cooke wrote:
That one's mine. You can tell because it's slow ;-) A 64-bit G5 build would be good too, as its longdouble semantics are different IIRC.
Well, I have a DualG5, running OS-X 10.4.9, and Python 2.3, 2.4, and 2.5 installed (though I'm only using 2.5 for anything new now). However, I don't think I've got 64bit anything running. Also, I'm behind a pretty rigid firewall -- what protocols are used to communicate between the buildbot and the nodes? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Stefan van der Walt schrieb:
If your platform is not currently on the list, please consider volunteering a machine as a build slave. This machine will be required to run the buildbot client, and to build a new version of numpy whenever changes are made to the repository. (The machine does not have to be dedicated to this task, and can be your own workstation.)
I have a windows XP 64-bit machine (a VMWare image, actually) that is only used as a buildbot client for python itself. I it helps, and if it isn't too much work for me I offer to install a numpy buildbot client on it. It has Visual Studio 2003 and 2005 installed, MS Platform SDK, and for the buildbot python 2.4 (32-bit), buildbot (0.7.5 ?), and twisted. Thomas
I have the potential to add OS X Server Intel (64-bit) and OS X Intel (32-bit) to the list, if I can convince my boss that the security risk (including DOS from compile times) is minimal. I've compiled both numpy and scipy many times, so I'm not worried about resources for a single compile/test, but can any of the regular developers tell me about how many commits there are per day that will trigger a compile/test? About the more general security risk of running a buildbot slave, from my reading of the buildbot manual (not the source, yet), it looks like the slave is a Twisted server that runs as a normal user process. Is there any sort of sandboxing built into the buildbot slave or is that the responsibility of the OS (an issue I'll have to discuss with our IT)? On a side note, buildbot.scipy.org goes to the DSP lab, Univ. of Stellenbosch's home page, not the buildbot status page. Thanks, Barry On 6/16/07, Stefan van der Walt <stefan@sun.ac.za> wrote:
Hi all,
Short version =============
We now have a numpy buildbot running at
Long version ============
Albert Strasheim and I set up a buildbot for numpy this week. For those of you unfamiliar with The Buildbot, it is
""" ...a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve.
The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin. """
While we are still working on automatic e-mail notifications, the system already provides valuable feedback -- take a look at the waterfall display:
If your platform is not currently on the list, please consider volunteering a machine as a build slave. This machine will be required to run the buildbot client, and to build a new version of numpy whenever changes are made to the repository. (The machine does not have to be dedicated to this task, and can be your own workstation.)
We'd like to thank Robert Kern, Jeff Strunk and Gert-Jan van Rooyen who helped us to get the ball rolling, as well as Neilen Marais for offering his workstation as a build slave.
Regards Stéfan _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Mon, 2 Jul 2007 17:26:15 -0700, "Barry Wark" <barrywark@gmail.com> wrote:
On a side note, buildbot.scipy.org goes to the DSP lab, Univ. of Stellenbosch's home page, not the buildbot status page.
Sorry about that -- I misconfigured Apache. Everything should be fine now. Cheers Stéfan
Stefan, No worries. I thought it was something like that. Any thoughts on my other questions? I'd love to have some ammunition to take to my boss. Thanks, Barry On 7/7/07, stefan <stefan@sun.ac.za> wrote:
On Mon, 2 Jul 2007 17:26:15 -0700, "Barry Wark" <barrywark@gmail.com> wrote:
On a side note, buildbot.scipy.org goes to the DSP lab, Univ. of Stellenbosch's home page, not the buildbot status page.
Sorry about that -- I misconfigured Apache. Everything should be fine now.
Cheers Stéfan
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hello On Mon, 02 Jul 2007, Barry Wark wrote:
I have the potential to add OS X Server Intel (64-bit) and OS X Intel (32-bit) to the list, if I can convince my boss that the security risk
Sounds good. We could definitely use these platforms.
(including DOS from compile times) is minimal. I've compiled both
Currently we don't allow builds to be forced from the web page, but this might change in future.
numpy and scipy many times, so I'm not worried about resources for a single compile/test, but can any of the regular developers tell me about how many commits there are per day that will trigger a compile/test?
We currently only build NumPy. SciPy should probably be added at some point, once we figure out how we want to configure the Buildbot to do this. NumPy averages close to 0 commits per day at this point. SciPy is more active. Between the two, on a busy day, you could expect more than 10 and less than 100 builds.
About the more general security risk of running a buildbot slave, from my reading of the buildbot manual (not the source, yet), it looks like the slave is a Twisted server that runs as a normal user process. Is there any sort of sandboxing built into the buildbot slave or is that the responsibility of the OS (an issue I'll have to discuss with our IT)?
Through the buildbot master configuration, we tell your buildslave what to check out and which commands to execute. We have set it up to do the build in terms of a Makefile, so the master will tell the slave to run "make build" followed by "make test". Here you can make your own machine do anything that hopefully involves running python setup.py, etc. However, the configuration on the master can be changed to make your slave execute any command. In short, any NumPy/SciPy committer or anyone who controls the build master configuration (i.e., me, Stefan, our admin person, a few other people who have root access on that machine and anybody who successfully breaks into it) can make your build machine execute arbitrary code as the build slave user. The chance of this happening is small, but it's not impossible, so if this risk is unacceptable to you/your IT people, running a build slave might not be for you. ;-) Cheers, Albert
Albert, Thanks for the info! I will run it by folks here and see what we can fiure out. We're using numpy and scipy very heavily in our internal software, so we have an interest in making sure it works on our platform. Hopefully we (here) can agree on a strategy that will satisfy me and the IT people. I'll get back to you ASAP. thanks, Barry On 7/8/07, Albert Strasheim <fullung@gmail.com> wrote:
Hello
On Mon, 02 Jul 2007, Barry Wark wrote:
I have the potential to add OS X Server Intel (64-bit) and OS X Intel (32-bit) to the list, if I can convince my boss that the security risk
Sounds good. We could definitely use these platforms.
(including DOS from compile times) is minimal. I've compiled both
Currently we don't allow builds to be forced from the web page, but this might change in future.
numpy and scipy many times, so I'm not worried about resources for a single compile/test, but can any of the regular developers tell me about how many commits there are per day that will trigger a compile/test?
We currently only build NumPy. SciPy should probably be added at some point, once we figure out how we want to configure the Buildbot to do this. NumPy averages close to 0 commits per day at this point. SciPy is more active. Between the two, on a busy day, you could expect more than 10 and less than 100 builds.
About the more general security risk of running a buildbot slave, from my reading of the buildbot manual (not the source, yet), it looks like the slave is a Twisted server that runs as a normal user process. Is there any sort of sandboxing built into the buildbot slave or is that the responsibility of the OS (an issue I'll have to discuss with our IT)?
Through the buildbot master configuration, we tell your buildslave what to check out and which commands to execute. We have set it up to do the build in terms of a Makefile, so the master will tell the slave to run "make build" followed by "make test". Here you can make your own machine do anything that hopefully involves running python setup.py, etc. However, the configuration on the master can be changed to make your slave execute any command.
In short, any NumPy/SciPy committer or anyone who controls the build master configuration (i.e., me, Stefan, our admin person, a few other people who have root access on that machine and anybody who successfully breaks into it) can make your build machine execute arbitrary code as the build slave user.
The chance of this happening is small, but it's not impossible, so if this risk is unacceptable to you/your IT people, running a build slave might not be for you. ;-)
Cheers,
Albert _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
participants (8)
-
Albert Strasheim -
Barry Wark -
Christopher Barker -
David Cournapeau -
David M. Cooke -
stefan -
Stefan van der Walt
-
Thomas Heller