Sad status of Python 3.x buildbots

Hi, I'm using Python buildbots to ensure that my changes don't fail on some platform. It's important for changes close to the operation system. The problem is that many buildbots are ill. Before, only a few buildbots had sporadic failures. Now most buildbots are always fail (are red). Here is an incomplete list of failures seen on buildbots. Before I opened an issue for each bug, but usually nobody takes care of them. So today I'm trying the mailing list. How can we fix all these issues to have more "green" buildbots? Can anyone help on this task? AMD64 OpenIndiana 3.x: a lot of tests fail with OSError(12, "Not enough space") or MemoryError. It's probably on issue on the host. AMD64 Snow Leop 3.x: many tests are not reliable (stable) on this platform. For example, test_logging.test_race() sometimes fail with PermissionError(1, "Operation not permitted: '/tmp/test_logging-3-bjulw8iz.log'"). Another example, test_asyncio.test_stdin_broken_pipe() sometimes fail with an assertion error because BrokenPipeError was not raised. Do we still support this old version of Mac OS X? Released in 2009, it is 5 years old. Is upgrading to Maverick (10.9) free and possible on old Mac computers? I don't have access to this old OS. x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it. x86 Windows Server 2003 [SB] 3.x: test_build_ext() of test_distutils hangs during 1 hang before being killed, it hangs something in the C compiler. x86 Windows7 3.x: test_nntplib fails with OSError("cannot read from timed out object"). x86 XP-4 3.x: test_distutils fails because Visual Studio linker (link.exe) fails with the error 1181 (cannot open input file). test_capi.test_forced_io_encoding() fails with an assertion error because sys.__stdin__ is None. AMD64 Solaris 11 [SB] 3.x: sometimes, tests fail with MemoryError. test_uuid.test_uuid4() fails with an assertion error. ctypes has issue with the sign of integer numbers (bug in libffi?). ARMv7 3.x: "Read-only file system", Mercurial fails... x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs. x86 FreeBSD 6.4 3.x: test_urllib2net.test_urlwithfrag() fails with urllib.error.URLError: <urlopen error unknown url type: https>. test_tcl has many issues. test_multiprocessing_spawn fails whereas the test should be skipped. x86 OpenBSD 5.5 3.x: many failing tests. x86 OpenIndiana 3.x: MemoryError. TestReadline.test_init() also fails. x86 Tiger 3.x: test_nntplib fails with OSError("cannot read from timed out object". I skipped "SPARC Solaris 10 OpenCSW 3.x" and "PPC64 AIX 3.x" in my list, I'm not really interested by these platforms. Victor

On 3 Sep 2014 08:15, "Victor Stinner" <victor.stinner@gmail.com> wrote:
x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it.
Sorry, I haven't been a very good maintainer for that buildbot (the main reason it never graduated to the "stable" list). If you send me your public SSH key, I can add it (I think - if not, I can ask Luke to do it). Alternatively, CentOS 6 may exhibit the same problem.
Cheers, Nick.

On 3 Sep 2014 09:00, "Brian Curtin" <brian@python.org> wrote:
On Tue, Sep 2, 2014 at 5:53 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
From a completely different perspective, does anyone have experience
with using BuildBot with OpenStack hosted clients? We may be able to take advantage of the PSF's new(ish) Rackspace infrastructure to provide more stable test VMs.
Is this a Buildbot feature (as in Buildbot master spins up VMs fresh for
a test run, or something), or do you just want to spin up a bunch of VMs, give access, and we configure them the same as we do today? The former would be better, but I have no idea if BuildBot supports it natively. The latter may still help if it replaces resource constrained VMs (and would also mean the PSF infra team could grant access for investigations). Cheers, Nick.

On Wed, 3 Sep 2014 08:53:51 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm not sure that's an answer to the problem. What we need is not more machines, but dedicated buildbot maintainers. That also means being willing to diagnose issues themselves rather than kindly offer SSH access, by the way ;-) (I can't speak for Victor, but being offered SSH access when I point out an issue on a remote machine is really a depressing answer to get, as far as I'm concerned) And, we also need some people to look after the master configuration and status - yes, that would have been me, but I've not been very motivated recently :-) Regards Antoine.

On 03/09/14 02:37, Antoine Pitrou wrote:
I'm not sure that's an answer to the problem. What we need is not more machines, but dedicated buildbot maintainers.
I would love to get an email if my buildbots are consistently RED for a few hours. In the past Antoine, Victor and others pinged me about issues in my buildbots (OpenIndiana). I feel shame and I think it is not very productive either. If red state in the buildbots were really trustable, the committer of that changelog should be bugged too. Since we have quite a few false positives, that would be probably annoying and no productive. Decreasing false positives should be a priority. In my case another issue is that my buildbots are hosted as a Solaris ZONE inside a OpenIndiana development machine I don't manage. Their own usage varies and can impact buildbots resource limits like memory, swap space, etc. Thanks for your patience and for notifying me issues when you suffer them. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On Sat, Oct 11, 2014 at 2:45 AM, Jesus Cea <jcea@jcea.es> wrote:
Likewise. In the earliest days of my buildbots, I kept a very close eye on them (which was good; they had some occasional network glitches, which I had to resolve), but once they became stable, I stopped watching. Now, I almost never check them - as long as the VM's running, I basically assume all's well. It'd be very helpful if a computer could auto-email any time they're down for any length of time. Is the info available in some way? Could I write a little monitor at my end that asks every hour if my buildbots can be seen? ChrisA

On 10/10/14 17:56, Chris Angelico wrote:
Could I write a little monitor at my end that asks every hour if my buildbots can be seen?
AFAIK maintainers already get an email if the buildbot vanishes long enough. I am more interested in getting an email when my buildbot is consistently red because somebody committed something it doesn't like two months ago... -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On Fri, 10 Oct 2014 18:00:06 +0200, Jesus Cea <jcea@jcea.es> wrote:
I think they are supposed to, but I've never gotten one, not even when my gentoo buildbots suffered a hardware failure. A month or so later Antoine emailed me, and I told him to remove them at least for now, since I don't currently have replacement hardware. (I'm hoping to fix that, but I have to find the time...) That said, there has never as far as I know been an active hook to email the maintainer when the buildbot is red. I don't know if buildbot supports that or not, so that would be the first thing to check. --David

It's https://bugs.python.org/issue19884 On Fri, Oct 10, 2014, at 12:08, R. David Murray wrote:

On Fri, 10 Oct 2014 12:08:54 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
As far as I remember it's a relaying problem on the SMTP server. Perhaps Benjamin can confirm.
I don't know if it would support e-mailing the maintainer rather than whoever committed the last changes. Besides, I'm afraid our test suite isn't stable enough for such e-mail notifications to be bearable... Regards Antoine.

On 10/10/14 17:45, Jesus Cea wrote:
Thanks for your patience and for notifying me issues when you suffer them.
Another issue is changes that actually breaks buildbots and I don't actually know where to start debugging. For instance, currently: <http://buildbot.python.org/all/builders/x86%20OpenIndiana%202.7/builds/2541/...> """ ====================================================================== FAIL: test_init (test.test_readline.TestReadline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/export/home/buildbot/32bits/2.7.cea-indiana-x86/build/Lib/test/test_readline.py", line 52, in test_init self.assertEqual(stdout, b'') AssertionError: '\x1b[?1034h' != '' """ No idea what to do now, beside forcing bisection builds and blaming somebody else :). -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

Nick Coghlan writes:
I wonder how many of these buildbots could be maintained by the kind of folks who show up on core-mentorship asking "how can I help?" Just a thought -- I wouldn't be surprised if the reaction is universal horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" And of course most want to write code, not sysadm.

On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Maintaining a buildbot isn't hard. (Although one thing I'm not sure of: If my bot goes down for an extended period of time, is any sort of automated email sent to me? I don't often check their status.) But it does mean a measure of trust in some external entity, or else some very careful rules (mainly firewall), which not every coder will know about. ChrisA

Hello all I don't mind helping out with maintaining buildbots / other build machines. Although I don't have a lot of experience with this sort of thing (I usually just ran a Jenkins for my own work), I think it would be a useful way to contribute. Let me know what I should do if you are all fine with this. Thanks Shorya Raj On Wed, Sep 3, 2014 at 2:52 PM, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Sep 4, 2014 at 3:32 AM, francis <francismb@email.de> wrote:
I'm more talking about how there's a (virtual) machine that I run, executing code sent to me by someone else (the Python devs). That machine needs a certain measure of access to the internet (to fetch code, and to run certain tests), and the easy and naive way to set it up is to give it full access to everything, which is the trust that I spoke of. Firewalling that box so it can't see anything it's not allowed to see is certainly possible, but that requires sysadmin skills, not coder skills, hence my comment. And it's very easy to make a mistake that you won't notice - everything works just fine, because normally that buildbot won't do anything it shouldn't. ChrisA

As mentioned, I don't mind sysadmining a bit, if required. My primary joy would be helping code python, but can't seem to figure out the ideal place to start doing so. Therefore, helping out as sysadmin would be a good start. On Wed, Sep 3, 2014 at 4:58 PM, Cameron Simpson <cs@zip.com.au> wrote:

Hi all I am using buildbot now for some time and i would be willing to contribute on that. I had small work on openstack buildbot slave but had not the proper infrastructure to get more value out of it. I like that project and automation. Anyway, if i could be of help let me know (; Short to my person: i am christian staffa and actually living in germany. Chris Sent from my Sony Xperia™ smartphone ---- Nick Coghlan wrote ----

On 03Sep2014 00:13, Victor Stinner <victor.stinner@gmail.com> wrote:
As a negative data point, my GF runs Snow Leopard by choice and will not upgrade that machine; we've both got Mavericks laptops and there are major regressions in the UI and OS behaviour (Apple UI and apps, not Python). I would imagine she's not alone in resisting change. Cheers, Cameron Simpson <cs@zip.com.au> Life IS pain, highness... anyone who tries to tell you different is trying to sell you something. - Wesley, The_Princess_Bride

2014-09-03 0:13 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:
I sent an email to Jesus Cea (owner of these buildbots).
x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it.
I sent my SSH key to Nick (owner).
I sent an email to the mailing list of the snakebite network.
I sent an email to David Bolen (owner).
ARMv7 3.x: "Read-only file system", Mercurial fails...
I sent an email to Gregory P. Smith (owner).
x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs.
I skipped the test on FreeBSD 7.2 (even if the test pass on FreeBSD 6.4): http://bugs.python.org/issue22331
x86 FreeBSD 6.4 3.x: (...) test_multiprocessing_spawn fails whereas the test should be skipped.
I skipped the test on FreeBSD 6.4: http://bugs.python.org/issue22332 Victor

I wonder if there is any interest in starting to use the opensuse build servers for continuous build and testing on redhat, fedora suse and (I think) debian: that will solve once for all the maintenance issues on those platforms (and provide a reliable build). Regards, Antonio

On 4 September 2014 11:07, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:
It sounds like a good idea in principle, but I suspect there would be a lot of devils in those details. That doesn't make it a bad idea, just something that would need a volunteer to investigate further before we could form a real opinion. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 September 2014 15:45, Shorya Raj <rajshorya@gmail.com> wrote:
Also, have you considered using Travis? It would allow us to add to the configurations, at least generally...
Many SaaS solution assume the use of specific centralised identity providers, which isn't a function we're prepared to outsource (we know we need to fix our current proliferation of identity silos, but that's a big project and there are other tasks ahead of it on the todo list for the existing infrastructure team). The limiting factor is generally time - various aspects of the current system are clumsy, but they work, so unless someone is particularly keen and willing to work through all the factors that led to the current setup and propose changes (including at least a rough concept of how ongoing maintenance will work), the status quo wins by default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 September 2014 16:59, Shorya Raj <rajshorya@gmail.com> wrote:
One that seems promising to me would be to experiment with dynamic testing in OpenStack: http://docs.buildbot.net/latest/manual/cfg-buildslaves.html#openstack Creating test VMs on demand avoids a lot of the security issues otherwise incurred by someone setting up a persistent build slave and provides a more consistent test environment. https://wiki.python.org/moin/BuildBot gives details of the test system configuration details. Due to the overlap with the infrastructure team (to make use of the Rackspace VM resources), infrastructure@python.org may be a better list to ask for more details on what OpenStack resources we have available (this idea really sits somewhere between infrastructure and python-dev, as Antoine does most of the BuildBot master maintenance). If we could get something like that running, then we might be able to trim the preconfigured buildbot fleet back to just the non-x86 machines. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yes there are "details" indeed. But not show stoppers. A prototype can be seen here: http://cclimited.webfactional.com The nice bit is testing will be executed in a total fresh instance, an added bonus On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Ok, I've put everything under the tools directory of the build server for the project home:cavallo71:opt-python-interpreters: https://build.opensuse.org/package/show/home:cavallo71:opt-python-interprete... To get started you need using the osc client (it uses the same subversion syntax, so is very simple): $> osc co home:cavallo71:opt-python-interpreters $> make -f tools/Makefile update (for the 2.7 branch) $> make -f tools/Makefile BRANCH=3 update (for the 3.x branch, I haven't fixed few issue with the build) I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if there's a general consensus on that (maybe having the python project official release build on susebuild?). Then there's the testing bit (it's separate from the python binaries build to insulate the two). It's very similar and I'll post more details when ready. I hope this helps, Thnaks Antonio Cavallo wrote:

On 8 Sep 2014 04:51, "Antonio Cavallo" <a.cavallo@cavallinux.eu> wrote:
Ok,
I've put everything under the tools directory of the build server for the
project home:cavallo71:opt-python-interpreters:
https://build.opensuse.org/package/show/home:cavallo71:opt-python-interprete...
To get started you need using the osc client (it uses the same subversion
syntax, so is very simple):
I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if
Thanks for posting this! there's a general consensus on that (maybe having the python project official release build on susebuild?). Integration into the release process will be up to the current release managers (Larry Hastings for 3.4/3.5 and Benjamin Peterson for 2.7). That may be worthwhile as a distro independent way of publishing "opt builds" of Python for Linux platforms (I'd suggest using the softwarecollections.org convention and put them under "/opt/psf/pythonXY"). Note that I think it's worth pursuing this as an integration testing tool, regardless of whether or not we end up incorporating it into the release process.
Thanks again! Regards, Nick.
https://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu

On Wed, Sep 3, 2014 at 2:52 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
ARMv7 3.x: "Read-only file system", Mercurial fails...
I sent an email to Gregory P. Smith (owner).
For mine, its a case of me not having any monitoring for it. Sending me an email worked. In this case the SSD (a very fast usb3 memory stick) died after 9 months of use. Not a surprise. I brought it back up on the internal flash for the time being. I am in the process of setting up a new system (nvidia jetson tk1) to replace this one that will be 1.5-3x faster depending on how you measure it. Monitoring idea for buildbots: If builders for all non-custom branches on a buildbot are failing for more than some period of time or number of builds each in a row, that is likely a sign of broken infrastructure or a common platform specific code change that needs to be made. anyone want to implement that? -gps

On 3 Sep 2014 08:15, "Victor Stinner" <victor.stinner@gmail.com> wrote:
x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it.
Sorry, I haven't been a very good maintainer for that buildbot (the main reason it never graduated to the "stable" list). If you send me your public SSH key, I can add it (I think - if not, I can ask Luke to do it). Alternatively, CentOS 6 may exhibit the same problem.
Cheers, Nick.

On 3 Sep 2014 09:00, "Brian Curtin" <brian@python.org> wrote:
On Tue, Sep 2, 2014 at 5:53 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
From a completely different perspective, does anyone have experience
with using BuildBot with OpenStack hosted clients? We may be able to take advantage of the PSF's new(ish) Rackspace infrastructure to provide more stable test VMs.
Is this a Buildbot feature (as in Buildbot master spins up VMs fresh for
a test run, or something), or do you just want to spin up a bunch of VMs, give access, and we configure them the same as we do today? The former would be better, but I have no idea if BuildBot supports it natively. The latter may still help if it replaces resource constrained VMs (and would also mean the PSF infra team could grant access for investigations). Cheers, Nick.

On Wed, 3 Sep 2014 08:53:51 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm not sure that's an answer to the problem. What we need is not more machines, but dedicated buildbot maintainers. That also means being willing to diagnose issues themselves rather than kindly offer SSH access, by the way ;-) (I can't speak for Victor, but being offered SSH access when I point out an issue on a remote machine is really a depressing answer to get, as far as I'm concerned) And, we also need some people to look after the master configuration and status - yes, that would have been me, but I've not been very motivated recently :-) Regards Antoine.

On 03/09/14 02:37, Antoine Pitrou wrote:
I'm not sure that's an answer to the problem. What we need is not more machines, but dedicated buildbot maintainers.
I would love to get an email if my buildbots are consistently RED for a few hours. In the past Antoine, Victor and others pinged me about issues in my buildbots (OpenIndiana). I feel shame and I think it is not very productive either. If red state in the buildbots were really trustable, the committer of that changelog should be bugged too. Since we have quite a few false positives, that would be probably annoying and no productive. Decreasing false positives should be a priority. In my case another issue is that my buildbots are hosted as a Solaris ZONE inside a OpenIndiana development machine I don't manage. Their own usage varies and can impact buildbots resource limits like memory, swap space, etc. Thanks for your patience and for notifying me issues when you suffer them. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On Sat, Oct 11, 2014 at 2:45 AM, Jesus Cea <jcea@jcea.es> wrote:
Likewise. In the earliest days of my buildbots, I kept a very close eye on them (which was good; they had some occasional network glitches, which I had to resolve), but once they became stable, I stopped watching. Now, I almost never check them - as long as the VM's running, I basically assume all's well. It'd be very helpful if a computer could auto-email any time they're down for any length of time. Is the info available in some way? Could I write a little monitor at my end that asks every hour if my buildbots can be seen? ChrisA

On 10/10/14 17:56, Chris Angelico wrote:
Could I write a little monitor at my end that asks every hour if my buildbots can be seen?
AFAIK maintainers already get an email if the buildbot vanishes long enough. I am more interested in getting an email when my buildbot is consistently red because somebody committed something it doesn't like two months ago... -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

On Fri, 10 Oct 2014 18:00:06 +0200, Jesus Cea <jcea@jcea.es> wrote:
I think they are supposed to, but I've never gotten one, not even when my gentoo buildbots suffered a hardware failure. A month or so later Antoine emailed me, and I told him to remove them at least for now, since I don't currently have replacement hardware. (I'm hoping to fix that, but I have to find the time...) That said, there has never as far as I know been an active hook to email the maintainer when the buildbot is red. I don't know if buildbot supports that or not, so that would be the first thing to check. --David

It's https://bugs.python.org/issue19884 On Fri, Oct 10, 2014, at 12:08, R. David Murray wrote:

On Fri, 10 Oct 2014 12:08:54 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
As far as I remember it's a relaying problem on the SMTP server. Perhaps Benjamin can confirm.
I don't know if it would support e-mailing the maintainer rather than whoever committed the last changes. Besides, I'm afraid our test suite isn't stable enough for such e-mail notifications to be bearable... Regards Antoine.

On 10/10/14 17:45, Jesus Cea wrote:
Thanks for your patience and for notifying me issues when you suffer them.
Another issue is changes that actually breaks buildbots and I don't actually know where to start debugging. For instance, currently: <http://buildbot.python.org/all/builders/x86%20OpenIndiana%202.7/builds/2541/...> """ ====================================================================== FAIL: test_init (test.test_readline.TestReadline) ---------------------------------------------------------------------- Traceback (most recent call last): File "/export/home/buildbot/32bits/2.7.cea-indiana-x86/build/Lib/test/test_readline.py", line 52, in test_init self.assertEqual(stdout, b'') AssertionError: '\x1b[?1034h' != '' """ No idea what to do now, beside forcing bisection builds and blaming somebody else :). -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

Nick Coghlan writes:
I wonder how many of these buildbots could be maintained by the kind of folks who show up on core-mentorship asking "how can I help?" Just a thought -- I wouldn't be surprised if the reaction is universal horror and the answer is "Are you crazy? Zero! Z-E-R-O!!" And of course most want to write code, not sysadm.

On Wed, Sep 3, 2014 at 12:47 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Maintaining a buildbot isn't hard. (Although one thing I'm not sure of: If my bot goes down for an extended period of time, is any sort of automated email sent to me? I don't often check their status.) But it does mean a measure of trust in some external entity, or else some very careful rules (mainly firewall), which not every coder will know about. ChrisA

Hello all I don't mind helping out with maintaining buildbots / other build machines. Although I don't have a lot of experience with this sort of thing (I usually just ran a Jenkins for my own work), I think it would be a useful way to contribute. Let me know what I should do if you are all fine with this. Thanks Shorya Raj On Wed, Sep 3, 2014 at 2:52 PM, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Sep 4, 2014 at 3:32 AM, francis <francismb@email.de> wrote:
I'm more talking about how there's a (virtual) machine that I run, executing code sent to me by someone else (the Python devs). That machine needs a certain measure of access to the internet (to fetch code, and to run certain tests), and the easy and naive way to set it up is to give it full access to everything, which is the trust that I spoke of. Firewalling that box so it can't see anything it's not allowed to see is certainly possible, but that requires sysadmin skills, not coder skills, hence my comment. And it's very easy to make a mistake that you won't notice - everything works just fine, because normally that buildbot won't do anything it shouldn't. ChrisA

As mentioned, I don't mind sysadmining a bit, if required. My primary joy would be helping code python, but can't seem to figure out the ideal place to start doing so. Therefore, helping out as sysadmin would be a good start. On Wed, Sep 3, 2014 at 4:58 PM, Cameron Simpson <cs@zip.com.au> wrote:

Hi all I am using buildbot now for some time and i would be willing to contribute on that. I had small work on openstack buildbot slave but had not the proper infrastructure to get more value out of it. I like that project and automation. Anyway, if i could be of help let me know (; Short to my person: i am christian staffa and actually living in germany. Chris Sent from my Sony Xperia™ smartphone ---- Nick Coghlan wrote ----

On 03Sep2014 00:13, Victor Stinner <victor.stinner@gmail.com> wrote:
As a negative data point, my GF runs Snow Leopard by choice and will not upgrade that machine; we've both got Mavericks laptops and there are major regressions in the UI and OS behaviour (Apple UI and apps, not Python). I would imagine she's not alone in resisting change. Cheers, Cameron Simpson <cs@zip.com.au> Life IS pain, highness... anyone who tries to tell you different is trying to sell you something. - Wesley, The_Princess_Bride

2014-09-03 0:13 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:
I sent an email to Jesus Cea (owner of these buildbots).
x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't have to this platform, I don't know how to fix it.
I sent my SSH key to Nick (owner).
I sent an email to the mailing list of the snakebite network.
I sent an email to David Bolen (owner).
ARMv7 3.x: "Read-only file system", Mercurial fails...
I sent an email to Gregory P. Smith (owner).
x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs.
I skipped the test on FreeBSD 7.2 (even if the test pass on FreeBSD 6.4): http://bugs.python.org/issue22331
x86 FreeBSD 6.4 3.x: (...) test_multiprocessing_spawn fails whereas the test should be skipped.
I skipped the test on FreeBSD 6.4: http://bugs.python.org/issue22332 Victor

I wonder if there is any interest in starting to use the opensuse build servers for continuous build and testing on redhat, fedora suse and (I think) debian: that will solve once for all the maintenance issues on those platforms (and provide a reliable build). Regards, Antonio

On 4 September 2014 11:07, Antonio Cavallo <a.cavallo@cavallinux.eu> wrote:
It sounds like a good idea in principle, but I suspect there would be a lot of devils in those details. That doesn't make it a bad idea, just something that would need a volunteer to investigate further before we could form a real opinion. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 September 2014 15:45, Shorya Raj <rajshorya@gmail.com> wrote:
Also, have you considered using Travis? It would allow us to add to the configurations, at least generally...
Many SaaS solution assume the use of specific centralised identity providers, which isn't a function we're prepared to outsource (we know we need to fix our current proliferation of identity silos, but that's a big project and there are other tasks ahead of it on the todo list for the existing infrastructure team). The limiting factor is generally time - various aspects of the current system are clumsy, but they work, so unless someone is particularly keen and willing to work through all the factors that led to the current setup and propose changes (including at least a rough concept of how ongoing maintenance will work), the status quo wins by default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 September 2014 16:59, Shorya Raj <rajshorya@gmail.com> wrote:
One that seems promising to me would be to experiment with dynamic testing in OpenStack: http://docs.buildbot.net/latest/manual/cfg-buildslaves.html#openstack Creating test VMs on demand avoids a lot of the security issues otherwise incurred by someone setting up a persistent build slave and provides a more consistent test environment. https://wiki.python.org/moin/BuildBot gives details of the test system configuration details. Due to the overlap with the infrastructure team (to make use of the Rackspace VM resources), infrastructure@python.org may be a better list to ask for more details on what OpenStack resources we have available (this idea really sits somewhere between infrastructure and python-dev, as Antoine does most of the BuildBot master maintenance). If we could get something like that running, then we might be able to trim the preconfigured buildbot fleet back to just the non-x86 machines. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yes there are "details" indeed. But not show stoppers. A prototype can be seen here: http://cclimited.webfactional.com The nice bit is testing will be executed in a total fresh instance, an added bonus On 4 September 2014 05:16:01 GMT+01:00, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Ok, I've put everything under the tools directory of the build server for the project home:cavallo71:opt-python-interpreters: https://build.opensuse.org/package/show/home:cavallo71:opt-python-interprete... To get started you need using the osc client (it uses the same subversion syntax, so is very simple): $> osc co home:cavallo71:opt-python-interpreters $> make -f tools/Makefile update (for the 2.7 branch) $> make -f tools/Makefile BRANCH=3 update (for the 3.x branch, I haven't fixed few issue with the build) I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if there's a general consensus on that (maybe having the python project official release build on susebuild?). Then there's the testing bit (it's separate from the python binaries build to insulate the two). It's very similar and I'll post more details when ready. I hope this helps, Thnaks Antonio Cavallo wrote:

On 8 Sep 2014 04:51, "Antonio Cavallo" <a.cavallo@cavallinux.eu> wrote:
Ok,
I've put everything under the tools directory of the build server for the
project home:cavallo71:opt-python-interpreters:
https://build.opensuse.org/package/show/home:cavallo71:opt-python-interprete...
To get started you need using the osc client (it uses the same subversion
syntax, so is very simple):
I don't think you'd bee able to use as it is because is still linked to my account (few hardcoded paths). But I can help in bootstrapping it if
Thanks for posting this! there's a general consensus on that (maybe having the python project official release build on susebuild?). Integration into the release process will be up to the current release managers (Larry Hastings for 3.4/3.5 and Benjamin Peterson for 2.7). That may be worthwhile as a distro independent way of publishing "opt builds" of Python for Linux platforms (I'd suggest using the softwarecollections.org convention and put them under "/opt/psf/pythonXY"). Note that I think it's worth pursuing this as an integration testing tool, regardless of whether or not we end up incorporating it into the release process.
Thanks again! Regards, Nick.
https://mail.python.org/mailman/options/python-dev/a.cavallo%40cavallinux.eu

On Wed, Sep 3, 2014 at 2:52 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
ARMv7 3.x: "Read-only file system", Mercurial fails...
I sent an email to Gregory P. Smith (owner).
For mine, its a case of me not having any monitoring for it. Sending me an email worked. In this case the SSD (a very fast usb3 memory stick) died after 9 months of use. Not a surprise. I brought it back up on the internal flash for the time being. I am in the process of setting up a new system (nvidia jetson tk1) to replace this one that will be 1.5-3x faster depending on how you measure it. Monitoring idea for buildbots: If builders for all non-custom branches on a buildbot are failing for more than some period of time or number of builds each in a row, that is likely a sign of broken infrastructure or a common platform specific code change that needs to be made. anyone want to implement that? -gps
participants (16)
-
A. Cavallo
-
Antoine Pitrou
-
Antonio Cavallo
-
Benjamin Peterson
-
Brian Curtin
-
Cameron Simpson
-
Chris Angelico
-
Christian Staffa
-
francis
-
Gregory P. Smith
-
Jesus Cea
-
Nick Coghlan
-
R. David Murray
-
Shorya Raj
-
Stephen J. Turnbull
-
Victor Stinner