It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release. I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Tue, Aug 3, 2010 at 20:08, Steve Holden <steve@holdenweb.com> wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
regards Steve
In my case, 2.6 in general just fell off the radar. 2/3 of the 2.6 Windows buildbots are fine, with the failing one being the case of a very slow VM machine and a time sensitive bsddb test. I looked into it this afternoon and it's something we can improve test-wise -- I'll see if I can work up a patch. It's not a problem with the module itself. As for getting the attention of more Windows devs, there are a few names I see pop up from time to time that would be nice to have on board if they have the time. I'll see if I can reach out and gauge their interest level.
On 4/08/2010 11:08 AM, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
I never go looking at the buildbots to look for problems - maybe some way of explicitly bringing such failures to peoples attention would be good - I don't recall seeing anything recently on python-dev which would prompt me to take a look. Visiting http://www.python.org/dev/buildbot/2.6/ shows a single Windows buildbot that seems to have been green for the last few builds - am I looking in the wrong place? Mark
On 04/08/2010 05:34, Mark Hammond wrote:
On 4/08/2010 11:08 AM, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
I never go looking at the buildbots to look for problems - maybe some way of explicitly bringing such failures to peoples attention would be good
Agree with that. This page looks hopeful: http://www.python.org/dev/buildbot/ with Atom/RSS feeds and an XML-RPC interface. I've subscribed to the RSS feeed which is -- from my perspective -- quite noisy. One could do something with the xml-rpc according to this: http://buildbot.net/buildbot/docs/0.7.11/#XMLRPC-server but does anyone know how easy it would be use setup a mail notifier to go to a specific Python mailing list on failure? I've looked at mail.python.org and Googled around and I can't see something which already does this, but I'm very happy to be wrong... There seems to be some previous discussion: http://mail.python.org/pipermail/python-dev/2006-October/069617.html but no sign of an outcome. TJG
On 04/08/2010 11.36, Tim Golden wrote:
On 04/08/2010 05:34, Mark Hammond wrote:
On 4/08/2010 11:08 AM, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
I never go looking at the buildbots to look for problems - maybe some way of explicitly bringing such failures to peoples attention would be good
Agree with that. This page looks hopeful:
http://www.python.org/dev/buildbot/
with Atom/RSS feeds and an XML-RPC interface. I've subscribed to the RSS feeed which is -- from my perspective -- quite noisy. One could do something with the xml-rpc according to this:
http://buildbot.net/buildbot/docs/0.7.11/#XMLRPC-server
but does anyone know how easy it would be use setup a mail notifier to go to a specific Python mailing list on failure? I've looked at mail.python.org and Googled around and I can't see something which already does this, but I'm very happy to be wrong...
There seems to be some previous discussion:
http://mail.python.org/pipermail/python-dev/2006-October/069617.html
but no sign of an outcome.
TJG _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ezio.melotti%40gmail.com
FWIW there's also http://code.google.com/p/bbreport/source/checkout We were planning to use bbreport to create weekly summary and mail them to python-dev, but someone should write some code (I could do that but it's quite low in my to-do list) and make it run once a week. Best Regards, Ezio Melotti
On 04/08/2010 02:08, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
My own problem is just the amount of ramp-up time (as a proportion of my own available time) to get hold of a problem even when I see it. (Speaking here in the more general sense of fixing Windows-related Python bugs). As one who has benefitted from the MSDN largesse I am certainly conscious of the responsibility to contribute benefits back to the Python community. On the basis that I'm far more likely to watch a buildbot which I actually administer, I have recently nudged my sysadmins here to see if they can make good on their promise to find me a spare server to use as a buildbot. I have watched the buildbot pages occasionally, especially when I see Windows-related commits going in, but several times "red" buildbots have turned out to be -- apparently -- environmental / local issues unrelated to commits. Obviously I could/should have contacted the buildbot owner to at least inform him or her that something was amiss. But somehow at that point one's technical enthusiasm for fixing a problem diminishes when it's not clear that there *is* a problem. (Grumble, grumble, mutter, mutter... :) ) While we'd certainly benefit from more Windows skills, we'd probably benefit more from people who have more *time* to look at Windows issues. OK; to propose something concrete: I'll write a blog post and advertise on python-win32 to ask for Windows people who perhaps might at least be interested in contributing time. I will also advertise (and maybe enhance) Brian Curtin's how-to doc on Windows Python core development... which I can't quite lay my hands on at the moment. Hopefully we can lower the perceived entry-bar for contribution at different levels. TJG
On 8/4/2010 3:49 AM, Tim Golden wrote:
On 04/08/2010 02:08, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I wonder if anyone can think of a way we can get some Windows skillz into the group that could assist at ties like this. Some brainstorming might find a way through.
My own problem is just the amount of ramp-up time (as a proportion of my own available time) to get hold of a problem even when I see it. (Speaking here in the more general sense of fixing Windows-related Python bugs).
As one who has benefitted from the MSDN largesse I am certainly conscious of the responsibility to contribute benefits back to the Python community. On the basis that I'm far more likely to watch a buildbot which I actually administer, I have recently nudged my sysadmins here to see if they can make good on their promise to find me a spare server to use as a buildbot.
I have watched the buildbot pages occasionally, especially when I see Windows-related commits going in, but several times "red" buildbots have turned out to be -- apparently -- environmental / local issues unrelated to commits. Obviously I could/should have contacted the buildbot owner to at least inform him or her that something was amiss. But somehow at that point one's technical enthusiasm for fixing a problem diminishes when it's not clear that there *is* a problem. (Grumble, grumble, mutter, mutter... :) )
While we'd certainly benefit from more Windows skills, we'd probably benefit more from people who have more *time* to look at Windows issues. OK; to propose something concrete: I'll write a blog post and advertise on python-win32 to ask for Windows people who perhaps might at least be interested in contributing time. I will also advertise (and maybe enhance) Brian Curtin's how-to doc on Windows Python core development... which I can't quite lay my hands on at the moment. Hopefully we can lower the perceived entry-bar for contribution at different levels.
Thanks, Tim. When I said "more Windows skillz" I should, of course, have said "more people with time to apply their Windows skillz". I imagine if we get some new blood in we can get them access to MSDN licenses should they be needed. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On 4 August 2010 08:49, Tim Golden <mail@timgolden.me.uk> wrote:
I have watched the buildbot pages occasionally, especially when I see Windows-related commits going in, but several times "red" buildbots have turned out to be -- apparently -- environmental / local issues unrelated to commits. Obviously I could/should have contacted the buildbot owner to at least inform him or her that something was amiss. But somehow at that point one's technical enthusiasm for fixing a problem diminishes when it's not clear that there *is* a problem. (Grumble, grumble, mutter, mutter... :) )
I agree that having a buildbot of your own to administer tends to encourage you to be more aware of issues - it certainly did for me. However, from my own experience, the Windows buildbot environment is fairly flaky, and I spend far too much time killing "stuck" python processes and VS JIT debugger processes, rather than actually usefully debugging real issues. (And I don't know of any way of finding out where the stuck processes came from or what they were doing at the time that they got stuck, so all I can do is kill them...) I don't have any Windows sysadmin experience, so I struggle to fix these types of problem, and doing so often takes up all the time that would otherwise go to areas where I *could* do some good, digging into genuine issues :-( I don't really have any answer to this problem right now. Is it possible to set up a local buildslave-like environment (so I can run the test suite on my development PC without needing to set up access and register that PC as a temporary buildslave, which wouldn't be practical for me)? If I can get an environment where I can run the tests as I need and debug them in place, I might be able to work on improving the reliability of things. Paul.
On Wed, Aug 4, 2010 at 8:25 AM, Paul Moore <p.f.moore@gmail.com> wrote:
However, from my own experience, the Windows buildbot environment is fairly flaky, and I spend far too much time killing "stuck" python processes and VS JIT debugger processes, rather than actually usefully debugging real issues.
I would turn off JIT debugging. On an unattended machine, it's more annoying than useful. http://msdn.microsoft.com/en-us/library/5hs4b7a6(VS.80).aspx -- Curt Hagenlocher curt@hagenlocher.org
I don't really have any answer to this problem right now. Is it possible to set up a local buildslave-like environment (so I can run the test suite on my development PC without needing to set up access and register that PC as a temporary buildslave, which wouldn't be practical for me)? If I can get an environment where I can run the tests as I need and debug them in place, I might be able to work on improving the reliability of things.
Not sure what you are asking. It's certainly possible to run the test suite yourself: just invoke "python Lib\test\regrtest.py" or some such. So I assume you are asking for something else. Regards, Martin
On 4 August 2010 18:42, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I don't really have any answer to this problem right now. Is it possible to set up a local buildslave-like environment (so I can run the test suite on my development PC without needing to set up access and register that PC as a temporary buildslave, which wouldn't be practical for me)? If I can get an environment where I can run the tests as I need and debug them in place, I might be able to work on improving the reliability of things.
Not sure what you are asking. It's certainly possible to run the test suite yourself: just invoke "python Lib\test\regrtest.py" or some such.
So I assume you are asking for something else.
Sorry, I wasn't clear. The issues I see tend to look like they come from the test suite being run under the control of the buildbot service - issues caused by not having a terminal session or window station (like the JIT debugger appearing, which Curt pointed out a way of addressing) or issues relating to my seeing orphaned python_d.exe processes which never occur for me when running the test suite manually. I'm guessing that these issues aren't "simple" test failures, but somehow triggered by the way in which the environment buildbot provides differs from a "normal" console session, and so I was looking for a way to reproduce those conditions. Paul.
Am 04.08.2010 21:03, schrieb Paul Moore:
On 4 August 2010 18:42, "Martin v. Löwis" <martin@v.loewis.de> wrote:
I don't really have any answer to this problem right now. Is it possible to set up a local buildslave-like environment (so I can run the test suite on my development PC without needing to set up access and register that PC as a temporary buildslave, which wouldn't be practical for me)? If I can get an environment where I can run the tests as I need and debug them in place, I might be able to work on improving the reliability of things.
Sorry, I wasn't clear. The issues I see tend to look like they come from the test suite being run under the control of the buildbot service - issues caused by not having a terminal session or window station (like the JIT debugger appearing, which Curt pointed out a way of addressing) or issues relating to my seeing orphaned python_d.exe processes which never occur for me when running the test suite manually.
Ah. It should certainly be possible to set this up locally - you just need to run a buildbot master as well, either on the same machine or a different one. The only thing you can't then get is automatic notifications on commits. You could emulate this with manually triggering builds over the master web pages, or by using buildbot's svnpoller change source (which defaults to a poll interval of 10min). To get started, I could share with you the master configuration of python.org (with account information stripped, of course). Regards, Martin
On 4 August 2010 20:17, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Ah. It should certainly be possible to set this up locally - you just need to run a buildbot master as well, either on the same machine or a different one. The only thing you can't then get is automatic notifications on commits. You could emulate this with manually triggering builds over the master web pages, or by using buildbot's svnpoller change source (which defaults to a poll interval of 10min).
To get started, I could share with you the master configuration of python.org (with account information stripped, of course).
Thanks, I'll have a look at what I need to set up when I next get some time (which may be a few weeks, as I'm on holiday soon :-)). When I get to a suitable point, I'll get back to you. It may be that setting up a full master/slave setup is more effort than it's going to be worth to me (after all, I have a buildslave to work on in any case...) - I was thinking more of something a little simpler, just a means of running the tests manually in a service-type environment. If the buildmaster/slave setup is more than I feel like setting up, I'll look into that as an alternative. Thanks for the suggestion. Paul.
On Tue, 03 Aug 2010 21:08:31 -0400 Steve Holden <steve@holdenweb.com> wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another. From time to time, a couple of "motivated" core developers (David, Ezio, Mark, Florent...) try to fix the situation by diagnosing all pending buildbot issues. It's a time-consuming task (because it's harder to diagnose problems when you aren't the one who introduced them), and it's somehow thankless and unmotivating (you'd rather work on your own issues rather than fix bugs introduced by others). Windows only exacerbates the problem because the aforementioned "motivated" core developers aren't necessary knowledgeable enough to write a fix, or even understand the problem. I would advocate a system were people are encouraged to take responsibility of the problems they introduce when committing changes. Of course, there are sometimes situations where it's not possible (triggering platform-specific oddities, for example). Regards Antoine.
On Wed, Aug 4, 2010 at 11:16, Antoine Pitrou <solipsis@pitrou.net> wrote:
I would advocate a system were people are encouraged to take responsibility of the problems they introduce when committing changes. Of course, there are sometimes situations where it's not possible (triggering platform-specific oddities, for example).
They might not be able to diagnose the problems, or fix them, but I think they should still take responsibility until handed over explicitly to someone else who knows the failing platform, right? It should be relatively straightforward to have some process send email to authors of checkins that have created new failures. Cheers, Dirkjan
On Wed, Aug 4, 2010 at 7:42 PM, Dirkjan Ochtman <dirkjan@ochtman.nl> wrote:
On Wed, Aug 4, 2010 at 11:16, Antoine Pitrou <solipsis@pitrou.net> wrote:
I would advocate a system were people are encouraged to take responsibility of the problems they introduce when committing changes. Of course, there are sometimes situations where it's not possible (triggering platform-specific oddities, for example).
They might not be able to diagnose the problems, or fix them, but I think they should still take responsibility until handed over explicitly to someone else who knows the failing platform, right?
It should be relatively straightforward to have some process send email to authors of checkins that have created new failures.
I believe that was the original intent, but the buildbot set generated too many false alarms (due to flaky tests which would fail intermittently) so it was never implemented. However, I believe the 3.x tests have had most of that flakiness eliminated (dropping bsddb helped greatly on that front), so it should be feasible to restore that feature for the stable buildbot set on the Py3k branch. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 8/4/2010 6:08 AM, Nick Coghlan wrote:
On Wed, Aug 4, 2010 at 7:42 PM, Dirkjan Ochtman <dirkjan@ochtman.nl> wrote:
On Wed, Aug 4, 2010 at 11:16, Antoine Pitrou <solipsis@pitrou.net> wrote:
I would advocate a system were people are encouraged to take responsibility of the problems they introduce when committing changes. Of course, there are sometimes situations where it's not possible (triggering platform-specific oddities, for example).
They might not be able to diagnose the problems, or fix them, but I think they should still take responsibility until handed over explicitly to someone else who knows the failing platform, right?
It should be relatively straightforward to have some process send email to authors of checkins that have created new failures.
I believe that was the original intent, but the buildbot set generated too many false alarms (due to flaky tests which would fail intermittently) so it was never implemented.
However, I believe the 3.x tests have had most of that flakiness eliminated (dropping bsddb helped greatly on that front), so it should be feasible to restore that feature for the stable buildbot set on the Py3k branch.
I think that would be good, and an attractive extra benefit of the work that's gone into the tests recently. The point of having the buildbots was to alert people to when one of their changes breaks a build they weren't able to test on. Indeed I seem to remember at one time I used to see "blame" emails, which were not always accurate (I haven't been subscribed to checkins for a while due to volume of other mail). That's a heavy load to carry for a new developer, so having defined people to go to for help on strange platforms would be useful. Assuming, of course, that such people can be identified. I'm well aware that developers have limits on the time they can spend on Python. There's no blame here, just a wish to improve the process and ensure that the Windows platform doesn't become the "poor relation". Thanks to everyone for the positive responses. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work. -Barry
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page? Jean-Paul
On 04/08/2010 16:15, exarkun@twistedmatrix.com wrote:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
The hard part is remembering. Michael
Jean-Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 03:17 pm, fuzzyman@voidspace.org.uk wrote:
On 04/08/2010 16:15, exarkun@twistedmatrix.com wrote:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
The hard part is remembering.
Seems to be setting the bar awfully low for Python developers. Jean-Paul
On Aug 04, 2010, at 03:15 PM, exarkun@twistedmatrix.com wrote:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers.
These post-commit verification steps should be proactive, and scream really >loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and >changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
That's not the right question :) The real questions are: how hard is it to remember how to find the appropriate web page, how hard is it to know which buildbots are *actually* stable enough to rely on, how hard is it to decipher the results to know what they're telling you? -Barry
On 03:31 pm, barry@python.org wrote:
On Aug 04, 2010, at 03:15 PM, exarkun@twistedmatrix.com wrote:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers.
These post-commit verification steps should be proactive, and scream really >loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and >changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
That's not the right question :)
The real questions are: how hard is it to remember how to find the appropriate web page
Oh, come on. I don't believe this.
how hard is it to know which buildbots are *actually* stable enough to rely on,
This is more plausible. But it's not the tools' fault that the test suite has intermittent failures. Developers choose to add new features or change existing ones instead of fixing bugs in existing code or tests. I'd call that a developer failure.
how hard is it to decipher the results to know what they're telling you?
Red bad, green good. A much more plausible explanation, to me, is that most developers don't really care if things are completely working most of the time. They're happy to push the work onto other developers and onto the release team. And as long as other developers let them get away with that, it's not likely to stop. But perhaps the people picking up the slack here don't mind and are happy to keep doing it, in which case nothing needs to change. Jean-Paul
On 8/4/2010 12:42 PM, exarkun@twistedmatrix.com wrote:
On 03:31 pm, barry@python.org wrote:
On Aug 04, 2010, at 03:15 PM, exarkun@twistedmatrix.com wrote:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers.
These post-commit verification steps should be proactive, and scream really >loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and >changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
That's not the right question :)
The real questions are: how hard is it to remember how to find the appropriate web page
Oh, come on. I don't believe this.
how hard is it to know which buildbots are *actually* stable enough to rely on,
This is more plausible. But it's not the tools' fault that the test suite has intermittent failures. Developers choose to add new features or change existing ones instead of fixing bugs in existing code or tests. I'd call that a developer failure.
how hard is it to decipher the results to know what they're telling you?
Red bad, green good.
A much more plausible explanation, to me, is that most developers don't really care if things are completely working most of the time. They're happy to push the work onto other developers and onto the release team. And as long as other developers let them get away with that, it's not likely to stop.
But perhaps the people picking up the slack here don't mind and are happy to keep doing it, in which case nothing needs to change.
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas. Otherwise a release manager could proceed by reading a web page an even, heaven help us, following specific links to ensure particular actions were taken. But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim. This doesn't give people any confidence that the rules actually mean much, and I think ignoring the latter rule can negatively affect quality. Establishing comprehensive procedures can be as difficult as programming, though, and requires skills that have eluded me through a fairly lengthy technical career. So it also boils down to shortage of manpower of a particular kind. People with programming skill would, understandably, rather invest their time in something they are good at. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
Wiadomość napisana przez Steve Holden w dniu 2010-08-04, o godz. 19:56:
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim. This doesn't give people any confidence that the rules actually mean much
Very good point. -- Best regards, Łukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/
On Wed, 04 Aug 2010 13:56:27 -0400 Steve Holden <steve@holdenweb.com> wrote:
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas. Otherwise a release manager could proceed by reading a web page an even, heaven help us, following specific links to ensure particular actions were taken.
This isn't about release management. If we care about buildbots only the few days before a release, we are doomed.
Establishing comprehensive procedures can be as difficult as programming, though, and requires skills that have eluded me through a fairly lengthy technical career. So it also boils down to shortage of manpower of a particular kind.
The problem is more of gathering support for a particular policy or procedure, giving that it isn't necessarily part of the established culture. There are certainly some of us who do care about buildbot stability and general test suite quality (a few months ago, we had a majority of green buildbots, quite consistently; but keeping those buildbots green in the face of daily SVN activity is an uphill battle if we don't get support from the majority of committers). Regards Antoine.
On Aug 04, 2010, at 01:56 PM, Steve Holden wrote:
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim. This doesn't give people any confidence that the rules actually mean much, and I think ignoring the latter rule can negatively affect quality.
I don't believe we've ever had a rule (as embodied in PEP 101) that a release requires all buildbots to be green. I think that would be unachievable due in large part to the buildbot infrastructure itself not being very stable. We have identified some buildbots as "stable" ones and PEP 101 says: ___ Check the stable buildbots. Go to http://www.python.org/dev/buildbot/stable/ (the trailing slash is required). Look at the buildbots for the release you're making. Ignore any that are offline (or inform the community so they can be restarted). If what remains are (mostly) green buildbots, you're good to go. If you have non-offline red buildbots, you may want to hold up the release until they are fixed. Review the problems and use your judgement, taking into account whether you are making an alpha, beta, or final release. Even having non-offline stable buildbots completely green for a release would take work we don't have the manpower for. When I'm doing a release, I certainly consult both the stable and unstable buildbots, but I always run the full test suite on local platforms I have access too, which covers Linux, OS X, and hopefully soon Windows. #python-dev is helpful here for providing some additional sanity checks. -Barry
Am 04.08.2010 20:25, schrieb Barry Warsaw:
On Aug 04, 2010, at 01:56 PM, Steve Holden wrote:
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim. This doesn't give people any confidence that the rules actually mean much, and I think ignoring the latter rule can negatively affect quality.
I don't believe we've ever had a rule (as embodied in PEP 101) that a release requires all buildbots to be green. I think that would be unachievable due in large part to the buildbot infrastructure itself not being very stable. We have identified some buildbots as "stable" ones and PEP 101 says:
...
Even having non-offline stable buildbots completely green for a release would take work we don't have the manpower for. When I'm doing a release, I certainly consult both the stable and unstable buildbots, but I always run the full test suite on local platforms I have access too, which covers Linux, OS X, and hopefully soon Windows. #python-dev is helpful here for providing some additional sanity checks.
Same here. For 3.2a1, I looked at the stable buildbots -- unfortunately, three out of six were offline. Of the other three, I fixed a few real bugs, and in the end two of them were green. The last one, a Windows 7 machine, had some obscure failures building OpenSSL, about which I consulted Martin and David, the owner, and some real issues with the new OpenSSL version were fixed, but in the end it was still red. So, I think in the end having the buildbots was a success, even if only 2/6 were green :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas.
No. It rather makes clear that people who never had the role of release manager
Otherwise a release manager could proceed by reading a web page an even, heaven help us, following specific links to ensure particular actions were taken.
When Barry had the keys to the time machine, he wrote PEP 101, to give you a web page with specific links in it: http://www.python.org/dev/peps/pep-0101/
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim.
What makes you say that?
Establishing comprehensive procedures can be as difficult as programming, though, and requires skills that have eluded me through a fairly lengthy technical career. So it also boils down to shortage of manpower of a particular kind. People with programming skill would, understandably, rather invest their time in something they are good at.
I think you are belittling the contributions of past and present release managers. Regards, Martin
On 8/4/2010 2:57 PM, "Martin v. Löwis" wrote:
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas.
No. It rather makes clear that people who never had the role of release manager
Otherwise a release manager could proceed by reading a web page an even, heaven help us, following specific links to ensure particular actions were taken.
When Barry had the keys to the time machine, he wrote PEP 101, to give you a web page with specific links in it:
http://www.python.org/dev/peps/pep-0101/
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim.
What makes you say that?
Establishing comprehensive procedures can be as difficult as programming, though, and requires skills that have eluded me through a fairly lengthy technical career. So it also boils down to shortage of manpower of a particular kind. People with programming skill would, understandably, rather invest their time in something they are good at.
I think you are belittling the contributions of past and present release managers.
I'd prefer to say I was displaying my ignorance. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
Am 04.08.2010 19:56, schrieb Steve Holden:
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas.
If you mean to imply that a release manager should care for the stability of "their" branch also in between of releases -- I'd love to do that, but I'd need a 36-hour day then. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On 8/4/2010 6:11 PM, Georg Brandl wrote:
Am 04.08.2010 19:56, schrieb Steve Holden:
This whole discussion seems to make it clear that the release manager procedures are still ill-defined in certain areas.
If you mean to imply that a release manager should care for the stability of "their" branch also in between of releases -- I'd love to do that, but I'd need a 36-hour day then.
I'll see if I can get God to extend it for you. I honestly do understand that everyone else works under the same restrictions of time that I do ... and I appreciate all the hard work the release managers continue to put in. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Aug 04, 2010, at 06:39 PM, Steve Holden wrote:
I'll see if I can get God to extend it for you.
No need to involve the supernatural Steve! Just approve that PSF grant I submitted so I can finish my (Python powered of course!) clone army.
I honestly do understand that everyone else works under the same restrictions of time that I do ... and I appreciate all the hard work the release managers continue to put in.
We know you love us Steve. :) -Barry
Am 05.08.2010 01:26, schrieb Barry Warsaw:
On Aug 04, 2010, at 06:39 PM, Steve Holden wrote:
I'll see if I can get God to extend it for you.
No need to involve the supernatural Steve! Just approve that PSF grant I submitted so I can finish my (Python powered of course!) clone army.
*Now* I know why the PSU abducted all these developers! To assemble a sufficient gene pool...
I honestly do understand that everyone else works under the same restrictions of time that I do ... and I appreciate all the hard work the release managers continue to put in.
We know you love us Steve. :)
In a PHBy way, it seems :) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On 8/5/2010 2:19 AM, Georg Brandl wrote:
Am 05.08.2010 01:26, schrieb Barry Warsaw:
On Aug 04, 2010, at 06:39 PM, Steve Holden wrote:
I'll see if I can get God to extend it for you.
No need to involve the supernatural Steve! Just approve that PSF grant I submitted so I can finish my (Python powered of course!) clone army.
*Now* I know why the PSU abducted all these developers! To assemble a sufficient gene pool...
I honestly do understand that everyone else works under the same restrictions of time that I do ... and I appreciate all the hard work the release managers continue to put in.
We know you love us Steve. :)
In a PHBy way, it seems :)
I'd have thought a pre-requisite for being a PHB was having hair. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Thu, Aug 05, 2010 at 07:49:31AM -0400, Steve Holden wrote:
I'd have thought a pre-requisite for being a PHB was having hair.
Not at all, not at all - being a PHB is a style of thinking, not hairdressing. ;) Oleg. -- Oleg Broytman http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
On 8/5/2010 8:12 AM, Oleg Broytman wrote:
On Thu, Aug 05, 2010 at 07:49:31AM -0400, Steve Holden wrote:
I'd have thought a pre-requisite for being a PHB was having hair.
Not at all, not at all - being a PHB is a style of thinking, not hairdressing. ;)
I was simply trying to avoid becoming "The Python PHB" in the folklore. Sadly I see it is now inevitable - thanks, Georg ;-) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On Wed, Aug 04, 2010 at 07:26:06PM -0400, Barry Warsaw wrote:
On Aug 04, 2010, at 06:39 PM, Steve Holden wrote:
I'll see if I can get God to extend it for you.
No need to involve the supernatural Steve! Just approve that PSF grant I submitted so I can finish my (Python powered of course!) clone army.
Someone else got there ahead of you... http://en.wikipedia.org/wiki/MASSIVE_%28software%29
If you mean to imply that a release manager should care for the stability of "their" branch also in between of releases -- I'd love to do that, but I'd need a 36-hour day then.
I'll see if I can get God to extend it for you. I honestly do understand that everyone else works under the same restrictions of time that I do ... and I appreciate all the hard work the release managers continue to put in.
And indeed, from an RM point of view, fixing the bugs that cause buildbot breakage isn't the right action for the RM. Instead, if the release procedures say that the buildbots must pass the test, and the tests don't actually pass, the natural consequence is not to release. Barry has often exercised that approach in the past, and in many cases, it actually helped in getting people motivated to fix bugs when he threatened not to release. Regards, Martin
Steve Holden writes:
But I see rules being established ("there's a language moratorium: no changes!", "no release should be made unless the buildbots are *all* green") and then ignored apparently on a whim. This doesn't give people any confidence that the rules actually mean much, and I think ignoring the latter rule can negatively affect quality.
I don't see this. In the first case, you've misstated the rule: it's "no changes to the Python language", and what is and is not part of the language is subject to a certain amount of interpretation. There are several PEPs waiting on the moratorium despite everybody loving them, and the decisions on borderline changes (which have gone both ways, mostly denials) are establishing precedents that narrow the scope for "interpretation". I think it's very reasonable to assess the moratorium as *very successful* with respect to it being a rule that is obeyed in spirit and according to the letter. In the second case, I don't recall it being stated as a project rule. The buildbots were considered untrustworthy by many from the get-go, and I do recall discussion of the "community buildbots" which effectively resulted in community 'bots being fully deprecated. Some RMs have nevertheless chosen to take them very seriously and want them fixed if they're broken, others consider them a useful indicator but are willing to proceed if there are strong indications that the 'bot is broken rather than CPython. That's something that can be left up to the release manager or not, as the project chooses, but my impression to date has been that this is a matter of RM policy, not project policy. Note that following the latter rule can also negatively affect quality, if scarce developer effort is devoted to fixing somebody else's software rather than working on Python. FWIW my assessment is that for the moment all of the RMs take the buildbots pretty seriously, which is good (that seems to be consensus opinion), with some variations in intensity, which is also (IMO YMMV) good. No need for change here yet (IMO YMMV), although community members (anybody who cares) should prod RMs who seem to be neglecting buildbot results. In another cycle or so, the bots will probably be ready for a project-wide rule. I agree with J-P's suggestion that the place to start is asking developers to bookmark the bot pages relevant to them, and visit it (with appropriate lag) after committing. For one thing, if people see the 'bots deprecating their perfectly good changes, they'll have some incentive to work on the bots and beat them into shape. That can help take some load off the people who have concentrated on the bots. It will also mean that a decision to condition releases on green bots will be taken based on much broader experience rather than hearsay about their reliability.
Am 04.08.2010 17:15, schrieb exarkun@twistedmatrix.com:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete. So if you look too soon, you won't see all the results, and usually the slow systems are the interesting ones. Now we could of course have a commit hook that counts down two hours and then sends an email to the committer "Now look at the buildbot!"... Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On Wed, Aug 4, 2010 at 11:53 AM, Georg Brandl <g.brandl@gmx.net> wrote:
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete. So if you look too soon, you won't see all the results, and usually the slow systems are the interesting ones.
Now we could of course have a commit hook that counts down two hours and then sends an email to the committer "Now look at the buildbot!"...
Why not have buildbot slaves email the committer when their change broke the build. I realize that if you break 25 slaves it would not be pleasant to receive 25 emails, but I've had worse things happen. Or buildbot slaves can report failures to a mailing list or IRC chat. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek
Am 04.08.2010 18:21, schrieb David Stanek:
On Wed, Aug 4, 2010 at 11:53 AM, Georg Brandl <g.brandl@gmx.net> wrote:
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete. So if you look too soon, you won't see all the results, and usually the slow systems are the interesting ones.
Now we could of course have a commit hook that counts down two hours and then sends an email to the committer "Now look at the buildbot!"...
Why not have buildbot slaves email the committer when their change broke the build. I realize that if you break 25 slaves it would not be pleasant to receive 25 emails, but I've had worse things happen. Or buildbot slaves can report failures to a mailing list or IRC chat.
This is what we did once (or the mails were sent to python-dev, I don't remember), but it became ugly fast whenever the buildbots became unstable. Also, some of the buildbot slave owners do not have endless spare time to look at issues on their side. Once we manage to get them into a mostly stable situation again, we should definitely enable notifications in on form or other again. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On 03:53 pm, g.brandl@gmx.net wrote:
Am 04.08.2010 17:15, schrieb exarkun@twistedmatrix.com:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete. So if you look too soon, you won't see all the results, and usually the slow systems are the interesting ones.
Now we could of course have a commit hook that counts down two hours and then sends an email to the committer "Now look at the buildbot!"...
I don't think it's that hard to take a look at the end of the day (or before starting anything else the next morning). All it really takes is a choice on the part of each developer to care whether or not their changes are correct. Jean-Paul
On Wed, 04 Aug 2010 16:45:37 -0000 exarkun@twistedmatrix.com wrote:
I don't think it's that hard to take a look at the end of the day (or before starting anything else the next morning). All it really takes is a choice on the part of each developer to care whether or not their changes are correct.
And as reported by Ezio, using bbreport makes it extra easy: $ bbreport.py -q 3.x Selected builders: 20 / 80 (branch: 3.x) 3.x.dmg 83717, 83666, 528, 403 ARMv4 Debian 3.x *** , 83681, 649, 636 # hung for 30 min ARMv7Thumb Ubuntu 3.x 83025, 82997, 985, 985 # failed slave lost [etc.] http://code.google.com/p/bbreport/ hg clone https://bbreport.googlecode.com/hg/ bbreport Regards Antoine.
On Aug 04, 2010, at 06:58 PM, Antoine Pitrou wrote:
On Wed, 04 Aug 2010 16:45:37 -0000 exarkun@twistedmatrix.com wrote:
I don't think it's that hard to take a look at the end of the day (or before starting anything else the next morning). All it really takes is a choice on the part of each developer to care whether or not their changes are correct.
And as reported by Ezio, using bbreport makes it extra easy:
$ bbreport.py -q 3.x Selected builders: 20 / 80 (branch: 3.x) 3.x.dmg 83717, 83666, 528, 403 ARMv4 Debian 3.x *** , 83681, 649, 636 # hung for 30 min ARMv7Thumb Ubuntu 3.x 83025, 82997, 985, 985 # failed slave lost [etc.]
http://code.google.com/p/bbreport/ hg clone https://bbreport.googlecode.com/hg/ bbreport
It would be kind of nice to integrate this with release.py. -Barry
Am 04.08.2010 18:45, schrieb exarkun@twistedmatrix.com:
How hard is it to look at a web page?
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete. So if you look too soon, you won't see all the results, and usually the slow systems are the interesting ones.
Now we could of course have a commit hook that counts down two hours and then sends an email to the committer "Now look at the buildbot!"...
I don't think it's that hard to take a look at the end of the day (or before starting anything else the next morning). All it really takes is a choice on the part of each developer to care whether or not their changes are correct.
That's true. However, when most of the time you're looking at the buildbots you have to check 10 different red ones, only to realize none of the failures is related to any recent commit, this gets tiresome fast. I know that the remedy is to get the buildbots stable, but hey, somebody has to do that. It's probably not you, and it's probably not me. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
Georg Brandl <g.brandl@gmx.net> wrote:
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete.
Based on this and other issues, I don't think it's practical to expect that people will always check that their commits pass tests on all platforms. Perhaps a slight change in culture is necessary. When someone finds a problem with a buildbot, the first response is should be to request that the committer attempt to fix the problem. Finding the guilty party can be easier said than done since there could be multiple problems or if the problem has been present for a long time. Another complication is that the commiter may not have access to the plaform and require it for debugging purposes. Basically, I think better communication could mostly solve the problem. If someone finds a test failing on a certain plaform, it's okay to complain about it. You don't have to fix it yourself. Perhaps the simpliest approach would be to open a new issue. Neil
Am 04.08.2010 17:53, schrieb Georg Brandl:
Am 04.08.2010 17:15, schrieb exarkun@twistedmatrix.com:
On 02:51 pm, barry@python.org wrote:
On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
I think the issue is that many core developers don't have the reflex to check buildbot state after they commit some changes (or at least on a regular, say weekly, basis), and so gradually the buildbots have a tendency to turn from green to red, one after another.
I'd classify this as a failure of the tools, not of the developers. These post-commit verification steps should be proactive, and scream really loud (or even prevent future commits) until everything is green again. Buildbots themselves can be unstable, so this may or may not be workable, and changing any of this will take valuable volunteer time. It's also unsexy work.
How hard is it to look at a web page?
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete.
That should be "1-2 hours", by the way... Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
On Wed, 04 Aug 2010 23:53:22 +0200 Georg Brandl <g.brandl@gmx.net> wrote:
The hard part is to know *when* to look. As you might have noticed, the Python test suite does not run in ten seconds, especially on some of the buildbots -- it can take 1-2 there to complete.
That should be "1-2 hours", by the way...
But if the buildbot is late, it can sometimes be 1-2 days. Antoine.
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl). If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken? -Barry
On Wed, Aug 4, 2010 at 09:48, Barry Warsaw <barry@python.org> wrote:
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
It's a little disappointing to discover that despite the relatively large number of developers who have received MSDN licenses from Microsoft, none if us have the time to make sure that the buildbots are green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
-Barry
I can expand the dev setup guide I wrote for the PSF Sprints to include the third-party stuff, then try to get that in wider circulation. I currently gloss over it to get a first-time contributor up and running quickly (since someone's first look into core dev is unlikely to be fixing sqlite). I haven't tried the current codebase on VS2010 yet, but it's on my todo list.
On 8/4/2010 11:00 AM, Brian Curtin wrote:
On Wed, Aug 4, 2010 at 09:48, Barry Warsaw <barry@python.org <mailto:barry@python.org>> wrote:
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
>It's a little disappointing to discover that despite the relatively >large number of developers who have received MSDN licenses from >Microsoft, none if us have the time to make sure that the buildbots are >green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
-Barry
I can expand the dev setup guide I wrote for the PSF Sprints to include the third-party stuff, then try to get that in wider circulation. I currently gloss over it to get a first-time contributor up and running quickly (since someone's first look into core dev is unlikely to be fixing sqlite).
I haven't tried the current codebase on VS2010 yet, but it's on my todo list.
I think that could be incredibly useful. I've tried building Windows Pythons in the past and stalled because I didn't have enough familiarity with the VS environment. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On 04/08/2010 16:38, Steve Holden wrote:
On 8/4/2010 11:00 AM, Brian Curtin wrote:
On Wed, Aug 4, 2010 at 09:48, Barry Warsaw<barry@python.org <mailto:barry@python.org>> wrote:
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
>It's a little disappointing to discover that despite the relatively >large number of developers who have received MSDN licenses from >Microsoft, none if us have the time to make sure that the buildbots are >green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
-Barry
I can expand the dev setup guide I wrote for the PSF Sprints to include the third-party stuff, then try to get that in wider circulation. I currently gloss over it to get a first-time contributor up and running quickly (since someone's first look into core dev is unlikely to be fixing sqlite).
I haven't tried the current codebase on VS2010 yet, but it's on my todo list.
I think that could be incredibly useful. I've tried building Windows Pythons in the past and stalled because I didn't have enough familiarity with the VS environment.
Brian: could you remind us where that doc is, please? I've lost track of it. In broad terms it's not too hard to get going once you've installed VS[*]; I've rebuild several different fresh Python checkouts several times today while these issues have been discussed, and generally it's a question of: cd py3k (or whatever) tools\buildbot\externals.bat cd py3k\pcbuild env build -d rt -d and you're done and tested. That said, I seem to be having build issues with ssl on 2.7 which I'll try to look into. But the technique is fairly straightforward. TJG [*] And a small clutter of other tools for certain bits
On Wed, Aug 4, 2010 at 10:49, Tim Golden <mail@timgolden.me.uk> wrote:
On 04/08/2010 16:38, Steve Holden wrote:
On 8/4/2010 11:00 AM, Brian Curtin wrote:
On Wed, Aug 4, 2010 at 09:48, Barry Warsaw<barry@python.org <mailto:barry@python.org>> wrote:
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
>It's a little disappointing to discover that despite the relatively >large number of developers who have received MSDN licenses from >Microsoft, none if us have the time to make sure that the buildbots are >green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
-Barry
I can expand the dev setup guide I wrote for the PSF Sprints to include the third-party stuff, then try to get that in wider circulation. I currently gloss over it to get a first-time contributor up and running quickly (since someone's first look into core dev is unlikely to be fixing sqlite).
I haven't tried the current codebase on VS2010 yet, but it's on my todo list.
I think that could be incredibly useful. I've tried building Windows Pythons in the past and stalled because I didn't have enough familiarity with the VS environment.
Brian: could you remind us where that doc is, please? I've lost track of it.
In broad terms it's not too hard to get going once you've installed VS[*]; I've rebuild several different fresh Python checkouts several times today while these issues have been discussed, and generally it's a question of:
cd py3k (or whatever) tools\buildbot\externals.bat cd py3k\pcbuild env build -d rt -d
and you're done and tested.
That said, I seem to be having build issues with ssl on 2.7 which I'll try to look into. But the technique is fairly straightforward.
TJG
[*] And a small clutter of other tools for certain bits
http://docs.pythonsprints.com/core_development/beginners.html Building ssl requires Perl and nasm, and gets completed as a post-build step. I haven't done that in a while but it's documented in PCBuild/readme.txt. That's the stuff I'll be adding to the above document.
http://docs.pythonsprints.com/core_development/beginners.html
Building ssl requires Perl and nasm, and gets completed as a post-build step. I haven't done that in a while but it's documented in PCBuild/readme.txt. That's the stuff I'll be adding to the above document.
Perl shouldn't be a requirement if you use the OpenSSL copy from svn.python.org. Regards, Martin
On 8/4/2010 11:00 AM, Brian Curtin wrote:
On Wed, Aug 4, 2010 at 09:48, Barry Warsaw <barry@python.org <mailto:barry@python.org>> wrote:
On Aug 03, 2010, at 09:08 PM, Steve Holden wrote:
>It's a little disappointing to discover that despite the relatively >large number of developers who have received MSDN licenses from >Microsoft, none if us have the time to make sure that the buildbots are >green for the 2.6.6 release.
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
-Barry
I can expand the dev setup guide I wrote for the PSF Sprints to include the third-party stuff, then try to get that in wider circulation. I currently gloss over it to get a first-time contributor up and running quickly (since someone's first look into core dev is unlikely to be fixing sqlite).
I haven't tried the current codebase on VS2010 yet, but it's on my todo list.
I think that could be incredibly useful. I've tried building Windows Pythons in the past and stalled because I didn't have enough familiarity with the VS environment. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
On 4 August 2010 15:48, Barry Warsaw <barry@python.org> wrote:
Should note that I did try to build Python using my MSDN license for Windows 7 and Visual Studio 2010. I only had an hour or so to attempt it, and did not succeed, though I think I got as far as trying to properly situate various dependent libraries (sqlite, ssl).
If anybody's successfully navigated the twisty maze, would you be able to write something up on the wiki to describe the steps you've taken?
I could probably reasonably easily set up a (VMWare/VirtualBox) VM image of a Windows 7 system with VS installed and all the infrastructure set up for Python development/testing, using my MSDN license (in fact, that's something I keep meaning to do for my own purposes, anyway). I don't know what issues there might be in making that available for others to use - on the one hand, it would need to be protected so that it couldn't be taken by just anyone, and it may be necessary to set something up to allow other users to re-register the software with their own license keys, but in principle that's just "a matter of programming"... Hosting might also be fun - I'm on an ADSL connection so uploading a (multi-)gigabyte image could be a challenge :-) If there's interest in this (and more particularly, if anyone could offer advice on what would be needed to be sure I do it legally!), let me know and I'll look into it. Paul.
participants (21)
-
"Martin v. Löwis" -
Antoine Pitrou -
Barry Warsaw -
Brian Curtin -
Curt Hagenlocher -
David Stanek -
Dirkjan Ochtman -
exarkun＠twistedmatrix.com -
Ezio Melotti -
Georg Brandl -
Jon Ribbens -
Mark Hammond -
Michael Foord -
Neil Schemenauer -
Nick Coghlan -
Oleg Broytman -
Paul Moore -
Stephen J. Turnbull -
Steve Holden -
Tim Golden -
Łukasz Langa