On 03:33 pm, guido@python.org wrote:

Too verbose, Glyph. :-)

Mea culpa. "Glyph" might be a less appropriate moniker than "Altogether too many glyphs".

It needs to be decided case-by-case.

If certain tests are to be ignored on a case-by-case basis, why not record that decision by disabling the test in the code? Otherwise, the decision inevitably gets boiled down to "it's okay to release the code with a bunch of tests failing, but I don't know which ones". As it was on Twisted when we used to make case-by-case decisions about failures, and as it is here now.

The beta is called beta because, well, it may break stuff and we may want to fix it.

That's also why the alpha is called an alpha. My informal understanding is that a beta should have no (or at least very few) known issues, with a medium level of confidence that no further ones will be found. An RC would have absolutely no known issues with a fairly high level of confidence that no further ones will be found. This would include the community buildbots basically working for the most part; I would not be surprised if there were a few tests that still had issues. But clearly the reality does not meet my informal expectations, so it would be nice to have something written down to check against. Still, I'm bringing this up now because it _is_ a beta, and I think it will be too late to talk about dealing with persistent test failures after the RCs start coming out. (Of course, I'm just being sneaky. I don't actually care if it's clearly documented, I just care that I stop having problems with incompatibility. But I believe having it clearly spelled out would actually prevent a lot of the problems that I've been having, since I don't think anyone would *intentionally* select a policy where every release breaks at least one major dependent project.)

I'm not particularly impressed by statistics like "all tests are red" -- this may all be caused by a single issue.

The issue, for me, is not specifically that tests are red. It's that there's no clear policy about what to do about that. Can a release go out with some of the tests being red? If so, what are the extenuating circumstances? Does this have to be fixed? If not, why not? Why are we talking about this now? Shouldn't the change which caused Django to become unimportable have been examined at the time, rather than months later? (In other words, if it *is* a single issue, great, it's easy to fix: revert that single issue.) If not, then shouldn't someone in Django-land have been notified so they could cope with the change? Sorry that there are so many questions here; if I had fewer, I'd use fewer words to ask them.

For example, I'd much rather read an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it?

I don't know. JP is already addressing the issues affecting Twisted in another thread (incompatible changes in the private-but-necessary-to- get-any-testing-done API of the warnings module). But I really think that whoever made the change which broke it should be the one investigating it, not me.

I do tend to ramble on, so here's an executive summary of my response: I want python developers to pay attention to the community buildbots and to treat breakages of existing projects as a serious issue. However, I don't think that maintaining those projects is the core team's job, so all I'm asking for is for core developers to: * treat breakages of 3rd party packages as a potentially serious issue, * if possible (i.e. if they find out about the breakage soon enough, which should be the case in any pybots failure) revert the change that caused the problem until the problem can be fixed, and * notify 3rd party maintainers when it's decided that the breakage will not be fixed. This only applies to breakages that the core developers find out about, which for all practical purposes means the ones on the community builders page. Those of you looking for point-by-point responses and some repetition of the above points, enjoy :). On 05:03 pm, guido@python.org wrote:

...

On 03:33 pm, guido@python.org wrote:

...

It needs to be decided case-by-case. (misunderstanding) No, I just meant that we need to figure out for each 3rd party test

On Thu, Jun 26, 2008 at 9:21 AM, wrote: that fails whether the failure is our fault (too incompatibile) or theirs (relied on undefined behavior) and what the best fix is (change our code or theirs -- note that even if it's there fault there are cases where the best fix is to change our code.

This is basically fine, as far as I'm concerned. I would like to suggest, however, that these issues be dealt with as soon as possible, rather than waiting for the release process to begin. A lot of decisions are made on this mailing list about the supposed properties of "average" python code, without any actual survey of said code. Sometimes the results of that survey can be really surprising. The end goal of any particular compatibility policy, of a distinction between "public" and "private" APIs, and so on, is to keep code working.

I'm sorry if your interpretation of the terminology is different, but this is mine and this is what we've always used, and it's not likely to change. (At least not for the 2.6/3.0 release.)

I have no problem with your definitions of these terms. I think that they should probably be in PEP 101 though. Would you accept a patch that added an edited / expanded version of this paragraph?

...

Still, I'm bringing this up now because it _is_ a beta,

Absolutely correct. The RCs are hoped to be as good as the final release. *Now* is the time to bring up issue.

Well, that's good, at least :)

But please bring up specific issues -- I don't want to have an extended discussion about process or quality or expectations. I just want the code to be fixed.

Well, one specific issue has been bumped in priority as a result of this thread, and others are under discussion. The code is getting fixed.

...

(I just care that I stop having problems with incompatibility.)

And here we seem to be parting our ways. We have a large amount of process already. I don't want more.

Looking at it from my perspective, I'm proposing a reduction in process. Under the current process, if a buildbot goes red, the developer makes a judgment call, the release manager makes a judgment call, there's discussion on a ticket, a ticket gets filed, it gets promoted, it gets demoted, the RM forgets to re-promote it... My suggestion is that the process be, simply: if a buildbot (community or otherwise) goes red, the change that broke it gets reverted. No questions asked! It's still there in the revision history, ready to be re-applied once the issues get worked out. Discussion can then take place and case-by-case judgments can be applied.

If you're talking about community buildbots (which I presume are testing 3rd party packages against the core release) being red, that's out of scope for the core developers.

I don't necessarily think that keeping the community buildbots red is the core developers' responsibility, but I don't think it should be entirely out of scope, either. The python test suite is, frankly, poor - and I hope I'm not surprising you by saying that. It's full of race conditions, tends to fail intermittently, and is frequently ignored. Not only that, but it is quite often changed, so tests for issues that affect real code are quite often removed. So, the community buildbots are not just making sure that 3rd-party code still works, they are an expanded, independently developed test suite to make sure that *python itself* still works. Sometimes they will not fill that role particularly well, but they are worth paying attention to. If python had a good, exhaustive regression test suite that was immutable between major versions, I'd probably feel differently. But that's not the world we live in. Right now, apparently, the *default* policy is that if the community buildbots go red, later, before a release, someone will maybe take a look at it. I'd suggest that the *default* policy ought to be that if a particular changeset breaks a community buildbot, it needs further examination before being merged to trunk. However, this is just the way I prefer to do development; if you think that would slow things down too much, the only thing I'm _really_ asking for is a clear statement that says "there should be no test failures on community buildbots that have not been explicitly accepted before a final release". I'm not even sure what "explicitly accepted" means - you have to sign off? the release manager, maybe? A discussion on this list? I don't really care, as long as _somebody_ does. Right now, my impression of the process is this: * The community buildbot goes red; no core developer looks at it. * If the project is Twisted, JP fixes the bug on Twisted's end. * If the project is Django, nobody notices. * Months later, a beta goes out. A few people try it out and report some bugs, but don't really understand the output. A good number go un- triaged. * A little while later, a final release comes out. Many projects are broken as a result. This is not a hypothetical concern. This is what happened with 2.5; Twisted was broken for months, and Zope *to this day* does not support Python 2.5. 2.6 looks like it's headed for the same trajectory. To be clear: this is with all of Python's _own_ tests passing, so it is specific to paying attention to community buildbots. (And the community buildbots only build django and twisted right now. I'm not talking about a massive pan-galactic survey of all possible python projects. I'm only talking about those popular enough to make this select list. Which should still be a slightly longer list, but I digress...)

Some of the core buildbots are red because, well, frankly, they run on a cranky old version of an OS that few people care about.

On Twisted, we have a distinction between "supported" and "unsupported" platforms, to provide the ability to run on platforms which aren't really supported and don't really run the whole suite, but we are nevertheless interested in. I don't believe the setup is too hard and we'll definitely help out with that if you want to do it. (I believe Thomas Herve volunteered to do this at PyCon...)

I hope the community buildbots can be used the same way: a red bot means someone needs to look into an issue. The issue could be with the core or with the 3rd party package being tested. I don't think a policy like "no community buildbots should be red" makes any sense.

These bots have been red for months. The issues exist, but have not been looked into. As a result, Barry made a specific commitment on a ticket (i.e. "this should block beta 1") which was not met. I think _something_ has to be changed to encourage people to do this more immediately or more seriously.

Whoever made what change? You can't seriously expect core developers investigating issues with 3rd party packages, no matter what the cause. The correct process is that someone who cares about the 3rd party package (could be an end user, a developer of that package, or a core developer who happens to care) looks into the issue enough to diagnose it, and then either proposes a fix or files a bug with the likely culprit, which could be the core or the 3rd party package. If nobody cares, well, that's open source too.

If the breakage is calculated and expected, and the benefits clearly oughtweigh the costs... oh well, too bad for the 3rd party people. It would be nice if the core developers would notify the third party if they find out about it so that it can be verified that the change in question wasn't obscuring some *other* problem, but from what I've seen, the breakages that I have been concerned about have not been intentional, calculated changes, but side-effects of other things. I'm talking about the case where the breakage reveals either a bug in Python, or an unintentional / side-effect change in behavior, which is surprisingly frequent.

glyph@divmod.com wrote:

I do tend to ramble on, so here's an executive summary of my response:

I want python developers to pay attention to the community buildbots and to treat breakages of existing projects as a serious issue.

Counter-proposal: * Interested developers or users of the major third party projects tested by the community buildbots should monitor the community buildbots, and start filing bug reports with the appropriate party as soon as the upcoming Python release hits 2.Xa1 (i.e. first alpha). * If the failure is due to an incompatibility between Python 2.X and 2.X-1, then the bug report should be filed against Python. While these issues may or may not be addressed before the first beta, they *must* be addressed before the first release candidate. * If the failure is due to an incompatibility between Python 2.X and 2.X-2 that was properly deprecated in 2.X-1, then the bug report should be filed against the third party project. Prioritising these is a question for the developers of that project. * Before filing a bug report against Python for a community buildbot failure, check if the relevant regression is also causing failures of the core buildbots. If it is, skip the bug report until the core buildbots are passing again. It's currently a fact of life that we do NOT keep the trunk in an always-releasable state. We just don't. It might be nice if we did, there are lots of reasons why that's a good way to run a project, but at this point in time it isn't the case with Python. Reacting every time a community buildbot goes red would be a serious waste of effort. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 05:14 pm, martin@v.loewis.de wrote:

...

I don't know. JP is already addressing the issues affecting Twisted in another thread (incompatible changes in the private-but-necessary-to- get-any-testing-done API of the warnings module). But I really think that whoever made the change which broke it should be the one investigating it, not me.

How could they have known that they broke it?

Because the relevant community buildbot turned red with that revision of trunk. Keep in mind I'm not talking about every piece of Python code in the universe; just the ones selected for the community buildbots. It would be nice if there were a dozen or so projects on there, but paying attention to the two builders that do actually run right now would be a good start.

On Thu, Jun 26, 2008 at 12:35 PM, wrote:

On 05:14 pm, martin@v.loewis.de wrote:

...

...

I don't know. JP is already addressing the issues affecting Twisted in another thread (incompatible changes in the private-but-necessary-to- get-any-testing-done API of the warnings module). But I really think that whoever made the change which broke it should be the one investigating it, not me.

How could they have known that they broke it?

Because the relevant community buildbot turned red with that revision of trunk. Keep in mind I'm not talking about every piece of Python code in the universe; just the ones selected for the community buildbots. It would be nice if there were a dozen or so projects on there, but paying attention to the two builders that do actually run right now would be a good start.

Sorry, this is an unacceptable burden on the core developers. The core developers aren't going to look at the community buildbots (unless voluntarily) and they aren't going to roll back changes just because some community buildbot goes red. In general someone outside the core developer group needs to bring the issue to the core developers' attention and then a fix will be created if appropriate. Rollbacks are generally reserved for accidental checkins or checkins against the process rules (e.g. during a code freeze). Heck, we don't even roll back if one of our own buildbots goes red. I'm fine with requesting that the core developers pay serious attention to reports about 3rd party code being broken. The community buildbots are a fine tool to find out about this. But any policy that requires an automatic rollback because a buildbot (community or core) goes red is unacceptable. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Jun 26, 2008 at 5:33 PM, wrote:

OK, fair enough. Taking a step back, I was pushing this really hard because to *me*, it seems like dealing with the influx of bug reports after the fact is an unreasonable amount of additional work, whereas immediate reverts are just an occasional, temporary inconvenience. Based on my experience, "We'll deal with the reports as they come in" sounds like "no".

No, that's not how it was intended. I'd rather hear you tell us about your pain than telling us how to fix it.

I think since the last time we had a similar conversation, other Twisted developers have been fairly diligent in reporting bugs. (In the time they've been reporting Python bugs, I've mostly been planning a wedding. Others who have survived it tell me that eventually, this process ends... and then I should be participating more directly.) I'll try to step up that feeback loop and get other projects involved. I hope I am wrong about generating an unreasonable amount of work.

Again, I'd much rather see you generate a huge pile of work for us in the form of specific bug reports than trying to tell us how to change our process.

Thanks. I appreciate it; an increased emphasis on 3rd party code being broken is really what I was looking for. This is fine with me. I mean, whatever your decision is, I'm going to have to live with it :), but in either case, we have to be looking for bugs and helping to investigate them. On my end of things I guess it's not going to make much difference.

Right. The beta period is the time to pay attention to 3rd party breakage. While we won't make any specific promises, we're taking 3rd party code breakage very seriously and, depending on the specifics, it can hold up a release. It just won't be automatic; many 3rd party breakages are most easily fixed by making small adjustments to the 3rd party code involved. (Imagine a 3rd party package still using string exceptions.) . . . On Thu, Jun 26, 2008 at 6:13 PM, wrote:

For the record, "automatic revert" does not mean "drop all other work". The changeset's still there. It's still in the revision history. It can easily be applied to anybody's working copy. It can easily be put into a branch. It can be automatically re-merged later. I do all of these things all the time, and I was *not* intending to suggest that any 3rd-party breakage should cause a code freeze or anything even remotely like that.

Still, a revert often holds up other work that depends on the reverted change. I expect that in most cases, finding the problem and applying a fix is much less of a burden for the core developer team as a whole than rolling back, working on a fix, and re-applying. I also expect that the policy of semi-automated merges from 2.6 into 3.0 would make a revert even more work, and more likely to cause second-order problem in the 3.0 branch.

...

...

(...) it would have been easier to convince a Twisted developer to do the work it to keep the community buildbot green rather than to make it a bit less red.

Why? That sounds like it's a problem with the psychology of the Twisted developers.

I hardly think it's unique to us. TDD test runners typically only know 2 colors and 2 states: "passed" and "fails". Once you're in the "fail" state, you tend to accumulate more failures; there's a much bigger bang for your buck. Better tools with nicer interfaces would let you easily mark individual tests as "usually intermittent" or something and make it a "slightly dirty green" or something, but we don't have those. The move from "red" to "green" is much more psychologically significant to just about anyone I know than the move from "red but 14 failures" to "red but 12 failures".

Still sounds like perhaps you're too much focused on the "green bar". I know this is encouraged by XP, but I don't want to have a religious debates about whether XP is the right development model for Python. (The time would be better spent on improving the buildbot reporting!)

Despite the idea's origins in a now-highly-controversial book on criminology, this is often referred to in XP discussion circles (wikis and the like) as the "fix broken windows" collaboration pattern. I notice this in lots of other areas of my life; a little pile of papers tends to become a big pile of papers, the first dent in a car makes the driver a little less careful, and so on.

For me, with that first dent comes the realization that it's really not worth it to obsess over scratches, and that makes me a more relaxed and hence *better* (because less stressed) driver. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Ralf Schmitt wrote:

TypeError: unhashable type: 'ManyToManyField' TypeError: unhashable type: 'PrimaryKeyConstraint'

and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html

Following the discussion to it's conclusions leads one to a tracker issue [1] that was supposed to be a blocker for the beta but (I assume) was forgotten about. Clearly this should be promoted to being a blocker again and some decision made. [1] http://bugs.python.org/issue2235 -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

And just to make my position perfectly clear, I've unassigned it, since I don't foresee to be able to give this issue the quality time it clearly needs. Mind you, I agree it's a release blocker. But I don't have time to personally investigate it. Sorry. On Thu, Jun 26, 2008 at 9:54 AM, wrote:

On 04:42 pm, schmir@gmail.com wrote:

...

On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum wrote:

...

an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it?

...

and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html

Following that trail of breadcrumbs, I ended up here:

http://bugs.python.org/issue2235

with a message from some guy named "Barry Warsaw" (anyone know him?) that says:

""" Guido, can you comment on Amaury's latest patch? I'm going to bump this back to critical so as not to hold up 2.6 alpha, but it should be marked as a release blocker for the first beta. """

I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt wrote:

this patch even make things worse for me. now py.test also dies.

Please add details to the tracker. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Jun 26, 2008 at 11:07 AM, Ralf Schmitt wrote:

On Thu, Jun 26, 2008 at 7:55 PM, Guido van Rossum wrote:

...

On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt wrote:

...

this patch even make things worse for me. now py.test also dies.

Please add details to the tracker.

Well, I think most probably the patch is correct and it just catches another class which defines __eq__ but does not define __hash__. So nothing to add there.

So you're saying py.test needs to be fixed? Fine with me, but then I don't understand why you bothered bringing it up here instead of fixing it (or reporting to the py.test maintainers, I don't know if you're one or not).

The question is if this behaviour should be kept or not. I guess there's no bug tracking that...

The best place is that very same issue. If this feature is really too annoying something needs to be done. Detailed reports about real code that gets broken without a good cause help making that case. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Jun 26, 2008 at 3:22 PM, Ralf Schmitt wrote:

On Thu, Jun 26, 2008 at 8:45 PM, Guido van Rossum wrote:

...

So you're saying py.test needs to be fixed? Fine with me, but then I don't understand why you bothered bringing it up here instead of fixing it (or reporting to the py.test maintainers, I don't know if you're one or not).

no, I'm not. Just wanted to point out, that this ticket/patch does not solve the django issue. (this is what I first thought it would do).

I'm still missing details. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jun 26, 2008, at 12:54 PM, glyph@divmod.com wrote:

On 04:42 pm, schmir@gmail.com wrote:

...

On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum wrote:

...

an explanation about *why* Django cannot even be imported than a blanket complaint that this is a disgrace. So why is it?

...

and already discussed: http://mail.python.org/pipermail/python-dev/2008-April/078421.html

Following that trail of breadcrumbs, I ended up here:

http://bugs.python.org/issue2235

with a message from some guy named "Barry Warsaw" (anyone know him?) that says:

""" Guido, can you comment on Amaury's latest patch? I'm going to bump this back to critical so as not to hold up 2.6 alpha, but it should be marked as a release blocker for the first beta. """

I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion.

Glyph, you did the right thing. I wish there was a way of marking a bug "not-release-blocker-for-now" and have it automatically update back to blocker at a certain time or after a certain event. I think it's valid for this issue to block beta2. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQF0XEjvBPtnXfVAQJwiQP/T9f33lNtAQ9/eooKEyVCDms7NXJE7mIf QPOQtXuTZdsfUl51OxkxewVj6ERZasHPwIB2c13HYuHRrMrmrE9EYStdbmMxh0BI 7EhsO4SfDI3ZnYFJvAEMrTvgY8Vz4v817LhzWNZ7RWxq/yOHG8C/ZgubPJIa8mnd EGWUVoLg77E= =OJHh -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jun 26, 2008, at 5:20 PM, Georg Brandl wrote:

Barry Warsaw schrieb:

...

...

I don't know if this "Barry" guy has the appropriate permissions on the bugtracker to increase priorities, so I've taken the liberty of upgrading it as a release blocker for the _second_ beta ... ;-). So, at least there's been one productive consequence of this discussion. Glyph, you did the right thing. I wish there was a way of marking a bug "not-release-blocker-for-now" and have it automatically update back to blocker at a certain time or after a certain event. I think it's valid for this issue to block beta2.

I just added a "deferred blocker" priority -- that implements one half of your wish. Mass issue updating must be done by someone who knows Roundup better than me, I'm afraid.

Cool, thanks! I should have thought of that before. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQNm3EjvBPtnXfVAQIy1wP/QPFwklmnay8VPPovHA6H4XA6tW+ziWPV hbeyAZX/MMxEYv/98Z4nOQU7M4AczlXHZks80UvDdKLwPe2wTwfOIgmCTeb+vciI 20GpxrHpWQNHy/XKeawEBxyLHJZ4qNGIAFmHwoiUusM6erTeVb9kWa0czG31En6r Z4MV0YKfu+A= =Zxh4 -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jun 26, 2008, at 5:54 PM, Martin v. Löwis wrote:

...

I just added a "deferred blocker" priority -- that implements one half of your wish. Mass issue updating must be done by someone who knows Roundup better than me, I'm afraid.

I doubt true mass update will be necessary. If you remind me that I should convert all "deferred blocker" issues to some other priority, I'll do that manually in a matter of minutes (using a query to find all issues with that priority).

Martin, I think that will work great. The idea being that after each release, we always bump deferred blockers back to release blocker. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSGQZbHEjvBPtnXfVAQIwWQQAqWoIoqV0S0O8ZpkXpWd3LO4Fo7MBfjRT LL5QgIPBWdQdZ4RR68LSfdAhiHGnZCZorpNksGaeIxP55qgS1z31fy3JF39UUbVR 3ojymtF6Is0qCB5lmkJIx9Na/2RUEOUWTQoQP3dG851oRRuSVTXb4uRaBlNlQErY oN9JxG5xQR0= =z9nN -----END PGP SIGNATURE-----

On 03:42 pm, ncoghlan@gmail.com wrote:

glyph@divmod.com wrote:

beta 1 has some trouble running *our* test suite - I'd be fairly surprised if the community buildbots were in significantly better shape.

That's another problem, yes :)

...

The community buildbots have been in a broken state for months now[1].

If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out.

I have mixed feelings about this: on the one hand, if this were a matter of public record, I would have known that this was too early to start complaining, and we could have saved everyone a lot of time reading these messages. That would be good; I would prefer to just let everyone work without bothering about process. On the other hand, it's much easier to deal with test failures as they arise than in one giant chunk of work at the end of the development process. I spoke to a few core developers at PyCon who thought that the buildbots should always be green and any change that broke them should be reverted. They may remain nameless unless they wish to raise their hands :-) but that's definitely how I feel about it.

Continuously running community buildbots on the maintenance trees makes sense, since those trees should always be in a releasable state. For the trunk, they're really only interesting when the Python core buildbots are reporting all green, but some of the community buildbots are reporting red.

(... which should be all the time ...)

One of the problems is what the term "beta" means to different groups - for us, this first beta was really about saying "zero new features from here on, focus on making what we have now work properly". The relatively late landing of a couple of major PEPs (371, 3108) also didn't do any favours for trunk stability.

A big part of why I wrote this message is that I wanted a clear understanding of the relationship between the definition of "alpha", "beta" and "RC" and the state of various buildbots. If that relationship exists already, just linking to it from http://python.org/download/releases/2.6/ would be good. By the way, that page still says these are "alpha" releases.

glyph@divmod.com schrieb:

On 03:42 pm, ncoghlan@gmail.com wrote:

...

glyph@divmod.com wrote:

...

beta 1 has some trouble running *our* test suite - I'd be fairly surprised if the community buildbots were in significantly better shape.

That's another problem, yes :)

...

...

The community buildbots have been in a broken state for months now[1].

...

If the community buildbots aren't largely green by the time beta 2 comes out, that's when I'll agree we have a problem - they should definitely be green by the time first release candidate comes out.

I have mixed feelings about this: on the one hand, if this were a matter of public record, I would have known that this was too early to start complaining, and we could have saved everyone a lot of time reading these messages. That would be good; I would prefer to just let everyone work without bothering about process.

The problem is that not enough people care about the community buildbots, and this isn't likely to improve since we have not enough manpower to do it.

On the other hand, it's much easier to deal with test failures as they arise than in one giant chunk of work at the end of the development process. I spoke to a few core developers at PyCon who thought that the buildbots should always be green and any change that broke them should be reverted. They may remain nameless unless they wish to raise their hands :-) but that's definitely how I feel about it.

I think no core developer will not tell you that the (core) buildbots should be green all the time. As for reverting changes that break, I'd support this only for changes that break *all* of them. For example, I only use one platform to develop on (and I guess it's the same for many others), having the buildbots go red on another platform means I can try to fix the issue. Georg

BuildBot has two ways to let you run your code on all builders before you commit it to trunk. You can force a build on a branch or you can try a build with a patch. I don't know if these options are enabled on Python's buildmaster. If they are, then if you want, you can use them to make sure your code works on all platforms before you put it into trunk, where it may cause problems for someone else.

Even if that's possible, it adds a burden. As likely not all people agree that such a procedure would be a good thing, it's necessary that somebody establishes a policy. Guido has already indicated that he is not interested in further policies; Barry (the canonical other person to set policies) has indicated in the past that he would like such a policy, but deems it unrealistic at this point. He gives other procedural changes higher priority, so that realistically, such a policy might not be established within the next two years. Regards, Martin

On 07:44 pm, g.brandl@gmx.net wrote:

At no time will a policy "the community buildbots must be green" be useful: the tests that run on these buildbots are not under our control, so if the tests test things we deem non-public we can't do anything about it. (And we may have a hard time convincing project authors to change the tests if we promised to make them run anyway.)

That's not what I'm suggesting. If there is a legitimate disagreement between Python developers and developers of a project about whether an API should continue to be supported, then clearly, the Python developers get to win. Welcome to open source. However, I believe that the core team is not paying attention to other projects breaking. I'm not saying that you want to make breaking changes, or that bug reports are not dealt with, but the problem is that dealing with these problems after the fact makes it _much_ more painful. When you get to the release, and you have 30 bug reports due to other projects breaking, they get triaged, some get left in, and some features of lots of different projects are left broken. And many projects do not bother to test with the beta. If developers were simply presented with the results of their changes immediately ("you broke twisted, django doesn't import, zope segfaults and mercurial destroys your repository") then perhaps they would weigh the benefits of compatibility differently, and do the trivial, obvious fix immediately, rather than waiting 6 months and having to figure out what the heck change caused the bug. I accept the fact that python core development is done differently than i.e. Java development, and some backward compatibility may simply be broken. Case in point: changes to the warnings module being discussed in a separate thread break a significant number of Twisted's tests, because they remove functionality which is not public but is required to test code that uses the warnings module. Would Brett have taken this into account if he knew about it immediately when revision 62303 was committed? Maybe not, but now that the code is written and done, there's significantly less motivation to go back and fix it. At the time it might have only been a few minutes' work to continue supporting the use-case which Twisted needs. Brett wouldn't even have necessarily needed to do the work: it would have been easier to convince a Twisted developer to do the work it to keep the community buildbot green rather than to make it a bit less red. Now, though, it's much easier to say "oh well, that's private, you lose". I don't ascribe this to malice - it really *would* be much harder to fix it now, for us as well as for him.

glyph@divmod.com schrieb:

On 07:44 pm, g.brandl@gmx.net wrote:

...

At no time will a policy "the community buildbots must be green" be useful: the tests that run on these buildbots are not under our control, so if the tests test things we deem non-public we can't do anything about it. (And we may have a hard time convincing project authors to change the tests if we promised to make them run anyway.)

That's not what I'm suggesting.

Sure, but I wanted to say that nevertheless :)

If there is a legitimate disagreement between Python developers and developers of a project about whether an API should continue to be supported, then clearly, the Python developers get to win. Welcome to open source.

However, I believe that the core team is not paying attention to other projects breaking.

Quite true. It is their duty to bring the breakage to our attention, if they want to see results.

I'm not saying that you want to make breaking changes, or that bug reports are not dealt with, but the problem is that dealing with these problems after the fact makes it _much_ more painful. When you get to the release, and you have 30 bug reports due to other projects breaking, they get triaged, some get left in, and some features of lots of different projects are left broken. And many projects do not bother to test with the beta.

I thought we're talking about the projects that *do* use the buildbots?

If developers were simply presented with the results of their changes immediately ("you broke twisted, django doesn't import, zope segfaults and mercurial destroys your repository") then perhaps they would weigh the benefits of compatibility differently, and do the trivial, obvious fix immediately, rather than waiting 6 months and having to figure out what the heck change caused the bug.

This sounds very nice in theory, but it must be implemented in a way that does not add a burden to the developer, as Martin said.

Case in point: changes to the warnings module being discussed in a separate thread break a significant number of Twisted's tests, because they remove functionality which is not public but is required to test code that uses the warnings module. Would Brett have taken this into account if he knew about it immediately when revision 62303 was committed? Maybe not, but now that the code is written and done, there's significantly less motivation to go back and fix it. At the time it might have only been a few minutes' work to continue supporting the use-case which Twisted needs. Brett wouldn't even have necessarily needed to do the work: it would have been easier to convince a Twisted developer to do the work it to keep the community buildbot green rather than to make it a bit less red. Now, though, it's much easier to say "oh well, that's private, you lose". I don't ascribe this to malice - it really *would* be much harder to fix it now, for us as well as for him.

I must say that I'm increasingly surprised by this discussion since it seems that not only the Python core devs neglect the community buildbots. Shouldn't the project authors have at least the same interest that their code runs on the next major Python version? And while they don't necessarily have more time to watch the buildbots than we have, the amount of what they need to keep an eye on. Georg

On 26 Jun, 09:24 pm, guido@python.org wrote:

On Thu, Jun 26, 2008 at 1:06 PM, wrote:

...

On 07:44 pm, g.brandl@gmx.net wrote:

Well, sorry, that's life. We're not going to deal with breakage in 3rd party code on a "drop all other work" basis.

For the record, "automatic revert" does not mean "drop all other work". The changeset's still there. It's still in the revision history. It can easily be applied to anybody's working copy. It can easily be put into a branch. It can be automatically re-merged later. I do all of these things all the time, and I was *not* intending to suggest that any 3rd-party breakage should cause a code freeze or anything even remotely like that.

...

Case in point: changes to the warnings module

I disagree. It's broken and should be fixed. Beta 1 just came out so this is the perfect time to file a bug.

I'll go back over the recent conversation and work out the specifics of the bug (if JP doesn't, or hasn't already beaten me to it).

...

(...) it would have been easier to convince a Twisted developer to do the work it to keep the community buildbot green rather than to make it a bit less red.

Why? That sounds like it's a problem with the psychology of the Twisted developers.

I hardly think it's unique to us. TDD test runners typically only know 2 colors and 2 states: "passed" and "fails". Once you're in the "fail" state, you tend to accumulate more failures; there's a much bigger bang for your buck. Better tools with nicer interfaces would let you easily mark individual tests as "usually intermittent" or something and make it a "slightly dirty green" or something, but we don't have those. The move from "red" to "green" is much more psychologically significant to just about anyone I know than the move from "red but 14 failures" to "red but 12 failures". Despite the idea's origins in a now-highly-controversial book on criminology, this is often referred to in XP discussion circles (wikis and the like) as the "fix broken windows" collaboration pattern. I notice this in lots of other areas of my life; a little pile of papers tends to become a big pile of papers, the first dent in a car makes the driver a little less careful, and so on.

Terry Reedy schrieb:

glyph@divmod.com wrote:

...

to what extent should Python actually be compatible between releases?

As I understand things from years of observation, the following are fair game to changed in ways possibly backward-incompatible for specific code: bugs, detailed float behavior (which may be system-specific anyway), error messages, private interfaces, non-enforcement of documented rules, and defined implementation-dependent behavior. But Guido has been somewhat conservative about this (least so for bug fixes, I think) and sometimes put off changes even when the fault lies with questionable usages.

Which is the source of many unresolved open issues in the tracker -- everyone agrees that "it" is a bug and should be fixed, perhaps there's even a patch, but people may be using it this way, so nothing is done forever. (And these bugs usually are the kind of things we don't want to change in 3.0 either since they are subtle things.) However, I don't have a solution here, so I won't complain about it anymore. Georg

On 01:02 am, grig.gheorghiu@gmail.com wrote:

To bring my $0.02 to this discussion: the Pybots 'community buildbots' turned out largely to be a failure.

Let's not say it's a failure. Let's instead say that it hasn't yet become a success :-).

I still haven't given up, and I hope this thread will spur project leaders into donating time, or resources, to the Pybots project. It has been my bitter observation about the Open Source world that people just LOVE to get stuff for free. As soon as you mention more involvement from them in the form of time, money, hardware resources, etc., the same brave proponents of cool things to be done are nowhere to be found.

I think this list is the wrong place to go to reach the people who need to be reached. It's python core developers and other people already involved in and aware of core development. That said I'm not sure what the *right* place is; I think your blog is syndicated on the unofficial planet python, so maybe that's a good place to start. Sadly, the right thing to do in terms of drumming up support is to get someone interested in PR and have them go to each project individually, but that might be more effort than setting up the buildbots themselves, at least initially... However, let's say that this were tremendously successful, and lots of people start paying attention. I think pybots.org needs to be updated to say exactly what a participant interested in python testing needs to do, beyond "here's how you set up a buildbot" (a page that is actually a daunting-looking blog post which admits it may be somewhat outdated), because setting up a buildbot might not be the only thing that the project needs. It's one thing to tell people that they need to be helping out (and I'm sure you're right) but it's much more useful to get the message out that "we really need people to do X, Y, and Z". One thing I will definitely commit to is that if you make a "cry for help" page, I'll blog about it to drive attention to it, and I'll encourage the other, perhaps better-read Python bloggers I know to do so as well. My personal interest at the moment is to get all of the irrelevant red off of the community builders page. Whether or not you believe in an XP "green bar" philosophy, the large number of spurious failures is distracting. Who is it that is capable of making appropriate changes? Is there something I could do to help with that? Note that I'm committing to say that I can do *that*, but, at least you could shut me up by making it my fault ;-). (I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?) It would be good to remove the perception that it's somebody else's problem as much as possible. Right now, all these dead buildbots suggest to the various communities, "oh, I guess that guy who runs that buildbot needs to fix it". The dead bots should just be killed off, and their projects removed from the list, so that if someone wants to get involved and set up a bot for lxml, they're not put off of it by the fact that it might be rude to the guy who is currently (allegedly) running it.

On 6 Jul, 05:29 pm, martin@v.loewis.de wrote:

...

(I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?)

It seems like you would like to edit the master configuration file. That can be arranged fairly easily.

How shall we arrange it, then? :) Whoever is interested, I've got a recent SSH key on https://launchpad.net/~glyph/+sshkeys

On Sun, Jul 6, 2008 at 8:46 AM, wrote:

However, let's say that this were tremendously successful, and lots of people start paying attention. I think pybots.org needs to be updated to say exactly what a participant interested in python testing needs to do, beyond "here's how you set up a buildbot" (a page that is actually a daunting-looking blog post which admits it may be somewhat outdated), because setting up a buildbot might not be the only thing that the project needs. It's one thing to tell people that they need to be helping out (and I'm sure you're right) but it's much more useful to get the message out that "we really need people to do X, Y, and Z". One thing I will definitely commit to is that if you make a "cry for help" page, I'll blog about it to drive attention to it, and I'll encourage the other, perhaps better-read Python bloggers I know to do so as well.

I have posted 'cries for help' repeatedly on my blog, with generally little success. See http://agiletesting.blogspot.com/search?q=pybots . But I will post again. If you can include here a paragraph of what your vision of the 'X, Y and Z' above is, that'd help too. I think I've been pretty clear about the benefits that the Pybots farm can bring to a given project, so all project leaders on this list should be aware of them IMO. If not, I'd be happy to rehash them. But the home page of pybots.org is pretty self-explanatory I think.

My personal interest at the moment is to get all of the irrelevant red off of the community builders page. Whether or not you believe in an XP "green bar" philosophy, the large number of spurious failures is distracting. Who is it that is capable of making appropriate changes? Is there something I could do to help with that? Note that I'm committing to say that I can do *that*, but, at least you could shut me up by making it my fault ;-).

I'll send a message to the pybots mailing list asking people whose buildbots are turned off if they're still interested in running them. Negative or no answers will mean we can remove them from the farm.

(I'd also like to improve the labels of the build slaves. What exactly is "x86 Red Hat 9 trunk" testing? Trunk of what? What project?)

It's not only a question of changing a static label in this case. A given buildslave can run the tests for multiple projects, in which case it becomes tricky to change the label on the fly accordingly. As an aside, the slave you mention was running on my machine, and I used it to run the Twisted tests, but I shut it down a while ago because the buildbot process was taking too many resources. If the Twisted project can donate a machine, I'd be happy to include it in the Pybots farm ASAP.

It would be good to remove the perception that it's somebody else's problem as much as possible. Right now, all these dead buildbots suggest to the various communities, "oh, I guess that guy who runs that buildbot needs to fix it". The dead bots should just be killed off, and their projects removed from the list, so that if someone wants to get involved and set up a bot for lxml, they're not put off of it by the fact that it might be rude to the guy who is currently (allegedly) running it.

As I said, I'll see what the current owners have to say, and then I'll report back to this list. Thanks for offering your help! Grig

On Sun, Jul 6, 2008 at 2:09 PM, Grig Gheorghiu wrote:

I'll send a message to the pybots mailing list asking people whose buildbots are turned off if they're still interested in running them. Negative or no answers will mean we can remove them from the farm.

OK, I posted a message to the pybots mailing list and I removed 2 slaves. Out of the 6 remaining, 4 are currently active, and one will hopefully soon be active starting next week. This leaves just one unanswered for so far. I also got an email from another person volunteering a buildslave, so we'll soon have 7 machines. As I said, if anybody else wants to participate in the Pybots project, please let me know! I'll also post a blog entry on this soon. Grig

On Tue, Jul 8, 2008 at 12:56 PM, wrote:

It looks to me like the main thing that Pybots needs is help with maintenance. Getting someone to set up an individual buildbot is one thing, but keeping it working is the important bit and it looks like people are not doing that. The project would also be greatly aided by having dedicated people diagnose errors, report bugs against Python core if they're real and report bugs against Pybots if they're spurious.

It would be good to have this effort be centralized and directed because it would otherwise be too easy to file duplicate bug reports, or to assume "oh, this has been failing for months, someone must have filed a bug already".

I agree with all you're saying here. As usual, the devil is in the details. Finding those 'dedicated people' and also people who would act as the central point of contact for bug reports etc. turns out to be very hard in practice. If you have any ideas, I'd be glad to hear them. Grig

On Mon, Jul 7, 2008 at 5:39 AM, Antoine Pitrou wrote:

- are more buildbots needed and if so, which kinds of platforms/architectures?

I can't really answer that question for the python code buildbot farm, but for the Pybots community project, the platforms we currently have are in a table on this page: http://pybots.org/ If you are able to offer something that's not on the list, that'd be good. But any help at all is appreciated. I believe Windows has traditionally been under-represented in all buildbot farms, and it's likely to stay that way...

- for which software? Python itself? third-party apps and libraries?

For Pybots, we're testing third-party apps and libraries against changes made to Python core. If you're interested in a 3rd party project, and you're willing to stay on top of that project's buildbot status, and notify both the project leaders and the Python core devs whenever you notice an ugly breakage -- then you're exactly the kind of guy we need on the Pybots project :-)

- how resource-consuming is it? CPU? memory? disk space? can it run along other services fine or does it need the whole machine for itself?

In my experience, buildbot runs fine on newer hardware. It does consume CPU, so if you have a slow machine, it might start impacting your other processes.

- how time-consuming is it (in terms of human work)? I may spend a bit of time at the start to set it up but I'd like it to it run quite flawlessly afterward. I'm really not a sysadmin at heart...

The initial learning curve can be a bit steep, but I'm here to help. Once you add your buildslave to the buildbot farm, things should run fairly smoothly. You will get notified via email / RSS about breakages, and then you'll have to invest the time to see what kind of breakage it is, and to notify the interested parties.

I suppose other interested people could ask themselves the same questions...

Just my 2 cents.

Antoine.

Thanks for the questions, they really help IMO. I also hope the answers helped. Grig