On 7/13/06, glyph@divmod.com wrote:

On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson wrote:

...

glyph@divmod.com writes:

...

...

For example, did anyone here know that the new-style exceptions stuff in 2.5 caused hundreds of unit-test failures in Twisted? I am glad the change was made, and one of our users did catch it, so the process isn't fatally broken, but it is still worrying.

When implementing this stuff, I could have (... snip ...)

To be clear, I agree with the decision you made in this particular case. I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen. (Of course I *can* always do that, and some other Twisted devs watch python-dev a bit more closely than I do, but the point is that the amount of effort required to do this is prohibitive for the average Python hacker, whereas the time to set up an individual buildbot might not be.)

One way to try to track python-dev is the python-dev Summaries. While Steven is only human and thus cannot always get them out immediately, they do happen frequently enough that major decisions will be covered before betas are hit. -Brett (Aside: IMHO, the sooner we can drop old-style classes entirely, the better.

That is one bumpy Python upgrade process that I will be _very_ happy to do. There's no way to have documentation that expresses the requirement that an implementation of an interface be new-style or old-style without reference to numerous historical accidents, which are bewildering and upsetting to people reading documentation for the first time.) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On 7/13/06, Giovanni Bajo wrote:

I think python should have a couple more of future imports. "from __future__ import new_classes" and "from __future__ import unicode_literals" would be really welcome, and would smooth the Py3k migration process

Along similar lines as glyph, after complaining about this for a long time "offline" with my friends, I decided it's probably a good idea to get involved and vote that the __future__ import for unicode_literals be implemented. python -U is a failure for obvious reasons and a __future__ import is clearly better. Does anyone want to pair on this? -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. Löwis\"" wrote:

Christopher Armstrong wrote:

...

python -U is a failure for obvious reasons and a __future__ import is clearly better.

I disagree.

I am surprised that you do, since I thought that Chris's conclusion was pretty obvious. Python -U doesn't work, even on the standard library. For example, glyph@legion:~% python -S -U -c 'import pickletools' Traceback (most recent call last): File "<string>", line 1, in ? File "/usr/lib/python2.4/pickletools.py", line 219, in ? doc="One-byte unsigned integer.") File "/usr/lib/python2.4/pickletools.py", line 187, in __init__ assert isinstance(name, str) AssertionError This was just the first convenient example. There are others. A __future__ import would allow these behaviors to be upgraded module-by-module. Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is.

On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan wrote:

glyph@divmod.com wrote:

...

A __future__ import would allow these behaviors to be upgraded module-by- module.

No it wouldn't.

Yes it would! :)

__future__ works solely on the semantics of different pieces of syntax, because any syntax changes are purely local to the current module. ... Changing all the literals in a module to be unicode instances instead of str instances is merely scratching the surface of the problem - such a module would still cause severe problems for any non-Unicode aware applications that expected it to return strings.

A module with the given __future__ import could be written to expect that literals are unicode instances instead of str, and encode them appropriately when passing to modules that expect str. This doesn't solve the problem, but unlike -U, you can make fixes which will work persistently without having to treat the type of every string literal as unknown. The obvious way to write code that works under -U and still works in normal Python is to .encode('charmap') every value intended to be an octet, and put 'u' in front of every string intended to be unicode. That would seem to defeat the purpose of changing the default literal type.

On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote:

Note that it also helps setting the default encoding to 'unknown'. That way you disable the coercion of strings to Unicode and all the places where this implicit conversion takes place crop up, allowing you to take proper action (i.e. explicit conversion or changing of the string to Unicode as appropriate).

I've tried that before to verify no such conversion issues occurred in Twisted, but, as the python stdlib isn't usable like that, it's hard to use it to find bugs in any other libraries. (in particular, the re module is badly broken, some other stuff was too). James

glyph@divmod.com wrote:

A module with the given __future__ import could be written to expect that literals are unicode instances instead of str, and encode them appropriately when passing to modules that expect str.

Such changes might have to be reverted in Python 3, since the module might then expect character strings instead of byte strings, and then might complain when it gets byte strings (which .encode will produce). So declaring that all string literals are Unicode objects might not help in the future, contrary to what the future import suggests.

The obvious way to write code that works under -U and still works in normal Python is to .encode('charmap') every value intended to be an octet, and put 'u' in front of every string intended to be unicode. That would seem to defeat the purpose of changing the default literal type.

The not so obvious way is to change the modules/methods receiving these strings to work with either string type if that is reasonable. Regards, Martin

On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. Löwis\"" wrote:

People can use [-U] to improve the Unicode support in the Python standard library. When they find that something doesn't work, they can study the problem, and ponder possible solutions. Then, they can contribute patches. -U has worked fine for me in the past, I contributed various patches to make it work better. It hasn't failed for me at all.

I guess it makes more sense as a development tool to work on zero-dependency tools like the standard library. Still, -Q has both a __future__ import and a command-line option, why not -U?

Bob Ippolito wrote:

"from __future__ import new_classes" exists, but the syntax is different:

__metaclass__ = type

Although it's not a very obvious spelling, particularly to the casual reader who may not be familiar with the intricacies of classes and metaclasses. I don't think it would hurt to have it available as a __future__ import as well. There's also the advantage that all of a module's future assumptions could then be documented uniformly in one place, i.e. in a __future__ import at the top. -- Greg

Giovanni Bajo wrote:

glyph@divmod.com wrote:

I think python should have a couple more of future imports. "from __future__ import new_classes" and "from __future__ import unicode_literals" would be really welcome, and would smooth the Py3k migration process

Actually - can we make new-style classes the default, but allow a way to switch to old-style classes if needed? Perhaps a command-line argument to set the default back to old-style? -- Talin

Talin wrote:

Actually - can we make new-style classes the default, but allow a way to switch to old-style classes if needed?

That sounds dangerously like a "from __past__" kind of feature, and Guido has said that there will never be a __past__ module. Also, this is probably not something that should be a program-wide setting. -- Greg

glyph> *can* always do that, and some other Twisted devs watch glyph> python-dev a bit more closely than I do, but the point is that glyph> the amount of effort required to do this is prohibitive for the glyph> average Python hacker, whereas the time to set up an individual glyph> buildbot might not be.) If you're concerned about noticing when a new release train is pulling out of the station I think it would be sufficient to subscribe to the low-volume python-announce mailing list. You will see announcements about alphas and betas there. Even lower volume would be a subscription to the RSS feed of Python announcements on the python.org front page (scroll to the bottom). The buildbot idea sounds excellent. Skip

>> If you're concerned about noticing when a new release train is >> pulling out of the station ... glyph> I am aware of when new releases come out :). What I'm not aware glyph> of is what features (may) have broken my code, and why. Hmmm... Well, you could subscribe to the python-checkins mailing list. Most of the time you'd probably decide rather quickly to delete the messages, but the stuff you care about should be in the checkin comments. Skip

On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy wrote:

Is the following something like what you are suggesting?

Something like it, but...

A Python Application Testing (PAT) machine is set up with buildbot and any needed custom scripts. Sometime after that and after 2.5 is released, when you have a version of, for instance, Twisted that passes its automated test suite when run on 2.5, you send it (or a URL) and an email address to PAT. Other developers do the same. Periodically (once a week?), when PAT is ^ "once per checkin to Python trunk" free and a new green development version of either the 2.5.x or 2.6 branches is available, PAT runs the test suites against that version. An email is sent for any that fail, perhaps accompanied by the concatenation of the relevant checkin message. Some possible options are to select just one of the branches for testing, to have more than one stable version being tested, and to receive pass emails.

Sending email also isn't really necessary; I would just like a web page I can look at (and draw the attention of the python core developers to).

glyph@divmod.com writes:

I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen.

I think something that has happened to some extent with this release is that there was a long-ish period where stuff got discussed and implemented (PEPs 343 and 344, new-style exceptions, the new compiler, the ssize_t branch) and then suddenly we went "crap! we have to release this!" and then the whole alpha/beta/release part has happened much more quickly. Maybe we should have had a more explicit discussion on what was going to be in 2.5 before beta1, but that kind of thing is difficult to make happen (it's entirely possible that it did happen and I just missed it by being snowed under, but I don't think so). Cheers, mwh -- Have you considered downgrading your arrogance to a reasonable level? -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll

On Thu, 13 Jul 2006 11:29:16 -0700, Aahz wrote:

There's been some recent discussion in the PSF wondering where it would make sense to throw some money to remove grit in the wheels; do you think this is a case where that would help?

Most likely yes. It's not a huge undertaking, and there are a lot of people out there in the community with the knowledge of Buildbot to make this happen.

On 7/13/06, Christopher Armstrong wrote:

On 7/13/06, glyph@divmod.com wrote:

...

On Thu, 13 Jul 2006 11:29:16 -0700, Aahz wrote:

...

There's been some recent discussion in the PSF wondering where it would make sense to throw some money to remove grit in the wheels; do you think this is a case where that would help?

Most likely yes. It's not a huge undertaking, and there are a lot of people out there in the community with the knowledge of Buildbot to make this happen.

I'm at least willing to set up the buildbots for projects I care about (Twisted, pydoctor, whatever), and perhaps help out with the setting up the buildmaster.

It would be really great to have someone setup buildbots for other important/large python projects to test python svn HEAD with. I can't even stay on top of the python buildbots, so I'm not volunteering for more work. I will greatly appreciate and thank whoever takes this on though. :-) The harder problem is keeping it running. Setting it up, in some ways is a lot easier. It's at least well defined. Maintaining it for several months is a different story. n

On Fri, Jul 14, 2006 at 01:37:28PM +0200, Fredrik Lundh wrote:

(add PEP announcements and python-dev summary items to the mix, and you have a high-quality development blog generated entirely from existing content) (hmm. maybe this could be put together by a robot? time to start hacking on "The Daily URL 2.0 Beta", I suppose...)

There's already an RSS feed for the python-dev summaries, so that could go into an aggregator. There isn't a feed for the PEPs, but those events are rarer and could be handled manually. --amk

On Thursday 13 July 2006 16:05, Barry Warsaw wrote:

This really is an excellent point and makes me think that we may want to consider elaborating on the Python release cycle to include a gamma phase or a longer release candidate cycle. OT1H I think there ... "absolutely no changes are allowed now unless it's to fix backward compatibility". No more sneaking in new sys functions or types module constants <wink> during the gamma phase.

+42 It feels like the release cycle from alpha1 to final has gotten increasingly rushed. While I'm sure some of that is just me having my attention elsewhere, I suspect a longer tail on the cycle to do gammas (or release candidates, or whatever) would definately encourage more testing with applications and the larger frameworks. No, it won't catch everything, but I think it would help. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On Friday 14 July 2006 00:32, skip@pobox.com wrote:

Same here. I believe there was some shortening of the 2.5 release cycle two or three months ago. I don't recall why or by how much, but I think the acceleration has resulted in a lot of the "can't we please squeeze this one little change in?" that's been happening. Shortening a micro release a bit seems reasonably easy to accommodate, but since minor releases occur so infrequently, I think it would be better to stretch them out if necessary.

The squeezing of the releases isn't where the problem is, I think. It's that, once squeezed, more releases aren't being added to compensate. We really need to determine what time we need to go from beta1 to (gamma|rc)1, and then from (gamma|rc)1 to final. Plenty of interim releases in the beta phase is good. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On 7/13/06, skip@pobox.com wrote:

Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed.

I think that's just because you are getting older and time goes by faster the less time you've got left. :-) It seems to be going quite quick, but I'm not so sure it's really different.

Same here. I believe there was some shortening of the 2.5 release cycle two or three months ago. I don't recall why or by how much, but I think the acceleration has resulted in a lot of the "can't we please squeeze this one little change in?" that's been happening. Shortening a micro release a bit seems reasonably easy to accommodate, but since minor releases occur so infrequently, I think it would be better to stretch them out if necessary.

Not exactly. The PEP was created Feb 7 (r42266) with a very relaxed schedule to allow enough time for ssize_t branch to solidify. Martin was ready sooner, so on Feb 26 (r42577) sped up the schedule to something more reasonable: - alpha 1: April 1, 2006 [planned] - alpha 2: April 29, 2006 [planned] - beta 1: June 24, 2006 [planned] - beta 2: July 15, 2006 [planned] - rc 1: August 5, 2006 [planned] - final: August 19, 2006 [planned] The current schedule is: + alpha 1: April 5, 2006 [completed] + alpha 2: April 27, 2006 [completed] + beta 1: June 20, 2006 [completed] + beta 2: July 11, 2006 [completed] + rc 1: August 1, 2006 [planned] + final: August 8, 2006 [planned] So beta1 was 4 days earlier than the date set at the end of Feb. The first beta was about 19 months after 2.4 was released if I'm counting correctly. In 2.4 beta1 to release was 1.5 months. For 2.5 it is about the same. The total 2.4 schedule (from a1 to release) was 4.5 months which included 3 alphas. For 2.5 we only had 2 alphas and the total schedule is just over 4 months. So 2.5 is just a little faster. But 2.5 has had a ton more testing than any previous released due to the buildbots, not to mention Coverity and now Klocwork. I definitely pushed the schedule, but I don't think it was really all that different from in the past. Given that several people here think we should lengthen the schedule in some way, I suspect we will do something. I'm not really against it, but I don't think it will provide much benefit either. A few more bugs will be fixed since we have more time. What I think might really make a difference if we could leverage the buildbots and copy the builds (especially WIndows) up to python.org and make them available for download. That might make it easier for people to try stuff out. n

On 7/13/06, Fredrik Lundh wrote:

Neal Norwitz wrote:

...

Given that several people here think we should lengthen the schedule in some way, I suspect we will do something. I'm not really against it, but I don't think it will provide much benefit either. A few more bugs will be fixed since we have more time.

you're still missing the point of this thread: it isn't bugs in the core Python distribution that's the problem, it's compatibility with third- party modules and applications.

...

a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.

This is the part I don't get. For the external developers, if they care about compatibility, why aren't they testing periodically, regardless of alpha/beta releases? How often is the python build broken or otherwise unusable? On Unix there's no more or less work to grab a tarball whether it's a nightly or a release. For Windows it's definitely easier to wait for a release. If there was an installable windows version made by the buildbots, would that mean there would be more testing? Is part of your point that these developers only care about something called "release" and they won't start testing before then? If that's the case why don't we start making (semi-)automated alpha releases every month? That will give people lots more time. Somehow I don't think it will make a difference. People mostly work on schedules or should I say avoid schedules. Give them 5 months and they will still wait to the last minute. Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments). n

On Friday 14 July 2006 17:00, Fredrik Lundh wrote:

because they don't want to waste time and effort on testing against releases that are not necessarily a close approximation of the full release? testing takes time, and time can be used for many different things.

Oddly enough, so does release management. Anthony -- Anthony Baxter It's never too late to have a happy childhood.

On Friday 14 July 2006 16:39, Neal Norwitz wrote:

Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments).

Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that. Anthony -- Anthony Baxter It's never too late to have a happy childhood.

On 7/14/06, Nick Coghlan wrote:

Anthony Baxter wrote:

...

On Friday 14 July 2006 16:39, Neal Norwitz wrote:

...

Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments).

Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that.

I think this is where release candidates can come into their own - the beta's can flush out the "just-one-more-thing" pleases, the maintenance branch is forked for rc1, and then any changes on the branch are purely to fix regressions.

As far as the idea of a suite of buildbots running the unit tests of large Python applications against the current SVN trunk goes, one problem is that failures in those buildbots will come from two sources: - Python itself fails (broken checkin) - the application unit tests fail (backwards incompatibility)

To weed out the false alarms, the slaves will need to be set up to run the Python unit tests first and then the application unit tests only if the Python test run is successful.

When the slave suffers a real failure due to a backwards incompatibility, it will take a developer of the application to figure out what it was that broke the application's tests.

So while I think it's a great idea, I also think it will need significant support from the application developers in debugging any buildbot failures to really make it work.

These buildbots should run the tests from stable, released versions of external packages, assuming that those packages don't ship releases with failing tests. If you ran the test suite for a Zope release and had a test failure, I think there would be a reasonable expectation that it was a Python bug. I'd definitely avoiding mixing a development version of Python and a development version of the test software in the automated build environment. Jeremy

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu

On 7/14/06, Anthony Baxter wrote:

On Friday 14 July 2006 16:39, Neal Norwitz wrote:

...

Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments).

Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that.

Maybe the basic question is right, but the emphasis needs to be changed. If we had a rule that said the final release was 90 days after the last submission that wasn't to fix a regression, we'd ask "Is this feature important enough to warrant delaying the release until three months from now?" I'm not sure what I think, but it doesn't seem like an implausible policy. Jeremy

Anthony -- Anthony Baxter It's never too late to have a happy childhood. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu

Anthony Baxter wrote:

On Friday 14 July 2006 22:45, Jeremy Hylton wrote:

...

Maybe the basic question is right, but the emphasis needs to be changed. If we had a rule that said the final release was 90 days after the last submission that wasn't to fix a regression, we'd ask "Is this feature important enough to warrant delaying the release until three months from now?" I'm not sure what I think, but it doesn't seem like an implausible policy.

I really really doubt that this would work. There's always pressure for "just one more feature" - and as the release drags on, this would increase, not decrease. Note that dragging the release process out has it's own costs, as mentioned previously - either the trunk is in some sort of near-frozen state for an extended period, or else we end up in the double-applying-bugfixes state by forking much earlier.

This approach would also make it extremely difficult to plan for releases. I know that my free time varies across the course of the year, and I need _some_ sort of plan for when the release will happen so I can make sure I have the time free to spend on it.

On the face of it, it seems to me that branching a new major release at the 1st beta would be one way of managing this. The trunk is not frozen for an extended period, and any "features" and bug fixes could probably be backported from the trunk on clearance from the RE with no more pain than is currently endured. One down side would be the possibility of a loss in emphasis in finishing the job on the release to be. I'm sure there are others... ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

Neal Norwitz wrote:

...

a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.

This is the part I don't get. For the external developers, if they care about compatibility, why aren't they testing periodically, regardless of alpha/beta releases?

Because it is a cost, and I don't want to waste my time if the trunk is unstable or whatnot. There is no official statement of the kind "all the real development is done in branches, the trunk is always very stable, feel free to grab it". Thus, we (external developers) assume that it's better to wait for the first alpha or beta before starting doing any testing. Personally, I won't touch something before the first beta: there are already enough known bugs when the alphas are shipped that I prefer to wait a little bit more. In my case, the beta period is too short. It's already a great luck if, during the beta cycle, I'm not deep into some milestone or release cycle for my own software. It might well happen that I have barely time to notice a Python beta is out, but I can't afford spending any time on it because of time constraint. By the time I'm done with my own deadlines, Python final is already out. But for the sake of the argument and the rest of this mail, let's assume I have tons of spare time to dedicate to Python 2.5 beta testing. My applications have several external dependencies (I wouldn't classify those as "many", but still around 6-7 libraries in average). For each of those libraries, there won't be Py2.5-ready RPM or Windows installer to grab and build. Most of the time, I have to download the source package, and try to build it. This also means hunting down and fixing Py2.5-related bugs in all those libraries *before* I can even boot my own application (which is what I would care about). Then I occasionally hit a core Python bug while doing so, which is good, but I have to sit and wait. Some libraries have very complex build process which are still distutils-based, but might require many other external C/C++ libraries, which need to be fetched and compiled just to setup the build environment. Alternatively, I could cross my fingers and wait for all the maintainers of the external libraries to have spare time, and dedicate to Python 2.5 upgrading. If I'm lucky, by the time RC1 is out, most of them might have binary packages available for download, or at least have their (unstable) CVS/SVN trunk fixed for Python 2.5 (which means that I'll have to fetch that unstable version and basically perform a forced upgrade of the library, which might trigger another class of compatibility/feature/regression bugs in my application, not related at all to Python 2.5 by itself, but still needed to be dealt). So I think it's useless to ask people to rush testing beta releases: it takes months to get the community synched, and will always take. It's also useless to point the finger to external developers if they don't test Python and there is a bug in a .0 release. Bugs happen. It's software that evolves. My own suggestion is that it's useless to stall a release for months to give external developers time to fix things. I think it's much better to "release early - release often", and just get the damn release out of the door. Usually, it's at that point that the whole community start working on it, and discover bugs. And so what? For production usage, .0 releases of libraries (let alone the interpreter of the language) are a no-go for all software, and I know that for a long time already. I don't ship an application of mine with a Python .0 release no matter what, no matter even if the whole testsuite passes and everything seems to work. I don't have enough benefits for the risks, so I'll wait for .1 or .2 release anyway. It's *very* fine by me if .0 release is actually the first "official" "beta" release for the community, when the core developers say "we're done" and external developers really start get things going. If you really care about .0 stability from a purely commercial point-of-view (it's bad advertisement if many people complain if a major release is broken, etc. etc.), I might suggest you to coordinate with a small group of selected external developers maintaing the larger external packages (Twisted and others). You could put a list of those packages in the release PEP as showstoppers for the release: you should not get a .0 out if those packages are broken. I think this could help smooth out the process. If important external libraries work, many applications relying on it will *mostly* work as well. I personally don't think it's such a big problem if one has to fix a couple of things in a 100K-line application to adjust it to the new .0 release, even if it's actually because of a bug in Python itself. Giovanni Bajo

"A.M. Kuchling" wrote in message news:20060714112137.GA891@Andrew-iBook2.local...

http://www.python.org/dev/tools/, in a discussion of checkin policies, does say:

The Python source tree is managed for stability, meaning that if you make a checkout at a random point in time the tree will almost always compile and be quite stable.

That is the goal, but when I watched the buildbot results last spring, the degree of stability (greenness) appeared to vary. Is it possible to tag particular versions as a 'green' version, or the 'most recent green version' worth playing with? tjr

On 7/15/06, "Martin v. Löwis" wrote:

Terry Reedy wrote:

...

That is the goal, but when I watched the buildbot results last spring, the degree of stability (greenness) appeared to vary. Is it possible to tag particular versions as a 'green' version, or the 'most recent green version' worth playing with?

Don't get confused by these colors. The tree compiled nearly all of the time, even if some tests were failing for an extended period of time on some platforms.

Right. It's very rare that the interpreter or stdlib is broken. There are 19 buildbots currently. 7 are not green. 5 of those are offline, most with known (to me and the person that donated the machines) hardware issues ranging from lack of air conditioning, to a bad router, to no power. 1 is cygwin which is running an old version. I just (an hour or so ago) got an account on a cygwin box to help diagnose the status and figure out if anything within Python or the tests are broken. That leaves 1 unexplained failure on a Windows bot. This started happening recently after being down for a while. I haven't had time to investigate. The reason why it was not very green in spring was because that's when we were getting it set up! The master looks like it was installed at the end of 2005/beginning of 2006. It took several months to get rid of many testing issues. Tests that couldn't be run in a particular order, tests that couldn't be run simultaneously (socket), bad compilation of sqlite/bsddb modules (not in python), misconfigured systems, tests verifying something that was system dependent, signed addresses, etc. Of all the problems there were, I only remember a single problem in Python. (There were probably more, I just remember one.) That was in test code (xxmodule or something like that. There were a bunch of problems with ctypes and/or sqlite when they got integrated having to deal with these new platforms. That may be what you are recalling. Right before alpha 2 was a particularly bad time. What we mean by stable is that at any time/any stage of development, if a change goes in that breaks the tests, it is likely to be fixed or reverted within hours. The amount of time the build or tests are broken on the majority of platforms is quite small. It took a while to get to this point due to a bunch of flaky tests, but those have mostly been fixed. The only known problems are mentioned on the wiki. When the buildbots fail, we get mail to python-checkins. Unfortunately, that's mostly spam at this point. I hope to fix that at some point. I also hope to change the main page to give more info in less space. n

At the risk of waking up a thread that was already declared dead, but perhaps this is usefull. So, what happens is pythons signal handler sets a flag and registrers a callback. Then the main thread should check the flag and make the callback to actually do something with the signal. However the main thread is blocked in GTK and can't check the flag. Nick Maclaren wrote: ...lots of reasons why you can't do anything reliably from within a signal handler... As far as I understand it, what could work is this: -PyGTK registrers a callback. -Pythons signal handler does not change at all. -All threads that run in the Python interpreter occasionally check the flag which the signal handler sets, like the main thread does nowadays. If it is set, the thread calls PyGTKs callback. It does not do anything else with the signal. -PyGTKs callback wakes up the main thread, which actually handles the signal just like it does now. PyGTKs callback could be called from any thread, but it would be called in a normal context, not in a signal handler. As the signal handler does not change, the risk of breaking anything or causing chaos is as large/small as it is under the current scheme. However, PyGTKs problem does get solved, as long as there is _a_ thread that returns to the interpreter within some timeframe. It seems plausible that this will happen.

On 9/8/06, Adam Olsen wrote:

Ensuring modifications to that array are atomic would be tricky, but I think it would be doable if we use a read-copy-update approach (with two alternating signal handler functions). Not sure how to ensure there's no currently running signal handlers in another thread though. Maybe have to rip the atomic read/write stuff out of the Linux sources to ensure it's *always* defined behavior.

Doh, except that's exactly what sig_atomic_t is for. Ah well, can't win them all. -- Adam Olsen, aka Rhamphoryncus

On 9/9/06, Jan Kanis wrote:

At the risk of waking up a thread that was already declared dead, but perhaps this is usefull.

So, what happens is pythons signal handler sets a flag and registrers a callback. Then the main thread should check the flag and make the callback to actually do something with the signal. However the main thread is blocked in GTK and can't check the flag.

Nick Maclaren wrote: ...lots of reasons why you can't do anything reliably from within a signal handler...

As far as I understand it, what could work is this: -PyGTK registrers a callback. -Pythons signal handler does not change at all. -All threads that run in the Python interpreter occasionally check the flag which the signal handler sets, like the main thread does nowadays. If it is set, the thread calls PyGTKs callback. It does not do anything else with the signal. -PyGTKs callback wakes up the main thread, which actually handles the signal just like it does now.

PyGTKs callback could be called from any thread, but it would be called in a normal context, not in a signal handler. As the signal handler does not change, the risk of breaking anything or causing chaos is as large/small as it is under the current scheme.

However, PyGTKs problem does get solved, as long as there is _a_ thread that returns to the interpreter within some timeframe. It seems plausible that this will happen.

No, it is not plausible at all. For instance, the GnomeVFS library usually has a pool of thread, not doing anything, waiting for some VFS task. It is likely that a signal will be delivered to one of these threads, which know nothing about Python, and sit idle most of the time. Regards.

Jan Kanis wrote:

However, PyGTKs problem does get solved, as long as there is _a_ thread that returns to the interpreter within some timeframe. It seems plausible that this will happen.

I don't see that this makes the situation much better, as it just shifts the need for polling to another thread.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg.ewing%40canterbury.ac...

On Fri, 14 Jul 2006 06:46:55 -0500, skip@pobox.com wrote:

Neal> How often is the python build broken or otherwise unusable?

Not very often.

I have to agree. The effort I'm talking about is not in fixing large numbers of problems, but simply gearing up to properly test to see if there *are* problems. Keep in mind though: just because the problems are small or easy to fix doesn't mean they're not severe. One tiny bug can prevent a program from even starting up.

Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I suspect that my usage of the language isn't all that different from most Python developers out there.

A huge percentage of Python developers are working with Zope, which means that although *their* code might not be terribly "sophisticated", it is nevertheless sitting on top of a rather large and intricate pile of implicit dependencies on interpreter behavior.

Neal> Is part of your point that these developers only care about Neal> something called "release" and they won't start testing before Neal> then? If that's the case why don't we start making Neal> (semi-)automated alpha releases every month?

How would that be any easier than a user setting up a read-only repository and svn-up-ing it once a month then using that as the default interpreter on that person's development machine? I maintain interpreters for 2.3, 2.4 and bleeding edge at the moment. If I need to it's fairly trivial (a symlink change) to fall back to the latest stable release.

Glyph, would that sort of scheme work for you?

No. I really think you're underestimating the sort of effort required to upgrade Python. First of all, I do have a job :) and it'd be very hard to make the case to an investor that it was worth tracking down every potential bug I had to determine whether it was a problem because I was working with an unreleased version of Python. This is the same reason I don't use beta versions of the kernel or libc for development. For that matter, I've had to avoid posting to this mailing list because even *this* is stretching my already overburdened schedule :). Secondly, I have to work with a few extension modules. To name a few: * ctypes * PyPAM * pysqlite2 * zope.interface (C optimizations) * xapian * pylucene * dspam * PyOpenSSL * PIL * PyCrypto * pyexpat * pygtk * pyvte Recompiling all of these is a project that takes a day. PyLucene, in fact, I've never managed to build myself, and I can only run because there happen to be debian packages which work (with some fiddling) on Ubuntu. There's no interactive warning during 'svn up' that it's time to recompile everything, either, so I don't even know when the ABIs are going to have changed. Even if everything works perfectly, and were perfectly automated, the compile itself would take a few hours. I also test work on Windows on occasion and recompiling these things _there_ is the work of a week and a half, not to mention it requires that I be sitting at the one machine where I have my Microsoft™ Developer™ Tools™ installed. I made the buildbot recommendation specifically because it would centralize the not inconsiderable effort of integrating these numerous dependencies (presuming that I could submit a Divmod buildbot as well as a Twisted one).

Fred L. Drake, Jr. wrote:

It feels like the release cycle from alpha1 to final has gotten increasingly rushed.

python 2.2: ~5 months python 2.3: ~7 months python 2.4: ~5 months python 2.5: ~4 months I think the biggest problem is the time between "basically stable beta" and "final"; it was ~90 days in 2.3 and is now ~50 days, most of which is vacation time in my part of the world. we could extend the beta period by playing silly games with naming, but as long as the "big things" go in *before* the beta period starts, I think it's easier to simply extend the beta period by adding another month to it. I'd prefer something like 90 days. </F>

Barry Warsaw wrote:

we may want to consider elaborating on the Python release cycle to include a gamma phase or a longer release candidate cycle.

Maybe there could be an "unstable" release phase that lasts for a whole release cycle. So you'd first release version 2.n as "unstable", and keep 2.(n-1) as the current "stable" release. Then when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would become the new "unstable". (Feel free to substitute other terms for "stable" and "unstable" if you don't like them.) That would give people plenty of warning and time to try things out with the new version. This wouldn't actually be much different to what is done now, except for the naming. But by not officially blessing the latest release as current for a while, it might give less of an impression that stuff is being sprung on the community unawares.

OT1H I think there will always be people or projects that won't try anything until the gold release,

Well, at least they've had a chance to try it. If they don't take that chance, they don't have much ground for complaint. -- Greg

skip@pobox.com wrote:

Greg> Maybe there could be an "unstable" release phase that lasts for a Greg> whole release cycle. So you'd first release version 2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable" release. Then Greg> when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would Greg> become the new "unstable".

In GCC don't they do an odd (stable)/even (unstable) release schedule? Same for Linux kernels? Would that help?

No. Linux kernel releases are more aggressive because of the fact that all the patches are mostly developed in different branch/repositories and get merged when they are already well tested and incorporated. Linus can merge literally hundreds of patches daily into his *stable* tree, and do releases from it even weekly, because most destabilizing works are being done in large branches carried on for months before they even are evaluated for being merged; or because patches were settled in the -mm tree for months. Linus' tree is kind-of a release branch, with the difference that he is the BDFL and does what he wants with his tree :) To keep this into perspective, remember also that they don't have *any* kind of testsuite (nor a debugger, if I might say). GCC has a more "old-fashioned" release process, where the trunk evolves through 3 stages: Stage 1 is open for all kind of changes (eg: from simple polishing/refactoring, to merging of large branches containing work of several man-years). Stage 2 is still open for new features, but not for big merges. Stage 3 is feature-freezed, bug-fixing only. Then, the trunk is branched into the new release branch, and the trunk gets back to Stage 1. Nominally, a stage lasts 3 months, but Stage 3 often stretches up to 6 months. The release branches are open for *only* regression fixes (that is, fixes that correct things that used to work in previous releases but do not work anymore). Any regression fix (with a corresponding Bugzilla entry, where it's marked and confirmed as "regression") is applied to trunk *and* the open release branches where the regression exists. For convoluted or dangerous regression fixes, usually maintainers prefer to wait 1 week for the patch to settle down on the trunk before applying it to the release branches. The release manager pulls dot releases from the release branch. Usually, the first release (.0) happens 3-4 months *after* the release branch was created, that is after several months of regression-only patches being merged to it (while new work is being done on the trunk in parallel, in its aggressive Stage 1). The 3-Stage work in the trunk is streamlined by the release manager. At the beginning of Stage 1, a detailed techinical list of on-going "projects" (new features) is presented to the mailing list, explaining the current status of the work and its ETA, and the release manager then publishes a work-plan for Stage 1 and 2, telling which projects will be merged when. This avoids multiple large projects to hit the trunk at the same time, causing may headaches to all the other developers. The work-plan is managed and updated in the GCC Wiki (which is off-line right now, but I'll post a link as example when it's back). -- Giovanni Bajo

Jeroen Ruigrok van der Werven wrote:

This is what Xenofarm for Pike has done for a long time now. See for example: http://pike.ida.liu.se/development/pikefarm/7.7.xml

This is also what Bitten (http://bitten.cmlenz.net/) has implemented for Trac (which is one of the bug/incident trackers for Python's call for trackers).

Are you aware of http://www.python.org/dev/buildbot/ ? We are not just talking about buildbots here (which the links you refer to seem to be); we are talking about buildbots that don't test Python, but test Python applications. We do know how to run a buildbot. Regards, Martin

Jeroen Ruigrok van der Werven wrote:

...

Are you aware of http://www.python.org/dev/buildbot/ ?

Yes. And it does not seem to be open for all

Ah, ok. It indeed isn't open for anonymous participation; the test results are open for all, though.

...

We are not just talking about buildbots here (which the links you refer to seem to be); we are talking about buildbots that don't test Python, but test Python applications.

Then I would dare to say you haven't fully investigated the links fully, Bitten, for example, also runs the unit-tests for any target you configure

I don't understand that comment. Bitten seems to be a software package, similar to buildbot. It doesn't do anything useful until I install and configure it, unlike, say, SourceForge, which is not (just) a soft package, but a running installation of it also. Right? The effort is in installing and configuring it, given that many such packages are already written. Distributed testing frameworks typically support running arbitrary target comments, so Bitten is not really special here (buildbot can also run about anything if you configure it to).

...

We do know how to run a buildbot.

How relevant was this comment in the entire? I am merely trying to help out here pointing out other similar projects when the community participating building issue was raised.

It would have been helpful if you had stated *why* you think these URLs are relevant in the context of the discussion. To me, it seemed you are pointing to the existence of distributed testing frameworks. I was pointing out that we (at least, I) am aware of the existence of such frameworks, and that we were doing it for quite some time now also, just like Pike. Regards, Martin