
On Wed, 12 Jul 2006 10:30:17 +1000, Michael Ellerman <michael@ellerman.id.au> wrote:
Having been exhorted (or maybe I mean "excoriated") by your friendly release manager earlier this week to post my comments and criticisms about Python here rather than vent in random IRC chatter, I feel morally compelled to comment. I see some responses to that post which indicate that the specific bug will be fixed, and that's good, but there is definitely a pattern he's talking about here, not just one issue. I think there is a general pattern of small, difficult to detect breakages in Python. Twisted (and the various other Divmod projects that I maintain) are thoroughly unit-tested, but there are still significant issues in each Python release that require changes. Unlike the jerk who posted that "python sucks" rant, I'm not leaving Python because one function changed in a major release; I do expect to have to maintain my projects myself, and major releases are "major" for a reason. I only wish that upgrading all my other dependencies were as easy as upgrading Python! I do see that the developers are working with some caution to avoid breaking code, and attempting to consider the cost of each change. However, although I've seen lots of discussions of what "average" code might break when exposed to new versions of Python, these discussions tend to be entirely hypothetical. Do the core Python developers typically run the test suites of major Python applications when considering major changes, such as Zope, Plone, TurboGears, and of course Twisted? Not that breakages would be considered unacceptable -- some gain is surely worth the cost -- but to establish some empirical level of burden on the community? I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to. I can see that certain Python developers expect that some of this work is the responsibility of the user community, and to some extent that's true, but at least half of the work needs to be done _before_ the changes are made. If some Python change breaks half of Twisted, I would like to know about it in time to complain about the implementation, rather than flailing around once the Python feature-freeze has set in and hoping that it's nothing too serious. For example, did anyone here know that the new-style exceptions stuff in 2.5 caused hundreds of unit-test failures in Twisted? I am glad the change was made, and one of our users did catch it, so the process isn't fatally broken, but it is still worrying. Another problem is simply that the Python developers don't have the same very public voice that other language developers do. It doesn't necessarily have to be a blog, but python-dev is fast-paced and intimidating, and a vehicle for discussion among those in the know, rather than dissimenation to the community at large. It's a constant source of anxiety to me that I might miss some key feature addition to Python which breaks or distorts some key bit of Twisted functionality (as the new-style exceptions, or recent ImportWarning almost did) because I don't have enough time to follow all the threads here. I really appreciate the work that Steve Bethard et. al. are doing on the python-dev summaries, but they're still pretty dry and low level. While the new python.org is very nice, I do note that there's no "blogs" entry on the front page, something which has become a fixture on almost every other website I visit regularly. The news page is not very personal, mainly a listing of releases and events. There's no forum for getting the community _excited_ about new features (especially new features which are explicitly enabled by potential breakages), and selling them on the cool uses. Who knows, maybe I'll even start using decorators syntax all over the place if I see them presented somewhere by someone who is very excited about the feature and thinks it's worthwhile, rather than as a defense on a mailing list against a criticism, or a simple announcement of the feature's existence. I've seen the other side of this problem as well, so I can appreciate that it is quite difficult to get this kind of thing right: lots of applications using Twisted break when we change broken or deprecated APIs. Twisted is lucky though; it has numerous subprojects, and I maintain a half-dozen unrelated projects in a different repository. Twisted itself is a fairly representative "application using Twisted", so when something breaks "average" Twisted code, I know about it right away. Python is less an application using Python: the standard library is rather passive (there is no "end-to-end" application path to test, only individual functions and classes), and the test coverage of included Python modules is rather spotty. I hope someone finds some of these suggestions helpful. I only wish I had the time and energy to act on some of them.

No real time to respond in detail here, but one small comment. glyph@divmod.com writes:
Remember these words...
When implementing this stuff, I could have merely made it possible for exceptions to be new-style, and left the builtin exceptions as classic classes. This didn't seem to be an especially good idea, as it would leave code working _most_ of the time, only to break horribly when confronted with a rare new-style exception. So I made the decision (and I don't think I ever got around to explicitly discussing this with anyone, nor if the people who actually updated my patch and checked it in thought about it at all) to make the built in exceptions new-style, precisely to make it a screamingly obvious change. I didn't _know_ when I was doing it that I'd break Twisted unit tests, but I was hardly a surprise. I think the idea of a buildbot running various projects tests suites with svn HEAD is a great idea -- back when I was doing 2.2.1 I did run a few third party test suites (zodb comes to mind) but as always with these things, automation is a good idea. Cheers, mwh --

On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson <mwh@python.net> wrote:
glyph@divmod.com writes:
To be clear, I agree with the decision you made in this particular case. I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen. (Of course I *can* always do that, and some other Twisted devs watch python-dev a bit more closely than I do, but the point is that the amount of effort required to do this is prohibitive for the average Python hacker, whereas the time to set up an individual buildbot might not be.) (Aside: IMHO, the sooner we can drop old-style classes entirely, the better. That is one bumpy Python upgrade process that I will be _very_ happy to do. There's no way to have documentation that expresses the requirement that an implementation of an interface be new-style or old-style without reference to numerous historical accidents, which are bewildering and upsetting to people reading documentation for the first time.)

On 7/13/06, glyph@divmod.com <glyph@divmod.com> wrote:
One way to try to track python-dev is the python-dev Summaries. While Steven is only human and thus cannot always get them out immediately, they do happen frequently enough that major decisions will be covered before betas are hit. -Brett (Aside: IMHO, the sooner we can drop old-style classes entirely, the better.

glyph@divmod.com wrote:
I think python should have a couple more of future imports. "from __future__ import new_classes" and "from __future__ import unicode_literals" would be really welcome, and would smooth the Py3k migration process -- Giovanni Bajo

On 7/13/06, Giovanni Bajo <rasky@develer.com> wrote:
Along similar lines as glyph, after complaining about this for a long time "offline" with my friends, I decided it's probably a good idea to get involved and vote that the __future__ import for unicode_literals be implemented. python -U is a failure for obvious reasons and a __future__ import is clearly better. Does anyone want to pair on this? -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. Löwis\"" <martin@v.loewis.de> wrote:
I am surprised that you do, since I thought that Chris's conclusion was pretty obvious. Python -U doesn't work, even on the standard library. For example, glyph@legion:~% python -S -U -c 'import pickletools' Traceback (most recent call last): File "<string>", line 1, in ? File "/usr/lib/python2.4/pickletools.py", line 219, in ? doc="One-byte unsigned integer.") File "/usr/lib/python2.4/pickletools.py", line 187, in __init__ assert isinstance(name, str) AssertionError This was just the first convenient example. There are others. A __future__ import would allow these behaviors to be upgraded module-by-module. Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is.

glyph@divmod.com wrote:
A __future__ import would allow these behaviors to be upgraded module-by-module.
No it wouldn't. __future__ works solely on the semantics of different pieces of syntax, because any syntax changes are purely local to the current module. This doesn't work for datatypes, because data types can cross module boundaries - other modules may still be using the old behaviour, and there's nothing the current module can do about it. Changing all the literals in a module to be unicode instances instead of str instances is merely scratching the surface of the problem - such a module would still cause severe problems for any non-Unicode aware applications that expected it to return strings.
Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is.
There's never been a concerted effort to make even the standard library work under -U. Maybe that should be a goal for Python 2.6 - figure out what tools or changes are needed to make code -U safe, add them, and then update the standard library to use them. PEP 349 (currently deferred, although I don't recall why) discusses some of the issues involved in making code unicode-safe, while still supporting non-unicode safe applications as clients. Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0349/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes it would! :)
A module with the given __future__ import could be written to expect that literals are unicode instances instead of str, and encode them appropriately when passing to modules that expect str. This doesn't solve the problem, but unlike -U, you can make fixes which will work persistently without having to treat the type of every string literal as unknown. The obvious way to write code that works under -U and still works in normal Python is to .encode('charmap') every value intended to be an octet, and put 'u' in front of every string intended to be unicode. That would seem to defeat the purpose of changing the default literal type.

glyph@divmod.com wrote:
Think of it this way: with the u'' literal prefix you can upgrade the behavior on a per-literal basis ;-) Seriously, it really does makes more sense to upgrade to Unicode on a per-literal basis than on a per-module or per- process basis. When upgrading to Unicode you have to carefully consider the path each changed object will take through the code. Note that it also helps setting the default encoding to 'unknown'. That way you disable the coercion of strings to Unicode and all the places where this implicit conversion takes place crop up, allowing you to take proper action (i.e. explicit conversion or changing of the string to Unicode as appropriate). Later on, you can then simply remove the u'' prefix when moving to Py3k. In fact, Py3k could simply ignore the prefix for you (and maybe generate an optional warning) to make the transition easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006)
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote:
I've tried that before to verify no such conversion issues occurred in Twisted, but, as the python stdlib isn't usable like that, it's hard to use it to find bugs in any other libraries. (in particular, the re module is badly broken, some other stuff was too). James

James Y Knight wrote:
True: it breaks a lot of code that was written to work with both strings and Unicode (or does so by accident ;-). The stdlib isn't too well prepared for Unicode yet, so if your code relies a lot on it, then the above may not be the right strategy for you. Perhaps a new ASCII codec that issues warnings for all these cases would help ?! (one that still converts to Unicode assuming ASCII, but issues a warning pointing to the location in the code where the conversion happend) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006)
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

glyph@divmod.com wrote:
Such changes might have to be reverted in Python 3, since the module might then expect character strings instead of byte strings, and then might complain when it gets byte strings (which .encode will produce). So declaring that all string literals are Unicode objects might not help in the future, contrary to what the future import suggests.
The not so obvious way is to change the modules/methods receiving these strings to work with either string type if that is reasonable. Regards, Martin

glyph@divmod.com wrote:
Sure, it doesn't work. That doesn't mean it's "a failure". It just hasn't been completed yet.
People can use it to improve the Unicode support in the Python standard library. When they find that something doesn't work, they can study the problem, and ponder possible solutions. Then, they can contribute patches. -U has worked fine for me in the past, I contributed various patches to make it work better. It hasn't failed for me at all. Regards, Martin

On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. Löwis\"" <martin@v.loewis.de> wrote:
I guess it makes more sense as a development tool to work on zero-dependency tools like the standard library. Still, -Q has both a __future__ import and a command-line option, why not -U?

Bob Ippolito wrote:
Although it's not a very obvious spelling, particularly to the casual reader who may not be familiar with the intricacies of classes and metaclasses. I don't think it would hurt to have it available as a __future__ import as well. There's also the advantage that all of a module's future assumptions could then be documented uniformly in one place, i.e. in a __future__ import at the top. -- Greg

Talin wrote:
Actually - can we make new-style classes the default, but allow a way to switch to old-style classes if needed?
That sounds dangerously like a "from __past__" kind of feature, and Guido has said that there will never be a __past__ module. Also, this is probably not something that should be a program-wide setting. -- Greg

On Thu, Jul 13, 2006, Talin wrote:
Nope. Python 3.0 will have new-style classes as the default, but there will probably not be any way to use old-style classes. As has been said before, if you want to make new-style classes the default for any module, just put __metaclass__ = type at the top. Please, just drop this subject. Guido has long since Pronounced. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan

glyph> *can* always do that, and some other Twisted devs watch glyph> python-dev a bit more closely than I do, but the point is that glyph> the amount of effort required to do this is prohibitive for the glyph> average Python hacker, whereas the time to set up an individual glyph> buildbot might not be.) If you're concerned about noticing when a new release train is pulling out of the station I think it would be sufficient to subscribe to the low-volume python-announce mailing list. You will see announcements about alphas and betas there. Even lower volume would be a subscription to the RSS feed of Python announcements on the python.org front page (scroll to the bottom). The buildbot idea sounds excellent. Skip

On Thu, 13 Jul 2006 23:27:56 -0500, skip@pobox.com wrote:
The buildbot idea sounds excellent.
Thanks. If someone can set this up, it pretty much addresses my concerns.
I am aware of when new releases come out :). What I'm not aware of is what features (may) have broken my code, and why. As long as Python's trunk@HEAD continues to run the test suites cleanly, I am mostly unconcerned. When it breaks, though, I want a chance to look at the cause of the breakage, *before* there is an alpha or beta Python release out and people are starting to write code that depends on its new features. Most importantly, python-dev is extremely context-sensitive. I want to be able to participate in the discussion when my concerns are relevant and still reasonable to act upon. Additionally I would like to know about these changes so that I can modify code to support Python releases and release a compatible version of Twisted _in advance_ of the Python release that's going to break them, so assuming users are keeping relatively current, there won't be a window where their most recent Twisted release will not work with the most recent Python release.

>> If you're concerned about noticing when a new release train is >> pulling out of the station ... glyph> I am aware of when new releases come out :). What I'm not aware glyph> of is what features (may) have broken my code, and why. Hmmm... Well, you could subscribe to the python-checkins mailing list. Most of the time you'd probably decide rather quickly to delete the messages, but the stuff you care about should be in the checkin comments. Skip

<glyph@divmod.com> wrote in message news:20060714051157.29014.803759518.divmod.quotient.35955@ohm...
Is the following something like what you are suggesting? A Python Application Testing (PAT) machine is set up with buildbot and any needed custom scripts. Sometime after that and after 2.5 is released, when you have a version of, for instance, Twisted that passes its automated test suite when run on 2.5, you send it (or a URL) and an email address to PAT. Other developers do the same. Periodically (once a week?), when PAT is free and a new green development version of either the 2.5.x or 2.6 branches is available, PAT runs the test suites against that version. An email is sent for any that fail, perhaps accompanied by the concatenation of the relevant checkin message. Some possible options are to select just one of the branches for testing, to have more than one stable version being tested, and to receive pass emails. Terry Jan Reedy

On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
Is the following something like what you are suggesting?
Something like it, but...
Sending email also isn't really necessary; I would just like a web page I can look at (and draw the attention of the python core developers to).

<glyph@divmod.com> wrote in message news:20060715181952.29014.1007814129.divmod.quotient.38416@ohm...
If the average rate of checkins is equal to or greater than the maximum rate of test runs then that is physically impossible. (Indeed, queueing theory suggests that keeping the queue size small requires that the checkin be more like half the test rate.) This appears to be currently true for the Python trunk buildbot and I would expect running the test of numerous large applications would take even longer, making the test rate even lower. If you want something like twisted run against multiple Python versions a day, I suspect you will have to do so on a system you control. Terry Jan Reedy

glyph@divmod.com writes:
I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen.
I think something that has happened to some extent with this release is that there was a long-ish period where stuff got discussed and implemented (PEPs 343 and 344, new-style exceptions, the new compiler, the ssize_t branch) and then suddenly we went "crap! we have to release this!" and then the whole alpha/beta/release part has happened much more quickly. Maybe we should have had a more explicit discussion on what was going to be in 2.5 before beta1, but that kind of thing is difficult to make happen (it's entirely possible that it did happen and I just missed it by being snowed under, but I don't think so). Cheers, mwh -- Have you considered downgrading your arrogance to a reasonable level? -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll

On Thu, Jul 13, 2006, glyph@divmod.com wrote:
There's been some recent discussion in the PSF wondering where it would make sense to throw some money to remove grit in the wheels; do you think this is a case where that would help? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes

On 7/13/06, glyph@divmod.com <glyph@divmod.com> wrote:
I'm at least willing to set up the buildbots for projects I care about (Twisted, pydoctor, whatever), and perhaps help out with the setting up the buildmaster. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On 7/13/06, Christopher Armstrong <radix@twistedmatrix.com> wrote:
It would be really great to have someone setup buildbots for other important/large python projects to test python svn HEAD with. I can't even stay on top of the python buildbots, so I'm not volunteering for more work. I will greatly appreciate and thank whoever takes this on though. :-) The harder problem is keeping it running. Setting it up, in some ways is a lot easier. It's at least well defined. Maintaining it for several months is a different story. n

On Thu, Jul 13, 2006 at 02:03:22PM -0400, glyph@divmod.com wrote:
An excellent idea!
So don't read threads -- just try the alphas and betas! The "What's New" documents have a 'porting' section that flags changes that may require application changes, but items are added only if I think of them or if someone suggests them (e.g. generator.gi_frame can be None in Python 2.5 -- I would never have thought people would have code that broke because of this).
A 'Blogs' link could be trivially added by linking to planet.python.org, though the blogs collected there are not in any way 'the Python developers', but a jumble of developers and users. I don't think enough core developers have weblogs (or write about Python) to make a 'python-dev only' planet very useful.
I think the venue for this would be weblog entries or news sites such as oreillynet.com. I do worry that we don't have enough people writing articles and things, but have no idea how to encourage people to write more. --amk

"A.M. Kuchling" wrote:
on the other hand, the material you're producing for the "what's new" document would be a rather excellent development blog in itself. just post new sections to the blog when they're ready... (add PEP announcements and python-dev summary items to the mix, and you have a high-quality development blog generated entirely from existing content) (hmm. maybe this could be put together by a robot? time to start hacking on "The Daily URL 2.0 Beta", I suppose...) </F>

On 7/14/06, A.M. Kuchling <amk@amk.ca> wrote:
I believe that's broken at the moment, at least in the sense that it's no longer updated: http://psf.pollenation.net/cgi-bin/trac.cgi/ticket/366 I don't have enough time right now to figure out the new website and how to fix it. Maybe when I get back from Sydney in a couple of weeks. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 13, 2006, at 2:03 PM, glyph@divmod.com wrote:
And I'm glad you did because you made some excellent comments.
I do plan on very soon running Mailman's meager test suite and running more manual functional tests on Python 2.5b2. Compared to Twisted, Mailman is pretty simple and doesn't do that much fancy stuff so I suspect that it'll mostly work, but you never know until you try it. I'd support adding Mailman to a community buildbot army, although I don't personally have the cycles to build out such a beast. I also plan on testing my Real Job's embedded application against Python 2.5b2 sometime this week or next. We're on Python 2.4.2 atm and that is much more complicated because of all the C API we use. I'm less sanguine about breakage there, but I'll post here if I find anything egregious (fortunately, we have an extensive test suite to rely upon).
This really is an excellent point and makes me think that we may want to consider elaborating on the Python release cycle to include a gamma phase or a longer release candidate cycle. OT1H I think there will always be people or projects that won't try anything until the gold release, and that if we've broken anything in their code we simply won't know it until after that, no matter how diligent we are. OTOH, a more formal gamma phase would allow us to say "absolutely no changes are allowed now unless it's to fix backward compatibility". No more sneaking in new sys functions or types module constants <wink> during the gamma phase. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLanoHEjvBPtnXfVAQJB6gP/SwVtejenN0/7tszePbJ4O20l98k2Z/7N rs9dfF2+Jcy/OLzSCcW/OW4iVf+iPJMWUNqHEPFDQO/+nVifh8pjjWGTQJMc8ynm 7HNCk/ZciZyNqeGbmGvzHoywmrldbDPkx5Y6Fo13yel0slw2Qo6gC6W8aygi7giP boJiovnaKH0= =wRxy -----END PGP SIGNATURE-----

On Thursday 13 July 2006 16:05, Barry Warsaw wrote:
+42 It feels like the release cycle from alpha1 to final has gotten increasingly rushed. While I'm sure some of that is just me having my attention elsewhere, I suspect a longer tail on the cycle to do gammas (or release candidates, or whatever) would definately encourage more testing with applications and the larger frameworks. No, it won't catch everything, but I think it would help. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed. Same here. I believe there was some shortening of the 2.5 release cycle two or three months ago. I don't recall why or by how much, but I think the acceleration has resulted in a lot of the "can't we please squeeze this one little change in?" that's been happening. Shortening a micro release a bit seems reasonably easy to accommodate, but since minor releases occur so infrequently, I think it would be better to stretch them out if necessary. Skip

On Friday 14 July 2006 00:32, skip@pobox.com wrote:
The squeezing of the releases isn't where the problem is, I think. It's that, once squeezed, more releases aren't being added to compensate. We really need to determine what time we need to go from beta1 to (gamma|rc)1, and then from (gamma|rc)1 to final. Plenty of interim releases in the beta phase is good. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

>> I believe there was some shortening of the 2.5 release cycle two or >> three months ago. Fred> The squeezing of the releases isn't where the problem is, I think. Had we been in alpha a bit longer (officially before feature freeze) I think there'd have been less back-and-forth about a couple otherwise noncontroversial checkins. Skip

On 7/13/06, skip@pobox.com <skip@pobox.com> wrote:
Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed.
I think that's just because you are getting older and time goes by faster the less time you've got left. :-) It seems to be going quite quick, but I'm not so sure it's really different.
Not exactly. The PEP was created Feb 7 (r42266) with a very relaxed schedule to allow enough time for ssize_t branch to solidify. Martin was ready sooner, so on Feb 26 (r42577) sped up the schedule to something more reasonable: - alpha 1: April 1, 2006 [planned] - alpha 2: April 29, 2006 [planned] - beta 1: June 24, 2006 [planned] - beta 2: July 15, 2006 [planned] - rc 1: August 5, 2006 [planned] - final: August 19, 2006 [planned] The current schedule is: + alpha 1: April 5, 2006 [completed] + alpha 2: April 27, 2006 [completed] + beta 1: June 20, 2006 [completed] + beta 2: July 11, 2006 [completed] + rc 1: August 1, 2006 [planned] + final: August 8, 2006 [planned] So beta1 was 4 days earlier than the date set at the end of Feb. The first beta was about 19 months after 2.4 was released if I'm counting correctly. In 2.4 beta1 to release was 1.5 months. For 2.5 it is about the same. The total 2.4 schedule (from a1 to release) was 4.5 months which included 3 alphas. For 2.5 we only had 2 alphas and the total schedule is just over 4 months. So 2.5 is just a little faster. But 2.5 has had a ton more testing than any previous released due to the buildbots, not to mention Coverity and now Klocwork. I definitely pushed the schedule, but I don't think it was really all that different from in the past. Given that several people here think we should lengthen the schedule in some way, I suspect we will do something. I'm not really against it, but I don't think it will provide much benefit either. A few more bugs will be fixed since we have more time. What I think might really make a difference if we could leverage the buildbots and copy the builds (especially WIndows) up to python.org and make them available for download. That might make it easier for people to try stuff out. n

Neal Norwitz wrote:
you're still missing the point of this thread: it isn't bugs in the core Python distribution that's the problem, it's compatibility with third- party modules and applications. I'm quite sure that we could ship the current beta2 as 2.5 final and have a Python release that, in itself, is more stable than all the ones before it, but since 2.5 introduces lots of new stuff, that stability doesn't necessarily translate to a better experience for people who wants to use their existing applications and libraries. a longer beta period gives *external* developers more time to catch up, and results in less work for the end users. </F>

On 7/13/06, Fredrik Lundh <fredrik@pythonware.com> wrote:
...
a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.
This is the part I don't get. For the external developers, if they care about compatibility, why aren't they testing periodically, regardless of alpha/beta releases? How often is the python build broken or otherwise unusable? On Unix there's no more or less work to grab a tarball whether it's a nightly or a release. For Windows it's definitely easier to wait for a release. If there was an installable windows version made by the buildbots, would that mean there would be more testing? Is part of your point that these developers only care about something called "release" and they won't start testing before then? If that's the case why don't we start making (semi-)automated alpha releases every month? That will give people lots more time. Somehow I don't think it will make a difference. People mostly work on schedules or should I say avoid schedules. Give them 5 months and they will still wait to the last minute. Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments). n

>> This is the part I don't get. For the external developers, if they >> care about compatibility, why aren't they testing periodically, >> regardless of alpha/beta releases? Fredrik> because they don't want to waste time and effort on testing Fredrik> against releases that are not necessarily a close approximation Fredrik> of the full release? testing takes time, and time can be used Fredrik> for many different things. How is that a waste of time? You've got the latest stable 2.4 installed I presume. Do the svn-up/make dance (at least on Unix-y systems - can it be much more difficult on Windows?). If it breaks something, either figure out why or drop back to 2.4 for a week or two and try again. If application breakage remains, investigate. Skip

On Friday 14 July 2006 16:39, Neal Norwitz wrote:
Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
I think this is where release candidates can come into their own - the beta's can flush out the "just-one-more-thing" pleases, the maintenance branch is forked for rc1, and then any changes on the branch are purely to fix regressions. As far as the idea of a suite of buildbots running the unit tests of large Python applications against the current SVN trunk goes, one problem is that failures in those buildbots will come from two sources: - Python itself fails (broken checkin) - the application unit tests fail (backwards incompatibility) To weed out the false alarms, the slaves will need to be set up to run the Python unit tests first and then the application unit tests only if the Python test run is successful. When the slave suffers a real failure due to a backwards incompatibility, it will take a developer of the application to figure out what it was that broke the application's tests. So while I think it's a great idea, I also think it will need significant support from the application developers in debugging any buildbot failures to really make it work. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 7/14/06, Nick Coghlan <ncoghlan@gmail.com> wrote:
These buildbots should run the tests from stable, released versions of external packages, assuming that those packages don't ship releases with failing tests. If you ran the test suite for a Zope release and had a test failure, I think there would be a reasonable expectation that it was a Python bug. I'd definitely avoiding mixing a development version of Python and a development version of the test software in the automated build environment. Jeremy

Jeremy Hylton wrote:
Definitely, but there's a difference between "bug that broke Python's own unit tests" and "change to a behaviour that package X depended on". It's the latter cases that the external buildbots would be looking for - and while some of those will be shallow enough that the regression is obvious from the unit test error message and the recent Python checkins, the non-obvious ones will require a collaborative resolution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 7/14/06, Anthony Baxter <anthony@interlink.com.au> wrote:
Maybe the basic question is right, but the emphasis needs to be changed. If we had a rule that said the final release was 90 days after the last submission that wasn't to fix a regression, we'd ask "Is this feature important enough to warrant delaying the release until three months from now?" I'm not sure what I think, but it doesn't seem like an implausible policy. Jeremy

On Friday 14 July 2006 22:45, Jeremy Hylton wrote:
I really really doubt that this would work. There's always pressure for "just one more feature" - and as the release drags on, this would increase, not decrease. Note that dragging the release process out has it's own costs, as mentioned previously - either the trunk is in some sort of near-frozen state for an extended period, or else we end up in the double-applying-bugfixes state by forking much earlier. This approach would also make it extremely difficult to plan for releases. I know that my free time varies across the course of the year, and I need _some_ sort of plan for when the release will happen so I can make sure I have the time free to spend on it. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
On the face of it, it seems to me that branching a new major release at the 1st beta would be one way of managing this. The trunk is not frozen for an extended period, and any "features" and bug fixes could probably be backported from the trunk on clearance from the RE with no more pain than is currently endured. One down side would be the possibility of a loss in emphasis in finishing the job on the release to be. I'm sure there are others... ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

On Monday 17 July 2006 20:27, Andrew MacIntyre wrote:
The major one is the doubling of the workload for bugfixes. SVN is better than CVS, but it's still not great when it comes to backporting fixes between branches. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Neal Norwitz <nnorwitz@gmail.com> wrote:
Because it is a cost, and I don't want to waste my time if the trunk is unstable or whatnot. There is no official statement of the kind "all the real development is done in branches, the trunk is always very stable, feel free to grab it". Thus, we (external developers) assume that it's better to wait for the first alpha or beta before starting doing any testing. Personally, I won't touch something before the first beta: there are already enough known bugs when the alphas are shipped that I prefer to wait a little bit more. In my case, the beta period is too short. It's already a great luck if, during the beta cycle, I'm not deep into some milestone or release cycle for my own software. It might well happen that I have barely time to notice a Python beta is out, but I can't afford spending any time on it because of time constraint. By the time I'm done with my own deadlines, Python final is already out. But for the sake of the argument and the rest of this mail, let's assume I have tons of spare time to dedicate to Python 2.5 beta testing. My applications have several external dependencies (I wouldn't classify those as "many", but still around 6-7 libraries in average). For each of those libraries, there won't be Py2.5-ready RPM or Windows installer to grab and build. Most of the time, I have to download the source package, and try to build it. This also means hunting down and fixing Py2.5-related bugs in all those libraries *before* I can even boot my own application (which is what I would care about). Then I occasionally hit a core Python bug while doing so, which is good, but I have to sit and wait. Some libraries have very complex build process which are still distutils-based, but might require many other external C/C++ libraries, which need to be fetched and compiled just to setup the build environment. Alternatively, I could cross my fingers and wait for all the maintainers of the external libraries to have spare time, and dedicate to Python 2.5 upgrading. If I'm lucky, by the time RC1 is out, most of them might have binary packages available for download, or at least have their (unstable) CVS/SVN trunk fixed for Python 2.5 (which means that I'll have to fetch that unstable version and basically perform a forced upgrade of the library, which might trigger another class of compatibility/feature/regression bugs in my application, not related at all to Python 2.5 by itself, but still needed to be dealt). So I think it's useless to ask people to rush testing beta releases: it takes months to get the community synched, and will always take. It's also useless to point the finger to external developers if they don't test Python and there is a bug in a .0 release. Bugs happen. It's software that evolves. My own suggestion is that it's useless to stall a release for months to give external developers time to fix things. I think it's much better to "release early - release often", and just get the damn release out of the door. Usually, it's at that point that the whole community start working on it, and discover bugs. And so what? For production usage, .0 releases of libraries (let alone the interpreter of the language) are a no-go for all software, and I know that for a long time already. I don't ship an application of mine with a Python .0 release no matter what, no matter even if the whole testsuite passes and everything seems to work. I don't have enough benefits for the risks, so I'll wait for .1 or .2 release anyway. It's *very* fine by me if .0 release is actually the first "official" "beta" release for the community, when the core developers say "we're done" and external developers really start get things going. If you really care about .0 stability from a purely commercial point-of-view (it's bad advertisement if many people complain if a major release is broken, etc. etc.), I might suggest you to coordinate with a small group of selected external developers maintaing the larger external packages (Twisted and others). You could put a list of those packages in the release PEP as showstoppers for the release: you should not get a .0 out if those packages are broken. I think this could help smooth out the process. If important external libraries work, many applications relying on it will *mostly* work as well. I personally don't think it's such a big problem if one has to fix a couple of things in a 100K-line application to adjust it to the new .0 release, even if it's actually because of a bug in Python itself. Giovanni Bajo

On Fri, Jul 14, 2006 at 12:00:07PM +0200, Giovanni Bajo wrote:
Where could we put such a statement? http://www.python.org/dev/tools/, in a discussion of checkin policies, does say: The Python source tree is managed for stability, meaning that if you make a checkout at a random point in time the tree will almost always compile and be quite stable. Large sets of changes, such as the 2.2 type/class rewrite, are usually tested on a branch first and merged into the main stream of development once they're believed to be complete. but this is buried pretty deeply. Maybe some text should be added to the release announcements about this.
And many people won't make binary packages until 2.5 is final, so you're probably not often lucky. --amk

"A.M. Kuchling" <amk@amk.ca> wrote in message news:20060714112137.GA891@Andrew-iBook2.local...
That is the goal, but when I watched the buildbot results last spring, the degree of stability (greenness) appeared to vary. Is it possible to tag particular versions as a 'green' version, or the 'most recent green version' worth playing with? tjr

On 7/15/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Right. It's very rare that the interpreter or stdlib is broken. There are 19 buildbots currently. 7 are not green. 5 of those are offline, most with known (to me and the person that donated the machines) hardware issues ranging from lack of air conditioning, to a bad router, to no power. 1 is cygwin which is running an old version. I just (an hour or so ago) got an account on a cygwin box to help diagnose the status and figure out if anything within Python or the tests are broken. That leaves 1 unexplained failure on a Windows bot. This started happening recently after being down for a while. I haven't had time to investigate. The reason why it was not very green in spring was because that's when we were getting it set up! The master looks like it was installed at the end of 2005/beginning of 2006. It took several months to get rid of many testing issues. Tests that couldn't be run in a particular order, tests that couldn't be run simultaneously (socket), bad compilation of sqlite/bsddb modules (not in python), misconfigured systems, tests verifying something that was system dependent, signed addresses, etc. Of all the problems there were, I only remember a single problem in Python. (There were probably more, I just remember one.) That was in test code (xxmodule or something like that. There were a bunch of problems with ctypes and/or sqlite when they got integrated having to deal with these new platforms. That may be what you are recalling. Right before alpha 2 was a particularly bad time. What we mean by stable is that at any time/any stage of development, if a change goes in that breaks the tests, it is likely to be fixed or reverted within hours. The amount of time the build or tests are broken on the majority of platforms is quite small. It took a while to get to this point due to a bunch of flaky tests, but those have mostly been fixed. The only known problems are mentioned on the wiki. When the buildbots fail, we get mail to python-checkins. Unfortunately, that's mostly spam at this point. I hope to fix that at some point. I also hope to change the main page to give more info in less space. n

[Neal Norwitz]
... That leaves 1 unexplained failure on a Windows bot.
It wasn't my Windows bot, but I believe test_profile has failed (rarely) on several of the bots, and in the same (or very similar) way. Note that the failure went away on the Windows bot in question the next time the tests were run on it. That's typical of test_profile failures. Unfortunately, because test_profile works by comparing stdout against a canned expected-output file under Lib/test/output/, when we re-run the test in verbose mode at the end, that comparison isn't done, so there's isn't a simple way to know whether the test passed or failed during its second run. I _bet_ it actually passed, but don't know. This problem is obscured because when you run regrtest.py "by hand" with -v, you get this message at the end: """ CAUTION: stdout isn't compared in verbose mode: a test that passes in verbose mode may fail without it. """ which is intended to alert you to the possible problem. But when we re-run failing tests in verbose mode "by magic" (via passing -w), that warning isn't produced. BTW, the best solution to this is to convert all the tests away from regrtest's original "compare stdout against a canned output/TEST_NAME file" scheme. That won't fix test_profile, but would make it less of a puzzle to figure out what's wrong with it.

At the risk of waking up a thread that was already declared dead, but perhaps this is usefull. So, what happens is pythons signal handler sets a flag and registrers a callback. Then the main thread should check the flag and make the callback to actually do something with the signal. However the main thread is blocked in GTK and can't check the flag. Nick Maclaren wrote: ...lots of reasons why you can't do anything reliably from within a signal handler... As far as I understand it, what could work is this: -PyGTK registrers a callback. -Pythons signal handler does not change at all. -All threads that run in the Python interpreter occasionally check the flag which the signal handler sets, like the main thread does nowadays. If it is set, the thread calls PyGTKs callback. It does not do anything else with the signal. -PyGTKs callback wakes up the main thread, which actually handles the signal just like it does now. PyGTKs callback could be called from any thread, but it would be called in a normal context, not in a signal handler. As the signal handler does not change, the risk of breaking anything or causing chaos is as large/small as it is under the current scheme. However, PyGTKs problem does get solved, as long as there is _a_ thread that returns to the interpreter within some timeframe. It seems plausible that this will happen.

On 9/8/06, Jan Kanis <jan-python@maka.demon.nl> wrote:
At the risk of waking up a thread that was already declared dead, but perhaps this is usefull.
I don't think we should let this die, at least not yet. Nick seems to be arguing that ANY signal handler is prone to random crashes or corruption (due to bugs). However, we already have a signal handler, so we should already be exposed to the random crashes/corruption. If we're going to rely on signal handling being correct then I think we should also rely on write() being correct. Note that I'm not suggesting an API that allows arbitrary signal handlers, but rather one that calls write() on an array of prepared file descriptors (ignoring errors). Ensuring modifications to that array are atomic would be tricky, but I think it would be doable if we use a read-copy-update approach (with two alternating signal handler functions). Not sure how to ensure there's no currently running signal handlers in another thread though. Maybe have to rip the atomic read/write stuff out of the Linux sources to ensure it's *always* defined behavior. Looking into the existing signalmodule.c, I see no attempts to ensure atomic access to the Handlers data structure. Is the current code broken, at least on non-x86 platforms? -- Adam Olsen, aka Rhamphoryncus

On 9/9/06, Adam Olsen <rhamph@gmail.com> wrote:
So, no, this is certainly not the same as linux kernel atomic operations, which allow you to do more interesting stuff like, test-and-clear, or decrement-and-test atomically. glib has those too, and so does mozilla's NSPR, but only on a few architectures does it do it without using mutexes. for instance, i686 onwards don't require mutexes, only special instructions, but i386 requires mutexes. And we all know mutexes in signal handlers cause deadlocks :-( And, yes, Py_AddPendingCall and Py_MakePendingCalls are most certainly not async safe! Just look at the source code of Py_MakePendingCalls and you'll see an interesting comment... Therefore, discussions about signal safety in whatever new API we may add to Python should be taken with a grain of salt. Regards.

On 9/9/06, Jan Kanis <jan-python@maka.demon.nl> wrote:
No, it is not plausible at all. For instance, the GnomeVFS library usually has a pool of thread, not doing anything, waiting for some VFS task. It is likely that a signal will be delivered to one of these threads, which know nothing about Python, and sit idle most of the time. Regards.

On Sat, 09 Sep 2006 12:59:23 +0200, Gustavo Carneiro <gjcarneiro@gmail.com> wrote:
Well, perhaps it isn't plausible in all cases. However, it is dependant on the libraries you're using and debuggable, which broken signal handlers apparently aren't. The approach would work if you don't use libraries that block threads, and if the libraries that do, co-operate with the interpreter. Open source libraries can be made to co-operate, and if you don't have the source and a library doesn't work correctly, all bets are off anyway. But having the signal handler itself write to a pipe seems to be a cleaner solution, if it can work reliable enough for some value of 'reliable'. Jan

Neal> How often is the python build broken or otherwise unusable? Not very often. I use the head branch of the repository as the source of the interpreter I run on my laptop. It generally takes just a couple minutes on my now-aging PowerBook to svn up and reinstall. I can only recall one time in the past where I had to temporarily fall back to 2.4 because of some change that broke an application. Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I suspect that my usage of the language isn't all that different from most Python developers out there. Neal> Is part of your point that these developers only care about Neal> something called "release" and they won't start testing before Neal> then? If that's the case why don't we start making Neal> (semi-)automated alpha releases every month? How would that be any easier than a user setting up a read-only repository and svn-up-ing it once a month then using that as the default interpreter on that person's development machine? I maintain interpreters for 2.3, 2.4 and bleeding edge at the moment. If I need to it's fairly trivial (a symlink change) to fall back to the latest stable release. Glyph, would that sort of scheme work for you? Skip

On Fri, 14 Jul 2006 06:46:55 -0500, skip@pobox.com wrote:
Neal> How often is the python build broken or otherwise unusable?
Not very often.
I have to agree. The effort I'm talking about is not in fixing large numbers of problems, but simply gearing up to properly test to see if there *are* problems. Keep in mind though: just because the problems are small or easy to fix doesn't mean they're not severe. One tiny bug can prevent a program from even starting up.
A huge percentage of Python developers are working with Zope, which means that although *their* code might not be terribly "sophisticated", it is nevertheless sitting on top of a rather large and intricate pile of implicit dependencies on interpreter behavior.
Glyph, would that sort of scheme work for you?
No. I really think you're underestimating the sort of effort required to upgrade Python. First of all, I do have a job :) and it'd be very hard to make the case to an investor that it was worth tracking down every potential bug I had to determine whether it was a problem because I was working with an unreleased version of Python. This is the same reason I don't use beta versions of the kernel or libc for development. For that matter, I've had to avoid posting to this mailing list because even *this* is stretching my already overburdened schedule :). Secondly, I have to work with a few extension modules. To name a few: * ctypes * PyPAM * pysqlite2 * zope.interface (C optimizations) * xapian * pylucene * dspam * PyOpenSSL * PIL * PyCrypto * pyexpat * pygtk * pyvte Recompiling all of these is a project that takes a day. PyLucene, in fact, I've never managed to build myself, and I can only run because there happen to be debian packages which work (with some fiddling) on Ubuntu. There's no interactive warning during 'svn up' that it's time to recompile everything, either, so I don't even know when the ABIs are going to have changed. Even if everything works perfectly, and were perfectly automated, the compile itself would take a few hours. I also test work on Windows on occasion and recompiling these things _there_ is the work of a week and a half, not to mention it requires that I be sitting at the one machine where I have my Microsoft™ Developer™ Tools™ installed. I made the buildbot recommendation specifically because it would centralize the not inconsiderable effort of integrating these numerous dependencies (presuming that I could submit a Divmod buildbot as well as a Twisted one).

On Thu, 13 Jul 2006 23:39:06 -0700, Neal Norwitz <nnorwitz@gmail.com> wrote:
On 7/13/06, Fredrik Lundh <fredrik@pythonware.com> wrote:
a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.
How often do you test new builds of Python against the most recent alpha of, e.g. the Linux kernel? This isn't just a hypothetical question: Twisted has broken because of changes to Linux as often as it has broken due to changes in Python :). In Linux's case we're all lucky because *any* regressions with existing software are considered bugs, whereas in Python's case, some breakagaes are considered acceptable since it's more feasible to have multiple builds of Python installed more than multiple kernels for different applications.

Fred L. Drake, Jr. wrote:
It feels like the release cycle from alpha1 to final has gotten increasingly rushed.
python 2.2: ~5 months python 2.3: ~7 months python 2.4: ~5 months python 2.5: ~4 months I think the biggest problem is the time between "basically stable beta" and "final"; it was ~90 days in 2.3 and is now ~50 days, most of which is vacation time in my part of the world. we could extend the beta period by playing silly games with naming, but as long as the "big things" go in *before* the beta period starts, I think it's easier to simply extend the beta period by adding another month to it. I'd prefer something like 90 days. </F>

Barry Warsaw wrote:
Maybe there could be an "unstable" release phase that lasts for a whole release cycle. So you'd first release version 2.n as "unstable", and keep 2.(n-1) as the current "stable" release. Then when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would become the new "unstable". (Feel free to substitute other terms for "stable" and "unstable" if you don't like them.) That would give people plenty of warning and time to try things out with the new version. This wouldn't actually be much different to what is done now, except for the naming. But by not officially blessing the latest release as current for a while, it might give less of an impression that stuff is being sprung on the community unawares.
Well, at least they've had a chance to try it. If they don't take that chance, they don't have much ground for complaint. -- Greg

Greg> Maybe there could be an "unstable" release phase that lasts for a Greg> whole release cycle. So you'd first release version 2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable" release. Then Greg> when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would Greg> become the new "unstable". In GCC don't they do an odd (stable)/even (unstable) release schedule? Same for Linux kernels? Would that help? Skip

skip@pobox.com wrote:
No. Linux kernel releases are more aggressive because of the fact that all the patches are mostly developed in different branch/repositories and get merged when they are already well tested and incorporated. Linus can merge literally hundreds of patches daily into his *stable* tree, and do releases from it even weekly, because most destabilizing works are being done in large branches carried on for months before they even are evaluated for being merged; or because patches were settled in the -mm tree for months. Linus' tree is kind-of a release branch, with the difference that he is the BDFL and does what he wants with his tree :) To keep this into perspective, remember also that they don't have *any* kind of testsuite (nor a debugger, if I might say). GCC has a more "old-fashioned" release process, where the trunk evolves through 3 stages: Stage 1 is open for all kind of changes (eg: from simple polishing/refactoring, to merging of large branches containing work of several man-years). Stage 2 is still open for new features, but not for big merges. Stage 3 is feature-freezed, bug-fixing only. Then, the trunk is branched into the new release branch, and the trunk gets back to Stage 1. Nominally, a stage lasts 3 months, but Stage 3 often stretches up to 6 months. The release branches are open for *only* regression fixes (that is, fixes that correct things that used to work in previous releases but do not work anymore). Any regression fix (with a corresponding Bugzilla entry, where it's marked and confirmed as "regression") is applied to trunk *and* the open release branches where the regression exists. For convoluted or dangerous regression fixes, usually maintainers prefer to wait 1 week for the patch to settle down on the trunk before applying it to the release branches. The release manager pulls dot releases from the release branch. Usually, the first release (.0) happens 3-4 months *after* the release branch was created, that is after several months of regression-only patches being merged to it (while new work is being done on the trunk in parallel, in its aggressive Stage 1). The 3-Stage work in the trunk is streamlined by the release manager. At the beginning of Stage 1, a detailed techinical list of on-going "projects" (new features) is presented to the mailing list, explaining the current status of the work and its ETA, and the release manager then publishes a work-plan for Stage 1 and 2, telling which projects will be merged when. This avoids multiple large projects to hit the trunk at the same time, causing may headaches to all the other developers. The work-plan is managed and updated in the GCC Wiki (which is off-line right now, but I'll post a link as example when it's back). -- Giovanni Bajo

On Friday 14 July 2006 06:05, Barry Warsaw wrote:
alpha 1: April 5, 2006 [completed] alpha 2: April 27, 2006 [completed] beta 1: June 20, 2006 [completed] beta 2: July 11, 2006 [completed] rc 1: August 1, 2006 [planned] final: August 8, 2006 [planned] Four months would seem to me to be quite long enough as a release cycle. Extending it means far more work for everyone - either we have special checkin rules for the trunk for a longer period of time (requiring extra monitoring by people like myself and Neal), or we branch earlier, requiring double commits to the trunk and the branch for bugfixes. I also strongly doubt that making a longer "release candidate" cycle would lead to any significant additional testing by end-users. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

FWIW, I tend to run a few project(*) test suites when doing minor releases (2.x.y), just to make sure I don't break things. For major releases, it's a fair bit more work - something like Twisted or Zope3 play at such a low level with the Python interfaces that there's nearly always breakages or massive warning dumps (from memory, zope.interface uses 'with' as an argument name, a lot). The "community buildbot" idea is a good one, although it should just be possible to write something for buildbot that checks out and builds the latest Python SVN, then installs it to a temporary location, then adds that location to the front of the path. Then each project could just add a "Python SVN buildbot" to their exists bbot install. Anthony (*) Typically includes Zope3 and Twisted, because I have these just lying around.

On 7/14/06, Anthony Baxter <anthony@interlink.com.au> wrote:
This is what Xenofarm for Pike has done for a long time now. See for example: http://pike.ida.liu.se/development/pikefarm/7.7.xml This is also what Bitten (http://bitten.cmlenz.net/) has implemented for Trac (which is one of the bug/incident trackers for Python's call for trackers). -- Jeroen Ruigrok van der Werven

Jeroen Ruigrok van der Werven wrote:
Are you aware of http://www.python.org/dev/buildbot/ ? We are not just talking about buildbots here (which the links you refer to seem to be); we are talking about buildbots that don't test Python, but test Python applications. We do know how to run a buildbot. Regards, Martin

On 7/15/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Are you aware of http://www.python.org/dev/buildbot/ ?
Yes. And it does not seem to be open for all, but then again, any documentation with regard to it seems to be very sparse or hidden so I can very well be wrong here. Ah, hidden away on the wiki: http://wiki.python.org/moin/BuildBot
Then I would dare to say you haven't fully investigated the links fully, Bitten, for example, also runs the unit-tests for any target you configure, saves all the outputs and provides quick overviews as well as coverage statistics. So that would fall perfectly into that category. See http://bitten.cmlenz.net/build/0.5.x/762 for an example.
We do know how to run a buildbot.
How relevant was this comment in the entire? I am merely trying to help out here pointing out other similar projects when the community participating building issue was raised. -- Jeroen Ruigrok van der Werven

Jeroen Ruigrok van der Werven wrote:
Are you aware of http://www.python.org/dev/buildbot/ ?
Yes. And it does not seem to be open for all
Ah, ok. It indeed isn't open for anonymous participation; the test results are open for all, though.
I don't understand that comment. Bitten seems to be a software package, similar to buildbot. It doesn't do anything useful until I install and configure it, unlike, say, SourceForge, which is not (just) a soft package, but a running installation of it also. Right? The effort is in installing and configuring it, given that many such packages are already written. Distributed testing frameworks typically support running arbitrary target comments, so Bitten is not really special here (buildbot can also run about anything if you configure it to).
It would have been helpful if you had stated *why* you think these URLs are relevant in the context of the discussion. To me, it seemed you are pointing to the existence of distributed testing frameworks. I was pointing out that we (at least, I) am aware of the existence of such frameworks, and that we were doing it for quite some time now also, just like Pike. Regards, Martin

No real time to respond in detail here, but one small comment. glyph@divmod.com writes:
Remember these words...
When implementing this stuff, I could have merely made it possible for exceptions to be new-style, and left the builtin exceptions as classic classes. This didn't seem to be an especially good idea, as it would leave code working _most_ of the time, only to break horribly when confronted with a rare new-style exception. So I made the decision (and I don't think I ever got around to explicitly discussing this with anyone, nor if the people who actually updated my patch and checked it in thought about it at all) to make the built in exceptions new-style, precisely to make it a screamingly obvious change. I didn't _know_ when I was doing it that I'd break Twisted unit tests, but I was hardly a surprise. I think the idea of a buildbot running various projects tests suites with svn HEAD is a great idea -- back when I was doing 2.2.1 I did run a few third party test suites (zodb comes to mind) but as always with these things, automation is a good idea. Cheers, mwh --

On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson <mwh@python.net> wrote:
glyph@divmod.com writes:
To be clear, I agree with the decision you made in this particular case. I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen. (Of course I *can* always do that, and some other Twisted devs watch python-dev a bit more closely than I do, but the point is that the amount of effort required to do this is prohibitive for the average Python hacker, whereas the time to set up an individual buildbot might not be.) (Aside: IMHO, the sooner we can drop old-style classes entirely, the better. That is one bumpy Python upgrade process that I will be _very_ happy to do. There's no way to have documentation that expresses the requirement that an implementation of an interface be new-style or old-style without reference to numerous historical accidents, which are bewildering and upsetting to people reading documentation for the first time.)

On 7/13/06, glyph@divmod.com <glyph@divmod.com> wrote:
One way to try to track python-dev is the python-dev Summaries. While Steven is only human and thus cannot always get them out immediately, they do happen frequently enough that major decisions will be covered before betas are hit. -Brett (Aside: IMHO, the sooner we can drop old-style classes entirely, the better.

glyph@divmod.com wrote:
I think python should have a couple more of future imports. "from __future__ import new_classes" and "from __future__ import unicode_literals" would be really welcome, and would smooth the Py3k migration process -- Giovanni Bajo

On 7/13/06, Giovanni Bajo <rasky@develer.com> wrote:
Along similar lines as glyph, after complaining about this for a long time "offline" with my friends, I decided it's probably a good idea to get involved and vote that the __future__ import for unicode_literals be implemented. python -U is a failure for obvious reasons and a __future__ import is clearly better. Does anyone want to pair on this? -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. Löwis\"" <martin@v.loewis.de> wrote:
I am surprised that you do, since I thought that Chris's conclusion was pretty obvious. Python -U doesn't work, even on the standard library. For example, glyph@legion:~% python -S -U -c 'import pickletools' Traceback (most recent call last): File "<string>", line 1, in ? File "/usr/lib/python2.4/pickletools.py", line 219, in ? doc="One-byte unsigned integer.") File "/usr/lib/python2.4/pickletools.py", line 187, in __init__ assert isinstance(name, str) AssertionError This was just the first convenient example. There are others. A __future__ import would allow these behaviors to be upgraded module-by-module. Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is.

glyph@divmod.com wrote:
A __future__ import would allow these behaviors to be upgraded module-by-module.
No it wouldn't. __future__ works solely on the semantics of different pieces of syntax, because any syntax changes are purely local to the current module. This doesn't work for datatypes, because data types can cross module boundaries - other modules may still be using the old behaviour, and there's nothing the current module can do about it. Changing all the literals in a module to be unicode instances instead of str instances is merely scratching the surface of the problem - such a module would still cause severe problems for any non-Unicode aware applications that expected it to return strings.
Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is.
There's never been a concerted effort to make even the standard library work under -U. Maybe that should be a goal for Python 2.6 - figure out what tools or changes are needed to make code -U safe, add them, and then update the standard library to use them. PEP 349 (currently deferred, although I don't recall why) discusses some of the issues involved in making code unicode-safe, while still supporting non-unicode safe applications as clients. Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0349/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes it would! :)
A module with the given __future__ import could be written to expect that literals are unicode instances instead of str, and encode them appropriately when passing to modules that expect str. This doesn't solve the problem, but unlike -U, you can make fixes which will work persistently without having to treat the type of every string literal as unknown. The obvious way to write code that works under -U and still works in normal Python is to .encode('charmap') every value intended to be an octet, and put 'u' in front of every string intended to be unicode. That would seem to defeat the purpose of changing the default literal type.

glyph@divmod.com wrote:
Think of it this way: with the u'' literal prefix you can upgrade the behavior on a per-literal basis ;-) Seriously, it really does makes more sense to upgrade to Unicode on a per-literal basis than on a per-module or per- process basis. When upgrading to Unicode you have to carefully consider the path each changed object will take through the code. Note that it also helps setting the default encoding to 'unknown'. That way you disable the coercion of strings to Unicode and all the places where this implicit conversion takes place crop up, allowing you to take proper action (i.e. explicit conversion or changing of the string to Unicode as appropriate). Later on, you can then simply remove the u'' prefix when moving to Py3k. In fact, Py3k could simply ignore the prefix for you (and maybe generate an optional warning) to make the transition easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006)
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote:
I've tried that before to verify no such conversion issues occurred in Twisted, but, as the python stdlib isn't usable like that, it's hard to use it to find bugs in any other libraries. (in particular, the re module is badly broken, some other stuff was too). James

James Y Knight wrote:
True: it breaks a lot of code that was written to work with both strings and Unicode (or does so by accident ;-). The stdlib isn't too well prepared for Unicode yet, so if your code relies a lot on it, then the above may not be the right strategy for you. Perhaps a new ASCII codec that issues warnings for all these cases would help ?! (one that still converts to Unicode assuming ASCII, but issues a warning pointing to the location in the code where the conversion happend) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006)
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

glyph@divmod.com wrote:
Such changes might have to be reverted in Python 3, since the module might then expect character strings instead of byte strings, and then might complain when it gets byte strings (which .encode will produce). So declaring that all string literals are Unicode objects might not help in the future, contrary to what the future import suggests.
The not so obvious way is to change the modules/methods receiving these strings to work with either string type if that is reasonable. Regards, Martin

glyph@divmod.com wrote:
Sure, it doesn't work. That doesn't mean it's "a failure". It just hasn't been completed yet.
People can use it to improve the Unicode support in the Python standard library. When they find that something doesn't work, they can study the problem, and ponder possible solutions. Then, they can contribute patches. -U has worked fine for me in the past, I contributed various patches to make it work better. It hasn't failed for me at all. Regards, Martin

On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. Löwis\"" <martin@v.loewis.de> wrote:
I guess it makes more sense as a development tool to work on zero-dependency tools like the standard library. Still, -Q has both a __future__ import and a command-line option, why not -U?

Bob Ippolito wrote:
Although it's not a very obvious spelling, particularly to the casual reader who may not be familiar with the intricacies of classes and metaclasses. I don't think it would hurt to have it available as a __future__ import as well. There's also the advantage that all of a module's future assumptions could then be documented uniformly in one place, i.e. in a __future__ import at the top. -- Greg

Talin wrote:
Actually - can we make new-style classes the default, but allow a way to switch to old-style classes if needed?
That sounds dangerously like a "from __past__" kind of feature, and Guido has said that there will never be a __past__ module. Also, this is probably not something that should be a program-wide setting. -- Greg

On Thu, Jul 13, 2006, Talin wrote:
Nope. Python 3.0 will have new-style classes as the default, but there will probably not be any way to use old-style classes. As has been said before, if you want to make new-style classes the default for any module, just put __metaclass__ = type at the top. Please, just drop this subject. Guido has long since Pronounced. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan

glyph> *can* always do that, and some other Twisted devs watch glyph> python-dev a bit more closely than I do, but the point is that glyph> the amount of effort required to do this is prohibitive for the glyph> average Python hacker, whereas the time to set up an individual glyph> buildbot might not be.) If you're concerned about noticing when a new release train is pulling out of the station I think it would be sufficient to subscribe to the low-volume python-announce mailing list. You will see announcements about alphas and betas there. Even lower volume would be a subscription to the RSS feed of Python announcements on the python.org front page (scroll to the bottom). The buildbot idea sounds excellent. Skip

On Thu, 13 Jul 2006 23:27:56 -0500, skip@pobox.com wrote:
The buildbot idea sounds excellent.
Thanks. If someone can set this up, it pretty much addresses my concerns.
I am aware of when new releases come out :). What I'm not aware of is what features (may) have broken my code, and why. As long as Python's trunk@HEAD continues to run the test suites cleanly, I am mostly unconcerned. When it breaks, though, I want a chance to look at the cause of the breakage, *before* there is an alpha or beta Python release out and people are starting to write code that depends on its new features. Most importantly, python-dev is extremely context-sensitive. I want to be able to participate in the discussion when my concerns are relevant and still reasonable to act upon. Additionally I would like to know about these changes so that I can modify code to support Python releases and release a compatible version of Twisted _in advance_ of the Python release that's going to break them, so assuming users are keeping relatively current, there won't be a window where their most recent Twisted release will not work with the most recent Python release.

>> If you're concerned about noticing when a new release train is >> pulling out of the station ... glyph> I am aware of when new releases come out :). What I'm not aware glyph> of is what features (may) have broken my code, and why. Hmmm... Well, you could subscribe to the python-checkins mailing list. Most of the time you'd probably decide rather quickly to delete the messages, but the stuff you care about should be in the checkin comments. Skip

<glyph@divmod.com> wrote in message news:20060714051157.29014.803759518.divmod.quotient.35955@ohm...
Is the following something like what you are suggesting? A Python Application Testing (PAT) machine is set up with buildbot and any needed custom scripts. Sometime after that and after 2.5 is released, when you have a version of, for instance, Twisted that passes its automated test suite when run on 2.5, you send it (or a URL) and an email address to PAT. Other developers do the same. Periodically (once a week?), when PAT is free and a new green development version of either the 2.5.x or 2.6 branches is available, PAT runs the test suites against that version. An email is sent for any that fail, perhaps accompanied by the concatenation of the relevant checkin message. Some possible options are to select just one of the branches for testing, to have more than one stable version being tested, and to receive pass emails. Terry Jan Reedy

On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
Is the following something like what you are suggesting?
Something like it, but...
Sending email also isn't really necessary; I would just like a web page I can look at (and draw the attention of the python core developers to).

<glyph@divmod.com> wrote in message news:20060715181952.29014.1007814129.divmod.quotient.38416@ohm...
If the average rate of checkins is equal to or greater than the maximum rate of test runs then that is physically impossible. (Indeed, queueing theory suggests that keeping the queue size small requires that the checkin be more like half the test rate.) This appears to be currently true for the Python trunk buildbot and I would expect running the test of numerous large applications would take even longer, making the test rate even lower. If you want something like twisted run against multiple Python versions a day, I suspect you will have to do so on a system you control. Terry Jan Reedy

glyph@divmod.com writes:
I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen.
I think something that has happened to some extent with this release is that there was a long-ish period where stuff got discussed and implemented (PEPs 343 and 344, new-style exceptions, the new compiler, the ssize_t branch) and then suddenly we went "crap! we have to release this!" and then the whole alpha/beta/release part has happened much more quickly. Maybe we should have had a more explicit discussion on what was going to be in 2.5 before beta1, but that kind of thing is difficult to make happen (it's entirely possible that it did happen and I just missed it by being snowed under, but I don't think so). Cheers, mwh -- Have you considered downgrading your arrogance to a reasonable level? -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll

On Thu, Jul 13, 2006, glyph@divmod.com wrote:
There's been some recent discussion in the PSF wondering where it would make sense to throw some money to remove grit in the wheels; do you think this is a case where that would help? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes

On 7/13/06, glyph@divmod.com <glyph@divmod.com> wrote:
I'm at least willing to set up the buildbots for projects I care about (Twisted, pydoctor, whatever), and perhaps help out with the setting up the buildmaster. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/

On 7/13/06, Christopher Armstrong <radix@twistedmatrix.com> wrote:
It would be really great to have someone setup buildbots for other important/large python projects to test python svn HEAD with. I can't even stay on top of the python buildbots, so I'm not volunteering for more work. I will greatly appreciate and thank whoever takes this on though. :-) The harder problem is keeping it running. Setting it up, in some ways is a lot easier. It's at least well defined. Maintaining it for several months is a different story. n

On Thu, Jul 13, 2006 at 02:03:22PM -0400, glyph@divmod.com wrote:
An excellent idea!
So don't read threads -- just try the alphas and betas! The "What's New" documents have a 'porting' section that flags changes that may require application changes, but items are added only if I think of them or if someone suggests them (e.g. generator.gi_frame can be None in Python 2.5 -- I would never have thought people would have code that broke because of this).
A 'Blogs' link could be trivially added by linking to planet.python.org, though the blogs collected there are not in any way 'the Python developers', but a jumble of developers and users. I don't think enough core developers have weblogs (or write about Python) to make a 'python-dev only' planet very useful.
I think the venue for this would be weblog entries or news sites such as oreillynet.com. I do worry that we don't have enough people writing articles and things, but have no idea how to encourage people to write more. --amk

"A.M. Kuchling" wrote:
on the other hand, the material you're producing for the "what's new" document would be a rather excellent development blog in itself. just post new sections to the blog when they're ready... (add PEP announcements and python-dev summary items to the mix, and you have a high-quality development blog generated entirely from existing content) (hmm. maybe this could be put together by a robot? time to start hacking on "The Daily URL 2.0 Beta", I suppose...) </F>

On 7/14/06, A.M. Kuchling <amk@amk.ca> wrote:
I believe that's broken at the moment, at least in the sense that it's no longer updated: http://psf.pollenation.net/cgi-bin/trac.cgi/ticket/366 I don't have enough time right now to figure out the new website and how to fix it. Maybe when I get back from Sydney in a couple of weeks. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 13, 2006, at 2:03 PM, glyph@divmod.com wrote:
And I'm glad you did because you made some excellent comments.
I do plan on very soon running Mailman's meager test suite and running more manual functional tests on Python 2.5b2. Compared to Twisted, Mailman is pretty simple and doesn't do that much fancy stuff so I suspect that it'll mostly work, but you never know until you try it. I'd support adding Mailman to a community buildbot army, although I don't personally have the cycles to build out such a beast. I also plan on testing my Real Job's embedded application against Python 2.5b2 sometime this week or next. We're on Python 2.4.2 atm and that is much more complicated because of all the C API we use. I'm less sanguine about breakage there, but I'll post here if I find anything egregious (fortunately, we have an extensive test suite to rely upon).
This really is an excellent point and makes me think that we may want to consider elaborating on the Python release cycle to include a gamma phase or a longer release candidate cycle. OT1H I think there will always be people or projects that won't try anything until the gold release, and that if we've broken anything in their code we simply won't know it until after that, no matter how diligent we are. OTOH, a more formal gamma phase would allow us to say "absolutely no changes are allowed now unless it's to fix backward compatibility". No more sneaking in new sys functions or types module constants <wink> during the gamma phase. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLanoHEjvBPtnXfVAQJB6gP/SwVtejenN0/7tszePbJ4O20l98k2Z/7N rs9dfF2+Jcy/OLzSCcW/OW4iVf+iPJMWUNqHEPFDQO/+nVifh8pjjWGTQJMc8ynm 7HNCk/ZciZyNqeGbmGvzHoywmrldbDPkx5Y6Fo13yel0slw2Qo6gC6W8aygi7giP boJiovnaKH0= =wRxy -----END PGP SIGNATURE-----

On Thursday 13 July 2006 16:05, Barry Warsaw wrote:
+42 It feels like the release cycle from alpha1 to final has gotten increasingly rushed. While I'm sure some of that is just me having my attention elsewhere, I suspect a longer tail on the cycle to do gammas (or release candidates, or whatever) would definately encourage more testing with applications and the larger frameworks. No, it won't catch everything, but I think it would help. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed. Same here. I believe there was some shortening of the 2.5 release cycle two or three months ago. I don't recall why or by how much, but I think the acceleration has resulted in a lot of the "can't we please squeeze this one little change in?" that's been happening. Shortening a micro release a bit seems reasonably easy to accommodate, but since minor releases occur so infrequently, I think it would be better to stretch them out if necessary. Skip

On Friday 14 July 2006 00:32, skip@pobox.com wrote:
The squeezing of the releases isn't where the problem is, I think. It's that, once squeezed, more releases aren't being added to compensate. We really need to determine what time we need to go from beta1 to (gamma|rc)1, and then from (gamma|rc)1 to final. Plenty of interim releases in the beta phase is good. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

>> I believe there was some shortening of the 2.5 release cycle two or >> three months ago. Fred> The squeezing of the releases isn't where the problem is, I think. Had we been in alpha a bit longer (officially before feature freeze) I think there'd have been less back-and-forth about a couple otherwise noncontroversial checkins. Skip

On 7/13/06, skip@pobox.com <skip@pobox.com> wrote:
Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed.
I think that's just because you are getting older and time goes by faster the less time you've got left. :-) It seems to be going quite quick, but I'm not so sure it's really different.
Not exactly. The PEP was created Feb 7 (r42266) with a very relaxed schedule to allow enough time for ssize_t branch to solidify. Martin was ready sooner, so on Feb 26 (r42577) sped up the schedule to something more reasonable: - alpha 1: April 1, 2006 [planned] - alpha 2: April 29, 2006 [planned] - beta 1: June 24, 2006 [planned] - beta 2: July 15, 2006 [planned] - rc 1: August 5, 2006 [planned] - final: August 19, 2006 [planned] The current schedule is: + alpha 1: April 5, 2006 [completed] + alpha 2: April 27, 2006 [completed] + beta 1: June 20, 2006 [completed] + beta 2: July 11, 2006 [completed] + rc 1: August 1, 2006 [planned] + final: August 8, 2006 [planned] So beta1 was 4 days earlier than the date set at the end of Feb. The first beta was about 19 months after 2.4 was released if I'm counting correctly. In 2.4 beta1 to release was 1.5 months. For 2.5 it is about the same. The total 2.4 schedule (from a1 to release) was 4.5 months which included 3 alphas. For 2.5 we only had 2 alphas and the total schedule is just over 4 months. So 2.5 is just a little faster. But 2.5 has had a ton more testing than any previous released due to the buildbots, not to mention Coverity and now Klocwork. I definitely pushed the schedule, but I don't think it was really all that different from in the past. Given that several people here think we should lengthen the schedule in some way, I suspect we will do something. I'm not really against it, but I don't think it will provide much benefit either. A few more bugs will be fixed since we have more time. What I think might really make a difference if we could leverage the buildbots and copy the builds (especially WIndows) up to python.org and make them available for download. That might make it easier for people to try stuff out. n

Neal Norwitz wrote:
you're still missing the point of this thread: it isn't bugs in the core Python distribution that's the problem, it's compatibility with third- party modules and applications. I'm quite sure that we could ship the current beta2 as 2.5 final and have a Python release that, in itself, is more stable than all the ones before it, but since 2.5 introduces lots of new stuff, that stability doesn't necessarily translate to a better experience for people who wants to use their existing applications and libraries. a longer beta period gives *external* developers more time to catch up, and results in less work for the end users. </F>

On 7/13/06, Fredrik Lundh <fredrik@pythonware.com> wrote:
...
a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.
This is the part I don't get. For the external developers, if they care about compatibility, why aren't they testing periodically, regardless of alpha/beta releases? How often is the python build broken or otherwise unusable? On Unix there's no more or less work to grab a tarball whether it's a nightly or a release. For Windows it's definitely easier to wait for a release. If there was an installable windows version made by the buildbots, would that mean there would be more testing? Is part of your point that these developers only care about something called "release" and they won't start testing before then? If that's the case why don't we start making (semi-)automated alpha releases every month? That will give people lots more time. Somehow I don't think it will make a difference. People mostly work on schedules or should I say avoid schedules. Give them 5 months and they will still wait to the last minute. Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments). n

>> This is the part I don't get. For the external developers, if they >> care about compatibility, why aren't they testing periodically, >> regardless of alpha/beta releases? Fredrik> because they don't want to waste time and effort on testing Fredrik> against releases that are not necessarily a close approximation Fredrik> of the full release? testing takes time, and time can be used Fredrik> for many different things. How is that a waste of time? You've got the latest stable 2.4 installed I presume. Do the svn-up/make dance (at least on Unix-y systems - can it be much more difficult on Windows?). If it breaks something, either figure out why or drop back to 2.4 for a week or two and try again. If application breakage remains, investigate. Skip

On Friday 14 July 2006 16:39, Neal Norwitz wrote:
Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
I think this is where release candidates can come into their own - the beta's can flush out the "just-one-more-thing" pleases, the maintenance branch is forked for rc1, and then any changes on the branch are purely to fix regressions. As far as the idea of a suite of buildbots running the unit tests of large Python applications against the current SVN trunk goes, one problem is that failures in those buildbots will come from two sources: - Python itself fails (broken checkin) - the application unit tests fail (backwards incompatibility) To weed out the false alarms, the slaves will need to be set up to run the Python unit tests first and then the application unit tests only if the Python test run is successful. When the slave suffers a real failure due to a backwards incompatibility, it will take a developer of the application to figure out what it was that broke the application's tests. So while I think it's a great idea, I also think it will need significant support from the application developers in debugging any buildbot failures to really make it work. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 7/14/06, Nick Coghlan <ncoghlan@gmail.com> wrote:
These buildbots should run the tests from stable, released versions of external packages, assuming that those packages don't ship releases with failing tests. If you ran the test suite for a Zope release and had a test failure, I think there would be a reasonable expectation that it was a Python bug. I'd definitely avoiding mixing a development version of Python and a development version of the test software in the automated build environment. Jeremy

Jeremy Hylton wrote:
Definitely, but there's a difference between "bug that broke Python's own unit tests" and "change to a behaviour that package X depended on". It's the latter cases that the external buildbots would be looking for - and while some of those will be shallow enough that the regression is obvious from the unit test error message and the recent Python checkins, the non-obvious ones will require a collaborative resolution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On 7/14/06, Anthony Baxter <anthony@interlink.com.au> wrote:
Maybe the basic question is right, but the emphasis needs to be changed. If we had a rule that said the final release was 90 days after the last submission that wasn't to fix a regression, we'd ask "Is this feature important enough to warrant delaying the release until three months from now?" I'm not sure what I think, but it doesn't seem like an implausible policy. Jeremy

On Friday 14 July 2006 22:45, Jeremy Hylton wrote:
I really really doubt that this would work. There's always pressure for "just one more feature" - and as the release drags on, this would increase, not decrease. Note that dragging the release process out has it's own costs, as mentioned previously - either the trunk is in some sort of near-frozen state for an extended period, or else we end up in the double-applying-bugfixes state by forking much earlier. This approach would also make it extremely difficult to plan for releases. I know that my free time varies across the course of the year, and I need _some_ sort of plan for when the release will happen so I can make sure I have the time free to spend on it. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
On the face of it, it seems to me that branching a new major release at the 1st beta would be one way of managing this. The trunk is not frozen for an extended period, and any "features" and bug fixes could probably be backported from the trunk on clearance from the RE with no more pain than is currently endured. One down side would be the possibility of a loss in emphasis in finishing the job on the release to be. I'm sure there are others... ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

On Monday 17 July 2006 20:27, Andrew MacIntyre wrote:
The major one is the doubling of the workload for bugfixes. SVN is better than CVS, but it's still not great when it comes to backporting fixes between branches. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Neal Norwitz <nnorwitz@gmail.com> wrote:
Because it is a cost, and I don't want to waste my time if the trunk is unstable or whatnot. There is no official statement of the kind "all the real development is done in branches, the trunk is always very stable, feel free to grab it". Thus, we (external developers) assume that it's better to wait for the first alpha or beta before starting doing any testing. Personally, I won't touch something before the first beta: there are already enough known bugs when the alphas are shipped that I prefer to wait a little bit more. In my case, the beta period is too short. It's already a great luck if, during the beta cycle, I'm not deep into some milestone or release cycle for my own software. It might well happen that I have barely time to notice a Python beta is out, but I can't afford spending any time on it because of time constraint. By the time I'm done with my own deadlines, Python final is already out. But for the sake of the argument and the rest of this mail, let's assume I have tons of spare time to dedicate to Python 2.5 beta testing. My applications have several external dependencies (I wouldn't classify those as "many", but still around 6-7 libraries in average). For each of those libraries, there won't be Py2.5-ready RPM or Windows installer to grab and build. Most of the time, I have to download the source package, and try to build it. This also means hunting down and fixing Py2.5-related bugs in all those libraries *before* I can even boot my own application (which is what I would care about). Then I occasionally hit a core Python bug while doing so, which is good, but I have to sit and wait. Some libraries have very complex build process which are still distutils-based, but might require many other external C/C++ libraries, which need to be fetched and compiled just to setup the build environment. Alternatively, I could cross my fingers and wait for all the maintainers of the external libraries to have spare time, and dedicate to Python 2.5 upgrading. If I'm lucky, by the time RC1 is out, most of them might have binary packages available for download, or at least have their (unstable) CVS/SVN trunk fixed for Python 2.5 (which means that I'll have to fetch that unstable version and basically perform a forced upgrade of the library, which might trigger another class of compatibility/feature/regression bugs in my application, not related at all to Python 2.5 by itself, but still needed to be dealt). So I think it's useless to ask people to rush testing beta releases: it takes months to get the community synched, and will always take. It's also useless to point the finger to external developers if they don't test Python and there is a bug in a .0 release. Bugs happen. It's software that evolves. My own suggestion is that it's useless to stall a release for months to give external developers time to fix things. I think it's much better to "release early - release often", and just get the damn release out of the door. Usually, it's at that point that the whole community start working on it, and discover bugs. And so what? For production usage, .0 releases of libraries (let alone the interpreter of the language) are a no-go for all software, and I know that for a long time already. I don't ship an application of mine with a Python .0 release no matter what, no matter even if the whole testsuite passes and everything seems to work. I don't have enough benefits for the risks, so I'll wait for .1 or .2 release anyway. It's *very* fine by me if .0 release is actually the first "official" "beta" release for the community, when the core developers say "we're done" and external developers really start get things going. If you really care about .0 stability from a purely commercial point-of-view (it's bad advertisement if many people complain if a major release is broken, etc. etc.), I might suggest you to coordinate with a small group of selected external developers maintaing the larger external packages (Twisted and others). You could put a list of those packages in the release PEP as showstoppers for the release: you should not get a .0 out if those packages are broken. I think this could help smooth out the process. If important external libraries work, many applications relying on it will *mostly* work as well. I personally don't think it's such a big problem if one has to fix a couple of things in a 100K-line application to adjust it to the new .0 release, even if it's actually because of a bug in Python itself. Giovanni Bajo

On Fri, Jul 14, 2006 at 12:00:07PM +0200, Giovanni Bajo wrote:
Where could we put such a statement? http://www.python.org/dev/tools/, in a discussion of checkin policies, does say: The Python source tree is managed for stability, meaning that if you make a checkout at a random point in time the tree will almost always compile and be quite stable. Large sets of changes, such as the 2.2 type/class rewrite, are usually tested on a branch first and merged into the main stream of development once they're believed to be complete. but this is buried pretty deeply. Maybe some text should be added to the release announcements about this.
And many people won't make binary packages until 2.5 is final, so you're probably not often lucky. --amk

"A.M. Kuchling" <amk@amk.ca> wrote in message news:20060714112137.GA891@Andrew-iBook2.local...
That is the goal, but when I watched the buildbot results last spring, the degree of stability (greenness) appeared to vary. Is it possible to tag particular versions as a 'green' version, or the 'most recent green version' worth playing with? tjr

On 7/15/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Right. It's very rare that the interpreter or stdlib is broken. There are 19 buildbots currently. 7 are not green. 5 of those are offline, most with known (to me and the person that donated the machines) hardware issues ranging from lack of air conditioning, to a bad router, to no power. 1 is cygwin which is running an old version. I just (an hour or so ago) got an account on a cygwin box to help diagnose the status and figure out if anything within Python or the tests are broken. That leaves 1 unexplained failure on a Windows bot. This started happening recently after being down for a while. I haven't had time to investigate. The reason why it was not very green in spring was because that's when we were getting it set up! The master looks like it was installed at the end of 2005/beginning of 2006. It took several months to get rid of many testing issues. Tests that couldn't be run in a particular order, tests that couldn't be run simultaneously (socket), bad compilation of sqlite/bsddb modules (not in python), misconfigured systems, tests verifying something that was system dependent, signed addresses, etc. Of all the problems there were, I only remember a single problem in Python. (There were probably more, I just remember one.) That was in test code (xxmodule or something like that. There were a bunch of problems with ctypes and/or sqlite when they got integrated having to deal with these new platforms. That may be what you are recalling. Right before alpha 2 was a particularly bad time. What we mean by stable is that at any time/any stage of development, if a change goes in that breaks the tests, it is likely to be fixed or reverted within hours. The amount of time the build or tests are broken on the majority of platforms is quite small. It took a while to get to this point due to a bunch of flaky tests, but those have mostly been fixed. The only known problems are mentioned on the wiki. When the buildbots fail, we get mail to python-checkins. Unfortunately, that's mostly spam at this point. I hope to fix that at some point. I also hope to change the main page to give more info in less space. n

[Neal Norwitz]
... That leaves 1 unexplained failure on a Windows bot.
It wasn't my Windows bot, but I believe test_profile has failed (rarely) on several of the bots, and in the same (or very similar) way. Note that the failure went away on the Windows bot in question the next time the tests were run on it. That's typical of test_profile failures. Unfortunately, because test_profile works by comparing stdout against a canned expected-output file under Lib/test/output/, when we re-run the test in verbose mode at the end, that comparison isn't done, so there's isn't a simple way to know whether the test passed or failed during its second run. I _bet_ it actually passed, but don't know. This problem is obscured because when you run regrtest.py "by hand" with -v, you get this message at the end: """ CAUTION: stdout isn't compared in verbose mode: a test that passes in verbose mode may fail without it. """ which is intended to alert you to the possible problem. But when we re-run failing tests in verbose mode "by magic" (via passing -w), that warning isn't produced. BTW, the best solution to this is to convert all the tests away from regrtest's original "compare stdout against a canned output/TEST_NAME file" scheme. That won't fix test_profile, but would make it less of a puzzle to figure out what's wrong with it.

At the risk of waking up a thread that was already declared dead, but perhaps this is usefull. So, what happens is pythons signal handler sets a flag and registrers a callback. Then the main thread should check the flag and make the callback to actually do something with the signal. However the main thread is blocked in GTK and can't check the flag. Nick Maclaren wrote: ...lots of reasons why you can't do anything reliably from within a signal handler... As far as I understand it, what could work is this: -PyGTK registrers a callback. -Pythons signal handler does not change at all. -All threads that run in the Python interpreter occasionally check the flag which the signal handler sets, like the main thread does nowadays. If it is set, the thread calls PyGTKs callback. It does not do anything else with the signal. -PyGTKs callback wakes up the main thread, which actually handles the signal just like it does now. PyGTKs callback could be called from any thread, but it would be called in a normal context, not in a signal handler. As the signal handler does not change, the risk of breaking anything or causing chaos is as large/small as it is under the current scheme. However, PyGTKs problem does get solved, as long as there is _a_ thread that returns to the interpreter within some timeframe. It seems plausible that this will happen.

On 9/8/06, Jan Kanis <jan-python@maka.demon.nl> wrote:
At the risk of waking up a thread that was already declared dead, but perhaps this is usefull.
I don't think we should let this die, at least not yet. Nick seems to be arguing that ANY signal handler is prone to random crashes or corruption (due to bugs). However, we already have a signal handler, so we should already be exposed to the random crashes/corruption. If we're going to rely on signal handling being correct then I think we should also rely on write() being correct. Note that I'm not suggesting an API that allows arbitrary signal handlers, but rather one that calls write() on an array of prepared file descriptors (ignoring errors). Ensuring modifications to that array are atomic would be tricky, but I think it would be doable if we use a read-copy-update approach (with two alternating signal handler functions). Not sure how to ensure there's no currently running signal handlers in another thread though. Maybe have to rip the atomic read/write stuff out of the Linux sources to ensure it's *always* defined behavior. Looking into the existing signalmodule.c, I see no attempts to ensure atomic access to the Handlers data structure. Is the current code broken, at least on non-x86 platforms? -- Adam Olsen, aka Rhamphoryncus

On 9/9/06, Adam Olsen <rhamph@gmail.com> wrote:
So, no, this is certainly not the same as linux kernel atomic operations, which allow you to do more interesting stuff like, test-and-clear, or decrement-and-test atomically. glib has those too, and so does mozilla's NSPR, but only on a few architectures does it do it without using mutexes. for instance, i686 onwards don't require mutexes, only special instructions, but i386 requires mutexes. And we all know mutexes in signal handlers cause deadlocks :-( And, yes, Py_AddPendingCall and Py_MakePendingCalls are most certainly not async safe! Just look at the source code of Py_MakePendingCalls and you'll see an interesting comment... Therefore, discussions about signal safety in whatever new API we may add to Python should be taken with a grain of salt. Regards.

On 9/9/06, Jan Kanis <jan-python@maka.demon.nl> wrote:
No, it is not plausible at all. For instance, the GnomeVFS library usually has a pool of thread, not doing anything, waiting for some VFS task. It is likely that a signal will be delivered to one of these threads, which know nothing about Python, and sit idle most of the time. Regards.

On Sat, 09 Sep 2006 12:59:23 +0200, Gustavo Carneiro <gjcarneiro@gmail.com> wrote:
Well, perhaps it isn't plausible in all cases. However, it is dependant on the libraries you're using and debuggable, which broken signal handlers apparently aren't. The approach would work if you don't use libraries that block threads, and if the libraries that do, co-operate with the interpreter. Open source libraries can be made to co-operate, and if you don't have the source and a library doesn't work correctly, all bets are off anyway. But having the signal handler itself write to a pipe seems to be a cleaner solution, if it can work reliable enough for some value of 'reliable'. Jan

Neal> How often is the python build broken or otherwise unusable? Not very often. I use the head branch of the repository as the source of the interpreter I run on my laptop. It generally takes just a couple minutes on my now-aging PowerBook to svn up and reinstall. I can only recall one time in the past where I had to temporarily fall back to 2.4 because of some change that broke an application. Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I suspect that my usage of the language isn't all that different from most Python developers out there. Neal> Is part of your point that these developers only care about Neal> something called "release" and they won't start testing before Neal> then? If that's the case why don't we start making Neal> (semi-)automated alpha releases every month? How would that be any easier than a user setting up a read-only repository and svn-up-ing it once a month then using that as the default interpreter on that person's development machine? I maintain interpreters for 2.3, 2.4 and bleeding edge at the moment. If I need to it's fairly trivial (a symlink change) to fall back to the latest stable release. Glyph, would that sort of scheme work for you? Skip

On Fri, 14 Jul 2006 06:46:55 -0500, skip@pobox.com wrote:
Neal> How often is the python build broken or otherwise unusable?
Not very often.
I have to agree. The effort I'm talking about is not in fixing large numbers of problems, but simply gearing up to properly test to see if there *are* problems. Keep in mind though: just because the problems are small or easy to fix doesn't mean they're not severe. One tiny bug can prevent a program from even starting up.
A huge percentage of Python developers are working with Zope, which means that although *their* code might not be terribly "sophisticated", it is nevertheless sitting on top of a rather large and intricate pile of implicit dependencies on interpreter behavior.
Glyph, would that sort of scheme work for you?
No. I really think you're underestimating the sort of effort required to upgrade Python. First of all, I do have a job :) and it'd be very hard to make the case to an investor that it was worth tracking down every potential bug I had to determine whether it was a problem because I was working with an unreleased version of Python. This is the same reason I don't use beta versions of the kernel or libc for development. For that matter, I've had to avoid posting to this mailing list because even *this* is stretching my already overburdened schedule :). Secondly, I have to work with a few extension modules. To name a few: * ctypes * PyPAM * pysqlite2 * zope.interface (C optimizations) * xapian * pylucene * dspam * PyOpenSSL * PIL * PyCrypto * pyexpat * pygtk * pyvte Recompiling all of these is a project that takes a day. PyLucene, in fact, I've never managed to build myself, and I can only run because there happen to be debian packages which work (with some fiddling) on Ubuntu. There's no interactive warning during 'svn up' that it's time to recompile everything, either, so I don't even know when the ABIs are going to have changed. Even if everything works perfectly, and were perfectly automated, the compile itself would take a few hours. I also test work on Windows on occasion and recompiling these things _there_ is the work of a week and a half, not to mention it requires that I be sitting at the one machine where I have my Microsoft™ Developer™ Tools™ installed. I made the buildbot recommendation specifically because it would centralize the not inconsiderable effort of integrating these numerous dependencies (presuming that I could submit a Divmod buildbot as well as a Twisted one).

On Thu, 13 Jul 2006 23:39:06 -0700, Neal Norwitz <nnorwitz@gmail.com> wrote:
On 7/13/06, Fredrik Lundh <fredrik@pythonware.com> wrote:
a longer beta period gives *external* developers more time to catch up, and results in less work for the end users.
How often do you test new builds of Python against the most recent alpha of, e.g. the Linux kernel? This isn't just a hypothetical question: Twisted has broken because of changes to Linux as often as it has broken due to changes in Python :). In Linux's case we're all lucky because *any* regressions with existing software are considered bugs, whereas in Python's case, some breakagaes are considered acceptable since it's more feasible to have multiple builds of Python installed more than multiple kernels for different applications.

Fred L. Drake, Jr. wrote:
It feels like the release cycle from alpha1 to final has gotten increasingly rushed.
python 2.2: ~5 months python 2.3: ~7 months python 2.4: ~5 months python 2.5: ~4 months I think the biggest problem is the time between "basically stable beta" and "final"; it was ~90 days in 2.3 and is now ~50 days, most of which is vacation time in my part of the world. we could extend the beta period by playing silly games with naming, but as long as the "big things" go in *before* the beta period starts, I think it's easier to simply extend the beta period by adding another month to it. I'd prefer something like 90 days. </F>

Barry Warsaw wrote:
Maybe there could be an "unstable" release phase that lasts for a whole release cycle. So you'd first release version 2.n as "unstable", and keep 2.(n-1) as the current "stable" release. Then when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would become the new "unstable". (Feel free to substitute other terms for "stable" and "unstable" if you don't like them.) That would give people plenty of warning and time to try things out with the new version. This wouldn't actually be much different to what is done now, except for the naming. But by not officially blessing the latest release as current for a while, it might give less of an impression that stuff is being sprung on the community unawares.
Well, at least they've had a chance to try it. If they don't take that chance, they don't have much ground for complaint. -- Greg

Greg> Maybe there could be an "unstable" release phase that lasts for a Greg> whole release cycle. So you'd first release version 2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable" release. Then Greg> when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would Greg> become the new "unstable". In GCC don't they do an odd (stable)/even (unstable) release schedule? Same for Linux kernels? Would that help? Skip

skip@pobox.com wrote:
No. Linux kernel releases are more aggressive because of the fact that all the patches are mostly developed in different branch/repositories and get merged when they are already well tested and incorporated. Linus can merge literally hundreds of patches daily into his *stable* tree, and do releases from it even weekly, because most destabilizing works are being done in large branches carried on for months before they even are evaluated for being merged; or because patches were settled in the -mm tree for months. Linus' tree is kind-of a release branch, with the difference that he is the BDFL and does what he wants with his tree :) To keep this into perspective, remember also that they don't have *any* kind of testsuite (nor a debugger, if I might say). GCC has a more "old-fashioned" release process, where the trunk evolves through 3 stages: Stage 1 is open for all kind of changes (eg: from simple polishing/refactoring, to merging of large branches containing work of several man-years). Stage 2 is still open for new features, but not for big merges. Stage 3 is feature-freezed, bug-fixing only. Then, the trunk is branched into the new release branch, and the trunk gets back to Stage 1. Nominally, a stage lasts 3 months, but Stage 3 often stretches up to 6 months. The release branches are open for *only* regression fixes (that is, fixes that correct things that used to work in previous releases but do not work anymore). Any regression fix (with a corresponding Bugzilla entry, where it's marked and confirmed as "regression") is applied to trunk *and* the open release branches where the regression exists. For convoluted or dangerous regression fixes, usually maintainers prefer to wait 1 week for the patch to settle down on the trunk before applying it to the release branches. The release manager pulls dot releases from the release branch. Usually, the first release (.0) happens 3-4 months *after* the release branch was created, that is after several months of regression-only patches being merged to it (while new work is being done on the trunk in parallel, in its aggressive Stage 1). The 3-Stage work in the trunk is streamlined by the release manager. At the beginning of Stage 1, a detailed techinical list of on-going "projects" (new features) is presented to the mailing list, explaining the current status of the work and its ETA, and the release manager then publishes a work-plan for Stage 1 and 2, telling which projects will be merged when. This avoids multiple large projects to hit the trunk at the same time, causing may headaches to all the other developers. The work-plan is managed and updated in the GCC Wiki (which is off-line right now, but I'll post a link as example when it's back). -- Giovanni Bajo

On Friday 14 July 2006 06:05, Barry Warsaw wrote:
alpha 1: April 5, 2006 [completed] alpha 2: April 27, 2006 [completed] beta 1: June 20, 2006 [completed] beta 2: July 11, 2006 [completed] rc 1: August 1, 2006 [planned] final: August 8, 2006 [planned] Four months would seem to me to be quite long enough as a release cycle. Extending it means far more work for everyone - either we have special checkin rules for the trunk for a longer period of time (requiring extra monitoring by people like myself and Neal), or we branch earlier, requiring double commits to the trunk and the branch for bugfixes. I also strongly doubt that making a longer "release candidate" cycle would lead to any significant additional testing by end-users. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

FWIW, I tend to run a few project(*) test suites when doing minor releases (2.x.y), just to make sure I don't break things. For major releases, it's a fair bit more work - something like Twisted or Zope3 play at such a low level with the Python interfaces that there's nearly always breakages or massive warning dumps (from memory, zope.interface uses 'with' as an argument name, a lot). The "community buildbot" idea is a good one, although it should just be possible to write something for buildbot that checks out and builds the latest Python SVN, then installs it to a temporary location, then adds that location to the front of the path. Then each project could just add a "Python SVN buildbot" to their exists bbot install. Anthony (*) Typically includes Zope3 and Twisted, because I have these just lying around.

On 7/14/06, Anthony Baxter <anthony@interlink.com.au> wrote:
This is what Xenofarm for Pike has done for a long time now. See for example: http://pike.ida.liu.se/development/pikefarm/7.7.xml This is also what Bitten (http://bitten.cmlenz.net/) has implemented for Trac (which is one of the bug/incident trackers for Python's call for trackers). -- Jeroen Ruigrok van der Werven

Jeroen Ruigrok van der Werven wrote:
Are you aware of http://www.python.org/dev/buildbot/ ? We are not just talking about buildbots here (which the links you refer to seem to be); we are talking about buildbots that don't test Python, but test Python applications. We do know how to run a buildbot. Regards, Martin

On 7/15/06, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Are you aware of http://www.python.org/dev/buildbot/ ?
Yes. And it does not seem to be open for all, but then again, any documentation with regard to it seems to be very sparse or hidden so I can very well be wrong here. Ah, hidden away on the wiki: http://wiki.python.org/moin/BuildBot
Then I would dare to say you haven't fully investigated the links fully, Bitten, for example, also runs the unit-tests for any target you configure, saves all the outputs and provides quick overviews as well as coverage statistics. So that would fall perfectly into that category. See http://bitten.cmlenz.net/build/0.5.x/762 for an example.
We do know how to run a buildbot.
How relevant was this comment in the entire? I am merely trying to help out here pointing out other similar projects when the community participating building issue was raised. -- Jeroen Ruigrok van der Werven

Jeroen Ruigrok van der Werven wrote:
Are you aware of http://www.python.org/dev/buildbot/ ?
Yes. And it does not seem to be open for all
Ah, ok. It indeed isn't open for anonymous participation; the test results are open for all, though.
I don't understand that comment. Bitten seems to be a software package, similar to buildbot. It doesn't do anything useful until I install and configure it, unlike, say, SourceForge, which is not (just) a soft package, but a running installation of it also. Right? The effort is in installing and configuring it, given that many such packages are already written. Distributed testing frameworks typically support running arbitrary target comments, so Bitten is not really special here (buildbot can also run about anything if you configure it to).
It would have been helpful if you had stated *why* you think these URLs are relevant in the context of the discussion. To me, it seemed you are pointing to the existence of distributed testing frameworks. I was pointing out that we (at least, I) am aware of the existence of such frameworks, and that we were doing it for quite some time now also, just like Pike. Regards, Martin
participants (29)
-
"Martin v. Löwis"
-
A.M. Kuchling
-
Aahz
-
Adam Olsen
-
Andrew MacIntyre
-
Anthony Baxter
-
Barry Warsaw
-
Bob Ippolito
-
Brett Cannon
-
Christopher Armstrong
-
Fred L. Drake, Jr.
-
Fredrik Lundh
-
Giovanni Bajo
-
glyph@divmod.com
-
Greg Ewing
-
Gustavo Carneiro
-
James Y Knight
-
Jan Kanis
-
Jeremy Hylton
-
Jeroen Ruigrok van der Werven
-
M.-A. Lemburg
-
Michael Hudson
-
Neal Norwitz
-
Nick Coghlan
-
skip@pobox.com
-
Steven Bethard
-
Talin
-
Terry Reedy
-
Tim Peters