Bug fix releases (was Re: Nested scopes resolution -- you can breathe again!)
[posted to c.l.py with cc to python-dev] [I apologize for the delay in posting this, but it's taken me some time to get my thoughts straight. I hope that by posting this right before IPC9 there'll be a chance to get some good discussion in person.] In article <mailman.982897324.9109.python-list@python.org>, Guido van Rossum <guido@digicool.com> wrote:
We have clearly underestimated how much code the nested scopes would break, but more importantly we have underestimated how much value our community places on stability.
I think so, yes, on that latter clause. I think perhaps it wasn't clear at the time, but I believe that much of the yelling over "print >>" was less over the specific design but because it came so close to the release of 2.0 that there wasn't *time* to sit down and talk things over rationally. As I see it, there's a natural tension between between adding features and delivering bug fixes. Particularly because of Microsoft, I think that upgrading to a feature release to get bug fixes has become anathema to a lot of people, and I think that seeing features added or changed close to a release reminds people too much of the Microsoft upgrade treadmill.
So here's the deal: we'll make nested scopes an optional feature in 2.1, default off, selectable on a per-module basis using a mechanism that's slightly hackish but is guaranteed to be safe. (See below.)
At the same time, we'll augment the compiler to detect all situations that will break when nested scopes are introduced in the future, and issue warnings for those situations. The idea here is that warnings don't break code, but encourage folks to fix their code so we can introduce nested scopes in 2.2. Given our current pace of releases that should be about 6 months warning.
As some other people have pointed out, six months is actually a rather short cycle when it comes to delivering enterprise applications across hundreds or thousands of machines. Notice how many people have said they haven't upgraded from 1.5.2 yet! Contrast that with the quickness of the 1.5.1 to 1.5.2 upgrade. I believe that "from __future__" is a good idea, but it is at best a bandage over the feature/bug fix tension. I think that the real issue is that in the world of core Python development, release N is always a future release, never the current release; as soon as release N goes out the door into production, it immediately becomes release N-1 and forever dead to development Rather than change that mindset directly, I propose that we move to a forked model of development. During the development cycle for any given release, release (N-1).1 is also a live target -- but strictly for bug fixes. I suggest that shortly after the release for Na1, there should also be a release for (N-1).1b1; shortly after the release of Nb1, there would be (N-1).1b2. And (N-1).1 would be released shortly after N. This means that each feature-based release gets one-and-only-one pure bugfix release. I think this will do much to promote the idea of Python as a stable platform for application development. There are a number of ways I can see this working, including setting up a separate project at SourceForge (e.g. pythonpatch.sourceforge.net). But I don't think this will work at all unless the PythonLabs team is at least willing to "bless" the bugfix release. Uncle Timmy has been known to make snarky comments about forever maintaining 1.5.2; I think this is a usable compromise that will take relatively little effort to keep going once it's set up. I think one key advantage of this approach is that a lot more people will be willing to try out a beta of a strict bugfix release, so the release N bugfixes will get more testing than they otherwise would. If there's interest in this idea, I'll write it up as a formal PEP. It's too late for my proposed model to work during the 2.1 release cycle, but I think it would be an awfully nice gesture to the community to take a month off after 2.1 to create 2.0.1, before going on to 2.2. BTW, you should probably blame Fredrik for this idea. ;-) If he had skipped providing 1.5.2 and 2.0 versions of sre, I probably wouldn't have considered this a workable idea. I was just thinking that it was too bad there wasn't a packaged version of 2.0 containing the new sre, and that snowballed into this. -- --- Aahz (Copyright 2001 by aahz@pobox.com) Androgynous poly kinky vanilla queer het <*> http://www.rahul.net/aahz/ Hugs and backrubs -- I break Rule 6 Nostalgia just ain't what it used to be -- --- Aahz (Copyright 2001 by aahz@pobox.com) Androgynous poly kinky vanilla queer het <*> http://www.rahul.net/aahz/ Hugs and backrubs -- I break Rule 6 Nostalgia just ain't what it used to be
Aahz writes:
[posted to c.l.py with cc to python-dev]
[I apologize for the delay in posting this, but it's taken me some time to get my thoughts straight. I hope that by posting this right before IPC9 there'll be a chance to get some good discussion in person.]
Excellent. Even in time for me to mention this in my keynote! :-)
In article <mailman.982897324.9109.python-list@python.org>, Guido van Rossum <guido@digicool.com> wrote:
We have clearly underestimated how much code the nested scopes would break, but more importantly we have underestimated how much value our community places on stability.
I think so, yes, on that latter clause. I think perhaps it wasn't clear at the time, but I believe that much of the yelling over "print >>" was less over the specific design but because it came so close to the release of 2.0 that there wasn't *time* to sit down and talk things over rationally.
In my eyes the issues are somewhat different: "print >>" couldn't possibly break existing code; nested scopes clearly do, and that's why we decided to use the __future__ statement. But I understand that you're saying that the community has grown so conservative that it can't stand new features even if they *are* fully backwards compatible. I wonder, does that extend to new library modules? Is there also resistance against the growth there? I don't think so -- if anything, people are clamoring for more stuff to become standard (while at the same time I feel some pressure to cut dead wood, like the old SGI multimedia modules). So that relegates us at PythonLabs to a number of things: coding new modules (boring), or trying to improve performance of the virtual machine (equally boring, and difficult to boot), or fixing bugs (did I mention boring? :-). So what can we do for fun? (Besides redesigning Zope, which is lots of fun, but runs into the same issues.)
As I see it, there's a natural tension between between adding features and delivering bug fixes. Particularly because of Microsoft, I think that upgrading to a feature release to get bug fixes has become anathema to a lot of people, and I think that seeing features added or changed close to a release reminds people too much of the Microsoft upgrade treadmill.
Actually, I though that the Microsoft way these days was to smuggle entire new subsystems into bugfix releases. What else are "Service Packs" for? :-)
So here's the deal: we'll make nested scopes an optional feature in 2.1, default off, selectable on a per-module basis using a mechanism that's slightly hackish but is guaranteed to be safe. (See below.)
At the same time, we'll augment the compiler to detect all situations that will break when nested scopes are introduced in the future, and issue warnings for those situations. The idea here is that warnings don't break code, but encourage folks to fix their code so we can introduce nested scopes in 2.2. Given our current pace of releases that should be about 6 months warning.
As some other people have pointed out, six months is actually a rather short cycle when it comes to delivering enterprise applications across hundreds or thousands of machines. Notice how many people have said they haven't upgraded from 1.5.2 yet! Contrast that with the quickness of the 1.5.1 to 1.5.2 upgrade.
Clearly, we're taking this into account. If we believed you all upgraded the day we announced a new release, we'd be even more conservative with adding new features (at least features introducing incompatibilities).
I believe that "from __future__" is a good idea, but it is at best a bandage over the feature/bug fix tension. I think that the real issue is that in the world of core Python development, release N is always a future release, never the current release; as soon as release N goes out the door into production, it immediately becomes release N-1 and forever dead to development
Rather than change that mindset directly, I propose that we move to a forked model of development. During the development cycle for any given release, release (N-1).1 is also a live target -- but strictly for bug fixes. I suggest that shortly after the release for Na1, there should also be a release for (N-1).1b1; shortly after the release of Nb1, there would be (N-1).1b2. And (N-1).1 would be released shortly after N.
Your math at first confused the hell out of me, but I see what you mean. You want us to spend time on 2.0.1 which should be a bugfix release for 2.0, while at the same time working on 2.1 which is a new feature release. Guess what -- I am secretly (together with the PSU) planning a 2.0.1 release. I'm waiting however for obtaining the ownership rights to the 2.0 release, so we can fix the GPL incompatibility issue in the license at the same time. (See the 1.6.1 release.) I promise that 2.0.1, unlike 1.6.1, will contain more than a token set of real bugfixes. Hey, we already have a branch in the CVS tree for 2.0.1 development! (Tagged "release20-maint".) We could use some checkins on that branch though.
This means that each feature-based release gets one-and-only-one pure bugfix release. I think this will do much to promote the idea of Python as a stable platform for application development.
Anything we can do to please those republicans! :-)
There are a number of ways I can see this working, including setting up a separate project at SourceForge (e.g. pythonpatch.sourceforge.net). But I don't think this will work at all unless the PythonLabs team is at least willing to "bless" the bugfix release. Uncle Timmy has been known to make snarky comments about forever maintaining 1.5.2; I think this is a usable compromise that will take relatively little effort to keep going once it's set up.
With the CVS branch it's *trivial* to keep it going. We should have learned from the Tcl folks, they've had 8.NpM releases for a while.
I think one key advantage of this approach is that a lot more people will be willing to try out a beta of a strict bugfix release, so the release N bugfixes will get more testing than they otherwise would.
Wait a minute! Now you're making it too complicated. Betas of bugfix releases? That seems to defeat the purpose. What kind of beta-testing does a pure bugfix release need? Presumably each individual bugfix applied has already been tested before it is checked in! Or are you thinking of adding small new features to a "bugfix" release? That ought to be a no-no according to your own philosophy!
If there's interest in this idea, I'll write it up as a formal PEP.
Please do.
It's too late for my proposed model to work during the 2.1 release cycle, but I think it would be an awfully nice gesture to the community to take a month off after 2.1 to create 2.0.1, before going on to 2.2.
It's not too late, as I mentioned. We'll also do this for 2.1.
BTW, you should probably blame Fredrik for this idea. ;-) If he had skipped providing 1.5.2 and 2.0 versions of sre, I probably wouldn't have considered this a workable idea. I was just thinking that it was too bad there wasn't a packaged version of 2.0 containing the new sre, and that snowballed into this.
So the new (2.1) sre code should be merged back into 2.0.1, right? Fredrik, go ahead! We'll start planning for the 2.0.1 release right after we're back from the conference. BTW, See you at the conference! --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
I wonder, does that extend to new library modules? Is there also resistance against the growth there? I don't think so -- if anything, people are clamoring for more stuff to become standard (while at the
There is still the issue of name clashes; introducing a new module in the top-level namespace introduces a potential conflict with someone's application-specific modules. This is a good reason for us to get the standard library packagized sooner rather than later (although this would have to be part of a "feature" release;).
Wait a minute! Now you're making it too complicated. Betas of bugfix releases? That seems to defeat the purpose. What kind of
Betas of the bugfix releases are important -- portability testing is fairly difficult to do when all we have are Windows and Linux/x86 boxes. There's definately a need for at least one beta. We probably don't need to lengthy, multi-phase alpha/alpha/beta/beta/candidate cycle we're using for feature releases now.
It's not too late, as I mentioned. We'll also do this for 2.1.
Managing the bugfix releases would also be an excellent task for someone who's expecting to use the bugfix releases more than the feature releases -- the mentality has to be right for the task. I know I'm much more of a "features" person, and would have a hard time not crossing the line if it were up to me what went into a bugfix release.
BTW, See you at the conference!
If we don't get snowed in! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Digital Creations
Guido van Rossum writes:
I wonder, does that extend to new library modules? Is there also resistance against the growth there? I don't think so -- if anything, people are clamoring for more stuff to become standard (while at the
There is still the issue of name clashes; introducing a new module in the top-level namespace introduces a potential conflict with someone's application-specific modules. This is a good reason for us to get the standard library packagized sooner rather than later (although this would have to be part of a "feature" release;).
But of course the library repackaging in itself would cause enormous outcries, because in a very real sense it *does* break code.
Wait a minute! Now you're making it too complicated. Betas of bugfix releases? That seems to defeat the purpose. What kind of
Betas of the bugfix releases are important -- portability testing is fairly difficult to do when all we have are Windows and Linux/x86 boxes. There's definately a need for at least one beta. We probably don't need to lengthy, multi-phase alpha/alpha/beta/beta/candidate cycle we're using for feature releases now.
OK, you can have *one* beta. That's it.
It's not too late, as I mentioned. We'll also do this for 2.1.
Managing the bugfix releases would also be an excellent task for someone who's expecting to use the bugfix releases more than the feature releases -- the mentality has to be right for the task. I know I'm much more of a "features" person, and would have a hard time not crossing the line if it were up to me what went into a bugfix release.
That's how all of us here at PythonLabs are feeling... I feel a community task coming. I'll bless a 2.0.1 release and the general idea of bugfix releases, but doing the grunt work won't be a PythonLabs task. Someone else inside or outside Python-dev will have to do some work. Aahz?
BTW, See you at the conference!
If we don't get snowed in!
Good point. East coasters flying to LA on Monday, watch your weather forecast! --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
But of course the library repackaging in itself would cause enormous outcries, because in a very real sense it *does* break code.
That's why it has to be a feature release. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Digital Creations
[Fred L. Drake, Jr.]
... Managing the bugfix releases would also be an excellent task for someone who's expecting to use the bugfix releases more than the feature releases -- the mentality has to be right for the task. I know I'm much more of a "features" person, and would have a hard time not crossing the line if it were up to me what went into a bugfix release.
Note there was never a bugfix release for 1.5.2, despite that 1.5.2 had some serious bugs, and that 1.5.2 was current for an unprecedentedly long time. Guido put out a call for volunteers to produce a 1.5.2 bugfix release, but nobody responded. Past is prelude ... everyone-is-generous-with-everyone-else's-time-ly y'rs - tim
Tim Peters writes:
Note there was never a bugfix release for 1.5.2, despite that 1.5.2 had some serious bugs, and that 1.5.2 was current for an unprecedentedly long time. Guido put out a call for volunteers to produce a 1.5.2 bugfix release, but nobody responded. Past is prelude ...
And as long as that continues, I'd have to conclude that the user base is largely happy with the way we've done things. *If* users want bugfix releases badly enough, someone will do them. If not, hey, features can be useful! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Digital Creations
[Fred L. Drake, Jr.]
... Managing the bugfix releases would also be an excellent task for someone who's expecting to use the bugfix releases more than the feature releases -- the mentality has to be right for the task. I know I'm much more of a "features" person, and would have a hard time not crossing the line if it were up to me what went into a bugfix release.
[Uncle Timmy]
Note there was never a bugfix release for 1.5.2, despite that 1.5.2 had some serious bugs, and that 1.5.2 was current for an unprecedentedly long time. Guido put out a call for volunteers to produce a 1.5.2 bugfix release, but nobody responded. Past is prelude ...
everyone-is-generous-with-everyone-else's-time-ly y'rs - tim
I understand the warning. How about the following (and then I really have to go write my keynote speech :-). PythonLabs will make sure that it will happen. But how much stuff goes into the bugfix release is up to the community. We'll give SourceForge commit privileges to individuals who want to do serious work on the bugfix branch -- but before you get commit privileges, you must first show that you know what you are doing by submitting useful patches through the SourceForge patch mananger. Since a lot of the 2.0.1 effort will be deciding which code from 2.1 to merge back into 2.0.1, it may not make sense to upload context diffs to SourceForge. Instead, we'll accept reasoned instructions for specific patches to be merged back. Instructions like "cvs update -j<rev1> -j<rev2> <file>" are very helpful; please also explain why! --Guido van Rossum (home page: http://www.python.org/~guido/)
Tim> Note there was never a bugfix release for 1.5.2, despite that 1.5.2 Tim> had some serious bugs, and that 1.5.2 was current for an Tim> unprecedentedly long time. Guido put out a call for volunteers to Tim> produce a 1.5.2 bugfix release, but nobody responded. Past is Tim> prelude ... Yes, but 1.5.2 source was managed differently. It was released while the source was still "captive" to CNRI and the conversion to Sourceforge was relatively speaking right before the 2.0 release and had the added complication that it more-or-less coincided with the formation of PythonLabs. With the source tree where someone can easily branch it, I think it's now feasible to create a bug fix branch and have someone volunteer to manage additions to it (that is, be the filter that decides if a code change is a bug fix or a new feature). Skip
FYI, in reviewing Misc/HISTORY, it appears that the last Python release *called* a "pure bugfix release" was in November of 1994 (1.1.1) -- although "a few new features were added to tkinter" anyway. fine-by-me-if-we-just-keep-up-the-good-work<wink>-ly y'rs - tim
On Sat, Mar 03, 2001 at 02:10:35PM -0500, Guido van Rossum wrote:
But I understand that you're saying that the community has grown so conservative that it can't stand new features even if they *are* fully backwards compatible.
There is an added dimension, especially with Python. Bugs in the new features. If it entails changes in the compiler or VM (like import-as, which changed the meaning of FROM_IMPORT and added a IMPORT_STAR opcode) or if modules get augmented to use the new features, these changes can introduce bugs into existing code that doesn't even use the new features itself.
I wonder, does that extend to new library modules? Is there also resistance against the growth there? I don't think so -- if anything, people are clamoring for more stuff to become standard (while at the same time I feel some pressure to cut dead wood, like the old SGI multimedia modules).
No (yes), bugfix releases should fix bugs, not add features (nor remove them). Modules in the std lib are just features.
So that relegates us at PythonLabs to a number of things: coding new modules (boring), or trying to improve performance of the virtual machine (equally boring, and difficult to boot), or fixing bugs (did I mention boring? :-).
How can you say this ? Okay, so *fixing* bugs isn't terribly exciting, but hunting them down is one of the best sports around. Same for optimizations: rewriting the code might be boring (though if you are a fast typist, it usually doesn't take long enough to get boring :) but thinking them up is the fun part. But who said PythonLabs had to do all the work ? You guys didn't do all the work in 2.0->2.1, did you ? Okay, so most of the major features are written by PythonLabs, and most of the decisions are made there, but there's no real reason for it. Consider the Linux kernel: Linus Torvalds releases the kernels in the devel 'tree' and usually the first few kernels in the 'stable' tree, and then Alan Cox takes over the stable tree and continues it. (Note that this analogy isn't quite correct: the stable tree often introduces new features, new drivers, etc, but avoids real incompatibilites and usually doesn't require extra upgrades of tools and such.) I hope you don't think any less of me if I volunteer *again* :-) but I'm perfectly willing to maintain the bugfix release(s). I also don't think we should necessarily stay at a single bugfix release. Whether or not a 'beta' for the bugfix release is necessary, I'm not sure. I don't think so, at least not if you release multiple bugfix releases. Holiday-Greetings-from-Long-Beach-ly y'rs, -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
"TW" == Thomas Wouters <thomas@xs4all.net> writes:
[GvR:]
So that relegates us at PythonLabs to a number of things: coding new modules (boring), or trying to improve performance of the virtual machine (equally boring, and difficult to boot), or fixing bugs (did I mention boring? :-).
TW> How can you say this ? Okay, so *fixing* bugs isn't terribly TW> exciting, but hunting them down is one of the best sports TW> around. Same for optimizations: rewriting the code might be TW> boring (though if you are a fast typist, it usually doesn't take TW> long enough to get boring :) but thinking them up is the fun TW> part. TW> But who said PythonLabs had to do all the work ? You guys didn't TW> do all the work in 2.0->2.1, did you ? Okay, so most of the TW> major features are written by PythonLabs, and most of the TW> decisions are made there, but there's no real reason for TW> it. Most of the work I did for Python 2.0 was fixing bugs. It was a lot of fairly tedious but necessary work. I have always imagined that this was work that most people wouldn't do unless they were paid to do it. (python-dev seems to have a fair number of exceptions, though.) Working on major new features has a lot more flash, so I imagine that volunteers would be more inclined to help. Neil's work on GC or yours on augmented assignment are examples. There's nothing that says we have to do all the work. In fact, I imagine we'll continue to collectively spend a lot of time on maintenance issues. We get paid to do it, and we get to hack on Zope and ZODB the rest of the time, which is also a lot of fun. Jeremy
participants (7)
-
aahz＠panix.com -
Fred L. Drake, Jr. -
Guido van Rossum -
Jeremy Hylton -
Skip Montanaro -
Thomas Wouters -
Tim Peters