Back in February, I tried to fix an issue of threads executed using 'threading' that did not shut themselves down before the interpreter started tearing down globals, the thread triggering an exception, and then a new exception being raised while trying to report the exception raised in the thread by accessing globals (see http://mail.python.org/pipermail/python-dev/2004-February/042457.html for the previous discussion). An initial fix was devised, but since there was no way to trigger the issue every time I was not sure if the fix was thorough enough.
Well, someone finally has an app that causes the error consistently and the previous fix did not go far enough. But with the help of the app (mnet, specifically) I was finally able to get rid of all global accesses while reporting exceptions so that the original exception caught in the thread can get printed. The patch is at http://python.org/sf/954922 while the bug report is at http://python.org/sf/754449 .
Trouble is that I had to basically fake traceback.format_exc() in the patch to a reasonable level since the 'traceback' module bounces around between its various functions too much to just store a local copy of the function and have it still work after globals are set to None. So I need someone to just look over the code to make sure that the code is not doing something stupid when this code is called (default way is still used when 'sys' is defined and thus the interpreter is not being torn down).
And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there.
-Brett
"Brett C." bac@OCF.Berkeley.EDU:
Back in February, I tried to fix an issue of threads executed using 'threading' that did not shut themselves down before the interpreter started tearing down globals, the thread triggering an exception, and then a new exception being raised while trying to report the exception raised in the thread by accessing globals
Seems to me the way to fix this sort of thing properly is to get rid of the need for explicitly clearing globals.
Now that we have a garbage collector, is it really necessary any more? What would happen if we simply set sys.modules to None and then invoked the collector?
Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Hello Greg,
On Tue, May 18, 2004 at 02:33:42PM +1200, Greg Ewing wrote:
Now that we have a garbage collector, is it really necessary any more? What would happen if we simply set sys.modules to None and then invoked the collector?
There is a patch on SF that does this. Brett, can you see if this patch helps in your case?
http://www.python.org/sf/812369
The problem I can see with my patch is that it's not clear to me that it eventually causes the destruction of enough objects. For that matter it's not clear why, or how many, objects we should try to destroy at shut-down time. I bet the goal is that if Python is embedded, and initialized and shut down repeatedly, it should not leak memory. Someone with such an application should give my patch a serious try before I can start to believe that no object is missed. There are moreover a few other pending issues raised on the patch's message board.
A bientot,
Armin.
Armin Rigo wrote:
Hello Greg,
On Tue, May 18, 2004 at 02:33:42PM +1200, Greg Ewing wrote:
Now that we have a garbage collector, is it really necessary any more? What would happen if we simply set sys.modules to None and then invoked the collector?
There is a patch on SF that does this. Brett, can you see if this patch helps in your case?
http://www.python.org/sf/812369
It segfaulted when it normally spits out a bunch of tracebacks about sys.stderr being None from the interpreter tearing itself down.
-Brett
Armin Rigo arigo@tunes.org:
I bet the goal is that if Python is embedded, and initialized and shut down repeatedly, it should not leak memory.
I think that's going to be impossible to achieve as long as there is no protocol for shutting down extension modules, which may be holding references to things in C globals.
Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Brett C. wrote:
And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there.
Not going to be there for 2.3.4.
My approach to the bug fix releases is that, in general, "there's always a next release". That is, the bug fix releases are relatively frequent (5-6 months) and they're "what's in the branch at the moment".
Starting with the next (2.3.5/2.4.1) releases, I plan on sending out a 1 month warning to python-list/python-dev, so that people that care can make sure their pet bug fix is checked in and done before the release. I'm really not interested in trying to slot in last minute changes that may, or may not, break the release.
In case it wasn't clear, I'm trying to get Python into a nice regular release cycle for the bugfix releases. While there will be the occasional "oh. my. god." type bug that drive a bug fix release now now now, I'd prefer that people can get used to a regular cycle for the bug fix releases. Part of this, of course, is being harsh on the no-new-features rule. I want people to feel confident that a bug fix release won't break their installed code base. In addition, I never ever want to see
try: True, False except NameError: True, False = 1, 0
or it's like ever again. I've seen it far far too often in code that wants to be "2.2" compatible, but the new True/False in the bugfix release of 2.2 made life unpleasant.
I'm open to suggestions for this - although emails about "but I really really want SF bug #XYZ fixed for 2.3.4" is not what I'm after.
Anthony
Anthony Baxter wrote:
Brett C. wrote:
And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there.
Not going to be there for 2.3.4.
Figured as much.
My approach to the bug fix releases is that, in general, "there's always a next release". That is, the bug fix releases are relatively frequent (5-6 months) and they're "what's in the branch at the moment".
OK. How long do you plan to do this for the 2.3 branch?
Starting with the next (2.3.5/2.4.1) releases, I plan on sending out a 1 month warning to python-list/python-dev, so that people that care can make sure their pet bug fix is checked in and done before the release. I'm really not interested in trying to slot in last minute changes that may, or may not, break the release.
Makes sense to me.
In case it wasn't clear, I'm trying to get Python into a nice regular release cycle for the bugfix releases. While there will be the occasional "oh. my. god." type bug that drive a bug fix release now now now, I'd prefer that people can get used to a regular cycle for the bug fix releases. Part of this, of course, is being harsh on the no-new-features rule. I want people to feel confident that a bug fix release won't break their installed code base.
Getting into a regular groove for releases sounds great to me, especially if we can state confidently that a bugfix release will be along every 6 months (+/- a month).
-Brett
Anthony Baxter wrote:
Starting with the next (2.3.5/2.4.1) releases, I plan on sending out a 1 month warning to python-list/python-dev, so that people that care can make sure their pet bug fix is checked in and done before the release. I'm really not interested in trying to slot in last minute changes that may, or may not, break the release.
Cool! That'll make some people happier. I'm tempted to suggest adding c.l.py.announce, but on second thought, anyone who's not paying reasonably close attention to at least one of python-list/python-dev probably doesn't have enough attention to properly champion a patch for a bugfix release.
Brett C. wrote:
Anthony Baxter wrote:
My approach to the bug fix releases is that, in general, "there's always a next release". That is, the bug fix releases are relatively frequent (5-6 months) and they're "what's in the branch at the moment".
OK. How long do you plan to do this for the 2.3 branch?
I'm anticipating doing this until 2.4.1. That is, the upcoming release train will be:
2.3.4 2.4a{1,2,...} 2.4b{1,2,...} 2.4rc{1,2,...} 2.4 2.3.5 2.4.1
Unless there's a pressing need for it, I don't see the point to cutting a 2.3.6 or later, once 2.4 is onto the release24-maint branch. Obviously, if there's one of the oh-my-gods type bugs in 2.3, this will probably require a 2.3.6. Hopefully, by the time I finish the 2.4 cycle, there'll be some sane release automation tools finished, too.
Once we've forked release24-maint, release23-maint can start to die. There's no point backporting patches to two separate -maint branches.
Getting into a regular groove for releases sounds great to me, especially if we can state confidently that a bugfix release will be along every 6 months (+/- a month).
Yep. I plan to update the bug fix release PEP and post it out to c.l.py and python-dev for feedback.
As ever, I'm open to feedback on any of the above. Except for the "can we please have this whizzo feature in the next bug fix release" feedback. You know the answer to that one ;)
Anthony Baxter wrote:
Brett C. wrote:
Anthony Baxter wrote:
My approach to the bug fix releases is that, in general, "there's always a next release". That is, the bug fix releases are relatively frequent (5-6 months) and they're "what's in the branch at the moment".
OK. How long do you plan to do this for the 2.3 branch?
I'm anticipating doing this until 2.4.1. That is, the upcoming release train will be:
2.3.4 2.4a{1,2,...} 2.4b{1,2,...} 2.4rc{1,2,...} 2.4 2.3.5 2.4.1
Unless there's a pressing need for it, I don't see the point to cutting a 2.3.6 or later, once 2.4 is onto the release24-maint branch. Obviously, if there's one of the oh-my-gods type bugs in 2.3, this will probably require a 2.3.6. Hopefully, by the time I finish the 2.4 cycle, there'll be some sane release automation tools finished, too.
Anything us normal folk can help you with in terms of the aforementioned tools?
Maybe this could encompass a tool to run the testing suite, re-run the failed tests with verbose output, and then email them to a Mailman list with system info so we have a partially automated testing framework for alphas, betas, and release candidates (Christopher Blunck proposed this idea back in March; email at http://mail.python.org/pipermail/python-dev/2004-March/043492.html) since we currently don't have a testing server farm running regularly anymore?
-Brett
Any chance someone can take a look at this before 2.4a1 goes out the door?
Brett C. wrote:
Back in February, I tried to fix an issue of threads executed using 'threading' that did not shut themselves down before the interpreter started tearing down globals, the thread triggering an exception, and then a new exception being raised while trying to report the exception raised in the thread by accessing globals (see http://mail.python.org/pipermail/python-dev/2004-February/042457.html for the previous discussion). An initial fix was devised, but since there was no way to trigger the issue every time I was not sure if the fix was thorough enough.
Well, someone finally has an app that causes the error consistently and the previous fix did not go far enough. But with the help of the app (mnet, specifically) I was finally able to get rid of all global accesses while reporting exceptions so that the original exception caught in the thread can get printed. The patch is at http://python.org/sf/954922 while the bug report is at http://python.org/sf/754449 .
Trouble is that I had to basically fake traceback.format_exc() in the patch to a reasonable level since the 'traceback' module bounces around between its various functions too much to just store a local copy of the function and have it still work after globals are set to None. So I need someone to just look over the code to make sure that the code is not doing something stupid when this code is called (default way is still used when 'sys' is defined and thus the interpreter is not being torn down).
And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there.
-Brett
participants (5)
-
Aahz -
Anthony Baxter -
Armin Rigo -
Brett C. -
Greg Ewing