Over the last week, I checked in fixes for two distinct broad causes of segfaults in code using weakrefs with callbacks. The bugs have been there since weakrefs were introduced, but for whatever reason nobody bumped into them (knowingly) until Jim Fulton and Thomas Heller happened to provoke both, independently, within a day of each other. It was especially easy under Thomas's scenario *not* to get a segfault in a release build, but to suffer random memory corruption instead (if the double-deallocation provoked pymalloc into handing out the same chunk of memory to two distinct objects alive at the same time -- and that is, alas, a likely outcome). I suspect these bugs hid for so long because it's taken Pythoneers a long time to discover why weakrefs can be so cool, and start to build serious apps on top of them. Casual programmers aren't likely to use weakrefs at all, but once you've built a cache based on weakrefs in a large app, weakrefs become critical to your code and your design. So I think either of these fixes is enough to justify a bugfix release, and having two of them makes a compelling case. What say we get 2.3.3 in motion? I did the weakref checkins already on the trunk and on release23-maint; Thomas Heller confirmed that his problems went away on release23-maint, and Jim Fulton confirmed that his Zope3 segfaults went away on the released 2.3.2 + a patch identical in all functional respects to what got checked in (the new test_weakref test cases, and some code comments, were different). If we get 2.3.3c1 out in early December, we could release 2.3.3 final before the end of the year, and start 2004 with a 100% bug-free codebase <wink>.
I was planning on a just-before-Christmas 2.3.3. Maybe a RC around the 15th of December, and a release around the 22nd? -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
[Anthony Baxter]
I was planning on a just-before-Christmas 2.3.3. Maybe a RC around the 15th of December, and a release around the 22nd?
That's good enough for me. I'd rather push the RC up a week earlier, though, to give more time for user testing. Many people take large blocks of time off around Christmas, and have major extra demands on their time the week before too (planning and shopping and endless bickering with family -- Christmas is great <wink>). What else does 2.3.3 need? IIRC, the sre tests still fail on 2.3 maint, and that's a showstopper. I'd like to "do something" about the 2.3 changes to Python finalization that have provoked new problems, but don't have time. If nothing else, I'd at least like to common out the second call to gc in Py_Finalize -- with hidnsight, that wasn't ready for prime time, and the # of things that can go wrong when trying to execute Python code after modules (particularly sys) have been torn down appears boundless. The only bad thing I've seen come out of the first call to gc in Py_Finalize is nonsense errors complaining that Python hasn't been initialized (when a __del__ or weakref callback triggered then tries to import a new module). What else does 2.3.3 need?
What else does 2.3.3 need? IIRC, the sre tests still fail on 2.3 maint, and that's a showstopper.
[Anthony Baxter]
I thought I'd fixed that.
I don't know. How did they fail? This is how they fail for me today (Windows): C:\Code\23\PCbuild>python ../lib/test/test_re.py test_anyall (__main__.ReTests) ... ok test_basic_re_sub (__main__.ReTests) ... ok test_bigcharset (__main__.ReTests) ... ok test_bug_113254 (__main__.ReTests) ... ok test_bug_114660 (__main__.ReTests) ... ok test_bug_117612 (__main__.ReTests) ... ok test_bug_418626 (__main__.ReTests) ... ERROR test_bug_448951 (__main__.ReTests) ... ok test_bug_449000 (__main__.ReTests) ... ok test_bug_449964 (__main__.ReTests) ... ok test_bug_462270 (__main__.ReTests) ... ok test_bug_527371 (__main__.ReTests) ... ok test_bug_545855 (__main__.ReTests) ... ok test_bug_612074 (__main__.ReTests) ... ok test_bug_725106 (__main__.ReTests) ... ok test_bug_725149 (__main__.ReTests) ... ok test_bug_764548 (__main__.ReTests) ... ok test_category (__main__.ReTests) ... ok test_constants (__main__.ReTests) ... ok test_expand (__main__.ReTests) ... ok test_finditer (__main__.ReTests) ... ok test_flags (__main__.ReTests) ... ok test_getattr (__main__.ReTests) ... ok test_getlower (__main__.ReTests) ... ok test_groupdict (__main__.ReTests) ... ok test_ignore_case (__main__.ReTests) ... ok test_non_consuming (__main__.ReTests) ... ok test_not_literal (__main__.ReTests) ... ok test_pickling (__main__.ReTests) ... ok test_qualified_re_split (__main__.ReTests) ... ok test_qualified_re_sub (__main__.ReTests) ... ok test_re_escape (__main__.ReTests) ... ok test_re_findall (__main__.ReTests) ... ok test_re_groupref (__main__.ReTests) ... ok test_re_groupref_exists (__main__.ReTests) ... ok test_re_match (__main__.ReTests) ... ok test_re_split (__main__.ReTests) ... ok test_re_subn (__main__.ReTests) ... ok test_repeat_minmax (__main__.ReTests) ... ok test_scanner (__main__.ReTests) ... ok test_search_coverage (__main__.ReTests) ... ok test_search_star_plus (__main__.ReTests) ... ok test_special_escapes (__main__.ReTests) ... ok test_sre_character_literals (__main__.ReTests) ... ok test_stack_overflow (__main__.ReTests) ... ERROR test_symbolic_refs (__main__.ReTests) ... ok ====================================================================== ERROR: test_bug_418626 (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_re.py", line 410, in test_bug_418626 self.assertEqual(re.search('(a|b)*?c', 10000*'ab'+'cd').end(0), 20001) File "C:\CODE\23\lib\sre.py", line 137, in search return _compile(pattern, flags).search(string) RuntimeError: maximum recursion limit exceeded ====================================================================== ERROR: test_stack_overflow (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_re.py", line 420, in test_stack_overflow self.assertEqual(re.match('(x)*', 50000*'x').group(1), 'x') File "C:\CODE\23\lib\sre.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded ---------------------------------------------------------------------- Ran 46 tests in 0.550s FAILED (errors=2)
I have a bunch of compatibility fixes that I'd like to work on. I'm also considering switching to the newer version of autoconf.
A newer & buggier version, or a newer & better version <wink>?
It looks like someone have backported the changes done in test_re.py. These tests were expected to fail with the SRE from 2.3. -- Gustavo Niemeyer http://niemeyer.net
[Gustavo Niemeyer]
It looks like someone have backported the changes done in test_re.py. These tests were expected to fail with the SRE from 2.3.
The tests are never expected to fail, so I think you mean that test_re in 2.3 should expect (and suppress) the RuntimeError in these cases. It looks like Anthony changed this most recently: test_re.py Revision 1.45.6.1 Tue Nov 4 14:11:01 2003 UTC (2 weeks, 3 days ago) by anthonybaxter Branch: release23-maint Changes since 1.45: +9 -7 lines get tests working again. partial backport of 1.46 - I fixed the recursive tests that used to fail, but left test_re_groupref_exists disabled, as it fails on the release23-maint branch. Maybe something else needs to be backported? We've got more than one problem here, then, because Barry reports that test_re on release23-maint, as it exists today, does *not* fail on a RedHat 9 build. So if Anthony reverted that change, test_re would pass again on Windows, but would start to fail on RH9.
Yes, that's what I meant. Sorry for not being clear.
Yes, he seems to belive that the new SRE scheme was introduced in 2.3, but these tests should still expect RuntimeError in 2.3.
That's strange indeed. Either other changes were introduced in 2.3 which changed the number of recursions, what I don't belive to be the case, or the fixed recursion limit was raised in that platform. -- Gustavo Niemeyer http://niemeyer.net
Gustavo Niemeyer <niemeyer@conectiva.com> writes:
Yes, he seems to belive that the new SRE scheme was introduced in 2.3, but these tests should still expect RuntimeError in 2.3.
I was under the impression (and slightly alarmed) that the recursion removal gimmicks had been backported from the trunk to the release23-maint branch. Was that not the case? (If so, phew). If that *wasn't* the case, then why were the tests failing for Anthony before he made that checkin? Cheers, mwh -- Or here's an even simpler indicator of how much C++ sucks: Print out the C++ Public Review Document. Have someone hold it about three feet above your head and then drop it. Thus you will be enlightened. -- Thant Tessman
FWIW, I'm having much more problems with 2.3cvs on RH7.3. test_re.py core dumps for me for instance. I'm doing a fresh build --with-pydebug and will try to get more information. -Barry
[Barry]
FWIW, I'm having much more problems with 2.3cvs on RH7.3. test_re.py core dumps for me for instance.
For Guido too.
I'm doing a fresh build --with-pydebug and will try to get more information.
It's one of two things: USE_RECURSION_LIMIT isn't #define'd or USE_RECURSION_LIMIT is #define'd, but to a value too large for that box There's a maze of #ifdef'ery near the start of _sre.c setting USE_RECURSION_LIMIT differently for different platforms. Windows doesn't use USE_RECURSION_LIMIT -- it uses a different gimmick based on being able to test for C stack overflow directly on Windows. test_re.py *should*, at this time, fail in exactly the same ways I reported it failing on Windows.
On Fri, 2003-11-21 at 13:09, Tim Peters wrote:
test_re.py *should*, at this time, fail in exactly the same ways I reported it failing on Windows.
Then Something Else is going on. As reported in another message, it doesn't fail for me on either RH9 or RH7.3, and a fresh debug build on RH7.3 also doesn't crash for me either. -Barry
On Fri, 21 Nov 2003, Barry Warsaw wrote:
sre in 2.3x is compiler sensitive - the stack frame size becomes critical in how many sre recursions are supported, and a core dump is certain if the sre recursion limit is more than the available stack space allows. Threads support may be mixed in with this, as the size of the stack for the primary or initial thread is what gets exercised by test_re. On FreeBSD 4.x the stack size for this thread is fixed at 1MB (pthreads implementation limitation, not OS limit). gcc versions < 3.0 don't cause problems with the default sre recursion limit of 10000, but later versions do. So I'd suggest trying a lower sre recursion limit to see whether this helps. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia
Never mind. A fresh debug build, test -u all yields no problems with 2.3cvs on RH7.3 either. -Barry
[Barry Warsaw]
Never mind. A fresh debug build, test -u all yields no problems with 2.3cvs on RH7.3 either.
test_re.py isn't supposed to pass on 2.3 maint today. If it passed, it's broken, and will start to fail as soon as the breakage is repaired. Find out what USE_RECURSION_LIMIT is set to on that box.
On Fri, 2003-11-21 at 13:13, Tim Peters wrote:
Is it possible that USE_RECURSION_LIMIT isn't defined for my RH builds?! I added the attached little bit of (seemingly useful) code to _sre.c, recompiled and then... % ./python Python 2.3.3a0 (#4, Nov 21 2003, 13:39:39) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Very odd. -Barry
[Barry Warsaw]
Is it possible that USE_RECURSION_LIMIT isn't defined for my RH builds?!
I can't see how: it's set by a giant maze of #ifdef's, which are almost as reliable as a giant maze of CVS branches <wink>. Because the #ifdef's nest 4 deep at one point, and the bodies aren't indented, it's damned hard to figure out what they're doing by eyeball. But I *think* this part: """ #else #define USE_RECURSION_LIMIT 10000 #endif #endif #endif """ which gives all the appearance of defining a default value (if nothing else triggers), is actually nested *inside* an #elif defined(__FreeBSD__) block (which is in turn nested in a !defined(USE_STACKCHECK) block, which is in turn nested in an ifndef SRE_RECURSIVE block). God only knows what the intent was. But I expect that, yes, USE_RECURSION_LIMIT isn't getting defined on anything other than FreeBSD and Win64.
OTOH, if you believe what it says, that leads directly to the cause <wink>.
On Fri, 2003-11-21 at 14:22, Tim Peters wrote:
Yep, you're right. If I hack _sre.c with the patch below, I think I get something closer to what we expect to see. % ./python Python 2.3.3a0 (#5, Nov 21 2003, 14:26:25) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
... ====================================================================== ERROR: test_bug_418626 (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_re.py", line 410, in test_bug_418626 self.assertEqual(re.search('(a|b)*?c', 10000*'ab'+'cd').end(0), 20001) File "/home/barry/projects/python23/Lib/sre.py", line 137, in search return _compile(pattern, flags).search(string) RuntimeError: maximum recursion limit exceeded ====================================================================== ERROR: test_stack_overflow (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_re.py", line 420, in test_stack_overflow self.assertEqual(re.match('(x)*', 50000*'x').group(1), 'x') File "/home/barry/projects/python23/Lib/sre.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded I'll leave it to someone else to check in the proper fix. (But does anybody else like exposing RECURSION_LIMIT in the _sre module?) -Barry
[Barry]
Yup, that's how they fail on Windows today, and is how they're *expected* to fail everywhere today.
I'll leave it to someone else to check in the proper fix.
I expect Anthony has the best shot at understanding why he did what he did before, so has the best shot at undoing it too without creating more new problems.
(But does anybody else like exposing RECURSION_LIMIT in the _sre module?)
For 2.3 maint it would be a new feature, so probably not. For 2.4, I believe all this code has become a mass of decoys (that is, it's still there, but is no longer used; I don't know why it hasn't been deleted) -- Gustavo reworked sre to stop using C-level recursion. BTW, Gustavo, we get a big pile of compiler warnings on the trunk (2.4 development) in _sre.c now, on Windows, and apparently under some-but-not-all gcc flavors. How about cleaning those up? See: http://mail.python.org/pipermail/python-dev/2003-October/039059.html
Thanks for pointing me this. I'll manage to clean these issues (the code and the warnings) ASAP. -- Gustavo Niemeyer http://niemeyer.net
Sorry - I (and a bunch of other folks, Alex included if I recall correctly) was seeing a bunch of test failures in test_re - I ported the "fixed" tests from the trunk, in the assumption that the relevant change had been made to the branch. I'll undo it, once someone's fixed _sre in the branch to be broken again <wink> Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Sat, 2003-11-22 at 00:10, Jeremy Hylton wrote:
Unfortunately, no so for me: test_mimetypes test test_mimetypes failed -- Traceback (most recent call last): File "/home/barry/projects/python23/Lib/test/test_mimetypes.py", line 52, in test_guess_all_types eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) File "/home/barry/projects/python23/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: ['.asc', '.bat', '.c', '.h', '.ksh', '.pl', '.txt'] != ['.bat', '.c', '.h', '.ksh', '.pl', '.txt'] But we've seen these before, right? Doesn't some test interfere with globals in a way that screws mimetypes occasionally? -Barry
Did a cvs update about 30 minutes ago. make test reports no errors. Running again with "-u all -r" to see what happens.
Also looks good. This was with a RH9 system.
[Barry Warsaw]
googling on test_guess_all_types nails it: http://mail.python.org/pipermail/python-dev/2003-September/038264.html Jeff Epler reported there, in a reply to you about the same thing in 2.3.1, that test_urllib2 interferes with test_mimetypes (when run in that order), and included a patch claimed to fix it. Of course, since he didn't put the patch on SF, it just got lost.
On Sat, 2003-11-22 at 07:59, Barry Warsaw wrote:
Yes and yes. Use of mimetypes causes the module's init() function to be run on a set of known files. test_mimetypes calls init() after zapping the list of knownfiles. init() does not clear out existing global state before re-initializing, which is why the test fails if mimetypes has been used before test_mimetypes. Jeremy
>> This needs fresh testing on all non-Win32 platforms ... >> Running the standard test_re.py is an adequate test. >> >> So start testing, or (my recommendation) upgrade to Win32 <wink>. Jeremy> Did a cvs update about 30 minutes ago. make test reports no Jeremy> errors. Running again with "-u all -r" to see what happens. "regrtest.py -u all -r" worked for me on Mac OS X. Skip
Tim> So ... upgrade to Win32 <wink>. I'll consider that after you've been in charge of software development at Microsoft for a couple years. Skip
[...]
It looks to be this patch's fault: ------------- From: loewis@users.sourceforge.net To: python-checkins@python.org Cc: Bcc: Subject: [Python-checkins] python/dist/src/Modules _sre.c,2.99,2.99.8.1 Reply-To: python-dev@python.org Update of /cvsroot/python/python/dist/src/Modules In directory sc8-pr-cvs1:/tmp/cvs-serv28127 Modified Files: Tag: release23-maint _sre.c Log Message: Patch #813391: Reduce limits for amd64 and sparc64. Index: _sre.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/_sre.c,v retrieving revision 2.99 retrieving revision 2.99.8.1 diff -C2 -d -r2.99 -r2.99.8.1 *** _sre.c 26 Jun 2003 14:41:08 -0000 2.99 --- _sre.c 20 Oct 2003 20:59:45 -0000 2.99.8.1 *************** *** 72,78 **** /* FIXME: maybe the limit should be 40000 / sizeof(void*) ? */ #define USE_RECURSION_LIMIT 7500 - #else ! #if defined(__GNUC__) && defined(WITH_THREAD) && defined(__FreeBSD__) /* the pthreads library on FreeBSD has a fixed 1MB stack size for the * initial (or "primary") thread, which is insufficient for the default --- 72,83 ---- /* FIXME: maybe the limit should be 40000 / sizeof(void*) ? */ #define USE_RECURSION_LIMIT 7500 ! #elif defined(__FreeBSD__) ! /* FreeBSD/amd64 and /sparc64 require even smaller limits */ ! #if defined(__amd64__) ! #define USE_RECURSION_LIMIT 6000 ! #elif defined(__sparc64__) ! #define USE_RECURSION_LIMIT 3000 ! #elif defined(__GNUC__) && defined(WITH_THREAD) /* the pthreads library on FreeBSD has a fixed 1MB stack size for the * initial (or "primary") thread, which is insufficient for the default _______________________________________________ Python-checkins mailing list Python-checkins@python.org http://mail.python.org/mailman/listinfo/python-checkins -- Gustavo Niemeyer http://niemeyer.net
[martin@v.loewis.de]
Sorry for causing so much confusion, and thanks to Tim for fixing it.
It's OK, Martin! It was a wonderful example of a simple mistake getting misdiagnosed and so leading to further mistakes, until the whole was much more confusing than the sum of its parts. And, as always, the root cause was trying to cover up Unix bugs with C's preprocessor <wink>.
"Tim Peters" <tim.one@comcast.net> writes:
I'm not expecting to be around much in between those dates, but could do some work for the RC with those dates.
What else does 2.3.3 need?
There are a bunch of build problems which my brain, sadly but not surprisingly, has thoroughly paged out. We should give the new autoconf a go, at least. Cheers, mwh -- "Well, the old ones go Mmmmmbbbbzzzzttteeeeeep as they start up and the new ones go whupwhupwhupwhooopwhooooopwhooooooommmmmmmmmm." -- Graham Reed explains subway engines on asr
Michael Hudson wrote We should give the new autoconf a go, at least.
I would strongly prefer to do this sooner than later, so I was thinking of doing the upgrade sometime this week. Does anyone have/know any reasons to not upgrade to the newer autoconf? It should fix a bunch of build annoyances (and I can get rid of aclocal.m4) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
Anthony Baxter <anthony@interlink.com.au> writes:
Well, there was an almost instananeous brown-paper-bag 2.59 release, but I don't know of any problems with 2.59. Not sure I would if there were, mind.
It should fix a bunch of build annoyances (and I can get rid of aclocal.m4)
That's the motivation :-) Cheers, mwh -- Make this IDLE version 0.8. (We have to skip 0.7 because that was a CNRI release in a corner of the basement of a government building on a planet circling Aldebaran.) -- Guido Van Rossum, in a checkin comment
Tim> What say we get 2.3.3 in motion? As long as a primary motivator for a 2.3.3 release seems to be weakref-related, perhaps someone who's familiar enough with their usage could beef up the docs enough to get rid of this comment at the top of the module doc: XXX -- need to say more here! I was motivated to take a look at the weakref docs for the first time after Tim mentioned: Casual programmers aren't likely to use weakrefs at all, but once you've built a cache based on weakrefs in a large app, weakrefs become critical to your code and your design. Skip
I was planning on a just-before-Christmas 2.3.3. Maybe a RC around the 15th of December, and a release around the 22nd? -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
[Anthony Baxter]
I was planning on a just-before-Christmas 2.3.3. Maybe a RC around the 15th of December, and a release around the 22nd?
That's good enough for me. I'd rather push the RC up a week earlier, though, to give more time for user testing. Many people take large blocks of time off around Christmas, and have major extra demands on their time the week before too (planning and shopping and endless bickering with family -- Christmas is great <wink>). What else does 2.3.3 need? IIRC, the sre tests still fail on 2.3 maint, and that's a showstopper. I'd like to "do something" about the 2.3 changes to Python finalization that have provoked new problems, but don't have time. If nothing else, I'd at least like to common out the second call to gc in Py_Finalize -- with hidnsight, that wasn't ready for prime time, and the # of things that can go wrong when trying to execute Python code after modules (particularly sys) have been torn down appears boundless. The only bad thing I've seen come out of the first call to gc in Py_Finalize is nonsense errors complaining that Python hasn't been initialized (when a __del__ or weakref callback triggered then tries to import a new module). What else does 2.3.3 need?
What else does 2.3.3 need? IIRC, the sre tests still fail on 2.3 maint, and that's a showstopper.
[Anthony Baxter]
I thought I'd fixed that.
I don't know. How did they fail? This is how they fail for me today (Windows): C:\Code\23\PCbuild>python ../lib/test/test_re.py test_anyall (__main__.ReTests) ... ok test_basic_re_sub (__main__.ReTests) ... ok test_bigcharset (__main__.ReTests) ... ok test_bug_113254 (__main__.ReTests) ... ok test_bug_114660 (__main__.ReTests) ... ok test_bug_117612 (__main__.ReTests) ... ok test_bug_418626 (__main__.ReTests) ... ERROR test_bug_448951 (__main__.ReTests) ... ok test_bug_449000 (__main__.ReTests) ... ok test_bug_449964 (__main__.ReTests) ... ok test_bug_462270 (__main__.ReTests) ... ok test_bug_527371 (__main__.ReTests) ... ok test_bug_545855 (__main__.ReTests) ... ok test_bug_612074 (__main__.ReTests) ... ok test_bug_725106 (__main__.ReTests) ... ok test_bug_725149 (__main__.ReTests) ... ok test_bug_764548 (__main__.ReTests) ... ok test_category (__main__.ReTests) ... ok test_constants (__main__.ReTests) ... ok test_expand (__main__.ReTests) ... ok test_finditer (__main__.ReTests) ... ok test_flags (__main__.ReTests) ... ok test_getattr (__main__.ReTests) ... ok test_getlower (__main__.ReTests) ... ok test_groupdict (__main__.ReTests) ... ok test_ignore_case (__main__.ReTests) ... ok test_non_consuming (__main__.ReTests) ... ok test_not_literal (__main__.ReTests) ... ok test_pickling (__main__.ReTests) ... ok test_qualified_re_split (__main__.ReTests) ... ok test_qualified_re_sub (__main__.ReTests) ... ok test_re_escape (__main__.ReTests) ... ok test_re_findall (__main__.ReTests) ... ok test_re_groupref (__main__.ReTests) ... ok test_re_groupref_exists (__main__.ReTests) ... ok test_re_match (__main__.ReTests) ... ok test_re_split (__main__.ReTests) ... ok test_re_subn (__main__.ReTests) ... ok test_repeat_minmax (__main__.ReTests) ... ok test_scanner (__main__.ReTests) ... ok test_search_coverage (__main__.ReTests) ... ok test_search_star_plus (__main__.ReTests) ... ok test_special_escapes (__main__.ReTests) ... ok test_sre_character_literals (__main__.ReTests) ... ok test_stack_overflow (__main__.ReTests) ... ERROR test_symbolic_refs (__main__.ReTests) ... ok ====================================================================== ERROR: test_bug_418626 (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_re.py", line 410, in test_bug_418626 self.assertEqual(re.search('(a|b)*?c', 10000*'ab'+'cd').end(0), 20001) File "C:\CODE\23\lib\sre.py", line 137, in search return _compile(pattern, flags).search(string) RuntimeError: maximum recursion limit exceeded ====================================================================== ERROR: test_stack_overflow (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "../lib/test/test_re.py", line 420, in test_stack_overflow self.assertEqual(re.match('(x)*', 50000*'x').group(1), 'x') File "C:\CODE\23\lib\sre.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded ---------------------------------------------------------------------- Ran 46 tests in 0.550s FAILED (errors=2)
I have a bunch of compatibility fixes that I'd like to work on. I'm also considering switching to the newer version of autoconf.
A newer & buggier version, or a newer & better version <wink>?
It looks like someone have backported the changes done in test_re.py. These tests were expected to fail with the SRE from 2.3. -- Gustavo Niemeyer http://niemeyer.net
[Gustavo Niemeyer]
It looks like someone have backported the changes done in test_re.py. These tests were expected to fail with the SRE from 2.3.
The tests are never expected to fail, so I think you mean that test_re in 2.3 should expect (and suppress) the RuntimeError in these cases. It looks like Anthony changed this most recently: test_re.py Revision 1.45.6.1 Tue Nov 4 14:11:01 2003 UTC (2 weeks, 3 days ago) by anthonybaxter Branch: release23-maint Changes since 1.45: +9 -7 lines get tests working again. partial backport of 1.46 - I fixed the recursive tests that used to fail, but left test_re_groupref_exists disabled, as it fails on the release23-maint branch. Maybe something else needs to be backported? We've got more than one problem here, then, because Barry reports that test_re on release23-maint, as it exists today, does *not* fail on a RedHat 9 build. So if Anthony reverted that change, test_re would pass again on Windows, but would start to fail on RH9.
Yes, that's what I meant. Sorry for not being clear.
Yes, he seems to belive that the new SRE scheme was introduced in 2.3, but these tests should still expect RuntimeError in 2.3.
That's strange indeed. Either other changes were introduced in 2.3 which changed the number of recursions, what I don't belive to be the case, or the fixed recursion limit was raised in that platform. -- Gustavo Niemeyer http://niemeyer.net
Gustavo Niemeyer <niemeyer@conectiva.com> writes:
Yes, he seems to belive that the new SRE scheme was introduced in 2.3, but these tests should still expect RuntimeError in 2.3.
I was under the impression (and slightly alarmed) that the recursion removal gimmicks had been backported from the trunk to the release23-maint branch. Was that not the case? (If so, phew). If that *wasn't* the case, then why were the tests failing for Anthony before he made that checkin? Cheers, mwh -- Or here's an even simpler indicator of how much C++ sucks: Print out the C++ Public Review Document. Have someone hold it about three feet above your head and then drop it. Thus you will be enlightened. -- Thant Tessman
FWIW, I'm having much more problems with 2.3cvs on RH7.3. test_re.py core dumps for me for instance. I'm doing a fresh build --with-pydebug and will try to get more information. -Barry
[Barry]
FWIW, I'm having much more problems with 2.3cvs on RH7.3. test_re.py core dumps for me for instance.
For Guido too.
I'm doing a fresh build --with-pydebug and will try to get more information.
It's one of two things: USE_RECURSION_LIMIT isn't #define'd or USE_RECURSION_LIMIT is #define'd, but to a value too large for that box There's a maze of #ifdef'ery near the start of _sre.c setting USE_RECURSION_LIMIT differently for different platforms. Windows doesn't use USE_RECURSION_LIMIT -- it uses a different gimmick based on being able to test for C stack overflow directly on Windows. test_re.py *should*, at this time, fail in exactly the same ways I reported it failing on Windows.
On Fri, 2003-11-21 at 13:09, Tim Peters wrote:
test_re.py *should*, at this time, fail in exactly the same ways I reported it failing on Windows.
Then Something Else is going on. As reported in another message, it doesn't fail for me on either RH9 or RH7.3, and a fresh debug build on RH7.3 also doesn't crash for me either. -Barry
On Fri, 21 Nov 2003, Barry Warsaw wrote:
sre in 2.3x is compiler sensitive - the stack frame size becomes critical in how many sre recursions are supported, and a core dump is certain if the sre recursion limit is more than the available stack space allows. Threads support may be mixed in with this, as the size of the stack for the primary or initial thread is what gets exercised by test_re. On FreeBSD 4.x the stack size for this thread is fixed at 1MB (pthreads implementation limitation, not OS limit). gcc versions < 3.0 don't cause problems with the default sre recursion limit of 10000, but later versions do. So I'd suggest trying a lower sre recursion limit to see whether this helps. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia
Never mind. A fresh debug build, test -u all yields no problems with 2.3cvs on RH7.3 either. -Barry
[Barry Warsaw]
Never mind. A fresh debug build, test -u all yields no problems with 2.3cvs on RH7.3 either.
test_re.py isn't supposed to pass on 2.3 maint today. If it passed, it's broken, and will start to fail as soon as the breakage is repaired. Find out what USE_RECURSION_LIMIT is set to on that box.
On Fri, 2003-11-21 at 13:13, Tim Peters wrote:
Is it possible that USE_RECURSION_LIMIT isn't defined for my RH builds?! I added the attached little bit of (seemingly useful) code to _sre.c, recompiled and then... % ./python Python 2.3.3a0 (#4, Nov 21 2003, 13:39:39) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
Very odd. -Barry
[Barry Warsaw]
Is it possible that USE_RECURSION_LIMIT isn't defined for my RH builds?!
I can't see how: it's set by a giant maze of #ifdef's, which are almost as reliable as a giant maze of CVS branches <wink>. Because the #ifdef's nest 4 deep at one point, and the bodies aren't indented, it's damned hard to figure out what they're doing by eyeball. But I *think* this part: """ #else #define USE_RECURSION_LIMIT 10000 #endif #endif #endif """ which gives all the appearance of defining a default value (if nothing else triggers), is actually nested *inside* an #elif defined(__FreeBSD__) block (which is in turn nested in a !defined(USE_STACKCHECK) block, which is in turn nested in an ifndef SRE_RECURSIVE block). God only knows what the intent was. But I expect that, yes, USE_RECURSION_LIMIT isn't getting defined on anything other than FreeBSD and Win64.
OTOH, if you believe what it says, that leads directly to the cause <wink>.
On Fri, 2003-11-21 at 14:22, Tim Peters wrote:
Yep, you're right. If I hack _sre.c with the patch below, I think I get something closer to what we expect to see. % ./python Python 2.3.3a0 (#5, Nov 21 2003, 14:26:25) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
... ====================================================================== ERROR: test_bug_418626 (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_re.py", line 410, in test_bug_418626 self.assertEqual(re.search('(a|b)*?c', 10000*'ab'+'cd').end(0), 20001) File "/home/barry/projects/python23/Lib/sre.py", line 137, in search return _compile(pattern, flags).search(string) RuntimeError: maximum recursion limit exceeded ====================================================================== ERROR: test_stack_overflow (__main__.ReTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "Lib/test/test_re.py", line 420, in test_stack_overflow self.assertEqual(re.match('(x)*', 50000*'x').group(1), 'x') File "/home/barry/projects/python23/Lib/sre.py", line 132, in match return _compile(pattern, flags).match(string) RuntimeError: maximum recursion limit exceeded I'll leave it to someone else to check in the proper fix. (But does anybody else like exposing RECURSION_LIMIT in the _sre module?) -Barry
[Barry]
Yup, that's how they fail on Windows today, and is how they're *expected* to fail everywhere today.
I'll leave it to someone else to check in the proper fix.
I expect Anthony has the best shot at understanding why he did what he did before, so has the best shot at undoing it too without creating more new problems.
(But does anybody else like exposing RECURSION_LIMIT in the _sre module?)
For 2.3 maint it would be a new feature, so probably not. For 2.4, I believe all this code has become a mass of decoys (that is, it's still there, but is no longer used; I don't know why it hasn't been deleted) -- Gustavo reworked sre to stop using C-level recursion. BTW, Gustavo, we get a big pile of compiler warnings on the trunk (2.4 development) in _sre.c now, on Windows, and apparently under some-but-not-all gcc flavors. How about cleaning those up? See: http://mail.python.org/pipermail/python-dev/2003-October/039059.html
Thanks for pointing me this. I'll manage to clean these issues (the code and the warnings) ASAP. -- Gustavo Niemeyer http://niemeyer.net
Sorry - I (and a bunch of other folks, Alex included if I recall correctly) was seeing a bunch of test failures in test_re - I ported the "fixed" tests from the trunk, in the assumption that the relevant change had been made to the branch. I'll undo it, once someone's fixed _sre in the branch to be broken again <wink> Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Sat, 2003-11-22 at 00:10, Jeremy Hylton wrote:
Unfortunately, no so for me: test_mimetypes test test_mimetypes failed -- Traceback (most recent call last): File "/home/barry/projects/python23/Lib/test/test_mimetypes.py", line 52, in test_guess_all_types eq(all, ['.bat', '.c', '.h', '.ksh', '.pl', '.txt']) File "/home/barry/projects/python23/Lib/unittest.py", line 302, in failUnlessEqual raise self.failureException, \ AssertionError: ['.asc', '.bat', '.c', '.h', '.ksh', '.pl', '.txt'] != ['.bat', '.c', '.h', '.ksh', '.pl', '.txt'] But we've seen these before, right? Doesn't some test interfere with globals in a way that screws mimetypes occasionally? -Barry
Did a cvs update about 30 minutes ago. make test reports no errors. Running again with "-u all -r" to see what happens.
Also looks good. This was with a RH9 system.
[Barry Warsaw]
googling on test_guess_all_types nails it: http://mail.python.org/pipermail/python-dev/2003-September/038264.html Jeff Epler reported there, in a reply to you about the same thing in 2.3.1, that test_urllib2 interferes with test_mimetypes (when run in that order), and included a patch claimed to fix it. Of course, since he didn't put the patch on SF, it just got lost.
On Sat, 2003-11-22 at 07:59, Barry Warsaw wrote:
Yes and yes. Use of mimetypes causes the module's init() function to be run on a set of known files. test_mimetypes calls init() after zapping the list of knownfiles. init() does not clear out existing global state before re-initializing, which is why the test fails if mimetypes has been used before test_mimetypes. Jeremy
>> This needs fresh testing on all non-Win32 platforms ... >> Running the standard test_re.py is an adequate test. >> >> So start testing, or (my recommendation) upgrade to Win32 <wink>. Jeremy> Did a cvs update about 30 minutes ago. make test reports no Jeremy> errors. Running again with "-u all -r" to see what happens. "regrtest.py -u all -r" worked for me on Mac OS X. Skip
Tim> So ... upgrade to Win32 <wink>. I'll consider that after you've been in charge of software development at Microsoft for a couple years. Skip
[...]
It looks to be this patch's fault: ------------- From: loewis@users.sourceforge.net To: python-checkins@python.org Cc: Bcc: Subject: [Python-checkins] python/dist/src/Modules _sre.c,2.99,2.99.8.1 Reply-To: python-dev@python.org Update of /cvsroot/python/python/dist/src/Modules In directory sc8-pr-cvs1:/tmp/cvs-serv28127 Modified Files: Tag: release23-maint _sre.c Log Message: Patch #813391: Reduce limits for amd64 and sparc64. Index: _sre.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/_sre.c,v retrieving revision 2.99 retrieving revision 2.99.8.1 diff -C2 -d -r2.99 -r2.99.8.1 *** _sre.c 26 Jun 2003 14:41:08 -0000 2.99 --- _sre.c 20 Oct 2003 20:59:45 -0000 2.99.8.1 *************** *** 72,78 **** /* FIXME: maybe the limit should be 40000 / sizeof(void*) ? */ #define USE_RECURSION_LIMIT 7500 - #else ! #if defined(__GNUC__) && defined(WITH_THREAD) && defined(__FreeBSD__) /* the pthreads library on FreeBSD has a fixed 1MB stack size for the * initial (or "primary") thread, which is insufficient for the default --- 72,83 ---- /* FIXME: maybe the limit should be 40000 / sizeof(void*) ? */ #define USE_RECURSION_LIMIT 7500 ! #elif defined(__FreeBSD__) ! /* FreeBSD/amd64 and /sparc64 require even smaller limits */ ! #if defined(__amd64__) ! #define USE_RECURSION_LIMIT 6000 ! #elif defined(__sparc64__) ! #define USE_RECURSION_LIMIT 3000 ! #elif defined(__GNUC__) && defined(WITH_THREAD) /* the pthreads library on FreeBSD has a fixed 1MB stack size for the * initial (or "primary") thread, which is insufficient for the default _______________________________________________ Python-checkins mailing list Python-checkins@python.org http://mail.python.org/mailman/listinfo/python-checkins -- Gustavo Niemeyer http://niemeyer.net
[martin@v.loewis.de]
Sorry for causing so much confusion, and thanks to Tim for fixing it.
It's OK, Martin! It was a wonderful example of a simple mistake getting misdiagnosed and so leading to further mistakes, until the whole was much more confusing than the sum of its parts. And, as always, the root cause was trying to cover up Unix bugs with C's preprocessor <wink>.
"Tim Peters" <tim.one@comcast.net> writes:
I'm not expecting to be around much in between those dates, but could do some work for the RC with those dates.
What else does 2.3.3 need?
There are a bunch of build problems which my brain, sadly but not surprisingly, has thoroughly paged out. We should give the new autoconf a go, at least. Cheers, mwh -- "Well, the old ones go Mmmmmbbbbzzzzttteeeeeep as they start up and the new ones go whupwhupwhupwhooopwhooooopwhooooooommmmmmmmmm." -- Graham Reed explains subway engines on asr
Michael Hudson wrote We should give the new autoconf a go, at least.
I would strongly prefer to do this sooner than later, so I was thinking of doing the upgrade sometime this week. Does anyone have/know any reasons to not upgrade to the newer autoconf? It should fix a bunch of build annoyances (and I can get rid of aclocal.m4) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
Anthony Baxter <anthony@interlink.com.au> writes:
Well, there was an almost instananeous brown-paper-bag 2.59 release, but I don't know of any problems with 2.59. Not sure I would if there were, mind.
It should fix a bunch of build annoyances (and I can get rid of aclocal.m4)
That's the motivation :-) Cheers, mwh -- Make this IDLE version 0.8. (We have to skip 0.7 because that was a CNRI release in a corner of the basement of a government building on a planet circling Aldebaran.) -- Guido Van Rossum, in a checkin comment
Tim> What say we get 2.3.3 in motion? As long as a primary motivator for a 2.3.3 release seems to be weakref-related, perhaps someone who's familiar enough with their usage could beef up the docs enough to get rid of this comment at the top of the module doc: XXX -- need to say more here! I was motivated to take a look at the weakref docs for the first time after Tim mentioned: Casual programmers aren't likely to use weakrefs at all, but once you've built a cache based on weakrefs in a large app, weakrefs become critical to your code and your design. Skip
participants (10)
-
Andrew MacIntyre -
Anthony Baxter -
Barry Warsaw -
Gustavo Niemeyer -
Jeremy Hylton -
martin＠v.loewis.de -
Michael Hudson -
Skip Montanaro -
Thomas Heller -
Tim Peters