Skip, [Sorry, this post is rushed...] On Thu, Jul 17, 2003 at 04:47:50PM -0500, Skip Montanaro wrote:

Hmmm... Some thoughts which come to my mind:

* Do I need a cygwin update?

Most likely not.

How can I tell if I do or not?

You can rerun Cygwin's setup.exe and it will inform you whether or not there are newer packages available.

* Would having installed mingw and msys the other day have polluted the environment in some way?

Very unlike. Note that I'm seeing the same (or at least very similar) behavior on my setup too. BTW, before updating Python's CVS today, I ran the Python regression test against the latest official Cygwin packages, but a fairly stale version of Python CVS. I got a couple of failures but no crashes. Next, I updated to the latest Python CVS, built, and reran the regression test. Then, I started to get crashes like the ones you observed. Maybe the latest Python CVS is tickling some lurking Cygwin bug? I hope to have better information tomorrow. Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

[Jason Tishler]

I get the following:

$ ./python.exe -E -tt ../Lib/test/regrtest.py -l -x test_poll \ test_popen test_popen2 test_pty test_queue test_select test_signal \ test_socket test_tempfile test_thread test_grammar [snip] test_zlib 210 tests OK. 4 tests failed: test___all__ test_csv test_netrc test_time 31 tests skipped: test_aepack test_al test_bsddb185 test_bsddb3 test_cd test_cl test_curses test_dbm test_email_codecs test_gl test_imgfile test_ioctl test_largefile test_linuxaudiodev test_locale test_macfs test_macostools test_mpz test_nis test_normalization test_ossaudiodev test_pep277 test_plistlib test_scriptpackages test_socketserver test_sunaudiodev test_timeout test_unicode_file test_urllibnet test_winreg test_winsound 1 skip unexpected on cygwin: test_ioctl

Note that the excluded tests above work fine when run separately:

$ ./python.exe -E -tt ../Lib/test/regrtest.py -l test_poll \ test_popen test_popen2 test_pty test_queue test_select test_signal \ test_socket test_tempfile test_thread test_poll test_popen test_popen2 test_pty test_queue test_select test_signal test_socket test_tempfile test_thread All 10 tests OK.

I appreciate the heads up. I will dig deeper...

I've got a bad feeling that we may have a wild store (or such like) in 2.3. A day or two ago, Jeremy got a senseless and irreproducible error when running a ZODB test after compiling with 2.3 CVS, and just now I tried running regrtest with -r on Win98SE (-r randomizes test case order). That triggered an error I've never seen before and can't reproduce: test test_time failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\CODE\PYTHON\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed. Nothing *screams* "wild store", though. For example, the failure in test_time may be due to some other test mucking with locale but not cleaning up after itself -- I simply don't know, and (worse) don't have time to pour into it. Possible helps: Check out regrtest's -f option (great for doing "binary searches" when a failure depends on running a certain subset of tests in a certain order). Try running the tests under a debug build (pymalloc's debugging armor in 2.3 debug builds catches many problems). If you get a reproducible baffling failure, try disabling cyclic gc (import gc; gc.disable()) to see whether the problem goes away (problems never get pinned on cyclic gc, but cyclic gc is often sensitive to out-of-bounds writes elsewhere).

[ failures in test_time, test_strptime, test_logging ] Tim> These don't smell like wild stores. What happens on your box? I've been unable to reproduce any of this on Mac OS X. I built --with-pydebug, tried those combinations you indicated as well as three regrtest -r runs (one with -uall), but saw nothing. I then returned to my build.O6 directory (OPT=-O6, -DNDEBUG), rebuilt and tried again. During the -r -uall -O6 run I got some assertion errors in the bsddb3 thread stuff: Exception in thread writer 0: Traceback (most recent call last): File "/Users/skip/src/python/head/dist/src/Lib/threading.py", line 436, in __bootstrap self.run() File "/Users/skip/src/python/head/dist/src/Lib/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) File "/Users/skip/src/python/head/dist/src/Lib/bsddb/test/test_thread.py", line 254, in writerThread self.assertEqual(data, self.makeData(key)) File "/Users/skip/src/python/head/dist/src/Lib/unittest.py", line 292, in failUnlessEqual raise self.failureException, \ AssertionError: None != '0002-0002-0002-0002-0002' During both -r -uall runs (debug and non-debug builds) I got a warning during the bsddb3 tests: /Users/skip/src/python/head/dist/src/Lib/bsddb/dbutils.py:67: RuntimeWarning: DB_INCOMPLETE: Cache flush was unable to complete return function(*_args, **_kwargs) If the problem is with the locale code, Mac OS X is the wrong place to debug the problem as its locale support is apparently minimal: test_locale skipped -- Locale support on MacOSX is minimal and cannot be tested I'll try some more cygwin tests tomorrow, though those failures looked a lot different than what you're seeing. Skip

On Thu, Jul 17, 2003 at 11:37:24PM -0400, Tim Peters wrote:

Partly because of this thing, but mostly because we want to give Jason a chance to stare at the Cygwin problems.

Unfortunately, I don't have that much to report: 1. It appears that Cygwin Python is crashing without even generating the normal Cygwin stackdump file. 2. When run inside of gdb and I get reproducible SEGVs when executing test_poll and test_popen (if test_poll is excluded). The SEGVs are inside of Cygwin so I will have to build a debuggable Cygwin DLL before I can dig deeper. At first glance, it doesn't look like Python is passing garbage into Cygwin. See attached for some backtraces, if interested. 3. I am observing the same (or at least similar) behavior that Tim is seeing with Win32 Python. I will try to do some debugging over the weekend but I may not have more until Monday. Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

On Mon, Jul 21, 2003 at 08:11:19AM -0400, Jason Tishler wrote:

I have started to play the binary search game to determine when the problem began.

Ding, ding, ding! We have a winner: http://sf.net/tracker/?group_id=5470&atid=305470&func=detail&aid=742741 The above patch enables HAVE_PTHREAD_SIGMASK under Cygwin which was not enabled before. I just tried the following with 2.3c1: $ configure ... $ # edit pyconfig.h to #undef HAVE_PTHREAD_SIGMASK $ make $ ./python.exe -E -tt ../Lib/test/regrtest.py -l test_grammar [snip] test_zlib 221 tests OK. 2 tests failed: test_netrc test_time 32 tests skipped: test_aepack test_al test_bsddb185 test_bsddb3 test_cd test_cl test_curses test_dbm test_email_codecs test_gl test_imgfile test_ioctl test_largefile test_linuxaudiodev test_locale test_macfs test_macostools test_mpz test_nis test_normalization test_ossaudiodev test_pep277 test_plistlib test_scriptpackages test_socket_ssl test_socketserver test_sunaudiodev test_timeout test_unicode_file test_urllibnet test_winreg test_winsound 1 skip unexpected on cygwin: test_ioctl Scanning the Cygwin code, I found the following: extern "C" int pthread_sigmask (int operation, const sigset_t *set, sigset_t *old_set) { pthread *thread = pthread::self (); // lock this myself, for the use of thread2signal ***> // two differt kills might clash: FIXME [snip] } Hence, it seems that my hypothesis that a Python change was tickling a Cygwin bug (i.e., the above FIXME or another issue) is correct. So, the question is how should we deal with this issue? 1. Leave the Python code base alone and assume that Cygwin's pthread_sigmask() will get fixed. Additionally, assume I will patch around this problem until that happens. 2. Patch the Python code base so that HAVE_PTHREAD_SIGMASK is undefined under Cygwin. I recommend option #1. Do others agree? Did I miss any other options? BTW, does anyone know if Python is triggering the FIXME in Cygwin's pthread_sigmask()? Or, should I look elsewhere? Thanks, Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

On Mon, Jul 21, 2003 at 03:55:31PM -0400, Tim Peters wrote:

[Jason Tishler]

...

So, the question is how should we deal with this issue?

1. Leave the Python code base alone and assume that Cygwin's pthread_sigmask() will get fixed. Additionally, assume I will patch around this problem until that happens. 2. Patch the Python code base so that HAVE_PTHREAD_SIGMASK is undefined under Cygwin.

I recommend option #1. Do others agree?

I don't know. It seems the only effect of HAVE_PTHREAD_SIGMASK is to decide whether Python uses pthread_sigmask() or sigprocmask(). If the latter works but the former doesn't, I would have guessed you'd like to use the latter <wink>.

Yup! But, would such a Cygwin specific change be accepted so close to the release date? This is one of the reasons that I recommended option #1. Any other opinions? Thanks, Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

On Mon, 2003-07-21 at 17:01, Tim Peters wrote:

[Jason Tishler]

...

Yup! But, would such a Cygwin specific change be accepted so close to the release date?

If the Cygwin-specific part is (as it seemed to me) isolated to the only line in the codebase that tests HAVE_PTHREAD_SIGMASK, I think the risk is too small to worry about. In one real sense, the HAVE_PTHREAD_SIGMASK patch introduced bugs, causing a formerly-working platform to fall over.

I'd agree. This isn't hurting any other platform, so hacking configure for this test on cygwin seems of minimal pain. The potential for other OS breakage is smaller the sooner you do it. <wink>

...

This is one of the reasons that I recommended option #1. Any other opinions?

Barry is the release manager again, so only his counts. Well, his and yours. OK, his, yours, and mine. OK, if you want to push it, his, yours, mine and Guido's. That's it, though. Unless you want to count Jeremy too. I suppose we should! OK, the only opinions that count are Skip's, Barry's, yours, Tim's, Jeremy's and Guido's. We shouldn't leave Jack out, should we? For sure, the only opinions that count are Fred's.

The only opinion that matters to me is our new Pylabs boss Frank Lomax, who, while being a genius, smells funny. He says to go for it, but I'm not sure if he was talking about this patch, or about eating the patch of chocolate pudding on his sleeve. 50/50. -Barry

I made a little progress narrowing down the cygwin errors. If I simply run ./python.exe Lib/test/regrtest.py the first test which fails is test_bz2. If I run test_bz2 by itself it succeeds. I created a shell script which just executed pairs of tests, one of the ones which was run before test_bz2 followed by test_bz2. A couple tests caused test_bz2 to fail: test___all__ test_asynchat Here's the error output: C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 44 [main] python 1476 sync_with_child: child 1532(0x18C) died before initialization with status code 0x1 184 [main] python 1476 sync_with_child: *** child state child loading dlls C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 61438 [main] python 1476 sync_with_child: child 896(0x150) died before initialization with status code 0x1 61612 [main] python 1476 sync_with_child: *** child state child loading dlls C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 118777 [main] python 1476 sync_with_child: child 1452(0x114) died before initialization with status code 0x1 118944 [main] python 1476 sync_with_child: *** child state child loading dlls C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 208810 [main] python 1476 sync_with_child: child 1664(0xDC) died before initialization with status code 0x1 208994 [main] python 1476 sync_with_child: *** child state child loading dlls C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 294015 [main] python 1476 sync_with_child: child 1584(0xA0) died before initialization with status code 0x1 294200 [main] python 1476 sync_with_child: *** child state child loading dlls C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 371845 [main] python 1476 sync_with_child: child 1600(0x68) died before initialization with status code 0x1 372041 [main] python 1476 sync_with_child: *** child state child loading dlls test test_bz2 failed -- errors occurred; run in verbose mode for details 1 test OK. 1 test failed: test_bz2 [18388 refs] I then tried the same tactic with another crasher, test_fork1, and discovered these possible culprits: test___all__ test_asynchat test_cgi test_email There's the test_fork1 error output: C:\cygwin\home\Administrator\src\python\dist\src\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0x11B0000) != 0x11C0000 77 [main] python 1504 sync_with_child: child 1452(0x190) died before initialization with status code 0x1 212 [main] python 1504 sync_with_child: *** child state child loading dlls test test_fork1 crashed -- exceptions.OSError: [Errno 11] Resource temporarily unavailable 1 test OK. 1 test failed: test_fork1 [18201 refs] I then went back and tried test_cgi and test_email with test_bz2 (they don't run before test_bz2 in the normal case). Test_bz2 failed when run after either of them. In all cases with both test scenarios, the first test succeeded. I took a quick peek at the code of the possible culprits but didn't see anything which seemed to link them all together. Skip

A little more poking around with Google using the error messages I saw led me to Jason's site: http://www.tishler.net/jason/software/rebase/rebase-2.2.README I installed the rebase package and ran "rebaseall -v" after a fresh reboot from a cmd window running bash, then rebuilt python.exe from scratch. That didn't help with the test_bz2 or test_fork1 failures. regrtest.py also still craps out in test_poll (though not if run by itself), so I tried my pairs-o-tests script again with test_poll as the second test. Nada. test_poll always succeeded, regardless of the result of the preceeding test (success, skip, failure). I then tried (unsuccessfully) another tack. I threw all the test names into a file then ran ./python.exe Lib/test/regrtest.py `head $i tests | tail -20` test_poll for $i ranging from 20 to 230 (the number of lines in the file) in steps of 10. This tickled a fork() problem with test_tempfile in one situation which seems like the thing rebaseall was supposed to fix. I never saw it croak in test_poll though. I'm not really equipped to deal with this, so I'm going to leave the debugging to Jason. If the evidence Jason collects suggests Python is tickling a Cygwin bug I think we should forge ahead with the release candidate unless a workaround is immediately obvious. There's a fairly hard deadline for the final release of July 31 so that Python 2.3 gets in the next version of Mac OS X. Jack, Just and the rest of the MacPython gang have put a lot of effort into Python between 2.2 and 2.3. It would be a shame to see it miss Apple's release train. Skip

Skip, On Fri, Jul 18, 2003 at 02:30:16PM -0500, Skip Montanaro wrote:

I then tried (unsuccessfully) another tack. I threw all the test names into a file then ran

./python.exe Lib/test/regrtest.py `head $i tests | tail -20` test_poll

for $i ranging from 20 to 230 (the number of lines in the file) in steps of 10. This tickled a fork() problem with test_tempfile in one situation which seems like the thing rebaseall was supposed to fix.

The "all" in rebaseall means all DLLs installed by Cygwin's setup.exe. Since your Python build has not been installed yet, its DLLs have not been rebased. Normally, rebasing /usr/bin/*.dll is sufficient. However, sometimes it's not. If you are interested in helping with the Cygwin hunt, then I can show you how to properly rebase the Python CVS build too. Let me know. Otherwise, I'm back to hunt... Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

Skip, On Mon, Jul 21, 2003 at 10:36:47AM -0500, Skip Montanaro wrote:

Jason> Normally, rebasing /usr/bin/*.dll is sufficient. However, Jason> sometimes it's not. If you are interested in helping with the Jason> Cygwin hunt, then I can show you how to properly rebase the Jason> Python CVS build too. Let me know. Otherwise, I'm back to Jason> hunt...

If it's not too much trouble. It's unlikely I'll be able to contribute anything useful though.

Since I think I have isolated the Cygwin Python problem, I would prefer to hold off on the above. My intention is to enhance rebaseall (and in turn rebase) so that rebasing additionally DLLs is easy. Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6

[Tim]

...

I've got a bad feeling that we may have a wild store (or such like) in 2.3. A day or two ago, Jeremy got a senseless and irreproducible error when running a ZODB test after compiling with 2.3 CVS, and just now I tried running regrtest with -r on Win98SE (-r randomizes test case order). That triggered an error I've never seen before and can't reproduce:

test test_time failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\CODE\PYTHON\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed.

[Jason Tishler]

Just to confirm, the above is Win32 Python -- not Cygwin. Right?

Yes, it was Win32 Python -- and also Jeremy's Linux. It "got fixed" by focing the locale to "C" at the end of test_logging.py. That's probably not a correct fix, just a bandaid. Brett is on vacation, and I have no real idea what strptime is trying to do with locale; it appears to be a bug that strptime tries to be locale-independent but that the strptime test in test_time.py fails if test_logging (which changes the locale) isn't hacked to force locale (back?) to "C" at its end.

[Brett C.]

... I can try to see what the problems are if someone can run::

...

...

...

import time time.strftime("%c") import _strptime _strptime.TimeRE()['c']

after running test_logging to trigger the failure.

There is no failure anymore, because Jermey added if cur_locale: locale.setlocale(locale.LC_ALL, "C") to the end of test_logging.py. This almost certainly isn't a correct fix, though (test_logging should restore the locale to what it was before test_logging started just as a matter of cleaning up after itself, but it remains surprising that test_strptime fails if test_logging doesn't). If you revert his change, then, e.g., C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_logging test_strptime test_logging test_strptime test test_strptime failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_strptime.py", line 96, in test_lang "Setting of lang failed") File "C:\Code\python\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError: Setting of lang failed 1 test OK. 1 test failed: test_strptime C:\Code\python\PCbuild> At the point test_strptime fails, on my box the relevant expressions have the following values: self.LT_ins.lang 'English_United States' locale.getdefaultlocale()[0] 'en_US' locale.getlocale(locale.LC_TIME) ['English_United States', '1252'] so self.failUnless(self.LT_ins.lang in (locale.getdefaultlocale()[0], locale.getlocale(locale.LC_TIME), ''), "Setting of lang failed") fails. It doesn't look like the test code expects locale.getlocale(locale.LC_TIME) to return a list, but I don't know what's expected here ...

[Brett]

Argh! The test is wrong. It should be locale.getlocale(locale.LC_TIME)[0] that is being checked against. Crap, sorry to have caused all of this trouble over such a stupid little mistake. I am personally amazed it took so long for the error to show up.

That's OK. There was another failure in test_time, but I don't remember that details anymore.

I can check this minute change in, but what branch/tag/thing do you want it done to?

The head, please.

[Tim]

...

That's OK. There was another failure in test_time, but I don't remember that details anymore.

[Brett]

Is it that tzset test that keeps failing that everyone who has Red Hat 6.2 seems to be about to test at some point today?

No -- see later email.

Done. Should Jeremy's change to test_logging be backed out?

test_logging should clean up after itself as a matter of principle. It wasn't. I'm not sure it is now, though -- hardcoding locale to "C" doesn't necessarily restore the initial value. Nobody seems to understand why test_logging changed the locale to begin with, though! The easiest way for it clean up after itself would be not to do anything in need of cleaning up.

The other error here (which Jeremy also saw on Linux) came from running 3 tests in a particular order. Here I'm running them with Jeremy's locale hack in test_logging reverted (there is no failure when that hack is in place): C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time test test_time failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\Code\python\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed. 2 tests OK. 1 test failed: test_time

[Tim]

...

The other error here (which Jeremy also saw on Linux) came from running 3 tests in a particular order. Here I'm running them with Jeremy's locale hack in test_logging reverted (there is no failure when that hack is in place):

C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time test test_time failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\Code\python\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed.

2 tests OK. 1 test failed: test_time

[Brett C.]

OK. This is where running:

...

...

...

import time time.strftime("%c") import _strptime _strptime.TimeRE()['c']

can help me try to diagnose this (I can't reproduce this under OS X). This will give me what strftime is spitting out and what regex strptime would be using to match against it.

After reverting Jeremy's hack to test_logging, and hacking regrtest.py to stop doing sys.exit(): C:\Code\python\PCbuild>python -i ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time test test_time failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\Code\python\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed. 2 tests OK. 1 test failed: test_time

...

...

import time time.strftime("%c") '07/22/2003 05:44:01 PM' import _strptime _strptime.TimeRE()['c'] '(?P<m>1[0-2]|0[1-9]|[1-9])/(?P<d>3[0-1]|[1-2]\\d|0[1-9]| [1-9]| [1-9])/(?P<y>\\d\\d)\\s*(?P<H>2[0-3]|[0-1]\\d|\\d) :(?P<M>[0-5]\\d|\\d):(?P<S>6[0-1]|[0-5]\\d|\\d)'

Now the same thing with Jeremy's test_logging hack restored: C:\Code\python\PCbuild>python -i ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time All 3 tests OK.

...

...

import time time.strftime("%c") '07/22/03 17:47:26' import _strptime _strptime.TimeRE()['c'] '(?P<m>1[0-2]|0[1-9]|[1-9])/(?P<d>3[0-1]|[1-2]\\d|0[1-9]| [1-9]| [1-9])/(?P<y>\\d\\d)\\s*(?P<H>2[0-3]|[0-1]\\d|\\d) :(?P<M>[0-5]\\d|\\d):(?P<S>6[0-1]|[0-5]\\d|\\d)'

The regexp appears to be the same, but strftime's result has switched from (apparently) 12-hour + AM/PM time to 24-hour time.

Note that I'm on Win98SE now (previous email today was on Win2K). Under current CVS, but with regrtest hacked not to do sys.exit(): C:\Code\python\PCbuild>python -i ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time All 3 tests OK.

...

...

import forbrett time.strftime("%c") '07/22/03 19:11:50'

_strptime.TimeRE()['c'] '(?P<m>1[0-2]|0[1-9]|[1-9])/(?P<d>3[0-1]|[1-2]\\d|0[1-9]| [1-9]| [1-9])/(?P<y>\\d\\d)\\s*(?P<H>2[0-3]|[0-1]\\d|\\d) :(?P<M>[0-5]\\d|\\d):(?P<S>6[0-1]|[0-5]\\d|\\d)' locale.getdefaultlocale() ('en_US', 'cp1252') locale.getlocale(locale.LC_TIME) (None, None)

...

...

Again, but reverting Jeremy's test_logging locale fiddling: C:\Code\python\PCbuild>python -i ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time test test_time failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\CODE\PYTHON\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed. 2 tests OK. 1 test failed: test_time

...

...

import forbrett time.strftime("%c") '07/22/2003 07:14:31 PM'

_strptime.TimeRE()['c'] '(?P<m>1[0-2]|0[1-9]|[1-9])/(?P<d>3[0-1]|[1-2]\\d|0[1-9]| [1-9]| [1-9])/(?P<y>\\d\\d)\\s*(?P<H>2[0-3]|[0-1]\\d|\\d) :(?P<M>[0-5]\\d|\\d):(?P<S>6[0-1]|[0-5]\\d|\\d)' locale.getdefaultlocale() ('en_US', 'cp1252') locale.getlocale(locale.LC_TIME) ['English_United States', '1252']

...

...

This is forbrett.py: """ def dump(tag, value): import pprint print tag pprint.pprint(value) print import time dump('time.strftime("%c")', time.strftime("%c")) import _strptime dump("_strptime.TimeRE()['c']", _strptime.TimeRE()['c']) import locale dump("locale.getdefaultlocale()", locale.getdefaultlocale()) dump("locale.getlocale(locale.LC_TIME)", locale.getlocale(locale.LC_TIME)) """ [Brett C.]

OK, the only thing I can think of is that the locale info has changed ever so subtly and the checking of locale.getlocale(locale.LC_TIME)[0] or locale.getdefaultlocale()[0] is not cutting it. Next thing to try is to see what values locale.getlocale(locale.LC_TIME) and locale.getdefaultlocale() have (notice I am asking for the full value and not just the 0 item) after each test is run. It is possible the cache in _strptime is not being cleared because it is not picking up the change in locale by only checking the 0 item.

If there is not some obvious difference in values then the cache will have to be deleted and strptime performance will drop but the problem will go away.

Barry agrees <wink>.

[Brett C.]

Tim (or someone else who can replicate the problem), can you give the attached patch a try? This should fix the problem. There are new tests on top of a proposed fix. If this doesn't solve it then I am giving up on getting the caching to work for this release.

Sorry, no joy! With the patch, and removing test_logging's restoration of locale, the symptom remains the same: C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_strptime test_logging test_time test_strptime test_logging test_time test test_time failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_time.py", line 49, in test_strptime self.fail('conversion specifier: %r failed.' % format) File "C:\CODE\PYTHON\lib\unittest.py", line 260, in fail raise self.failureException, msg AssertionError: conversion specifier: ' %c' failed. 2 tests OK. 1 test failed: test_time Remove test_strptime, or test_logging, from the mix, and test_time passes. There's a different failure mode if the order of the first two is swapped: C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_logging test_strptime test_time test_logging test_strptime test test_strptime failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_strptime.py", line 433, in test_NoneNone_cl earing self.failUnlessEqual(len(_strptime._regex_cache), 1) File "C:\CODE\PYTHON\lib\unittest.py", line 292, in failUnlessEqual raise self.failureException, \ AssertionError: 4 != 1 test_time 2 tests OK. 1 test failed: test_strptime

On Thu, 2003-07-24 at 02:15, Brett C. wrote:

OK, then I am ripping out the caching mechanism and will have it committed by midnight PT.

Did you update the NEWS file? <wink> -Barry

Tim Peters wrote:

[Brett C.]

...

OK, then I am ripping out the caching mechanism and will have it committed by midnight PT.

It was a mistake to check this in without confirming first that it actually cured the problem. In fact, the same failure still occurs when test_logging's restoration of locale is removed:

Yes, it was a mistake on my part. This was an instance of me becoming very frustrated with myself and just wanting to get this thing solved and stop being the last nagging issue for this release. Sorry to everyone (and especially the plutocratic release team of Tim and Barry) for causing all of this grief.

There's no more time for this, so anything more should wait for 2.3.1.

Well, I think I found the actual problem. Guess what the TimeRE had as its parameter list for __init__ : ``locale_time = LocaleTime()`` . Aaaaaahhhhhh!!!!!! I can't believe I didn't spot that sooner! The damn caching was probably working but every instance was using the default value that is created at module initialization instead of recreating it on every instantiation of TimeRE. Barry OK'ed fixing this and I am still going to leave the caching out. I need to really go over this module and work on it with my new Dutch knowledge of Python for 2.4 to get rid of these nagging newbie mistakes that I initially made in this module. -Brett

Tim Peters wrote:

[Tim "Hairy Thunderer" Peters]

...

...

It was a mistake to check this in without confirming first that it actually cured the problem. In fact, the same failure still occurs when test_logging's restoration of locale is removed:

[Brett C.]

...

Yes, it was a mistake on my part. This was an instance of me becoming very frustrated with myself and just wanting to get this thing solved and stop being the last nagging issue for this release. Sorry to everyone (and especially the plutocratic release team of Tim and Barry) for causing all of this grief.

OK, that's enough groveling -- thanks! Not all mistakes are catastrophes, and this one was pretty minor.

Damn. And I had this poetic apology letter that would have brought everyone to tears. OK, I will stop apologizing for my mistake now.

...

... Well, I think I found the actual problem. Guess what the TimeRE had as its parameter list for __init__ : ``locale_time = LocaleTime()`` . Aaaaaahhhhhh!!!!!! I can't believe I didn't spot that sooner! The damn caching was probably working but every instance was using the default value that is created at module initialization instead of recreating it on every instantiation of TimeRE.

Ah. As most people suspected, then, it was really Guido's fault! Default arguments should only be used as low-level speed hacks to avoid lookups in the builtin namespace.

I know I have learned my lesson: set initial values to None unless it is a constant type for default parameter values.

...

Barry OK'ed fixing this and I am still going to leave the caching out. I need to really go over this module and work on it with my new Dutch knowledge of Python for 2.4 to get rid of these nagging newbie mistakes that I initially made in this module.

You did fine, Brett, and I'm glad we've got _strptime.py. Overall, it was almost a win <wink>. BTW, I can confirm that the goofy locale-related buglets have gone away now on Win2K.

Damn it. Now I have to get something in that is a complete win. =) Glad to hear it's fixed now. -Brett