Mailman 3 January 2006 - Python-Dev

Including zlib...
by skip＠pobox.com 03 Jan '06

03 Jan '06

Martin checked in zlib to the Python svn repository. Are we really sure that including zlib is the only path to whatever it is that it achieves? If security holes in zlib turn up (they have in the past), new Python releases will have to be released quickly. Skip

5 6

Re: [Python-Dev] [Python-checkins] commit of r41906 - python/branches/ssize_t/Objects/obmalloc.c
by Tim Peters 03 Jan '06

03 Jan '06

> Author: martin.v.loewis > Date: Tue Jan 3 14:16:53 2006 > New Revision: 41906 > > Modified: > python/branches/ssize_t/Objects/obmalloc.c > Log: > Disable 32-bit size limitation for 64-bit mode. > > > Modified: python/branches/ssize_t/Objects/obmalloc.c > ============================================================================== > --- python/branches/ssize_t/Objects/obmalloc.c (original) > +++ python/branches/ssize_t/Objects/obmalloc.c Tue Jan 3 14:16:53 2006 > @@ -1005,6 +1005,8 @@ > > bumpserialno(); > total = nbytes + 16; > +#if SIZEOF_SIZE_T < 8 > + /* XXX do this check only on 32-bit machines */ > if (total < nbytes || (total >> 31) > 1) { > /* overflow, or we can't represent it in 4 bytes */ > /* Obscure: can't do (total >> 32) != 0 instead, because > @@ -1013,6 +1015,7 @@ > size_t is an unsigned type. */ > return NULL; > } > +#endif This checkin should be reverted for now. It's in _PyObject_DebugMalloc, and at present the layout of the extra debug info in a PYMALLOC_DEBUG build is hard-coded to use 4-byte fields, no matter what sizeof(size_t) may be. One of the extra fields recorded in a PYMALLOC_DEBUG build is the number of bytes requested, and at present it's simply not capable of recording a value that doesn't fit in 4 bytes. Even after (if ever ;-)) this is changed to support recording 8-byte values on a box where sizeof(size_t) == 8, the "total < nbytes" part of the test would still be appropriate: PyObject_DebugMalloc requests more memory (`total`) than the user asked for (`nbytes`), and the computation of `total` may have overflowed. That's what "total < nbytes" is checking, and that's the right way to spell the overflow check on any box.

2 3

Re: [Python-Dev] [Python-checkins] commit of r41880 - python/trunk/Python/Python-ast.c
by Barry Warsaw 03 Jan '06

03 Jan '06

I think we have a fundamental problem with Python-ast.c and Python-ast.h. These files should not be both auto-generated and checked into Subversion. The problem is that if you do a "make distclean", these files will get removed and "svn stat" will give you a ! flag. Of course, you can "svn up" to get them back, but that just masks the problem. Worse, if you distclean, then configure, then make, it is possible these files will change. Say you're working on a patch that touches a lot of files and you do an "svn commit". It's certainly possible that you'll accidentally commit new versions of the Python-ast.* files, possibly even broken ones. The general rule should be that no file that is ever generated can be checked into Subversion. Probably the right approach is to check in a template file that will not get removed by a distclean, and modify the build process to generate Python-ast.* from those template files. -Barry

6 6

a quit that actually quits
by Fredrik Lundh 03 Jan '06

03 Jan '06

sourceforge just went off the air, so I'm posting this patch here, in order to distract you all from Christian's deque thread. this silly little patch changes the behaviour of the interpreter so that "quit" and "exit" actually exits the interpreter. it does this by installing a custom excepthook that looks for NameErrors at the top level, in interactive mode only. whaddya think? </F> Index: Lib/site.py =================================================================== --- Lib/site.py (revision 41831) +++ Lib/site.py (working copy) @@ -60,6 +60,7 @@ import sys import os import __builtin__ +import traceback def makepath(*paths): @@ -222,19 +223,20 @@ def setquit(): - """Define new built-ins 'quit' and 'exit'. - These are simply strings that display a hint on how to exit. + """Set default exception handler for the interactive mode.""" + def defaultexcepthook(exc_type, exc_value, exc_info): + if hasattr(sys, "ps1"): + # interactive mode + if isinstance(exc_value, NameError) and not exc_info.tb_next: + text = exc_value[0] + if (text == "name 'exit' is not defined" or + text == "name 'quit' is not defined"): + # XXX: print helpful "Use control-D etc" message here? + raise SystemExit + # XXX: add if text == "help" ? + traceback.print_exception(exc_type, exc_value, exc_info) + sys.excepthook = defaultexcepthook - """ - if os.sep == ':': - exit = 'Use Cmd-Q to quit.' - elif os.sep == '\\': - exit = 'Use Ctrl-Z plus Return to exit.' - else: - exit = 'Use Ctrl-D (i.e. EOF) to exit.' - __builtin__.quit = __builtin__.exit = exit - - class _Printer(object): """interactive prompt objects for printing the license text, a list of contributors and the copyright notice."""

28 79

API changes
by Thomas Heller 03 Jan '06

03 Jan '06

The ast-branch merge apparently changed some formerly public functions to macros. The two that I found out about are PyRun_SimpleString and PyRun_InteractiveLoop, I have not checked if there are more or not. This breaks apps which dynamically link at runtime to the Python dll (the latest py2exe does this). Was this change intentional, or can it be reverted? Thanks, Thomas

2 4

mac memory leaks
by Neal Norwitz 03 Jan '06

03 Jan '06

2 1

automated builds and tests
by Neal Norwitz 03 Jan '06

03 Jan '06

I've updated the script that builds python and the docs, runs the tests, and copies everything up to docs.python.org. Currently, this only works for the trunk. I'm not sure if there's much of a benefit to set this up for branch(es) also. The docs are here: http://docs.python.org/dev/ The results of the build are here: http://docs.python.org/dev/results/ The script is checked in to Misc/build.sh. If the script is modified, it doesn't pick up the changes until the next run. There is a hook to pickup the changes on each run, but I didn't finish that. Look for one of the several FIXMEs. A brief description of what the script does is: svn up ; ./configure ; make ; make test ; make install ; cd Doc ; make ; rsync Below is a description of everything that's done in the script. Most commands are checked for errors and the results are emailed iff there are problems. Right now I'm the only one that gets spammed. This ran all last week without any problems. I made a bunch of changes and probably sometime this week, I'll change the mail to go to python-checkins. svn update svn status ./configure --prefix=$INSTALL_DIR --with-pydebug make | grep warning if make fails, goto building docs make install make test | grep fail ./python ./Lib/test/regrtest.py -R 4:3:$REFLOG -u network | grep leak ./python -E -tt ./Lib/test/regrtest.py -uall -x test_curses test_linuxaudiodev test_ossaudiodev | grep fail cd Doc ; make rsync doc rsync build results --- test_curses is ignored because this script is run from cron and there is no terminal. The test stops prematurely. The sound tests are ignored because the sound device is not configured on this computer. Some of the failure counts are off because it does a simple grep and make test runs everything twice. I've generally tried to filter out stuff we don't care about. There is potential for the tests to fail due to them running at the same time as the buildbot test run. These tests are run every 12 hours. It currently takes over 1.25 hours to build and run everything. There is no make clean step for either the code or doc, perhaps there should be? n

1 0

current test problems
by Neal Norwitz 03 Jan '06

03 Jan '06

test_bsddb3 fails for me with BSD DB 4.1 and 4.2 on gentoo Linux. I think it has for a long time. Is anyone else seeing these failures? You must use -u bsddb when running regrtest. 4.2 has 2 failures: test02_associateAfterDB (bsddb.test.test_associate.ShelveAssociateRecnoTestCase) test02_associateAfterDB (bsddb.test.test_associate.ThreadedAssociateRecnoTestCase) 4.1 fails like this: Traceback (most recent call last): File "Lib/threading.py", line 444, in __bootstrap self.run() File "Lib/threading.py", line 424, in run self.__target(*self.__args, **self.__kwargs) File "Lib/bsddb/test/test_thread.py", line 275, in readerThread rec = dbutils.DeadlockWrap(c.next, max_retries=10) File "Lib/bsddb/dbutils.py", line 62, in DeadlockWrap return function(*_args, **_kwargs) DBLockDeadlockError: (-30996, 'DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock') When looking for leaks, the following tests fail: test_decimal test_difflib test_optparse test_warnings test_zipimport To test: ./python -E -tt ./Lib/test/regrtest.py -R 3:2: \ test_decimal test_difflib test_optparse test_warnings test_zipimport The following test has actual leaks which was reported a while ago. Armin found the problem to be that itertools.tee needs to participate with GC IIRC: test_generators The following tests do not report stable leaks (at least sometimes): test_capi test_cfgparser test_charmapcodec test_cmd_line test_compiler test_filecmp test_logging test_threadedtempfile test_threading test_threading_local test_urllib2 n

1 0

Re: [Python-Dev] When do sets shrink?
by Raymond Hettinger 01 Jan '06

01 Jan '06

[Noam] > For example, iteration over a set which once had > 1,000,000 members and now has 2 can take 1,000,000 operations every > time you traverse all the (2) elements. Do you find that to be a common or plausible use case? Was Guido's suggestion of s=set(s) unworkable for some reason? dicts and sets emphasize fast lookups over fast iteration -- apps requiring many iterations over a collection may be better off converting to a list (which has no dummy entries or empty gaps between entries). Would the case be improved by incurring the time cost of 999,998 tests for possible resizing (one for each pop) and some non-trivial number of resize operations along the way (each requiring a full-iteration over the then current size)? Even if this unique case could be improved, what is the impact on common cases? Would a downsizing scheme risk thrashing with the over-allocation scheme in cases with mixed adds and pops? Is there any new information/research beyond what has been obvious from the moment the dict resizing scheme was born? Raymond

5 7

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 01 Jan '06

01 Jan '06

Patch / Bug Summary ___________________ Patches : 382 open ( +3) / 3003 closed ( +1) / 3385 total ( +4) Bugs : 903 open (-11) / 5479 closed (+27) / 6382 total (+16) RFE : 203 open ( -1) / 195 closed ( +2) / 398 total ( +1) New / Reopened Patches ______________________ NotImplemented->TypeError in __add__ and __mul__ (2005-12-26) CLOSED http://python.org/sf/1390657 opened by Armin Rigo dict.merge (2005-12-27) http://python.org/sf/1391204 opened by Nicolas Lehuen cookielib LWPCookieJar and MozillaCookieJar exceptions (2005-02-06) http://python.org/sf/1117398 reopened by jjlee Optional second argument for startfile (2005-12-29) http://python.org/sf/1393157 opened by Thomas Heller Add restart debugger command to pdb.py (2005-12-30) http://python.org/sf/1393667 opened by Rocky Bernstein Patches Closed ______________ NotImplemented->TypeError in __add__ and __mul__ (2005-12-26) http://python.org/sf/1390657 closed by arigo weakref callbacks are called only if the weakref is alive (2005-12-12) http://python.org/sf/1379023 closed by arigo New / Reopened Bugs ___________________ Incorrectly docs for return values of set update methods (2005-12-24) CLOSED http://python.org/sf/1389673 opened by Collin Winter Fxn call in _elementtree.c has incorrect signedness (2005-12-24) CLOSED http://python.org/sf/1389809 opened by Brett Cannon _winreg specifies EnvironmentError instead of WindowsError (2005-12-21) CLOSED http://python.org/sf/1386675 reopened by birkenfeld ScrolledText hungs up in some conditions (2005-12-25) http://python.org/sf/1390086 opened by dani_filth README mention --without-cxx (2005-12-25) http://python.org/sf/1390321 opened by Aahz time docs lack %F in strftime! (2005-12-26) CLOSED http://python.org/sf/1390605 opened by Nikos Kouremenos split() breaks no-break spaces (2005-12-26) CLOSED http://python.org/sf/1390608 opened by MvR time.strftime('%F', local_time) is okay but time.strptime no (2005-12-26) CLOSED http://python.org/sf/1390629 opened by Nikos Kouremenos lambda functions confused when mapped in dictionary object (2005-12-27) CLOSED http://python.org/sf/1390991 opened by Samuel Hsiung missing module names in email package (2005-12-27) http://python.org/sf/1391608 opened by Gabriel Genellina floating point literals don't work in non-US locale in 2.5 (2005-12-28) http://python.org/sf/1391872 opened by Fredrik Lundh build fails on BSD 3.8 (2005-12-30) http://python.org/sf/1392915 opened by George Yoshida cannot build SVN trunk on old systems (2005-12-29) http://python.org/sf/1393109 opened by Fredrik Lundh Deleting first item causes anydbm.first() to fail (2005-12-30) http://python.org/sf/1394135 opened by Dan Bisalputra urllib2 raises exception when page redirects to itself (2005-12-31) CLOSED http://python.org/sf/1394453 opened by RenÃ© Pijlman SimpleHTTPServer doesn't understand query arguments (2005-12-31) http://python.org/sf/1394565 opened by Aaron Swartz 'Plus' filemode exposes uninitialized memory on win32 (2005-12-31) http://python.org/sf/1394612 opened by Cory Dodt Bugs Closed ___________ Decimal sqrt() ignores rounding (2005-12-23) http://python.org/sf/1388949 closed by facundobatista Incorrect docs for return values of set update methods (2005-12-24) http://python.org/sf/1389673 closed by birkenfeld Fxn call in _elementtree.c has incorrect signedness (2005-12-25) http://python.org/sf/1389809 closed by effbot _winreg specifies EnvironmentError instead of WindowsError (2005-12-21) http://python.org/sf/1386675 closed by birkenfeld time docs lack %F in strftime! (2005-12-26) http://python.org/sf/1390605 closed by birkenfeld split() breaks no-break spaces (2005-12-26) http://python.org/sf/1390608 closed by lemburg time.strftime('%F', local_time) is okay but time.strptime no (2005-12-26) http://python.org/sf/1390629 closed by birkenfeld metaclasses, __getattr__, and special methods (2003-04-29) http://python.org/sf/729913 closed by arigo special methods become static (2004-11-15) http://python.org/sf/1066490 closed by arigo len() on class broken (2005-12-16) http://python.org/sf/1382740 closed by arigo urllib.url2pathname, pathname2url doc strings inconsistent (2002-12-07) http://python.org/sf/649974 closed by birkenfeld PyLong_AsVoidPtr()/PyLong_FromVoidPtr() (2002-12-14) http://python.org/sf/653542 closed by birkenfeld Acrobat Reader 5 compatibility (2003-04-14) http://python.org/sf/721160 closed by birkenfeld Calling socket.recv() with a large number breaks (2003-06-17) http://python.org/sf/756104 closed by birkenfeld Automated daily documentation builds (2002-06-26) http://python.org/sf/574241 closed by birkenfeld String formatting operator % badly documented (2003-11-10) http://python.org/sf/839585 closed by birkenfeld Document that highly recursive data cannot be pickled (2003-11-10) http://python.org/sf/839075 closed by birkenfeld os.exec* and first 'arg' (2003-11-19) http://python.org/sf/845342 closed by birkenfeld Error in urllib2 Examples (2004-05-16) http://python.org/sf/954981 closed by birkenfeld Lib/profile.doc should be updated (2001-12-05) http://python.org/sf/489256 closed by birkenfeld exec statement link in index broken (2005-12-19) http://python.org/sf/1385004 closed by birkenfeld Tutorial errors (2005-12-07) http://python.org/sf/1375599 closed by birkenfeld Dict docstring error Python-2.3.5 (2005-02-26) http://python.org/sf/1152424 closed by birkenfeld lambda functions confused when mapped in dictionary object (2005-12-27) http://python.org/sf/1390991 closed by birkenfeld Can't build Python on POSIX w/o $HOME (2004-05-24) http://python.org/sf/959576 closed by birkenfeld Setup.local ignored by setup.py (2004-07-29) http://python.org/sf/999767 closed by birkenfeld PyXxx_Check(x) trusts x->ob_type->tp_mro (2005-02-27) http://python.org/sf/1153075 closed by arigo urllib2 raises exception when page redirects to itself (2005-12-31) http://python.org/sf/1394453 closed by rpijlman New / Reopened RFE __________________ tempfile misses usecase which requirs renaming (2005-12-25) http://python.org/sf/1390197 opened by Dennis Allison RFE Closed __________ split() string method has two splitting algorithms (2005-11-28) http://python.org/sf/1367936 closed by rhettinger future warning in commets (2005-02-18) http://python.org/sf/1144057 closed by mcherm

1 0