Mailman 3 February 2008 - Python-Dev

svn repo out of disk?
by Douglas Napoleone 12 Feb '08

12 Feb '08

Not sure if this is effecting the core svn repository or not, but the conference software (also hosted at svn.python.org) is reporting that the disk is full: Transmitting file data ....svn: Commit failed (details follow): svn: Can't create directory '/data/repos/conference/db/transactions/557-1.txn': No space left on device I have not seen anything on the list about this, but then again, there was room at 7pm EST, so I might be the first to notice it. I fear this could be effecting more than just the pycon software, and figured people would want to be forewarned. (Also if someone could fix the problem somehow I would be ever grateful. ;-) -Doug Napoleone

3 3

Confused about test_bsddb failures on Mac OS X
by skip＠pobox.com 11 Feb '08

11 Feb '08

I'm having trouble with test_bsddb on my new MacBook Pro (OS X 10.5.1). Many tests give this error: Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_bsddb.py", line 18, in setUp self.f = self.openmethod[0](self.fname, self.openflag, cachesize=32768) File "/Users/skip/src/python/trunk/Lib/bsddb/__init__.py", line 327, in btopen d.open(file, db.DB_BTREE, flags, mode) DBFileExistsError: (17, 'File exists -- __fop_file_setup: Retry limit (100) exc eeded') In TestBSDDB.setUp I added a print statement to the start: print os.path.abspath(self.fname), os.path.exists(self.fname) It always prints: /Users/skip/src/python/trunk/build/@test False I'm using the MacPorts version of Berkeley DB. I've tried both 4.5 and 4.4: % otool -L build/lib.macosx-10.3-i386-2.6/_bsddb.so build/lib.macosx-10.3-i386-2.6/_bsddb.so: /opt/local/lib/db44/libdb-4.4.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0) Any idea what the problem might be?

3 3

Re: [Python-Dev] int/float freelists vs pymalloc
by Christian Heimes 11 Feb '08

11 Feb '08

Andrew MacIntyre wrote: > For the int case, that patch is slower than no free list at all. For the > float case its very slightly faster (55.3s vs 55.5s) than no free list at > all. Slower than the trunk code for both cases. Did you run the test > scripts on your own machine? I've used a simple timeit call to bench mark the memory allocation. The first statement saturates the free lists and the second one benchmarks memory allocation and float creation. ./python -m timeit -n10 -s "[float(x) for x in list(range(1000))]" "for i in range(100): [float(x) for x in list(range(1000))]" I've done some experiments with a fixed length *free_list[] like dict and listobject.c. A fixed length list is faster than a single linked list. I'm going to upload the patch later. > In addition to the pure performance aspect, there is the issue of memory > utilisation. The current trunk code running the int test case in my > original post peaks at 151MB according to top on my FreeBSD box, dropping > back to about 62MB after the dict is destroyed (without a compaction). > The same script running on the no-freelist build of the interpreter peaks > at 119MB, with a minima of around 57MB. I wonder why the free list has such a huge impact in memory usage. Int objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte value). A thousand int object should consume less than 20kB including overhead and padding. > On the 2 platforms I have, the current trunk code for ints would appear > to be as good as it gets speed-wise. If the memory utilisation issue > were to be considered significant, dropping the int freelist would have > its attractions... > > As far as floats go, all the indications I have suggest that the current > freelist code has a performance problem. Ditching the freelist and > reverting to inlined PyObject_New()/PyObject_Del() semantics would be an > attractive course as it appears to perform to within a whisker of the > best alternatives while simplifying the object's code and taking > advantage of pymalloc, however there are a number of other > platforms/compilers that need testing before it would be prudent to do > so. I can test the code on Linux and Windows XP. My tests have shown that a linked free list doesn't give a considerable performance boost - even for code that allocates and frees a lot of ints and floats at once. But a fixed length pointer array gives a measurable performance gain. > The compaction routine you added to trunk gets the job done as far as > freelist pruning goes, but I can't say I find it anything other than an > ugly solution (though there is precedent for the concept via the gc > module). Yeah, I won't argue with that. The compaction function is a crutch. Christian

3 3

int/float freelists vs pymalloc
by Andrew MacIntyre 11 Feb '08

11 Feb '08

I note that in r60567 Christian checked in support for compacting the int and float freelists. I have no problems with the implementation, but would like to note that I have been experimenting with an alternate approach which doesn't require the addition of a knob to initiate the compacting. Probably in response to the same stimulus as Christian it occurred to me that the freelist approach had been adopted long before PyMalloc was enabled as standard (in 2.3), and that much of the performance gains between 2.2 and 2.3 were in fact due to PyMalloc. So I've been testing with the freelists ripped out and ints and floats reverted to fairly standard PyObject_New allocation (retaining the small int interning) and thus relying on PyMalloc to do any compaction. The results have been somewhat surprising... The short version is that: - for ints, the freelist is ahead of PyMalloc by a very small margin (<4%) - for floats, the freelist is a long way behind PyMalloc (>35% slower) This without invoking freelist compaction by the way (though PyMalloc is releasing empty arenas). I don't know what's pessimising the float freelist, but the results are similar on 2 boxes: - AMD Athlon XP-1600+, 512MB RAM, FreeBSD 6.1, gcc 3.4.4 (~27k pystones) - AMD Athlon XP-2500+, 512MB RAM, OS/2 v4 with EMX, gcc 2.8.1 (~38k pystones) If there's interest in following this up, I can put my patches to intobject.c and floatobject.c into the tracker. Test scripts: a) int <code> # test routine - integer import time def b(time_now=time.clock): limit_val = 2000000 start_time = time_now() for j in xrange(25): d = {} for i in xrange(limit_val): d[i] = i + limit_val return time_now() - start_time if __name__ == '__main__': print 'elapsed: %s s' % b() </code> b) float <code> # test routine - float import time def b(time_now=time.clock): limit_val = 1000000 vals = range(limit_val) offset = limit_val * 1.0 start_time = time_now() for j in xrange(25): d = {} for i in [float(x) for x in vals]: d[i] = i + offset return time_now() - start_time if __name__ == '__main__': print 'elapsed: %s s' % b() </code> The times I've obtained were: case XP1600/Fbsd XP2500/OS2 (*) 1) int freelist 38.1s 24.6s 2) int pymalloc 39.0s 25.3s 3) float freelist (with int freelist) 75s 46.1s 4) float pymalloc -with int freelist 55s 29.0s -with int pymalloc 55.5s 29.5s (*) OS/2 tests based on patched 2.5.1 sources rather than svn trunk. FreeBSD tests based on svn trunk checked out at 1050UTC 7Feb08. -- ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac(a)bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac(a)pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia

6 18

Python Bug Day on Feb. 23
by A.M. Kuchling 10 Feb '08

10 Feb '08

After the success of January's bug day, which closed 37 issues, let's have another one this month! Here's the brief announcement: Python Bug Day: Saturday, February 23 2008. Meet in the #python-dev IRC channel on irc.freenode.net and help improve Python. For more information, see http://wiki.python.org/moin/PythonBugDay . --amk

1 0

Backporting tempfile.NamedTemporaryFile context management fix
by Nick Coghlan 10 Feb '08

10 Feb '08

In 2.5, tempfile.NamedTemporaryFile passes requests for __enter__ and __exit__ through to the underlying file object, which ends up breaking in a couple of different ways (by a fortuitous coincidence, it actually works in most respects on Windows, so tempfile.TemporaryFile will mostly do the right thing in with statements no matter what platform you are on). This has been fixed on the trunk in response to issue 2021, but I'm not sure of the current status of the 2.5 maintenance branch. Should I wait until after 2.5.2 is released before backporting? Or backport it now? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

3 2

Error from SVN commit
by Nick Coghlan 10 Feb '08

10 Feb '08

I'm getting an error from SVN when trying to commit a NEWS entry for r60695: -------------- Sending Misc/NEWS Transmitting file data .svn: Commit failed (details follow): svn: Can't create directory '/data/repos/projects/db/transactions/60708-1.txn': No space left on device -------------- I've tried it a few times over the last 5-10 minutes and got the same error each time. I checked r60695 in last night from that same checkout, so I wouldn't expect it to be a local problem. Could the repository drive on svn.python.org just be full? Or is something else going on? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

4 3

Initial attempt to PyCon sprint tutorial slides are up
by Brett Cannon 10 Feb '08

10 Feb '08

The 1 MB PDF can be found at http://www.cs.ubc.ca/~drifty/pycon/sprint_tutorial.pdf . If you find any bad info or some info that is really lacking, let me know. But please realize that my slides are never really meant to be read on their own as it just goes against my presentation style. So don't think that some slide doesn't go into enough detail unless there is some URL I am missing. Every slide will be discussed more during the presentation than what is on the slide. -Brett

4 8

trunc()
by Raymond Hettinger 09 Feb '08

09 Feb '08

Can anyone explain to me why we need both trunc() and int()? We used to be very resistant to adding new built-ins and magic method protocols. In days not long past, this would have encountered fierce opposition. ISTM that numbers.py has taken on a life of its own and is causing non-essential offspring to sprout-up everywhere. Also, it seems that ABC is starting to evolve from an optional tool into something embedded in the language in a way that you can't escape it. I would prefer that it not be on the list of concepts that a beginner *must* know in order to be functional in the language. There are a handful of needs met by the numeric tower but they only warrant minimal changes to the language. Raymond P.S. The docstring for trunc() makes it sound like an imprecisely specified version of round(). ---- trunc() looks like int() but it isn't --------------------------- >>> v = [-4.9, -4.5, -4.1, -4.0, -3.5, -0.0, 0.0, 3.5, 4.0, 4.1, 4.5, 4.9] >>> map(int, v) [-4, -4, -4, -4, -3, 0, 0, 3, 4, 4, 4, 4] >>> map(trunc, v) [-4, -4, -4, -4, -3, 0, 0, 3, 4, 4, 4, 4] >>> trunc is int False >>> print trunc.__doc__ trunc(Real) -> Integral returns the integral closest to x between 0 and x.

9 13

Summary of Tracker Issues
by Tracker 08 Feb '08

08 Feb '08

ACTIVITY SUMMARY (02/01/08 - 02/08/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1691 open (+30) / 12209 closed (+18) / 13900 total (+48) Open issues with patches: 437 Average duration of open issues: 754 days. Median duration of open issues: 1065 days. Open Issues Breakdown open 1667 (+30) pending 24 ( +0) Issues Created Or Reopened (49) _______________________________ localeconv() does not encode returned strings 02/01/08 http://bugs.python.org/issue1995 created pitrou float.as_integer_ratio() needs to return fraction in lowest term 02/01/08 CLOSED http://bugs.python.org/issue1996 created rhettinger unicode and string compare should not cause an exception 02/01/08 CLOSED http://bugs.python.org/issue1997 created aaron_watters documentation grammatical error 02/02/08 CLOSED http://bugs.python.org/issue1998 created mickbeaver easy wrong tracker 02/02/08 CLOSED http://bugs.python.org/issue1999 created mickbeaver Undefined symbols: _PyOS_mystrnicmp on Mac OS X 02/02/08 CLOSED http://bugs.python.org/issue2000 created belopolsky Pydoc interactive browsing enhancement 02/02/08 http://bugs.python.org/issue2001 created ron_adam Make int() fall back to trunc() 02/02/08 CLOSED http://bugs.python.org/issue2002 created jyasskin patch Incorrect definition of new-style class 02/03/08 CLOSED http://bugs.python.org/issue2003 created thaneplummer tarfile extractall() allows local attacker to overwrite files wh 02/03/08 CLOSED http://bugs.python.org/issue2004 created mebrown posixmodule expects sizeof(pid_t/gid_t/uid_t) <= sizeof(long) 02/03/08 http://bugs.python.org/issue2005 created tiran easy asyncore loop lacks timers and work tasks 02/04/08 http://bugs.python.org/issue2006 created janssen cookielib lacks FileCookieJar class for Internet Explorer 02/04/08 http://bugs.python.org/issue2007 created janssen cookielib lacks FileCookieJar class for Safari 02/04/08 http://bugs.python.org/issue2008 created janssen Grammar change to prevent shift/reduce problem with varargslist 02/05/08 http://bugs.python.org/issue2009 created dalke Link to howto section on re module documentation incorrect 02/05/08 CLOSED http://bugs.python.org/issue2010 created knorby compiler.parse("1;") adds unexpected extra Discard(Const(None)) 02/05/08 CLOSED http://bugs.python.org/issue2011 created dalke Add migration step for DictMixin -> collections.MutableMapping 02/05/08 http://bugs.python.org/issue2012 created tiran Long object free list optimization 02/05/08 http://bugs.python.org/issue2013 created tiran patch xmlrpclib cannot send datetime objects with dates before 1900 02/05/08 http://bugs.python.org/issue2014 created schmir patch Possible optimisations in kwargs handling 02/06/08 http://bugs.python.org/issue2015 created amaury.forgeotdarc Crash when modifying the **kwargs passed to a function. 02/06/08 http://bugs.python.org/issue2016 created amaury.forgeotdarc Calendar.yeardatescalendar etc. do not take 'month' argument 02/06/08 CLOSED http://bugs.python.org/issue2017 created mft easy TextCalendar.formatmonth is not influenced by setfirstweekday 02/06/08 CLOSED http://bugs.python.org/issue2018 created mft easy API to clear most free lists 02/06/08 http://bugs.python.org/issue2019 created tiran patch _sha256 module missing if openssl is not in a "normal" directory 02/06/08 http://bugs.python.org/issue2020 created pajs(a)fodder.org.uk Turn NamedTemporaryFile into a context manager 02/06/08 http://bugs.python.org/issue2021 created belopolsky patch test_al converted to unittest 02/06/08 http://bugs.python.org/issue2022 created giampaolo.rodola test_cd converted to unittest 02/06/08 http://bugs.python.org/issue2023 created giampaolo.rodola test_gl.py converted to unittest 02/06/08 http://bugs.python.org/issue2024 created giampaolo.rodola tuples are instances of collections.Sequence but do not support 02/06/08 CLOSED http://bugs.python.org/issue2025 created rhettinger test_largefile.py converted to unittest 02/06/08 http://bugs.python.org/issue2026 created giampaolo.rodola Module containing C implementations of common text algorithms 02/07/08 http://bugs.python.org/issue2027 created mchaput _fmode = O_TEXT is obsolete 02/07/08 CLOSED http://bugs.python.org/issue2028 created zde "python -m pydoc -g" fails 02/07/08 http://bugs.python.org/issue2029 created bpb win32pdh.EnumObjects fails on Windows Server 2003 R2 02/07/08 CLOSED http://bugs.python.org/issue2038 created rmason Pymalloc patch for int and float objects 02/07/08 http://bugs.python.org/issue2039 created tiran patch Class instance attributes that are property() should appear in _ 02/07/08 CLOSED http://bugs.python.org/issue2040 created jag __getslice__ still called 02/07/08 CLOSED http://bugs.python.org/issue2041 created stefan test_audioop.py converted to unittest 02/07/08 http://bugs.python.org/issue2042 created giampaolo.rodola test_cl.py converted to unittest 02/07/08 http://bugs.python.org/issue2043 created giampaolo.rodola easy test_sunaudiodev.py converted to unittest 02/07/08 http://bugs.python.org/issue2044 created giampaolo.rodola defaultdict subclasses segfault with a bound method set as a def 02/07/08 CLOSED http://bugs.python.org/issue2045 created jek patch to fix_import: UserDict -> collections 02/07/08 http://bugs.python.org/issue2046 created eopadoan shutil.destinsrc returns wrong result when source path matches b 02/08/08 http://bugs.python.org/issue2047 created computercrustie Python 2.5.1 woes on IRIX, Solaris 02/08/08 http://bugs.python.org/issue2048 created atossava IDLE - Restart Shell & Run Module 02/08/08 http://bugs.python.org/issue2049 created taleinat patch IDLE - make ScriptBinding event handlers return 'break' 02/08/08 http://bugs.python.org/issue2050 created taleinat Thread shutdown exception in Thread.notify() 02/05/08 http://bugs.python.org/issue1722344 reopened brett.cannon Issues Now Closed (39) ______________________ libffi needs an update to support mips64, arm and armeabi on lin 112 days http://bugs.python.org/issue1292 doko urllib2 302 POST 92 days http://bugs.python.org/issue1401 facundobatista test_plistlib fails 11 days http://bugs.python.org/issue1914 tiran test_descr.py converted to unittest 7 days http://bugs.python.org/issue1935 georg.brandl patch re.search hangs on this 9 days http://bugs.python.org/issue1946 facundobatista Exception exceptions.AttributeError '_shutdown' in <module 'thre 11 days http://bugs.python.org/issue1947 facundobatista test_wave.py converted to unittest 10 days http://bugs.python.org/issue1951 giampaolo.rodola patch, easy test_contains.py converted to unittest 9 days http://bugs.python.org/issue1959 facundobatista Move trunc() to math module 3 days http://bugs.python.org/issue1965 jyasskin pybsddb leak in using cursors 4 days http://bugs.python.org/issue1976 gregory.p.smith Make Decimal comparisons with NaN less arbitrary 7 days http://bugs.python.org/issue1979 marketdickinson patch float.as_integer_ratio() needs to return fraction in lowest term 0 days http://bugs.python.org/issue1996 rhettinger unicode and string compare should not cause an exception 1 days http://bugs.python.org/issue1997 gvanrossum documentation grammatical error 0 days http://bugs.python.org/issue1998 georg.brandl easy wrong tracker 0 days http://bugs.python.org/issue1999 loewis Undefined symbols: _PyOS_mystrnicmp on Mac OS X 0 days http://bugs.python.org/issue2000 georg.brandl Make int() fall back to trunc() 1 days http://bugs.python.org/issue2002 jyasskin patch Incorrect definition of new-style class 0 days http://bugs.python.org/issue2003 georg.brandl tarfile extractall() allows local attacker to overwrite files wh 2 days http://bugs.python.org/issue2004 georg.brandl Link to howto section on re module documentation incorrect 0 days http://bugs.python.org/issue2010 georg.brandl compiler.parse("1;") adds unexpected extra Discard(Const(None)) 0 days http://bugs.python.org/issue2011 ncoghlan Calendar.yeardatescalendar etc. do not take 'month' argument 0 days http://bugs.python.org/issue2017 doerwalter easy TextCalendar.formatmonth is not influenced by setfirstweekday 2 days http://bugs.python.org/issue2018 mft easy tuples are instances of collections.Sequence but do not support 0 days http://bugs.python.org/issue2025 rhettinger _fmode = O_TEXT is obsolete 0 days http://bugs.python.org/issue2028 loewis win32pdh.EnumObjects fails on Windows Server 2003 R2 0 days http://bugs.python.org/issue2038 amaury.forgeotdarc Class instance attributes that are property() should appear in _ 1 days http://bugs.python.org/issue2040 tiran __getslice__ still called 0 days http://bugs.python.org/issue2041 marketdickinson defaultdict subclasses segfault with a bound method set as a def 0 days http://bugs.python.org/issue2045 amaury.forgeotdarc Issue warning when LC_NUMERIC is not "C" 2477 days http://bugs.python.org/issue419153 draghuram Memory Usage Reporting 2128 days http://bugs.python.org/issue540952 draghuram os.listdir-alike that includes file type 1948 days http://bugs.python.org/issue619222 draghuram SRE bugs with capturing groups in negative assertions 1753 days http://bugs.python.org/issue725149 draghuram Check for signals during regular expression matches 1537 days http://bugs.python.org/issue846388 gvanrossum patch pdb should hash stdout 1437 days http://bugs.python.org/issue908316 rhettinger easy NaN comparison in Decimal broken 587 days http://bugs.python.org/issue1514428 marketdickinson parser stack overflow 486 days http://bugs.python.org/issue1572320 amaury.forgeotdarc Adding an index method to tuples 305 days http://bugs.python.org/issue1696444 rhettinger patch Python 2.5+ skips while statements in debuggers 211 days http://bugs.python.org/issue1750076 amaury.forgeotdarc Top Issues Most Discussed (10) ______________________________ 11 Move Demo/classes/Rat.py to Lib/rational.py and fix it up. 49 days open http://bugs.python.org/issue1682 9 asyncore loop lacks timers and work tasks 5 days open http://bugs.python.org/issue2006 9 Thread shutdown exception in Thread.notify() 3 days open http://bugs.python.org/issue1722344 8 tarfile extractall() allows local attacker to overwrite files w 2 days closed http://bugs.python.org/issue2004 6 TextCalendar.formatmonth is not influenced by setfirstweekday 2 days closed http://bugs.python.org/issue2018 6 Make int() fall back to trunc() 1 days closed http://bugs.python.org/issue2002 5 _fmode = O_TEXT is obsolete 0 days closed http://bugs.python.org/issue2028 5 Add inspect.isgenerator 16 days open http://bugs.python.org/issue1916 4 Class instance attributes that are property() should appear in 1 days closed http://bugs.python.org/issue2040 4 Module containing C implementations of common text algorithms 2 days open http://bugs.python.org/issue2027

1 0