Mailman 3 October 2006 - Python-Dev

Python 2.5 performance
by Kristján V. Jónsson 13 Oct '06

13 Oct '06

Hello there. I just got round to do some comparative runs of 2.5 32 bit Release, built with visual studio 2003 and 2005. Here the figures (pybench with default arguments) .NET 2003: Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 262ms 304ms 0.60us 0.251ms BuiltinMethodLookup: 232ms 267ms 0.25us 0.312ms CompareFloats: 148ms 170ms 0.14us 0.377ms CompareFloatsIntegers: 183ms 216ms 0.24us 0.261ms CompareIntegers: 144ms 163ms 0.09us 0.527ms CompareInternedStrings: 157ms 186ms 0.12us 1.606ms CompareLongs: 153ms 174ms 0.17us 0.300ms CompareStrings: 156ms 198ms 0.20us 1.166ms CompareUnicode: 180ms 205ms 0.27us 0.731ms ConcatStrings: 410ms 457ms 0.91us 0.579ms ConcatUnicode: 473ms 610ms 2.03us 0.466ms CreateInstances: 248ms 290ms 2.59us 0.432ms CreateNewInstances: 206ms 243ms 2.89us 0.352ms CreateStringsWithConcat: 164ms 200ms 0.20us 0.971ms CreateUnicodeWithConcat: 268ms 295ms 0.74us 0.343ms DictCreation: 152ms 186ms 0.47us 0.358ms DictWithFloatKeys: 378ms 410ms 0.46us 0.660ms DictWithIntegerKeys: 133ms 161ms 0.13us 0.907ms DictWithStringKeys: 152ms 184ms 0.15us 0.927ms ForLoops: 125ms 133ms 5.32us 0.069ms IfThenElse: 109ms 131ms 0.10us 1.019ms ListSlicing: 193ms 223ms 15.90us 0.072ms NestedForLoops: 147ms 164ms 0.11us 0.021ms NormalClassAttribute: 176ms 195ms 0.16us 0.579ms NormalInstanceAttribute: 171ms 198ms 0.17us 0.598ms PythonFunctionCalls: 207ms 240ms 0.73us 0.326ms PythonMethodCalls: 234ms 287ms 1.27us 0.163ms Recursion: 294ms 328ms 6.56us 0.563ms SecondImport: 191ms 210ms 2.10us 0.241ms SecondPackageImport: 197ms 220ms 2.20us 0.217ms SecondSubmoduleImport: 257ms 276ms 2.76us 0.213ms SimpleComplexArithmetic: 191ms 208ms 0.24us 0.445ms SimpleDictManipulation: 158ms 178ms 0.15us 0.625ms SimpleFloatArithmetic: 183ms 211ms 0.16us 0.703ms SimpleIntFloatArithmetic: 122ms 133ms 0.10us 0.745ms SimpleIntegerArithmetic: 106ms 121ms 0.09us 0.680ms SimpleListManipulation: 132ms 149ms 0.13us 0.750ms SimpleLongArithmetic: 170ms 198ms 0.30us 0.322ms SmallLists: 246ms 274ms 0.40us 0.437ms SmallTuples: 204ms 235ms 0.43us 0.497ms SpecialClassAttribute: 177ms 201ms 0.17us 0.561ms SpecialInstanceAttribute: 257ms 290ms 0.24us 0.598ms StringMappings: 881ms 949ms 3.77us 0.584ms StringPredicates: 321ms 366ms 0.52us 3.207ms StringSlicing: 243ms 286ms 0.51us 1.032ms TryExcept: 87ms 110ms 0.05us 0.957ms TryRaiseExcept: 164ms 197ms 3.08us 0.434ms TupleSlicing: 195ms 230ms 0.88us 0.065ms UnicodeMappings: 158ms 187ms 5.20us 0.699ms UnicodePredicates: 191ms 233ms 0.43us 3.954ms UnicodeProperties: 209ms 251ms 0.63us 3.234ms UnicodeSlicing: 306ms 345ms 0.70us 0.933ms ------------------------------------------------------------------------------- Totals: 11202ms 12875ms .NET 2005: Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 254ms 279ms 0.55us 0.280ms BuiltinMethodLookup: 269ms 290ms 0.28us 0.327ms CompareFloats: 136ms 147ms 0.12us 0.375ms CompareFloatsIntegers: 158ms 178ms 0.20us 0.268ms CompareIntegers: 118ms 141ms 0.08us 0.603ms CompareInternedStrings: 152ms 203ms 0.14us 1.666ms CompareLongs: 152ms 171ms 0.16us 0.335ms CompareStrings: 118ms 140ms 0.14us 1.374ms CompareUnicode: 160ms 180ms 0.24us 0.730ms ConcatStrings: 430ms 472ms 0.94us 0.681ms ConcatUnicode: 488ms 535ms 1.78us 0.458ms CreateInstances: 249ms 286ms 2.56us 0.437ms CreateNewInstances: 220ms 254ms 3.02us 0.356ms CreateStringsWithConcat: 174ms 204ms 0.20us 1.123ms CreateUnicodeWithConcat: 271ms 294ms 0.74us 0.348ms DictCreation: 151ms 169ms 0.42us 0.365ms DictWithFloatKeys: 350ms 387ms 0.43us 0.666ms DictWithIntegerKeys: 140ms 151ms 0.13us 1.020ms DictWithStringKeys: 154ms 176ms 0.15us 1.070ms ForLoops: 96ms 111ms 4.42us 0.069ms IfThenElse: 115ms 130ms 0.10us 0.697ms ListSlicing: 221ms 261ms 18.66us 0.093ms NestedForLoops: 146ms 167ms 0.11us 0.022ms NormalClassAttribute: 182ms 205ms 0.17us 0.502ms NormalInstanceAttribute: 174ms 192ms 0.16us 0.457ms PythonFunctionCalls: 203ms 221ms 0.67us 0.337ms PythonMethodCalls: 266ms 309ms 1.37us 0.149ms Recursion: 286ms 329ms 6.57us 0.459ms SecondImport: 170ms 197ms 1.97us 0.181ms SecondPackageImport: 187ms 215ms 2.15us 0.178ms SecondSubmoduleImport: 243ms 275ms 2.75us 0.215ms SimpleComplexArithmetic: 177ms 199ms 0.23us 0.370ms SimpleDictManipulation: 159ms 185ms 0.15us 0.498ms SimpleFloatArithmetic: 177ms 196ms 0.15us 1.502ms SimpleIntFloatArithmetic: 109ms 126ms 0.10us 0.574ms SimpleIntegerArithmetic: 108ms 124ms 0.09us 0.611ms SimpleListManipulation: 145ms 169ms 0.14us 0.619ms SimpleLongArithmetic: 167ms 190ms 0.29us 0.324ms SmallLists: 247ms 274ms 0.40us 0.339ms SmallTuples: 204ms 224ms 0.42us 0.429ms SpecialClassAttribute: 193ms 216ms 0.18us 0.558ms SpecialInstanceAttribute: 255ms 280ms 0.23us 0.470ms StringMappings: 297ms 321ms 1.28us 0.474ms StringPredicates: 229ms 274ms 0.39us 3.892ms StringSlicing: 238ms 258ms 0.46us 0.962ms TryExcept: 86ms 102ms 0.05us 0.755ms TryRaiseExcept: 155ms 173ms 2.70us 0.357ms TupleSlicing: 188ms 217ms 0.83us 0.050ms UnicodeMappings: 103ms 118ms 3.29us 0.595ms UnicodePredicates: 176ms 207ms 0.38us 3.950ms UnicodeProperties: 187ms 212ms 0.53us 3.228ms UnicodeSlicing: 312ms 342ms 0.70us 0.834ms ------------------------------------------------------------------------------- Totals: 10343ms 11677ms This is an improvement of more than 7%. In addition, here is a run of the PGO optimized .NET 2005: Test minimum average operation overhead ------------------------------------------------------------------------------- BuiltinFunctionCalls: 232ms 250ms 0.49us 0.330ms BuiltinMethodLookup: 276ms 296ms 0.28us 0.382ms CompareFloats: 130ms 142ms 0.12us 0.451ms CompareFloatsIntegers: 150ms 166ms 0.18us 0.326ms CompareIntegers: 130ms 155ms 0.09us 0.729ms CompareInternedStrings: 152ms 197ms 0.13us 1.947ms CompareLongs: 136ms 146ms 0.14us 0.390ms CompareStrings: 151ms 174ms 0.17us 1.583ms CompareUnicode: 131ms 167ms 0.22us 0.965ms ConcatStrings: 417ms 485ms 0.97us 0.681ms ConcatUnicode: 483ms 551ms 1.84us 0.484ms CreateInstances: 224ms 252ms 2.25us 0.600ms CreateNewInstances: 186ms 216ms 2.58us 0.407ms CreateStringsWithConcat: 155ms 175ms 0.18us 1.264ms CreateUnicodeWithConcat: 275ms 306ms 0.76us 0.437ms DictCreation: 160ms 186ms 0.47us 0.443ms DictWithFloatKeys: 349ms 375ms 0.42us 0.924ms DictWithIntegerKeys: 143ms 173ms 0.14us 1.296ms DictWithStringKeys: 157ms 177ms 0.15us 1.184ms ForLoops: 140ms 155ms 6.21us 0.074ms IfThenElse: 107ms 127ms 0.09us 0.955ms ListSlicing: 217ms 256ms 18.29us 0.103ms NestedForLoops: 166ms 194ms 0.13us 0.018ms NormalClassAttribute: 163ms 179ms 0.15us 0.564ms NormalInstanceAttribute: 151ms 169ms 0.14us 0.536ms PythonFunctionCalls: 210ms 235ms 0.71us 0.313ms PythonMethodCalls: 237ms 260ms 1.15us 0.167ms Recursion: 285ms 334ms 6.68us 0.538ms SecondImport: 147ms 169ms 1.69us 0.243ms SecondPackageImport: 155ms 200ms 2.00us 0.215ms SecondSubmoduleImport: 202ms 234ms 2.34us 0.203ms SimpleComplexArithmetic: 162ms 187ms 0.21us 0.446ms SimpleDictManipulation: 162ms 181ms 0.15us 0.627ms SimpleFloatArithmetic: 171ms 201ms 0.15us 1.335ms SimpleIntFloatArithmetic: 119ms 137ms 0.10us 0.659ms SimpleIntegerArithmetic: 114ms 128ms 0.10us 0.668ms SimpleListManipulation: 145ms 161ms 0.14us 0.764ms SimpleLongArithmetic: 161ms 178ms 0.27us 0.423ms SmallLists: 234ms 271ms 0.40us 0.454ms SmallTuples: 182ms 203ms 0.38us 0.497ms SpecialClassAttribute: 174ms 201ms 0.17us 0.716ms SpecialInstanceAttribute: 230ms 252ms 0.21us 0.558ms StringMappings: 285ms 313ms 1.24us 0.514ms StringPredicates: 233ms 275ms 0.39us 3.475ms StringSlicing: 225ms 242ms 0.43us 1.037ms TryExcept: 78ms 89ms 0.04us 0.961ms TryRaiseExcept: 133ms 156ms 2.44us 0.454ms TupleSlicing: 186ms 202ms 0.77us 0.078ms UnicodeMappings: 103ms 118ms 3.29us 0.520ms UnicodePredicates: 186ms 216ms 0.40us 3.414ms UnicodeProperties: 180ms 214ms 0.54us 2.530ms UnicodeSlicing: 299ms 318ms 0.65us 0.815ms ------------------------------------------------------------------------------- Totals: 9974ms 11345ms This is an improvement of another 3.5 %. In all, we have a performance increase of more than 10%. Granted, this is from a single set of runs, but I think we should start considering to make PCBuild8 a "supported" build. Cheers, Kristján

7 7

Re: [Python-Dev] 2.3.6 for the unicode buffer overrun
by Brian Warner 13 Oct '06

13 Oct '06

"Gregory P. Smith" <greg(a)electricrain.com> writes: > as for buildbot, i haven't looked at its design but from the chatter > i've seen i was under the impression that it operates on a continually > updated sandbox rather than a 100% fresh checkout for each build? It's a configuration option. If you use mode="update" then your builds will re-use the same source directory over and over, if you use mode="clobber" then your builds will get a brand new checkout each time, and if you use mode="copy" then the source is updated in-place in one directory, but each build is performed from a copy of that checkout. Each offers different tradeoffs between disk usage, network usage, and which sorts of Makefile bugs they are likely to discover. cheers, -Brian (Buildbot author)

1 0

Re: [Python-Dev] Why spawnvp not implemented on Windows?
by "Martin v. Löwis" 13 Oct '06

13 Oct '06

Alexey Borzenkov schrieb: > On 10/13/06, "Martin v. Löwis" <martin(a)v.loewis.de> wrote: >> > Umm... do you mean that spawn*p* on python 2.5 is an absolute no? >> Yes. No new features can be added to Python 2.5.x; Python 2.5 has >> already been released. > > Ugh... that's just not fair. Because of this there will be no spawn*p* > in python for another two years. x_x It may be inconvenient, but it is certainly fair: the same rule is applied to *all* proposed new features. It would be unfair if that feature was accepted, and other features were rejected. Please try to see this from "our" view. If new features are added to a bugfix release (say, 2.5.1), then users (programmers) would quickly consider Python as unstable, moving target. They would use the feature, claiming that you need Python 2.5, and not knowing that it is really 2.5.*1* that you need. Users would try to run the program, and find out that it doesn't work, and complain to the author. Unhappy users, unhappy programmers, and unhappy maintainers (as the programmers would then complain which idiot allowed that feature in - they do use strong language at times). It happened once, in 2.2.1 (IIRC) with the introduction of True and False. It was very painful and lead to a lot of bad code, and it still hasn't settled. As you already have a work-around: what is the problem waiting for 2.6, for you personally? If you want to see the feature eventually, please do submit it to sourceforge, anyway. Regards, Martin

1 0

Exceptions and slicing
by Thomas Heller 13 Oct '06

13 Oct '06

Is it an oversight that exception instances do no longer support slicing in Python 2.5? This code works in 2.4, but no longer in 2.5: try: open("", "r") except IOError, details: print details[:] Thomas

3 6

Re: [Python-Dev] Signals, threads, blocking C functions
by Nick Maclaren 12 Oct '06

12 Oct '06

=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin(a)v.loewis.de> wrote: Michael Hudson schrieb: > > >> According to [1], all python needs to do to avoid this problem is > >> block all signals in all but the main thread; > > > > Argh, no: then people who call system() from non-main threads end up > > running subprocesses with all signals masked, which breaks other > > things in very mysterious ways. Been there... > > Python should register a pthread_atfork handler then, which clears > the signal mask. Would that not work? No. It's not the only such problem. Personally, I think that anyone who calls system(), fork(), spawn() or whatever from threads is cuckoo. It is precisely the sort of thing that is asking for trouble, because there are so many ways of doing it 'right' that you can't be sure exactly what mental model the system developers will have. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Signals, threads, blocking C functions
by Gustavo Carneiro 12 Oct '06

12 Oct '06

We have to resort to timeouts in pygtk in order to catch unix signals in threaded mode. The reason is this. We call gtk_main() (mainloop function) which blocks forever. Suppose there are threads in the program; then any thread can receive a signal (e.g. SIGINT). Python catches the signal, but doesn't do anything; it simply sets a flag in a global structure and calls Py_AddPendingCall(), and I guess it expects someone to call Py_MakePendingCalls(). However, the main thread is blocked calling a C function and has no way of being notified it needs to give control back to python to handle the signal. Hence, we use a 100ms timeout for polling. Unfortunately, timeouts needlessly consume CPU time and drain laptop batteries. According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. Another alternative would be to add a new API like Py_AddPendingCallNotification, which would let python notify extensions that new pending calls exist and need to be processed. But I would really prefer the first alternative, as it could be fixed within python 2.5; no need to wait for 2.6. Please, let's make Python ready for the enterprise! [2] [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 [2] http://perkypants.org/blog/2006/09/02/rfte-python/

6 11

RELEASED Python 2.4.4, release candidate 1
by Anthony Baxter 12 Oct '06

12 Oct '06

On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.4.4 (release candidate 1). Python 2.4.4 is a bug-fix release. While Python 2.5 is the latest version of Python, we're making this release for people who are still running Python 2.4. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of the more than 80 bugs squished in this release, including a number found by the Coverity and Klocwork static analysis tools. We'd like to offer our thanks to both these companies for making this available for open source projects. * Python 2.4.4 contains a fix for PSF-2006-001, a buffer overrun * * in repr() of unicode strings in wide unicode (UCS-4) builds. * * See http://www.python.org/news/security/PSF-2006-001/ for more. * Assuming no major problems crop up, a final release of Python 2.4.4 will follow in about a week's time. This will be the last planned release in the Python 2.4 series - future maintenance releases will be in the 2.5 line. For more information on Python 2.4.4, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.4.4/ Highlights of this new release include: - Bug fixes. According to the release notes, at least 80 have been fixed. - A fix for PSF-2006-001, a bug in repr() for unicode strings on UCS-4 (wide unicode) builds. Highlights of the previous major Python release (2.4) are available from the Python 2.4 page, at http://www.python.org/2.4/highlights.html Enjoy this release, Anthony Anthony Baxter anthony(a)python.org Python Release Manager (on behalf of the entire python-dev team)

1 0

2.4 vs Windows vs bsddb
by Tim Peters 11 Oct '06

11 Oct '06

I just noticed that the bsddb portion of Python fails to compile on the 2.4 Windows buildbots, but for some reason the buildbot machinery doesn't notice the failure: """ Compiling... _bsddb.c Linking... Creating library .\./_bsddb_d.lib and object .\./_bsddb_d.exp _bsddb.obj : warning LNK4217: locally defined symbol _malloc imported in function __db_associateCallback _bsddb.obj : warning LNK4217: locally defined symbol _free imported in function __DB_consume _bsddb.obj : warning LNK4217: locally defined symbol _fclose imported in function _DB_verify _bsddb.obj : warning LNK4217: locally defined symbol _fopen imported in function _DB_verify _bsddb.obj : warning LNK4217: locally defined symbol _strncpy imported in function _init_pybsddb _bsddb.obj : error LNK2019: unresolved external symbol __imp__strncat referenced in function _makeDBError _bsddb.obj : error LNK2019: unresolved external symbol __imp___assert referenced in function _makeDBError ./_bsddb_d.pyd : fatal error LNK1120: 2 unresolved externals ... _bsddb - 3 error(s), 5 warning(s) Build: 15 succeeded, 1 failed, 0 skipped """ The warnings there are old news, but no idea when the errors started. The test suite doesn't care that bsddb is missing either, just ending with: 1 skip unexpected on win32: test_bsddb Same kind of things when building from my 2.4 checkout. No clues.

4 9

Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?
by Raymond Hettinger 11 Oct '06

11 Oct '06

No need to backport. Py_TPFLAGS_DEFAULT implies Py_TPFLAGS_HAVE_WEAKREFS. The change was for clarity -- most things that have the weakref slots filled-in will also make the flag explicit -- that makes it easier on the brain when verifying code that checks the weakref flag. Raymond -----Original Message----- From: python-dev-bounces+rhettinger=ewtllc.com(a)python.org [mailto:python-dev-bounces+rhettinger=ewtllc.com@python.org] On Behalf Of A.M. Kuchling Sent: Friday, October 06, 2006 6:41 AM To: python-dev(a)python.org Subject: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS? I was looking at the logs for classobject.c and noticed this commit that adds Py_TPFLAGS_HAVE_WEAKREFS to the instance type. Should it be backported to 2.4? (It looks to me like it should, but I don't know anything about weakref implementation and want to get approval from someone who knows.) --amk r39038 | rhettinger | 2005-06-19 04:42:20 -0400 (Sun, 19 Jun 2005) | 2 lines Insert missing flag. ------------------------------------------------------------------------ Index: classobject.c =================================================================== --- classobject.c (revision 39037) +++ classobject.c (revision 39038) @@ -2486,7 +2486,7 @@ (getattrofunc)instancemethod_getattro, /* tp_getattro */ PyObject_GenericSetAttr, /* tp_setattro */ 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_HAVE_WEAKREFS, /* tp_flags */ instancemethod_doc, /* tp_doc */ (traverseproc)instancemethod_traverse, /* tp_traverse */ 0, /* tp_clear */ svn merge -r 39037:39038 svn+ssh://pythondev@svn.python.org/python/trunk _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/rhettinger%40ewtllc.co m

3 2

BRANCH FREEZE, release24-maint for 2.4.4c1. 00:00UTC, 11 October 2006
by Anthony Baxter 10 Oct '06

10 Oct '06

The release24-maint branch is frozen for the 2.4.4c1 release from 00:00UTC on the 11th of October. That's about 7 hours from now. Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

1 0