Mailman 3 September 2006 - Python-Dev

Re: [Python-Dev] Removing __del__
by Nick Coghlan 21 Sep '06

21 Sep '06

Adding pydev back in, since these seem like reasonable questions to me :) Jim Jewett wrote: > On 9/20/06, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> # Create a class with the same instance attributes >> # as the original >> class attr_holder(object): >> pass >> finalizer_arg = attr_holder() >> finalizer_arg.__dict__ = self.__dict__ > > Does this really work? It works for normal user-defined classes at least: >>> class C1(object): ... pass ... >>> class C2(object): ... pass ... >>> a = C1() >>> b = C2() >>> b.__dict__ = a.__dict__ >>> a.x = 1 >>> b.x 1 > (1) for classes with a dictproxy of some sort, you might get either a > copy (which isn't updated) Classes that change the way __dict__ is handled would probably need to define their own __del_arg__. > (2) for other classes, self might be added to the dict later Yeah, that's the strongest argument I know of against having that default fallback - it can easily lead to a strong reference from sys.finalizers into an otherwise unreachable cycle. I believe it currently takes two __del__ methods to prevent a cycle from being collected, whereas in this set up it would only take one. OTOH, fixing it would be much easier than it is now (by setting __del_args__ to something that holds only the subset of attributes that require finalization). > and of course, if it isn't added later, then it doesn't hvae the full > power of current finalizers -- just the __close__ subset. True, but most finalizers I've seen don't really *need* the full power of the current __del__. They only need to get at a couple of their internal members in order to explicitly release external resources. And more sophisticated usage is still possible by assigning an appropriate value to __del_arg__. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

2 1

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 21 Sep '06

21 Sep '06

Patch / Bug Summary ___________________ Patches : 419 open ( +3) / 3410 closed ( +2) / 3829 total ( +5) Bugs : 910 open (+12) / 6185 closed ( +5) / 7095 total (+17) RFE : 235 open ( +1) / 238 closed ( +0) / 473 total ( +1) New / Reopened Patches ______________________ Practical ctypes example (2006-09-15) http://python.org/sf/1559219 opened by leppton pyclbr reports different module for Class and Function (2006-09-18) http://python.org/sf/1560617 opened by Peter Otten Exec stacks in python 2.5 (2006-09-18) http://python.org/sf/1560695 opened by Chaza Patches Closed ______________ test_grp.py doesn't skip special NIS entry, fails (2006-06-22) http://python.org/sf/1510987 closed by martineau New / Reopened Bugs ___________________ some section links (previous, up, next) missing last word (2006-09-15) http://python.org/sf/1559142 opened by Tim Smith time.strptime() access non exitant attribute in calendar.py (2006-09-15) CLOSED http://python.org/sf/1559515 opened by betatim shutil.copyfile incomplete on NTFS (2006-09-16) http://python.org/sf/1559684 opened by Roger Upole gcc trunk (4.2) exposes a signed integer overflows (2006-08-24) http://python.org/sf/1545668 reopened by arigo 2.5c2 pythonw does not execute (2006-09-16) http://python.org/sf/1559747 opened by Ron Platten list.sort does nothing when both cmp and key are given (2006-09-16) CLOSED http://python.org/sf/1559818 opened by Marcin 'Qrczak' Kowalczyk confusing error msg from random.randint (2006-09-17) http://python.org/sf/1560032 opened by paul rubin Tutorial: incorrect info about package importing and mac (2006-09-17) http://python.org/sf/1560114 opened by C L Better/faster implementation of os.path.split (2006-09-17) CLOSED http://python.org/sf/1560161 opened by Michael Gebetsroither Better/faster implementation of os.path.basename/dirname (2006-09-17) http://python.org/sf/1560179 reopened by gbrandl Better/faster implementation of os.path.basename/dirname (2006-09-17) http://python.org/sf/1560179 opened by Michael Gebetsroither copy() method of dictionaries is not "deep" (2006-09-17) http://python.org/sf/1560327 reopened by gbrandl copy() method of dictionaries is not "deep" (2006-09-17) http://python.org/sf/1560327 opened by daniel hahler python 2.5 fails to build with --as-needed (2006-09-18) http://python.org/sf/1560984 opened by Chaza mac installer profile patch vs. .bash_login (2006-09-19) http://python.org/sf/1561243 opened by Ronald Oussoren -xcode=pic32 option is not supported on Solaris x86 Sun C (2006-09-19) http://python.org/sf/1561333 opened by James Lick Dedent with Italian keyboard (2006-09-20) http://python.org/sf/1562092 opened by neclepsio Fails to install on Fedora Core 5 (2006-09-20) http://python.org/sf/1562171 opened by Mark Summerfield IDLE Hung up after open script by command line... (2006-09-20) http://python.org/sf/1562193 opened by Faramir^ uninitialized memory read in parsetok() (2006-09-20) http://python.org/sf/1562308 opened by Luke Moore Bugs Closed ___________ 2.5c2 macosx installer aborts during "GUI Applications" (2006-09-15) http://python.org/sf/1558983 closed by ronaldoussoren time.strptime() access non existant attribute in calendar.py (2006-09-15) http://python.org/sf/1559515 closed by bcannon list.sort does nothing when both cmp and key are given (2006-09-16) http://python.org/sf/1559818 closed by qrczak Better/faster implementation of os.path.split (2006-09-17) http://python.org/sf/1560161 deleted by einsteinmg Better/faster implementation of os.path.basename/dirname (2006-09-17) http://python.org/sf/1560179 deleted by einsteinmg copy() method of dictionaries is not "deep" (2006-09-17) http://python.org/sf/1560327 closed by gbrandl New / Reopened RFE __________________ Exception need structured information associated with them (2006-09-15) http://python.org/sf/1559549 opened by Ned Batchelder String searching performance improvement (2006-09-19) CLOSED http://python.org/sf/1561634 opened by Nick Welch RFE Closed __________ String searching performance improvement (2006-09-19) http://python.org/sf/1561634 deleted by mackstann

1 0

deja-vu .. python locking
by Martin Devera 20 Sep '06

20 Sep '06

Hello, as someone has written in FAQ, sometimes someone starts a thread about finer grained locking in Python. Ok here is one. I don't want to start a flamewar. I only seek suggestions and constructive critic. I have some ideas whose are new in this context (I believe) and I only wanted to make them public in case someone finds them interesting. Comments are welcome. Martin ------------ Round 1, Greg Stein's patch The patch removes GIL from version 1.6 and replaces locking of list, dict and other structures with finer grained locking. The major slowdown seems to be in list and dict structures, dicts are used for object attributes and these are accessed quite often. Because (IIUC) mutex struct is quite heavy, dict and list are locked via pool of locks. When you lock this pooled lock you have to lock two locks in reality. One locks pool itself, and other locks the pooled lock (the second locking can be omited in non contended case because locks in the pool are in locked state). One lock take about 25 cycles on UP P4 (mainly pipeline flush during memory barrier) and can be even more expensive (hundreds of cycles) due to cacheline move between CPUs on SMP machine. "Global" pool lock is subject to cacheline pinpong as it will be often reacquired by competing CPUs. In mappinglookup there is lookmapping guarded by this locking scheme, lookmapping itself has about 20 cycles in the best (one hope typical) case plus compareobj cost (in case of string keys say ... 50..100 cycles?). Thus locking/unlocking the read takes 50..100 cycles and operation itself is 70-120 cycles. One might expect about 50% slowdown in dict read path. RCU like locking Solution I have in mind is similar to RCU. In Python we have quiscent state - when a thread returns to main loop of interpreter. Let's add "owner_thread" field to locked object. It reflects last thread (its id) which called any lockable method on the object. Each LOCK operation looks like: while (ob->owner_thread != self_thread()) { unlock_mutex(thread_mutex[self_thread()]) // wait for owning thread to go to quiscent state lock_mutex(thread_mutex[ob->owner_thread]) ob->owner_thread = self_thread() unlock_mutex(thread_mutex[ob->owner_thread]) lock_mutex(thread_mutex[self_thread()]) } Unlock is not done - we own the object now and can use it without locking (until we return to interpreter loop or we call LOCK on other object). For non-shared objects there is only penalty of ob->owner_thread != self_thread() condition. Not sure about Windows, but in recent Linuxes one can use %gs register as thread id, thus compare is about 3 cycles (and owner_thread should be in object's cacheline anyway). In contended case there is some cache pingpong with ob and mutex but it is as expected. Deadlocks Our object ownership is long - from getting it in LOCK to next quiscent state of the thread. Thus when two threads want to step each on other's object, they will deadlock. Simple solution is to extend set of quiscent states. It is when thread releases its thread_mutex in main loop (and immediately reacquires). Additionaly it can release it just before it is going to wait on another thread's mutex, like in LOCK (already in code above). If you use LOCK correctly then when you are LOCKing an object you can't be in vulnerable part of OTHER object. So that let other threads to get ownership of your own objects in that time. One can also want to release his lock when going to lock mutex in threading package and in other places where GIL is released today. However I admit that I did no formal proof regarding deadlock, I plan to do it if nobody can find other flaw in the proposal. Big reader lock While above scheme might work well, it'd impose performance penalty for shared dicts which are almost read only (module.__dict__). For these similar locking can be used, only writer has to wait until ALL other threads enter quiscent state (take locks of them), then perform change and unlock them all. Readers can read without any locking. Compatibilty with 3rd party modules I've read this argument on pydev list. Maybe I'm not understanding something, but is it so complex for Py_InitModule4 to use extra flag in apiver for example ? When at least one non-freethreaded module is loaded, locking is done in old good way...

6 16

Download URL typo
by Michael Walter 20 Sep '06

20 Sep '06

Hiho, in case noone didn't notice yet: the "Windows MSI Installer" link at http://www.python.org/download/releases/2.5/ points to Python 2.4! Regards, Michael

3 4

release25-maint branch - please keep frozen for a day or two more.
by Anthony Baxter 19 Sep '06

19 Sep '06

Could people please treat the release25-maint branch as frozen for a day or two, just in case we have to cut an ohmygodnononokillme release? Thanks, Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

2 1

RELEASED Python 2.5 (FINAL)
by Anthony Baxter 19 Sep '06

19 Sep '06

It's been nearly 20 months since the last major release of Python (2.4), and 5 months since the first alpha release of this cycle, so I'm absolutely thrilled to be able to say: On behalf of the Python development team and the Python community, I'm happy to announce the FINAL release of Python 2.5. This is a *production* release of Python 2.5. Yes, that's right, it's finally here. Python 2.5 is probably the most significant new release of Python since 2.2, way back in the dark ages of 2001. There's been a wide variety of changes and additions, both user-visible and underneath the hood. In addition, we've switched to SVN for development and now use Buildbot to do continuous testing of the Python codebase. Much more information (as well as source distributions and Windows and Universal Mac OSX installers) are available from the 2.5 website: http://www.python.org/2.5/ The new features in Python 2.5 are described in Andrew Kuchling's What's New In Python 2.5. It's available from the 2.5 web page. Amongst the new features of Python 2.5 are conditional expressions, the with statement, the merge of try/except and try/finally into try/except/finally, enhancements to generators to produce coroutine functionality, and a brand new AST-based compiler implementation underneath the hood. There's a variety of smaller new features as well. New to the standard library are hashlib, ElementTree, sqlite3, wsgiref, uuid and ctypes. As well, a new higher-performance profiling module (cProfile) was added. Extra-special thanks on behalf of the entire Python community should go out to Neal Norwitz, who's done absolutely sterling work in shepherding Python 2.5 through to it's final release. Enjoy this new release, (and Woo-HOO! It's done!) Anthony Anthony Baxter anthony(a)python.org Python Release Manager (on behalf of the entire python-dev team)

1 0

Testsuite fails on Windows if a space is in the path
by "Martin v. Löwis" 18 Sep '06

18 Sep '06

The test suite currently (2.5) has two failures on Windows if Python is installed into a directory with a space in it (such as "Program Files"). The failing tests are test_popen and test_cmd_line. The test_cmd_line failure is shallow: the test fails to properly quote sys.executable when passing it to os.popen. I propose to fix this in Python 2.5.1; see #1559413 test_popen is more tricky. This code has always failed AFAICT, except that the test itself is a recent addition. The test tries to pass the following command to os.popen "c:\Program Files\python25\python.exe" -c "import sys;print sys.version" For some reason, os.popen invokes doesn't directly start Python as a new process, but install invokes cmd.exe /c "c:\Program Files\python25\python.exe" -c "import sys;print sys.version" Can somebody remember what the reason is to invoke cmd.exe (or COMSPEC) in os.popen? In any case, cmd.exe fails to execute this, claiming that c:\Program is not a valid executable. It would run cmd.exe /c "c:\Program Files\python25\python.exe" just fine, so apparently, the problem is with argument that have multiple pairs of quotes. I found, through experimentation, that it *will* accept cmd.exe /c ""c:\Program Files\python25\python.exe" -c "import sys;print sys.version"" (i.e. doubling the quotes at the beginning and the end). I'm not quite sure what algorithm cmd.exe uses for parsing, but it appears that adding a pair of quotes works in all cases (at least those I could think of). See # 1559298 Here are my questions: 1. Should this be fixed before the final release of Python 2.5? 2. If not, should it be fixed in Python 2.5.1? I'd say not: there is a potential of breaking existing applications. Applications might be aware of this mess, and deliberately add a pair of quotes already. If popen then adds yet another pair of quotes, cmd.exe will again fail. 3. If not, should this be fixed in 2.6 in the way I propose in the patch (i.e. add quotes around the command line)? Or can anybody propose a different fix? 4. How should we deal with different values of COMSPEC? Should this patch only apply for cmd.exe, or should we assume that other shells are quirk-compatible with cmd.exe in this respect (or that people stopped setting COMSPEC, anyway)? Any comments appreciated, Martin

5 6

Before 2.5 - More signed integer overflows
by Armin Rigo 18 Sep '06

18 Sep '06

Hi all, There are more cases of signed integer overflows in the CPython source code base... That's on a 64-bits machine: [GCC 4.1.2 20060715 (prerelease) (Debian 4.1.1-9)] on linux2 abs(-sys.maxint-1) == -sys.maxint-1 I'd expect the same breakage everywhere when GCC 4.2 is used. Note that the above is Python 2.4.4c0 - apparently Python 2.3 compiled with GCC 4.1.2 works, although that doesn't make much sense to me because intobject.c didn't change here - 2.3, 2.4, 2.5, trunk are all the same. Both tested Pythons are Debian packages, not in-house compiled. Humpf! Looks like one person or two need to do a quick last-minute review of all places trying to deal with -sys.maxint-1, and replace them all with the "official" fix from Tim [SF 1545668]. A bientot, Armin

5 9

BRANCH FREEZE/IMMINENT RELEASE: Python 2.5 (final). 2006-09-19, 00:00UTC
by Anthony Baxter 18 Sep '06

18 Sep '06

Ok, time to bring down the hammer. The release25-maint branch is absolutely frozen to everyone but the release team from 00:00UTC, Tuesday 19th September. That's just under 20 hours from now. This is for Python 2.5 FINAL, so anyone who breaks this release will make me very, very sad. Based on the last few releases, I'd expect the release process to take around 18 hours (timezones are a swine). Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

1 0

python, lipo and the future?
by howarth＠bromo.msbb.uc.edu 17 Sep '06

17 Sep '06

I am curious if there are any plans to support the functionality provided by lipo on MacOS X to create a python release that could operate at either 32-bit or 64-bit on Darwin ppc and Darwin intel? My understanding was that the linux developers are very interested in lipo as well as an approach to avoid the difficulty of maintaining separate lib directories for 32 and 64-bit libraries. Thanks in advance for any insights on this issue. Jack

6 14