Mailman 3 February 2005 - Python-Dev

Prospective Peephole Transformation
by Raymond Hettinger 22 Feb '05

22 Feb '05

Based on some ideas from Skip, I had tried transforming the likes of "x in (1,2,3)" into "x in frozenset([1,2,3])". When applicable, it substantially simplified the generated code and converted the O(n) lookup into an O(1) step. There were substantial savings even if the set contained only a single entry. When disassembled, the bytecode is not only much shorter, it is also much more readable (corresponding almost directly to the original source). The problem with the transformation was that it didn't handle the case where x was non-hashable and it would raise a TypeError instead of returning False as it should. That situation arose once in the email module's test suite. To get it to work, I would have to introduce a frozenset subtype: class Searchset(frozenset): def __contains__(self, element): try: return frozenset.__contains__(self, element) except TypeError: return False Then, the transformation would be "x in Searchset([1, 2, 3])". Since the new Searchset object goes in the constant table, marshal would have to be taught how to save and restore the object. This is a more complicated than the original frozenset version of the patch, so I would like to get feedback on whether you guys think it is worth it. Raymond Hettinger

11 25

Store x Load x --> DupStore
by Raymond Hettinger 21 Feb '05

21 Feb '05

Any objections to new peephole transformation that merges a store/load pair into a single step? There is a tested patch at: www.python.org/sf/1144842 It folds the two steps into a new opcode. In the case of store_name/load_name, it saves one three byte instruction, a trip around the eval-loop, two stack mutations, a incref/decref pair, a dictionary lookup, and an error check (for the lookup). While it acts like a dup followed by a store, it is implemented more simply as a store that doesn't pop the stack. The transformation is broadly applicable and occurs thousands of times in the standard library and test suite. Raymond Hettinger

8 16

Re: Welcome to the "Python-Dev" mailing list
by z-axis 21 Feb '05

21 Feb '05

hi,friends i am a python newbie but i used Java for about 5 years. when i saw python introduce in a famous magzine called <<programmer>> in China, i am immediately absorbed by its pretty code. i hope i can use Python to do real development. regards! 　　 ======== 2005-02-21 14:28:00 您在来信中写道： ======== Welcome to the Python-Dev(a)python.org mailing list! If you are a new subscriber, please take the time to introduce yourself briefly in your first post. It is appreciated if you lurk around for a while before posting! :-) Additional information on Python's development process can be found in the Python Developer's Guide: http://www.python.org/dev/ To post to this list, send your email to: python-dev(a)python.org General information about the mailing list is at: http://mail.python.org/mailman/listinfo/python-dev If you ever want to unsubscribe or change your options (eg, switch to or from digest mode, change your password, etc.), visit your subscription page at: http://mail.python.org/mailman/options/python-dev/z_axis%40163.com You can also make such adjustments via email by sending a message to: Python-Dev-request(a)python.org with the word `help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You must know your password to change your options (including changing the password, itself) or to unsubscribe. It is: zpython999 Normally, Mailman will remind you of your python.org mailing list passwords once every month, although you can disable this if you prefer. This reminder will also include instructions on how to unsubscribe or change your account options. There is also a button on your options page that will email your current password to you. = = = = = = = = = = = = = = = = = = = = = = 　　　　　　　　致礼！　　　　　　　　　　　　　z-axis z_axis(a)163.com 　　　　　　　　　　　　　　　2005-02-21

1 0

builtin_id() returns negative numbers
by Troels Walsted Hansen 19 Feb '05

19 Feb '05

Hi all, The Python binding in libxml2 uses the following code for __repr__(): class xmlNode(xmlCore): def __init__(self, _obj=None): self._o = None xmlCore.__init__(self, _obj=_obj) def __repr__(self): return "<xmlNode (%s) object at 0x%x>" % (self.name, id (self)) With Python 2.3.4 I'm seeing warnings like the one below: <frozen module libxml2>:2357: FutureWarning: %u/%o/%x/%X of negative int will return a signed string in Python 2.4 and up I believe this is caused by the memory address having the sign bit set, causing builtin_id() to return a negative integer. I grepped around in the Python standard library and found a rather awkward work-around that seems to be slowly propagating to various module using the "'%x' % id(self)" idiom: Lib/asyncore.py: # On some systems (RH10) id() can be a negative number. # work around this. MAX = 2L*sys.maxint+1 return '<%s at %#x>' % (' '.join(status), id(self)&MAX) $ grep -r 'can be a negative number' * Lib/asyncore.py: # On some systems (RH10) id() can be a negative number. Lib/repr.py: # On some systems (RH10) id() can be a negative number. Lib/tarfile.py: # On some systems (RH10) id() can be a negative number. Lib/test/test_repr.py: # On some systems (RH10) id() can be a negative number. Lib/xml/dom/minidom.py: # On some systems (RH10) id() can be a negative number. There are many modules that do not have this work-around in Python 2.3.4. Wouldn't it be more elegant to make builtin_id() return an unsigned long integer? Is the performance impact too great? A long integer is used on platforms where SIZEOF_VOID_P > SIZEOF_LONG (most 64 bit platforms?), so all Python code must be prepared to handle it already... Troels

10 14

license issues with profiler.py and md5.h/md5c.c
by Matthias Klose 18 Feb '05

18 Feb '05

A Debian user pointed out (http://bugs.debian.org/293932), that the current license for the Python profiler is not conforming to the DFSG (Debian free software guidelines). http://www.python.org/doc/current/lib/node829.html states "This permission is explicitly restricted to the copying and modification of the software to remain in Python, compiled Python, or other languages (such as C) wherein the modified or derived code is exclusively imported into a Python module." The DFSG, http://www.debian.org/doc/debian-policy/ch-archive.html#s-dfsg, third paragraph state: "Derived Works The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software." - Does somebody knows about the history of this license, why it is more restricted than the Python license? - Is there a chance to change the license for these two modules (profile.py, pstats.py)? The md5.h/md5c.c files allow "copy and use", but no modification of the files. There are some alternative implementations, i.e. in glibc, openssl, so a replacement should be sage. Any other requirements when considering a replacement? Matthias

14 34

RE: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15%
by Gfeller Martin 18 Feb '05

18 Feb '05

Hi, what immediately comes to mind are Modules/cPickle.c and Modules/cStringIO.c, which (I believe) are heavily used by ZODB (which in turn is heavily used by the application). The lists also get fairly large, although not huge - up to typically 50000 (complex) objects in the tests I've measured. As I said, I don't speak C, so I can only speculate - do the lists at some point grow beyond the upper limit of obmalloc, but are handled by the LFH (which has a higher upper limit, if I understood Tim Peters correctly)? Best regards, Martin -----Original Message----- From: Evan Jones [mailto:ejones@uwaterloo.ca] Sent: Thursday, 17 Feb 2005 02:26 To: Python Dev Cc: Gfeller Martin; Martin v. Löwis Subject: Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15% On Feb 16, 2005, at 18:42, Martin v. Löwis wrote: > I must admit that I'm surprised. I would have expected > that most allocations in Python go through obmalloc, so > the heap would only see "large" allocations. > > It would be interesting to find out, in your application, > why it is still an improvement to use the low-fragmentation > heaps. Hmm... This is an excellent point. A grep through the Python source code shows that the following files call the native system malloc (I've excluded a few obviously platform specific files). A quick visual inspection shows that most of these are using it to allocate some sort of array or string, so it likely *should* go through the system malloc. Gfeller, any idea if you are using any of the modules on this list? If so, it would be pretty easy to try converting them to call the obmalloc functions instead, and see how that affects the performance. Evan Jones Demo/pysvr/pysvr.c Modules/_bsddb.c Modules/_curses_panel.c Modules/_cursesmodule.c Modules/_hotshot.c Modules/_sre.c Modules/audioop.c Modules/bsddbmodule.c Modules/cPickle.c Modules/cStringIO.c Modules/getaddrinfo.c Modules/main.c Modules/pyexpat.c Modules/readline.c Modules/regexpr.c Modules/rgbimgmodule.c Modules/svmodule.c Modules/timemodule.c Modules/zlibmodule.c PC/getpathp.c Python/strdup.c Python/thread.c

3 5

Fixing _PyEval_SliceIndex so that integer-like objects can be used
by Travis Oliphant 18 Feb '05

18 Feb '05

Hello again, There is a great discussion going on the numpy list regarding a proposed PEP for multidimensional arrays that is in the works. During this discussion as resurfaced regarding slicing with objects that are not IntegerType objects but that have a tp_as_number->nb_int method to convert to an int. Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number->nb_int method and calls it instead. If this is acceptable, it is an easy patch. Thanks, -Travis Oliphant

5 8

Fix _PyEval_SliceIndex (Take two)
by Travis Oliphant 18 Feb '05

18 Feb '05

(More readable second paragraph) Hello again, There is a great discussion going on the numpy list regarding a proposed PEP for multidimensional arrays that is in the works. During this discussion a problem has resurfaced regarding slicing with objects that are not IntegerType objects but that have a tp_as_number->nb_int method. Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than raising an exception if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number->nb_int method and trys it before raising an exception. If this is acceptable, it is an easy patch. Thanks, -Travis Oliphant

1 0

Prospective Peephole Transformation
by Jim Jewett 18 Feb '05

18 Feb '05

Raymond Hettinger: > tried transforming the likes of "x in (1,2,3)" into "x in frozenset([1,2,3])". >... There were substantial savings even if the set contained only a single entry. >... where x was non-hashable and it would raise a TypeError instead of > returning False as it should. I read the objection as saying that it should not return False, because an unhashable object might pretend it is equal to a hashable one in the set. """ class Searchset(frozenset): def __contains__(self, element): try: return frozenset.__contains__(self, element) except TypeError: return False """ So instead of return False it should be return x in frozenset.__iter__() This would be a net loss if there were many unhashable x. You could restrict the iteration to x that implement a custom __eq__, if you ensured that none of the SearchSet elements do... but it starts to get uglier and less general. Raymond has already look at http://www.python.org/sf/1141428, which contains some test case patches to enforce this implicit "sequences always use __eq__; only mappings can short-circuit on __hash__" contract. -jJ

1 0

Five review rule on the /dev/ page?
by Skip Montanaro 18 Feb '05

18 Feb '05

I am frantically trying to get ready to be out of town for a week of vacation. Someone sent me some patches for datetime and asked me to look at them. I begged off but referred him to http://www.python.org/dev/ and made mention of the five patch review idea. Can someone make sure that's explained on the /dev/ site? Thx, Skip

3 3