Mailman 3 November 2001 - Python-Dev

pickle/cPickle raises SystemError
by barry＠zope.com 14 Nov '01

14 Nov '01

Did anybody else know that pickle and cPickle can raise a SystemError? I'm working on a rewrite of the pickle documentation and was playing around with some of the undefined corners when I stumbled across this. SystemError is clearly the wrong exception to raise, given the way the exception is documented in the exceptions module! I've submitted SF bug #481882 which describes the problem and contains candidate patches (currently only for pickle.py). I've no idea why pickle.py's find_class() is masking KeyError, AttributeError, and ImportError and transforming them into SystemError. I think the exceptions should just be allowed to percolate up. Alternatively (although less correct, IMO) would be to transform them into UnpicklingErrors. The Python test suite seems to not cover this point, and it's not documented anywhere that I can tell, so I feel justified in fixing this for Python 2.2. I'm concerned about losing backwards compatibility with commonly held lore though, so I'm wondering if anybody out there counts on exactly a SystemError being raised in this situation? -Barry -------------------- snip snip -------------------- Python 2.2b1+ (#3, Nov 13 2001, 18:06:11) [GCC 2.95.3 19991030 (prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pickle >>> pickle.loads('c__builtin__\noops\np0\n.') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/barry/projects/python/Lib/pickle.py", line 979, in loads return Unpickler(file).load() File "/home/barry/projects/python/Lib/pickle.py", line 588, in load dispatch[key](self) File "/home/barry/projects/python/Lib/pickle.py", line 805, in load_global klass = self.find_class(module, name) File "/home/barry/projects/python/Lib/pickle.py", line 815, in find_class raise SystemError, \ SystemError: Failed to import class oops from module __builtin__ >>> import cPickle >>> cPickle.loads('c__builtin__\noops\np0\n.') Traceback (most recent call last): File "<stdin>", line 1, in ? SystemError: Failed to import class oops from module __builtin__

2 1

Python's footprint
by Gustavo Niemeyer 14 Nov '01

14 Nov '01

Hello! I've been thinking about ways to reduce python's footprint, so we'd be able to include the interpreter in environments requiring reduced applications. One of the tests I've done was to remove every inlined documentation. Please, take a look at the results using a stripped binary of python 2.2: With inline docs: -rwxrwxr-x 1 niemeyer niemeyer 634452 Nov 8 17:05 python Without inline docs: -rwxrwxr-x 1 niemeyer niemeyer 576852 Nov 8 17:12 python It means that about 10% of python's executable is documentation. Now I'm wondering if something like a DOCSTRING("foo") macro would be valid in that case. If the user disabled it trough --disable-doc, for example, DOCSTRING() would return "". Thanks! -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

4 7

Experimental features and __slots__
by Martin v. Loewis 14 Nov '01

14 Nov '01

Recently, there has been a concern that if __slots__ is added to the language, we'll be stuck with it forever. I want to propose a strategy that doesn't require things to be cast in stone right now. PEP 5 should be enhanced to allow the notion of experimental features. Programs using experimental features should expect to break from one release to another, with no need to deprecate the experimental feature. If desired, __future__ import could be required to activate experimental features, with a prospect that they disappear in the next release. However, this is not strictly necessary: Documenting experimental features may be sufficient. For 2.2, I propose that all __ attributes relating to new-style classes are experimental. Regards, Martin

1 0

__slots__ came from?
by Paul Dubois 14 Nov '01

14 Nov '01

I wanted to learn more about this __slots__ (proposal?)(decision?) because of the possible relationship to the Properties package that is part of Numeric/MA. I can't find the PEP that relates to it. Help?

2 1

PyDict_GetItem & dict_subscript
by Thomas Wouters 14 Nov '01

14 Nov '01

While trying to figure out why passing an unhashable object to switch didn't cause a TypeError: unhashable object, I found out PyDict_GetItem and dict_subscript behave differently in that respect. PyDict_GetItem does: hash = PyObject_Hash(key); if (hash == -1) { PyErr_Clear(); return NULL; } whereas dict_subscript does: hash = PyObject_Hash(key); if (hash == -1) return NULL; PyDict_SetItem and PyDict_DelItem both behave like dict_subscript (that is, they propagate the error upwards, instead of clearing the error.) Is this intentional ? Why ? :P And what is the 'right' way to do a PyDict_GetItem() that gives the proper error-message without relying on the need for that particular dict (which could be a subtype that has more or less restrictions) to have hashable keys ? -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

3 3

Re: [Python-Dev] Re: psyco
by Michael Hudson 14 Nov '01

14 Nov '01

[python.net mail unblocks again, sigh] Armin Rigo <arigo(a)ulb.ac.be> writes: > Hello everybody, > > > 2X on pystone is nothing to snort at -- it's good! > > Thanks, but I believe that much better results can be > obtained. However, I have a problem here. I am starting to run out > of free time for Psyco. Shame! > I would like to put efforts in helping people understand how Psyco > works, so that it can be developped up to a really usable state -- > it is currently more of a proof-of-concept, but it would be just too > bad to loose this opportunity, I guess. Besides, I believe that with > some more work we could really turn the current version into > something great. I've been meaning to take a good look at pysco since you first announced it, but have never found the time/inspiration needed. It would be a shame if it were to die. > That's just too much for me alone now (I should be doing a math > Ph.D. these days you know ;-). Same here! ...googles... logic, eh? Nutter <wink>. Out of curiosity, do you know people at vub? I was at a conference with Michel van den Bergh and Tor Lowen in September... (small world). > So my request is, Does anyone feel like investing some time in Psyco > just on the basis of the current preliminary results ? Shall I > write a technical in-depth overview ? I you can find the time to write it, I'll do my best to find the time to read it properly. What else is the first year of a PhD for? Cheers, M. -- [Perl] combines all the worst aspects of C and Lisp: a billion different sublanguages in one monolithic executable. It combines the power of C with the readability of PostScript. -- Jamie Zawinski

1 0

Re: psyco
by Armin Rigo 13 Nov '01

13 Nov '01

Hello Kjetil, > tried to give the latest psyco a whirl. more specifically twist pystone a > bit to use a psyco proxy to get an idea of the speedup for the benchmark. > now, the downsize is that psyco segfaults with an assertion failure ;) A few trivial and one more subtle bug prevented any seriously-sized test to complete. I fixed it. http://homepages.ulb.ac.be/~arigo/psyco The pystone benchmark is not at all typical Python code :-) psyco still performs a x2 speed-up but this is not representative of possible results (it suffers from missing knowledge about integer multiplications and divisions as well as (more importantly) methods of user classes). Armin

3 3

Caching directory files in import.c
by James C. Ahlstrom 13 Nov '01

13 Nov '01

I have a new version of my zip importing code. As before, it reads the file names from zipfiles and records them in a global dictionary to speed up finding zip imports. But what about imports from directories? Looking at the code, I saw that I could do an os.listdir(path), and record the directory file names into the same dictionary. Then it would not be necessary to perform a large number of fopen()'s. The same dictionary lookup is used instead. Is this a good idea??? It seems it should be faster when a "large" percentage of files in a directory are imported. It should be slower when only one file is imported from a directory with many names. I think I remember people discussing this before. Is the speedup real and worth the slight amount of additional code? JimA

1 3

PEP 275: "Switching on Multiple Values", Rev 1.1
by M.-A. Lemburg 13 Nov '01

13 Nov '01

Here's an updated version of the PEP. You will also find an HTML copy at: http://python.sourceforge.net/peps/pep-0275.html -- PEP: 0275 Title: Switching on Multiple Values Version: $Revision: 1.1 $ Author: mal(a)lemburg.com (Marc-Andre Lemburg) Status: Draft Type: Standards Track Python-Version: 2.3 Created: 10-Nov-2001 Post-History: Abstract This PEP proposes strategies to enhance Python's performance with respect to handling switching on a single variable having one of multiple possible values. Problem Up to Python 2.2, the typical way of writing multi-value switches has been to use long switch constructs of the following type: if x == 'first state': ... elif x == 'second state': ... elif x == 'third state': ... elif x == 'fourth state': ... else: # default handling ... This works fine for short switch constructs, since the overhead of repeated loading of a local (the variable x in this case) and comparing it to some constant is low (it has a complexity of O(n) on average). However, when using such a construct to write a state machine such as is needed for writing parsers the number of possible states can easily reach 10 or more cases. The current solution to this problem lies in using a dispatch table to find the case implementing method to execute depending on the value of the switch variable (this can be tuned to have a complexity of O(1) on average, e.g. by using perfect hash tables). This works well for state machines which require complex and lengthy processing in the different case methods. It does not perform well for ones which only process one or two instructions per case, e.g. def handle_data(self, data): self.stack.append(data) A nice example of this is the state machine implemented in pickle.py which is used to serialize Python objects. Other prominent cases include XML SAX parsers and Internet protocol handlers. Proposed Solutions This PEP proposes two different but not necessarily conflicting solutions: 1. Adding an optimization to the Python compiler and VM which detects the above if-elif-else construct and generates special opcodes for it which use an read-only dictionary for storing jump offsets. 2. Adding new syntax to Python which mimics the C style switch statement. The first solution has the benefit of not relying on adding new keywords to the language, while the second looks cleaner. Both involve some run-time overhead to assure that the switching variable is immutable and hashable. Solution 1: Optimizing if-elif-else XXX This section currently only sketches the design. Issues: The new optimization should not change the current Python semantics (by reducing the number of __cmp__ calls and adding __hash__ calls in if-elif-else constructs which are affected by the optimiztation). To assure this, switching can only safely be implemented either if a "from __future__" style flag is used, or the switching variable is one of the builtin immutable types: int, float, string, unicode, etc. (not subtypes, since it's not clear whether these are still immutable or not) To prevent post-modifications of the jump-table dictionary (which could be used to reach protected code), the jump-table will have to be a read-only type (e.g. a read-only dictionary). The optimization should only be used for if-elif-else constructs which have a minimum number of n cases (where n is a number which has yet to be defined depending on performance tests). Implementation: It should be possible for the compiler to detect an if-elif-else construct which has the following signature: if x == 'first':... elif x == 'second':... else:... i.e. the left hand side always references the same variable, the right hand side a hashable immutable builtin type. The right hand sides need not be all of the same type, but they should be comparable to the type of the left hand switch variable. The compiler could then setup a read-only (perfect) hash table, store it in the constants and add an opcode SWITCH in front of the standard if-elif-else byte code stream which triggers the following run-time behaviour: At runtime, SWITCH would check x for being one of the well-known immutable types (strings, unicode, numbers) and use the hash table for finding the right opcode snippet. If this condition is not met, the interpreter should revert to the standard if-elif-else processing by simply skipping the SWITCH opcode and procedding with the usual if-elif-else byte code stream. Solutions 2: Adding a switch statement to Python XXX This section currently only sketches the design. Syntax: switch EXPR: case CONSTANT: SUITE case CONSTANT: SUITE ... else: SUITE (modulo indentation variations) The "else" part is optional. If no else part is given and none of the defined cases matches, a ValueError is raised. Implementation: The compiler would have to compile this into byte code similar to this: def whatis(x): switch(x): case 'one': print '1' case 'two': print '2' case 'three': print '3' else: print "D'oh!" into (ommitting POP_TOP's and SET_LINENO's): 6 LOAD_FAST 0 (x) 9 LOAD_CONST 1 (switch-table-1) 12 SWITCH 26 (to 38) 14 LOAD_CONST 2 ('1') 17 PRINT_ITEM 18 PRINT_NEWLINE 19 JUMP 43 22 LOAD_CONST 3 ('2') 25 PRINT_ITEM 26 PRINT_NEWLINE 27 JUMP 43 30 LOAD_CONST 4 ('3') 33 PRINT_ITEM 34 PRINT_NEWLINE 35 JUMP 43 38 LOAD_CONST 5 ("D'oh!") 41 PRINT_ITEM 42 PRINT_NEWLINE >>43 LOAD_CONST 0 (None) 46 RETURN_VALUE Where the 'SWITCH' opcode would jump to 14, 22, 30 or 38 depending on 'x'. Issues: The switch statement should not implement fall-through behaviour (as does the switch statement in C). Each case defines a complete and independent suite; much like in a if-elif-else statement. This also enables using break in switch statments inside loops. If the interpreter finds that the switch variable x is not hashable, it should raise a TypeError at run-time pointing out the problem. There have been other proposals for the syntax which reuse existing keywords and avoid adding two new ones ("switch" and "case"). Others have argued that the keywords should use new terms to avoid confusion with the C keywords of the same name but slightly different semantics (e.g. fall-through without break). Some of the proposed variants: case EXPR: of CONSTANT: SUITE of CONSTANT: SUITE else: SUITE case EXPR: if CONSTANT: SUITE if CONSTANT: SUITE else: SUITE when EXPR: in CONSTANT_TUPLE: SUITE in CONSTANT_TUPLE: SUITE ... else: SUITE The switch statement could be extended to allow tuples of values for one section (e.g. case 'a', 'b', 'c': ...). Another proposed extension would allow ranges of values (e.g. case 10..14: ...). These should probably be post-poned, but already kept in mind when designing and implementing a first version. Examples: switch EXPR: switch x: case CONSTANT: case "first": SUITE print x case CONSTANT: case "second": SUITE x = x**2 ... print x else: else: SUITE print "whoops!" case EXPR: case x: of CONSTANT: of "first": SUITE print x of CONSTANT: of "second": SUITE print x**2 else: else: SUITE print "whoops!" case EXPR: case state: if CONSTANT: if "first": SUITE state = "second" if CONSTANT: if "second": SUITE state = "third" else: else: SUITE state = "first" when EXPR: when state: in CONSTANT_TUPLE: in ("first", "second"): SUITE print state in CONSTANT_TUPLE: state = next_state(state) SUITE in ("seventh",): ... print "done" else: break # out of loop! SUITE else: print "middle state" state = next_state(state) Scope XXX Explain "from __future__ import switch" Credits Martin von Löwis (issues with the optimization idea) Thomas Wouters (switch statement + byte code compiler example) Skip Montanaro (dispatching ideas, examples) Donald Beaudry (switch syntax) Greg Ewing (switch syntax) Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil End: -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

4 5

switch in Python.
by Thomas Wouters 13 Nov '01

13 Nov '01

I uploaded a proof-of-concept of the switch statement to SourceForge: https://sourceforge.net/tracker/index.php?func=detail&aid=481118&group_id=5… It's a diff against the current CVS tree, taken from 'src/' -- so apply in src/ with -p0. It's a quick and almost certainly buggy hack. It bypasses the c_consts_dict in an ugly way, and might break code that tries to do stuff with code objects. It probably doesn't keep track of the stack properly, and skips SET_LINENO opcodes. It only allows string or numerical literals, though it does get the '1000 == 1000.0 == 1000L' trick for free. It doesn't contain changes to docs or dis.py, and it doesn't use a future statement. It _does_ contain a change to distutils, because it uses a variable 'switch' :-) And I haven't profiled it in any way. In my defense, I don't recall writing it, so I was probably sleeping at the time (or at least I should have been.) Anyway, here's how it works: """ def whatis(x): switch x: case "number one": return "The Larch." case "spam", "Spam", "SPAM": return "It's " + "SPAM "*31 case 1.1: return "one dot one" case 1000L: return "a lot." case 1j: return "Not *really* there." else: return "D'oh, donno that!" for val in (1j, "number one", 1.1, 1000, 1000L, 1000.0, 1000.0005, "SPAM", "spam", "spaM", "eggs and ham"): print "What is", repr(val), "?" print whatis(val) print """ thomas@stalker:~/python-cvs/dist/src$ ./python example.py What is 1j ? Not *really* there. What is 'number one' ? The Larch. What is 1.1000000000000001 ? one dot one What is 1000 ? a lot. What is 1000L ? a lot. What is 1000.0 ? a lot. What is 1000.0005 ? D'oh, donno that! What is 'SPAM' ? It's SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM What is 'spam' ? It's SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM SPAM What is 'spaM' ? D'oh, donno that! What is 'eggs and ham' ? D'oh, donno that! -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

2 2