Mailman 3 November 2001 - Python-Dev

RE: [Python-Dev] Replacing __slots__ with addmembers()
by Michael McLay 13 Nov '01

13 Nov '01

Barry Warsaw writes: > Without BDFL override, yes, it does. > > >>>>> "TP" == Tim Peters <tim.one(a)home.com> writes: > > TP> I expect we will. Ditto __metatype__, long-winded super(), > TP> function-based property "declarations", and all the other new > TP> stuff. We're aiming for progress with the new features, not > TP> perfection <wink>. > > I believe Guido knows that it will be impossible to get all this stuff > right the first time, and even the 2.2 beta cycle won't shake out all > the problems. I think his intention was to get the basic > functionality in place for Python 2.2, and to clean up and improve the > syntax and semantics in future releases. I hope Python Labs doesn't rush Python 2.2 out early just to meet an artificial schedule. MAL's concerns about cluttering Python with several variations for spelling the same kind of feature is shared by me. I'm sure other long time users of Python would rather wait a few weeks to get something with fewer new warts. Not all of these changes have the same level of impact on day to day programming. The __metatype__ and super() feature will be used far less than __slots__. Spelling __slots__ in a more pythonic manner now will eliminate having lots of ugly code being supported forever. Tim suggested that the type checking feature might be enough to get the patch rejected. For the moment let's assume I had not added that feature. The patch does still makes adding member_descriptors more consistent with the syntax used to add properties. There is consistency in usage with property(fget, fset, doc) addmember(default, doc) Adding the default capability to adding a member is important because it will eliminate a very common initialization pattern. Instead of writing: class B(object): __slots__ = ['a','b'] def __init__(self,a=1,b="cats and dogs"): self.a=a self.b=b The addmember() feature reduces this to an easy to read: class B(object): a = addmember(default=1, doc="the ever popular a attribute") b = addmember(default="cats and dogs", doc="the b docstring") This also allows docstrings to be added to the attributes, a feature that isn't possible with the current __slots__ spelling. The type checking feature can easily be removed. I hope this doesn't happen. This feature replaces another very common design pattern. While dynamic typing is very powerful, there are many occasions when data is required to be of a specific type. For instance, engineering data exchange standards are filled with type checking requirements. Spacecraft crash into Mars for a lack of good checks. This is not only embarrassing, it's expensive. Examples of type checking in Python are described in: - http://mail.python.org/pipermail/python-dev/2001-October/017852.html The approach to adding this capability using addmember() is very pythonic. It is completely unintrusive to anyone who doesn't need it and it is very easy to use for those occasions when you must use it.

4 4

Re: [Python-Dev] RE: test_file failing on Windows
by Jack Jansen 12 Nov '01

12 Nov '01

Recently, "Tim Peters" <tim.one(a)home.com> said: > [Jack Jansen] > > Ah. Then, shouldn't we have an option WITHOUT_STDIO_ERRNO or > > somesuch, and ignore the errno values if it is defined? > > We already have NO_FOPEN_ERRNO; the comment says "Metroworks [sic] only". You're absolutely right. And I know it, I wrote it:-) I'll try to turn my brain ON before posting in the future, -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

order of unittest.TestCase execution?
by Skip Montanaro 12 Nov '01

12 Nov '01

I'm trying to write a regression test for dumbdbm (well, officially I'm updating the current test, but it's taking a beating). I'm unclear on the order of method execution. Can I rely on multiple test methods to be run in alphabetical order? That's the case currently. If possible, I'd like to guarantee that the db creation test is run before the others. I couldn't find the code in test_support.py or unittest.py where a test case instance with multiple methods is run. Thx, Skip

4 4

Re: [Python-Dev] RE: test_file failing on Windows
by Jack Jansen 12 Nov '01

12 Nov '01

> > Same here with CodeWarrior on the Mac: stdio errors return NULL or -1 > > and that is it, errno isn't touched, not even for fopen() file not > > found, etc. > > MS does set errno in most cases; the failure to set it for bad fopen() mode > strings appears to be a bug in their code. > > > If the ANSI standard requires errno to be set and people can point me > > to the right section I can submit an error report... > > No such luck, Jack: errno has always been mostly folklore in the C std, and > is almost pure folklore in C99. Ah. Then, shouldn't we have an option WITHOUT_STDIO_ERRNO or somesuch, and ignore the errno values if it is defined? "Cannot open file" doesn't say much, but it's better than "Errno 0". If I'm right in guessing that configure is only used on unix (am I?) then adding support for this option to configure isn't needed, I guess. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

Re: [Python-Dev] order of unittest.TestCase execution?
by Michael Hudson 12 Nov '01

12 Nov '01

Skip Montanaro <skip(a)pobox.com> writes: > Andreas> Have you been trying to put the database creation into the > Andreas> setUp() method that is executed before every testcase ? > > No, I was just trying to mimic the structure of the test_xmlrpc.py module. > It put all the cases into one class, so I did to. I realized I wanted to > run the creation test before the others and noticed that it was executing > them in alphabetical order. I was just wondering if I could rely on that or > if that was just a quirk of either the current test_support or unittest > implementations and couldn't be relied on. Well, there's this: def getTestCaseNames(self, testCaseClass): """Return a sorted sequence of method names found within testCaseClass """ testFnNames = filter(lambda n,p=self.testMethodPrefix: n[:len(p)] == p, dir(testCaseClass)) for baseclass in testCaseClass.__bases__: for testFnName in self.getTestCaseNames(baseclass): if testFnName not in testFnNames: # handle overridden methods testFnNames.append(testFnName) if self.sortTestMethodsUsing: testFnNames.sort(self.sortTestMethodsUsing) return testFnNames in unittest.py & self.sortTestMethodsUsing seems to be cmp() by default. Don't know if this is documented or subject to change, though. It seems you could define a subclass of TestLoader, override getTestCaseNames and pass an instance as the testLoader arg to unitest.main() if you want to be sure. Cheers, M. -- ARTHUR: Why should he want to know where his towel is? FORD: Everybody should know where his towel is. ARTHUR: I think your head's come undone. -- The Hitch-Hikers Guide to the Galaxy, Episode 7

2 1

Re: [Python-Dev] PyObject_GenericGetAttr vs cygwin
by Michael Hudson 12 Nov '01

12 Nov '01

Jason Tishler <jason(a)tishler.net> writes: > --CGQ4QxZ4DCp/f9YC > Content-Type: text/plain; charset=us-ascii > Content-Disposition: inline > > Michael, > > On Thu, Nov 08, 2001 at 01:52:01PM -0500, Michael Hudson wrote: > > It works. Do you want to see the patches or shall I just check the > > changes in? > > If they are just the standard "PyObject_HEAD_INIT(NULL)" style fix, then > please just commit them. Done. > > BTW, _cursesmodule.c doesn't compile; you get things like: > > > > Warning: resolving _stdscr by linking to __imp__stdscr (auto-import) > > [snip] > > build/temp.cygwin-1.3.3-i686-2.2/_cursesmodule.o: In function `PyCurses_InitScr': > > /cygdrive/c/src/python/dist/src/Modules/_cursesmodule.c:1842: undefined reference to `acs_map' > > [snip] > > I'm getting a feeling of deja vu from the above, but I can't quite > remember... I believe that the above is related to the auto-import > features recently added to Cygwin's binutils. Please send me your > > /etc/setup/installed.db > > via private email. In the meantime, I will dig some on my own. OK, this will follow (from a different machine). > > and contrary to README, > > The Cygwin section of Python README has become stale -- I need to submit > a doco patch. Sigh... > > > test_poll > > The test_poll problem was fixed by: > > http://sources.redhat.com/ml/cygwin-patches/2001-q3/msg00109.html > > and released in Cygwin 1.3.4. OK... > > and threads seem to work fine. > > There is still one known Cygwin pthreads hang. If interested, see the > following for the current state of affairs: > > http://sources.redhat.com/ml/cygwin-developers/2001-10/msg00193.html makes little sense to me, I'm afraid. Haven't had test_thread die on me yet, but then I've only run it a few times. > > test_strftime is still bust, though: > > The test_strftime problem was fixed by: > > http://sources.redhat.com/ml/newlib/2001/msg00504.html > > and released in Cygwin 1.3.4. ... but if test_poll works, how come this doesn't? How do I find out which version of cygwin I have? > I've attached the latest README, since it is the most complete description > of the current state. Thanks. Cheers, M. -- NUTRIMAT: That drink was individually tailored to meet your personal requirements for nutrition and pleasure. ARTHUR: Ah. So I'm a masochist on a diet am I? -- The Hitch-Hikers Guide to the Galaxy, Episode 9

2 1

First Draft: PEP "Switching on Multiple Values"
by M.-A. Lemburg 11 Nov '01

11 Nov '01

Here's a first draft of a PEP on the subject we recently discussed on python-dev. I intend to do a few more rounds here and then post the PEP for final discussion to python-list and python-dev. Barry, could you assign a PEP number ? Thanks. -- PEP: 02XX Title: Switching on Multiple Values Version: $Revision: 1.0 $ Author: mal(a)lemburg.com (Marc-Andre Lemburg) Status: Draft Type: Standards Track Python-Version: 2.3 Created: 10-Nov-2001 Post-History: Abstract This PEP proposes strategies to enhance Python's performance with respect to handling switching on a single variable having one of multiple possible values. Problem Up to Python 2.1, the typical way of writing multi-value switches has been to use long switch constructs of the following type: if x == 'first state': ... elif x == 'second state': ... elif x == 'third state': ... elif x == 'fourth state': ... else: # default handling ... This works fine for short switch constructs, since the overhead of repeated loading of a local (the variable x in this case) and comparing it to some constant is low (it has a complexity of O(n) on average). However, when using such a construct to write a state machine such as is needed for writing parsers the number of possible states can easily reach 10 or more cases. The current solution to this problem lies in using a dispatch table to find the case implementing method to execute depending on the value of the switch variable (this can be tuned to have a complexity of O(1) on average, e.g. by using perfect hash tables). This works well for state machines which require complex and lengthy processing in the different case methods. It does perform well for ones which only process one or two instructions per case, e.g. def handle_data(self, data): self.stack.append(data) A nice example of this is the state machine implemented in pickle.py which is used to serialize Python objects. Other prominent cases include XML SAX parsers and Internet protocol handlers. Proposed Solutions This PEP proposes two different but not necessarily conflicting solutions: 1. Adding an optimization to the Python compiler and VM which detects the above if-elif-else construct and generates special opcodes for it which use an read-only dictionary for storing jump offsets. 2. Adding new syntax to Python which mimics the C style switch statement. The first solution has the benefit of not relying on new keywords to the language, while the second looks cleaner. Both involve some run-time overhead to assure that the switching variable is immutable and hashable. Solution 1: Optimizing if-elif-else XXX This section currently only sketches the design. Issues: The new optimization should not change the current Python semantics (by reducing the number of __cmp__ calls and adding __hash__ calls in if-elif-else constructs which are affected by the optimiztation). To assure this, switching can only safely be implemented either if a "from __future__" style flag is used, or the switching variable is one of the builtin immutable types: int, float, string, unicode, etc. To prevent post-modifications of the jump-table dictionary (which could be used to reach protected code), the jump-table will have to be a read-only type (e.g. a read-only dictionary). The optimization should only be used for if-elif-else constructs which have a minimum number of n cases (where n is a number which has yet to be defined depending on performance tests). Implementation: It should be possible for the compiler to detect an if-elif-else construct which has the following signature: if x == 'first':... elif x == 'second':... else:... (ie. the left hand side alwys references the same variable, the right hand side some hashable immutable builtin type) The compiler could then setup a read-only (perfect) hash table, store it in the constants and add an opcode SWITCH which triggers the following run-time behaviour: At runtime, SWITCH would check x for being one of the well-known immutable types (strings, unicode, numbers) and use the hash table for finding the right opcode snippet. Solutions 2: Adding a switch statement to Python XXX This section currently only sketches the design. Syntax: switch EXPR: case CONSTANT: [suite] case CONSTANT: [suite] ... else: [suite] (modulo indentation variations) Implementation: The compiler would have to generate code similar to this: def whatis(x): switch(x): case 'one': print '1' case 'two': print '2' case 'three': print '3' else: print "D'oh!" into (ommitting POP_TOP's and SET_LINENO's): 6 LOAD_FAST 0 (x) 9 LOAD_CONST 1 (switch-table-1) 12 SWITCH 26 (to 38) 14 LOAD_CONST 2 ('1') 17 PRINT_ITEM 18 PRINT_NEWLINE 19 JUMP 43 22 LOAD_CONST 3 ('2') 25 PRINT_ITEM 26 PRINT_NEWLINE 27 JUMP 43 30 LOAD_CONST 4 ('3') 33 PRINT_ITEM 34 PRINT_NEWLINE 35 JUMP 43 38 LOAD_CONST 5 ("D'oh!") 41 PRINT_ITEM 42 PRINT_NEWLINE >>43 LOAD_CONST 0 (None) 46 RETURN_VALUE Where the 'SWITCH' opcode would jump to 14, 22, 30 or 38 depending on 'x'. Issues: The switch statement should not implement fall-through behaviour (as does the switch statement in C). Each case defines a complete and independent suite; much like in a if-elif-else statement. This also enables using break in switch statments inside loops. There have been other proposals for the syntax which reuse existing keywords and avoid adding two new ones ("switch" and "case"). Others have argued that the keywords should use new terms to avoid confusion with the C keywords of the same name but slightly different semantics (e.g. fall-through without break). Some of the proposed variants: case EXPR: of CONSTANT: [suite] of CONSTANT: [suite] else: [suite] case EXPR: if CONSTANT: [suite] if CONSTANT: [suite] else: [suite] when EXPR: in CONSTANT_TUPLE: [suite] in CONSTANT_TUPLE: [suite] ... else: [suite] The switch statement could be extended to allow tuples of values for one section (e.g. case 'a', 'b', 'c': ...). Another proposed extension would allow ranges of values (e.g. case 10..14: ...). These should probably be post-poned, but already kept in mind when designing and implementing a first version. Scope XXX Explain "from __future__ import switch" Credits Martin von Löwis (issues with the optimization) Thomas Wouters (switch statement + byte code compiler example) Skip Montanaro (dispatching ideas) Donald Beaudry (switch syntax) Greg Ewing (switch syntax) Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil End: -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/

2 2

Re: [Python-Dev] Re: Proposal for a modified import mechanism.
by Michael McLay 11 Nov '01

11 Nov '01

mal> will have to be a read-only type (e.g. a read-only mal> dictionary). SM>There was discussion once upon a time about adding a read-write flag to SM>dictionaries to prevent modification during sensitive times. I gather SM> lists already have such a flag to allow in-place sorting to work. The member description returned by addmembers() allows types to be specified for an attribute. It would be trivial to add another flag to mark the member as read only. You could also add methods to the PyExtendedMemberDescrObject to allow the state to be turned on at the begining of the switch and off at the end of the switch.

1 0

Proposal for a modified import mechanism.
by Prabhu Ramachandran 11 Nov '01

11 Nov '01

Hi, Sorry about the cross posting. Over the last couple of weeks I described a problem and inconsistency with the way Python imports modules. For more details look at this thread: http://mail.python.org/pipermail/python-list/2001-November/070719.html In short, currently, import allows one to use non-absolute module names for modules that are in the current directory. If a module is not in the same directory, import then looks for modules in sys.path. Consequently, dealing with packages that are re-nested is a pain. Complex package structure also causes problems. I'd like to note that I was also not the only person who suffered from this issue -- four others on mentioned similar problems and some asked me to let them know if I found a solution. Subsequently, I proposed another approach that first looks in the local directory and then walks up the current package tree looking for modules before looking at sys.path. I also modified knee.py to obtain a working solution. More information is here: http://mail.python.org/pipermail/python-list/2001-November/071212.html the threading is messed up and starts here: http://mail.python.org/pipermail/python-list/2001-November/071218.html You can find the new module and a simple test package here: http://av.stanford.edu/~prabhu/download/ There is also a slightly enhanced version of knee.py included that supports caching module lookup failures suggested by Rainer and Gordon in: http://mail.python.org/pipermail/python-list/2001-November/071218.html it also fixes a bug where the parent package is an extension module. I therefore have a working import style that seems to handle importing packages in a more natural(?) and consistent(?) manner. I've also tested it out with a large package like scipy (http://www.scipy.org) with no trouble or significant performance problems: http://mail.python.org/pipermail/python-list/2001-November/071325.html I'd like to ask the Python developers if they'd consider (a) changing the way the current import works to do what I proposed, or, (b) add a new keyword like 'rimport' (or something) that does this recursive search through parent packages and loads modules. This was actually suggested by Gordon McMillan. Gordon actually suggested something stronger -- import only supports absolute names, rimport is relative import and rrimport is a recursive relative import. But this would break the current import since import currently aupports some relative lookup. So maybe import and rimport is a workable solution? (c) patch the existing knee.py with my fixes. Note: these fixes have nothing to do with the recirsive module lookup stuff -- knee.py is merely an improved version of the older one. Thanks for listening patiently and sorry again for all the cross posting. prabhu

2 2

Psyco 0.3.1
by Armin Rigo 10 Nov '01

10 Nov '01

Hello everybody, Finally, the first usable version of Psyco. It compiles small but realistic code examples. http://homepages.ulb.ac.be/~arigo/psyco/psyco-0.3.1.tgz The included text files can also be read at http://homepages.ulb.ac.be/~arigo/psyco/psyco-0.3.1/README.txt and http://homepages.ulb.ac.be/~arigo/psyco/psyco-0.3.1/ISSUES.txt . Not all Python bytecodes are implemented, but you can already do quite a lot. The missing ones are the ones that cannot be really optimized any more, so they should be easy to implement as a call to a run-time function. I used Python 2.2b1's new typing system to figure out when an attribute is a method. It should still compile with older Python version but you will not get method access optimization. There are a few cases in which Psyco does not exactly follow Python's semantics, mainly with globals/builtins. Builtins are assumed never to change, and we assume that the user will not later add or remove a global variable that will shadow or expose a builtin. This issue and others are discussed in detail in ISSUES.txt. Produced code buffers are never freed. This is a problem for larger examples. See ISSUES.txt. This file will also give you cool examples of Python code which are known tricks to speed up the interpreter but which are no longer needed -- Psyco is able to optimize them correctly. One such nice example is 'range', which is now actually faster than 'xrange', and completely transparent (for x in sequence is equivalent to for i in range(len(sequence)): x = sequence[i]). Inspecting the produced machine code is now possible if you have GDB (the GNU debugger) installed. There is a script that presents the produced cross-calling code buffers as cross-linked HTML pages. And finally... (drums...) some timing results. This is for my notebook Intel PIII 700Mhz with all optimization turned on. Note however that none of these small examples is typical of Python code; they are all small compact algorithms not using complex data structures. f1() is designed specifically for the purpose of showing how fast Psyco is ;-) It works on integers only, and is not at all a real-world algorithm. f2() and f3() are too trivial to be timed. f4() is a function that loads a list of files and counts the number of times each character appears. Ok, this is also a case in which Psyco will clearly win. f5() is like f4() but completely loads the files in memory, whereas f4() loads them line by line. f6() computes the factorial modulo p, using long integers, whose operations are not optimized by Psyco. f7() is the Mandelbrot set. Float/complex not optimized by Psyco. In the latter two examples the gain is only on the stuff around the actual computations. The table is in seconds, as returned by time.clock(), so the numbers don't include delays for loading the (relatively large) code of Psyco itself nor any swapping. Psyco, first call Psyco, next calls Python -OO go1() 0.03 0.02 to 0.03 3.63 go4() 0.35 0.33 3.59 go5() 0.29 0.27 3.47 go6() 0.87 0.85 1.03 to 1.07 go7() 2.16 to 2.18 2.16 to 2.18 2.83 Interesting, isn't it ? A bientot, Armin.

1 0