Python-Dev September 2004

python-dev@python.org

113 participants
122 discussions

by Neil Schemenauer

Perhaps this is more approprate for python-list but I looks like a bug to me. Example code: class A: def __str__(self): return u'\u1234' '%s' % u'\u1234' # this works '%s' % A() # this doesn't work It will work if 'A' subclasses from 'unicode' but should not be necessary, IMHO. Any reason why this shouldn't be fixed? Neil

15 years

RE: [Python-Dev] Those import related syntax errors again...

by Samuele Pedroni

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

15 years, 1 month

mimetypes and _winreg

by Mike Brown

I thought it would be nice to try to improve the mimetypes module by having it, on Windows, query the Registry to get the mapping of filename extensions to media types, since the mimetypes code currently just blindly checks posix-specific paths for httpd-style mapping files. However, it seems that the way to get mappings from the Windows registry is excessively slow in Python. I'm told that the reason has to do with the limited subset of APIs that are exposed in the _winreg module. I think it is that EnumKey(key, index) is querying for the entire list of subkeys for the given key every time you call it. Or something. Whatever the situation is, the code I tried below is way slower than I think it ought to be. Does anyone have any suggestions (besides "write it in C")? Could _winreg possibly be improved to provide an iterator or better interface to get the subkeys? (or certain ones? There are a lot of keys under HKEY_CLASSES_ROOT, and I only need the ones that start with a period). Should I file this as a feature request? Thanks -Mike from _winreg import HKEY_CLASSES_ROOT, OpenKey, EnumKey, QueryValueEx i = 0 typemap = {} try: while 1: subkeyname = EnumKey(HKEY_CLASSES_ROOT, i) try: subkey = OpenKey(HKEY_CLASSES_ROOT, subkeyname) if subkeyname[:1] == '.': data = QueryValueEx(subkey, 'Content Type')[0] print subkeyname, '=', data typemap[subkeyname] = data # data will be unicode except EnvironmentError, WindowsError: pass i += 1 except WindowsError: pass

15 years, 2 months

assert failure on obmalloc

by Jeremy Hylton

Failure running the test suite today with -u compiler enabled on Windows XP. test_logging Assertion failed: bp != NULL, file \code\python\dist\src\Objects\obmalloc.c, line 604 The debugger says the error is here: msvcr71d.dll!_assert(const char * expr=0x1e22bcc0, const char * filename=0x1e22bc94, unsigned int lineno=604) Line 306 C python24_d.dll!PyObject_Malloc(unsigned int nbytes=100) Line 604 + 0x1b C python24_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=84) Line 1014 + 0x9 C python24_d.dll!PyThreadState_New(_is * interp=0x00951028) Line 136 + 0x7 C python24_d.dll!PyGILState_Ensure() Line 430 + 0xc C python24_d.dll!t_bootstrap(void * boot_raw=0x02801d48) Line 431 + 0x5 C python24_d.dll!bootstrap(void * call=0x04f0d264) Line 166 + 0x7 C msvcr71d.dll!_threadstart(void * ptd=0x026a2320) Line 196 + 0xd C I've been seeing this sort of error on-and-off for at least a year with my Python 2.3 install. It's the usual reason my spambayes popproxy dies. I can't recell seeing it before on Windows or while running the test suite. Jeremy

15 years, 5 months

[OT] stats on type-inferencing atomic types in local variables in the stdlib

by Brett C.

My thesis (which, for those who don't know, was to come up with a way to do type inferencing in the compiler without requiring any semantic changes; basically type inferencing atomic types assigned to local variables) is now far enough long that I have the algorithm done and I can generate statistics on what opcodes are called with the most common types that I can specifically infer (can also do static type checking on occasion; only triggers were actual unit tests making sure TypeError was raised for certain things like ``~4.2`` and such). Thought some of you might get a kick out of this since the numbers are rather blatent for certain opcodes and methods. To read the stats, the number to the left is the number of times the opcode was compiled (not executed!) with the specific type(s) known for the opcode (if it took two args, then both types are listed; order was considered irrelevant). Now they are listed as integers, so here is the conversion:: Basestring 4 IntegralType 8 FloatType 16 ImagType 32 DictType 64 ListType 128 TupleType 256 For the things named "meth_<something>" that is the method being called immediately on the type. Now realize these numbers are only for opcodes where I could definitely infer the type; ones where it could be more than one type, regardless if those possibilities were very specific, I just ignored it and did not include in the stats. I also tweaked some opcodes knowing how they are more often used. So, for instance, BINARY_MODULO checks specifically for the case of when the left side is a basestring and then just doesn't worry about the other args. Other ones I just didn't bother with all the args since it was not interesting to me in terms of deciding what type-specific opcodes I want to come up with. Anyway, here are the numbers on Lib sans Lib/test (129,814 lines according to SLOCCount) for the ones above 100:: (101, ('BINARY_MULTIPLY', (8, 4))), (106, ('BINARY_SUBSCR', 128)), (118, ('GET_ITER', 128)), (124, ('BINARY_MODULO', None)), (195, ('meth_join', 4)), (204, ('BINARY_ADD', (8, 8))), (331, ('BINARY_ADD', (4, 4))), (513, ('BINARY_LSHIFT', (8, 8))), (840, ('meth_append', 128)), (1270, ('PRINT_ITEM', 4)), (1916, ('BINARY_MODULO', 4)), (12302, ('STORE_SUBSCR', 64))] We sure like our dictionaries (for those that don't know, dictionaries are created by making an empty dict and then basically doing an indivual assignment for each value). We also seem to love to use string interpolation, and printing stuff. Using list.append is also popular. Now the BINARY_LSHIFT is rather interesting, and that ties into the whole issue of how much I can actually infer; since binary work tends to be with all constants I can infer it really easily and so its frequency is rather high. Its actual frequency of use, though, compared to other things probably is not high, though. Plus I doubt Plone, for instance, uses ``<<`` very often so I suspect the opcode will get weeded out when I incorporate stats from the other apps I am taking stats from. As for the stuff I cut out, the surprising thing from those numbers was how few mathematical expressions could be inferred. I checked my numbers with grep and there really is only 3 times where a float constant is divided by a float constant (and they are all in colorsys). I was not expecting that at all. Guess global variables or object attributes tend to have them or I just can't infer the values. Either way I just wasn't expecting that. Anyway, as I said I just thought some people might find this interesting. Don't read into this too much since I am just using these numbers as guidelines for type-specific opcodes to write for use as a quantifiable measurement of the usefulness of type inferencing like this. -Brett P.S.: anyone who is *really* interested I can send you the full stats for the apps I have run my modified version of compile.c against.

15 years, 6 months

Proposing a sys.special_exceptions tuple

by Nick Coghlan

I spent some time the other day looking at the use of bare except statements in the standard library. Many of them seemed to fall into the category of 'need to catch anything user code is likely to throw, but shouldn't be masking SystemExit, StopIteration, KeyboardInterrupt, MemoryError, etc'. Changing them to "except Exception:" doesn't help, since all of the above still fit into that category (Tim posted a message recently about rearranging the Exception heirarchy to fix this. Backwards compatibility woes pretty much killed the discussion though). However, another possibility occurred to me: try: # Do stuff except sys.special_exceptions: raise except: # Deal with all the mundane stuff With an appropriately defined tuple, that makes it easy for people to "do the right thing" with regards to critical exceptions. Such a tuple could also be useful for invoking isinstance() and issubclass(). Who knows? If something like this caught on, it might some day be possible to kill a Python script with a single press of Ctrl-C };> Cheers, Nick. -- Nick Coghlan Brisbane, Australia

15 years, 6 months

Running a module as a script

by Nick Coghlan

Patch # 1035498 attempts to implement the semantics suggested by Ilya and Anthony and co. "python -m module" Means: - find the source file for the relevant module (using the standard locations for module import) - run the located script as __main__ (note that containing packages are NOT imported first - it's as if the relevant module was executed directly from the command line) - as with '-c' anything before the option is an argument to the interpreter, anything after is an argument to the script The allowed modules are those whose associated source file meet the normal rules for a command line script. I believe that means .py and .pyc files only (e.g. "python -m profile" works, but "python -m hotshot" does not). Special import hooks (such as zipimport) almost certainly won't work (since I don't believe they work with the current command line script mechanism). Cheers, Nick. -- Nick Coghlan Brisbane, Australia

15 years, 6 months

Another test_compiler mystery

by Tim Peters

I've noticed several times now, in both debug and release builds, that if I run regrtest.py with -uall, *sometimes* it just stops after running test_compiler: $ python_d regrtest.py -uall test_grammar test_opcodes ... test_compare test_compile test_compiler $ There's no indication of error, it just ends. It's not consistent. Happened once when I was running with -v, and test_compiler's output ended here: ... compiling C:\Code\python\lib\test\test_operator.py compiling C:\Code\python\lib\test\test_optparse.py compiling C:\Code\python\lib\test\test_os.py compiling C:\Code\python\lib\test\test_ossaudiodev.py compiling C:\Code\python\lib\test\test_parser.py In particular, there's no Ran M tests in Ns output, so it doesn't look like unittest (let alone regrtest) ever got control back. Hmm. os.listdir() is in sorted order on NTFS, so test_compiler should be chewing over a lot more files after test_parser.py. *This* I could blame on a blown C stack -- although I'd expect a much nastier symptom then than just premature termination. Anyone else?

15 years, 6 months

Re: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration

by Phillip J. Eby

At 11:36 PM 9/30/04 +0100, Michael Sparks wrote: >On Thu, 30 Sep 2004, Phillip J. Eby wrote: >... > > A mechanism to pass values or exceptions into generators > >[ Possibly somewhat off topic, and apologies if it is, and I'm positive > someone's done something similar before, but I think it's relevant to > the discussion in hand -- largely because the above use case *doesn't* > require changes to python... ] I know it doesn't; peak.events does this now, but in order to have a decent programmer interface, the implementation involves a fair amount of magic. Similarly, PEP 334 doesn't call for anything you can't do with a bit of work and magic. I was suggesting, however, that a true "simple co-routine" (similar to a generator but with bidirectional communication of values and exceptions) would be a valuable addition to the language, in the area of simplifying async programming in e.g. Twisted and peak.events. To put it another way: Slap the current title of PEP 334 onto the body of PEP 288, and change its syntax so you have a way to pass values and exceptions *in* to a suspended "coroutine" (a new and different animal from a generator), and I'm sold.

15 years, 6 months

PEP 334 - Simple Coroutines via SuspendIteration

by Clark C. Evans

I've packaged up the idea of a coroutine facility using iterators and an exception, SuspendIteration. This would require some rather deep changes to how generators are implemented, however, it seems backwards compatible, implementable /w JVM or CLR, and would make most of my database/web development work far more pleasant. http://www.python.org/peps/pep-0334.html Cheers! Clark ... PEP: 334 Title: Simple Coroutines via SuspendIteration Version: $Revision: 1.1 $ Last-Modified: $Date: 2004/09/08 00:11:18 $ Author: Clark C. Evans <info(a)clarkevans.com> Status: Draft Type: Standards Track Python-Version: 3.0 Content-Type: text/x-rst Created: 26-Aug-2004 Post-History: Abstract ======== Asynchronous application frameworks such as Twisted [1]_ and Peak [2]_, are based on a cooperative multitasking via event queues or deferred execution. While this approach to application development does not involve threads and thus avoids a whole class of problems [3]_, it creates a different sort of programming challenge. When an I/O operation would block, a user request must suspend so that other requests can proceed. The concept of a coroutine [4]_ promises to help the application developer grapple with this state management difficulty. This PEP proposes a limited approach to coroutines based on an extension to the iterator protocol [5]_. Currently, an iterator may raise a StopIteration exception to indicate that it is done producing values. This proposal adds another exception to this protocol, SuspendIteration, which indicates that the given iterator may have more values to produce, but is unable to do so at this time. Rationale ========= There are two current approaches to bringing co-routines to Python. Christian Tismer's Stackless [6]_ involves a ground-up restructuring of Python's execution model by hacking the 'C' stack. While this approach works, its operation is hard to describe and keep portable. A related approach is to compile Python code to Parrot [7]_, a register-based virtual machine, which has coroutines. Unfortunately, neither of these solutions is portable with IronPython (CLR) or Jython (JavaVM). It is thought that a more limited approach, based on iterators, could provide a coroutine facility to application programmers and still be portable across runtimes. * Iterators keep their state in local variables that are not on the "C" stack. Iterators can be viewed as classes, with state stored in member variables that are persistent across calls to its next() method. * While an uncaught exception may terminate a function's execution, an uncaught exception need not invalidate an iterator. The proposed exception, SuspendIteration, uses this feature. In other words, just because one call to next() results in an exception does not necessarily need to imply that the iterator itself is no longer capable of producing values. There are four places where this new exception impacts: * The simple generator [8]_ mechanism could be extended to safely 'catch' this SuspendIteration exception, stuff away its current state, and pass the exception on to the caller. * Various iterator filters [9]_ in the standard library, such as itertools.izip should be made aware of this exception so that it can transparently propagate SuspendIteration. * Iterators generated from I/O operations, such as a file or socket reader, could be modified to have a non-blocking variety. This option would raise a subclass of SuspendIteration if the requested operation would block. * The asyncore library could be updated to provide a basic 'runner' that pulls from an iterator; if the SuspendIteration exception is caught, then it moves on to the next iterator in its runlist [10]_. External frameworks like Twisted would provide alternative implementations, perhaps based on FreeBSD's kqueue or Linux's epoll. While these may seem dramatic changes, it is a very small amount of work compared with the utility provided by continuations. Semantics ========= This section will explain, at a high level, how the introduction of this new SuspendIteration exception would behave. Simple Iterators ---------------- The current functionality of iterators is best seen with a simple example which produces two values 'one' and 'two'. :: class States: def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_two return "one" def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration print list(States()) An equivalent iteration could, of course, be created by the following generator:: def States(): yield 'one' yield 'two' print list(States()) Introducing SuspendIteration ---------------------------- Suppose that between producing 'one' and 'two', the generator above could block on a socket read. In this case, we would want to raise SuspendIteration to signal that the iterator is not done producing, but is unable to provide a value at the current moment. :: from random import randint from time import sleep class SuspendIteration(Exception): pass class NonBlockingResource: """Randomly unable to produce the second value""" def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_suspend return "one" def state_suspend(self): rand = randint(1,10) if 2 == rand: self._next = self.state_two return self.state_two() raise SuspendIteration() def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration def sleeplist(iterator, timeout = .1): """ Do other things (e.g. sleep) while resource is unable to provide the next value """ it = iter(iterator) retval = [] while True: try: retval.append(it.next()) except SuspendIteration: sleep(timeout) continue except StopIteration: break return retval print sleeplist(NonBlockingResource()) In a real-world situation, the NonBlockingResource would be a file iterator, socket handle, or other I/O based producer. The sleeplist would instead be an async reactor, such as those found in asyncore or Twisted. The non-blocking resource could, of course, be written as a generator:: def NonBlockingResource(): yield "one" while True: rand = randint(1,10) if 2 == rand: break raise SuspendIteration() yield "two" It is not necessary to add a keyword, 'suspend', since most real content generators will not be in application code, they will be in low-level I/O based operations. Since most programmers need not be exposed to the SuspendIteration() mechanism, a keyword is not needed. Application Iterators --------------------- The previous example is rather contrived, a more 'real-world' example would be a web page generator which yields HTML content, and pulls from a database. Note that this is an example of neither the 'producer' nor the 'consumer', but rather of a filter. :: def ListAlbums(cursor): cursor.execute("SELECT title, artist FROM album") yield '<html><body><table><tr><td>Title</td><td>Artist</td></tr>' for (title, artist) in cursor: yield '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) yield '</table></body></html>' The problem, of course, is that the database may block for some time before any rows are returned, and that during execution, rows may be returned in blocks of 10 or 100 at a time. Ideally, if the database blocks for the next set of rows, another user connection could be serviced. Note the complete absence of SuspendIterator in the above code. If done correctly, application developers would be able to focus on functionality rather than concurrency issues. The iterator created by the above generator should do the magic necessary to maintain state, yet pass the exception through to a lower-level async framework. Here is an example of what the corresponding iterator would look like if coded up as a class:: class ListAlbums: def __init__(self, cursor): self.cursor = cursor def __iter__(self): self.cursor.execute("SELECT title, artist FROM album") self._iter = iter(self._cursor) self._next = self.state_head return self def next(self): return self._next() def state_head(self): self._next = self.state_cursor return "<html><body><table><tr><td>\ Title</td><td>Artist</td></tr>" def state_tail(self): self._next = self.state_stop return "</table></body></html>" def state_cursor(self): try: (title,artist) = self._iter.next() return '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) except StopIteration: self._next = self.state_tail return self.next() except SuspendIteration: # just pass-through raise def state_stop(self): raise StopIteration Complicating Factors -------------------- While the above example is straight-forward, things are a bit more complicated if the intermediate generator 'condenses' values, that is, it pulls in two or more values for each value it produces. For example, :: def pair(iterLeft,iterRight): rhs = iter(iterRight) lhs = iter(iterLeft) while True: yield (rhs.next(), lhs.next()) In this case, the corresponding iterator behavior has to be a bit more subtle to handle the case of either the right or left iterator raising SuspendIteration. It seems to be a matter of decomposing the generator to recognize intermediate states where a SuspendIterator exception from the producing context could happen. :: class pair: def __init__(self, iterLeft, iterRight): self.iterLeft = iterLeft self.iterRight = iterRight def __iter__(self): self.rhs = iter(iterRight) self.lhs = iter(iterLeft) self._temp_rhs = None self._temp_lhs = None self._next = self.state_rhs return self def next(self): return self._next() def state_rhs(self): self._temp_rhs = self.rhs.next() self._next = self.state_lhs return self.next() def state_lhs(self): self._temp_lhs = self.lhs.next() self._next = self.state_pair return self.next() def state_pair(self): self._next = self.state_rhs return (self._temp_rhs, self._temp_lhs) This proposal assumes that a corresponding iterator written using this class-based method is possible for existing generators. The challenge seems to be the identification of distinct states within the generator where suspension could occur. Resource Cleanup ---------------- The current generator mechanism has a strange interaction with exceptions where a 'yield' statement is not allowed within a try/finally block. The SuspendIterator exception provides another similar issue. The impacts of this issue are not clear. However it may be that re-writing the generator into a state machine, as the previous section did, could resolve this issue allowing for the situation to be no-worse than, and perhaps even removing the yield/finally situation. More investigation is needed in this area. API and Limitations ------------------- This proposal only covers 'suspending' a chain of iterators, and does not cover (of course) suspending general functions, methods, or "C" extension function. While there could be no direct support for creating generators in "C" code, native "C" iterators which comply with the SuspendIterator semantics are certainly possible. Low-Level Implementation ======================== The author of the PEP is not yet familiar with the Python execution model to comment in this area. References ========== .. [1] Twisted (http://twistedmatrix.com) .. [2] Peak (http://peak.telecommunity.com) .. [3] C10K (http://www.kegel.com/c10k.html) .. [4] Coroutines (http://c2.com/cgi/wiki?CallWithCurrentContinuation) .. [5] PEP 234, Iterators (http://www.python.org/peps/pep-0234.html) .. [6] Stackless Python (http://stackless.com) .. [7] Parrot /w coroutines (http://www.sidhe.org/~dan/blog/archives/000178.html) .. [8] PEP 255, Simple Generators (http://www.python.org/peps/pep-0255.html) .. [9] itertools - Functions creating iterators (http://docs.python.org/lib/module-itertools.html) .. [10] Microthreads in Python, David Mertz (http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ *

15 years, 6 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev September 2004