Mailman 3 September 2004 - Python-Dev

PEP 334 - Simple Coroutines via SuspendIteration
by Clark C. Evans 30 Sep '04

30 Sep '04

I've packaged up the idea of a coroutine facility using iterators and an exception, SuspendIteration. This would require some rather deep changes to how generators are implemented, however, it seems backwards compatible, implementable /w JVM or CLR, and would make most of my database/web development work far more pleasant. http://www.python.org/peps/pep-0334.html Cheers! Clark ... PEP: 334 Title: Simple Coroutines via SuspendIteration Version: $Revision: 1.1 $ Last-Modified: $Date: 2004/09/08 00:11:18 $ Author: Clark C. Evans <info(a)clarkevans.com> Status: Draft Type: Standards Track Python-Version: 3.0 Content-Type: text/x-rst Created: 26-Aug-2004 Post-History: Abstract ======== Asynchronous application frameworks such as Twisted [1]_ and Peak [2]_, are based on a cooperative multitasking via event queues or deferred execution. While this approach to application development does not involve threads and thus avoids a whole class of problems [3]_, it creates a different sort of programming challenge. When an I/O operation would block, a user request must suspend so that other requests can proceed. The concept of a coroutine [4]_ promises to help the application developer grapple with this state management difficulty. This PEP proposes a limited approach to coroutines based on an extension to the iterator protocol [5]_. Currently, an iterator may raise a StopIteration exception to indicate that it is done producing values. This proposal adds another exception to this protocol, SuspendIteration, which indicates that the given iterator may have more values to produce, but is unable to do so at this time. Rationale ========= There are two current approaches to bringing co-routines to Python. Christian Tismer's Stackless [6]_ involves a ground-up restructuring of Python's execution model by hacking the 'C' stack. While this approach works, its operation is hard to describe and keep portable. A related approach is to compile Python code to Parrot [7]_, a register-based virtual machine, which has coroutines. Unfortunately, neither of these solutions is portable with IronPython (CLR) or Jython (JavaVM). It is thought that a more limited approach, based on iterators, could provide a coroutine facility to application programmers and still be portable across runtimes. * Iterators keep their state in local variables that are not on the "C" stack. Iterators can be viewed as classes, with state stored in member variables that are persistent across calls to its next() method. * While an uncaught exception may terminate a function's execution, an uncaught exception need not invalidate an iterator. The proposed exception, SuspendIteration, uses this feature. In other words, just because one call to next() results in an exception does not necessarily need to imply that the iterator itself is no longer capable of producing values. There are four places where this new exception impacts: * The simple generator [8]_ mechanism could be extended to safely 'catch' this SuspendIteration exception, stuff away its current state, and pass the exception on to the caller. * Various iterator filters [9]_ in the standard library, such as itertools.izip should be made aware of this exception so that it can transparently propagate SuspendIteration. * Iterators generated from I/O operations, such as a file or socket reader, could be modified to have a non-blocking variety. This option would raise a subclass of SuspendIteration if the requested operation would block. * The asyncore library could be updated to provide a basic 'runner' that pulls from an iterator; if the SuspendIteration exception is caught, then it moves on to the next iterator in its runlist [10]_. External frameworks like Twisted would provide alternative implementations, perhaps based on FreeBSD's kqueue or Linux's epoll. While these may seem dramatic changes, it is a very small amount of work compared with the utility provided by continuations. Semantics ========= This section will explain, at a high level, how the introduction of this new SuspendIteration exception would behave. Simple Iterators ---------------- The current functionality of iterators is best seen with a simple example which produces two values 'one' and 'two'. :: class States: def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_two return "one" def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration print list(States()) An equivalent iteration could, of course, be created by the following generator:: def States(): yield 'one' yield 'two' print list(States()) Introducing SuspendIteration ---------------------------- Suppose that between producing 'one' and 'two', the generator above could block on a socket read. In this case, we would want to raise SuspendIteration to signal that the iterator is not done producing, but is unable to provide a value at the current moment. :: from random import randint from time import sleep class SuspendIteration(Exception): pass class NonBlockingResource: """Randomly unable to produce the second value""" def __iter__(self): self._next = self.state_one return self def next(self): return self._next() def state_one(self): self._next = self.state_suspend return "one" def state_suspend(self): rand = randint(1,10) if 2 == rand: self._next = self.state_two return self.state_two() raise SuspendIteration() def state_two(self): self._next = self.state_stop return "two" def state_stop(self): raise StopIteration def sleeplist(iterator, timeout = .1): """ Do other things (e.g. sleep) while resource is unable to provide the next value """ it = iter(iterator) retval = [] while True: try: retval.append(it.next()) except SuspendIteration: sleep(timeout) continue except StopIteration: break return retval print sleeplist(NonBlockingResource()) In a real-world situation, the NonBlockingResource would be a file iterator, socket handle, or other I/O based producer. The sleeplist would instead be an async reactor, such as those found in asyncore or Twisted. The non-blocking resource could, of course, be written as a generator:: def NonBlockingResource(): yield "one" while True: rand = randint(1,10) if 2 == rand: break raise SuspendIteration() yield "two" It is not necessary to add a keyword, 'suspend', since most real content generators will not be in application code, they will be in low-level I/O based operations. Since most programmers need not be exposed to the SuspendIteration() mechanism, a keyword is not needed. Application Iterators --------------------- The previous example is rather contrived, a more 'real-world' example would be a web page generator which yields HTML content, and pulls from a database. Note that this is an example of neither the 'producer' nor the 'consumer', but rather of a filter. :: def ListAlbums(cursor): cursor.execute("SELECT title, artist FROM album") yield '<html><body><table><tr><td>Title</td><td>Artist</td></tr>' for (title, artist) in cursor: yield '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) yield '</table></body></html>' The problem, of course, is that the database may block for some time before any rows are returned, and that during execution, rows may be returned in blocks of 10 or 100 at a time. Ideally, if the database blocks for the next set of rows, another user connection could be serviced. Note the complete absence of SuspendIterator in the above code. If done correctly, application developers would be able to focus on functionality rather than concurrency issues. The iterator created by the above generator should do the magic necessary to maintain state, yet pass the exception through to a lower-level async framework. Here is an example of what the corresponding iterator would look like if coded up as a class:: class ListAlbums: def __init__(self, cursor): self.cursor = cursor def __iter__(self): self.cursor.execute("SELECT title, artist FROM album") self._iter = iter(self._cursor) self._next = self.state_head return self def next(self): return self._next() def state_head(self): self._next = self.state_cursor return "<html><body><table><tr><td>\ Title</td><td>Artist</td></tr>" def state_tail(self): self._next = self.state_stop return "</table></body></html>" def state_cursor(self): try: (title,artist) = self._iter.next() return '<tr><td>%s</td><td>%s</td></tr>' % (title, artist) except StopIteration: self._next = self.state_tail return self.next() except SuspendIteration: # just pass-through raise def state_stop(self): raise StopIteration Complicating Factors -------------------- While the above example is straight-forward, things are a bit more complicated if the intermediate generator 'condenses' values, that is, it pulls in two or more values for each value it produces. For example, :: def pair(iterLeft,iterRight): rhs = iter(iterRight) lhs = iter(iterLeft) while True: yield (rhs.next(), lhs.next()) In this case, the corresponding iterator behavior has to be a bit more subtle to handle the case of either the right or left iterator raising SuspendIteration. It seems to be a matter of decomposing the generator to recognize intermediate states where a SuspendIterator exception from the producing context could happen. :: class pair: def __init__(self, iterLeft, iterRight): self.iterLeft = iterLeft self.iterRight = iterRight def __iter__(self): self.rhs = iter(iterRight) self.lhs = iter(iterLeft) self._temp_rhs = None self._temp_lhs = None self._next = self.state_rhs return self def next(self): return self._next() def state_rhs(self): self._temp_rhs = self.rhs.next() self._next = self.state_lhs return self.next() def state_lhs(self): self._temp_lhs = self.lhs.next() self._next = self.state_pair return self.next() def state_pair(self): self._next = self.state_rhs return (self._temp_rhs, self._temp_lhs) This proposal assumes that a corresponding iterator written using this class-based method is possible for existing generators. The challenge seems to be the identification of distinct states within the generator where suspension could occur. Resource Cleanup ---------------- The current generator mechanism has a strange interaction with exceptions where a 'yield' statement is not allowed within a try/finally block. The SuspendIterator exception provides another similar issue. The impacts of this issue are not clear. However it may be that re-writing the generator into a state machine, as the previous section did, could resolve this issue allowing for the situation to be no-worse than, and perhaps even removing the yield/finally situation. More investigation is needed in this area. API and Limitations ------------------- This proposal only covers 'suspending' a chain of iterators, and does not cover (of course) suspending general functions, methods, or "C" extension function. While there could be no direct support for creating generators in "C" code, native "C" iterators which comply with the SuspendIterator semantics are certainly possible. Low-Level Implementation ======================== The author of the PEP is not yet familiar with the Python execution model to comment in this area. References ========== .. [1] Twisted (http://twistedmatrix.com) .. [2] Peak (http://peak.telecommunity.com) .. [3] C10K (http://www.kegel.com/c10k.html) .. [4] Coroutines (http://c2.com/cgi/wiki?CallWithCurrentContinuation) .. [5] PEP 234, Iterators (http://www.python.org/peps/pep-0234.html) .. [6] Stackless Python (http://stackless.com) .. [7] Parrot /w coroutines (http://www.sidhe.org/~dan/blog/archives/000178.html) .. [8] PEP 255, Simple Generators (http://www.python.org/peps/pep-0255.html) .. [9] itertools - Functions creating iterators (http://docs.python.org/lib/module-itertools.html) .. [10] Microthreads in Python, David Mertz (http://www-106.ibm.com/developerworks/linux/library/l-pythrd.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- Clark C. Evans Prometheus Research, LLC. http://www.prometheusresearch.com/ o office: +1.203.777.2550 ~/ , mobile: +1.203.444.0557 // (( Prometheus Research: Transforming Data Into Knowledge \\ , \/ - Research Exchange Database /\ - Survey & Assessment Technologies ` \ - Software Tools for Researchers ~ *

9 18

Finding the module from PyTypeObject?
by Nick Bastin 30 Sep '04

30 Sep '04

Is there any way to (reliably) find the module that defined the class represented by a given PyTypeObject in C? -- Nick

3 6

Odd compile errors for bad genexps
by Tim Peters 29 Sep '04

29 Sep '04

>>> (i for i in x) = 2 SystemError: assign to generator expression not possible 1. Why is that a SystemError instead of a SyntaxError? SystemError doesn't make sense. 2. Why didn't it echo the offending line? >>> (i for i in x) += 2 SyntaxError: augmented assign to tuple literal not possible 3. That's not a tuple literal. 4. See #2 <wink>.

2 3

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser 29 Sep '04

29 Sep '04

Patch / Bug Summary ___________________ Patches : 235 open ( +0) / 2637 closed ( +4) / 2872 total ( +4) Bugs : 768 open ( +1) / 4480 closed (+17) / 5248 total (+18) RFE : 152 open ( +1) / 131 closed ( +0) / 283 total ( +1) New / Reopened Patches ______________________ unittest.py patch: add skipped test functionality (2004-09-24) http://python.org/sf/1034053 opened by Remy Blank Remove CoreServices / CoreFoundation dependencies in core (2004-09-26) http://python.org/sf/1035255 opened by Bob Ippolito -m option to run a module as a script (2004-09-28) http://python.org/sf/1035498 opened by Nick Coghlan Add New RPM-friendly record option to setup.py (2004-09-28) http://python.org/sf/1035576 opened by Jeff Pitman Patches Closed ______________ Add API to logging package to allow intercooperation. (2004-09-21) http://python.org/sf/1032206 closed by vsajip SystemError generated by struct.pack('P', 'notanumber') (2004-08-18) http://python.org/sf/1011240 closed by arigo (bug 952953) execve empty 2nd arg fix (2004-08-14) http://python.org/sf/1009075 closed by arigo atexit decorator (2004-09-21) http://python.org/sf/1031687 closed by rhettinger New / Reopened Bugs ___________________ idle -n crashes (2004-09-22) CLOSED http://python.org/sf/1032395 opened by Matthias Klose Odd behavior with unicode.translate on OSX. (2004-09-22) http://python.org/sf/1032615 opened by Jeremy Fincher ftplib has incomplete transfer when sending files in Windows (2004-09-22) http://python.org/sf/1032875 opened by Ed Sanville Confusing description of strict option for email.Parser (2004-09-23) http://python.org/sf/1032960 opened by Andrew Bennetts Misleading error message in random.choice (2004-09-22) CLOSED http://python.org/sf/1033038 opened by Nefarious CodeMonkey, Jr. build doesn't pick up bsddb w/Mandrake 9.2 (2004-09-23) http://python.org/sf/1033390 opened by Alex Martelli buffer() object broken. (2004-09-23) CLOSED http://python.org/sf/1033720 opened by James Y Knight Can't inherit slots from new-style classes implemented in C (2004-09-24) http://python.org/sf/1034178 opened by Phil Thompson More buffer object brokenness (2004-09-24) CLOSED http://python.org/sf/1034242 opened by James Y Knight Why does Python link to Foundation? (2004-09-24) http://python.org/sf/1034277 opened by Bob Ippolito Configure uses GNU ld flags with non-GNU compilers/linkers (2004-09-25) CLOSED http://python.org/sf/1034496 opened by Drew Schatt hex() and oct() documentation is incorrect (2004-09-27) http://python.org/sf/1035279 opened by Nick Coghlan distutils.util.get_platform() should include sys.version[:2] (2004-09-27) http://python.org/sf/1035703 opened by Bob Ippolito Tix.Grid widgets not implemented yet (2004-09-28) http://python.org/sf/1036406 opened by Christos Georgiou unicode strings cannot be dictionary keys (2004-09-28) http://python.org/sf/1036490 opened by Morten Kjeldgaard Email module's feed parser (2004-09-28) CLOSED http://python.org/sf/1036506 opened by Matthew Cowles file.next() info hidden (2004-09-28) http://python.org/sf/1036626 opened by Nick Jacobson printf() in dynload_shlib.c should be PySys_WriteStderr (2004-09-28) http://python.org/sf/1036752 opened by Jp Calderone Bugs Closed ___________ rfc822 __iter__ problem (2004-09-17) http://python.org/sf/1030125 closed by rhettinger Fold tuples of constants into a single constant (2004-09-20) http://python.org/sf/1031667 closed by rhettinger Misleading error message in random.choice (2004-09-22) http://python.org/sf/1033038 closed by rhettinger PEP 302 loader not carried through by reload function (2004-09-16) http://python.org/sf/1029475 closed by pje Float/long comparison anomaly (2002-02-06) http://python.org/sf/513866 closed by tim_one buffer() object broken. (2004-09-23) http://python.org/sf/1033720 closed by nascheme ConfigParser's get method gives utf-8 for a utf-16 config... (2004-01-10) http://python.org/sf/874354 closed by goodger More buffer object brokenness (2004-09-24) http://python.org/sf/1034242 closed by nascheme embedding in multi-threaded & multi sub-interpreter environ (2004-03-22) http://python.org/sf/921077 closed by bcannon Configure uses GNU ld flags with non-GNU compilers/linkers (2004-09-25) http://python.org/sf/1034496 closed by loewis 2.4 asyncore breaks Zope (2004-08-18) http://python.org/sf/1011606 closed by tim_one CPU usage shoots up with asyncore (2004-08-16) http://python.org/sf/1010098 closed by arigo execve rejects empty argument list (2004-05-13) http://python.org/sf/952953 closed by arigo email.Message.Message.__getitem__ doc string wrong (2004-06-25) http://python.org/sf/979924 closed by bwarsaw Email module's feed parser (2004-09-28) http://python.org/sf/1036506 closed by bwarsaw idle -n crashes (2004-09-22) http://python.org/sf/1032395 closed by kbk IDLE hangs when inactive more than 2 hours (2004-08-02) http://python.org/sf/1001869 closed by kbk

1 0

Fwd: Python binding to Rendezvous
by Guido van Rossum 28 Sep '04

28 Sep '04

I've been asked to help getting Rendevous (AKA zeroconf I believe) bindings for Python implemented. Anyone interested in helping out? (The person to contact if you're interested is Daniel Steinberg. Please cc me on the initial emails so I know contact is being made. --Guido ---------- Forwarded message ---------- From: Joey Trevino Date: Mon, 27 Sep 2004 17:09:48 -0700 Subject: Python binding to Rendezvous To: Guido van Rossum Cc: Daniel H Steinberg Hey Guido, Here's the description of the task from my friend Daniel Steinberg (CC'd here) at O'Reilly. I understand that you don't have time to do the work yourself, but I'm hopeful that you know of someone who would. >>>> Apple has just provided a daemon for Rendezvous for Mac, for >>>> Windows, for Linux, and for UNIX. Rich Kilmer spent an afternoon >>>> with Stuart Cheshire and wrote Ruby bindings for this daemon and so >>>> Ruby developers can easily Rendezvous enable their application. >>>> There are also Parrot and Perl bindings on the way. The question >>>> was whether someone had an afternoon free to spend with Stuart to >>>> write the Python bindings. Stuart is not a Python expert but he is >>>> good at teaming up with someone who is to help them see what hooks >>>> are needed. Thanks, Joey -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 2

open('/dev/null').read() -> MemoryError
by Armin Rigo 28 Sep '04

28 Sep '04

Hi, On my system, which is admittedly an old Linux box (2.2 kernel), one test fails: >>> file('/dev/null').read() Traceback (most recent call last): File "<stdin>", line 1, in ? MemoryError This is because: >>> os.stat('/dev/null').st_size 4540321280L This looks very broken indeed. I have no idea where this number comes from. I'd also complain if I was asked to allocate a buffer large enough to hold that many bytes. If we cared, we could "enhance" the file.read() method to account for the possibility that maybe stat() lied; maybe it is desirable, instead of allocating huge amounts of memory, to revert to something like the following above some large threshold: result = [] while 1: buf = f.read(16384) if not buf: return ''.join(result) result.append(buf) Of course for genuinely large reads it's a disaster to have to allocate twice as much memory. Anyway I'm not sure we care about going around broken behaviour. I'm just wondering if os.stat() could lie in other situations too. Armin

2 2

using openssh's pty code
by J Raynor 28 Sep '04

28 Sep '04

Since openssh must handle pty allocation, its support for pty operations across various platforms is more robust than python's. I'd like to use openssh's code to improve on python's pty handling. I know the licenses for openssh and python are different. Can anyone tell me if it's legal to mix openssh code into python? Assuming it is, are the python maintainers willing to accept a python patch that contains some openssh code?

6 8

[Fwd: Re: [Python-Dev] using openssh's pty code]
by J Raynor 28 Sep '04

28 Sep '04

I forgot to CC the list with my response.

2 3

upcoming stable release?
by Jason Lunz 27 Sep '04

27 Sep '04

I'm not a regular around here, so forgive me if this is obvious: Can anyone give me an idea of when to expect the next 2.3 point release? 2.3.4 dates from May, and I have a vague idea that I need some more-recent fix from the release23-maint branch. [background: I have a python-gtk app that may be affected by this bug: http://bugzilla.gnome.org/show_bug.cgi?id=149845. The changelog for the debian unstable python2.3 package says: python2.3 (2.3.4-5) unstable; urgency=medium * Updated to CVS release23-maint 20040705. - Remove threading patch, integrated upstream. I have a vague idea that this may address the same issue, and if so, the 2.3.5 release (being based on release23-maint, I assume) will be safe to use on all platforms, not just Debian sid. If I were on debian unstable everything would be fine. Unfortunately, I wish to also support windows, and on that platform I prefer to use official python.org releases, which makes me wonder whether 2.3.5 is imminent.] thanks, Jason

2 1

Socket/Asyncore bug needs attention
by Raymond Hettinger 27 Sep '04

27 Sep '04

Anyone who has worked on sockets or asyncore should take a look at SF bug #1010098: CPU usage shoots up with asyncore. Since Py2.3, the behavior changed for the worse. The bug report has been around for about five weeks and doesn't look like it is actively being solved. If you worked on those modules, please review your check-ins to see if they were the cause: www.python.org/sf/1010098 Thx, Raymond

4 3