Mailman 3 October 2005 - Python-Dev

PEP 342 suggestion: start(), __call__() and unwind_call() methods
by Nick Coghlan Oct. 10, 2005

Oct. 10, 2005

I'm lifting Jason's PEP 342 suggestions out of the recent PEP 343 thread, in case some of the folks interested in coroutines stopped following that discussion. Jason suggested two convenience methods, .start() and .finish(). start() simply asserted that the generator hadn't been started yet, and I find the parallel with "Thread.start()" appealing: def start(self): """ Convenience method -- exactly like next(), but assert that this coroutine hasn't already … [View More]been started. """ if self.__started: raise RuntimeError("Coroutine already started") return self.next() I've embellished Jason's suggested finish() method quite a bit though. 1. Use send() rather than next() 2. Call it __call__() rather than finish() 3. Add an unwind_call() variant that gives similar semantics for throw() 4. Support getting a return value from the coroutine using the syntax "raise StopIteration(val)" 5. Add an exception "ContinueIteration" that is used to indicate the generator hasn't finished yet, rather than expecting the generator to finish and raising RuntimeError if it doesn't It ends up looking like this: def __call__(self, value=None): """ Call a generator as a coroutine Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration with the value yielded as the argument if the generator yields a value """ if not self.__started: raise RuntimeError("Coroutine not started") try: if exc: yield_val = self.throw(value, *exc) else: yield_val = self.send(value) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val) def unwind_call(self, *exc): """Raise an exception in a generator used as a coroutine. Returns the first argument supplied to StopIteration or None if no argument was supplied. Raises ContinueIteration if the generator yields a value with the value yield as the argument """ try: yield_val = self.throw(*exc) except (StopIteration), ex: if ex.args: return args[0] else: raise ContinueIteration(yield_val) Now here's the trampoline scheduler from PEP 342 using this idea: import collections class Trampoline: """Manage communications between coroutines""" running = False def __init__(self): self.queue = collections.deque() def add(self, coroutine): """Request that a coroutine be executed""" self.schedule(coroutine) def run(self): result = None self.running = True try: while self.running and self.queue: func = self.queue.popleft() result = func() return result finally: self.running = False def stop(self): self.running = False def schedule(self, coroutine, stack=(), call_result=None, *exc): # Define the new pseudothread def pseudothread(): try: if exc: result = coroutine.unwind_call(call_result, *exc) else: result = coroutine(call_result) except (ContinueIteration), ex: # Called another coroutine callee = ex.args[0] self.schedule(callee, (coroutine,stack)) except: if stack: # send the error back to the caller caller = stack[0] prev_stack = stack[1] self.schedule( caller, prev_stack, *sys.exc_info() ) else: # Nothing left in this pseudothread to # handle it, let it propagate to the # run loop raise else: if stack: # Finished, so pop the stack and send the # result to the caller caller = stack[0] prev_stack = stack[1] self.schedule(caller, prev_stack, result) # Add the new pseudothread to the execution queue self.queue.append(pseudothread) Notice how a non-coroutine callable can be yielded, and it will still work happily with the scheduler, because the desire to continue execution is indicated by the ContinueIteration exception, rather than by the type of the returned value. With this relatively dumb scheduler, that doesn't provide any particular benefit - the specific pseudothread doesn't block, but eventually the scheduler itself blocks when it executes the non-coroutine call. However, it wouldn't take too much to make the scheduler smarter and give it a physical thread pool that it used whenever it encountered a non-coroutine call And that's the real trick here: with these additions to PEP 342, the decision of how to deal with blocking calls could be made in the scheduler, without affecting the individual coroutines. All the coroutine writers would need to remember to do is to write any potentially blocking operations as yielded lambda expressions or functional.partial invocations, rather than as direct function calls. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com [View Less]

8 17

RFC: readproperty
by Jim Fulton Oct. 10, 2005

Oct. 10, 2005

Read descriptors (defining only __get__) and data descripttors have (defining __get__ and __set__) different semantics. In particular, read descriptors are overridden by instance data, while data descriptors are not. A common use of read descriptors is for lazily computed data: class readproperty(object): "Create a read descriptor from a function" def __init__(self, func): self.func = func def __get__(self, inst, class_): if inst is None: … [View More]

10 19

Re: [Python-Dev] Removing the block stack (was Re: PEP 343 and __with__)
by Phillip J. Eby Oct. 10, 2005

Oct. 10, 2005

At 10:09 PM 10/5/2005 -0700, Neal Norwitz wrote: >I've also been thinking about avoiding tuple creation when calling >python functions. The change I have in mind would probably have to >wait until p3k, but could yield some speed ups. > >Warning: half baked idea follows. Yeah, I've been baking that idea for a long time, and it's a bit more complex than you've suggested, due to generators, sys._getframe(), and tracebacks. >My thoughts are to dynamically allocate the … [View More]Python stack memory (e.g., >void *stack = malloc(128MB)). Then all calls within each thread uses >its own stack. So things would be pushed onto the stack like they are >currently, but we wouldn't need to do create a tuple to pass to a >method, they could just be used directly. Basically more closely >simulate the way it currently works in hardware. Actually, Python/ceval.c already skips creating a tuple when calling Python functions with a fixed number of arguments (caller and callee) and no cell vars (i.e., not a closure). It copies them straight from the calling frame stack to the callee frame's stack. >This would mean all the PyArg_ParseTuple()s would have to change. It >may be possible to fake it out, but I'm not sure it's worth it which >is why it would be easier to do this for p3k. Actually, I've been thinking that replacing the arg tuple with a PyObject* array would allow us to skip tuple creation when calling C functions, since you could just give the C functions a pointer to the arguments on the caller's stack. That would let us get rid of most remaining tuple allocations. I suppose we'd also need either an argcount parameter. The old APIs taking tuples for calls could trivially convert the tuples to a array pointer and size, then call the new APIs. Actually, we'd probably have to have a tp_arraycall slot or something, with the existing tp_call forwarding to tp_arraycall in most cases, but occasionally the reverse. The tricky part is making sure you don't end up with cases where you call a tuple API that converts to an array that then turns it back into a tuple! >The general idea is to allocate the stack in one big hunk and just >walk up/down it as functions are called/returned. This only means >incrementing or decrementing pointers. This should allow us to avoid >a bunch of copying and tuple creation/destruction. Frames would >hopefully be the same size which would help. Note that even though >there is a free list for frames, there could still be >PyObject_GC_Resize()s often (or unused memory). WIth my idea, >hopefully there would be better memory locality, which could speed >things up. Yeah, unfortunately for your idea, generators would have to copy off bits of the stack and then copy them back in, making generators slower. If it weren't for that part, the idea would probably be a good one, as arguments, locals, cells, and the block and value stacks could all be handled that way, with the compiler treating all operations as base-pointer offsets, thereby eliminating lots of more-complex pointer management in ceval.c and frameobject.c. Another possible fix for generators would be of course to give them their own stack arena, but then you have the problem of needing to copy overflows from one such stack to another - at which point you're basically back to having frames. On the other hand, maybe the good part of this idea is just eliminating all the pointer fudging and having the compiler determine stack offsets. Then, the frame object layout would just consist of a big hunk of stack space, laid out as a PyObject* array. The main problem with this concept is that it would change the meaning of certain opcodes, since right now the offsets of free variables in opcodes start over the numbering, but this approach would add the number of locals to those offsets. [View Less]

2 1

Re: [Python-Dev] PEP 343 and __with__
by Phillip J. Eby Oct. 9, 2005

Oct. 9, 2005

At 12:37 PM 10/3/2005 -0400, Jason Orendorff wrote: >I'm -1 on PEP 343. It seems ...complex. And even with all the >complexity, I *still* won't be able to type > > with self.lock: ... > >which I submit is perfectly reasonable, clean, and clear. Which is why it's proposed to add __enter__/__exit__ to locks, and somewhat more controversially, file objects. (Guido objected on the basis that people might reuse the file object, but reusing a closed file object results in … [View More]a sensible error message and so doesn't seem like a problem to me.) >[snip] >__with__() always returns a new context manager object. Just as with >iterators, a context manager object has "cm.__with__() is cm". > >The 'with' statement would call __with__(), of course. You didn't offer any reasons why this would be useful and/or good. >2. More radical: Let's get rid of __enter__() and __exit__(). The >only example in PEP 343 that uses them is Example 4, which exists only >to show that "there's more than one way to do it". It all seems fishy >to me. Why not get rid of them and use only __with__()? In this >scenario, Python would expect __with__() to return a coroutine (not to >say "iterator") that yields exactly once. Because this multiplies the difficulty of implementing context managers in C. It's easy to define a pair of C methods for __enter__ and __exit__, but an iterator requires creating another class in C. The yield-based syntax is just syntax sugar, not the essence of the proposal. >The only drawback I see is that context manager methods implemented in >C will work differently from those implemented in Python. Since C >doesn't have coroutines, I imagine there would have to be enter() and >exit() slots. Maybe this is a major design concern; I don't know. Considering your argument that locks should be contextmanagers, it would seem like a good idea for C implementations to be easy. :) >My apologies if this is redundant or unwelcome at this date. Since the PEP is accepted and has patches for both its implementation and a good part of its documentation, a major change like this would certainly need a better rationale. If your idea was that __with__ would somehow make it easier for locks to be context managers, it's based on a flawed premise. All that's required now is to have __enter__ and __exit__ call acquire() and release(). At this point, it's simply an open issue as to which stdlib objects will be context managers, and which will have helper functions or classes to serve as context managers. The actual API used to implement them has little or no bearing on that issue. [View Less]

13 32

Re: [Python-Dev] Proposed changes to PEP 343
by Anders J. Munch Oct. 9, 2005

Oct. 9, 2005

Nick Coghlan did a +1 job to write: > 1. Amend the statement specification such that: > > with EXPR as VAR: > BLOCK > > is translated as: > > abc = (EXPR).__with__() > exc = (None, None, None) > VAR = abc.__enter__() > try: > try: > BLOCK > except: > exc = sys.exc_info() > raise > finally: > abc.__exit__(*exc) … [View More]

4 3

Re: [Python-Dev] Sandboxed Threads in Python
by Phillip J. Eby Oct. 8, 2005

Oct. 8, 2005

At 06:12 PM 10/7/2005 -0600, Adam Olsen wrote: >Okay, basic principal first. You start with a sandboxed thread that >has access to nothing. No modules, no builtins, *nothing*. This >means it can run without the GIL but it can't do any work. It sure can't. You need at least the threadstate and a builtins dictionary to do any work. > To make it >do something useful we need to give it two things: first, immutable >types that can be safely accessed without locks, This … [View More]is harder than it sounds. Integers, for example, have a custom allocator and a free list, not to mention a small-integer cache. You would somehow need to duplicate all that for each sandbox, or else you have to make those integers immortal using your "magic constant". >Turns out it's quite easy and it doesn't harm performance of existing >code or require modification (but a recompile is necessary). The idea >is to only use a cyclic garbage collector for cleaning them up, Um, no, actually. You need a mark-and-sweep GC or something of that ilk. Python's GC only works with objects that *have refcounts*, and it works by clearing objects that are in cycles. The clearing causes DECREF-ing, which then causes objects to be freed. If you have objects without refcounts, they would be immortal and utterly unrecoverable. >which >means we need to disable the reference counting. That requires we >modify Py_INCREF and Py_DECREF to be a no-op if ob_refcnt is set to a >magic constant (probably a negative value). And any object with the magic refcount will live *forever*, unless you manually deallocate it. >That's all it takes. Modify Py_INCREF and Py_DECREFs to check for a >magic constant. Ahh, but the performance? See for yourself. First, you need to implement a garbage collection scheme that can deal with not having refcounts. Otherwise you're not comparing apples to apples here, and your programs will leak like crazy. Note that implementing a root-based GC for Python is non-trivial, since extension modules can store pointers to PyObjects anywhere they like. Further, many Python objects don't even support being tracked by the current cycle collector. So, changing this would probably require a lot of C extensions to be rewritten to support the needed API changes for the new garbage collection strategy. >So to sum up, by prohibiting mutable objects from being transferred >between sandboxes we can achieve scalability on multiple CPUs, making >threaded programming easier and more reliable, as a bonus get secure >sandboxes[1], and do that all while maintaining single-threaded >performance and requiring minimal changes to existing C modules >(recompiling). Unfortunately, you have only succeeded in restating the problem, not reducing its complexity. :) In fact, you may have increased the complexity, since now you need a threadsafe garbage collector, too. Oh, and don't forget - newstyle classes keep weak references to all their subclasses, which means for example that every time you subclass 'dict', you're modifying the "immutable" 'dict' class. So, unless you recreate all the classes in each sandbox, you're back to needing locking. And if you recreate everything in each sandbox, well, I think you've just reinvented "processes". :) [View Less]

5 9

test_cmd_line failure on Kubuntu 5.10 with GCC 4.0
by Nick Coghlan Oct. 8, 2005

Oct. 8, 2005

Anyone else seeing any problems with test_cmd_line? I've got a few failures in test_cmd_line on Kubuntu 5.10 with GCC 4.0 relating to a missing "\n" line ending. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

5 4

Re: [Python-Dev] Sandboxed Threads in Python
by Phillip J. Eby Oct. 8, 2005

Oct. 8, 2005

At 07:17 PM 10/7/2005 -0600, Adam Olsen wrote: >On 10/7/05, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > At 06:12 PM 10/7/2005 -0600, Adam Olsen wrote: > > >Turns out it's quite easy and it doesn't harm performance of existing > > >code or require modification (but a recompile is necessary). The idea > > >is to only use a cyclic garbage collector for cleaning them up, > > > > Um, no, actually. You need a mark-and-sweep GC or … [View More]something of that > > ilk. Python's GC only works with objects that *have refcounts*, and it > > works by clearing objects that are in cycles. The clearing causes > > DECREF-ing, which then causes objects to be freed. If you have objects > > without refcounts, they would be immortal and utterly unrecoverable. > >Perhaps I wasn't clear enough, I was assuming appropriate changes to >the GC would be done. The important thing is it can be done without >changing the interface that the existing modules use. No, it can't. See more below. > > >That's all it takes. Modify Py_INCREF and Py_DECREFs to check for a > > >magic constant. Ahh, but the performance? See for yourself. > > > > First, you need to implement a garbage collection scheme that can deal with > > not having refcounts. Otherwise you're not comparing apples to apples > > here, and your programs will leak like crazy. > > > > Note that implementing a root-based GC for Python is non-trivial, since > > extension modules can store pointers to PyObjects anywhere they > > like. Further, many Python objects don't even support being tracked by the > > current cycle collector. > > > > So, changing this would probably require a lot of C extensions to be > > rewritten to support the needed API changes for the new garbage collection > > strategy. > >They only need to be rewritten if you want them to provide an >immutable type that can be transferred between sandboxes. No. You're missing my point. If they are able to *reference* these objects, then the garbage collector has to know about it, or else it can't know when to reclaim them. Ergo, these objects will leak, or else extensions will crash when they refer to the deallocated memory. In other words, you can't handwave the whole problem away by assuming "a garbage collector". The garbage collector has to actually be able to work, and you haven't specified *how* it can work without changing the C API. >I was aware that weakrefs needed some special handling (I just forgot >to mention it), but I didn't know it was used by subclassing. >Unfortunately I don't know what purpose it serves so I can't >contemplate how to deal with it. It allows changes to a supertype's C-level slots to propagate to subclasses. [View Less]

2 1

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser Oct. 8, 2005

Oct. 8, 2005

Patch / Bug Summary ___________________ Patches : 341 open ( +4) / 2953 closed ( +6) / 3294 total (+10) Bugs : 884 open (-28) / 5321 closed (+43) / 6205 total (+15) RFE : 196 open ( +1) / 187 closed ( +0) / 383 total ( +1) New / Reopened Patches ______________________ Make fcntl work properly on AMD64 (2005-09-30) http://python.org/sf/1309352 opened by Brad Hards A wait4() implementation (2005-09-30) http://python.org/sf/1309579 opened by chads httplib … [View More]

1 0

Sourceforge CVS access
by Nick Coghlan Oct. 7, 2005

Oct. 7, 2005

Could one of the Sourceforge powers-that-be grant me check in access so I can update PEP 343 directly? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

3 2