functional continuations
i'm working on some minimalistic asynchronous framework for python, somewhat like twisted or stackless, but for different purposes. i came to the conclusion i want to be able to "freeze" functions, and resume them later, when some condition is matched. the idea i came up with is, using exceptions for functional continuations: after all, the exception's traceback holds the entire context... so with some help from black magic (i.e., ctypes), i can resurrect dead stackframes. i will have a reactor in the background which manages the execution flow, and blocking operations (I/O, etc.) will be wrapped with a thin wrapper, such as class socket2(socket): def recv(self, count = None): raise WaitFor(self, "r") return socket.recv(self, count) now, when read()ing from file2 objects, first an exception will propagate up to the reactor, which will catch it, create a continuation object (which holds the traceback) and register it with the desired event. then, when the event is triggered, the reactor will resurrect the continuation object to the point where it stopped (f_lasti + 2), and execution would continue normally from there. the idea itself is similar to generator objects, and requires some messing-around with f_back and f_lasti which is quite ugly. on the other hand, the beauty of it is, it doesn't require any special design patterns (unlike twisted) or replacing the interpreter (unlike stackless), so it can be transparently used with any third-party libs. all it takes is wrapping I/O objects to "raise WaitFor(...)" prior to doing blocking I/O. moreover, unlike generators, i gain the ability to "propagate" up to the reactor (like normal exceptions), so the code in between the I/O and the reactor needn't be aware of anything (only avoiding unconstrained try-except blocks). i wanted to get some feedback on the issue (i tried c.l.p, but they didn't understand me well enough): * has it been tried before? * do you suppose it will work? are there any drawbacks i didn't anticipate? * is it useful enough to be added to the core interpreter (like generators)? of course if it is added, it could be implemented with a dedicated mechanism, rather than abusing exceptions for that. -tomer
On Sat, Dec 15, 2007, tomer filiba wrote:
i wanted to get some feedback on the issue (i tried c.l.p, but they didn't understand me well enough):
python-ideas is the best place for topics like this. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Typing is cheap. Thinking is expensive." --Roy Smith
tomer filiba wrote:
the idea i came up with is, using exceptions for functional continuations: after all, the exception's traceback holds the entire context...
The problem with this is that, if the call chain has passed through a C-implemented function at some point, the traceback *doesn't* contain the entire context -- some of it is in the C stack frames that got unwound during the propagation of the exception. So you may be able to make this work where all the code involved is pure Python, but it can't work in general, unless you redesign the whole interpreter the way the original Stackless did. Having said that, it might still be a useful thing to have, as the pure-Python case is probably fairly common. But I'd want to know what the failure mode is like when the pure-Python case doesn't hold. Do you get a clear indication of what is wrong, or does it misbehave in some obscure way? A test case for this would be to call map() with a function that tries to suspend itself using this mechanism. What happens when you try to resume it? -- Greg
By the way, I wouldn't call this a "continuation", as that word implies a bit more (reusability). It's more like a coroutine or lightweight thread. -- Greg
yes, well, as i suspected and as PJE pointed out, extension or builtin
functions would
make life difficult, but these are difficulties i would be willing to
accept. however, my attempts
are futile anyways, as the evaluation stack is cleared before the exception
propagates up
to the frames above... which would explain the NULL deref exceptions i was
getting :)
-tomer
On Dec 16, 2007 12:54 AM, Greg Ewing
tomer filiba wrote:
the idea i came up with is, using exceptions for functional continuations: after all, the exception's traceback holds the entire context...
The problem with this is that, if the call chain has passed through a C-implemented function at some point, the traceback *doesn't* contain the entire context -- some of it is in the C stack frames that got unwound during the propagation of the exception.
So you may be able to make this work where all the code involved is pure Python, but it can't work in general, unless you redesign the whole interpreter the way the original Stackless did.
Having said that, it might still be a useful thing to have, as the pure-Python case is probably fairly common. But I'd want to know what the failure mode is like when the pure-Python case doesn't hold. Do you get a clear indication of what is wrong, or does it misbehave in some obscure way?
A test case for this would be to call map() with a function that tries to suspend itself using this mechanism. What happens when you try to resume it?
-- Greg
-- An NCO and a Gentleman
participants (3)
-
Aahz -
Greg Ewing -
tomer filiba