Replacing threads with swapcontext()

Hello. I have went and replaced Python threads with explicitly-scheduled coroutines using swapcontext() family of functions (basically out of frustration with the greenlet). I have two questions. 1. Is it as dangerous to switch C stacks and not switch PyThreadState as I suspect? In particular, how tightly is PyThreadState::frame coupled to current C stack? 2. How do you all look at merging my work as another threading-related configure switch? It's only another thread_something.h file and a small patch to threadmodule.c. Project is at http://code.google.com/p/coev/ -- ./lxnt

Alexander Sabourenkov wrote:
Hello.
I have went and replaced Python threads with explicitly-scheduled coroutines using swapcontext() family of functions (basically out of frustration with the greenlet).
I have two questions.
1. Is it as dangerous to switch C stacks and not switch PyThreadState as I suspect? In particular, how tightly is PyThreadState::frame coupled to current C stack?
Off the top of my head, recursion limit checks will definitely go haywire, and I would expect threading.local() and exception handling to explode at some point (although the latter depends on what you mean by "switching C stacks" - if you really are just fiddling the stack pointer without affecting the thread local heap storage mechanisms, the latter two should be fine. Still doesn't sound like a good idea though). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Nick Coghlan wrote:
Off the top of my head, recursion limit checks will definitely go haywire, and I would expect threading.local() and exception handling to explode at some point (although the latter depends on what you mean by "switching C stacks" - if you really are just fiddling the stack pointer without affecting the thread local heap storage mechanisms, the latter two should be fine. Still doesn't sound like a good idea though).
In my implementation, each context looks like a thread to Python, with its own PyThreadState, thread-local storage and locks are taken care of, so I'm reasonably sure there's nothing scary lurking unseen. I'm trying to confirm my understanding as to how unreliable the greenlet is. The greenlet module basically just switches C stacks and does not touch anything else. There evidently are problems with this when there's an exception raised, as it's stored in PyThreadState::curexc_*/exc_*. PyThreadState's profiling/tracing -related members, the tick_counter, and per-thread state dict don't appear as a problem at the first glance. recursion_depth will be messed up unpredictably, as you kindly pointed out. This leaves PyThreadState::frame a PyFrameObject to be examined. Do they form a python-side stack of a thread in question? -- ./lxnt
participants (2)
-
Alexander Sabourenkov
-
Nick Coghlan