These CPU intensive threads are python/C extensions generated with
pyrex. Instead of messing with the GIL I will use processes as David
suggests. It would be nice to have a process pool implementation that
was as easy to use as deferToThread.
Evidently they use the perspective broker for inter process
communication in quotient.searchup, and that may be a good starting
place.
Thanks,
-Nate
On 27 Jul 2005 17:10:42 -0400, David Bolen
Jp Calderone
writes: POSH doesn't elimate the transport overhead. I've done some basic investigation, and it's incredibly slow. mmap() is probably a better solution in most cases, although I am not convinced multiple processes are called for in this case.
Me neither, but it would be a logical way to progress if you couldn't resolve things in the single process. Good to know about POSH though.
A solution which hasn't been suggested yet is to drop the native thread and use a cooperative Python thread. With this approach, you can choose to schedule it however you like, including /not/ scheduling it when you have other more important tasks to complete.
Well, but you'd still have the problem of ensuring that it was yielding back at a reasonable frequency wouldn't you? So it would be similar to sprinkling in a time.sleep(0) in a non-cooperative thread, and still subject to cases where it might not be that simple.
-- David
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python