Re: [Twisted-Python] Re: CPU intensive threads

These CPU intensive threads are python/C extensions generated with pyrex. Instead of messing with the GIL I will use processes as David suggests. It would be nice to have a process pool implementation that was as easy to use as deferToThread. Evidently they use the perspective broker for inter process communication in quotient.searchup, and that may be a good starting place. Thanks, -Nate On 27 Jul 2005 17:10:42 -0400, David Bolen wrote:

Jp Calderone writes:

...

POSH doesn't elimate the transport overhead. I've done some basic investigation, and it's incredibly slow. mmap() is probably a better solution in most cases, although I am not convinced multiple processes are called for in this case.

Me neither, but it would be a logical way to progress if you couldn't resolve things in the single process. Good to know about POSH though.

...

A solution which hasn't been suggested yet is to drop the native thread and use a cooperative Python thread. With this approach, you can choose to schedule it however you like, including /not/ scheduling it when you have other more important tasks to complete.

Well, but you'd still have the problem of ensuring that it was yielding back at a reasonable frequency wouldn't you? So it would be similar to sprinkling in a time.sleep(0) in a non-cooperative thread, and still subject to cases where it might not be that simple.

-- David

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python