[Python-Dev] Cloning threading.py using proccesses

[Josiah Carlson]

Fredrik Lundh <fredrik at pythonware.com> wrote:
> 
> Josiah Carlson wrote:
> 
> > Presumably with this library you have created, you have also written a
> > fast object encoder/decoder (like marshal or pickle).  If it isn't any
> > faster than cPickle or marshal, then users may bypass the module and opt
> > for fork/etc. + XML-RPC
> 
> XML-RPC isn't close to marshal and cPickle in performance, though, so 
> that statement is a bit misleading.

You are correct, it is misleading, and relies on a few unstated
assumptions.

In my own personal delving into process splitting, RPC, etc., I usually
end up with one of two cases; I need really fast call/return, or I need
not slow call/return.  The not slow call/return is (in my opinion)
satisfactorally solved with XML-RPC.  But I've personally not been
satisfied with the speed of any remote 'fast call/return' packages, as
they usually rely on cPickle or marshal, which are slow compared to
even moderately fast 100mbit network connections.  When we are talking
about local connections, I have even seen cases where the
cPickle/marshal calls can make it so that forking the process is faster
than encoding the input to a called function.

I've had an idea for a fast object encoder/decoder (with limited support
for certain built-in Python objects), but I haven't gotten around to
actually implementing it as of yet.


> the really interesting thing here is a ready-made threading-style API, I 
> think.  reimplementing queues, locks, and semaphores can be a reasonable 
> amount of work; might as well use an existing implementation.

Really, it is a matter of asking what kind of API is desireable.  Do we
want to have threading plus other stuff be the style of API that we want
to replicate?  Do we want to have shared queue objects, or would an
XML-RPC-esque remote.queue_put('queue_X', value) and
remote.queue_get('queue_X', blocking=1) be better?


 - Josiah