[Python-Dev] Cloning threading.py using proccesses

[M.-A. Lemburg]

Josiah Carlson wrote:
> Fredrik Lundh <fredrik at pythonware.com> wrote:
>> Josiah Carlson wrote:
>>
>>> Presumably with this library you have created, you have also written a
>>> fast object encoder/decoder (like marshal or pickle).  If it isn't any
>>> faster than cPickle or marshal, then users may bypass the module and opt
>>> for fork/etc. + XML-RPC
>> XML-RPC isn't close to marshal and cPickle in performance, though, so 
>> that statement is a bit misleading.
> 
> You are correct, it is misleading, and relies on a few unstated
> assumptions.
> 
> In my own personal delving into process splitting, RPC, etc., I usually
> end up with one of two cases; I need really fast call/return, or I need
> not slow call/return.  The not slow call/return is (in my opinion)
> satisfactorally solved with XML-RPC.  But I've personally not been
> satisfied with the speed of any remote 'fast call/return' packages, as
> they usually rely on cPickle or marshal, which are slow compared to
> even moderately fast 100mbit network connections.  When we are talking
> about local connections, I have even seen cases where the
> cPickle/marshal calls can make it so that forking the process is faster
> than encoding the input to a called function.

This is hard to believe. I've been in that business for a few
years and so far have not found an OS/hardware/network combination
with the mentioned features.

Usually the worst part in performance breakdown for RPC is network
latency, ie. time to connect, waiting for the packets to come through,
etc. and this parameter doesn't really depend on the OS or hardware
you're running the application on, but is more a factor of which
network hardware, architecture and structure is being used.

It also depends a lot on what you send as arguments, of course,
but I assume that you're not pickling a gazillion objects :-)

> I've had an idea for a fast object encoder/decoder (with limited support
> for certain built-in Python objects), but I haven't gotten around to
> actually implementing it as of yet.

Would be interesting to look at.

BTW, did you know about http://sourceforge.net/projects/py-xmlrpc/ ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 11 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::