[Python-Dev] Cloning threading.py using proccesses

[Chetan Pandya]

I just got around to reading the messages.

When I first saw this, I thought this is so that the processes that need to
share and work on shared objects. That is where the locks are required.
However, all shread objects are managed by the object manager and thus all
such operations are in effect sequential, even acquires on different locks.
Thus other shared objects in the object manager will actually not require
any (additional) synchronization. Of course, the argument here is that it is
still possible to use that code.

Cleanup of shared objects seems to be another thing to look out for. This is
a problem that subprocesses seem to avoid and has been already suggested.

-Chetan

On 10/11/06, python-dev-request at python.org <python-dev-request at python.org>
wrote:
>
> Message: 5
> Date: Wed, 11 Oct 2006 10:23:40 +0200
> From: "M.-A. Lemburg" <mal at egenix.com>
> Subject: Re: [Python-Dev] Cloning threading.py using proccesses
> To: Josiah Carlson <jcarlson at uci.edu>
> Cc: python-dev at python.org
> Message-ID: <452CAA0C.6030306 at egenix.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Josiah Carlson wrote:
> > Fredrik Lundh <fredrik at pythonware.com> wrote:
> >> Josiah Carlson wrote:
> >>
> >>> Presumably with this library you have created, you have also written a
> >>> fast object encoder/decoder (like marshal or pickle).  If it isn't any
> >>> faster than cPickle or marshal, then users may bypass the module and
> opt
> >>> for fork/etc. + XML-RPC
> >> XML-RPC isn't close to marshal and cPickle in performance, though, so
> >> that statement is a bit misleading.
> >
> > You are correct, it is misleading, and relies on a few unstated
> > assumptions.
> >
> > In my own personal delving into process splitting, RPC, etc., I usually
> > end up with one of two cases; I need really fast call/return, or I need
> > not slow call/return.  The not slow call/return is (in my opinion)
> > satisfactorally solved with XML-RPC.  But I've personally not been
> > satisfied with the speed of any remote 'fast call/return' packages, as
> > they usually rely on cPickle or marshal, which are slow compared to
> > even moderately fast 100mbit network connections.  When we are talking
> > about local connections, I have even seen cases where the
> > cPickle/marshal calls can make it so that forking the process is faster
> > than encoding the input to a called function.
>
> This is hard to believe. I've been in that business for a few
> years and so far have not found an OS/hardware/network combination
> with the mentioned features.
>
> Usually the worst part in performance breakdown for RPC is network
> latency, ie. time to connect, waiting for the packets to come through,
> etc. and this parameter doesn't really depend on the OS or hardware
> you're running the application on, but is more a factor of which
> network hardware, architecture and structure is being used.
>
> It also depends a lot on what you send as arguments, of course,
> but I assume that you're not pickling a gazillion objects :-)
>
> > I've had an idea for a fast object encoder/decoder (with limited support
> > for certain built-in Python objects), but I haven't gotten around to
> > actually implementing it as of yet.
>
> Would be interesting to look at.
>
> BTW, did you know about http://sourceforge.net/projects/py-xmlrpc/ ?
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Oct 11 2006)
> >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
>
>
> ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20061013/b836ab79/attachment.html