I just got around to reading the messages. When I first saw this, I thought this is so that the processes that need to share and work on shared objects. That is where the locks are required. However, all shread objects are managed by the object manager and thus all such operations are in effect sequential, even acquires on different locks. Thus other shared objects in the object manager will actually not require any (additional) synchronization. Of course, the argument here is that it is still possible to use that code. Cleanup of shared objects seems to be another thing to look out for. This is a problem that subprocesses seem to avoid and has been already suggested. -Chetan On 10/11/06, python-dev-request@python.org <python-dev-request@python.org> wrote:
Message: 5 Date: Wed, 11 Oct 2006 10:23:40 +0200 From: "M.-A. Lemburg" <mal@egenix.com> Subject: Re: [Python-Dev] Cloning threading.py using proccesses To: Josiah Carlson <jcarlson@uci.edu> Cc: python-dev@python.org Message-ID: <452CAA0C.6030306@egenix.com> Content-Type: text/plain; charset=ISO-8859-1
Josiah Carlson wrote:
Fredrik Lundh <fredrik@pythonware.com> wrote:
Josiah Carlson wrote:
Presumably with this library you have created, you have also written a fast object encoder/decoder (like marshal or pickle). If it isn't any faster than cPickle or marshal, then users may bypass the module and opt for fork/etc. + XML-RPC XML-RPC isn't close to marshal and cPickle in performance, though, so that statement is a bit misleading.
You are correct, it is misleading, and relies on a few unstated assumptions.
In my own personal delving into process splitting, RPC, etc., I usually end up with one of two cases; I need really fast call/return, or I need not slow call/return. The not slow call/return is (in my opinion) satisfactorally solved with XML-RPC. But I've personally not been satisfied with the speed of any remote 'fast call/return' packages, as they usually rely on cPickle or marshal, which are slow compared to even moderately fast 100mbit network connections. When we are talking about local connections, I have even seen cases where the cPickle/marshal calls can make it so that forking the process is faster than encoding the input to a called function.
This is hard to believe. I've been in that business for a few years and so far have not found an OS/hardware/network combination with the mentioned features.
Usually the worst part in performance breakdown for RPC is network latency, ie. time to connect, waiting for the packets to come through, etc. and this parameter doesn't really depend on the OS or hardware you're running the application on, but is more a factor of which network hardware, architecture and structure is being used.
It also depends a lot on what you send as arguments, of course, but I assume that you're not pickling a gazillion objects :-)
I've had an idea for a fast object encoder/decoder (with limited support for certain built-in Python objects), but I haven't gotten around to actually implementing it as of yet.
Would be interesting to look at.
BTW, did you know about http://sourceforge.net/projects/py-xmlrpc/ ?
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Source (#1, Oct 11 2006)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
------------------------------