[Python-3000] Kill GIL?

[Calvin Spealman]

On 9/17/06, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> "Giovanni Bajo" <rasky at develer.com> wrote:
> > Josiah Carlson <jcarlson at uci.edu> wrote:
> >
> > > It would be substantially easier if there were a distributed RPC
> > > mechanism that auto distributed to the "least-working" process in a
> > > set
> > > of potential working processes on a single machine.  [...]
> >
> > I'm not sure I follow you. Would you mind providing an example of a plausible
> > API for this mechanism (aka how the code would look like, compared to the
> > current Python threading classes)?
>
>     import autorpc
>     caller = autorpc.init_processes(autorpc.num_processors())
>
>     import callables
>     caller.register_module(callables)
>
>     result = caller.fcn1(arg1, arg2, arg3)
>
> The point is not to compare API/etc., with threading, but to compare it
> with XMLRPC.  Because ultimately, what I would like to see, is a
> mechanic similar to XMLRPC; call a method on an instance, that is
> automatically executed perhaps in some other thread in some other
> process, or maybe even in the same thread on the same process (depending
> on load, etc.), and which returns the result in-place.
>
> It's just much easier to handle (IMO).  The above example highlights an
> example of single call/return.  What if you don't care about getting a
> result back before continuing, or perhaps you have a bunch of things you
> want to get done?
>
>     ...
>     q = Queue.Queue()
>
>     caller.delayed(q.put).fcn1(arg1, arg2, arg3)
>     r = q.get() #will be delayed until q gets something
>
> What to do about exceptions happening in fcn1 remotely?  A fellow over
> in the wxPython mailing list brought up the idea of exception objects;
> perhaps not stackframes, etc., but perhaps an object with information
> like exception type and traceback, used for both delayed and non-delayed
> tracebacks.
>
>
>  - Josiah

I would be thrilled to see this kind of api brought into python. It
could very likely be implemented in time for Python 2.6, which would
be spawning processes to handle the load. At the very least, a Python
2.4 or older compatible module could be release to test the waters and
see what works and doesnt in this idea. I tried to wrap my head around
different options on how the GIL might go away, but the in end you
just realize you would hate to see  it go away.

I'm sure Twisted would have a field day with such a facility.

If this kind of thing gets brought into Python, it would almost
require some form of MapReduce come along with it. Of course, with the
existing talks about removing map as a built-in in favor of list
comprehensions, it makes one consider if listcomps and genexps might
have some way to utilize a distributed model natively. Some
consideration toward that end would be valuable.