xmlrpclib and generators

[Ken Seehart]

It would be really cool if an rpc call could return a generator.  I know 
that there are a few reasons why one would not expect it to be part of 
the library (e.g. the client would need to receive asynchronous 
messages, and it's not part of the rpc standard), however below I show a 
possible implementation, given that the client is also able to be a 
server...

I am aware of MultiCall, but that is actually something like the inverse 
of what I want.  I need a consumer.

Example:
------------------------------------------------------------------------

# hypothetical client code (running at "http://rpc.myserver.com")

from fancyxmlrpc import FancyXMLRPCServer

server = FancyXMLRPCServer(('localhost', 9000))

def primes():
    n = 2
    p = []
    while True:
        if not any( n % f == 0 for f in p ):
            yield n
            p.append( n )
        n += 1

server.register_generator(primes)
server.serve_forever()

------------------------------------------------------------------------
# hypothetical client code (running at http://www.mywebsite.com):

from fancyxmlrpc import FancyServerProxy

server_url = "http://rpc.myserver.com"
local_url = "http://www.mywebsite.com"

server = FancyServerProxy(server_url, local_url)

g = server.examples.getStateNames()

for x in g:
    print x
------------------------------------------------------------------------

Okay, so to implement this, the trick would be to implement the 
generator wrapper as a hidden xmlrpc conversation.  On the client side, 
FancyServerProxy would encapsulate a hidden RPC server with a function 
that receives each item that is yielded by the server and queues them 
while the client generator consumes the queue and yields the items.

The practical constraints of my specific application are:
1. The rpc server is a highly specialized slave system that does heavy 
duty work.
2. The rpc client is itself a web server that dispatches work requests 
to the rpc server(s) and displays the current status of work done so far.
3. The generators will typically run for a long time (hours) and yield 
data periodically (perhaps once a minute).
4. Trusted users will write generators on the server and consumers on 
the client (web site) and use the web site to make requests.
5. It would be even better if my generator yields numpy arrays rather 
than lists.
6. It would be nice to be able to scale to allow the web site to 
dispatch to multiple work servers.


So my questions are:
1. Does using xmlrpc make any sense for this?
2. I am missing an easier way to do this?
3. Any good examples of things like this?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090610/6d46acc3/attachment.html>