[sapug] Stateless Iteration
chris at inetd.com.au
Mon Jun 12 14:50:48 CEST 2006
On Mon, 12 Jun 2006, Daryl Tester wrote:
> One of the reasons I'm looking at using XML-RPC is to allow me to
> get Python into some areas where it hasn't previously been
> allowed [...]
That's a good idea. I had a recent need to use a piece of commercial
software where the only interface was a Perl module, so I wrapped it in
an XML-RPC server so I could use it from Python :-)
> I guess (and I'm still mulling over the problem definition in
> my head) that what I'm aiming for is to retain as little state
> as possible in the transport (could be XML-RPC, but could also
> easily be traditional RPC, or REST) and the backend server so
> that the server could be restarted without loss of state,
> because it won't have any to lose (helps minimise the need
> for persistent session management).
> Dammit, this where my thinking is breaking down - I'm not
> able to articulate the problem in all its gory details.
> OK, a concrete example. I offer the following service
> customer.fetch(next, limit)
> customer.fetch() returns the first customer (as an array of
> 1) and an ID of some sort. This returned ID can be fed into
> customer.fetch() as parameter "next" to return the next
> customer ID (in this example, it could be the primary key
> of the customer - it would map easily) and also return
> another ID, which if subsequently fed into customer.fetch()
> returns the next customer, and so forth. The "limit"
> parameter allows you to return a bunch of customers, as
> round trips can be expensive (and as before, the returned
> ID would point to the customer immediately after "limit"
> customers). In this example, no state is maintained by the
> server, so it could be rebooted in between requests and the
> client could be none the wiser. If the client was rebooted
> (or otherwise disappeared), the server doesn't have any
> session details to clean up. Sweet and simple. (There
> are other issues here that I'm glossing over, concurrent
> updates leaping to mind).
> The problem arises when I'm talking to something that's not
> a table with a primary key, like some sort of computed list
> (or a table without a primary key :-). "next" needs to be
> something that can be used by the server to reconstruct
> enough state to be able to resume the list evaluation from
> where it left off. "Next" is a nonce, and given a complicated
> enough nonce this could be done - i.e. the nonce is the remainder
> of the list - but then the client wouldn't need to go back
> to the server to fetch the remainder of the list when it
> already has it (it could be hidden by the client API, but
> then my goal in the first place was to lazy evaluate the list
> so it wasn't pulled across the link in its entirety).
> So I guess what I'm after is some method/pattern where, given
> an arbitrary function running on a server, I can extract enough
> state from it to be handed over to the client (ignoring for the
> moment issues like security) so that when the client says "give
> me more data", the server can reconstruct the previous state
> from the client supplied data, and take off from there.
Is maintaining the list of objects on the server for each client a real
resource problem ?
If not, and if your desired outcome as you say is:
"my goal in the first place was to lazy evaluate the list so it
wasn't pulled across the link in its entirety"
then what about putting the smarts into the client ?
e.g. If you created a server XMLRPC method customers.getsome(limit=1)
and get the client to manage loading of the data via a seperate thread
where it puts items from the list into a queue, then you could watch
the size of the queue and call customers.getsome(limit=50) in advance
of the queue being empty. If the client doesn't need all of the data,
it just needs to toggle a flag on the queue loader to not bother calling
 By queue, I mean something like Queue.Queue() which has locking
support so that different threads can safely use it.
Chris Foote <chris at inetd.com.au>
Inetd Pty Ltd T/A HostExpress
Phone: (08) 8410 4566
More information about the sapug