[sapug] Stateless Iteration

Chris Foote chris at inetd.com.au
Mon Jun 12 14:50:48 CEST 2006

On Mon, 12 Jun 2006, Daryl Tester wrote:

> One of the reasons I'm looking at using XML-RPC is to allow me to
> get Python into some areas where it hasn't previously been
> allowed [...]

That's a good idea.  I had a recent need to use a piece of commercial
software where the only interface was a Perl module, so I wrapped it in
an XML-RPC server so I could use it from Python :-)

> I guess (and I'm still mulling over the problem definition in
> my head) that what I'm aiming for is to retain as little state
> as possible in the transport (could be XML-RPC, but could also
> easily be traditional RPC, or REST) and the backend server so
> that the server could be restarted without loss of state,
> because it won't have any to lose (helps minimise the need
> for persistent session management).
> Dammit, this where my thinking is breaking down - I'm not
> able to articulate the problem in all its gory details.
> OK, a concrete example.  I offer the following service
> interface:
> customer.fetch()
> customer.fetch(next)
> customer.fetch(next, limit)
> customer.fetch() returns the first customer (as an array of
> 1) and an ID of some sort.  This returned ID can be fed into
> customer.fetch() as parameter "next" to return the next
> customer ID (in this example, it could be the primary key
> of the customer - it would map easily) and also return
> another ID, which if subsequently fed into customer.fetch()
> returns the next customer, and so forth.  The "limit"
> parameter allows you to return a bunch of customers, as
> round trips can be expensive (and as before, the returned
> ID would point to the customer immediately after "limit"
> customers).  In this example, no state is maintained by the
> server, so it could be rebooted in between requests and the
> client could be none the wiser. If the client was rebooted
> (or otherwise disappeared), the server doesn't have any
> session details to clean up.  Sweet and simple.  (There
> are other issues here that I'm glossing over, concurrent
> updates leaping to mind).
> The problem arises when I'm talking to something that's not
> a table with a primary key, like some sort of computed list
> (or a table without a primary key :-).  "next" needs to be
> something that can be used by the server to reconstruct
> enough state to be able to resume the list evaluation from
> where it left off.  "Next" is a nonce, and given a complicated
> enough nonce this could be done - i.e. the nonce is the remainder
> of the list - but then the client wouldn't need to go back
> to the server to fetch the remainder of the list when it
> already has it (it could be hidden by the client API, but
> then my goal in the first place was to lazy evaluate the list
> so it wasn't pulled across the link in its entirety).
> So I guess what I'm after is some method/pattern where, given
> an arbitrary function running on a server, I can extract enough
> state from it to be handed over to the client (ignoring for the
> moment issues like security) so that when the client says "give
> me more data", the server can reconstruct the previous state
> from the client supplied data, and take off from there.

Is maintaining the list of objects on the server for each client a real
resource problem ?

If not, and if your desired outcome as you say is:
 	"my goal in the first place was to lazy evaluate the list so it
 	wasn't pulled across the link in its entirety"
then what about putting the smarts into the client ?

e.g.  If you created a server XMLRPC method customers.getsome(limit=1)
and get the client to manage loading of the data via a seperate thread
where it puts items from the list into a queue[1], then you could watch
the size of the queue and call customers.getsome(limit=50) in advance
of the queue being empty.  If the client doesn't need all of the data,
it just needs to toggle a flag on the queue loader to not bother calling
customers.getsome() anymore.

[1] By queue, I mean something like Queue.Queue() which has locking
support so that different threads can safely use it.


Chris Foote <chris at inetd.com.au>
Inetd Pty Ltd T/A HostExpress
Web:   http://www.hostexpress.com.au
Blog:  http://www.hostexpress.com.au/drupal/chris
Phone: (08) 8410 4566

More information about the sapug mailing list