[sapug] Stateless Iteration

Daryl Tester Daryl.Tester at iocane.com.au
Sun Jun 11 09:14:20 CEST 2006

Sub-Subject: Sad man seeks life on long weekend.

I'm puttering around with XML-RPC, which is all
relatively straight forward and such, and have come
across an issue that I've struck before in the past
and never adequately resolved, and that is the
burning issue of today - "Scalable APIs".

I want to provide an access method for a large list
of objects (say, Customers, or Accounts, or records
of some type - they will be uniform types).  If I
provide a method that says "return list", then it
may crunch away on a a very large dataset and occupy:

1) a large amount of bandwidth traversing the network
(although if the data is required, the data is
required), so this reason is a bit of a furfy.
B) a large amount of time to transfer the result
(variation on 1, but more important).

III) a large amount of memory to store the result
on the client.

(also an inability to stick to a consistent numbering
format, but that's the fault of the author's).

B & III are the ones I'm most concerned about.  If
the results were driving (for example) a GUI listbox,
I'd like the results to start populating that immediately
rather than waiting for the entire result set to be
returned.  If I'm driving an export or conversion process
then I don't need to store the entire result set on the
client, I can process each record as it comes in.

Cool, so some form of iterator would probably be ideal.
Unfortunately, to use an iterator I would need to retain
some form of state (insert dramatic chord: or would I?)
on the server, which, like cliches, I try to avoid like
the plague.  I could probably "cheat" (although I'm not
really sure that it's cheating) and provide an interface
that not only returns the data in question, but some form
of nonce that I could use to retrieve the next (or next
'x') records from.  The nonce could probably even be the
primary key of the next record, if it were against a
database table that had primary keys; I tend to use views
a lot (which may not have primary keys), and the result
set may not even be the output of a database.

So I was wondering if anyone has ever come across this
problem, found any naff solutions, or sunk any thought
into the problem?

  Daryl Tester, IOCANE Pty. Ltd.

More information about the sapug mailing list