Sending Python statement over socket in chunks

[Diez B. Roggisch]

Jeffrey Barish schrieb:
> I have a python module that contains an assignment statement binding a long
> list of things to a name:
> 
> list_of_things = [thing1, thing2, ...]
> 
> where each thing instantiates class Thing when executed.  I send this
> statement through a socket to a remote module that executes it.  The
> problem is that it takes too long to send the entire statement. 
> Accordingly, I am thinking of a solution that sends the list in chunks. 
> The client then executes each chunk and reassembles the complete list. 
> Thus, I would first send something like:
> 
> chunk = [thing1, thing2, ... thing10]
> 
> and then
> 
> chunk = [thing11, thing12, ... thing20]
> 
> until all things have been transmitted.  At the client end, I would execute
> each chunk statement and then do
> 
> list_of_things.append(chunk)
> 
> The point of this solution is that I can start doing useful work in the
> client as soon as I receive the first chunk, and the others can arrive in
> the background and be available by the time I need them.
> 
> One way I could implement this solution is to execute the statement for the
> entire list_of_things in the server and then create chunk = [...]
> statements with the lists filled using the repr of the class.  I believe
> that this solution will work, but it seems a shame to execute the
> list_of_things statement in the server rather than have the server stupidly
> handle strings (because executing the statement takes time and because the
> server currently doesn't understand "Thing").  Should I investigate using a
> parser to carve up the list_of_things = [...] statement?  Basically, I just
> need to identify each Thing(...) expression and then count out some number
> of them.  I should be able to do that using re.  Or perhaps I should write
> my own parser using Python string operations as all I need to do is count
> out some number of Things delimited by "Thing(" at one end and "),\nThing("
> at the other (or ")]\n" at the end of the list).  Did I just answer my own
> question?
> 
> Of course, I need to balance the complexity of any alternative solution
> against simply executing the statement on the server.


Stop reinventing the wheel, start using pyro. Then either return the 
list as whole, or if it _really_ is to big, return subsequent slices of it.

Diez