New subject: [Twisted-Python] Unjelly - recursion limit reached

Nov. 6, 2005

      ...
When a view_ method is called in a cacheable and the return value is a
graph of interconnected cacheables, is the return value completely
serialized at that point?
Do you mean that some view_ method is being invoked on a pb.Referenceable (or
pb.ViewPoint, I suppose) ? And that method is returning an instance that
inherits from pb.Cacheable? And this pb.Cacheable instance holds references
to other pb.Cacheables? Which may or may not have been transmitted across
this wire once already?

The general answer is yes, the return value is completely serialized right
then. The serialized form of the return value is transmitted in pieces as
jelly recurses down the retval object graph, each new node visited causes a
little bit more data to be sent. However, it does not stop for anything, and
there is no way for an object being serialized to indicate that it wants to
put off serialization for a while[1]. On the receiving end, the
callRemote()'s Deferred will not be fired until the retval has completely
finished deserialization. Intermediate objects may be constructed while
deserialization is taking place, but that ought to be invisible to the
caller.

I'm not sure what would happen if, say, your getStateToCacheAndObserveFor
method did a callRemote though the same wire. newpb has a queue to handle
this sort of thing (the callRemote doesn't get transmitted until at least
after the current operation has finished), but I don't know what oldpb does.
If it isn't clever enough to be reentrant, the receiving end will get
interleaved object state from the two operations and the results will be very
very messy.

getStateToCacheAndObserveFor and getStateToCopy are called from PB internals,
and as a result it may not be safe to make other PB calls from there. view_*
is called when the wire is in a stable state (i.e. remote methods calls are
top-level objects on the wire), so I think it should be safe to make
arbitrary PB calls from it.
...
I'm concerned that my hierarchy of cacheables received by the client  
is inconsistent because it is being modified by another client view_  
call into the server before the previous hierarchy has been fully  
serialized to the first client.
Note that each callRemote is more-or-less atomic, and the creation of a whole
object graph is more-or-less atomic, but it is entirely possible that other
method calls will happen during the middle of a huge slew of RemoteCache
updates. So if object A modifies a dozen Cacheables at once, and object B has
a RemoteCache that is watching them, object C might sneak in a callRemote
while only half of the updates have been processed. The same thing will
happen if your Cacheable update method does multiple callRemotes to do its
job.

hope that helps somehow,
 -Brian

[1]: in newpb, if you enable it, Slicers can return a Deferred to indicate
     that they want to stall serialization for a while. This raises concerns
     about knowing when, if ever, serialization will resume, and increases
     uncertainty about state coherency, because all sorts of stuff could
     happen by the time serialization is resumed.

Re: [Twisted-Python] Unjelly - recursion limit reached

Brian Warner

David K. Hess

David K. Hess

tags

participants (2)