Hmmmm. You're implying that the state can't hold references to other objects? That might explain an exception I'm getting, which I'll describe below. If this is true, is there some easy way around this, or do I need a custom setCopyableState()?
The state *can* hold references to other objects (as long as those objects are themselves somehow transferrable, like pb.Copyable). I draw a distinction between having or not having references to other objects because data that is standalone makes more sense when transferred to other memory spaces. Imagine your data structures make up a big directed graph. The edges are references, each contained in one object and pointing at another. When you copy one of the objects out to another memory space, you're plucking a node out of the graph and putting it somewhere else. What happens to the edges? It depends upon what flavor the referenced objects inherit from. If they are pb.Referenceable, the edges turn into pb.RemoteReferences, and it's as if the edge-arrows are stretched to run from the object's new location back to the home memory space. If they are pb.Copyable, the reference is followed and the target object copied just like the original object was. If they are neither, you get an InsecureJelly exception. The idea is to prevent you from accidentially copying out objects that you didn't intend to be shared. So if the object points to a lot of other objects, those referents make up an environment. If the environment doesn't come with the object, then that object could be said to have a "home", and then it makes sense to talk about the "home" version of an object versus a copy that lives somewhere "away" from that home. If the object is mostly standalone, then it doesn't matter where the object lives and the home/away distinction is moot.
However, I'm sometimes getting an exception when the actual dictionary copying is done, as something other than a dict is being copied into __dict__. At this point the "jelType" is "dereference"...
Which exception is being raised? If it's the InsecureJelly, then you're referencing an object that doesn't inherit from one of the PB flavors. You either need to remove that reference in your getStateToCopy() method (cut the edge-arrow) or you need to make the referenced objects inherit from something like pb.Copyable. If it's something else, let us know (and provide a small test case??) so we can fix it at the sprint.
Damn, it looks like this might be the cuplrit. "reference" jelyTypes are recursively descended into before they are stored, and if a dereference is found before it's stored... some sort of _Dereference object is created? An attempt is then made to copy this into __dict__, and boom.
Yes, the current Jelly code looks for objects that are referenced multiple times in the same jellying call and marks them with "reference" tags. When another reference to the same object is detected, it is jellied with a "dereference" tag that points to the earlier "reference" marker. The "cook", "prepare", and "preserve" methods are used to implement these multiple phases. Circular or recursive references are handled because the reference number is allocated when we start to jelly the object, even though the state is not yet known. This scheme will change on Tuesday. The "reference" tags will go away and be replaced by an implicit marker that is notionally inserted every time we start jellying a new mutable object. The "dereference" tags will then point to these implicit markers. This should improve performance quite a bit, and will pave the way to a combined jelly+banana extension module that should give an enormous speedup (doing everything in C).
I'll have to look into this more closely latter. At first glance it appears to be something best fixed in twisted itself rather than a local setCopyableState()... at least to this twisted newbie. ;-)
Definitely. Twisted should "just handle" arbitrary reference graphs with no problems right now.. the change planned for the PyCon sprint will make it handle them faster and with less on-wire traffic than before.
I've since looked at this code more closely as well. The "else" branch in question is most definitely _not_ dead code, but it is confusing. ;-)
I think _unjelly_instance might be dead code, because I don't see anywhere an "instance" tag could be inserted into the stream. That might be compatibility with an older version of the jelly side, though. Another item on the PB sprint will be to implement proper version markers so this sort of thing can be done properly next time. cheers, -Brian