[Twisted-Python] Unjellying and circular references?
I'm using perspective broker to transfer objects in a networked game, which I'm having trouble unjellying -- the remote versions wind up with dangling twisted.persisted.crefutil._Dereference instances, so don't match the original objects. I'm seeing this for objects that have circular references to each other. I've refactored things to mostly avoid circular references and sidestep this, but have one remaining case where I find circular references mean clearer code that I'm reluctant to refactor. Is there some trick I'm missing to avoid _Dereferences?
On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips <jasperisjaded@gmail.com> wrote:
I'm using perspective broker to transfer objects in a networked game, which I'm having trouble unjellying -- the remote versions wind up with dangling twisted.persisted.crefutil._Dereference instances, so don't match the original objects.
Hi! I'd strongly suggest you switch to a different protocol. HTTP is quite capable. It has some annoying corners but you can avoid them if you control the client and the server. If you really want a Twisted-originated protocol, AMP is a much better choice than PB. It offers a little bit more functionality than HTTP without going to the extreme complexity of PB (I am not going to try to explain those complexities here, I'm just offering my personal conclusion based on 20 years of experience developing and/or using Twisted). This is not to say that PB is the wrong choice for all applications ... but probably almost all.
I'm seeing this for objects that have circular references to each other. I've refactored things to mostly avoid circular references and sidestep this, but have one remaining case where I find circular references mean clearer code that I'm reluctant to refactor.
Is there some trick I'm missing to avoid _Dereferences?
No, it's supposed to Just Work™ so you've found a bug in some part of the implementation. If you can produce a minimal reproducing example then it may be worth a bug report. But after that, I'd suggest investigating HTTP or AMP as a replacement for PB. Jean-Paul
On Fri, Dec 11, 2020 at 6:08 AM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips <jasperisjaded@gmail.com> wrote:
I'm using perspective broker to transfer objects in a networked game, which I'm having trouble unjellying -- the remote versions wind up with dangling twisted.persisted.crefutil._Dereference instances, so don't match the original objects.
I'm seeing this for objects that have circular references to each other. I've refactored things to mostly avoid circular references and sidestep this, but have one remaining case where I find circular references mean clearer code that I'm reluctant to refactor.
Is there some trick I'm missing to avoid _Dereferences?
No, it's supposed to Just Work™ so you've found a bug in some part of the implementation. If you can produce a minimal reproducing example then it may be worth a bug report. But after that, I'd suggest investigating HTTP or AMP as a replacement for PB.
Here's a test case demonstrating the bug: import sys from twisted.persisted.crefutil import _Dereference from twisted.spread import pb, jelly class RemoteCopyable( pb.Copyable, pb.RemoteCopy ): pass jelly.globalSecurity.allowInstancesOf( RemoteCopyable ) pb.setCopierForClassTree( sys.modules[__name__], pb.Copyable ) if __name__ == '__main__': # circular object ref cell = RemoteCopyable() cell.link = RemoteCopyable() cell.link.cell = cell # Mimic sending across network broker = pb.Broker() serializedCell = broker.serialize( cell ) remoteCell = broker.unserialize( serializedCell ) print( _Dereference is type(remoteCell.link.cell) ) This bug stems from twisted.spread.flavors.RemoteCopy.setCopyableState(), which works fine for Python 2! But the UTF fix for Python 3 broke circular references. It looks like this: def setCopyableState(self, state): if _PY3: state = {x.decode('utf8') if isinstance(x, bytes) else x:y for x,y in state.items()} self.__dict__ = state Here is a simple tweak that fixes the problem: def setCopyableState(self, state): if _PY3: for x, y in list(state.items()): if isinstance(x, bytes): del state[x] state[x.decode('utf8')] = y self.__dict__ = state Basically the reference unwinding that takes place in twisted.spread.jelly._Unjellier._unjelly_reference()'s call to ref.resolveDependants() relies upon setCopyableStates()'s passed in state being used directly, such that all matching references' __dict__ point to the same object. Hope this helps clarify. Is there some more formal location than this list that you'd like a bug report filed?
Out official bug tracker is at https://twistedmatrix.com/trac/newticket — if you could file it there it would be great. (Github login required.) Thanks!
On Dec 16, 2020, at 2:21 AM, Jasper Phillips <jasperisjaded@gmail.com> wrote:
On Fri, Dec 11, 2020 at 6:08 AM Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips <jasperisjaded@gmail.com> wrote:
I'm using perspective broker to transfer objects in a networked game, which I'm having trouble unjellying -- the remote versions wind up with dangling twisted.persisted.crefutil._Dereference instances, so don't match the original objects.
I'm seeing this for objects that have circular references to each other. I've refactored things to mostly avoid circular references and sidestep this, but have one remaining case where I find circular references mean clearer code that I'm reluctant to refactor.
Is there some trick I'm missing to avoid _Dereferences?
No, it's supposed to Just Work™ so you've found a bug in some part of the implementation. If you can produce a minimal reproducing example then it may be worth a bug report. But after that, I'd suggest investigating HTTP or AMP as a replacement for PB.
Here's a test case demonstrating the bug:
import sys from twisted.persisted.crefutil import _Dereference from twisted.spread import pb, jelly
class RemoteCopyable( pb.Copyable, pb.RemoteCopy ): pass jelly.globalSecurity.allowInstancesOf( RemoteCopyable ) pb.setCopierForClassTree( sys.modules[__name__], pb.Copyable )
if __name__ == '__main__': # circular object ref cell = RemoteCopyable() cell.link = RemoteCopyable() cell.link.cell = cell
# Mimic sending across network broker = pb.Broker() serializedCell = broker.serialize( cell ) remoteCell = broker.unserialize( serializedCell )
print( _Dereference is type(remoteCell.link.cell) )
This bug stems from twisted.spread.flavors.RemoteCopy.setCopyableState(), which works fine for Python 2! But the UTF fix for Python 3 broke circular references. It looks like this:
def setCopyableState(self, state): if _PY3: state = {x.decode('utf8') if isinstance(x, bytes) else x:y for x,y in state.items()} self.__dict__ = state
Here is a simple tweak that fixes the problem:
def setCopyableState(self, state): if _PY3: for x, y in list(state.items()): if isinstance(x, bytes): del state[x] state[x.decode('utf8')] = y self.__dict__ = state
Basically the reference unwinding that takes place in twisted.spread.jelly._Unjellier._unjelly_reference()'s call to ref.resolveDependants() relies upon setCopyableStates()'s passed in state being used directly, such that all matching references' __dict__ point to the same object.
Hope this helps clarify. Is there some more formal location than this list that you'd like a bug report filed? _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Glyph -
Jasper Phillips -
Jean-Paul Calderone