Out official bug tracker is at https://twistedmatrix.com/trac/newticket — if you could file it there it would be great.  (Github login required.)


On Dec 16, 2020, at 2:21 AM, Jasper Phillips <jasperisjaded@gmail.com> wrote:

On Fri, Dec 11, 2020 at 6:08 AM Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips <jasperisjaded@gmail.com> wrote:
I'm using perspective broker to transfer objects in a networked game, which I'm having trouble unjellying -- the remote versions wind up with dangling twisted.persisted.crefutil._Dereference instances, so don't match the original objects.

I'm seeing this for objects that have circular references to each other. I've refactored things to mostly avoid circular references and sidestep this, but have one remaining case where I find circular references mean clearer code that I'm reluctant to refactor.

Is there some trick I'm missing to avoid _Dereferences?

No, it's supposed to Just Work™ so you've found a bug in some part of the implementation.  If you can produce a minimal reproducing example then it may be worth a bug report.  But after that, I'd suggest investigating HTTP or AMP as a replacement for PB.

Here's a test case demonstrating the bug:

import sys
from twisted.persisted.crefutil import _Dereference
from twisted.spread             import pb, jelly

class RemoteCopyable( pb.Copyable, pb.RemoteCopy ):  pass
jelly.globalSecurity.allowInstancesOf( RemoteCopyable )
pb.setCopierForClassTree( sys.modules[__name__], pb.Copyable )

if __name__ == '__main__':
    # circular object ref
    cell           = RemoteCopyable()
    cell.link      = RemoteCopyable()
    cell.link.cell = cell

    # Mimic sending across network
    broker         = pb.Broker()
    serializedCell = broker.serialize( cell )
    remoteCell     = broker.unserialize( serializedCell )

    print( _Dereference is type(remoteCell.link.cell) )

This bug stems from twisted.spread.flavors.RemoteCopy.setCopyableState(), which works fine for Python 2! But the UTF fix for Python 3 broke circular references. It looks like this:

def setCopyableState(self, state):
    if _PY3:
        state = {x.decode('utf8') if isinstance(x, bytes)
                 else x:y for x,y in state.items()}
    self.__dict__ = state

Here is a simple tweak that fixes the problem:

def setCopyableState(self, state):
    if _PY3:
        for x, y in list(state.items()):
            if isinstance(x, bytes):
                del state[x]
                state[x.decode('utf8')] = y
    self.__dict__ = state

Basically the reference unwinding that takes place in twisted.spread.jelly._Unjellier._unjelly_reference()'s call to ref.resolveDependants() relies upon setCopyableStates()'s passed in state being used directly, such that all matching references' __dict__ point to the same object.

Hope this helps clarify. Is there some more formal location than this list that you'd like a bug report filed?
Twisted-Python mailing list