[Twisted-Python] Testing AMP-based code
Hi,
following up from ticket #6502, I'm looking for recommendations/best practices for writing unit-tests for AMP-based code.
As described in the ticket, the issue I'm currently facing is that the AMP implementation is subtly not re-entrant safe and doesn't work with a synchronous transport, for example this code raises an exception:
http://twistedmatrix.com/trac/attachment/ticket/6502/example.py
I'm starting to think that the most appropriate testing strategy would be to mock/stub AMP.callRemote (or the protocol class altogether) instead of trying to use a fake transport.
Thoughts?
Cheers,
Free
On 5 Jun, 07:13 pm, free@64studio.com wrote:
Hi,
following up from ticket #6502, I'm looking for recommendations/best practices for writing unit-tests for AMP-based code.
As described in the ticket, the issue I'm currently facing is that the AMP implementation is subtly not re-entrant safe and doesn't work with a synchronous transport, for example this code raises an exception:
http://twistedmatrix.com/trac/attachment/ticket/6502/example.py
I'm starting to think that the most appropriate testing strategy would be to mock/stub AMP.callRemote (or the protocol class altogether) instead of trying to use a fake transport.
Thoughts?
I think this is thinking in the right direction. Twisted generally tries to be responsible for testing its own code, and the serialization from commands to bytes (and the reverse) that AMP does is part of Twisted, so you should really be free from the burden of testing that that stuff works.
One thing it's worth noticing is that the AMP class itself contains very little code. Instead, it inherits most functionality from a few base classes. It's worth learning about the division of responsibility between these classes because they can be helpful in writing cleaner AMP code and - relevant to this topic - writing AMP unit tests.
One of the base classes, BinaryBoxProtocol, is almost entirely concerned with serialization logic. You can probably ignore this one entirely to begin with (although consider the consequences of serialization/deserialization living in this one class, independent of the rest of the protocol logic: perhaps you want to exchange AMP commands with a web browser and would find JSON an easier format to work with than AMP's int16 string scheme... etc).
Next, BoxDispatcher. This one is what actually implements `callRemote` and the reverse - ampBoxReceived, turning an AMP box (already parsed), into an incoming method call or the result of a previous outgoing method call. It operates on locator (to look up how to handle incoming method calls) and a box sender (to send out boxes representing method calls or responses). It doesn't know about the network, so your box sender can be a purely in-memory thing, implementing some box handling logic purely as Python code and no I/O.
As far as the locator goes, if you want the standard `@SomeCommand.responder` functionality, then you can easily get this by re-using the next base class of AMP, CommandLocator. This one's pretty straight-forward: subclass it and those decorators will work for you.
Ignore the last one, SimpleStringLocator, it's for extremely old-style AMP code that no one should be writing anymore.
So this all means that your application logic can all live on a CommandLocator subclass. When you really want to put this on an AMP server, you can hook an AMP instance up to your CommandLocator subclass (AMP takes a locator as an __init__ argument). When you want to test your command implementations, you can hook the CommandLocator up to a BoxDispatcher and a box sender and throw boxes straight at it with no network interation.
Some pieces are probably still missing from the public API - for example, you do want to test that your objects all get properly serialized and deserialized through AMP, particularly if you're implementing custom Argument types. There are some private APIs, _objectsToStrings and _stringsToObjects mostly, that really help with testing this, and we should think about how to expose this functionality publically. Also, we should document this whole pile of stuff. Maybe you'd be interested in writing something up after you've had a chance to play with these ideas?
Jean-Paul
On Jun 12, 2013, at 4:16 PM, exarkun@twistedmatrix.com wrote:
[snip]
So this all means that your application logic can all live on a CommandLocator subclass. When you really want to put this on an AMP server, you can hook an AMP instance up to your CommandLocator subclass (AMP takes a locator as an __init__ argument). When you want to test your command implementations, you can hook the CommandLocator up to a BoxDispatcher and a box sender and throw boxes straight at it with no network interation.
Thanks a lot for this description. Someone (maybe you if you're feeling generous, or someone else if they'd like to volunteer) should take most of the prose from this post, edit it a little bit to fit into the flow, and use it to expand the somewhat bare explanation of these concepts in http://twistedmatrix.com/documents/current/core/howto/amp.html.
Some pieces are probably still missing from the public API - for example, you do want to test that your objects all get properly serialized and deserialized through AMP, particularly if you're implementing custom Argument types. There are some private APIs, _objectsToStrings and _stringsToObjects mostly, that really help with testing this, and we should think about how to expose this functionality publically. Also, we should document this whole pile of stuff. Maybe you'd be interested in writing something up after you've had a chance to play with these ideas?
It would also be nice if someone could find a link to the tickets for this functionality - or, file one, if none exist.
Again, thanks a lot for the great explanation, JP.
-glyph
|--==> On Wed, 12 Jun 2013 23:16:12 -0000, exarkun@twistedmatrix.com said:
I think this is thinking in the right direction. Twisted generally tries to be responsible for testing its own code, and the serialization from commands to bytes (and the reverse) that AMP does is part of Twisted, so you should really be free from the burden of testing that that stuff works.
Agreed. Thanks for the detailed explanation about the AMP internals, I hadn't take time yet to fully read the implementation, I did now and your directions helped.
I'm certainly happy to integrate the docs with this information as Glyph suggested.
So this all means that your application logic can all live on a CommandLocator subclass.
Indeed, I had figured out this particular bit already.
When you really want to put this on an AMP server, you can hook an AMP instance up to your CommandLocator subclass (AMP takes a locator as an __init__ argument). When you want to test your command implementations, you can hook the CommandLocator up to a BoxDispatcher and a box sender and throw boxes straight at it with no network interation.
Okay this is where it get interesting and I'd like to hear for opinions for a few different possibilities. Following the your idea above, I've put together this little example of application code with tests (it expands the one attached to the ticket):
I've included some comments that are not really code comments, but rather just express issues/considerations.
I believe there are two parts of application code that you want to test (possibly differently). One part are the application-defined AMP commands and their responders, the other part is the application client code that make use of them via callRemote. So basically two different layers.
My question is probably if folks have any opinion about whether to mock the parts of your applications that invoke callRemote (typically by providing fakes, so you don't invoke callRemote at all), or alternatively to fake callRemote itself. As I wrote in the paste, this probably drifts towards a "state-based vs behavior-based" matter.
Some pieces are probably still missing from the public API - for example, you do want to test that your objects all get properly serialized and deserialized through AMP, particularly if you're implementing custom Argument types. There are some private APIs, _objectsToStrings and _stringsToObjects mostly, that really help with testing this, and we should think about how to expose this functionality publically. Also, we should document this whole pile of stuff.
I do have custom Argument types. Those could be exercised indirectly by faking callRemote the way the FakeBoxDispatcher class in the paste does, but there might be nicer approaches.
Maybe you'd be interested in writing something up after you've had a chance to play with these ideas?
Sure. It feels part of these questions are not purely AMP-specific and are generally relevant for testing application code that uses protocols indirectly. I guess a popular approach is to wrap protocols inside "Client" classes and then provide fake versions of those clients, which at the end is perhaps what I should do in this case too (e.g. fake the MathClient in the code).
Cheers,
Free
<exarkun <at> twistedmatrix.com> writes:
I think this is thinking in the right direction. Twisted generally tries to be responsible for testing its own code, and the serialization from commands to bytes (and the reverse) that AMP does is part of Twisted, so you should really be free from the burden of testing that that stuff works.
....
Some pieces are probably still missing from the public API - for example, you do want to test that your objects all get properly serialized and deserialized through AMP, particularly if you're implementing custom Argument types. There are some private APIs, _objectsToStrings and _stringsToObjects mostly, that really help with testing this, and we should think about how to expose this functionality publically. Also, we should document this whole pile of stuff. Maybe you'd be interested in writing something up after you've had a chance to play with these ideas?
Jean-Paul
Greetings!
Perhaps I should create a new thread, but I think that my problem could be related with this question.
I want to implement master/slave protocol based on AMP, here is a slave side code https://github.com/MichaelMayorov/buildbot/compare/fake-master-slave- amp
The problem is that I want to pass few arguments to Bot https://github.com/MichaelMayorov/buildbot/compare/fake-master-slave- amp#L0R38 from this place https://github.com/MichaelMayorov/buildbot/compare/fake-master-slave- amp#L0R124 but current implementation not allow me to do this. I've read twisted API docs and discussed my idea on #buildbot where finally take advice to post my question here.
Thanks!
On 16 Jul, 07:13 pm, marchael@ya.ru wrote:
<exarkun <at> twistedmatrix.com> writes:
I think this is thinking in the right direction. Twisted generally tries to be responsible for testing its own code, and the serialization from commands to bytes (and the reverse) that AMP does is part of Twisted, so you should really be free from the burden of testing that that stuff works.
....
Some pieces are probably still missing from the public API - for example, you do want to test that your objects all get properly serialized and deserialized through AMP, particularly if you're implementing custom Argument types. There are some private APIs, _objectsToStrings and _stringsToObjects mostly, that really help with testing this, and we should think about how to expose this functionality publically. Also, we should document this whole pile of stuff. Maybe you'd be interested in writing something up after you've had a chance to play with these ideas?
Jean-Paul
Greetings!
Perhaps I should create a new thread, but I think that my problem could be related with this question.
I want to implement master/slave protocol based on AMP, here is a slave side code https://github.com/MichaelMayorov/buildbot/compare/fake-master- slave- amp
The problem is that I want to pass few arguments to Bot https://github.com/MichaelMayorov/buildbot/compare/fake-master-slave- amp#L0R38 from this place https://github.com/MichaelMayorov/buildbot/compare/fake-master-slave- amp#L0R124 but current implementation not allow me to do this. I've read twisted API docs and discussed my idea on #buildbot where finally take advice to post my question here.
There are a lot of ways to go about this. The smallest change required to pass an argument to the `Bot` protocol in your code is to use a lambda:
factory = Factory() factory.protocol = lambda: Bot(arg1, arg2, ...)
This isn't necessarily the best solution. Some other options include:
- use the newly introduced Factory.forProtocol(Bot, arg1, arg2, ...) - use connectProtocol(endpoint, Bot(arg1, arg2, ...)) (only slightly less new)
Jean-Paul
Great thanks! That helps me a lot!
participants (4)
-
exarkun＠twistedmatrix.com -
Free Ekanayaka -
Glyph -
MichaelMayorov