[Twisted-Python] Avoiding network when testing Perspective Broker
Hi! I am learning to develop TDD way. I want to create a server that understands PB protocol. Initially I thought it would be a good idea to avoid real network connections in my tests, so I tried to use `proto_helpers.StringTransport`: ---------- import cStringIO from twisted.spread import pb from twisted.trial import unittest from twisted.test import proto_helpers class Document(pb.Root): def remote_convert(self, props): self.props = props class DocTestCase(unittest.TestCase): def setUp(self): # set up server self.doc = Document() factory = pb.PBServerFactory(self.doc) self.broker = factory.buildProtocol(('127.0.0.1', 0)) tr = proto_helpers.StringTransport() self.broker.makeConnection(tr) # this is what a client sends self.props = {'name': 'MyDoc', 'path': '/path/'} # prepare data serialized_props = self.broker.serialize(self.props) msg = ('message', 1, 'root', 'convert', 1, ['tuple', serialized_props], ['dictionary']) io = cStringIO.StringIO() self.broker._encode(msg, io.write) self.chunk = io.getvalue() def test_convert(self): # data arrived self.broker.dataReceived(self.chunk) self.assertEqual(self.props, self.doc.props) ---------- However, `Document.remote_convert` is never executed so the test above fails. After debugging I discovered that `Broker._encode` produces different results depending on whether `self.broker.makeConnection(tr)` is called or not. And I created a test case that shows a difference. Here `test_convert1` succeeds while `test_convert2` fails: ---------- class DocTestCase(unittest.TestCase): def setUp(self): self.doc = Document() factory = pb.PBServerFactory(self.doc) self.broker = factory.buildProtocol(('127.0.0.1', 0)) self.props = {'name': 'MyDoc', 'path': '/path/'} serialized_props = self.broker.serialize(self.props) self.msg = ('message', 1, 'root', 'convert', 1, ['tuple', serialized_props], ['dictionary']) def test_convert1(self): self.broker.currentDialect = 'pb' self.broker.setPrefixLimit(64) self.broker.transport = proto_helpers.StringTransport() io = cStringIO.StringIO() self.broker._encode(self.msg, io.write) self.broker.dataReceived(io.getvalue()) self.assertEqual(self.props, self.doc.props) def test_convert2(self): self.tr = proto_helpers.StringTransport() self.broker.makeConnection(self.tr) io = cStringIO.StringIO() self.broker._encode(self.msg, io.write) self.broker.dataReceived(io.getvalue()) self.assertEqual(self.props, self.doc.props) ---------- I wonder what causes this behavior and, in general, if `StringTransport` is suitable for testing PB protocol. Thanks in advance. For your convenience I attached files with these test cases. -- with regards, Maxim
On 09:18 am, lacrima.maxim@gmail.com wrote:
Hi!
I am learning to develop TDD way. I want to create a server that
Hooray!
understands PB protocol. Initially I thought it would be a good idea to avoid real network connections in my tests, so I tried to use
Yes, that's definitely what you want to do.
`proto_helpers.StringTransport`:
StringTransport is frequently what you want in order to test a protocol implementation, so you're on the right track. Except... actually you're not implementing a protocol. You're *using* a protocol implementation that exists already and has its own unit tests. It would be better if you found a way to test your code without involving the PB protocol implementation or StringTransport. However, I admit that the tools for doing this with PB are almost non-existent. On the other hand, application code written for use with AMP is much more amenable to testing. Just something to think about.
So far so good. You made a factory, got it to make you a protocol, and hooked the protocol up to a StringTransport you made. That's all good stuff.
This part isn't quite as good. `broker.serialize` is basically an implementation detail. `broker._encode` is *definitely* an implementation detail. The exact structure of the message isn't quite an implementation detail, but it's such a low-level detail that you really don't want to be thinking about it while writing tests for something like Document.remote_convert. Instead, you should probably use the PB client API (ie, PBClientFactory and what comes out of it) to interact with this server. Let the PB client implementation figure out what bytes to "send" to your server.
You'll definitely need to call dataReceived at some point. Perhaps more than once. So this is pretty good.
I haven't bothered to look into what might be going on here, because I think you should forget about the `._encode` code and start using a client instead. The client is much more likely to produce the correct network traffic to exercise the code you want to exercise. The same goes for the two further tests below - which seem to differ only by a call to setPrefixLength, which should make absolutely no difference to this code when it is being used in practice, so the difference it makes in the tests is probably due to driving the code wrong, which problem will go away if you start using PBClientFactory etc. Hope this helps, Jean-Paul
On Oct 25, 2012, at 2:22 PM, exarkun@twistedmatrix.com wrote:
Yes, hooray! I wish that everyone did this. Sorry that you've encountered trouble; I very much hope you will stick with this and help us make the process for doing TDD on PB (and generally, Twisted) applications much easier.
While I agree with everything exarkun has said here, I think we should note that using StringTransport is basically the state of the art in protocol testing at this point. It's unfortunate that we don't have anything better, but you shouldn't feel the need to go scouring the code base for a good example or a testing utility that you've missed; unless it's immediately obvious to you how to remove byte-level parsing from the equation, this is just an area that Twisted needs to work on.
I'm curious, since we rarely get to see the positive impact of documentation, and only hear about it when it didn't exist - did you discover this testing style from http://twistedmatrix.com/documents/current/core/howto/trial.html#auto5 ? :)
To be clear, you should use PBClientFactory in the same way you've been these other abstractions: create it with a string transport, don't hook it up to any sockets or anything. There are three (count them, three!) classes called "IOPump" in various Twisted test-support modules. Please note that unlike the stuff in proto_helpers these are not supported and will likely go away in a future version of Twisted. However, they do provide a simple demonstration of how to hook up a client and server (calling dataReceived() with what transport.write() produced). If you'd like to help Twisted itself, we could really use a patch that cleans up all three of those implementations of IOPump, tests them, documents them, and exposes them in a way that would be beneficial to those writing tests like yours. Thanks for doing TDD, -glyph
Hi! Your replies are very encouraging. Thank you!
Yes, I started from that document. It describes well how to call certain parts of Twisted to check some behavior and this stimulated me to discover Twisted API. I think I have finally discovered a good way for writing tests that use PB. I used `test.test_pb.IOPump` `test.test_pb.connectedServerAndClient` as a basis for my tests. -------- from twisted.spread import pb from twisted.trial import unittest from twisted.test import proto_helpers class Document(pb.Root): def remote_convert(self): return 'I was called' class IOPump: def __init__(self, client, server, clientIO, serverIO): self.client = client self.server = server self.clientIO = clientIO self.serverIO = serverIO def pump(self): cData = self.clientIO.value() sData = self.serverIO.value() self.clientIO.clear() self.serverIO.clear() self.server.dataReceived(cData) self.client.dataReceived(sData) def connect(root): serverFactory = pb.PBServerFactory(root()) serverBroker = serverFactory.buildProtocol(()) clientFactory = pb.PBClientFactory() clientBroker = clientFactory.buildProtocol(()) clientTransport = proto_helpers.StringTransport() serverTransport = proto_helpers.StringTransport() clientBroker.makeConnection(clientTransport) serverBroker.makeConnection(serverTransport) pump = IOPump(clientBroker, serverBroker, clientTransport, serverTransport) # initial communication pump.pump() return clientFactory, serverFactory, pump class DocTestCase(unittest.TestCase): def test_convert(self): def cb0(doc): d = doc.callRemote('convert') return d def cb1(res): self.assertEqual('I was called', res) return res client, server, pump = connect(Document) d = client.getRootObject() d.addCallback(cb0) d.addCallback(cb1) pump.pump() pump.pump() return d -------- The only caveat here is that if I forget to call pump.pump() sufficient number of times, then the callback with an assertion may not be executed and this can lead to false positives -- with regards, Maxim
On Oct 26, 2012, at 7:16 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:
Nope, it has two other transport implementations - _LoopbackTransport and LoopbackRelay; plus, unlike what IOPump does, loopback is written to encourage you to return a Deferred from your test, which is generally a bad idea if you can easily avoid it. -glyph
On 09:18 am, lacrima.maxim@gmail.com wrote:
Hi!
I am learning to develop TDD way. I want to create a server that
Hooray!
understands PB protocol. Initially I thought it would be a good idea to avoid real network connections in my tests, so I tried to use
Yes, that's definitely what you want to do.
`proto_helpers.StringTransport`:
StringTransport is frequently what you want in order to test a protocol implementation, so you're on the right track. Except... actually you're not implementing a protocol. You're *using* a protocol implementation that exists already and has its own unit tests. It would be better if you found a way to test your code without involving the PB protocol implementation or StringTransport. However, I admit that the tools for doing this with PB are almost non-existent. On the other hand, application code written for use with AMP is much more amenable to testing. Just something to think about.
So far so good. You made a factory, got it to make you a protocol, and hooked the protocol up to a StringTransport you made. That's all good stuff.
This part isn't quite as good. `broker.serialize` is basically an implementation detail. `broker._encode` is *definitely* an implementation detail. The exact structure of the message isn't quite an implementation detail, but it's such a low-level detail that you really don't want to be thinking about it while writing tests for something like Document.remote_convert. Instead, you should probably use the PB client API (ie, PBClientFactory and what comes out of it) to interact with this server. Let the PB client implementation figure out what bytes to "send" to your server.
You'll definitely need to call dataReceived at some point. Perhaps more than once. So this is pretty good.
I haven't bothered to look into what might be going on here, because I think you should forget about the `._encode` code and start using a client instead. The client is much more likely to produce the correct network traffic to exercise the code you want to exercise. The same goes for the two further tests below - which seem to differ only by a call to setPrefixLength, which should make absolutely no difference to this code when it is being used in practice, so the difference it makes in the tests is probably due to driving the code wrong, which problem will go away if you start using PBClientFactory etc. Hope this helps, Jean-Paul
On Oct 25, 2012, at 2:22 PM, exarkun@twistedmatrix.com wrote:
Yes, hooray! I wish that everyone did this. Sorry that you've encountered trouble; I very much hope you will stick with this and help us make the process for doing TDD on PB (and generally, Twisted) applications much easier.
While I agree with everything exarkun has said here, I think we should note that using StringTransport is basically the state of the art in protocol testing at this point. It's unfortunate that we don't have anything better, but you shouldn't feel the need to go scouring the code base for a good example or a testing utility that you've missed; unless it's immediately obvious to you how to remove byte-level parsing from the equation, this is just an area that Twisted needs to work on.
I'm curious, since we rarely get to see the positive impact of documentation, and only hear about it when it didn't exist - did you discover this testing style from http://twistedmatrix.com/documents/current/core/howto/trial.html#auto5 ? :)
To be clear, you should use PBClientFactory in the same way you've been these other abstractions: create it with a string transport, don't hook it up to any sockets or anything. There are three (count them, three!) classes called "IOPump" in various Twisted test-support modules. Please note that unlike the stuff in proto_helpers these are not supported and will likely go away in a future version of Twisted. However, they do provide a simple demonstration of how to hook up a client and server (calling dataReceived() with what transport.write() produced). If you'd like to help Twisted itself, we could really use a patch that cleans up all three of those implementations of IOPump, tests them, documents them, and exposes them in a way that would be beneficial to those writing tests like yours. Thanks for doing TDD, -glyph
Hi! Your replies are very encouraging. Thank you!
Yes, I started from that document. It describes well how to call certain parts of Twisted to check some behavior and this stimulated me to discover Twisted API. I think I have finally discovered a good way for writing tests that use PB. I used `test.test_pb.IOPump` `test.test_pb.connectedServerAndClient` as a basis for my tests. -------- from twisted.spread import pb from twisted.trial import unittest from twisted.test import proto_helpers class Document(pb.Root): def remote_convert(self): return 'I was called' class IOPump: def __init__(self, client, server, clientIO, serverIO): self.client = client self.server = server self.clientIO = clientIO self.serverIO = serverIO def pump(self): cData = self.clientIO.value() sData = self.serverIO.value() self.clientIO.clear() self.serverIO.clear() self.server.dataReceived(cData) self.client.dataReceived(sData) def connect(root): serverFactory = pb.PBServerFactory(root()) serverBroker = serverFactory.buildProtocol(()) clientFactory = pb.PBClientFactory() clientBroker = clientFactory.buildProtocol(()) clientTransport = proto_helpers.StringTransport() serverTransport = proto_helpers.StringTransport() clientBroker.makeConnection(clientTransport) serverBroker.makeConnection(serverTransport) pump = IOPump(clientBroker, serverBroker, clientTransport, serverTransport) # initial communication pump.pump() return clientFactory, serverFactory, pump class DocTestCase(unittest.TestCase): def test_convert(self): def cb0(doc): d = doc.callRemote('convert') return d def cb1(res): self.assertEqual('I was called', res) return res client, server, pump = connect(Document) d = client.getRootObject() d.addCallback(cb0) d.addCallback(cb1) pump.pump() pump.pump() return d -------- The only caveat here is that if I forget to call pump.pump() sufficient number of times, then the callback with an assertion may not be executed and this can lead to false positives -- with regards, Maxim
On Oct 26, 2012, at 7:16 AM, Itamar Turner-Trauring <itamar@futurefoundries.com> wrote:
Nope, it has two other transport implementations - _LoopbackTransport and LoopbackRelay; plus, unlike what IOPump does, loopback is written to encourage you to return a Deferred from your test, which is generally a bad idea if you can easily avoid it. -glyph
participants (4)
-
exarkun＠twistedmatrix.com -
Glyph -
Itamar Turner-Trauring -
Maxim Lacrima