On 09:29 am, adi@roiban.ro wrote:

On 22 January 2013 02:21, <exarkun@twistedmatrix.com> wrote:

...

On 20 Jan, 02:35 pm, adi@roiban.ro wrote:

...

Hi,

I start my pet project using Python Nose as test runner. At the start, Twisted not used at all. Later I discovered Twisted and start using it... but testing Twisted requires trial.

I resist migrating to Trial and ended up with a custom code for running Twisted tests together with Nose:

https://github.com/chevah/chevah-rhinoplasty

I though I can share this code on the list, in case there is someone else wrestling with deferres testing outside of Trial.

Any feedback or comments is much appreciated.

The implementation is somewhat saddening in its use of Twisted internals (which makes it fragile and subject to easy breakage by new Twisted releases) and limited platform support (it won't work on Windows, afaik).

You may also find problems with direct use of `reactor.iterate`, since use of this API is highly discouraged due to re-entrancy issues and its implementation is not all that well tested by Twisted's own test suite.

If it works for you, great, but I would hesitate to recommend it to new projects as a testing tool. As Itamar suggests, nose should be able to run xUnit-style tests.

Jean-Paul

Thanks for your feedback.

----

The new methods successResultOf and failureResultOf looks good, but I think they only work with deferred that already have a result.

----

I am aware that the whole thing is fragile... and I am aware of the consequences of using internal non-public methods.

I checked the code from twisted.trial.unittest.TestCase but I could not spot how it waits for deferreds execution.

What part of twisted.trial.unittest.TestCase executes and waits for the deferred?

Can I use that code to implement something similar to successResultOf and failureResultOf but which also executes the deferred?

Hi Adi, trial does what it does by touching a lot of internal stuff as well. This is still bad, but at least it's our fault if it ever breaks instead of yours. There's also a long term plan (or "plan" may be putting it too strongly, perhaps I should say "hope") that this part of trial will change to only use public interfaces. This will probably require reactors actually implementing restartability, or it will require changing the trial feature slightly (eg, so it starts a reactor, runs all tests, then stops the reactor - if it did this, I'm sure you can imagine how "waiting" for a Deferred would just be adding a callback to the right place, as in any other Twisted-based application). Are you interested in helping out with making reactors restartable? :)

This code also works on Windows and on other Unix systems (OSX, Solaris)

Okay. I expected otherwise when I saw `_UnixWaker` in the implementation, but reading more carefully I now see that the code doesn't ever try to instantiate or use one of these, it just needs it for type checking. My mistake! Jean-Paul

On 18 Mar, 07:32 pm, adi@roiban.ro wrote:

On 22 January 2013 22:03, <exarkun@twistedmatrix.com> wrote:

...

On 09:29 am, adi@roiban.ro wrote:

...

On 22 January 2013 02:21, <exarkun@twistedmatrix.com> wrote:

...

On 20 Jan, 02:35 pm, adi@roiban.ro wrote:

I agree that this is a ugly hack and I removed the project.

...

Hi Adi,

trial does what it does by touching a lot of internal stuff as well. This is still bad, but at least it's our fault if it ever breaks instead of yours. There's also a long term plan (or "plan" may be putting it too strongly, perhaps I should say "hope") that this part of trial will change to only use public interfaces. This will probably require reactors actually implementing restartability, or it will require changing the trial feature slightly (eg, so it starts a reactor, runs all tests, then stops the reactor - if it did this, I'm sure you can imagine how "waiting" for a Deferred would just be adding a callback to the right place, as in any other Twisted-based application).

Are you interested in helping out with making reactors restartable? :)

Sorry for the late reply.

I am still clumsy when working with Twisted so I don't know if I can help to much here.

I don't know what is expected from a restartable reactor.

The way I am testing deferreds is by starting the reactor, allow for the deferred to execute and then stop the reactor.

I don't want to pause it and then continue the execution from where it was stopped.

To help with debugging I am also printing a snapshot of reactor state at a certain time.

----

I prefer the Arrange/Act/Assert way of writing test:

checker = mk.credentialsChecker() credentials = mk.credentials()

deferred = checker.requestAvatarId(credentials) failure = self.getDeferredFailure(deferred)

self.assertFailureType(AuthentiationError, failure) self.assertEqual(credentials.username, failure.value.username)

I found it easier to read than this version:

checker = mk.credentialsChecker() credentials = mk.credentials()

def check_result(result_or_failure): self.assertFailureType(AuthentiationError, failure) self.assertEqual(credentials.username, failure.value.username)

deferred = checker.requestAvatarId(credentials) deferred.addBoth(check_result)

return deferred

--------

I have updated the code to use as many public reactor members as possible.

The following private member are still use: reactor._startedBefore, reactor._started

It uses the following public methods: startRunning(), doIteration(), stop(), iterate()

Here is the main part that blocks the execution until the deferred got a result. It executes the deferred in the reactor loop.

https://github.com/chevah/empirical/blob/master/chevah/empirical/testcase.py...

------

Maybe this is has only limited usage, but I just wanted to share this work. For me, this makes writing test a much nicer experience.

Hi Adi, This basically looks like an implementation of the old, now-removed `TestCase.wait` API. We got rid of `wait` for several reasons: * It was hard to implement. By the end, it sort of worked with most reactors - but not all of them. * It is a tool for building non-deterministic, slow tests. If tests are written *not* to do real I/O and *not* to wait for real time to pass, then they don't need to let a real reactor spin. We replaced these ideas: * with returning a `Deferred` from a test method (which works even if you don't use trial to run your tests - but not if you don't subclass trial's `TestCase`). We eventually moved on from this idea, though many parts of Twisted itself are still tested using this feature, to... * things like `twisted.test.proto_helpers.MemoryReactor`, `twisted.internet.task.Clock`, and most recently `TestCase.successResultOf` and `TestCase.failureResultOf` (but don't confuse these with your `getDeferredFailure` - they are significantly less capable). I'd encourage you to explore testing strategies that use reactor/transport/time fakes and give us feedback about where they're not making your job easy enough. I think ultimately you'll be happier with the resulting tests, and you won't have to maintain so much hairy reactor manipulation code. Jean-Paul

On 25 March 2013 15:48, <exarkun@twistedmatrix.com> wrote:

On 18 Mar, 07:32 pm, adi@roiban.ro wrote:

...

On 22 January 2013 22:03, <exarkun@twistedmatrix.com> wrote:

...

On 09:29 am, adi@roiban.ro wrote:

...

On 22 January 2013 02:21, <exarkun@twistedmatrix.com> wrote:

...

On 20 Jan, 02:35 pm, adi@roiban.ro wrote:

I agree that this is a ugly hack and I removed the project.

...

Hi Adi,

trial does what it does by touching a lot of internal stuff as well. This is still bad, but at least it's our fault if it ever breaks instead of yours. There's also a long term plan (or "plan" may be putting it too strongly, perhaps I should say "hope") that this part of trial will change to only use public interfaces. This will probably require reactors actually implementing restartability, or it will require changing the trial feature slightly (eg, so it starts a reactor, runs all tests, then stops the reactor - if it did this, I'm sure you can imagine how "waiting" for a Deferred would just be adding a callback to the right place, as in any other Twisted-based application).

Are you interested in helping out with making reactors restartable? :)

Sorry for the late reply.

I am still clumsy when working with Twisted so I don't know if I can help to much here.

I don't know what is expected from a restartable reactor.

The way I am testing deferreds is by starting the reactor, allow for the deferred to execute and then stop the reactor.

I don't want to pause it and then continue the execution from where it was stopped.

To help with debugging I am also printing a snapshot of reactor state at a certain time.

----

I prefer the Arrange/Act/Assert way of writing test:

checker = mk.credentialsChecker() credentials = mk.credentials()

deferred = checker.requestAvatarId(credentials) failure = self.getDeferredFailure(deferred)

self.assertFailureType(AuthentiationError, failure) self.assertEqual(credentials.username, failure.value.username)

I found it easier to read than this version:

checker = mk.credentialsChecker() credentials = mk.credentials()

def check_result(result_or_failure): self.assertFailureType(AuthentiationError, failure) self.assertEqual(credentials.username, failure.value.username)

deferred = checker.requestAvatarId(credentials) deferred.addBoth(check_result)

return deferred

--------

I have updated the code to use as many public reactor members as possible.

The following private member are still use: reactor._startedBefore, reactor._started

It uses the following public methods: startRunning(), doIteration(), stop(), iterate()

Here is the main part that blocks the execution until the deferred got a result. It executes the deferred in the reactor loop.

https://github.com/chevah/empirical/blob/master/chevah/empirical/testcase.py...

------

Maybe this is has only limited usage, but I just wanted to share this work. For me, this makes writing test a much nicer experience.

Hi Adi,

This basically looks like an implementation of the old, now-removed `TestCase.wait` API.

We got rid of `wait` for several reasons:

* It was hard to implement. By the end, it sort of worked with most reactors - but not all of them.

* It is a tool for building non-deterministic, slow tests. If tests are written *not* to do real I/O and *not* to wait for real time to pass, then they don't need to let a real reactor spin.

We replaced these ideas:

* with returning a `Deferred` from a test method (which works even if you don't use trial to run your tests - but not if you don't subclass trial's `TestCase`). We eventually moved on from this idea, though many parts of Twisted itself are still tested using this feature, to...

* things like `twisted.test.proto_helpers.MemoryReactor`, `twisted.internet.task.Clock`, and most recently `TestCase.successResultOf` and `TestCase.failureResultOf` (but don't confuse these with your `getDeferredFailure` - they are significantly less capable).

I'd encourage you to explore testing strategies that use reactor/transport/time fakes and give us feedback about where they're not making your job easy enough. I think ultimately you'll be happier with the resulting tests, and you won't have to maintain so much hairy reactor manipulation code.

I already use StringTransport in various forms for unit and integration tests and it is great! Clock is also great for delayedCalls. I also have various mock/dummy/spy implementations for transports/channels. The whole Twisted architecture make writing tests a fun task! Thanks! I am "spining" the reactor to "resolve" all deferreds involved in a StringTransport conversation or in an DeferredList or other kind of chained deferreds. These are what I call "integration tests" and they only use memory, no external I/O. In most of my calles of result = self.getDeferredResult(deferred), the deferred's callback() method was already called and I just want to resolve the callbacks chain. Is there something in _synctest that can "resolve" a list/chain of deferreds? I was not aware of MemoryReactor. Thanks for the note. From what I can read in the code, it does not help with spinning the reactor. ----- I only do real I/O for what I call, "system tests" these are checking integration of my code with the outside world (the system). They are kept to a minimum. Since there should not be to many of these tests, returning a deferred is not a big annoyance... just that the code is harder to read. Thanks again for all your feedback! -- Adi Roiban

On 26 March 2013 13:40, Itamar Turner-Trauring <itamar@itamarst.org> wrote:

On 03/26/2013 06:39 AM, Adi Roiban wrote:

...

I am "spining" the reactor to "resolve" all deferreds involved in a StringTransport conversation or in an DeferredList or other kind of chained deferreds. These are what I call "integration tests" and they only use memory, no external I/O. If you're only using StringTransport or Clock, there is no need for a real reactor. Deferreds have nothing to do with the reactor as such.

...

In most of my calles of result = self.getDeferredResult(deferred), the deferred's callback() method was already called and I just want to resolve the callbacks chain.

Again, that does not require the reactor. E.g.:

...

...

...

d = Deferred() d.callback(1) l = [] d.addCallback(l.append) l [1]

You might also want to look at the latest version of http://twistedmatrix.com/documents/current/core/howto/trial.html, in particular the new successResultOf and similar APIs.

Many thanks for you comment. I am stupid :) and successResultOf is great! -- Adi Roiban