[Twisted-Python] Blocking on deferreds during unit tests (e.g. deferredResult)
I'm starting a new thread on this because It's very important to me, and we need to have some kind of resolution on this issue. On Fri, 06 May 2005 16:00:50 +1000, Jp Calderone <exarkun@divmod.com> wrote: [...]
I would like to have more details on this considering that I use deferredResult and deferredError quite extensively, and very much appreciate their existence. If methods suck this bad we need to deprecate them and provide something better. Is there a better way to synchronously spin the reactor until a deferred fires? There any many cases in which I prefer to do that instead of returning a deferred from the test case method. -Eric
Eric Mangold wrote:
I'm starting a new thread on this because It's very important to me, and we need to have some kind of resolution on this issue.
Is there a better way to synchronously spin the reactor until a deferred fires?
Yep. Write a unit test framework that uses Stackless and create a tasklet for each deferred that you want to run.
There any many cases in which I prefer to do that instead of returning a deferred from the test case method.
Without language support, this is an exercise in futility. Don't do it. Trial is broken, and everyone laments it, but the *reason* that it's broken (the root cause, anyway, if not the only reason) is that it bends over backwards to accomodate this use-case when _every_ _other_ _part_ of Twisted is very explicit that this is a bad, bad thing to do and you should never do it because it doesn't work. I am sorry that so many people want to block on deferreds in test methods, but I can guarantee you that support for doing it *WILL* be removed in a future version of Twisted. If we don't eventually kill this feature, our test suite is never going to work reliably.
On 5/7/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
I'd suggest 1) not writing a unit test framework, and instead using trial, and 2) instead of using stackless, use greenlet + my gthreadless module that integrates greenlets and Deferreds (sandbox/radix/gthreadless.py). Alternatively, if you don't want to use greenlet, as Bob said, just use deferred generators (twisted.internet.defer.{deferredGenerator,waitForDeferred}). HTH, HAND :-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On 5/6/05, Christopher Armstrong <radeex@gmail.com> wrote:
gthreadless looks quite cool! From things I've seen along the way, it seems that many people are turned off by the callback style of programming required by an async framework like Twisted. Using gthreadless, your code looks just like normal synchronous code. I'm only a little familiar with greenlet. Is there any significant disadvantage to using this approach extensively? For example, is the overhead very large? Using gthreadless, one could interface sqlobject and Twisted's db interface to take advantage of the work done in Twisted Enterprise to make sqlobject-style database access safe for Twisted. (It's undoubtedly possible without gthreadless, but it seems like it would be a lot easier) Kevin
On Sat, 7 May 2005 11:02:09 -0400, Kevin Dangoor <dangoor@gmail.com> wrote:
The main advantage is also the main disadvantage: when using gthreadless, rather than function callbacks, it becomes less obvious where context switches occur, leading to an increased likelihood of introducing concurrency-related bugs (primarily due to switches out of non-atomic operations). Unlike pre-emptive threads, such programming errors can be avoided entirely without the use of locks, but doing so may require more skill and attention to detail than would be required when writing the same code without gthreadless. This is somewhat unfortunate, since programmers least familiar with the concurrency problems solved by Twisted's current mechanisms -- function callbacks -- seem to be the most likely to be attracted by solutions like gthreadless while at the same time being the least prepared to develop software correctly with them. Of course, this isn't to say gthreadless isn't useful or an improvement over explicit use of Deferreds, just that caution needs to be taken in its use. gthreadless documentation to the effect of the above paragraphs might go some ways towards letting people use it correctly, too (though since the module is still only in Chris' sandbox, I don't expect him to go out and write this right away; conversely, if someone were to contribute documentation, this might help gthreadless get released, either as part of Twisted or in a stand-alone capacity). Jp
On 5/8/05, Jp Calderone <exarkun@divmod.com> wrote:
If you replace all occurences of "gthreadless" there with "greenlets in general", then you're correct. However, if one is talking about gthreadless, this argument is not true. All context switches still *are* obvious. Every asynchronous operation is still wrapped in a blockOn(), just like deferred generators. gthreadless is basically deferred generators with much nicer syntax. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On 5/8/05, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
I understand that... The Twisted Enterprise database api, if I understand correctly, uses the Twisted threadpool to wrap otherwise blocking database calls behind new calls that return deferreds. sqlobject, on the otherhand, was written in a traditional (non-callback) style. My thought was to use gthreadless to create an interface between Twisted Enterprise (ADBAPI?) and sqlobject. For sqlobject, it would look just like the blocking interface it was used to. But, it would be able to take advantage of the thread management already done by ADBAPI. Kevin
On Tue, 10 May 2005 08:27:27 -0400, Kevin Dangoor <dangoor@gmail.com> wrote:
SQLObject doesn't implement DB-API, so it is not usable with t.e.adbapi. The interface it presents is not very amenable to isolation in a separate thread, so any mechanism to using it asynchronously will be awkward. Jp
On 5/10/05, Kevin Dangoor <dangoor@gmail.com> wrote:
This won't work because gthreadless doesn't allow you to write interfaces that look synchronous. It only lets *you* use asynchronous interfaces in a synchronous-looking way. That is to say, your code that wants to be synchronous-looking still isn't 100% transparent; you need to call blockOn() around every Deferred you get. And a gthreadless-function always returns a Deferred itself, so there's really no way you could use the SQLObject code to talk to an asynchronous backend. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
Eric Mangold wrote:
I'm starting a new thread on this because It's very important to me, and we need to have some kind of resolution on this issue.
Is there a better way to synchronously spin the reactor until a deferred fires?
Yep. Write a unit test framework that uses Stackless and create a tasklet for each deferred that you want to run.
There any many cases in which I prefer to do that instead of returning a deferred from the test case method.
Without language support, this is an exercise in futility. Don't do it. Trial is broken, and everyone laments it, but the *reason* that it's broken (the root cause, anyway, if not the only reason) is that it bends over backwards to accomodate this use-case when _every_ _other_ _part_ of Twisted is very explicit that this is a bad, bad thing to do and you should never do it because it doesn't work. I am sorry that so many people want to block on deferreds in test methods, but I can guarantee you that support for doing it *WILL* be removed in a future version of Twisted. If we don't eventually kill this feature, our test suite is never going to work reliably.
On 5/7/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
I'd suggest 1) not writing a unit test framework, and instead using trial, and 2) instead of using stackless, use greenlet + my gthreadless module that integrates greenlets and Deferreds (sandbox/radix/gthreadless.py). Alternatively, if you don't want to use greenlet, as Bob said, just use deferred generators (twisted.internet.defer.{deferredGenerator,waitForDeferred}). HTH, HAND :-) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On 5/6/05, Christopher Armstrong <radeex@gmail.com> wrote:
gthreadless looks quite cool! From things I've seen along the way, it seems that many people are turned off by the callback style of programming required by an async framework like Twisted. Using gthreadless, your code looks just like normal synchronous code. I'm only a little familiar with greenlet. Is there any significant disadvantage to using this approach extensively? For example, is the overhead very large? Using gthreadless, one could interface sqlobject and Twisted's db interface to take advantage of the work done in Twisted Enterprise to make sqlobject-style database access safe for Twisted. (It's undoubtedly possible without gthreadless, but it seems like it would be a lot easier) Kevin
On Sat, 7 May 2005 11:02:09 -0400, Kevin Dangoor <dangoor@gmail.com> wrote:
The main advantage is also the main disadvantage: when using gthreadless, rather than function callbacks, it becomes less obvious where context switches occur, leading to an increased likelihood of introducing concurrency-related bugs (primarily due to switches out of non-atomic operations). Unlike pre-emptive threads, such programming errors can be avoided entirely without the use of locks, but doing so may require more skill and attention to detail than would be required when writing the same code without gthreadless. This is somewhat unfortunate, since programmers least familiar with the concurrency problems solved by Twisted's current mechanisms -- function callbacks -- seem to be the most likely to be attracted by solutions like gthreadless while at the same time being the least prepared to develop software correctly with them. Of course, this isn't to say gthreadless isn't useful or an improvement over explicit use of Deferreds, just that caution needs to be taken in its use. gthreadless documentation to the effect of the above paragraphs might go some ways towards letting people use it correctly, too (though since the module is still only in Chris' sandbox, I don't expect him to go out and write this right away; conversely, if someone were to contribute documentation, this might help gthreadless get released, either as part of Twisted or in a stand-alone capacity). Jp
On 5/8/05, Jp Calderone <exarkun@divmod.com> wrote:
If you replace all occurences of "gthreadless" there with "greenlets in general", then you're correct. However, if one is talking about gthreadless, this argument is not true. All context switches still *are* obvious. Every asynchronous operation is still wrapped in a blockOn(), just like deferred generators. gthreadless is basically deferred generators with much nicer syntax. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On 5/8/05, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
I understand that... The Twisted Enterprise database api, if I understand correctly, uses the Twisted threadpool to wrap otherwise blocking database calls behind new calls that return deferreds. sqlobject, on the otherhand, was written in a traditional (non-callback) style. My thought was to use gthreadless to create an interface between Twisted Enterprise (ADBAPI?) and sqlobject. For sqlobject, it would look just like the blocking interface it was used to. But, it would be able to take advantage of the thread management already done by ADBAPI. Kevin
On Tue, 10 May 2005 08:27:27 -0400, Kevin Dangoor <dangoor@gmail.com> wrote:
SQLObject doesn't implement DB-API, so it is not usable with t.e.adbapi. The interface it presents is not very amenable to isolation in a separate thread, so any mechanism to using it asynchronously will be awkward. Jp
On 5/10/05, Kevin Dangoor <dangoor@gmail.com> wrote:
This won't work because gthreadless doesn't allow you to write interfaces that look synchronous. It only lets *you* use asynchronous interfaces in a synchronous-looking way. That is to say, your code that wants to be synchronous-looking still isn't 100% transparent; you need to call blockOn() around every Deferred you get. And a gthreadless-function always returns a Deferred itself, so there's really no way you could use the SQLObject code to talk to an asynchronous backend. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
participants (6)
-
Christopher Armstrong -
Eric Mangold -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Jp Calderone -
Kevin Dangoor