Re: [Twisted-Python] Really Basic clarification on defers

A point I missed out on: The adbapi module seems to be a good example of using deferreds and threads. The adbapi module returns a deferred it has created, you add your callbacks to it. It calls your callback when ready. It does seem like the examplar for doing deferreds. The db stuff will normally block so put it in a thread and use deferreds to get the result or failure. A point about the db calls is that they can be very intensive. If you need to run some db calls every 30 secs or 60 secs and the db takes 50% or more of the time to generate the results, you won't have much time to service any incoming requests to see the data results. The remote connections will be failing bigtime. So then I suppose you should break the code into 2 programs. One that does the db stuff, the other to handle the remote connections. The db code when it has a result will then connect to the other program and pass across its results. There may be better ways of doing this of course. So as Jarrod points out, deferToThread is an easy way of solving blocking code, but not always. It seems good for short blocks, but you do need the bulk of the time available for handling connections. If anyone sees howlers in this, please let me know. Regards John Aherne On Wed, Aug 5, 2009 at 11:04 PM, John Aherne wrote:

On Wed, Aug 5, 2009 at 4:17 PM, wrote:

...

On 09:33 am, johnaherne@rocs.co.uk wrote:

...

On Wed, Aug 5, 2009 at 12:14 AM, Johann Borck wrote:

...

[snip]

Sendline is not blocking so as you say we can avoid the use of deferreds and continue to use sendline directly.

LineReceiver.sendLine is not blocking, correct. However, your statement implies that if it were blocking, you could use Deferreds to address this problem. This is incorrect. Deferreds do not make blocking APIs into non-blocking APIs.

...

[snip]

Our option 3 using defertothread does use sendline from the thread. Your response implies that is OK since you say defertothread is threadsafe. Did you really mean that.

deferToThread is not thread-safe: you may only call it from the reactor thread (the thread in which you called reactor.run).

Since deferToThread runs the function you pass to it in a non-reactor thread, you may not use any non-thread-safe Twisted APIs in the function you pass to it.

...

Once again thanks for a very good response. That has cleared up a lot of confusion.

I suppose it would help if there was a paragraph at the start of the twisted documentation detailing what you have just said. So when they start on deferreds you have some sort of context in which to interpret what is being said

A significant effort is presently underway to improve the documentation about Deferreds. Any specific feedback you have about it would be much appreciated. :)

Jean-Paul

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Jean-Paul

Thanks for the clarifications.

I assume what I meant by blocking was I would have to put a function into deferToThread and add some callbacks to return the result. And not directly calling sendline from the thread. This is what Jarrod suggested in his reply. If I'm wrong, please say so.

I have read the thread regarding deferreds with interest, but did not feel I knew enough to contribute.

I do feel qualified to ask some very daft questions. Unfortunately, I don't see too many daft questions being asked in the list. I reckon if I need to know the answer to them then some other people probably do as well but don't put themselves forward. I see it as a way to document information that is difficult to come by. And I really do appreciate the very good answers I get

However, after this little foray, I probably feel able to comment:--

The concept of deferreds is very simple. Everyone understands the concept - even I do. The issue is how and why and where you should use them in twisted.

Some Basic getting Started Points.

1. For simple network activity do not use deferreds. They are not necessary. You can get a lot done without deferreds. And you don't know how to use them yet. The reactor and the select will process the outgoing and incoming buffers without blocking. Anyone familiar with networking and select will already understand this. Anyone not familiar will not realise it and needs to be made aware of how the select works.

2. If you have blocking code - please define blocking :), then first think about putting it into deferToThread with appropriate callbacks and return the deferred. As suggested by Jarrod in his response.

3. John Goerzen in his Apress book Python Network Fundamentals has a very simple chat server example. With a few comments for the uninitiated, this would be a good starter. Possibly I could ask for permission to include it in some twisted HOWTO documentation for beginners with suitable copyright recognition.

4. With these few points as starters, maybe more people will be encouraged to get started with twisted. And if you know you can ignore deferreds until later you will find twisted is very simple to use and get some good results with little effort.

5. The emphasis on how deferreds work probably ought to be counterbalanced by some insight into how and why and where you would use them. For example, if you have a text file of 10000 lines you need to read in and summarize, presumably you would run this with deferToThread(+other options) and get the result via the callback. If someone has a better example please let me know.

6. Blocking code is always put into a thread or like, and a deferred callback or errback used to return the result or failure of the blocking code from the thread. See jarrod's response above

You asked :) I'm giving

I may be completely off track in what I have said above. And I would not want anyone to fall upon this mail and think it represents the gospel truth.

Thank you for the response..

John Aherne