[Twisted-Python] When do calls to transport.write() block ?
Hi everyone! I have looked through quite a bit of Twisted's code by now, but I just can't manage to figure out how those transport.write()s work. The servers I looked through always come back to this without putting any special treatment into it. My question is: If a server starts sending as fast as it can, what will happen to the output queue? Will Twisted just eat up all memory with buffers? Will it start blocking? Will it open up a thread? What will it do? Since this really is an important question, I'm sort of worried that it's not a big issue in the documentation... Thanks for any hints, Stefan
On Tue, 2004-05-11 at 11:12, Stefan Behnel wrote:
It will eat up all your memory. You should, therefore, not do that. Instead, use producers and consumers (this is e.g. how the web server sends large files): http://itamarst.org/writings/etech04/twisted_internet-101.html -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
Thank you for your quick answer! Itamar Shtull-Trauring wrote:
That's about the idea I had about it, but I couldn't quite grasp it. I'm curious though: wouldn't it be much more pythonic to allow users to register a generator with the transport? A generator usually does exactly what is expected from a pull-producer. Do you/does anyone know if that's foreseen somewhere in Twisted, either now or in a later version? Stefan
On Tue, 2004-05-11 at 11:49, Stefan Behnel wrote:
An iterator, you mean. Hmm, because even a pull producer may not have data on hand, you'd need to have a special object which means "I don't have data, sorry" that the iterator can return. And you'd still need API for push producers. OTOH it probably would be more efficient. What we were thinking of doing is getting rid of the concept of producers altogether and just exposing "buffers empty and writable" and "buffers full" callback events similar to dataReceived or connectionLost. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
Hi! Itamar Shtull-Trauring wrote:
That would be "None", I guess. What else could it possibly mean to write None to the transport?
And you'd still need API for push producers.
True. However, the registration methods could easily distinguish producers and iterators. So maybe this could be handled transparent for the API. It would not even break anything if the methods defaulted to using the object as a producer if it has the right methods and only check for iterator capabilities if it does not.
Sounds reasonable to me. Thanks again for your help, Stefan
On Tue, 2004-05-11 at 11:12, Stefan Behnel wrote:
It will eat up all your memory. You should, therefore, not do that. Instead, use producers and consumers (this is e.g. how the web server sends large files): http://itamarst.org/writings/etech04/twisted_internet-101.html -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
Thank you for your quick answer! Itamar Shtull-Trauring wrote:
That's about the idea I had about it, but I couldn't quite grasp it. I'm curious though: wouldn't it be much more pythonic to allow users to register a generator with the transport? A generator usually does exactly what is expected from a pull-producer. Do you/does anyone know if that's foreseen somewhere in Twisted, either now or in a later version? Stefan
On Tue, 2004-05-11 at 11:49, Stefan Behnel wrote:
An iterator, you mean. Hmm, because even a pull producer may not have data on hand, you'd need to have a special object which means "I don't have data, sorry" that the iterator can return. And you'd still need API for push producers. OTOH it probably would be more efficient. What we were thinking of doing is getting rid of the concept of producers altogether and just exposing "buffers empty and writable" and "buffers full" callback events similar to dataReceived or connectionLost. -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html
Hi! Itamar Shtull-Trauring wrote:
That would be "None", I guess. What else could it possibly mean to write None to the transport?
And you'd still need API for push producers.
True. However, the registration methods could easily distinguish producers and iterators. So maybe this could be handled transparent for the API. It would not even break anything if the methods defaulted to using the object as a producer if it has the right methods and only check for iterator capabilities if it does not.
Sounds reasonable to me. Thanks again for your help, Stefan
participants (3)
-
Bob Ippolito -
Itamar Shtull-Trauring -
Stefan Behnel