[Twisted-Python] Clarification or pausing a consumer
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it. The consumer keeps calling resumeProducing, but there is no data to send often. What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer? Then when the next block of data is available what is the correct way to resume the consumer? registerProducer? Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush? Barry
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
This sounds like a bug in the consumer. There are two cases. If the producer is "streaming" or "push" (two words, one meaning) then the consumer should call resumeProducing once and let data come. If it wants to slow down the data, it can call pauseProducing. There's nothing else it should be doing. If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or switch to tubes which has a simpler model and (in principle) fewer places to make mistakes.
Probably as an attempt to simplify the implementation - to make it so the consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone). Jean-Paul
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error to include it and it should be removed from Twisted eventually.
My guess is not that the consumer is broken, but rather, you registered your producer as the bad kind of producer ("pull") and it's just doing as it was told, which is to keep calling resumeProducing over and over again every time it wants data. Let us know how it works out!
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
Oh. This is surprising to here what is the rational behind this "good" vs. "bad"? My use case is that I'm a proxy and I can only produce what has been received from the other side. A hard CPU loop calling resumeProducing is not appropiate for this use case. I had to work around what looked like a bug that the streaming parameter is ignored and call unregisterProducer/registedProduced to get back usable behaviour. Barry
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
What makes it "bad" is that it creates extra work for every consumer implementation. The extra work is implementing largely the same thing every time so it's also redundant work. The logic for knowing when to call resumeProducing again is always exactly the same regardless of the consumer. It doesn't belong inside every consumer.
This sounds like it should be a push producer to me.
A hard CPU loop calling resumeProducing is not appropiate for this use case.
This is appropriate behavior for neither kind of producer. As I wrote in my first reply, a consumer with a pull ("not streaming", "bad") producer should call resumeProducing once and then wait until write is called on it. After write is called on it, it may call resumeProducing again if it wants. Jean-Paul
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 23 October 2017 17:31:19 BST Jean-Paul Calderone wrote:
o.k. I see. So you want to have the consumer logic implemented once and reused as needed?
Yes it is.
But that is what happens is you convert a push to a pull which is why I noticed this.
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
It sure looks like a bug to give me that API that is the shape to take push or pull but inside then force into pull. Barry
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
This sounds like a bug in the consumer. There are two cases. If the producer is "streaming" or "push" (two words, one meaning) then the consumer should call resumeProducing once and let data come. If it wants to slow down the data, it can call pauseProducing. There's nothing else it should be doing. If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or switch to tubes which has a simpler model and (in principle) fewer places to make mistakes.
Probably as an attempt to simplify the implementation - to make it so the consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone). Jean-Paul
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error to include it and it should be removed from Twisted eventually.
My guess is not that the consumer is broken, but rather, you registered your producer as the bad kind of producer ("pull") and it's just doing as it was told, which is to keep calling resumeProducing over and over again every time it wants data. Let us know how it works out!
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
Oh. This is surprising to here what is the rational behind this "good" vs. "bad"? My use case is that I'm a proxy and I can only produce what has been received from the other side. A hard CPU loop calling resumeProducing is not appropiate for this use case. I had to work around what looked like a bug that the streaming parameter is ignored and call unregisterProducer/registedProduced to get back usable behaviour. Barry
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.scott@forcepoint.com> wrote:
What makes it "bad" is that it creates extra work for every consumer implementation. The extra work is implementing largely the same thing every time so it's also redundant work. The logic for knowing when to call resumeProducing again is always exactly the same regardless of the consumer. It doesn't belong inside every consumer.
This sounds like it should be a push producer to me.
A hard CPU loop calling resumeProducing is not appropiate for this use case.
This is appropriate behavior for neither kind of producer. As I wrote in my first reply, a consumer with a pull ("not streaming", "bad") producer should call resumeProducing once and then wait until write is called on it. After write is called on it, it may call resumeProducing again if it wants. Jean-Paul
![](https://secure.gravatar.com/avatar/cf223b7cf77583c0a2665bad01f84f11.jpg?s=120&d=mm&r=g)
On Monday, 23 October 2017 17:31:19 BST Jean-Paul Calderone wrote:
o.k. I see. So you want to have the consumer logic implemented once and reused as needed?
Yes it is.
But that is what happens is you convert a push to a pull which is why I noticed this.
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
It sure looks like a bug to give me that API that is the shape to take push or pull but inside then force into pull. Barry
participants (3)
-
Barry Scott
-
Glyph
-
Jean-Paul Calderone