[Twisted-Python] Clarification or pausing a consumer
I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to send often.
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Then when the next block of data is available what is the correct way to resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Barry
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott barry.scott@forcepoint.com wrote:
I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to send often.
This sounds like a bug in the consumer. There are two cases.
If the producer is "streaming" or "push" (two words, one meaning) then the consumer should call resumeProducing once and let data come. If it wants to slow down the data, it can call pauseProducing. There's nothing else it should be doing.
If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or switch to tubes which has a simpler model and (in principle) fewer places to make mistakes.
Then when the next block of data is available what is the correct way to resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Probably as an attempt to simplify the implementation - to make it so the consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone).
Jean-Paul
On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone exarkun@twistedmatrix.com wrote:
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <barry.scott@forcepoint.com mailto:barry.scott@forcepoint.com> wrote: I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to send often.
This sounds like a bug in the consumer. There are two cases.
If the producer is "streaming" or "push" (two words, one meaning) then the consumer should call resumeProducing once and let data come. If it wants to slow down the data, it can call pauseProducing. There's nothing else it should be doing.
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error to include it and it should be removed from Twisted eventually.
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or switch to tubes which has a simpler model and (in principle) fewer places to make mistakes.
My guess is not that the consumer is broken, but rather, you registered your producer as the bad kind of producer ("pull") and it's just doing as it was told, which is to keep calling resumeProducing over and over again every time it wants data.
Let us know how it works out!
Then when the next block of data is available what is the correct way to resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Probably as an attempt to simplify the implementation - to make it so the consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone).
Jean-Paul
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone exarkun@twistedmatrix.com wrote:
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <barry.scott@forcepoint.com mailto:barry.scott@forcepoint.com> wrote: I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to send often.
This sounds like a bug in the consumer. There are two cases.
If the producer is "streaming" or "push" (two words, one meaning) then the consumer should call resumeProducing once and let data come. If it wants to slow down the data, it can call pauseProducing. There's nothing else it should be doing.
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then the consumer should call resumeProducing and wait for a write call. Then it should call resumeProducing again (if it wants more data). It should not call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error to include it and it should be removed from Twisted eventually.
Oh. This is surprising to here what is the rational behind this "good" vs. "bad"?
My use case is that I'm a proxy and I can only produce what has been received from the other side.
A hard CPU loop calling resumeProducing is not appropiate for this use case.
I had to work around what looked like a bug that the streaming parameter is ignored and call unregisterProducer/registedProduced to get back usable behaviour.
Barry
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or switch to tubes which has a simpler model and (in principle) fewer places to make mistakes.
My guess is not that the consumer is broken, but rather, you registered your producer as the bad kind of producer ("pull") and it's just doing as it was told, which is to keep calling resumeProducing over and over again every time it wants data.
Let us know how it works out!
Then when the next block of data is available what is the correct way to resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Probably as an attempt to simplify the implementation - to make it so the consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone).
Jean-Paul
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott barry.scott@forcepoint.com wrote:
On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone exarkun@twistedmatrix.com wrote:
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
barry.scott@forcepoint.com
mailto:barry.scott@forcepoint.com> wrote: I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to
send
often.
This sounds like a bug in the consumer. There are two cases.
If the producer is "streaming" or "push" (two words, one meaning) then
the
consumer should call resumeProducing once and let data come. If it
wants
to slow down the data, it can call pauseProducing. There's nothing
else
it should be doing.
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then
the
consumer should call resumeProducing and wait for a write call. Then
it
should call resumeProducing again (if it wants more data). It should
not
call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error
to
include it and it should be removed from Twisted eventually.
Oh. This is surprising to here what is the rational behind this "good" vs. "bad"?
What makes it "bad" is that it creates extra work for every consumer implementation. The extra work is implementing largely the same thing every time so it's also redundant work. The logic for knowing when to call resumeProducing again is always exactly the same regardless of the consumer. It doesn't belong inside every consumer.
My use case is that I'm a proxy and I can only produce what has been received from the other side.
This sounds like it should be a push producer to me.
A hard CPU loop calling resumeProducing is not appropiate for this use case.
This is appropriate behavior for neither kind of producer. As I wrote in my first reply, a consumer with a pull ("not streaming", "bad") producer should call resumeProducing once and then wait until write is called on it. After write is called on it, it may call resumeProducing again if it wants.
Jean-Paul
I had to work around what looked like a bug that the streaming parameter is ignored and call unregisterProducer/registedProduced to get back usable behaviour.
Barry
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or
switch
to tubes which has a simpler model and (in principle) fewer places to make mistakes.
My guess is not that the consumer is broken, but rather, you registered
your
producer as the bad kind of producer ("pull") and it's just doing as it
was
told, which is to keep calling resumeProducing over and over again every time it wants data.
Let us know how it works out!
Then when the next block of data is available what is the correct way
to
resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Probably as an attempt to simplify the implementation - to make it so
the
consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone).
Jean-Paul
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Monday, 23 October 2017 17:31:19 BST Jean-Paul Calderone wrote:
On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott barry.scott@forcepoint.com
wrote:
On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone exarkun@twistedmatrix.com wrote:
On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
barry.scott@forcepoint.com
mailto:barry.scott@forcepoint.com> wrote: I have a app that needs to send data as an http response. But the data is produced slowly. More slowly then the time taken to transmit it.
The consumer keeps calling resumeProducing, but there is no data to
send
often.
This sounds like a bug in the consumer. There are two cases.
If the producer is "streaming" or "push" (two words, one meaning) then
the
consumer should call resumeProducing once and let data come. If it
wants
to slow down the data, it can call pauseProducing. There's nothing
else
it should be doing.
This is the good kind of producer.
If the producer is "not streaming" or "pull" (again, one meaning) then
the
consumer should call resumeProducing and wait for a write call. Then
it
should call resumeProducing again (if it wants more data). It should
not
call resumeProducing again while waiting for a write call.
This is the bad kind. It is basically pointless. It was a design error
to
include it and it should be removed from Twisted eventually.
Oh. This is surprising to here what is the rational behind this "good" vs. "bad"?
What makes it "bad" is that it creates extra work for every consumer implementation. The extra work is implementing largely the same thing every time so it's also redundant work. The logic for knowing when to call resumeProducing again is always exactly the same regardless of the consumer. It doesn't belong inside every consumer.
o.k. I see. So you want to have the consumer logic implemented once and reused as needed?
My use case is that I'm a proxy and I can only produce what has been received from the other side.
This sounds like it should be a push producer to me.
Yes it is.
A hard CPU loop calling resumeProducing is not appropiate for this use case.
This is appropriate behavior for neither kind of producer. As I wrote in my first reply, a consumer with a pull ("not streaming", "bad") producer should call resumeProducing once and then wait until write is called on it. After write is called on it, it may call resumeProducing again if it wants.
But that is what happens is you convert a push to a pull which is why I noticed this.
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
It sure looks like a bug to give me that API that is the shape to take push or pull but inside then force into pull.
Barry
Jean-Paul
I had to work around what looked like a bug that the streaming parameter is ignored and call unregisterProducer/registedProduced to get back usable behaviour.
Barry
What is the correct way to pause the consumer so that it does not keep calling resumeProducing? unregisterProducer?
Probably just fix the consumer implementation to not be broken. Or
switch
to tubes which has a simpler model and (in principle) fewer places to make mistakes.
My guess is not that the consumer is broken, but rather, you registered
your
producer as the bad kind of producer ("pull") and it's just doing as it
was
told, which is to keep calling resumeProducing over and over again every time it wants data.
Let us know how it works out!
Then when the next block of data is available what is the correct way
to
resume the consumer? registerProducer?
Related to that why does HTTPChannel.registerProducer convert a IPullProducer into a IPushProducer using _PullToPush?
Probably as an attempt to simplify the implementation - to make it so
the
consumer can pretend there's just one kind of producer instead of two (which would be nice for everyone).
Jean-Paul
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Scanned by Forcepoint Email Security Gateway
To report this email as SPAM, please forward it to spam@forcepoint.com
participants (3)
-
Barry Scott -
Glyph -
Jean-Paul Calderone