On 04:13 pm, markus@bluegap.ch wrote:

I have yet another question... a producer has to learn about it's consumer somehow, otherwise it would not know where to write it's data to.

Why is that? What's the best way to associate a producer with a consumer? Is something like the following okay:

This is a really deep problem, unfortunately, and Valentino has correctly identified that it's the same one I was going to talk about when responding to his twisted-web posting :). Your solution doesn't really solve anything, since neither producers nor consumers are documented to have the attributes you are assigning, nor do anything with them. Consumers are documented to have a "write" method, but the documentation is misleading. The most common type of producer - tcp.Connection - does not put data into its consumer's "write" method, it calls its protocol's dataReceived method. Many other producers are actually written specifically to call higher-level methods on something other than their consumer (such as callRemote) rather than "write". The basic use-case here is that the responsibility for producing bytes and the responsibility for throttling data production are usually separated in an application. I agree that this is not really the best way to separate them. The API grew up almost by accident, it wasn't really designed intentionally, although it has worked surprisingly well considering. In many cases it does seem that we want a high-level API for setting up pipelines of producer-to-consumer-to-producer, or (in the case of web2 streams) end-to-end producer-producer-producer sequences. While many of these use-cases are superficially similar, my own attempts to resolve this shows that there is a surprising diversity of requirements which are difficult to resolve simultaneously by "simply fixing" the existing consumer / producer mess. Keep in mind, also, that a great deal of code has been written (and will be written before everyone has upgraded to the next version of Twisted) that uses the current IConsumer/IProducer APIs, and all of that will need to keep working for the forseeable future. You can find some discussion, as well as our attempts to create something better, here: http://twistedmatrix.com/trac/ticket/1956 If you have new information, feel free to comment there, but please read all the comments and try to understand them first: there has been a *lot* of back-and-forth over this issue.

On 04:13 pm, markus@bluegap.ch wrote:

I have yet another question... a producer has to learn about it's consumer somehow, otherwise it would not know where to write it's data to.

Why is that? What's the best way to associate a producer with a consumer? Is something like the following okay:

This is a really deep problem, unfortunately, and Valentino has correctly identified that it's the same one I was going to talk about when responding to his twisted-web posting :). Your solution doesn't really solve anything, since neither producers nor consumers are documented to have the attributes you are assigning, nor do anything with them. Consumers are documented to have a "write" method, but the documentation is misleading. The most common type of producer - tcp.Connection - does not put data into its consumer's "write" method, it calls its protocol's dataReceived method. Many other producers are actually written specifically to call higher-level methods on something other than their consumer (such as callRemote) rather than "write". The basic use-case here is that the responsibility for producing bytes and the responsibility for throttling data production are usually separated in an application. I agree that this is not really the best way to separate them. The API grew up almost by accident, it wasn't really designed intentionally, although it has worked surprisingly well considering. In many cases it does seem that we want a high-level API for setting up pipelines of producer-to-consumer-to-producer, or (in the case of web2 streams) end-to-end producer-producer-producer sequences. While many of these use-cases are superficially similar, my own attempts to resolve this shows that there is a surprising diversity of requirements which are difficult to resolve simultaneously by "simply fixing" the existing consumer / producer mess. Keep in mind, also, that a great deal of code has been written (and will be written before everyone has upgraded to the next version of Twisted) that uses the current IConsumer/IProducer APIs, and all of that will need to keep working for the forseeable future. You can find some discussion, as well as our attempts to create something better, here: http://twistedmatrix.com/trac/ticket/1956 If you have new information, feel free to comment there, but please read all the comments and try to understand them first: there has been a *lot* of back-and-forth over this issue.