On Mon, Jun 23, 2003 at 12:51:40PM +0200, Matthias Urlichs wrote:
Hi, Jp Calderone wrote:
Hmm, this is the only part of the patch I am unsure about. The API seems a little too tuned to your use-case. I think the way to go for this would be to have a Producer passed in and make the NNTP protocol a Consumer for that (in turn acting as a Producer for its transport object). Would you be willing to make this change? (If you need an example of how this might work, check out smtp.py)
Hmm. Looking at the SMTP case, it needs refactoring in that the producer isn't decoupled from the file object it ends up reading, thus I can't plug a different kind of producer in there.
Yea, it isn't perfect.
Looking further, there's a slightly-broken data-from-file producer in doc/historic/2003/pycon/twisted-internet/twisted-internet.py ...
Anyway, I need NNTP to be a streaming protocol. With a Deferred, this is easy -- each Deferred gets a callback which writes TAKETHIS plus the message to the server (and queues the result code recognition), so these can complete in any order, which is very nice.
They can complete in any order because the code responsible for the Deferred's callback must buffer the message in memory.
A producer interface would instead have an arbitrary number of producers, all of which are active, but as soon as the first one returns anything I have to temporarily deactivate all the others. That sounds dangerous -- what if a producer can't be paused? Then I need to buffer their data, too, which is no different from the current case where the article data also end up in memory.
At the very least, the object that is currently a string should become a file object (This should have minimal impact on your code -- if buffering in memory is acceptable to you, you only have to build a StringIO object from your string), and the NNTP protocol class should consume it in a manner similar to that of smtp.py. It -does- seem that this is becoming a common task (smtp.py and pop3.py do it, imap4.py needs to do, and now nntp.py), so maybe I'll write a utility mixin to help out, and I'll see if I can generalize it away from just file-like objects too. This gives clients at least the opportunity to avoid loading entire messages into memory, which is what I am looking for. Am I right in thinking this leaves you with all the functionality you want? Jp -- "Minerals are inexhaustible and will never be depleted. A stream of investment creates additions to proved reserves, a very large in-ground inventory, constantly renewed as it is extracted... How much was in the ground at the start and how much will be left at the end are unknown and irrelevant." -- Morry Adelman, World Renowned Economist