Hi, Jp Calderone wrote:
They can complete in any order because the code responsible for the Deferred's callback must buffer the message in memory.
Nope. Converting the code to assemble the articles in file-like objects, or whatever, instead, wouldn't make much of a difference to my current code -- they would still be able to complete in any order. Part of the niceness of streaming CHECK and TAKETHIS messages to an NNTP server is that there is no ordering required; you can't do that with SMTP. On the other hand, thinking about this I just had an idea how to manage the transition from a bunch of parallel Deferreds to a serialized producer/consumer scheme (addCallback() a code snipped which enqueues the Deferred onto a queue, and take it from there one-by-one). It's still going to start out as a Deferred, though; "a producer which needs to do something special when it first delivers data and which might trigger an error instead" is complicated code which lumps at least three concepts into one piece of code, while using a Deferred in the first step leads to nicely separated stages with distinct code -- always a win in my book.
At the very least, the object that is currently a string should become a file object (This should have minimal impact on your code -- if buffering in memory is acceptable to you, you only have to build a StringIO object from your string)
That would be no problem. Whether the Deferred returns an in-memory string or a file object doesn't make a whole lot of difference, esp. as a simple 'if not hasattr(f,"read"): f = StringIO.StringIO(f)' is sufficient for turning one into the other.
It -does- seem that this is becoming a common task (smtp.py and pop3.py do it, imap4.py needs to do, and now nntp.py), so maybe I'll write a utility mixin to help out, and I'll see if I can generalize it away from just file-like objects too.
That would be nice to have. I'd write it as an adapter or similar, i.e. a class which turns an LF-based dots-ignorant possibly-without-final-LF producer into one which has CRLFs, dot-escapes, and a guaranteed CRLF at the end. That way it'd be useable for things which aren't a streaming protocol. Direct back-end access to the NNTP storage comes to mind; it stores articles in the latter format. -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de -- There will always be some delightful mysteries in your life.