Web2 and Half-Baked Reinvention of Wheels
I have been debugging an odd problem that apparently is arising in web2.stream.FileStream. One thing I noticed while poring over the source is that its read() method will try to instantiate SendfileBuffer if you try to use its sendfile option. However, there's no reference To such a class anywhere in the module or anywhere else in the Twisted source tree, for that matter. I understand that sendfile isn't supported, but it's horrible coding practice to let that just sit there in the source. As requested by Dialtone, and to borrow liberally from what he wrote in an IRC chat with me today, web2.stream code is pretty badly tested and someone should do something about it. (I would also add that it is pretty badly documented, but that problem is widespread in Twisted's source, despite my ranting about it for years and offering bounties for people to contribute docstrings.) Dialtone says that there are numerous other problems about streams. He says, and I wholeheartedly agree, that they are mostly a substitution for something that already exists in twisted, namely Producer/Consumer. (Please guys, stop chasing butterflies and direct your considerable talents toward improving what already works!) Dialtone says that glyph once wrote a replacement for the CompundStream using Producer/Consumer paradigm; instead of streams, we could use functions that combine different producers or consumers. I can't believe that the world's greatest networking platform *still* doesn't have an adequately supported HTTP server! Best regards, Ed
Hi Ed, On Mon, 02 Jul 2007 17:21:13 -0500, Ed Suominen <general@eepatents.com> wrote:
I have been debugging an odd problem that apparently is arising in web2.stream.FileStream. One thing I noticed while poring over the source is that its read() method will try to instantiate SendfileBuffer if you try to use its sendfile option. However, there's no reference To such a class anywhere in the module or anywhere else in the Twisted source tree, for that matter. I understand that sendfile isn't supported, but it's horrible coding practice to let that just sit there in the source.
I know you are a regular in these parts, but isn't your tone a trifle accusatory? I can't speak for any of the core devs, but posts like this don't tend to motivate people to fix code; rather, I find they tend to lead to people like myself responding about the tone of the post ;)
As requested by Dialtone, and to borrow liberally from what he wrote in an IRC chat with me today, web2.stream code is pretty badly tested and someone should do something about it. (I would also add that it is pretty badly documented, but that problem is widespread in Twisted's source, despite my ranting about it for years and offering bounties for people to contribute docstrings.)
OK, I'll grant that it is a shame nobody has taken you up on the bounties, but are you really surprised that "ranting about it for years" hasn't produced much of a result?
Dialtone says that there are numerous other problems about streams. He says, and I wholeheartedly agree, that they are mostly a substitution for something that already exists in twisted, namely Producer/Consumer. (Please guys, stop chasing butterflies and direct your considerable talents toward improving what already works!) Dialtone says that glyph once wrote a replacement for the CompundStream using Producer/Consumer paradigm; instead of streams, we could use functions that combine different producers or consumers.
Right, I vaguely recall a sandbox module, called consprod.py, which was supposed to be a brief sketch of how to implement HTTP in the consumer/producer vein. It definitely wasn't a replacement, just a proof of concept, with no tests or docs. What is it that makes you think that twisted devs are, as you put it, "chasing butterflies"? Specifically, what code should the twisted devs be improving instead of web2?
I can't believe that the world's greatest networking platform *still* doesn't have an adequately supported HTTP server!
This is overly dramatic, to be sure. The supported HTTP server is twisted.web, and it receives bugfixes when needed. There is a mega-ticket describing what needs to be done in order for web2 to fully replace twisted.web. That this work hasn't been completed has to do with developer priorities, and the fact that twisted.web, along with nevow, is perfectly adequate for all sorts of production websites, and has been for some time. I know you've produced plenty of code yourself, so I'm really curious as to why you'd adopt such a confrontational tone, when you must realize that such tactics seldom produce the result you desire. L. Daniel Burr
On Jul 2, 2007, at 3:21 PM, Ed Suominen wrote:
I have been debugging an odd problem that apparently is arising in web2.stream.FileStream. One thing I noticed while poring over the source is that its read() method will try to instantiate SendfileBuffer if you try to use its sendfile option. However, there's no reference To such a class anywhere in the module or anywhere else in the Twisted source tree, for that matter. I understand that sendfile isn't supported, but it's horrible coding practice to let that just sit there in the source.
There is a bug, please feel free to contribute a patch. For a while now web2 has been kind of a solitary effort, and I just don't have the energy right now, but I will gladly review and apply a patch.
As requested by Dialtone, and to borrow liberally from what he wrote in an IRC chat with me today, web2.stream code is pretty badly tested and someone should do something about it. (I would also add that it is pretty badly documented, but that problem is widespread in Twisted's source, despite my ranting about it for years and offering bounties for people to contribute docstrings.)
It is conceptually pretty bad, and counter to existing abstractions in Twisted. I believe there is a bug about this, and even a branch that foom started, I do not know the status of this branch, nor the quality of the code that lies within it. Feel free to review the branch, and post your findings in the associated ticket.
Dialtone says that there are numerous other problems about streams. He says, and I wholeheartedly agree, that they are mostly a substitution for something that already exists in twisted, namely Producer/ Consumer. (Please guys, stop chasing butterflies and direct your considerable talents toward improving what already works!) Dialtone says that glyph once wrote a replacement for the CompundStream using Producer/Consumer paradigm; instead of streams, we could use functions that combine different producers or consumers.
Right, that would be awesome and I fully support completely replacing streams with something based on the more traditional Producer/ Consumer model.
I can't believe that the world's greatest networking platform *still* doesn't have an adequately supported HTTP server!
I'm sorry. Despite much interest in web programming (in the python community,) there is a definite lack of interest in writing a proper HTTP server. I really do love working on web2, and I'm sorry I haven't been able to give it the attention it deserves, but such is life. I would welcome any contribution you're willing to make (though I am probably unable to directly accept a financial contribution.) -David
On 03:06 am, dreid@dreid.org wrote:
On Jul 2, 2007, at 3:21 PM, Ed Suominen wrote:
I have been debugging an odd problem that apparently is arising in web2.stream.FileStream. (...)
There is a bug, please feel free to contribute a patch.
http://twistedmatrix.com/trac/ticket/2296
As requested by Dialtone, and to borrow liberally from what he wrote in an IRC chat with me today, web2.stream code is pretty badly tested (...)
It is conceptually pretty bad, and counter to existing abstractions in Twisted. I believe there is a bug about this, (...)
http://twistedmatrix.com/trac/ticket/1937
I can't believe that the world's greatest networking platform *still* doesn't have an adequately supported HTTP server!
http://twistedmatrix.com/trac/milestone/Web2-Gold-Master
I'm sorry.
While this is unfortunate, I don't think that there's really any reason to apologize. This is a volunteer effort all around, and it's as much my fault (and Ed's) that such a server doesn't exist. In fact, David, if there are any people whose fault it *isn't*, it would be you and dialtone. Thank you for your sustained (albeit intermittent) efforts to get this done. (Of course, the bears and I reserve the right to hassle you mercilessly to complete it.) As I've annotated here, the issues that Ed outlined in his email are pre-existing, with tickets describing them. If anyone out there shares his frustration and would like to help us along towards a supported web2 release, or any other part of Twisted for that matter, just have a look at the Twisted tracker and steal a ticket from some of the obviously overcommitted and overworked maintainers. The milestone link I posted is the best start for web2. Many of those tickets are underspecified, but don't let that stop you. If you find a ticket that you think sounds interesting but you don't understand how to proceed on it, feel free to comment and ask for clarification - it is often far easier for a maintainer to respond to a prompt for some answers to questions than to drive tests, documentation, and code all the way through the review process.
On Tue, 03 Jul 2007 19:52:06 -0000, glyph@divmod.com wrote:
If anyone out there shares his frustration and would like to help us along towards a supported web2 release, or any other part of Twisted for that matter, just have a look at the Twisted tracker and steal a ticket from some of the obviously overcommitted and overworked maintainers. The milestone link I posted is the best start for web2.
I'm already fixing some bugs with the current code as part of the SoC, and I've currently fixed 4 or 5 of them (the last one waiting for review in branch 1895). I'm currently starting, or I'm hoping to start, work on no-stream-1937 branch...
Many of those tickets are underspecified, but don't let that stop you. If you find a ticket that you think sounds interesting but you don't understand how to proceed on it, feel free to comment and ask for clarification - it is often far easier for a maintainer to respond to a prompt for some answers to questions than to drive tests, documentation, and code all the way through the review process.
Since you kindly offered for clarifications: I see one problem with the current IConsumer/IProducer API and it's returning deferreds when a producer finished its job. The current IConsumer API uses registerProducer() but doesn't allow to return anything from the method, and in fact this is what the transport does. Unfortunately it would be useful to deal with the finish event of producing to a consumer for many reasons, like you don't know in advance when the produced content will finish. In Twisted this problem was solved, partially, by FileSender which uses a beginFileTransfer(fobj, consumer) method to return a deferred that fires at the end of the file object. Unfortunately this is not really a reliable method since it's not part of the IProducer interface and any producer might use whatever method they like and, worst of all, I'd anyway need to rewrite or add a method to FileSender because I can't really use beginFileTransfer() for a MemoryBufferProducer() object. One of the needs is the chance to chain Producers/Consumers to do input filtering and output filtering. While the first could also be avoided (meaning that the current implementation is good enough IMHO), output filtering is actually pretty cool to have since it allows the user to dynamically add behavior to a request's response like it's currently done. Taking from JP's consumer-sketch.py code: def _chain(hook, chainedElements): assert len(chainedElements) >= 2 elements = iter(chainedElements) producer = elements.next() if isinstance(producer, Deferred): producer = yield producer for ele in elements: if isinstance(ele, Deferred): ele = yield ele finishedProducing = hook(producer, ele) producer = ele yield finishedProducing _chain = defgen(_chain) def forwardHook(producer, consumer): return producer.produceTo(consumer) def chainOneWay(*x): """ chainOneWay(producer,[chainedElement1,...,chainedElementN],consumer)->Deferred Connect a producer to a consumer/producer to a consumer/producer to a ... ETC For example:: a = FileSender('filename') b = GzipCompressor() c = ChunkEncoder() d = TCPTransport() finished = chainOneWay(a, b, c, d) def cbSentChunkedGzippedFile(ign): print 'Yaaaaaaaaaaaaay' d.loseConnection() finished.addCallback(cbSentChunkedGzippedFile) Will do the obvious thing. """ return _chain(forwardHook, x) This would be a really cool way to solve the problem and would also present a pretty clean and simple API. My question then is: how should I proceed? Clearly using this new API is hard because everything else uses the current one, but the current one lacks a fairly important functionality. Providing adapter in web2 for current producer/consumer stuff might seem overkill and not very clean but might be a solution. Introducing this new API in Twisted would be a nightmare. Is there any way to implement this using the current IProducer/IConsumer API? I seek advice and some code examples if anyone is willing to spend his mind on the problem a bit.
participants (5)
-
David Reid -
Ed Suominen -
glyph＠divmod.com -
L. Daniel Burr -
Valentino Volonghi aka Dialtone