On 5 Sep, 11:25 pm, thostrup@gmail.com wrote:
2008/9/5 Jean-Paul Calderone <exarkun@divmod.com>
On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen < thostrup@gmail.com> wrote:
[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).
Please feel free to use twisted.web2 if it is meeting your needs currently. However, our goal remains to focus maintenance effort on twisted.web and provide a gradual, compatible evolution to a better webserver. The HTTP 1.1 features that you need will eventually be added to twisted.web, and I hope that the split with paisley can be resolved at that point. I personally don't use couchdb or have any interest in its bindings, but it sounds like you have some other misconceptions about twisted and its development process which I'd like to correct.
Which other HTTP 1.1 features do you need?
Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy.
640k limitation? In twisted.web? As far as I know there's no such thing; the only 640k limitation I know of anywhere in Twisted is in PB, and that's just on individual message size. A brief skim through the twisted.web client and server code doesn't reveal any obvious size limitations that I can see. (Is there some 640k limitation in the 'cgi' module?) If there is such a limitation it should be more clearly documented somewhere. True, twisted.web doesn't provide a graceful way for resources to handle streaming file uploads as they arrive, but for dedicated servers it is certainly *possible*; Request.handleContentChunk says it's "not for users", but you could easily write a few simple patches which would make it "for users", or just subclass it and ignore the docstring. Nothing in twisted.web is really that hard to fix.
Ticket #1956 fixes some of these things. I'm not quite sure how it handles errors (which can happen both ways), but I've just had a quick look. It's been a whlie since i've had a look at all this. I write a bit about it, which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt
Ticket #1956 is still kind of stuck defining the problem. If you have comments or thoughts on #1956, please feel free to add them as comments. Text files lying around the web are not as helpful as discussions on tickets. Your text file suffers from the same problems as the existing discussion on the ticket, though: a lack of concrete use-cases and example code. There are lots of proposed solutions but not a lot of proposed problems. (I am certainly a guilty party here, I really should describe the problem in more detail.) We all feel there is some inelegance, but there is no example of some code that we can all agree is gross and would be fixed by some better API. Please come up with an example - the shorter the better - of a deficiency in the streaming API and put it on the ticket. Hopefully that will further the discussion there.
IMHO, getting (and deciding on) a proper producer/consumer interface need to happen, to make whatever web things twisted will have usefull. Changing the producer/consumer might be massive API break, however it is a needed one.
No. 1. We will never break compatibility with an API this old and this stable. When we do start to move over to a new API, the transition will be particularly gradual and slow given the core nature of this API and the fact that pretty much everything uses it. Even if it were only around for one release, see http://twistedmatrix.com/trac/wiki/CompatibilityPolicy 2. It's not clear what the new API would be. Every suggestion proposed thus far has some issues, and there's no clear winner. Unless someone takes the time to really thoroughly explain the problem and describe what effect the solution would have, with examples, then we're not going to make any progress. 3. There isn't anything that you *can't do* with ITransport/IProducer/IConsumer/IProtocol. It can be slightly awkward sometimes, but I have yet to see an example where you really can't make something work. For example, IConsumer doesn't have .finish(), but that's fine, because you probably mean IProtocol.connectionLost(...) anyway. Some users of these interfaces don't provide nice ways to get data in and out, and yes, a uniform and consistent way to deal with slinging data between endpoints (for example, a standard, featureful object to remix a 'protocol' into a 'transport' and vice versa) would probably help with that. Given that the deficiency creates momentary inconveniences rather than impossibilities, there's no way we're going to break *every single twisted program ever* just to make it slightly more convenient. 4. Given that the semantics are already there, it will be very easy to evolve new code to call old interfaces, provide adapters which wrap around things and generally make it easy to have some hypothetical shiny new interface cooperate just fine with the crufty old ones.