On Tue, Aug 19, 2008 at 6:59 AM, Henrik Thostrup Jensen wrote:

Hi

I've written a small python/twisted client library for CouchDB.

It's mostly a toy, but it can be fetched from here: http://www.cs.aau.dk/~htj/code/couched-0.1.tar.gz if anyone is interested. License: MIT

I'm aware of paisley (https://launchpad.net/paisley), The main difference from paisley is that the connection and database abstraction are separated. Also couched does connection caching and uses web2 instead of web. Currently paisley is more feature rich, but couched has slightly nicer error handling.

Hey Henrik, Thanks for sharing! I would, however, encourage you to create a Paisley branch in Launchpad and improve the error handling and connection abstraction there. This would make it much easier for us to merge your changes into Paisley. At the very least, you may want to register your project with Launchpad and associate it with the tx project group, so people can find it easily (the tx project group, or "super project" is for Twisted community code). Also, are you aware that web2 features are being slowly ported to web, and that twisted.web will continue to be the official web library? Thanks again! d

On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen wrote:

[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).

Which other HTTP 1.1 features do you need? Jean-Paul

2008/9/5 Jean-Paul Calderone

On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen < thostrup@gmail.com> wrote:

...

[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).

Which other HTTP 1.1 features do you need?

HTTP 1.1 status codes, notable 303 (see other). I could probably use 302, but 303 is the "correct" behaviour for my case. This should be relatively easy to implement. Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy. I'm not saying streams are the answer but it works for me (actually I use the producer/consumer interface in my lower level code and then adapt it in the web layer). Unfortunately I've had to rewrite a lot of the producer/consumer code and stream stuff and glue it together, as some of the stuff in twisted (not just in web2) is rather broken. For complete insanity (sorry to whoever wrote it ) see twisted.protocols.basic.FileSender, which implements its own magic method beginFileTransfer, which takes BOTH a file and a consumer as argument. This makes is impossible to seperate the construction of the producer and starting the transfer. Ticket #1956 fixes some of these things. I'm not quite sure how it handles errors (which can happen both ways), but I've just had a quick look. It's been a whlie since i've had a look at all this. I write a bit about it, which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt IMHO, getting (and deciding on) a proper producer/consumer interface need to happen, to make whatever web things twisted will have usefull. Changing the producer/consumer might be massive API break, however it is a needed one. There are simply things the current API cannot do (i usually return a deferred along with any producer/consumer object to indiciate how the data moving went). I might be interested in participating in this, however I'm rather work struck as I have to hand my ph.d. thesis at the end of this year. Btw. I really am a huge fan of twisted, however the web and producer/consumer is a rather dark spot in an otherwise bright world :-) -- - Henrik

On 5 Sep, 11:25 pm, thostrup@gmail.com wrote:

2008/9/5 Jean-Paul Calderone

...

On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen < thostrup@gmail.com> wrote:

...

[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).

Please feel free to use twisted.web2 if it is meeting your needs currently. However, our goal remains to focus maintenance effort on twisted.web and provide a gradual, compatible evolution to a better webserver. The HTTP 1.1 features that you need will eventually be added to twisted.web, and I hope that the split with paisley can be resolved at that point. I personally don't use couchdb or have any interest in its bindings, but it sounds like you have some other misconceptions about twisted and its development process which I'd like to correct.

...

Which other HTTP 1.1 features do you need?

Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy.

640k limitation? In twisted.web? As far as I know there's no such thing; the only 640k limitation I know of anywhere in Twisted is in PB, and that's just on individual message size. A brief skim through the twisted.web client and server code doesn't reveal any obvious size limitations that I can see. (Is there some 640k limitation in the 'cgi' module?) If there is such a limitation it should be more clearly documented somewhere. True, twisted.web doesn't provide a graceful way for resources to handle streaming file uploads as they arrive, but for dedicated servers it is certainly *possible*; Request.handleContentChunk says it's "not for users", but you could easily write a few simple patches which would make it "for users", or just subclass it and ignore the docstring. Nothing in twisted.web is really that hard to fix.

Ticket #1956 fixes some of these things. I'm not quite sure how it handles errors (which can happen both ways), but I've just had a quick look. It's been a whlie since i've had a look at all this. I write a bit about it, which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt

Ticket #1956 is still kind of stuck defining the problem. If you have comments or thoughts on #1956, please feel free to add them as comments. Text files lying around the web are not as helpful as discussions on tickets. Your text file suffers from the same problems as the existing discussion on the ticket, though: a lack of concrete use-cases and example code. There are lots of proposed solutions but not a lot of proposed problems. (I am certainly a guilty party here, I really should describe the problem in more detail.) We all feel there is some inelegance, but there is no example of some code that we can all agree is gross and would be fixed by some better API. Please come up with an example - the shorter the better - of a deficiency in the streaming API and put it on the ticket. Hopefully that will further the discussion there.

IMHO, getting (and deciding on) a proper producer/consumer interface need to happen, to make whatever web things twisted will have usefull. Changing the producer/consumer might be massive API break, however it is a needed one.

No. 1. We will never break compatibility with an API this old and this stable. When we do start to move over to a new API, the transition will be particularly gradual and slow given the core nature of this API and the fact that pretty much everything uses it. Even if it were only around for one release, see http://twistedmatrix.com/trac/wiki/CompatibilityPolicy 2. It's not clear what the new API would be. Every suggestion proposed thus far has some issues, and there's no clear winner. Unless someone takes the time to really thoroughly explain the problem and describe what effect the solution would have, with examples, then we're not going to make any progress. 3. There isn't anything that you *can't do* with ITransport/IProducer/IConsumer/IProtocol. It can be slightly awkward sometimes, but I have yet to see an example where you really can't make something work. For example, IConsumer doesn't have .finish(), but that's fine, because you probably mean IProtocol.connectionLost(...) anyway. Some users of these interfaces don't provide nice ways to get data in and out, and yes, a uniform and consistent way to deal with slinging data between endpoints (for example, a standard, featureful object to remix a 'protocol' into a 'transport' and vice versa) would probably help with that. Given that the deficiency creates momentary inconveniences rather than impossibilities, there's no way we're going to break *every single twisted program ever* just to make it slightly more convenient. 4. Given that the semantics are already there, it will be very easy to evolve new code to call old interfaces, provide adapters which wrap around things and generally make it easy to have some hypothetical shiny new interface cooperate just fine with the crufty old ones.