[Twisted-Python] Couched 0.1: CouchDB twisted client
Hi I've written a small python/twisted client library for CouchDB. It's mostly a toy, but it can be fetched from here: http://www.cs.aau.dk/~htj/code/couched-0.1.tar.gz if anyone is interested. License: MIT I'm aware of paisley (https://launchpad.net/paisley), The main difference from paisley is that the connection and database abstraction are separated. Also couched does connection caching and uses web2 instead of web. Currently paisley is more feature rich, but couched has slightly nicer error handling. -- - Henrik
On Tue, Aug 19, 2008 at 6:59 AM, Henrik Thostrup Jensen <thostrup@gmail.com> wrote:
Hi
I've written a small python/twisted client library for CouchDB.
It's mostly a toy, but it can be fetched from here: http://www.cs.aau.dk/~htj/code/couched-0.1.tar.gz if anyone is interested. License: MIT
I'm aware of paisley (https://launchpad.net/paisley), The main difference from paisley is that the connection and database abstraction are separated. Also couched does connection caching and uses web2 instead of web. Currently paisley is more feature rich, but couched has slightly nicer error handling.
Hey Henrik, Thanks for sharing! I would, however, encourage you to create a Paisley branch in Launchpad and improve the error handling and connection abstraction there. This would make it much easier for us to merge your changes into Paisley. At the very least, you may want to register your project with Launchpad and associate it with the tx project group, so people can find it easily (the tx project group, or "super project" is for Twisted community code). Also, are you aware that web2 features are being slowly ported to web, and that twisted.web will continue to be the official web library? Thanks again! d
Hi Duncan, thanks for heads up. 2008/9/5 Duncan McGreggor <duncan.mcgreggor@gmail.com>
I would, however, encourage you to create a Paisley branch in Launchpad and improve the error handling and connection abstraction there. This would make it much easier for us to merge your changes into Paisley.
Changing the connection abstraction and error handling in paisley would make it extremely similar to couched. The main difference would be the name :-). I might do it. Btw. I've created a second release of couched which supports views and probably fixes a few other things. Just forgot to release it. Couched was mainly an exercise in understanding the CouchDB interface. At the very least, you may want to register your project with
Launchpad and associate it with the tx project group, so people can find it easily (the tx project group, or "super project" is for Twisted community code).
I'll try to do this over the weekend.
Also, are you aware that web2 features are being slowly ported to web, and that twisted.web will continue to be the official web library?
I'm aware that something might or might not happen in the twisted web world :-). <begin twisted web rant which would be much better suited for twisted-web> Frankly I've given up following the web vs. web2 discussions. I used web2 as it is what I am familiar with, and I wanted connection caching. I have a significant code base using web2 and beside a few issues (for which I've submitted patches and bugs, but nothing happens) it works fine, though I have to maintain a patchset for it :-/. Twisted web is currently not an option (I need request pipelining and http 1.1). <end twisted web rant> -- - Henrik
2008/9/5 Jean-Paul Calderone <exarkun@divmod.com>
On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen < thostrup@gmail.com> wrote:
[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).
Which other HTTP 1.1 features do you need?
HTTP 1.1 status codes, notable 303 (see other). I could probably use 302, but 303 is the "correct" behaviour for my case. This should be relatively easy to implement. Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy. I'm not saying streams are the answer but it works for me (actually I use the producer/consumer interface in my lower level code and then adapt it in the web layer). Unfortunately I've had to rewrite a lot of the producer/consumer code and stream stuff and glue it together, as some of the stuff in twisted (not just in web2) is rather broken. For complete insanity (sorry to whoever wrote it ) see twisted.protocols.basic.FileSender, which implements its own magic method beginFileTransfer, which takes BOTH a file and a consumer as argument. This makes is impossible to seperate the construction of the producer and starting the transfer. Ticket #1956 fixes some of these things. I'm not quite sure how it handles errors (which can happen both ways), but I've just had a quick look. It's been a whlie since i've had a look at all this. I write a bit about it, which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt IMHO, getting (and deciding on) a proper producer/consumer interface need to happen, to make whatever web things twisted will have usefull. Changing the producer/consumer might be massive API break, however it is a needed one. There are simply things the current API cannot do (i usually return a deferred along with any producer/consumer object to indiciate how the data moving went). I might be interested in participating in this, however I'm rather work struck as I have to hand my ph.d. thesis at the end of this year. Btw. I really am a huge fan of twisted, however the web and producer/consumer is a rather dark spot in an otherwise bright world :-) -- - Henrik
On 5 Sep, 11:25 pm, thostrup@gmail.com wrote:
2008/9/5 Jean-Paul Calderone <exarkun@divmod.com>
On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen < thostrup@gmail.com> wrote:
[snip] Twisted web is currently not an option (I need request pipelining and http 1.1).
Please feel free to use twisted.web2 if it is meeting your needs currently. However, our goal remains to focus maintenance effort on twisted.web and provide a gradual, compatible evolution to a better webserver. The HTTP 1.1 features that you need will eventually be added to twisted.web, and I hope that the split with paisley can be resolved at that point. I personally don't use couchdb or have any interest in its bindings, but it sounds like you have some other misconceptions about twisted and its development process which I'd like to correct.
Which other HTTP 1.1 features do you need?
Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy.
640k limitation? In twisted.web? As far as I know there's no such thing; the only 640k limitation I know of anywhere in Twisted is in PB, and that's just on individual message size. A brief skim through the twisted.web client and server code doesn't reveal any obvious size limitations that I can see. (Is there some 640k limitation in the 'cgi' module?) If there is such a limitation it should be more clearly documented somewhere. True, twisted.web doesn't provide a graceful way for resources to handle streaming file uploads as they arrive, but for dedicated servers it is certainly *possible*; Request.handleContentChunk says it's "not for users", but you could easily write a few simple patches which would make it "for users", or just subclass it and ignore the docstring. Nothing in twisted.web is really that hard to fix.
Ticket #1956 fixes some of these things. I'm not quite sure how it handles errors (which can happen both ways), but I've just had a quick look. It's been a whlie since i've had a look at all this. I write a bit about it, which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt
Ticket #1956 is still kind of stuck defining the problem. If you have comments or thoughts on #1956, please feel free to add them as comments. Text files lying around the web are not as helpful as discussions on tickets. Your text file suffers from the same problems as the existing discussion on the ticket, though: a lack of concrete use-cases and example code. There are lots of proposed solutions but not a lot of proposed problems. (I am certainly a guilty party here, I really should describe the problem in more detail.) We all feel there is some inelegance, but there is no example of some code that we can all agree is gross and would be fixed by some better API. Please come up with an example - the shorter the better - of a deficiency in the streaming API and put it on the ticket. Hopefully that will further the discussion there.
IMHO, getting (and deciding on) a proper producer/consumer interface need to happen, to make whatever web things twisted will have usefull. Changing the producer/consumer might be massive API break, however it is a needed one.
No. 1. We will never break compatibility with an API this old and this stable. When we do start to move over to a new API, the transition will be particularly gradual and slow given the core nature of this API and the fact that pretty much everything uses it. Even if it were only around for one release, see http://twistedmatrix.com/trac/wiki/CompatibilityPolicy 2. It's not clear what the new API would be. Every suggestion proposed thus far has some issues, and there's no clear winner. Unless someone takes the time to really thoroughly explain the problem and describe what effect the solution would have, with examples, then we're not going to make any progress. 3. There isn't anything that you *can't do* with ITransport/IProducer/IConsumer/IProtocol. It can be slightly awkward sometimes, but I have yet to see an example where you really can't make something work. For example, IConsumer doesn't have .finish(), but that's fine, because you probably mean IProtocol.connectionLost(...) anyway. Some users of these interfaces don't provide nice ways to get data in and out, and yes, a uniform and consistent way to deal with slinging data between endpoints (for example, a standard, featureful object to remix a 'protocol' into a 'transport' and vice versa) would probably help with that. Given that the deficiency creates momentary inconveniences rather than impossibilities, there's no way we're going to break *every single twisted program ever* just to make it slightly more convenient. 4. Given that the semantics are already there, it will be very easy to evolve new code to call old interfaces, provide adapters which wrap around things and generally make it easy to have some hypothetical shiny new interface cooperate just fine with the crufty old ones.
Henrik Thostrup Jensen ha scritto:
2008/9/5 Jean-Paul Calderone <exarkun@divmod.com <mailto:exarkun@divmod.com>>
On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen <thostrup@gmail.com <mailto:thostrup@gmail.com>> wrote:
[snip]
Twisted web is currently not an option (I need request pipelining and http 1.1).
Which other HTTP 1.1 features do you need?
HTTP 1.1 status codes, notable 303 (see other). I could probably use 302, but 303 is the "correct" behaviour for my case. This should be relatively easy to implement.
Furthermore I need the server to support large uploads (last i checked the 640K limitation was still there) and downloads, and with reasonable efficiancy.
You can try to use Nginx with the upload module http://www.grid.net.ru/nginx/upload.en.html The upload module parse the multipart/form-data entity body, and saves all file fields to temporary files. The backend server receive a multipart/form-data entity body where each file field is replaced with several fields: *_name, *_content_type, *_file_name, *_tmp_path (names are configurable).
[...]
Manlio Perillo
Hi Henrik, Patches to Paisley would be greatly appreciated. On Sep 5, 2008, at 12:48 AM, Henrik Thostrup Jensen wrote:
Also, are you aware that web2 features are being slowly ported to web, and that twisted.web will continue to be the official web library?
I'm aware that something might or might not happen in the twisted web world :-).
<begin twisted web rant which would be much better suited for twisted-web> Frankly I've given up following the web vs. web2 discussions. I used web2 as it is what I am familiar with, and I wanted connection caching. I have a significant code base using web2 and beside a few issues (for which I've submitted patches and bugs, but nothing happens) it works fine, though I have to maintain a patchset for it :-/. Twisted web is currently not an option (I need request pipelining and http 1.1).
Please file tickets about making twisted.web support the HTTP/1.1 features you need, this is the best way to get that support into Paisley. It just saddens me to see a 3rd couchdb python client library spring up. 2 is bad enough but one being asynchronous and one being synchronous means it can't really be helped. Paisley development hasn't stopped it just hasn't really needed much work. -David "The Unmaintainer of Web2"
2008/9/5 David Reid <dreid@dreid.org>
Hi Henrik, Patches to Paisley would be greatly appreciated.
Will you accept a switch to web2, otherwise there is very little point to it. Otherwise the patch kit needs to look like this: Proper producer/consumer infrastructure in twisted Integrate the above in twisted.web along with http 1.1 response codes, and request pipelining. Some patches to paisley, which would be quite small compared to the other. Sorry, but it is not that simple :-) Please file tickets about making twisted.web support the HTTP/1.1 features
you need, this is the best way to get that support into Paisley.
See my response to JP.
It just saddens me to see a 3rd couchdb python client library spring up. 2 is bad enough but one being asynchronous and one being synchronous means it can't really be helped. Paisley development hasn't stopped it just hasn't really needed much work.
Without getting into some large discussion here, I think that some diversity is actually a good thing. Of course it is no use having several half implementations lying around (the python community excels at this :-)), than having a single working library. However. the interface of couchdb is simple enough that a full client library can be written rather fast. This is not sad, but very nice :-). -- - Henrik
participants (6)
-
David Reid -
Duncan McGreggor -
glyph＠divmod.com -
Henrik Thostrup Jensen -
Jean-Paul Calderone -
Manlio Perillo