But it blocks, mm? I could break things into small chuncks, so I could likely get away with not using a thread. But shouldn't it ideally be accessed from a thread? I'll note that you can open bsddb (using the new interface, not the one documented in Python) in a thread-safe way, but any given iteration (using a cursor object) must be done from a single thread--cursors are not thread-free. One approach might be to create a cursor in a server thread and then use a fixed-size queue to retrieve the iterations, but then you'll need at least as many server threads as you have open cursors. So the best approach might be to have a cursor manager which handles many such cursors/queues running on a single thread. (Invoking the manager, it returns a master queue for handling subsequent requests.) But then it sounds like a dedicated thread is again the best approach. It seems to me that a nice general solution would be a project in its own right. (I've found bsddb to be pretty powerful/useful/fast/handy.) Ah, Atop looks interesting. Thanks! Bill Matt Goodall <matt@pollenation.net> wrote: On Fri, 2004-05-14 at 08:17, Bill la Forge wrote:

New to twisted, interested in porting a project, but I can't find anything on using bsddb.

Open a bdb at the start of the app, use it as necessary, close the bdb at the end of the app. What could be easier ;-). It's probably worth creating a Service (http://twistedmatrix.com/documents/current/api/) to help manage the bsddb resource.

I assume I need a dedicated thread. Has anyone else worked on this?

You probably only need a thread for operations that you know will take a long time. Retrieving a single, keyed object should be so quick with bsdbd that it's probably not with the effort/expense of a thread. On the other hand, iterating an entire bdb will (possibly) take quite some time, so a thread may be a good idea. (Note that Python's bsdbd is not threadsafe so you will have to manage that yourself.) I don't know enough about the bsddb api but I suspect it's possible to avoid threads altogether by iterating an entire bdb in small chunks and relinquishing control to the event loop at the end of each chunk to allow other events to run.

(I did a web search, but couldn't relate what I found to any code in the 1.2 release I just downloaded. I also read the May list archive.)

You may find Atop (http://www.divmod.org/Home/Projects/Atop/index.html) interesting. Hope this helps. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python Bill la Forge http://www.geocities.com/laforge49/ Yahoo! India Matrimony: Find your partner online.

Itamar Shtull-Trauring wrote:

Atop is designed so that each operation is either very quick and non-blocking, *or* long running operations like iterating over results that can be easily broken up (and which is which is very clear). As a result atop works quite well without threads.

The flip side oft his is that it can be quite difficult to build complex queries without deep structural knowledge of the database (and they are therefore somewhat fragile). However, we're working on that. A somewhat unsettling thing about using atop is you start to realize how incredibly inefficient some of the things that are really easy to say with SQL are, even in SQL DBs ;). If an operation is synchronous in atop, it's fast. Right now this means, "if it's easy" - but a more advanced query mechanism will fix this. If you have any ideas, please let me know.

But it blocks, mm? I could break things into small chuncks, so I could likely get away with not using a thread. But shouldn't it ideally be accessed from a thread? I'll note that you can open bsddb (using the new interface, not the one documented in Python) in a thread-safe way, but any given iteration (using a cursor object) must be done from a single thread--cursors are not thread-free. One approach might be to create a cursor in a server thread and then use a fixed-size queue to retrieve the iterations, but then you'll need at least as many server threads as you have open cursors. So the best approach might be to have a cursor manager which handles many such cursors/queues running on a single thread. (Invoking the manager, it returns a master queue for handling subsequent requests.) But then it sounds like a dedicated thread is again the best approach. It seems to me that a nice general solution would be a project in its own right. (I've found bsddb to be pretty powerful/useful/fast/handy.) Ah, Atop looks interesting. Thanks! Bill Matt Goodall <matt@pollenation.net> wrote: On Fri, 2004-05-14 at 08:17, Bill la Forge wrote:

New to twisted, interested in porting a project, but I can't find anything on using bsddb.

Open a bdb at the start of the app, use it as necessary, close the bdb at the end of the app. What could be easier ;-). It's probably worth creating a Service (http://twistedmatrix.com/documents/current/api/) to help manage the bsddb resource.

I assume I need a dedicated thread. Has anyone else worked on this?

You probably only need a thread for operations that you know will take a long time. Retrieving a single, keyed object should be so quick with bsdbd that it's probably not with the effort/expense of a thread. On the other hand, iterating an entire bdb will (possibly) take quite some time, so a thread may be a good idea. (Note that Python's bsdbd is not threadsafe so you will have to manage that yourself.) I don't know enough about the bsddb api but I suspect it's possible to avoid threads altogether by iterating an entire bdb in small chunks and relinquishing control to the event loop at the end of each chunk to allow other events to run.

(I did a web search, but couldn't relate what I found to any code in the 1.2 release I just downloaded. I also read the May list archive.)

You may find Atop (http://www.divmod.org/Home/Projects/Atop/index.html) interesting. Hope this helps. Cheers, Matt -- __ / \__ Matt Goodall, Pollenation Internet Ltd \__/ \ w: http://www.pollenation.net __/ \__/ e: matt@pollenation.net / \__/ \ t: +44 (0)113 2252500 \__/ \__/ / \ Any views expressed are my own and do not necessarily \__/ reflect the views of my employer. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python Bill la Forge http://www.geocities.com/laforge49/ Yahoo! India Matrimony: Find your partner online.

Itamar Shtull-Trauring wrote:

Atop is designed so that each operation is either very quick and non-blocking, *or* long running operations like iterating over results that can be easily broken up (and which is which is very clear). As a result atop works quite well without threads.

The flip side oft his is that it can be quite difficult to build complex queries without deep structural knowledge of the database (and they are therefore somewhat fragile). However, we're working on that. A somewhat unsettling thing about using atop is you start to realize how incredibly inefficient some of the things that are really easy to say with SQL are, even in SQL DBs ;). If an operation is synchronous in atop, it's fast. Right now this means, "if it's easy" - but a more advanced query mechanism will fix this. If you have any ideas, please let me know.