Re: [Twisted-Python] recommended twisted database coding
"Jesus" == Jesus Cea <jcea@argo.es> writes:
Jesus> I recomend you "durus" Jesus> (http://www.mems-exchange.org/software/durus/) and, depending of Jesus> your usage profile, my storage backend: Jesus> <http://www.argo.es/~jcea/programacion/durus-berkeleydbstorage.htm> Jesus> Yes, durus supports transactions. Jesus> Durus is not async, but wrapping it in a thread should be easy enough. Jesus> I'm fairly involved with Durus, so I could help you there. Hi Jesus Thanks for the info. I think I've seen Durus before, but was not aware that you had implemented a BSD backend. Does that mean the qualification on the Durus home page "Durus is best suited to collections of less than a million instances with relatively stable state." does not apply here? After looking around at some Pyhon persistence options, I decided to use ZODB. I'd be happy to hear your thoughts on Durus+BSD versus ZODB. I was disappointed to see that ZODB had dropped BSD as a backend as BSD is battle hardened and I wanted to use its secondary DBs. BTW, I had no reply to email sent to greg@electricrain.com, the maintainer of the Python bsddb interface. I presume you are using that code? Anyway, I am using the BTrees provided by ZCatalog in Zope. Apart from things like robustness, speed, and number of objects I could store, I very much wanted to avoid having multiple persistence systems. Using ZODB makes this very easy. It sounds like you have that advantage too. How well supported is Durus? What about your interface? Are you in Spain? We can discuss that offline I guess. Regards, Terry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Terry Jones wrote:
Thanks for the info. I think I've seen Durus before, but was not aware that you had implemented a BSD backend. Does that mean the qualification on the Durus home page "Durus is best suited to collections of less than a million instances with relatively stable state." does not apply here?
Right. My storage engine allows multigigabytes repositories, with a lot of writes. Actually, I use it myself with a 933 gygabytes repository (mailbox storage) and a couple of days ago a guy reported me his profile: 16 terabytes storage managing medical scanning images.
After looking around at some Pyhon persistence options, I decided to use ZODB. I'd be happy to hear your thoughts on Durus+BSD versus ZODB. I was disappointed to see that ZODB had dropped BSD as a backend as BSD is battle hardened and I wanted to use its secondary DBs.
ZODB is fine, but its storage engine is ugly. I miss also the BSD backend. Without it, ZODB behaviour when you have a lot of writes is not better than stock Durus. The main advantages of ZODB over Durus is the distributed caching client (ZEO) and the multithreading support. I'm working with the Durus guys to improve durus in those areas. Progress is slow but working on it :-p For my environment, Durus simplicity (you can keep the entiry durus code in your head) and performance, plus my storage backend goodies, are my favourite poisons :).
BTW, I had no reply to email sent to greg@electricrain.com, the maintainer of the Python bsddb interface. I presume you are using that code?
Yes I use pybddb. In fact, next release (in python 2.5 beta 1) will include some patches of mine :-). Greg is a bit slow answering email. I suffer it myself from time to time :). The inminent Python 2.5 beta could increase his workload, also, I guess. So, forgiveness...
Anyway, I am using the BTrees provided by ZCatalog in Zope. Apart from things like robustness, speed, and number of objects I could store, I very much wanted to avoid having multiple persistence systems. Using ZODB makes this very easy. It sounds like you have that advantage too.
Durus has a BTree class also :). I also prefer to use a single persistence system. I'm decommisioning SQL, for example, migrating all my systems to Durus backends. My Durus preference over ZODB started because I could understand Durus completely, and its "limitations" were not important to me. Nowadays Durus was improved a lot and I still see additional potential to grow. BTW, I see very few people using ZODB outside ZOPE. Ah, my personal webserver runs Zope :p
How well supported is Durus? What about your interface?
My only complain about Durus support is that it is not a open source proyect in the sense that you can't directly contribute code. This is an issue I had talked to the coder guys several times in the past. I guess that if durus could reach a critical mass, the issue would be solved by itself. Meantime, the difficulty is convincing the coder guys that your suggestions a) do not hurt their performance and b) are valuable enough for their workload. Of course, their workload is not your workload. In any case, I must say that they are smart and recognize a good idea when they see it. The mailing list is quiet and supportive, also. About my backend... currently I try to do a monthly release, although current releases are already production ready. I'm very motivated and Durus + my backend are top priorities now for me, so... I'm planning for replication/redundancy also, if Durus guys implement some suggestions I did recently :).
Are you in Spain? We can discuss that offline I guess.
Yes. I live in Madrid. We are getting a bit offtopic. If you use jabber/XMPP, you can contact me online. See details in my signature. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRI9CoJlgi5GaxT1NAQL/pwP+LQQnzJANoDaAjp2/7nAMl1CMcrCgBIrP WUapc2h6W8SsO4XYRuWd8d/yZAvhG7ZnsGZOHYnn4qvQ9BPBwNDy1z2vej/ZdQpd KbGOSwW2hNm8burdmHvbVvRAPuuZGS+0u0/YApOQ+XTsiMOxpPHeFdRPncw+J9Lg uzcnUVXpZC4= =nhHW -----END PGP SIGNATURE-----
participants (2)
-
Jesus Cea -
Terry Jones