New release of BerkeleyDB Backend Storage Engine for DURUS (20060629)
jcea en argo.es
Jue Jun 29 18:48:57 CEST 2006
-----BEGIN PGP SIGNED MESSAGE-----
See the CHANGES and the UPGRADING documents.
Also updated the "KNOW_HOW-DURUS" to Durus version 3.4:
20060629 - r191 - jcea en argo.es
Document the upgrade process.
20060620 - r188 - jcea en argo.es
"get_size()" is very costly. With current implementation,
the storage must access all the database. I can implement a manual
counter or migrate to btree and use DB_RECNUM:
Finally we implement a manually managed counter. Now "get_size()"
is instantaneous, but it require an storage upgrade. So, you can't
use previous backend versions.
20060620 - r184 - jcea en argo.es
When creating a new storage database, be able to
choose between "btree" and "hash" schema.
20060620 - r183 - jcea en argo.es
Document "KNOW_HOW-DURUS" updated to Durus 3.4.
20060524 - r157 - jcea en argo.es
We add a new "validate_garbage" optional parameter to the
storage constructor. If that parameter is True, the storage will
do a garbage check. Read the documentation in the README file
20060523 - r156 - jcea en argo.es
If the storage was stopped before garbage
collection was completed, the storage could leak
some unreachable objects.
Solving this issue, we make sure also that the garbage
collection makes progress even if the storage is stopped
repeatly before garbage collection completion. That is,
we store partial state to avoid to try to clean
already collected objects.
*** Release 20060629:
Storage databases created with this release are not
compatible with previous releases.
First time you use this release to open a storage
database created by previous releases, it will be
transparently "upgraded" to current format, so:
- - The storage will become incompatible with previous
- - In order to be able to upgrade the storage, you can't
open it in "read only" mode. Once upgraded, you can
use "read only" mode freely.
- - The upgrade process can be a bit slow, since the
backend needs to analyze the entire database. So, if
your database size is 1 Gigabyte and your hard disk
can push 50MB/s, the upgrade process will take about
20 seconds to finish.
- - While upgrading, the database will NOT serve requests.
- - The upgrade process doesn't take any RAM.
- - If the upgrade process is aborted (program quits, crashes,
machine reboot, etc), the database will be stable
and clean. That is, the upgrade process is transactional
Also, in previous backend releases there was a bug in
the garbage collection code that would skip over
"to be deleted" objects, leaving some garbage behind.
So with this release you have three options:
- - Ignore the issue: This release will not leak new garbage,
but garbage already leaked will remain there. You lose
some diskspace. No other side effect.
- - Do a dump+load migration, using the storage backend
"migrate" method. Your diskspace requirement will
double (temporaly), since you need space for the
source and the destination databases. The migration
will copy garbage, but this new backend release can
cope with it and it will be freed. You don't need
extra RAM to do this process, even if your database
is in the petabyte range :-).
- - Initialize the storage with the "validate_garbage"
parameter sets to "True": This option will force
a full sweep over the database, to examine
cross-object links and locate garbage. This scan
is done in-place, so you don't need extra diskspace.
It is transactional, so it is SAFE if something
It takes RAM proportional to object count
in your storage, so beware if you have a huge
Of course you only need to pass this parameter
once, to catch garbage leaked by previous releases.
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jcea en argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jcea en jabber.org _/_/ _/_/ _/_/_/_/_/
_/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v220.127.116.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
-----END PGP SIGNATURE-----
Más información sobre la lista de distribución Python-es