Special logging module needed

[Laszlo Nagy]

2011.03.23. 19:33 keltezéssel, Dan Stromberg írta:
>
> On Wed, Mar 23, 2011 at 7:37 AM, Laszlo Nagy <gandalf at shopzeus.com 
> <mailto:gandalf at shopzeus.com>> wrote:
>
>     I was also thinking about storing data in a gdbm database. One
>     file for each month storing at most 100 log messages for every key
>     value. Then one file for each day in the current month, storing
>     one message for each key value. Incremental backup would be easy,
>     and reading back old messages would be fast enough (just need to
>     do a few hash lookups). However, implementing a high availability
>     service around this is not that easy.
>
>
> I think a slight variation of this sounds like a good bet for you.  
> But when you "open" a database, create a temporary copy, and when you 
> close the database, rename it back to its original name.  Then your 
> backups should be able to easily get a self-consistent (if not up to 
> the millisecond) snapshot.
My idea was to open all database in read-only mode, except the one for 
the last day. So it will be possible to archive these files (except the 
last day). No need to make copies. I have also developed an algorithm 
that merges the database from the "previous day" with the database of 
the "month of the previous day". This happens when a day switch occurs. 
The algorithm detects this, and it can merge the database and at the 
same time, log messages can be added. The service is only suspended two 
times a day, when fragmented and defragged database are switched. But 
that is only a single "file rename" operation and it takes less than 0.1 
seconds to do. So the alg. is ready. I can implement it. But It is not 
easy to do it. I can spend many days with it and then it may turn out 
that it is not that efficient than I thought.

I cannot believe that others din't run into the same problem. This is 
why I posted to the list. I don't want to reinvent the wheel if I don't 
need to.

> Or did you have some other problem in mind for the gdbm version?
Nope.
>
> BTW, avoid huge directories of course, especially if you don't have 
> hashed or btree directories.  One way is to come up with a longish 
> hash key (sha?), and use a trie-like structure in the filesystem on 
> fibonnaci-length chunks of the hash keys becoming directories and 
> subdirectories.
Hmm that's a good idea. Thanks!

    L


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110324/cfa05a8a/attachment-0001.html>