[Tutor] python logging module: two handlers writing to the same file - okay?

Ricardo Aráoz ricaraoz at gmail.com
Mon Dec 17 11:56:17 CET 2007

Hans Fangohr wrote:
>>>> Ricardo Aráoz wrote:
>>>>> Kent Johnson wrote:
>>>>>> I don't know the answer, but it has nothing to do with the logging
>>>>>> module. The question is, can the same file reliably be opened
>>>>>> twice for
>>>>>> writing in the same module.
>>>>> Well, the question would actually be if the logging module is smart
>>>>> enough to find out that both your filehandlers are referring to the
>>>>> same
>>>>> file and open it only once.
>>>> A quick glance at the logging module shows that it is not that
>>>> smart. It
>>>> just opens the file twice; hence my original question.
>>> To summarise the situation: the logging module is not clever enough to
>>> detect that I am opening the same file twice. If I use 'w' as the
>>> opening mode, then I know (and have an example program for this) that
>>> the logging 'fails': I suppose this can be tracked to the first part of
>>> the log file being overriden when the second filehandler attempts to log
>>> its first message, and thus opens the file (again) in 'w' mode.
>>> Using 'a' (append) or 'a+' (?)  *seems* to solve this problem but we are
>>> still short of knowing as to whether this is guaranteed to work (on all
>>> platforms...) or whether it just happened to be okay on Mac OS X and
>>> (Debian) Linux (on which I have tested this).
>>> I think my example raises another question (in particular in context
>>> with openening the files in 'w' mode): would it be worth contacting the
>>> developers of the logging module to suggest that it
>>> - tracks which files it writes to and
>>> - warns if the same file is opened twice (or more ofter) using 'w'?
>>> Alternatively, one could enhance the internal logic of the logging
>>> module so that it is aware of the same file being used twice (or more
>>> times) so that it only opens the file once, but I suppose this raises
>>> some other questions (if the two filehandlers have been given different
>>> opening modes, say 'a' and 'w', which one should it take when opening
>>> the file only once?). Therefore, just issueing a warning may be the
>>> better solution: should be easy to implement, doesn't change the
>>> interface, and prevents (naive) users (like myself) from wasting time
>>> trying to track down the problem of the beginning of the log file
>>> missing.
>> As far as I can see, the only reason in your example program to open the
>> same file twice is to use two different formatters (i.e. two different
>> type of lines) in the same log, 
> Absolutely right.
>> if you'd drop the requirement for two
>> different format of line in the same log (which is itself somewhat
>> questionable) then you could use the same handler for both loggers, thus
>> opening only one file.
> True.
> FYI: The reason for wanting to use two different file formats is this:
> we have a somewhat larger project (http://nmag.soton.ac.uk) where we
> combine high-level Python code with low-level Objective Caml code (and
> a number of libraries). We would like to use the Python-logging module
> to log all sorts of messages coming from various parts of the code. In
> particular, we find having the line number (and function name if the
> python version provides this) useful information in the log messages
> (which can bedone for Python code issueing log messages). However,
> this information cannot be obtained from the OCaml code; thus we do
> not want to waste space in the log messages, and therefore we'd like
> to use different formatters.
> I'll reply to Ken's email in a minute, and I think that closes my query.
> Best wishes,
> Hans

One last thing Hans. If you are concerned about the space the log files
will use maybe it would be interesting to check out the zlib module
which could be used to compress your log strings before sending them to
the logging module, then a utility to read the modules would close the
cycle (caveat: I've never done this, just a crazy idea, but... what if
it works?).
Best of luck.


More information about the Tutor mailing list