[stdlib-sig] Evolving the Standard Library

[Vinay Sajip]

Armin Ronacher <armin.ronacher at ...> writes:

> I agree that nobody did.  And there is a reason for it, and that reason
> is probably something that could be discussed in a separate thread too,
> because it affects a lof stuff that is currently in the standard libary.
> 
> I looked at the logging library and my point of view is that logging is
> "broken by design".  Eg: you can't fix it without breaking backwards
> compatibility.

It's easy for you to just say "it's broken by design", and that's only a more
polite way of saying "it sucks". It doesn'nt strike me as a basis for
constructive dialogue, unless you provide some more specifics.

> I'm sorry for the wait I was expressing my disagreement with design
> decisions made in the standard library and I would love to change that.
>  One of the biggest griefs I have with how the standard library works it
> that it does not work for me and everytime I discuss that topic with
> anyone else I immediately get the feedback that "this does not happen in
> the real world" or "works for me" etc.

Well, I'm happy to discuss any problems you are having with logging in a
practical sense. Obviously I can't do much with "I just don't like how it's
designed", but if you want to spell out a specific problem - something you want
to do with it that you just can't - I'll gladly listen and see if I can help
with it.

> I consider any kind of shared state a design mistake, no matter if it
> may work for some people or not, the logging package is no exception.
> However I have to admit that global loggers appears to be one of the
> easiest solutions for the problem.

I think you are being dogmatic, rather than pragmatic. The Zen of Python says,
"practicality beats purity." Your bugbear here seems to be how shared state
causes problems in web applications. Despite having shared state, AFAIK the
logging module is quite usable in a web context - as well as the usage with
Django and Tornado that I mentioned earlier, Google App Engine uses it too
(meaning, all the web applications developed with GAE can use it). So if it
doesn't work for you in a practical way, give me some details. But the logging
design isn't meant to be a candidate in a beauty contest, and I don't claim it's
perfect. You're a very smart guy, Armin, but you perhaps need to consider that
it is possible to not like a design because it doesn't suit your personal taste
- but that doesn't necessarily make it a bad design. 

> 
> If I would have to create my own logging library, I would go a total
> different part from design to implementation.  I would create a system
> where the "sender" is an arbitrary Python object and the system that
> handles it would have to check for its own if it may process that
> message.  Only *if* there was any handler that wants to do anything with
> that message, it would pull the details form the stack and format the

Currently, Python logging doesn't do formatting of stack traces etc. until it's
sure that the message is severe enough to require handling, based on the current
logger configuration. When handling a message, each handler checks its
configuration before formatting and dealing with the formatted message. The
system tries not to do unnecessary work, and if you have found some cases where
it does unnecessary work, please tell me. So at present, Python logging conforms
to your statement "Only if there was any handler that wants to do anything with
that message, it would pull the details form the stack and format ..."

That's been there from day one, so if you got the idea it worked differently,
I'm not sure why.

> message.  That also would get rid of the formatters and all the other
> stuff we currently have and avoids a global registry of loggers.

In Python logging, you never have to instantiate a Formatter in your code unless
you want some specific formatting functionality or format. In your scheme, if a
user wanted certain messages to be formatted in certain ways (and other messages
in other ways - e.g. for a log file as opposed to console display), you would do
this without any formatter classes - how, exactly?

The global registry of loggers is there to avoid the need to pass loggers around
the system - you just access them by name. It's a bit like thread locals - do
you have a problem with thread locals, too? Sure, it's state shared across
threads, unlike thread locals. But if it's causing you a specific problem
because you've found some non-thread-safe behaviour, I'd really like to know.

> How would I configure a logger in a library then?  I library would
> *never* by default print anything that is logged.  Default
> configurations for the logging system currently are the biggest reason
> for me to hate that library.  Many libraries do not even give you the
> ability to turn of logging in general, and documentation for the logging
> system was probably one of the reasons.  I don't know when the
> "NullHandler" example appeared in the docs, but I'm sure it was not
> there when I started using the logging system.

Agreed, the logging from a library leading to warning messages was an annoyance
in the library as first released. But *you* never raised an issue about it. When
other people did (I think it was Thomas Heller), I immediately updated the
documentation to explain how to use a NullHandler to avoid annoying messages,
and NullHandler appeared in a subsequent release soon afterwards.

If default configuration for the logging system is the biggest reason for you to
hate that package, don't get mad - get even. Tell me how I messed up and how to
make it better.

> 
> I cannot use logging for anything serious because it is slow, it has
> shared state that often causes frustration, you cannot delete loggers
> and much more.  Of course I do use logging for libraries because it's
> the standard and the best we have got.

Slow? Sure, everything has a cost. It's all about tradeoffs. What specific
performance problems have you come up against? What mitigating strategies did
you use? What were the observed performance metrics, as against what you
expected? Did you do any profiling to be sure that logging was definitely the
culprit? Show me the numbers.

If your frustration with shared state is from an aesthetic point of view, I
can't really help. After all, sys.modules (say) is shared state too. If the
frustration comes from some specific thing you're trying to do, please do tell
what that is.

You cannot delete loggers because in a multithreaded application, other threads
may still be using them. You can, however, disable loggers - which, from a
functional point of view, seems as good. Or are you finding that loggers are
taking up too much memory?

> The only reasons (in my opinion) that logging is still around is that
> it's in the stdlib, not because it's any good.  Especially when it comes

That's a cheap remark, I would have expected better from you. So everybody who
uses logging (and is smart, like you) hates it but uses it because they have no
choice, right? It *is* widely used, AFAIK. If everyone shared your opinion, I
don't think that would be the case.

> to highly optimized code in web applications you will quickly discover
> that half a dozen log calls take up more CPU cycles than the actual
> application code.
>

Please show me the numbers.

> I don't know how I could contribute constructive criticism.  I'm sorry
> for that.

You can't contribute constructive criticism because you don't know how? Well,
you've already made some criticisms in your post, so how about putting some in
more detail? I really like Jinja2 (a lot), but if someone said to you "Jinja2
sucks, it's really slow, it does all that conversion to bytecode stuff, that's
really gross. I can't really say any more than that, excuse me while I vomit"
then how would you feel? That's pretty much how you're coming across right now.

> > Am I expecting too much?
> I'm afraid you are.  It's hard to accept that some people think of your
> system as "it just sucks", but you can't change that.  I know that

Actually I have no problem with that - people thinking that it sucks. You can't
please everyone, and that goes for when you have a roomful of smart people, too.
The problem I have is when people just vent without trying to make it better,
talk about replacing it or removing it without even having the courtesy to talk
to me first. (I'm not talking here about what you post on your blog - those are
your opinions and of course you're entitled to say whatever you like. I'm
talking about this discussion and perhaps on python-dev and other other SIG
mailing lists.)

Perhaps you think I'm being oldskool by mentioning "courtesy", but I prefer to
think if it as an attribute of being a grown-up. Open source is very political,
but that doesn't mean it doesn't have to be polite.

> feeling from some of the stuff I wrote.  It's just a little bit worse
> for you because logging is the standard (and only) one, everybody uses.
>  For the stuff I wrote*, people have choice.  If they don't want to use
> it, they don't have to.

Actually they don't have to use Python's built-in logging either. Who's twisting
your arm? Talking of twisting, I recently had a discussion with Glyph Lefkowitz
of Twisted Matrix about twisted.log after he posted some of his thoughts on
logging in general and Python's implementation in particular. I believe I
rebutted, or at least addressed, all of his points. They will continue to use
twisted.log for aesthetic and pragmatic reasons, which is fine by me. They've
built a bridge to Python logging, presumably because some customers wanted that.

By the way, if you think that having choice has no downsides, perhaps you should
step back and take a look at the Python web application framework space -
Django, Pylons, TurboGears, web.py, Werkzeug, ... wow, so much choice! Sorry if
I missed one. Each one doing it a little differently, even for something as
basic as HTTP request/response objects. (I know the issues aren't trivial, but
are they rocket surgery? ;-) Look at Graham Dumpleton's frustration over WSGI
and Python 3.0 - I feel more than a little sympathy for his situation. He's
actually trying to create something useful, just as I am. There's no shortage of
approaches, and opinions, and in general, I believe competition is good. But
some of us are just trying to get some work done here.