[stdlib-sig] Evolving the Standard Library

[Vinay Sajip]

Armin Ronacher <armin.ronacher at ...> writes:

> Django has a global settings module and yet there are tons of developers
> hating that one.  Even Simon Willison agrees that the settings module
> was the biggest mistake made in Django.

No system is perfect, and even systems that are good to start with improve
over time. I don't say that every use of shared state is justified. There's no
connection I can see between haters of Django's global settings module and
problems with the logging package. Care to stop waving your arms about over
general principles like "shared state", and get down to specifics about
logging?

> That does not have anything to do with personal taste or not.  Some
> things cannot work in some situations, and logging is currently one of
> them.  Logging was designed to be based on persistent loggers.  You get

You're still talking in very general terms - "some things cannot work in some
situations" - that could really be said about most APIs. What things? What
situations? I'm not saying we have to do this on this thread, else it might
become off-topic - just that it should be done somewhere.

> one, it's there, you can log into it.  SQLAlchemy for example does an
> incredible dance to get separate loggers for temporary database connections.

Neither Mike Bayer nor anyone else on the SQLAlchemy team have raised this as
an issue, AFAIK. Mike, if you or other SQLAlchemy committers are reading
this, please get in touch. I'm not sure why they'd need to get separate
loggers for temporary database connections - logging has mechanisms to log to
the same logger while separating out different contextual information for
transient entities (for example, database or socket connections).

> That is actually something where I had a small problem with the logging
> implementation that could be fixed: the exc_text is currently only set
> in the format() function of the formatter, instead of being an attribute
> on the log record.  And yes, it could be an attribute of the log record
> if it was a property.  (I guess the reason it isn't currently is that
> the logging module is backwards compatible down to 1.5.2 or something)

It *is* an attribute of the LogRecord, and always has been - not sure where
you get your "facts" ;-). The reason it is not set in the constructor is to
avoid doing unnecessary work - something else you complained about. The
reason the exc_text is set in the formatter is this: the logger receives
the event/message, conditionally passes it on to handlers (i.e. only when
necessary), the handlers format and send out the message (again, only when
necessary). The exc_text is stored as an attribute of the LogRecord, but only
computed when the exception is formatted for output by the overridable
formatException method of the Formatter. If I hadn't done it that way, you
might well have said, "Bah! There's no way to control how an exception gets
formatted. Logging's not reusable." Once computed, it's stored there so that
later handling operations don't need to compute it again (again, to save
unnecessary work).

> My point was that there are no loggers, no registry of loggers, just
> senders and senders are arbitrary Python objects.

The point of having logger names is to control, by easy configuration,
logging verbosity at different levels in your application. Different levels
imply a hierarchy. A dotted-namespace is a reasonable way of codifying a
hierarchy - Java, Python and many other systems use it. If you don't want to
pass loggers around between function calls (and in case you say a logger
could be an instance attribute, remember logging works for non-OOP
function-based scripts, too), then you need some way to get hold of a
certain logger from anywhere in your application. A registry provides that.
It means that at runtime, without bringing down a running application, it is
possible to change the verbosity of any logger.

If you want arbitrary objects to send notifications to arbitrary objects, you
can use something like pydispatcher. It works, and it fulfills a need, but
it's not, to my mind, a logging system. You've merely stated the germ of an
idea ("no loggers, no registry of loggers, just senders, and senders are
arbitrary Python objects"), but I see no evidence that you've thought it
through in terms of addressing a whole host of practical issues,
such as configuration.

> In my personal experience the formatting is based on the handler.  For
> example if I want to log into a text file, i have one formatting, if I
> log to stderr I have a different one.  If I log into an email I have a
> completely different one.  For example I tried a long ago to log my

Yes, that's how it is because the audiences are different. The end user may
not need a timestamp and probably will not want a stack trace.

> exceptions from a logging handler into a new ticket in the trac.  I
> ended up replacing a log of code from both log record and formatter
> inside the log handler because of the way the message was assembled. I
> had to escape text and could not rely on string formatting to work.

I don't know, without knowing your problem in more detail, if you could have
avoided copying and changing code from LogRecord and Formatter. Obviously
I've tried to provide enough hooks so that people can subclass and override
methods for specific requirements, such as adding to a Trac ticket. If you
describe the problem in more detail, I may be able to indicate a better
solution. If it turns out that I need to provide more hooks where people can
override functionality, I'm open to doing that.

> I have a problem with unnecessarily used thread locals.  Not with the
> concept in general.

That's okay then. I don't feel that the use of shared state in logging is
unnecessary, because of the benefits it confers. It was done in a thoughtful
way, not just because I don't know any better. But then, I would say that,
right?

> I don't complain that it's not threadsafe, I'm pretty sure if logging
> was thread unsafe someone would have noticed by now.

So your complaint is really just a philosophical diatribe against shared
state? 

> If I would know how to improve it so that I'm happy with it, I would
> have told you.

Then you're really saying something roughly like "It's Not Invented Here, so
I don't like it. The only way it could be better is if I had thought of it.
Period."

> > can't really help. After all, sys.modules (say) is shared state too.
> http://lucumr.pocoo.org/2009/7/24/singletons-and-their-problems-in-python
> *cough*

*cough* *cough*. I've already read that post, as I referred to it earlier in
this thread. Since sys.modules is shared state at a much more fundamental
level than logging's logger registry, why not focus your energies on getting
that changed first? If you're successful at pulling it off, it'll no doubt
lead to a whole slew of changes in the Python ecosystem, of which logging is
just a tiny part.

> > ... because in a multithreaded application, other threads
> > ... be using them. You can, however, disable loggers - which, from a
> > functional point of view, seems as good.
> Which is why I would not design a logging library based on loggers.

You didn't really answer the point that you don't need to delete loggers,
since disabling them is just as good.

> > Or are you finding that loggers are taking up too much memory?
> Even if a logger would be just 8 bytes in size, it would steal leak if
> you cannot control the number of loggers created.  (see SQLAlchemy for a
> nice example).

You (as the application developer or library developer) *can* control it,
because you decide exactly which loggers get created. I'll ask Mike Bayer
about the specifics of the SQLAlchemy issue.

> A while ago when I was blogging about logging I wrote this:
> 
> > [...] It's called "logging" and does exactly that --- it logs errors.
> > I don't know why so many people miss it or just don't use it, but it's
> > really one of the good things in the python standard library.
> 
> to which Robert Brewer replied:
> 
> > That's also why it's so bad: it's so extensible and configurable
> > that's it's far too slow for high-performance websites.
> 
> And I'm pretty sure Rober knows what he's talking about.
>

No doubt he does. Logging's extensible and configurable because it has to
work for *lots* of different scenarios - not *just* high-performance web
sites. Obviously, that generality involves tradeoffs which may make the
logging package less suitable for high-performance scenarios. Did Robert
confide in what they *do* use for logging functionality?

Robert - if you're reading, do you have numbers? Obviously *any* logging
will involve *some* penalty. I'll be very happy if someone says, "we profiled
the application with logging enabled, and we found that there was a penalty
of X%. That's hard for us, but we can live with an overhead of Y%. Here are
the profiler reports, can you do anything to improve it?" I might *not* be
able to make much, if any, improvement, but I'd certainly take a shot at it.

> I gave up on defending Jinja2 a while ago.  Because people from the
> Django world constantly call me names for enabling logic in templates ;)
>

There you go. Is Jinja2 "broken by design", then?

Aso, I prefer Python to Ruby. Should I say "Ruby? Broken, by design"?

> I'm sorry for the way I expressed my disagreement with the library's
> design here and in what way.  To make you feel better: from all the
> modules in the standard library, logging is still one of the best
> designed and implemented, despite my disagreement with it.

What, sarcasm now? From "broken by design" to "paragon of virtue, exemplar
to the world"? Please. I've been around the block a few times, no longer the
new kid, and my skin is reasonably thick. I'm not crying. But keep it
constructive (i.e. with improvement as a goal), that's all I'm asking.

> and WSGI for Python 3 is something that has to be discussed.  Graham's
> master plan is one proposal, but in my opinion not the best one.

My point was not about whether Graham's plan was or was not the best. It's
that he wants to get *something* reasonable out there, knows the issues and
is frustrated with the constant back-and-forth between conflicting opinions.
We could be waiting forever for a resolution, what good is that to anybody?

Remember Voltaire: "Le mieux est l'ennemi du bien." (The best is the enemy of
the good.) Am I showing my age, or what? ;-)

Regards,

Vinay Sajip