Hi everybody,
I'm known for my dislike of the standard libray. In the past I wrote
some blog posts about this topic into my personal blog already. However
as many people pointed out earlier, a blog is not the place for this
kind of criticism. Not only that, also just ranting about a topic does
not help at all. Yesterday I subscribed to the stdlib-sig and
immediately tons of mails ended up in my inbox. A quick look at the
mail archives confirms what I was afraid of: this list is really high
traffic. I tried to read up some of the discussions I missed but it's
nearly impossible to do that.
I would love to sum up my thoughts about the standard library here and
my ideas to improve it. This list of ideas and improvements does not
include any unrealistic plans such as rewriting the standard library, an
approach I was a big fan of.
I can see a couple of problems with the standard library currently, and
some reasons why that is the case. If we look back on the history of
Python it's obvious that a large number of modules in the standard
library appeared out of the need of a single developer or company a
while ago. Many of these libraries finally disappered or where renamed
in the big standard library reorganization in Python 3 and I'm very
happy that this happened. However at the same time a large number of
the modules still continue to show their age.
Python is currently heading into a new direction many people would not
have thought about a few years ago. And that are web applications.
For web applications different rules apply than for desktop
applications. Command line scripts or GUI applications are mostly fine
with shared state on module level, web applications are not. It is true
that Python currently has some issues with high concurrency and people
try to fix that by forking and spawning new processes which certainly
hides away the problem of shared state, but that does not solve it. In
fact, very recently Facebook open sourced the Tornado framework which
does very well at high concurrency by using async IO. Also this recent
interest in Tornado will probably also motivate Twisted developers to
improve their project's documentation and performance, because
competition is often the what causes projects to improve.
Now if we look at the standard library, we can see many modules that
just do not work in such environments because they have some sort of
shared state. The most obvious ones are certainly the `locale` module
and all the other modules that change behavior based on the locale
settings. Did you know that every major Python framework reimplements
time formatting even for something as simple as HTTP headers, because
Python does not provide a way to format the time to english strings
reliably? But there are certainly more modules that have this sort of
problem.
Also we have many modules in the standard library that in my opinion
just do not belong there. From my point of view, stuff like XML does
not belong into the standard library. But it appears that not many
people agree with me on this one. But even if everybody would,
backwards compatibility would still be a good reason to keep these
modules around.
Besides modules that do not work in every environment or modules that
were probably a mistake to include, we also have modules in the standard
library with a hideous implementation or no reusability, forcing people
to reinvent what's already there. For a long time, `urllib` was a
module I would have listed there, but as of Python 2.6, the module
largely improved by exposing the underlaying socket more which finally
alllows us to set the timeout in a reliable way. But there are still a
ton of modules in the library that cause troubles for people. `dis` is
one of them. The implementation of dis prints to stdout no matter what
you do. Of course you can replace sys.stdout with something else for a
brief moment, but again: this is not something we should aim for or
advertise because it breaks for many people. `Cookie` is a module
people monkey patched for a while (badly) to support the http only flag.
Not only does the code expose a weird API, it is also nearly impossible
to extend and even ships cookie subclasses that use unsigned pickles and
trust the client. `cgi` has again, shared state on the global namespace
that alters the behavior of the lirbary. Of course it was never
intended to be used by anything but `cgi`, but that leaves people
reimplementing it or abusing it.
So when the discussion started replacing `optparse` with `argparse`,
because the former is unmaintained I became alerted. My wishes have
always been the standard library to be a reliable fallback to be used if
everything else fails. Something I can rely on which will not change,
except for maybe some additions or modules moved to different locations.
As Python developers we became used to moving import locations a lot.
It it's `cPickle` or any of the element tree implementations, you name it.
I wonder if the solution to this problem wouldn't be a largely improved
packaging system and some sort of standardized reviewing process for the
standard library. Currently there is not even an accepted style for
modules ending up in the Python distribution. That, and a group of
people, dedicated to standard library refactoring. The majority of
libraries in the standard library are small and easy to understand, I'm
sure they are perfectly suited for students on projects like GSOC or
GHOP to work on. They could even be used as some sort of "playground"
for new Python developers.
Ubuntu recently started the "100 paper cuts" project. There people work
on tiny little patches to improve the system, rather to replace
components. Even though a large place of the standard library appears
to be broken by design they could still be redesigned on the small
scale, without breaking backwards compatibility.
Of course libraries like `locale` and `logging` are hard to change, but
it would still be possible. For `locale` it would probably a useful
idea to go into the direction of datetime, where the timezone
information is left to a 3rd party library. `locale` could provide some
hooks for libraries like `babel` to fill the gap. On the other hand
`Cookie` would be very easy to fix by moving the parsing code into a
separate function and refactoring the cookie objects.
We could probably also start a poll out there with well-selected
questions of what users think about parts of the library. And for that
poll it would make a lot of sense to not just ask the questions and
evaluating the results, but also track the area the user is coming from
(small size company, open / closed source, web development etc.).
Because we all are biased and seeing results grouped by some of these
factoids could be enlightening. That said, it could tell us that I'm
completely wrong with my ideas of how the state of the standard library.
But how realistic is it to refactor the standard library? I don't know.
For a long time people were pretty sure Python will not get any faster
and yet Unleaden Swallow is doing some really amazing progress.
If we want to push Python foward into new areas, and the web is one of
them, it is necessary to jump into the cold water and start things.
Any maybe we should have some elected task forces for things like the
standard library. Judging from the mailinglist it appears that far too
many people are discussing *every detail* of it. It is a good idea to
ask as many people as possible, but I am not sure if the mailinglist is
the way to do that. It is currently very hard to see the direction in
which development is heading.
Please think of this email just as a suggestion. I don't have too much
trust into myself to follow the discussions on this list camely enough
to become a real part of a solution, but I would love to help shifting
the development into a better direction, no matter which one it will be.
Regards,
Armin
