
Hi everybody, I'm known for my dislike of the standard libray. In the past I wrote some blog posts about this topic into my personal blog already. However as many people pointed out earlier, a blog is not the place for this kind of criticism. Not only that, also just ranting about a topic does not help at all. Yesterday I subscribed to the stdlib-sig and immediately tons of mails ended up in my inbox. A quick look at the mail archives confirms what I was afraid of: this list is really high traffic. I tried to read up some of the discussions I missed but it's nearly impossible to do that. I would love to sum up my thoughts about the standard library here and my ideas to improve it. This list of ideas and improvements does not include any unrealistic plans such as rewriting the standard library, an approach I was a big fan of. I can see a couple of problems with the standard library currently, and some reasons why that is the case. If we look back on the history of Python it's obvious that a large number of modules in the standard library appeared out of the need of a single developer or company a while ago. Many of these libraries finally disappered or where renamed in the big standard library reorganization in Python 3 and I'm very happy that this happened. However at the same time a large number of the modules still continue to show their age. Python is currently heading into a new direction many people would not have thought about a few years ago. And that are web applications. For web applications different rules apply than for desktop applications. Command line scripts or GUI applications are mostly fine with shared state on module level, web applications are not. It is true that Python currently has some issues with high concurrency and people try to fix that by forking and spawning new processes which certainly hides away the problem of shared state, but that does not solve it. In fact, very recently Facebook open sourced the Tornado framework which does very well at high concurrency by using async IO. Also this recent interest in Tornado will probably also motivate Twisted developers to improve their project's documentation and performance, because competition is often the what causes projects to improve. Now if we look at the standard library, we can see many modules that just do not work in such environments because they have some sort of shared state. The most obvious ones are certainly the `locale` module and all the other modules that change behavior based on the locale settings. Did you know that every major Python framework reimplements time formatting even for something as simple as HTTP headers, because Python does not provide a way to format the time to english strings reliably? But there are certainly more modules that have this sort of problem. Also we have many modules in the standard library that in my opinion just do not belong there. From my point of view, stuff like XML does not belong into the standard library. But it appears that not many people agree with me on this one. But even if everybody would, backwards compatibility would still be a good reason to keep these modules around. Besides modules that do not work in every environment or modules that were probably a mistake to include, we also have modules in the standard library with a hideous implementation or no reusability, forcing people to reinvent what's already there. For a long time, `urllib` was a module I would have listed there, but as of Python 2.6, the module largely improved by exposing the underlaying socket more which finally alllows us to set the timeout in a reliable way. But there are still a ton of modules in the library that cause troubles for people. `dis` is one of them. The implementation of dis prints to stdout no matter what you do. Of course you can replace sys.stdout with something else for a brief moment, but again: this is not something we should aim for or advertise because it breaks for many people. `Cookie` is a module people monkey patched for a while (badly) to support the http only flag. Not only does the code expose a weird API, it is also nearly impossible to extend and even ships cookie subclasses that use unsigned pickles and trust the client. `cgi` has again, shared state on the global namespace that alters the behavior of the lirbary. Of course it was never intended to be used by anything but `cgi`, but that leaves people reimplementing it or abusing it. So when the discussion started replacing `optparse` with `argparse`, because the former is unmaintained I became alerted. My wishes have always been the standard library to be a reliable fallback to be used if everything else fails. Something I can rely on which will not change, except for maybe some additions or modules moved to different locations. As Python developers we became used to moving import locations a lot. It it's `cPickle` or any of the element tree implementations, you name it. I wonder if the solution to this problem wouldn't be a largely improved packaging system and some sort of standardized reviewing process for the standard library. Currently there is not even an accepted style for modules ending up in the Python distribution. That, and a group of people, dedicated to standard library refactoring. The majority of libraries in the standard library are small and easy to understand, I'm sure they are perfectly suited for students on projects like GSOC or GHOP to work on. They could even be used as some sort of "playground" for new Python developers. Ubuntu recently started the "100 paper cuts" project. There people work on tiny little patches to improve the system, rather to replace components. Even though a large place of the standard library appears to be broken by design they could still be redesigned on the small scale, without breaking backwards compatibility. Of course libraries like `locale` and `logging` are hard to change, but it would still be possible. For `locale` it would probably a useful idea to go into the direction of datetime, where the timezone information is left to a 3rd party library. `locale` could provide some hooks for libraries like `babel` to fill the gap. On the other hand `Cookie` would be very easy to fix by moving the parsing code into a separate function and refactoring the cookie objects. We could probably also start a poll out there with well-selected questions of what users think about parts of the library. And for that poll it would make a lot of sense to not just ask the questions and evaluating the results, but also track the area the user is coming from (small size company, open / closed source, web development etc.). Because we all are biased and seeing results grouped by some of these factoids could be enlightening. That said, it could tell us that I'm completely wrong with my ideas of how the state of the standard library. But how realistic is it to refactor the standard library? I don't know. For a long time people were pretty sure Python will not get any faster and yet Unleaden Swallow is doing some really amazing progress. If we want to push Python foward into new areas, and the web is one of them, it is necessary to jump into the cold water and start things. Any maybe we should have some elected task forces for things like the standard library. Judging from the mailinglist it appears that far too many people are discussing *every detail* of it. It is a good idea to ask as many people as possible, but I am not sure if the mailinglist is the way to do that. It is currently very hard to see the direction in which development is heading. Please think of this email just as a suggestion. I don't have too much trust into myself to follow the discussions on this list camely enough to become a real part of a solution, but I would love to help shifting the development into a better direction, no matter which one it will be. Regards, Armin