[Python-ideas] stdlib upgrades

[Brett Cannon]

On Tue, Jun 1, 2010 at 11:13, Ian Bicking <ianb at colorstudy.com> wrote:
> Threading will probably break here as I wasn't on the list for the first
> email...
>
> My concern with the standard library is that there's a couple things going
> on:
>
> 1. The standard library represents "accepted" functionality, kind of best
> practice, kind of just conventional.  Everyone (roughly) knows what you are
> talking about when you use things from the standard library.
> 2. The standard library has some firm backward compatibility guarantees.  It
> also has some firm stability guarantees, especially within releases (though
> in practice, nearly for eternity).
> 3. The standard library is kind of collectively owned; it's not up to the
> whims of one person, and can't be abandoned.
> 4. The standard library is one big chunk of functionality, upgraded all
> under one version number, and specifically works together (though in
> practice cross-module refactorings are uncommon).
>
> There's positive things about these features, but 4 really drives me nuts,
> and I think is a strong disincentive to putting stuff into the standard
> library.  For packaging I think 4 actively damages maintainability.
>
> Packaging is at the intersection of several systems:
>
> * Python versions
> * Forward and backward compatibility with distributed libraries
> * System policies (e.g., Debian has changed things around a lot in the last
> few years)
> * A whole other ecosystem of libraries outside of Python (e.g., binding to C
> libraries)
> * Various developer toolkits, some Python specific (e.g., Cython) some not
> (gcc)
>
> I don't think it's practical to think that we can determine some scope of
> packaging where it will be stable in the long term, all these things are
> changing and many are changing without any particular concern for how it
> affects Python (i.e., packaging must be reactive).  And frankly we clearly
> do not have packaging figured out, we're still circling in on something...
> and I think the circling will be more like a Strange Attractor than a sink
> drain.
>
> The issues exist for other libraries that aren't packaging-related, of
> course, it's just worse for packaging.  argparse for instance is not
> "done"... it has bugs that won't be fixed before release, and functionality
> that it should reasonably include.  But there's no path for it to get
> better.  Will it have new and better features in Python 3.3?  Who seriously
> wants to write code that is only compatible with Python 3.3+ just because of
> some feature in argparse?  Instead everyone will work around argparse as it
> currently exists.  In the process they'll probably use undocumented APIs,
> further calcifying the library and making future improvements disruptive.
>
> It's not very specific to argparse, I think ElementTree has similar issues.
> The json library is fairly unique in that it has a scope that can be
> "done".  I don't know what to say about wsgiref... it's completely
> irrelevant in Python 3 because it was upgraded along the Python schedule
> despite being unready to be released (this is relatively harmless as I don't
> think anyone is using wsgiref in Python 3).
>
> So, this is the tension I see.  I think aspects of the standard library
> process and its guarantees are useful, but the current process means
> releasing code that isn't ready or not releasing code that should be
> released, and neither is good practice and both compromise those
> guarantees.  Lots of moving versions can indeed be difficult to manage...
> though it can be made a lot easier with good practices.  Though even then
> distutils2 (and pip) does not even fit into that... they both enter into the
> workflow before you start working with libraries and versions, making them
> somewhat unique (though also giving them some more flexibility as they are
> not so strongly tied to the Python runtime, which is where stability
> requirements are most needed).

I can only see two scenarios that might be considered acceptable to
address these issues.

One is that when new modules are accepted into the stdlib they are
flagged with a ExpermintalWarning so that people know that no
backwards-compatibility promises have been made yet. That gets the
module more exposure and gets python-dev real-world feedback to fix
issues before the module calcifies into a strong
backwards-compatibility. With that experience more proper decisions
can be made as to how to change things (e.g. the logging module's
default timestamp including microseconds which strptime cannot parse).

Otherwise we shift to an annual release schedule, but alternate Python
versions have a language moratorium. That would mean only new language
features every two years, but a new stdlib annually.

But one thing I can tell you is that having separate module releases
of what is in the stdlib under the same name or doing a separate
stdlib release will not happen. Python-dev as a whole does not like
this idea and I don't see that changing.