On Tue, Jun 1, 2010 at 4:54 AM, Tarek Ziadé wrote:

Hello,

That's not a new idea, but I'd like to throw it here again.

Some modules/packages in the stdlib are pretty isolated, which means that they could be upgraded with no harm, independently from the rest. For example the unittest package, or the email package.

Here's an idea:

1 - add a version number in each package or module of the stdlib that is potentially upgradable

2 - create standalone releases of these modules/packages at PyPI, in a restricted area 'stdlib upgrades'     that can be used only by core devs to upload new versions. Each release lists the precise     Python versions it's compatible with.

3 - once distutils2 is back in the stdlib, provide a command line interface to list upgradable packages, and make     it possible to upgrade them

4 - an upgraded package lands in a new specific site-packages directory and is loaded *before* the one in Lib

Regards Tarek

I dislike this more than I thought I would - I would rather have the stdlib broken out from core and have it have more releases than the whole of python then allowing for piecemeal "blessed" upgrades. Allowing piecemeal upgrades of the stdlib means you have to say something akin to: "I support Python 2.6, with the upgraded unittest (2.6.1.3), socket (2.6.1.2) and multiprocessing modules" And so on. Sure, API compatibility should be "fine" - but we all know that there are exceptions to the rule all the time, and that alone is enough to put the nix on allowing arbitrary upgrades of individual modules within the standard lib. For package authors, and users, the simple "I support 2.6" statement is key. For corporations with strict upgrade checks and verifications, the same applies. jesse

On Tue, Jun 1, 2010 at 4:46 PM, Jesse Noller wrote: [..]

I dislike this more than I thought I would - I would rather have the stdlib broken out from core and have it have more releases than the whole of python then allowing for piecemeal "blessed" upgrades. Allowing piecemeal upgrades of the stdlib means you have to say something akin to:

"I support Python 2.6, with the upgraded unittest (2.6.1.3), socket (2.6.1.2) and multiprocessing modules"

And so on. Sure, API compatibility should be "fine" - but we all know that there are exceptions to the rule all the time, and that alone is enough to put the nix on allowing arbitrary upgrades of individual modules within the standard lib. For package authors, and users, the simple "I support 2.6" statement is key. For corporations with strict upgrade checks and verifications, the same applies.

What I expect would be for some projects to state : "I support Python 2.6, with the upgraded unittest (2.6.1.3), or Python 3.2" Instead of: "I support Python 2.6, with unittest2 or Python 3.2 with its own unittest" Because the latter makes more work in the project itself, (and no difference on the corporation/end user side) where it has to deal with two different unittest versions. Well, the same, but with a different namespace that is used to be able to install it on previous Python versions besides the stdlib one. At some point, if a package or module in the stdlib evolve in a backward compatible way, it would be nice to be able to upgrade an existing Python installation. And this is going to be more and more true with the moratorium I guess: what people are creating now for Python should work in a wider range of Pythons. Now, releasing the stdlib on its own and shortening its cycle would also resolve the problem we have. But then, while there will be less combinations, the problems you have mentioned will remain the same. Just replace in your example "I support Python 2.6, with the upgraded unittest (2.6.1.3), socket (2.6.1.2) and multiprocessing modules" by "I support Python 2.6, with the upgraded stdlib 2.6.1.2". Regards, Tarek -- Tarek Ziadé | http://ziade.org

On Tue, Jun 1, 2010 at 5:40 PM, Tarek Ziadé wrote:

On Tue, Jun 1, 2010 at 4:46 PM, Jesse Noller wrote: [..]

...

I dislike this more than I thought I would - I would rather have the stdlib broken out from core and have it have more releases than the whole of python then allowing for piecemeal "blessed" upgrades. Allowing piecemeal upgrades of the stdlib means you have to say something akin to:

"I support Python 2.6, with the upgraded unittest (2.6.1.3), socket (2.6.1.2) and multiprocessing modules"

And so on. Sure, API compatibility should be "fine" - but we all know that there are exceptions to the rule all the time, and that alone is enough to put the nix on allowing arbitrary upgrades of individual modules within the standard lib. For package authors, and users, the simple "I support 2.6" statement is key. For corporations with strict upgrade checks and verifications, the same applies.

What I expect would be for some projects to state :

 "I support Python 2.6, with the upgraded unittest (2.6.1.3), or Python 3.2"

Instead of:

  "I support Python 2.6, with unittest2 or Python 3.2 with its own unittest"

Because the latter makes more work in the project itself, (and no difference on the corporation/end user side) where it has to deal with two different unittest versions. Well, the same, but with a different namespace that is used to be able to install it on previous Python versions besides the stdlib one.

Either fight is a losing one. In the first, you're requiring that someone *fundamentally alter* their standard library install to monkey patch in something with the same name, which means it may or may not break something else - which makes it a non zero risk, and therefore, unacceptable to a lot of people. The second requires that the user install an external package; which; until we include something as a standard, is a fools errand, only to be taken on by the bravest of people (that might be hyperbole ;)) In all seriousness - the second you ask people to alter one tiny sliver of the stdlib for the sake of your unique-snowflake project or app, you've lost. The stdlib is a critical piece of Python - and it's relative integrity is assumed when people download it from the python.org website. Asking them to download it from the site, and then possibly install piecemeal upgrades seems like a bad idea. Imagine a future where project dependencies look like this: Python 2.7.1 Python 2.7.1 with upgraded unittest Python 2.7.1 with upgraded unittest, socket, multiprocessing Python 2.7.1 with upgraded unittest, socket, multiprocessing, httplib And so on - sure, eventually (say, 6 months later) there might be a 2.7.2 with all of those changes rolled in, but that begs the question - why release them individually when you know there's another release coming shortly, and avoid the confusion?

At some point, if a package or module in the stdlib evolve in a backward compatible way, it would be nice to be able to upgrade an existing Python installation.

Yes, but wouldn't it also be nice to simply have a built in package installation script, and a shorter (say, 6 month) release cycle for patch releases which maintain the backwards compatibility guarantee? This way, bug fixes can move more quickly. We're really discussing the window from a release, to the next - which could easily be shortened lacking language changes (stdlib bugfixes only).

And this is going to be more and more true with the moratorium I guess: what people are creating now for Python should work in a wider range of Pythons.

Now, releasing the stdlib on its own and shortening its cycle would also resolve the problem we have. But then, while there will be less combinations, the problems you have mentioned will remain the same. Just replace in your example "I support Python 2.6, with the upgraded unittest (2.6.1.3), socket (2.6.1.2) and multiprocessing modules" by  "I support Python 2.6, with the upgraded stdlib 2.6.1.2".

Then don't fragment it - just release everything more rapidly. And then suffer from the fact that OS vendors don't pick up your releases quickly, and so on and so forth. All I'm trying to say, is that allowing piecemeal upgrades of stdlib modules is a risky prospect - I know plenty of people (myself included) who write code which is ONLY dependent on the standard library intentionally, to save ourselves from the packaging/installation/etc heartache. This isn't just because of the pain of installing, or dependency management issues - it's because the stdlib is known, stable and the one requirement we can rely on, other than the language itself. The stdlib doesn't require anyone to install from github, or bitbucket, or figure out distribute vs. distutils2 - it's just there, despite all it's warts and dusty corners. jesse

On Wed, Jun 2, 2010 at 5:13 AM, M.-A. Lemburg wrote:

While I played with this idea a long time ago as well, I have since found that it causes more trouble than it's worth.

Apart from having the user to maintain at least two different versioned packages (Python and (part of) the stdlib), it also causes problems if you use this Python installation for more than one project: it's easily possible to have project A require version 2 or a stdlib module and project B version 3 of that same module.

This exists for normal libraries currently, and using virtualenv I've found it to be manageable. It does require process separation (and sys.path separation) in some cases. I agree that global upgrades are dangerous. distutils2/pip may be different because projects don't generally get used except when managing a project, and very few projects will require any particular version of these libraries. If you then load both projects in an application, you end up

either with a broken project A or B (depending on whether you have version 2 or 3 of that stdlib module installed), or you allow loading multiple versions of the same module, in which case you will likely break you application, since it will find multiple class implementations (and objects) for the the same instances.

Things like exception catching, pickling (and esp. unpickling), security checks based on classes, interface adapters and even simply isinstance() checks would then fail in various hard to reproduce ways.

Yes, multiple versions of a library loaded at the same time is not a good idea.

IMHO, we've so far done well by issuing new Python patch level releases whenever there was a problem in the stdlib (and only then).

Introducing new features by way of updates is left to minor releases, which then require more testing by the user.

This additional testing is what causes many corporates to not follow the Python release cycle or skip a few minor releases: the work involved often just doesn't warrant the advantages of the added new features.

Yes, and so applications and libraries have to work around bugs instead of using fixed versions, generally making upgrades even more danger-prone. In the case of package management, the hardest libraries to support are those libraries that have included a large number of fixes for installation problems in their setup.py. Futzing around with most of the standard library right now would just add complexity, and applying changes that might be more aesthetic than functional would be a really bad choice and lead to tedious discussions. But new functionality can't usefully *just* exist in the standard library because basically no one is using 2.7, few people are using 3.x, and lots of people are using 2.5 or at best 2.6 -- so new functionality should be available to all those people. Which means there *has* to be releases of any new functionality. argparse was already released, and so there will be "argparse" out in the wild that anyone can install on any version of Python shadowing the existing module. unittest improvements are being released as unittest2... meaning I guess that the "proper" way to use that functionality would be: import sys if sys.version_info >= (2, 7): import unittest else: import unittest2 as unittest

The situations won't get any better if we start releasing partial or complete stdlib updates even more often.

That a stdlib release means potentially *any* part of the standard library could have been upgraded (even though it probably won't be) probably will throw people off. The advantage of versions on specific functionality is that you can upgrade just what you care about. It's much less burdensome to test something that actually fixes a problem for you, and of course people do that all the time with non-standard libraries. -- Ian Bicking | http://blog.ianbicking.org

Tarek Ziadé wrote:

Hello,

That's not a new idea, but I'd like to throw it here again.

Some modules/packages in the stdlib are pretty isolated, which means that they could be upgraded with no harm, independently from the rest. For example the unittest package, or the email package.

Here's an idea:

1 - add a version number in each package or module of the stdlib that is potentially upgradable

2 - create standalone releases of these modules/packages at PyPI, in a restricted area 'stdlib upgrades' that can be used only by core devs to upload new versions. Each release lists the precise Python versions it's compatible with.

Not a packaging expert, but I think in the context of a virtualenv this all makes sense. The ability to have a pip requirements file (for example) with stdlib-email==2.6 stdlib-unittest==2.7 would be a useful flexibility in my view. Any given application or library will only exercise a certain subset of stdlib after all. Also it might give you more confidence to upgrade to a higher python if you had this flexibility. Whether you wanted to incorporate this in the absence of a virtualenv is another question, I suppose.

4 - an upgraded package lands in a new specific site-packages directory and is loaded *before* the one in Lib

For a quick test, I added a "prioritize_site_packages" function to a virtualenv's site.py, which just rearranged sys.path so that anything containing the string 'site-packages' was prior to anything else. Would this be sufficient in the general case?