Moving development out of the standard library
OK... after a bit of off-list discussion I realize what I am really concerned about with respect to the standard library wasn't well expressed. So here's my real assertion: There is no reason any new library or functionality should be tied to a Python release. Outside of a few exceptions (like ast or importlib) functionality in the standard library seldom relies on anything in a particular Python release; e.g., code might use conditional expressions, but it never *has* to use conditional expressions. The standard library that most people know and love is really the least common denominator of Python's that person has to handle; for someone writing an open source library that's probably 2.5, for someone using Zope 2 that's traditionally been 2.4, and if you have a controlled environment (e.g., internal development) maybe you can do 2.6. I think there is a general consensus that functionality should not be tied to a Python release, but the results are ad hoc. That is, truly useful libraries that are added to the stdlib are backported, or more often were originally maintained as a library with backward compatibility before being integrated into the standard library. I think we should have a more formalized process about how this functionality is maintained, including a process that considers the years of ongoing maintenance and improvement that should happen on these libraries. (Most specifically without serious thought about this development process I am pessimistic about an orderly or positive inclusion of distutils2 in packaging workflows.) Another alternative is to simply not make improvements to the standard library beyond a very well-defined set of appropriate functionality. This would be much closer to the status quo. Defining what categories would be "appropriate" would be contentious, I am sure, but would sharply focus future discussions. -- Ian Bicking | http://blog.ianbicking.org
On Mon, 7 Jun 2010 11:35:16 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
I think there is a general consensus that functionality should not be tied to a Python release, but the results are ad hoc.
I disagree with this. Tying new functionality to a Python release vastly simplifies dependency management (instead of having to track the versions of N external libraries, sometimes with inter-dependencies).
Without any discussion of specifics, I find it hard to understand what you are concerned about.
On Mon, Jun 7, 2010 at 1:14 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I say there is consensus because as far as I know anything substantial has a maintained version outside the standard library; argparse is implicitly, unittest is unittest2, ElementTree always has maintained a separate existence, simplejson implicitly.
1. How will distutils2 updates be made available between Python releases? 2. How will distutils2 features be made available in older Python releases? 3. How will old standard library releases of distutils2 be managed? E.g., if pip starts using distutils2, at the time distutils2 ends up in the standard library there is a version of distutils2 I cannot simply reject as incompatible, meaning I can't make use of any new features or bug fixes. -- Ian Bicking | http://blog.ianbicking.org
On Mon, 7 Jun 2010 13:33:48 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
"Anything substantial" is more than exagerated. The modules you are mentioning are exceptions, two of which may even be temporary (argparse and unittest2). Most sdtlib modules don't have external releases, and many of them are still "substantial".
1. How will distutils2 updates be made available between Python releases? 2. How will distutils2 features be made available in older Python releases?
Why are you expecting any of these to happen? I don't know what Tarek intends to do in that respect, but he certainly doesn't have any moral obligation to do external releases. Regards Antoine.
On Mon, Jun 7, 2010 at 1:52 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Most other modules are very old. In cases where it hasn't happened, e.g., doctest in 2.4, I at least personally have had to backport that module on my own.
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 9:20 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Distutils2 will be provided for Python 2.4 to 3.x with frequent releases. Then, once it's added in 3.2 (maybe 3.3 if not ready by then), its release cycle will be driven by Python's one, with backports released at the same pace for Python versions that didn't have it in the stdlib. Regards Tarek -- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 2:36 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
So let's say distutils2 gets version parity with Python, so distutils2 3.3 is released with Python 3.3. Then distutils2 3.4 is released with Python 3.4, and is backported to all previous versions of Python... except for Python 3.3, which will always have distutils2 3.3? -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 9:39 PM, Ian Bicking <ianb@colorstudy.com> wrote:
yes, in other word, it's part of the stdlib
Distutils2 will behave exactly like other packages in the standard library. That is: - the development is going on in trunk - bug fixes are backported to older python releases that have this package present - some feature or small refactoring are also backported when it makes sense So Python 3.3.2 will have Distutils 3.3.2, but also os.path 3.3.2, and shutil 3.3.2 Like Python 3.4 its own versions. Now for older versions of Python, I will provide a backport at PyPI, so people can use it under Python 2.x. This backport will probably be made with the trunk so the 2.x line has the latest code. IOW the latest 2.7 release might be more advanced than the one provided in 3.3 for example, but I don't see this as a problem. Tarek -- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 2:57 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
It means that, at least for pip, distutils2 3.3 would be effectively the last version. If there are important bugs we'll have to work around them. If there are added features we'll have to ignore them. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 10:00 PM, Ian Bicking <ianb@colorstudy.com> wrote:
To make sure it's clear: The latest would be 3.4 here, with its backport in 2.7 and an older version in 3.3.
If there are important bugs we'll have to work around them. If there are added features we'll have to ignore them.
Not for the bug fixes because they will likely to be backported in all versions. (3.3 and 2.7) Now for new features, if pip uses the latest 2.x and the latest 3.x versions, you will get them. I am not sure why you would have to ignore them. You would probably want to use the new features when they are released, and still make your code work with older versions. This is not a new problem btw: if you want to support several versions of Python, you have to work around the differences. Example: There's a big bug in tarfile in Python 2.4, and I had to backport part of the 2.5 version for a while in my 2.4 projects. That's doesn't mean I don't want tarfile to be in the stdlib. Tarek
Tarek Ziadé wrote:
There's no way for the new features to show up in 3.3, is there? You can't add them to a micro release, and you can't replace a module in the standard library. I think that's Ian's point. pip could use the new features in 3.4, and it could get the new features in 2.x if the users were willing to install the updated library, since it's not in the stdlib. But for 3.3 you'd be stuck. -- Eric.
On 7 June 2010 22:20, Eric Smith <eric@trueblade.com> wrote:
But that's no different to pip using *any* standard library module. If you want to support Python 2.4 you can't use os.path.relpath (or you have to provide it yourself anyway) for example. Michael
On Mon, Jun 7, 2010 at 5:45 PM, Michael Foord <fuzzyman@voidspace.org.uk>wrote:
This is part of why I don't care about reforming or modifying what's in the standard library now -- I know the constraints well, and they can't be changed. I'm solely concerned about new functionality which need not repeat this pattern. -- Ian Bicking | http://blog.ianbicking.org
On Tue, Jun 8, 2010 at 3:49 AM, Ian Bicking <ianb@colorstudy.com> wrote:
Are you suggesting to freeze the stdlib development ? So you don't have to deal with different Python version at your level ? If so, that doesn't sound right because making the "batteries included" evolve is part (imho) of the Python spirit, and the constraints we are talking about right now is not a huge problem as you seem to think in my opinion. I don't find it extremely hard to cope with various Python version.
-- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 11:20 PM, Eric Smith <eric@trueblade.com> wrote:
Yes, I understood that. My point is that you can adapt your software when the Python version you use is an old version, and you don't have the latest feature. That's how we work with all modules/packages from the stdlib, and features are added at every Python version. The stdlib is not frozen.
Not stuck, but you will definitely need to deal with it in your project. And this is not new. That's why I have suggested earlier authorizing partial updates of the stdlib, but it seemed that the idea was disliked because of the complexity it would bring. Regards Tarek
-- Eric.
-- Tarek Ziadé | http://ziade.org
On 09/06/10 05:33, Terry Reedy wrote:
I believe Eric was pointing out the fact that, by default, there is no directory on sys.path that will override the standard library versions of a module or package for all applications using that interpreter installation. So you're forced to either resort to destructive replacement (actually overwriting the standard library module on disk) or else tinkering with sys.path in each app or a library to insert an "override" directory before the normal standard library paths. It sometimes seems to me that, for the advocates of a more granular standard library, proposing standardisation of such an override directory would be an interesting way to test the waters (since it would make it much easier to drop in backported updates to standard library modules). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Tue, Jun 8, 2010 at 3:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Setuptools uses .pth file hackery to handle this case (not to intentionally override the stdlib, but it would also do that), and I think it could apply here as well. Also the way Setuptools installs eggs might be useful in this case, as it makes it very obvious in tracebacks what version of a module is being used (and if it is from the standard library). Even if the particular mechanics were revisited, I think these basic ideas would be helpful. -- Ian Bicking | http://blog.ianbicking.org
On Tue, Jun 8, 2010 at 11:01 PM, Ian Bicking <ianb@colorstudy.com> wrote:
The problem is, any project would start overriding the stdlib to fix things or change some behavior, unless this is somehow controlled. In the proposal I've made earlier to update partially the stdlib, I've proposed to have a specific area at PyPI for distributions that are "blessed" to override the stdlib packages/modules.
Tarek Ziadé writes:
But in the end, that's precisely what you propose to do yourself with "partial stdlib upgrades"! It's just that you trust yourself more than you trust "any project". But that just doesn't fly from the point of the third party clients of the stdlib. Either stability of any particular version's stdlib applies to the stdlib developers too, or it doesn't really apply at all.
On Wed, Jun 9, 2010 at 2:30 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
If the maintainer of unittest for example, provides an upgrade for this package, don't you think we can trust that he will provide a more stable upgrade for the unittest package in the stdlib than another project that would implement a unittest package ? So no, I don't think you can compare a potential upgrade from a stdlib package maintainer with an upgrade issued from someone else. So, by "controlled" I mean releasing official upgrades of the stdlib, people know they were built by the same maintainers.
-- Tarek Ziadé | http://ziade.org
Tarek Ziadé writes:
For the users who really care about this, it's not a question of relative stability. Either the only changes in documented behavior involve successful completion of jobs that used to fail, or instability has been introduced. For many people (though a small fraction) that is *very bad*, and they have complained vociferously in the past. I really don't understand where the big benefits are to having minor improvements introduced in bugfix releases. People who want those benefits should upgrade to a more recent series. The people who really need them but must stick to an older series for Python itself can get the most recent version of the few packages that have "must-have" improvements from PyPI. "No behavior changes in micro releases" is an easily understood, reasonably easily followed policy. The policy you propose requires judgment calls that will differ from module maintainer to module maintainer, and every upgrade will involve discussion on python-dev. Yuck.
On Wed, Jun 9, 2010 at 10:25 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote: [..]
I really don't understand where the big benefits are to having minor improvements introduced in bugfix releases.
To try to find a solution to the problems described in this thread earlier. If I summarize so far the threads as I understood them, people don't want to rely on stdlib packages because their release cycles are too slow for what they want/need to do with it *today*. A package that enters the stdlib suffers from being slowed down. That's also a huge benefit for many reasons: stability, blessing, etc. The initial reason is that Ian doesn't want Pip do depend on distutils2 if it's in the stdlib, because he will have to cope with various versions of Python to make sure his users will have the same set of features I guess. So he needs to provide his own backports of any new distutils2 features. If we can find a way to facilitate this work, that would be great. IOW, if we can provide somehow a backport of these features so some projects can use it no matter what the python version is... And "not putting distutils2 in the stdlib" is not the solution because this is a problem for all packages in there. That's exactly what unitest currently do (but with a new name "unittest2") and as soon as Python 2.7 final will be out, unittest will have the same problem: it won't be able to backport new features anymore under the same namespace. [..]
This idea I have was kind of rejected earlier in Python-ideas, if you read back the two/three latest threads on the topic. I have reintroduced here because someone proposed to allow overriding stdlib packages. If this was to happen, I'd rather have upgrades decided in python-dev than arbitrary ones. But yes, that's too much burden... Maybe the status quo would is the best idea :) Regards Tarek
On 09/06/10 19:05, Tarek Ziadé wrote:
Something we may want to seriously consider is maintaining parallel releases of packages indefinitely when the benefits are deemed to justify the additional overheads. So, for example, many of the current features of unittest2 will be available in unittest in 2.7 and 3.2. This makes those features available to those that rely almost entirely on standard library functionality rather than third party packages. In the meantime, users of previous versions of Python can already use unittest2, and that package will work *without name conflicts* in both 2.7 and 3.2, even though many of its features have been added to the standard library. unittest2 may then even go through a few external releases before being synced up again with the standard library's unittest when 3.3 comes around. Something similar may turn out to be a good idea for distutils2: rather than consider the anticipated merge back into distutils prior to 3.3 the end of the road, instead continue to use distutils2 to release faster updates while evolving the API design towards 3.4. Users then have the choice - the solid, stable standard library version, or the distinctly named, more rapidly updated PyPI version. As others have suggested, this namespace separation approach could be standardised through the use of a PEP 382 namespace package so that users could choose between (e.g.) "unittest" and "distutils" and "cutting_edge.unittest" and "cutting_edge.distutils" (with the latter being the regularly updated, new features and all, PyPI versions, and the former the traditional stable API, bugfix-only standard library versions). That would probably be an improvement over the current ad hoc approach to naming separation for standard library updates. I don't see any way to ever resolve the two competing goal sets (stability vs latest features) without permanently maintaining separate namespaces. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Jun 9, 2010 at 7:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I had a very similar thought. Why not have all the real development of those packages happen outside of the standard lib and just grab the latest stable version when cutting a new version of Python. Namespaces are fine, but I'd be happy enough with a way for these packages to show up on the Python path before the stdlib. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek
On Wed, Jun 9, 2010 at 6:38 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
New releases could also be cut from the Python tree. I believe everyone here agrees that entering the standard library in any form should imply a greater sense of collective ownership of a package. -- Ian Bicking | http://blog.ianbicking.org
On Wed, Jun 9, 2010 at 08:17, Ian Bicking <ianb@colorstudy.com> wrote:
But this makes the assumption that core developers are going to choose to develop modules such that they can be released before the next release of Python goes out. There seems to be a lot of extrapolation from the fact that Michael takes the time to do unittest2 (which, now that I think about it, should probably have been named unittest_external or something as it really isn't a sequel to unittest, just an external release) and that Tarek is doing the initial development of distutils2 externally. Out of all the core developers that is not that many. Sure you could maybe toss in ElementTree, but that probably is it (e.g. simplejson only gets new releases when Bob fixes stuff for a new Python release or simply makes an external release of stdlib fixes). For me, importlib will never work in this environment. When new features in modules or the language come in I try to update code to use it when I can. That means that importlib is not going to have a stable release cycle outside of Python's. Nor am I going to be willing to change that practice as that makes development harder for me -- I don't want to have to check if some new feature has been made available externally -- and honestly more boring -- I like using the new features of the language as that helps make core development fun. The only way I see this working is for individual core developers to decide they want to keep a module dependent only on the last minor release of Python (or older). At that point you can try to convince the developers to use a common package name (stdlib, py, stdlib_ext, etc.) and then use namespace packages or pkgutil.extend_path to pull them in under the common package name to signify that they are "early" releases of the modules from the stdlib. That gives you an easier way to gauge usage as you can look at use of the package as a whole to see what kind of community pickup there is. If you can do that *and* show that there was a clear benefit to the community then you might have a chance to get more developers to participate in this development scheme. But in terms of a general python-dev policy, it simply won't happen as it is just too much extra work to force upon everyone (Nick seems to be the only one I can think of still throwing ideas out there, but even with that I don't know how much work he wants to put in).
On 10/06/10 05:52, Brett Cannon wrote:
Pure speculation as to ways this could be done such that it produces a net benefit for the Python ecosystem as a whole. I have close to zero interest in actually doing the work involved myself, since I'm one of those that just uses the standard library for stuff and doesn't worry if it isn't the latest and greatest. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On 09/06/10 21:34, David Stanek wrote:
Another parallel that occurs to me is the stable/testing/unstable distinction in Debian - if you want rock solid (but old) you stick with stable, if you want comparatively cutting edge you go with unstable and testing splits the difference. Ubuntu's "normal release" vs "Long Term Support release" distinction is also worth thinking about. Most home desktop users will upgrade every 6 months (modulo rocky transitions like the upgrade to KDE 4), while corporate users will wait for the next LTS to arrive. To bring that back to a Python context and revisit the proposal that Guido rejected for biennial Python releases with annual standard library releases, suppose that, instead of the standard library itself having annual releases, there was a python-dev maintained (or even third-party maintained, python-dev blessed) "pybackports" project. The idea being, that package would not only make many new standard library features of the current Python release available on widely used previous versions, but also make upcoming features of the *next* Python release available on the current version. The periods of support would be shorter to reflect the faster release Third party libraries and applications could then either target the "rock solid" market and depend solely on the standard library, or go for the "latest and greatest" crowd and require pybackports. Make no mistake, something like this would be quite a bit of work, but it would be far from impossible and could go a long way towards fighting the impression that the standard library is the place where modules go to die. TLDR version: Year A: Python 3.x release, pybackports 3.x.0 release Year B: No Python release, pybackports 3.x.5 release (with 3.y features) Year C: Python 3.y release, pybackports 3.y.0 release Year D: No Python release, pybackports 3.y.5 release (with 3.z features) ...etc Unlike Python itself, Pybackports would not provide bugfix support for prior releases (since it would only be for the latest-and-greatest crowd, those that want the longer support should stick with the standard library). Something to think about, anyway. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, 09 Jun 2010 22:13:51 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Isn't that "impression" largely constructed, and propagated by a limited number of people who apparently don't like the very idea of a "batteries included" stdlib? There has been an amount of anti-stdlib activism (including in this thread) that I find both antagonizing and unconstructive. Outside of that vocal minority, there doesn't seem to be that much criticism against the stdlib. The reality is that there are regularly feature requests on the tracker, and many of them get accepted and committed (of course, when no patch is submitted and no core developer is interested, things have a tendency to linger on; but it's the same for outside libraries too; take a look at bug trackers for e.g. nose or twisted, and you'll see many open entries that have been resting for years). Regards Antoine.
On 09/06/10 22:40, Antoine Pitrou wrote:
The "where modules go to die" version of it is overstated, but the standard library definitely evolves more slowly than many third party packages. To use numpy as an example (just going off their SF file dates): Dec 2007: 1.0.4 May 2008: 1.1.0 Sep 2008: 1.2.0 Jul 2009: 1.3.0 Apr 2010: 1.4.1 Faster cycle times allow developers to be much more responsive to feedback when changes don't turn out as well as was hoped. The comparatively slow evolution of the standard library is the grain of truth that underlies the exaggerated form. The trick is to explore avenues that make these faster cycle times available to those that want them, while still providing a stable foundation for those that need it. It is exactly this situation that the Ubuntu release cycle is designed around: regular 6-monthly releases for most people, less frequent Long Term Support releases for those that need stability. That said, the status quo, with ad hoc porting to PyPI by module maintainers that consider doing so to be worthwhile is certainly a viable option going forward as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Jun 9, 2010 at 8:40 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I don't think so. I've no particular dislike of "batteries included" per se. What I don't like is dealing with packages that may or may not be in the standard library (that affects the requirements for my software), or that may have different names or update policies depending on whether they're part of the standard library (because that affects my code, always negatively).
Unconstructive in what way? Writing cross-Python-version code that deals with the differences between the stdlib and 3rd-party versions of packages is certainly unconstructive, but that's an argument to avoid moving packages into the standard library. One thing that seems to be happening is that the so-called "vocal minority" is growing. I think that should be expected as the acceptance of Python and applications built on it gain wider penetration. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Le mercredi 09 juin 2010 à 09:19 -0400, Fred Drake a écrit :
I don't understand what you mean. Are these packages in the stdlib or aren't they? It can't be both.
I don't see how, really. You make it sound like you have to deal with several versions of the stdlib, but not with several versions of external packages. I wonder why: do you force your users to install version X of module Y? More generally, it seems that people are reproaching things to the stdlib that are equally true for non-stdlib modules. This is quite bewildering to me.
On Wed, Jun 9, 2010 at 3:10 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Yes, as N grows, any constant fraction f times N will grow. In fact, f*N can grow even if f shrinks. but just more slowly than N grows.
True. Since there's no empirical measure of either N (number of Python users) or VM (the Vocal Minority), it's hard to tell if f is a constant fraction or something more (or less) interesting. I also suspect that VM is less likely to be readily measurable than N. For those in the VM, the severity of their objections only increases over time, until they exit both N and VM. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 9 June 2010 13:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
I agree - I think the "where modules go to die" argument is very overstated (but sadly, just as a result of repetition, it seems to be gaining traction :-(). Certainly, stdlib modules evolve at a slower rate than 3rd party ones in many cases. But they do evolve, as Antoine points out, and the slower evolution can just as easily be viewed as stability. What I don't understand is why the "activists" actually care if the stdlib is big or small. Surely if you don't like the "fat stdlib", just ignore it? Why inconvenience those of us who find it a benefit? So can someone clarify (from the point of view of a "thin stdlib" proponent) - what is the benefit to you, personally (i.e., ignoring things like "frees up core developer time" - let them speak for themselves if they feel that is a benefit), of actually removing items from the stdlib, rather than just ignoring them? Paul.
On Wed, Jun 9, 2010 at 7:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
And this provides an opportunity to bring the workload back under control, as well: if we continue the non-stdlib releases for these packages, all we really need to do is remember not to add them to the stdlib in the first place. The one case where we clearly get some win by having the newer package rolled in is distutils2, since that's really about a core service that needs to be available for everyone. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 6/9/2010 5:05 AM, Tarek Ziadé wrote:
I do not see that changing the 'no new features in micro (bugfix) releases' policy would solve Ian's problem at all. Suppose unittest2 were sufficiently done in September to go into 3.2 as unittest. Suppose the policy were changed and unittest2 were also backported into (3.1.final) and 2.7.1. That still would not help Ian with respect to 2.6, 2.5, 2.4, and however far back he wants to support. Since whatever solution he uses for 2.6- should also work for 2.7 (and 3.1) what is the use? Perhaps the reason for the policy needs to be restated: if new features are introduced in every x.y.z release, then 'Python x.y' has no particular meaning. This *was* the case somewhat during Python 1 and early Python 2 days, when the choice between moving from x.y.z to either x.y.(z+1) and x.(y+1) was somewhat arbitrary. (The move from '1.6' to '2.0' rather than '1.7' was for legal reasons only. Python2 really began with 2.2) The precipitating event for the new policy was the introduction of bool() and prebound True and False late in 2.2 series. People who download the latest '2.2' release and who used the new feature found that their code would not run on the great majority of '2.2' installations. It caused enough problems for enough people that Guido decided that he should have waited for 2.3 for official introduction, and perhaps first released bool as a separate module for earlier use. People who propose to change the policy back (are part way back) to what it used to be should at least be aware that it is a reversion and that there are reasons, and not just arbitrary whim or accidental happenstance, for the change. Terry Jan Reedy
On Mon, Jun 7, 2010 at 3:40 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The latest version pip could *depend on* would be 3.3, meaning all subsequent releases would be relatively unimportant. If you have to work around a known-bad version of something, then all later versions become liabilities instead of improvements (unless the workaround is egregiously ugly). I'd be okay saying that pip could require 3.4+, but not if it means Python 3.3 users would be excluded.
In minor releases, which are new Python releases and take a long time to be widely enough distributed to depend on the bug fix. Though admittedly less time than waiting for a major release to die. Now for new features, if pip uses the latest 2.x and the latest 3.x
A feature that may or may not be available is not a useful feature. We'll just have to backport distutils2 features in an ad hoc way using conditional imports and other nuisances. This is not a new problem btw: if you want to support several versions
of Python, you have to work around the differences.
Yes, but because the standard library changes so little it's not too bad, and in some cases we can rely on backports, and otherwise we simply ignore new functionality. Example: There's a big bug in tarfile in Python 2.4, and I had to
Instead of conditionally monkeypatching tarfile, I'd rather I just had a known-good version of tarfile. And maybe in a sense that is a solution; why try to patch tarfile at all, why not just include swaths of the standard library inline with libraries? Right now typically in projects I've noticed we carefully tease apart a libraries bugs when monkeypatching in an upgrade... but that's probably not worth it. OTOH, I don't think people would be happy if I just included all of distutils2 in pip with some sys.path magic to "upgrade" distutils2 as needed. But then... it might be the most reasonable approach. Or another option, allow versioning of portions of the standard library as need demands. Versioned portions of the standard library may still be quite constrained with respect to backward compatibility, but at least there would be an orderly way to handle backports and for libraries to require bugfixes instead of monkeypatching them in. Maybe an additional constraint would be that all new features have to be in new modules or via new names, and so upgrades would be additive and less likely to affect backward compatibility. And we just keep bad stuff out of the standard library (since perhaps the PyXML lessons is conflate a namespace issue with what was simply a lot of bad code). As an example of how this might have worked, unittest enhancements would have been in a separate module or with as a TestCase subclass (TestCase2) or something else highly unintrusive, and the result could be installed in any version of Python with little danger of conflicts. I.e., API versioning (for the standard library only) gets pushed into module and class names and isn't separate metadata. -- Ian Bicking | http://blog.ianbicking.org
On 7 June 2010 20:20, Ian Bicking <ianb@colorstudy.com> wrote:
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant.
I find it hard to interpret this in any way that doesn't pretty much imply "Python 3 is irrelevant". While I disagree with that, I suspect it isn't what you mean. So can you clarify? Paul.
On Mon, Jun 7, 2010 at 2:42 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It's kind of a moot point as Tarek isn't planning to only support Python 3. But developing packaging libraries for *only* Python 3 would mean that an alternate ecosystem would have to continue to support Python 2, and that alternate ecosystem would produce superior tools because that's where all the users actually are. For the next few years at least Python 3 needs to ride on Python 2's coat tails if it's going to keep up. -- Ian Bicking | http://blog.ianbicking.org
On Mon, 7 Jun 2010 14:20:41 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
Well, even if that's true (I haven't checked and I guess we wouldn't agree on the meaning of "old"), so what? I guess what I'm asking is: what is your line of reasoning? You started with a contention that: “There is no reason any new library or functionality should be tied to a Python release” and, in my humble opinion, you failed to demonstrate that. In particular, you haven't replied to my argument that it dramatically eases dependency management.
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant.
That's your opinion and I guess some people would disagree. Besides, decreeing that distutils2 be kept out of the stdlib won't make it its code magically compatible with Python 2.x. Regards Antoine.
On Mon, Jun 7, 2010 at 2:56 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
It only eases dependency management in closed systems with a fixed Python version. If you support more than one Python version than you still have dependency management, it is just tied to ad hoc workarounds when there are compatibility problems, or ignoring new functionality. The importance of "old" vs. "new" modules is that people tend to have a lowest version of Python they support, as simply a hard stop. This is currently Python 2.5 for most people, 2.4 for some groups (and just a very few stragglers with 2.3). So long as you never use anything beyond what 2.5 provides then it's okay, which is most of the standard library. I am not aware of anything added since 2.5 that isn't backported or previously available as a separate library (I'm sure there's *something*, just nothing I can think of). There is no clear policy about how backports are managed. There's some disfavor for backporting under the same name (PyXML being a primary source of this disfavor), but implicitly argparse *will* be backported as such as it is already installed under argparse and that name matches the stdlib name. It's unclear what should happen when you install a backport in a version of Python that already has the module. E.g., if distutils2 is distributed as distutils2 it would be able to override the standard library when installed unless there was code specifically to disallow it. Also we have implicit dependency management and versioning already because a few libraries in the standard library have .egg-info files installed. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 10:09 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Also we have implicit dependency management and versioning already because a few libraries in the standard library have .egg-info files installed.
If you are talking about wsgiref, that's the only one that has the egg-info in the stdlib, and it will go away soonish (probably in 3.2) since: - this format will be replaced by the new dist-info (PEP 376). - this was a mistake imho to add it since wsgiref was not a distribution in that case
-- Tarek Ziadé | http://ziade.org
On Mon, 7 Jun 2010 15:09:36 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
We're misunderstanding each other. I'm talking about dependency management for users (or application developers), you seem to be talking about dependency management for library developers. As for "ad hoc workarounds" and various "compatibility problems", they wouldn't disappear if the stdlib became smaller; I have trouble understanding what kind of solution you think would eliminate these issues.
The importance of "old" vs. "new" modules is that people tend to have a lowest version of Python they support, as simply a hard stop.
Yes, and it's the same for external modules too. For example, they will support Twisted 8.0 and upper. So what's the difference?
Really, I'm too lazy to go and read the changelogs, but there definitely are many improvements that are not available in Python 2.5 and older.
There is no clear policy about how backports are managed.
That's because, contrary to what you seem to think, external backports are the exception and not the rule. Regards Antoine.
On Mon, Jun 7, 2010 at 4:27 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I offered examples, you were too lazy to read the changelogs, your sweeping declaration does not seem justified.
From What's New in Python 2.6:
contextlib: attached to a language feature. multiprocessing: backported new string formatting: though a method, it'd be easy to produce in a module form. I'm not aware of a backport. abstract base classes: I don't think this can be backported wouldn't all kinds of contortions ast: associated with the implementation/language json: backported (implicitly, as it is simplejson) plistlib: I'm guessing this was added to support packaging. It exists separately. What's New in Python 2.7: argparse: implicitly backported changes to logging: not sure what will happen with this; the module has been backported in the past memoryview: not aware of it being backported importlib: technically including it in 2.7 is a backport, but otherwise no ttk: appears to be backported (http://pypi.python.org/pypi/pyttk/) unittest: backported ElementTree: backported Digging deeper into 2.5: functools: apparently backported at one time, now defunct ctypes: the backport appears to be dead sqlite3: actively developed (different name?) wsgiref: backported hashlib: backported Every release there's some additions to collections, which have not been backported. So in summary, of 17 additions which seemed "backportable" to me (not counting 3 modules that seemed tied to language features): * 4 were not backported * 3 have defunct or ambiguous backports * 10 were backported -- Ian Bicking | http://blog.ianbicking.org
Le lundi 07 juin 2010 à 17:16 -0500, Ian Bicking a écrit :
I offered examples, you were too lazy to read the changelogs, your sweeping declaration does not seem justified.
Sure, but my sweeping declaration is justified by the fact that I'm a daily contributor to Python core and know what kinds of things happen here.
What's New in Python 2.7: [snip]
Your list seems to forget lots of module-specific improvements. There are many more things in http://docs.python.org/dev/whatsnew/2.7.html#new-and-improved-modules , and most of them aren't "backported" in any fashion.
Of course this all depends on your definition of "backportable". If you remove that arbitrary conditional, the fact remains that most improvements, small or big, don't get backported. Regards Antoine.
On 7 June 2010 22:09, Ian Bicking <ianb@colorstudy.com> wrote:
Which gives me an idea: What if all the backports were managed within a single PyPI package, (e.g. backport26, backport27) with clear policy on older Python versions supported. then I could write in my py2.4 script: from backport26.os.path import relpath Konrad
Konrad Delong writes:
This is an interesting idea. I thought about something similar, but I decided that this would basically end up being the same problem as managing the stdlib (I could be wrong, of course, but I'm reasonably confident of that :-), while decreasing returns would have long since set in. Ie,q even though the effort would probably be quite a bit smaller than managing the stdlib itself, the benefits would decrease more rapidly (that's just a wild-ass guess, and since you bring it up, I'm curious to see what folks will say).
On 9 June 2010 10:20, Konrad Delong <konryd@gmail.com> wrote:
It's not a bad idea. A key benefit seems to be that it can be done by anyone, whether or not they are a core developer. So it can be set up right now, without taking up any of the limited core-dev resource. Of course, conversely, the disadvantage is that nobody's done this already, implying that either nobody's thought of it before or there's actually little motivation for someone to put the work into such a solution :-) If nobody's thought of it, you may be lucky and someone will pick up the idea and run with it. Paul.
Yeah, I am aware of that :-) Another question is whether such a package is going to find use. I doubt Michael would introduce a dependency into unittest2 just to get functools.wrap and os.path.relpath out of his code. Distutils2 contains a whole _backport module [1] which could go away, but again: at the cost of introducing dependency into the package. Konrad [1] http://bitbucket.org/tarek/distutils2/src/tip/src/distutils2/_backport/
On Mon, Jun 7, 2010 at 1:42 PM, Brett Cannon <brett@python.org> wrote:
Indeed, each person projects different ideas and motivations onto the standard library and I don't see a great deal of shared understanding about what it is. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 8:50 PM, Ian Bicking <ianb@colorstudy.com> wrote:
There's one thing that is clear though: Distutils is in the standard library, and it would be a non sense not to have Distutils2 included to replace it. While I understand your motivations not to see Pip included in the standard library, I will strongly object the exclusion Distutils2 from the standard library. I have agreed to temporarily develop Distutils2 outside the stdlib with the sole condition that it will be included back as soon as it is ready, because we badly need such a system in a vanilla Python. Distutils2 is the place where we are implementing the PEP that where accepted lately, and its inclusion to the standard library will provide a working packaging system for Python (that is *batteries included*) and a blessed playground for third party packaging tools. Regards Tarek -- Tarek Ziadé | http://ziade.org
On 08/06/10 02:35, Ian Bicking wrote:
Rather than rehash the point Antoine has already made regarding dependency management, I'll mention the other key benefit of standard library inclusion: Inclusion of a module or package in the standard library makes sense when the benefits of having "One Obvious Way To Do It" simplifies teaching of Python and development and maintenance of future Python code sufficiently to justify the slower rate of release associated with standard library inclusion. I'm one of those that believes that the volume of all currently written Python code is a small subset of the Python code that will ever be written, hence it makes sense to "raise the bar" in improving the quality of the baseline toolset provided to developers. Additions like argparse and the new futures module in PEP 3148, as well as historical additions like itertools, collections, ElementTree, simplejson, etc all serve that purpose well. However, I *also* like the pattern that has emerged of many standard library modules being kept backwards compatible with previous Python releases, and sometimes even being separately released on PyPI (or elsewhere). This approach allows the "one obvious way" to be extended back to earlier Python versions, since eventual standard library inclusion is a big point in a module or package's favour, even if a developer isn't currently using the latest Python release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mon, 7 Jun 2010 11:35:16 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
I think there is a general consensus that functionality should not be tied to a Python release, but the results are ad hoc.
I disagree with this. Tying new functionality to a Python release vastly simplifies dependency management (instead of having to track the versions of N external libraries, sometimes with inter-dependencies).
Without any discussion of specifics, I find it hard to understand what you are concerned about.
On Mon, Jun 7, 2010 at 1:14 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I say there is consensus because as far as I know anything substantial has a maintained version outside the standard library; argparse is implicitly, unittest is unittest2, ElementTree always has maintained a separate existence, simplejson implicitly.
1. How will distutils2 updates be made available between Python releases? 2. How will distutils2 features be made available in older Python releases? 3. How will old standard library releases of distutils2 be managed? E.g., if pip starts using distutils2, at the time distutils2 ends up in the standard library there is a version of distutils2 I cannot simply reject as incompatible, meaning I can't make use of any new features or bug fixes. -- Ian Bicking | http://blog.ianbicking.org
On Mon, 7 Jun 2010 13:33:48 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
"Anything substantial" is more than exagerated. The modules you are mentioning are exceptions, two of which may even be temporary (argparse and unittest2). Most sdtlib modules don't have external releases, and many of them are still "substantial".
1. How will distutils2 updates be made available between Python releases? 2. How will distutils2 features be made available in older Python releases?
Why are you expecting any of these to happen? I don't know what Tarek intends to do in that respect, but he certainly doesn't have any moral obligation to do external releases. Regards Antoine.
On Mon, Jun 7, 2010 at 1:52 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Most other modules are very old. In cases where it hasn't happened, e.g., doctest in 2.4, I at least personally have had to backport that module on my own.
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 9:20 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Distutils2 will be provided for Python 2.4 to 3.x with frequent releases. Then, once it's added in 3.2 (maybe 3.3 if not ready by then), its release cycle will be driven by Python's one, with backports released at the same pace for Python versions that didn't have it in the stdlib. Regards Tarek -- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 2:36 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
So let's say distutils2 gets version parity with Python, so distutils2 3.3 is released with Python 3.3. Then distutils2 3.4 is released with Python 3.4, and is backported to all previous versions of Python... except for Python 3.3, which will always have distutils2 3.3? -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 9:39 PM, Ian Bicking <ianb@colorstudy.com> wrote:
yes, in other word, it's part of the stdlib
Distutils2 will behave exactly like other packages in the standard library. That is: - the development is going on in trunk - bug fixes are backported to older python releases that have this package present - some feature or small refactoring are also backported when it makes sense So Python 3.3.2 will have Distutils 3.3.2, but also os.path 3.3.2, and shutil 3.3.2 Like Python 3.4 its own versions. Now for older versions of Python, I will provide a backport at PyPI, so people can use it under Python 2.x. This backport will probably be made with the trunk so the 2.x line has the latest code. IOW the latest 2.7 release might be more advanced than the one provided in 3.3 for example, but I don't see this as a problem. Tarek -- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 2:57 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
It means that, at least for pip, distutils2 3.3 would be effectively the last version. If there are important bugs we'll have to work around them. If there are added features we'll have to ignore them. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 10:00 PM, Ian Bicking <ianb@colorstudy.com> wrote:
To make sure it's clear: The latest would be 3.4 here, with its backport in 2.7 and an older version in 3.3.
If there are important bugs we'll have to work around them. If there are added features we'll have to ignore them.
Not for the bug fixes because they will likely to be backported in all versions. (3.3 and 2.7) Now for new features, if pip uses the latest 2.x and the latest 3.x versions, you will get them. I am not sure why you would have to ignore them. You would probably want to use the new features when they are released, and still make your code work with older versions. This is not a new problem btw: if you want to support several versions of Python, you have to work around the differences. Example: There's a big bug in tarfile in Python 2.4, and I had to backport part of the 2.5 version for a while in my 2.4 projects. That's doesn't mean I don't want tarfile to be in the stdlib. Tarek
Tarek Ziadé wrote:
There's no way for the new features to show up in 3.3, is there? You can't add them to a micro release, and you can't replace a module in the standard library. I think that's Ian's point. pip could use the new features in 3.4, and it could get the new features in 2.x if the users were willing to install the updated library, since it's not in the stdlib. But for 3.3 you'd be stuck. -- Eric.
On 7 June 2010 22:20, Eric Smith <eric@trueblade.com> wrote:
But that's no different to pip using *any* standard library module. If you want to support Python 2.4 you can't use os.path.relpath (or you have to provide it yourself anyway) for example. Michael
On Mon, Jun 7, 2010 at 5:45 PM, Michael Foord <fuzzyman@voidspace.org.uk>wrote:
This is part of why I don't care about reforming or modifying what's in the standard library now -- I know the constraints well, and they can't be changed. I'm solely concerned about new functionality which need not repeat this pattern. -- Ian Bicking | http://blog.ianbicking.org
On Tue, Jun 8, 2010 at 3:49 AM, Ian Bicking <ianb@colorstudy.com> wrote:
Are you suggesting to freeze the stdlib development ? So you don't have to deal with different Python version at your level ? If so, that doesn't sound right because making the "batteries included" evolve is part (imho) of the Python spirit, and the constraints we are talking about right now is not a huge problem as you seem to think in my opinion. I don't find it extremely hard to cope with various Python version.
-- Tarek Ziadé | http://ziade.org
On Mon, Jun 7, 2010 at 11:20 PM, Eric Smith <eric@trueblade.com> wrote:
Yes, I understood that. My point is that you can adapt your software when the Python version you use is an old version, and you don't have the latest feature. That's how we work with all modules/packages from the stdlib, and features are added at every Python version. The stdlib is not frozen.
Not stuck, but you will definitely need to deal with it in your project. And this is not new. That's why I have suggested earlier authorizing partial updates of the stdlib, but it seemed that the idea was disliked because of the complexity it would bring. Regards Tarek
-- Eric.
-- Tarek Ziadé | http://ziade.org
On 09/06/10 05:33, Terry Reedy wrote:
I believe Eric was pointing out the fact that, by default, there is no directory on sys.path that will override the standard library versions of a module or package for all applications using that interpreter installation. So you're forced to either resort to destructive replacement (actually overwriting the standard library module on disk) or else tinkering with sys.path in each app or a library to insert an "override" directory before the normal standard library paths. It sometimes seems to me that, for the advocates of a more granular standard library, proposing standardisation of such an override directory would be an interesting way to test the waters (since it would make it much easier to drop in backported updates to standard library modules). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Tue, Jun 8, 2010 at 3:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Setuptools uses .pth file hackery to handle this case (not to intentionally override the stdlib, but it would also do that), and I think it could apply here as well. Also the way Setuptools installs eggs might be useful in this case, as it makes it very obvious in tracebacks what version of a module is being used (and if it is from the standard library). Even if the particular mechanics were revisited, I think these basic ideas would be helpful. -- Ian Bicking | http://blog.ianbicking.org
On Tue, Jun 8, 2010 at 11:01 PM, Ian Bicking <ianb@colorstudy.com> wrote:
The problem is, any project would start overriding the stdlib to fix things or change some behavior, unless this is somehow controlled. In the proposal I've made earlier to update partially the stdlib, I've proposed to have a specific area at PyPI for distributions that are "blessed" to override the stdlib packages/modules.
Tarek Ziadé writes:
But in the end, that's precisely what you propose to do yourself with "partial stdlib upgrades"! It's just that you trust yourself more than you trust "any project". But that just doesn't fly from the point of the third party clients of the stdlib. Either stability of any particular version's stdlib applies to the stdlib developers too, or it doesn't really apply at all.
On Wed, Jun 9, 2010 at 2:30 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
If the maintainer of unittest for example, provides an upgrade for this package, don't you think we can trust that he will provide a more stable upgrade for the unittest package in the stdlib than another project that would implement a unittest package ? So no, I don't think you can compare a potential upgrade from a stdlib package maintainer with an upgrade issued from someone else. So, by "controlled" I mean releasing official upgrades of the stdlib, people know they were built by the same maintainers.
-- Tarek Ziadé | http://ziade.org
Tarek Ziadé writes:
For the users who really care about this, it's not a question of relative stability. Either the only changes in documented behavior involve successful completion of jobs that used to fail, or instability has been introduced. For many people (though a small fraction) that is *very bad*, and they have complained vociferously in the past. I really don't understand where the big benefits are to having minor improvements introduced in bugfix releases. People who want those benefits should upgrade to a more recent series. The people who really need them but must stick to an older series for Python itself can get the most recent version of the few packages that have "must-have" improvements from PyPI. "No behavior changes in micro releases" is an easily understood, reasonably easily followed policy. The policy you propose requires judgment calls that will differ from module maintainer to module maintainer, and every upgrade will involve discussion on python-dev. Yuck.
On Wed, Jun 9, 2010 at 10:25 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote: [..]
I really don't understand where the big benefits are to having minor improvements introduced in bugfix releases.
To try to find a solution to the problems described in this thread earlier. If I summarize so far the threads as I understood them, people don't want to rely on stdlib packages because their release cycles are too slow for what they want/need to do with it *today*. A package that enters the stdlib suffers from being slowed down. That's also a huge benefit for many reasons: stability, blessing, etc. The initial reason is that Ian doesn't want Pip do depend on distutils2 if it's in the stdlib, because he will have to cope with various versions of Python to make sure his users will have the same set of features I guess. So he needs to provide his own backports of any new distutils2 features. If we can find a way to facilitate this work, that would be great. IOW, if we can provide somehow a backport of these features so some projects can use it no matter what the python version is... And "not putting distutils2 in the stdlib" is not the solution because this is a problem for all packages in there. That's exactly what unitest currently do (but with a new name "unittest2") and as soon as Python 2.7 final will be out, unittest will have the same problem: it won't be able to backport new features anymore under the same namespace. [..]
This idea I have was kind of rejected earlier in Python-ideas, if you read back the two/three latest threads on the topic. I have reintroduced here because someone proposed to allow overriding stdlib packages. If this was to happen, I'd rather have upgrades decided in python-dev than arbitrary ones. But yes, that's too much burden... Maybe the status quo would is the best idea :) Regards Tarek
On 09/06/10 19:05, Tarek Ziadé wrote:
Something we may want to seriously consider is maintaining parallel releases of packages indefinitely when the benefits are deemed to justify the additional overheads. So, for example, many of the current features of unittest2 will be available in unittest in 2.7 and 3.2. This makes those features available to those that rely almost entirely on standard library functionality rather than third party packages. In the meantime, users of previous versions of Python can already use unittest2, and that package will work *without name conflicts* in both 2.7 and 3.2, even though many of its features have been added to the standard library. unittest2 may then even go through a few external releases before being synced up again with the standard library's unittest when 3.3 comes around. Something similar may turn out to be a good idea for distutils2: rather than consider the anticipated merge back into distutils prior to 3.3 the end of the road, instead continue to use distutils2 to release faster updates while evolving the API design towards 3.4. Users then have the choice - the solid, stable standard library version, or the distinctly named, more rapidly updated PyPI version. As others have suggested, this namespace separation approach could be standardised through the use of a PEP 382 namespace package so that users could choose between (e.g.) "unittest" and "distutils" and "cutting_edge.unittest" and "cutting_edge.distutils" (with the latter being the regularly updated, new features and all, PyPI versions, and the former the traditional stable API, bugfix-only standard library versions). That would probably be an improvement over the current ad hoc approach to naming separation for standard library updates. I don't see any way to ever resolve the two competing goal sets (stability vs latest features) without permanently maintaining separate namespaces. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Jun 9, 2010 at 7:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I had a very similar thought. Why not have all the real development of those packages happen outside of the standard lib and just grab the latest stable version when cutting a new version of Python. Namespaces are fine, but I'd be happy enough with a way for these packages to show up on the Python path before the stdlib. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek
On Wed, 9 Jun 2010 07:34:56 -0400 David Stanek <dstanek@dstanek.com> wrote:
-1. We have had too much trouble with externally-maintained modules such as elementtree and json. The Python SVN (or hg) tree should be the primary place where development takes place. Regards Antoine.
On Wed, Jun 9, 2010 at 6:38 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
New releases could also be cut from the Python tree. I believe everyone here agrees that entering the standard library in any form should imply a greater sense of collective ownership of a package. -- Ian Bicking | http://blog.ianbicking.org
On Wed, Jun 9, 2010 at 08:17, Ian Bicking <ianb@colorstudy.com> wrote:
But this makes the assumption that core developers are going to choose to develop modules such that they can be released before the next release of Python goes out. There seems to be a lot of extrapolation from the fact that Michael takes the time to do unittest2 (which, now that I think about it, should probably have been named unittest_external or something as it really isn't a sequel to unittest, just an external release) and that Tarek is doing the initial development of distutils2 externally. Out of all the core developers that is not that many. Sure you could maybe toss in ElementTree, but that probably is it (e.g. simplejson only gets new releases when Bob fixes stuff for a new Python release or simply makes an external release of stdlib fixes). For me, importlib will never work in this environment. When new features in modules or the language come in I try to update code to use it when I can. That means that importlib is not going to have a stable release cycle outside of Python's. Nor am I going to be willing to change that practice as that makes development harder for me -- I don't want to have to check if some new feature has been made available externally -- and honestly more boring -- I like using the new features of the language as that helps make core development fun. The only way I see this working is for individual core developers to decide they want to keep a module dependent only on the last minor release of Python (or older). At that point you can try to convince the developers to use a common package name (stdlib, py, stdlib_ext, etc.) and then use namespace packages or pkgutil.extend_path to pull them in under the common package name to signify that they are "early" releases of the modules from the stdlib. That gives you an easier way to gauge usage as you can look at use of the package as a whole to see what kind of community pickup there is. If you can do that *and* show that there was a clear benefit to the community then you might have a chance to get more developers to participate in this development scheme. But in terms of a general python-dev policy, it simply won't happen as it is just too much extra work to force upon everyone (Nick seems to be the only one I can think of still throwing ideas out there, but even with that I don't know how much work he wants to put in).
On 10/06/10 05:52, Brett Cannon wrote:
Pure speculation as to ways this could be done such that it produces a net benefit for the Python ecosystem as a whole. I have close to zero interest in actually doing the work involved myself, since I'm one of those that just uses the standard library for stuff and doesn't worry if it isn't the latest and greatest. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On 09/06/10 21:34, David Stanek wrote:
Another parallel that occurs to me is the stable/testing/unstable distinction in Debian - if you want rock solid (but old) you stick with stable, if you want comparatively cutting edge you go with unstable and testing splits the difference. Ubuntu's "normal release" vs "Long Term Support release" distinction is also worth thinking about. Most home desktop users will upgrade every 6 months (modulo rocky transitions like the upgrade to KDE 4), while corporate users will wait for the next LTS to arrive. To bring that back to a Python context and revisit the proposal that Guido rejected for biennial Python releases with annual standard library releases, suppose that, instead of the standard library itself having annual releases, there was a python-dev maintained (or even third-party maintained, python-dev blessed) "pybackports" project. The idea being, that package would not only make many new standard library features of the current Python release available on widely used previous versions, but also make upcoming features of the *next* Python release available on the current version. The periods of support would be shorter to reflect the faster release Third party libraries and applications could then either target the "rock solid" market and depend solely on the standard library, or go for the "latest and greatest" crowd and require pybackports. Make no mistake, something like this would be quite a bit of work, but it would be far from impossible and could go a long way towards fighting the impression that the standard library is the place where modules go to die. TLDR version: Year A: Python 3.x release, pybackports 3.x.0 release Year B: No Python release, pybackports 3.x.5 release (with 3.y features) Year C: Python 3.y release, pybackports 3.y.0 release Year D: No Python release, pybackports 3.y.5 release (with 3.z features) ...etc Unlike Python itself, Pybackports would not provide bugfix support for prior releases (since it would only be for the latest-and-greatest crowd, those that want the longer support should stick with the standard library). Something to think about, anyway. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, 09 Jun 2010 22:13:51 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Isn't that "impression" largely constructed, and propagated by a limited number of people who apparently don't like the very idea of a "batteries included" stdlib? There has been an amount of anti-stdlib activism (including in this thread) that I find both antagonizing and unconstructive. Outside of that vocal minority, there doesn't seem to be that much criticism against the stdlib. The reality is that there are regularly feature requests on the tracker, and many of them get accepted and committed (of course, when no patch is submitted and no core developer is interested, things have a tendency to linger on; but it's the same for outside libraries too; take a look at bug trackers for e.g. nose or twisted, and you'll see many open entries that have been resting for years). Regards Antoine.
On 09/06/10 22:40, Antoine Pitrou wrote:
The "where modules go to die" version of it is overstated, but the standard library definitely evolves more slowly than many third party packages. To use numpy as an example (just going off their SF file dates): Dec 2007: 1.0.4 May 2008: 1.1.0 Sep 2008: 1.2.0 Jul 2009: 1.3.0 Apr 2010: 1.4.1 Faster cycle times allow developers to be much more responsive to feedback when changes don't turn out as well as was hoped. The comparatively slow evolution of the standard library is the grain of truth that underlies the exaggerated form. The trick is to explore avenues that make these faster cycle times available to those that want them, while still providing a stable foundation for those that need it. It is exactly this situation that the Ubuntu release cycle is designed around: regular 6-monthly releases for most people, less frequent Long Term Support releases for those that need stability. That said, the status quo, with ad hoc porting to PyPI by module maintainers that consider doing so to be worthwhile is certainly a viable option going forward as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Jun 9, 2010 at 8:40 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I don't think so. I've no particular dislike of "batteries included" per se. What I don't like is dealing with packages that may or may not be in the standard library (that affects the requirements for my software), or that may have different names or update policies depending on whether they're part of the standard library (because that affects my code, always negatively).
Unconstructive in what way? Writing cross-Python-version code that deals with the differences between the stdlib and 3rd-party versions of packages is certainly unconstructive, but that's an argument to avoid moving packages into the standard library. One thing that seems to be happening is that the so-called "vocal minority" is growing. I think that should be expected as the acceptance of Python and applications built on it gain wider penetration. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Le mercredi 09 juin 2010 à 09:19 -0400, Fred Drake a écrit :
I don't understand what you mean. Are these packages in the stdlib or aren't they? It can't be both.
I don't see how, really. You make it sound like you have to deal with several versions of the stdlib, but not with several versions of external packages. I wonder why: do you force your users to install version X of module Y? More generally, it seems that people are reproaching things to the stdlib that are equally true for non-stdlib modules. This is quite bewildering to me.
On Wed, Jun 9, 2010 at 3:10 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Yes, as N grows, any constant fraction f times N will grow. In fact, f*N can grow even if f shrinks. but just more slowly than N grows.
True. Since there's no empirical measure of either N (number of Python users) or VM (the Vocal Minority), it's hard to tell if f is a constant fraction or something more (or less) interesting. I also suspect that VM is less likely to be readily measurable than N. For those in the VM, the severity of their objections only increases over time, until they exit both N and VM. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 9 June 2010 13:40, Antoine Pitrou <solipsis@pitrou.net> wrote:
I agree - I think the "where modules go to die" argument is very overstated (but sadly, just as a result of repetition, it seems to be gaining traction :-(). Certainly, stdlib modules evolve at a slower rate than 3rd party ones in many cases. But they do evolve, as Antoine points out, and the slower evolution can just as easily be viewed as stability. What I don't understand is why the "activists" actually care if the stdlib is big or small. Surely if you don't like the "fat stdlib", just ignore it? Why inconvenience those of us who find it a benefit? So can someone clarify (from the point of view of a "thin stdlib" proponent) - what is the benefit to you, personally (i.e., ignoring things like "frees up core developer time" - let them speak for themselves if they feel that is a benefit), of actually removing items from the stdlib, rather than just ignoring them? Paul.
On Wed, Jun 9, 2010 at 7:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
And this provides an opportunity to bring the workload back under control, as well: if we continue the non-stdlib releases for these packages, all we really need to do is remember not to add them to the stdlib in the first place. The one case where we clearly get some win by having the newer package rolled in is distutils2, since that's really about a core service that needs to be available for everyone. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 6/9/2010 5:05 AM, Tarek Ziadé wrote:
I do not see that changing the 'no new features in micro (bugfix) releases' policy would solve Ian's problem at all. Suppose unittest2 were sufficiently done in September to go into 3.2 as unittest. Suppose the policy were changed and unittest2 were also backported into (3.1.final) and 2.7.1. That still would not help Ian with respect to 2.6, 2.5, 2.4, and however far back he wants to support. Since whatever solution he uses for 2.6- should also work for 2.7 (and 3.1) what is the use? Perhaps the reason for the policy needs to be restated: if new features are introduced in every x.y.z release, then 'Python x.y' has no particular meaning. This *was* the case somewhat during Python 1 and early Python 2 days, when the choice between moving from x.y.z to either x.y.(z+1) and x.(y+1) was somewhat arbitrary. (The move from '1.6' to '2.0' rather than '1.7' was for legal reasons only. Python2 really began with 2.2) The precipitating event for the new policy was the introduction of bool() and prebound True and False late in 2.2 series. People who download the latest '2.2' release and who used the new feature found that their code would not run on the great majority of '2.2' installations. It caused enough problems for enough people that Guido decided that he should have waited for 2.3 for official introduction, and perhaps first released bool as a separate module for earlier use. People who propose to change the policy back (are part way back) to what it used to be should at least be aware that it is a reversion and that there are reasons, and not just arbitrary whim or accidental happenstance, for the change. Terry Jan Reedy
On Mon, Jun 7, 2010 at 3:40 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
The latest version pip could *depend on* would be 3.3, meaning all subsequent releases would be relatively unimportant. If you have to work around a known-bad version of something, then all later versions become liabilities instead of improvements (unless the workaround is egregiously ugly). I'd be okay saying that pip could require 3.4+, but not if it means Python 3.3 users would be excluded.
In minor releases, which are new Python releases and take a long time to be widely enough distributed to depend on the bug fix. Though admittedly less time than waiting for a major release to die. Now for new features, if pip uses the latest 2.x and the latest 3.x
A feature that may or may not be available is not a useful feature. We'll just have to backport distutils2 features in an ad hoc way using conditional imports and other nuisances. This is not a new problem btw: if you want to support several versions
of Python, you have to work around the differences.
Yes, but because the standard library changes so little it's not too bad, and in some cases we can rely on backports, and otherwise we simply ignore new functionality. Example: There's a big bug in tarfile in Python 2.4, and I had to
Instead of conditionally monkeypatching tarfile, I'd rather I just had a known-good version of tarfile. And maybe in a sense that is a solution; why try to patch tarfile at all, why not just include swaths of the standard library inline with libraries? Right now typically in projects I've noticed we carefully tease apart a libraries bugs when monkeypatching in an upgrade... but that's probably not worth it. OTOH, I don't think people would be happy if I just included all of distutils2 in pip with some sys.path magic to "upgrade" distutils2 as needed. But then... it might be the most reasonable approach. Or another option, allow versioning of portions of the standard library as need demands. Versioned portions of the standard library may still be quite constrained with respect to backward compatibility, but at least there would be an orderly way to handle backports and for libraries to require bugfixes instead of monkeypatching them in. Maybe an additional constraint would be that all new features have to be in new modules or via new names, and so upgrades would be additive and less likely to affect backward compatibility. And we just keep bad stuff out of the standard library (since perhaps the PyXML lessons is conflate a namespace issue with what was simply a lot of bad code). As an example of how this might have worked, unittest enhancements would have been in a separate module or with as a TestCase subclass (TestCase2) or something else highly unintrusive, and the result could be installed in any version of Python with little danger of conflicts. I.e., API versioning (for the standard library only) gets pushed into module and class names and isn't separate metadata. -- Ian Bicking | http://blog.ianbicking.org
On 7 June 2010 20:20, Ian Bicking <ianb@colorstudy.com> wrote:
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant.
I find it hard to interpret this in any way that doesn't pretty much imply "Python 3 is irrelevant". While I disagree with that, I suspect it isn't what you mean. So can you clarify? Paul.
On Mon, Jun 7, 2010 at 2:42 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It's kind of a moot point as Tarek isn't planning to only support Python 3. But developing packaging libraries for *only* Python 3 would mean that an alternate ecosystem would have to continue to support Python 2, and that alternate ecosystem would produce superior tools because that's where all the users actually are. For the next few years at least Python 3 needs to ride on Python 2's coat tails if it's going to keep up. -- Ian Bicking | http://blog.ianbicking.org
On Mon, 7 Jun 2010 14:20:41 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
Well, even if that's true (I haven't checked and I guess we wouldn't agree on the meaning of "old"), so what? I guess what I'm asking is: what is your line of reasoning? You started with a contention that: “There is no reason any new library or functionality should be tied to a Python release” and, in my humble opinion, you failed to demonstrate that. In particular, you haven't replied to my argument that it dramatically eases dependency management.
distutils2 won't be in 2.7 at least, and any packaging system not available for Python 2 would be irrelevant.
That's your opinion and I guess some people would disagree. Besides, decreeing that distutils2 be kept out of the stdlib won't make it its code magically compatible with Python 2.x. Regards Antoine.
On Mon, Jun 7, 2010 at 2:56 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
It only eases dependency management in closed systems with a fixed Python version. If you support more than one Python version than you still have dependency management, it is just tied to ad hoc workarounds when there are compatibility problems, or ignoring new functionality. The importance of "old" vs. "new" modules is that people tend to have a lowest version of Python they support, as simply a hard stop. This is currently Python 2.5 for most people, 2.4 for some groups (and just a very few stragglers with 2.3). So long as you never use anything beyond what 2.5 provides then it's okay, which is most of the standard library. I am not aware of anything added since 2.5 that isn't backported or previously available as a separate library (I'm sure there's *something*, just nothing I can think of). There is no clear policy about how backports are managed. There's some disfavor for backporting under the same name (PyXML being a primary source of this disfavor), but implicitly argparse *will* be backported as such as it is already installed under argparse and that name matches the stdlib name. It's unclear what should happen when you install a backport in a version of Python that already has the module. E.g., if distutils2 is distributed as distutils2 it would be able to override the standard library when installed unless there was code specifically to disallow it. Also we have implicit dependency management and versioning already because a few libraries in the standard library have .egg-info files installed. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 10:09 PM, Ian Bicking <ianb@colorstudy.com> wrote: [..]
Also we have implicit dependency management and versioning already because a few libraries in the standard library have .egg-info files installed.
If you are talking about wsgiref, that's the only one that has the egg-info in the stdlib, and it will go away soonish (probably in 3.2) since: - this format will be replaced by the new dist-info (PEP 376). - this was a mistake imho to add it since wsgiref was not a distribution in that case
-- Tarek Ziadé | http://ziade.org
On Mon, 7 Jun 2010 15:09:36 -0500 Ian Bicking <ianb@colorstudy.com> wrote:
We're misunderstanding each other. I'm talking about dependency management for users (or application developers), you seem to be talking about dependency management for library developers. As for "ad hoc workarounds" and various "compatibility problems", they wouldn't disappear if the stdlib became smaller; I have trouble understanding what kind of solution you think would eliminate these issues.
The importance of "old" vs. "new" modules is that people tend to have a lowest version of Python they support, as simply a hard stop.
Yes, and it's the same for external modules too. For example, they will support Twisted 8.0 and upper. So what's the difference?
Really, I'm too lazy to go and read the changelogs, but there definitely are many improvements that are not available in Python 2.5 and older.
There is no clear policy about how backports are managed.
That's because, contrary to what you seem to think, external backports are the exception and not the rule. Regards Antoine.
On Mon, Jun 7, 2010 at 4:27 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I offered examples, you were too lazy to read the changelogs, your sweeping declaration does not seem justified.
From What's New in Python 2.6:
contextlib: attached to a language feature. multiprocessing: backported new string formatting: though a method, it'd be easy to produce in a module form. I'm not aware of a backport. abstract base classes: I don't think this can be backported wouldn't all kinds of contortions ast: associated with the implementation/language json: backported (implicitly, as it is simplejson) plistlib: I'm guessing this was added to support packaging. It exists separately. What's New in Python 2.7: argparse: implicitly backported changes to logging: not sure what will happen with this; the module has been backported in the past memoryview: not aware of it being backported importlib: technically including it in 2.7 is a backport, but otherwise no ttk: appears to be backported (http://pypi.python.org/pypi/pyttk/) unittest: backported ElementTree: backported Digging deeper into 2.5: functools: apparently backported at one time, now defunct ctypes: the backport appears to be dead sqlite3: actively developed (different name?) wsgiref: backported hashlib: backported Every release there's some additions to collections, which have not been backported. So in summary, of 17 additions which seemed "backportable" to me (not counting 3 modules that seemed tied to language features): * 4 were not backported * 3 have defunct or ambiguous backports * 10 were backported -- Ian Bicking | http://blog.ianbicking.org
Le lundi 07 juin 2010 à 17:16 -0500, Ian Bicking a écrit :
I offered examples, you were too lazy to read the changelogs, your sweeping declaration does not seem justified.
Sure, but my sweeping declaration is justified by the fact that I'm a daily contributor to Python core and know what kinds of things happen here.
What's New in Python 2.7: [snip]
Your list seems to forget lots of module-specific improvements. There are many more things in http://docs.python.org/dev/whatsnew/2.7.html#new-and-improved-modules , and most of them aren't "backported" in any fashion.
Of course this all depends on your definition of "backportable". If you remove that arbitrary conditional, the fact remains that most improvements, small or big, don't get backported. Regards Antoine.
On 7 June 2010 22:09, Ian Bicking <ianb@colorstudy.com> wrote:
Which gives me an idea: What if all the backports were managed within a single PyPI package, (e.g. backport26, backport27) with clear policy on older Python versions supported. then I could write in my py2.4 script: from backport26.os.path import relpath Konrad
Konrad Delong writes:
This is an interesting idea. I thought about something similar, but I decided that this would basically end up being the same problem as managing the stdlib (I could be wrong, of course, but I'm reasonably confident of that :-), while decreasing returns would have long since set in. Ie,q even though the effort would probably be quite a bit smaller than managing the stdlib itself, the benefits would decrease more rapidly (that's just a wild-ass guess, and since you bring it up, I'm curious to see what folks will say).
On 9 June 2010 10:20, Konrad Delong <konryd@gmail.com> wrote:
It's not a bad idea. A key benefit seems to be that it can be done by anyone, whether or not they are a core developer. So it can be set up right now, without taking up any of the limited core-dev resource. Of course, conversely, the disadvantage is that nobody's done this already, implying that either nobody's thought of it before or there's actually little motivation for someone to put the work into such a solution :-) If nobody's thought of it, you may be lucky and someone will pick up the idea and run with it. Paul.
Yeah, I am aware of that :-) Another question is whether such a package is going to find use. I doubt Michael would introduce a dependency into unittest2 just to get functools.wrap and os.path.relpath out of his code. Distutils2 contains a whole _backport module [1] which could go away, but again: at the cost of introducing dependency into the package. Konrad [1] http://bitbucket.org/tarek/distutils2/src/tip/src/distutils2/_backport/
On Mon, Jun 7, 2010 at 1:42 PM, Brett Cannon <brett@python.org> wrote:
Indeed, each person projects different ideas and motivations onto the standard library and I don't see a great deal of shared understanding about what it is. -- Ian Bicking | http://blog.ianbicking.org
On Mon, Jun 7, 2010 at 8:50 PM, Ian Bicking <ianb@colorstudy.com> wrote:
There's one thing that is clear though: Distutils is in the standard library, and it would be a non sense not to have Distutils2 included to replace it. While I understand your motivations not to see Pip included in the standard library, I will strongly object the exclusion Distutils2 from the standard library. I have agreed to temporarily develop Distutils2 outside the stdlib with the sole condition that it will be included back as soon as it is ready, because we badly need such a system in a vanilla Python. Distutils2 is the place where we are implementing the PEP that where accepted lately, and its inclusion to the standard library will provide a working packaging system for Python (that is *batteries included*) and a blessed playground for third party packaging tools. Regards Tarek -- Tarek Ziadé | http://ziade.org
On 08/06/10 02:35, Ian Bicking wrote:
Rather than rehash the point Antoine has already made regarding dependency management, I'll mention the other key benefit of standard library inclusion: Inclusion of a module or package in the standard library makes sense when the benefits of having "One Obvious Way To Do It" simplifies teaching of Python and development and maintenance of future Python code sufficiently to justify the slower rate of release associated with standard library inclusion. I'm one of those that believes that the volume of all currently written Python code is a small subset of the Python code that will ever be written, hence it makes sense to "raise the bar" in improving the quality of the baseline toolset provided to developers. Additions like argparse and the new futures module in PEP 3148, as well as historical additions like itertools, collections, ElementTree, simplejson, etc all serve that purpose well. However, I *also* like the pattern that has emerged of many standard library modules being kept backwards compatible with previous Python releases, and sometimes even being separately released on PyPI (or elsewhere). This approach allows the "one obvious way" to be extended back to earlier Python versions, since eventual standard library inclusion is a big point in a module or package's favour, even if a developer isn't currently using the latest Python release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (13)
-
Antoine Pitrou -
Brett Cannon -
David Stanek -
Eric Smith -
Fred Drake -
Ian Bicking -
Konrad Delong -
Michael Foord -
Nick Coghlan -
Paul Moore -
Stephen J. Turnbull -
Tarek Ziadé -
Terry Reedy