As for PEP 4: I don't know whether it needs to be listed there. It appears that the PEP is largely unmaintained (I, personally, do not really maintain it). So one option would be to just stop using PEP 4 for recording deprecations, since we now have the warnings module.

+1 Raymond

Raymond Hettinger wrote:

* Deprecated modules just get moved to the lib-old directory. If someone has ancient code relying on the module, it is a somewhat trivial maintenance step to add lib-old to their PYTHONPATH. IOW, I fail to see the harm.

I have never considered this as an official policy. For example, when deprecated C modules are removed, they are never moved to lib-old.

* For this particular module, xmllib, about six years will have elapsed between its original deprecation in Py2.0 and us taking it out in a Py2.5 release.

Correct. For regex, much more time has passed (it is deprecated since 1.5 or something).

* The number one current python complaint is the state of the standard library: http://www.internetnews.com/dev-news/article.php/3441931 . Some of this may just be the fruits of AMK's highly publicized journal entry; however, I think the concerns about library quality will persist.

I agree. Removing old modules might change this, indeed: the complaint about unstability and incompatibility might then become the number one complaint :-)

* The average quality of the library improves as we take out junk (the tzparse module for example) and put in high quality modules like logging, csv, decimal, etc.

I am not convinced that all these more recent modules are really higher quality than the modules that have been added ten years ago. Some are (especially those which has seen extensive testing before being integrated), some are not (especially those added in an ad-hoc manner). Regards, Martin

Anthony Baxter wrote:

The library docs for, e.g. xmllib already say deprecated at the top - maybe it should be larger?

I think it would be better to *remove* the xmllib documentation. Existing code which needs the module will continue to work even without documentation, and new code is unlikely to be written for a module that has no documentation, and where importing the module gives a DeprecationWarning.

Alternately, we could lock the folks complaining about the stdlib's state in a room with the folks who complain that every new thing has "wrecked the language that they knew and loved" and let them fight it out.

That sounds like a fair and democratic way of solving the issue. Regards, Martin

On Sun, 2004-12-05 at 18:49, Brett C. wrote:

And as for the mention of dropping PEP 4, I agree with the running consensus that it isn't needed thanks to the warning module. Do we need to have a more formal vote to drop PEP 4, or should one the PEP maintainers just delete it?

There's really no such thing as "dropping" a PEP, but in any event, we should still keep PEP 4 to document the procedure for deprecating modules. It just doesn't need to list any modules (i.e. remove the sections labeled Obsolete modules, Deprecated modules, and Undeprecated modules). -Barry

On Mon, 2004-12-06 at 16:28, "Martin v. Löwis" wrote: Martin, +1 on everything you wrote, with one minor quibble.

Removal ======= If the module has been deprecated for atleast a year and atleast a version, it can be removed. Removal should move it to old-libs for pure Python modules; a removal procedure for pure C modules has not been defined yet.

I wonder if the one year/one version rule is too short. Given that new versions come about every 18 months, I'd probably give it a 2 year/one version limit.

Modules that have currently deprecation messages in them often fail to identify the Python version in which removal will occur. For modules that have been deprecated since 2.1, I would suggest to remove them for 2.5, with the option of bringing them back in 2.5.1 if people complain

+1 here too. -Barry

I would do what Barry suggests: rephrase the module to document the deprecation/removal procedure. This, of course, needs consensus/pronouncement, too, but I think I would put the following aspects into it:

Requirements ============ Removal of module needs proper advance warning for users; the module should be removed only if - continued usage poses a security threat to the application using it, or - there are no reported usages of the module anymore for quite some time - the module is unmaintained (i.e. there are serious outstanding unfixed bugs for it), and there are alternatives which are maintained. There must be a way for users to stop removal of the module, e.g. by volunteering to maintain an unmaintained module.

The effect of coding this into the PEP is to make deprecation unnecessarily involved. Also, the list of reasons is insufficiently comprehensive. In the past we've gotten along fine with discussing a situation on python-dev and either reaching a consensus or having Guido give an okay.

2. a comment must be added just above the warning, giving a the date and version of deprecation, and a rationale for removal (e.g.: no known users, is security risk)

We should only talk in terms of versions. Dates are irrelevant and confusing. In the last version of the PEP, it was far from clear the significane of deprecating or removing on date xx-xx-xx. All that matters is the version the warning first appears and the version where the module is removed.

Modules that have currently deprecation messages in them often fail to identify the Python version in which removal will occur. For modules that have been deprecated since 2.1, I would suggest to remove them for 2.5, with the option of bringing them back in 2.5.1 if people complain.

I do not think the version number messages are useful. When someone imports a file and gets a warning, they know a deprecation is pending and to stop writing code with that module. Further, I want to avoid the previous PEP 4 situation where one nit or another wasn't followed to the letter so we had to keep the module around for another two versions. It can and should be as simple as having us agree to deprecation, inserting a warning, and then removing it two versions later. Also, we need to shy away from the idea of having 2.5.1 with different capabilities than 2.5. This sort of thing is a portability disaster. Anthony, please speak up. General comments: I had thought the purpose of rewording the PEP was just to take out language naming specific modules and to add language on using the warning module. Instead, it looks like an attempt to make it much more difficult and slow to deprecate a module. If so, I would like to understand what situation is motivating it. Is there some module that had been deprecated but should not have been? Why the change in procedure? It doesn't affect me personally (I haven't proposed any modules for deprecation and do not have any in mind); however, I think the discussion should take place in the broader context of what we intend to do with the standard library -- do we want regex still there in 2010. Also, as proposed, the PEP lists many forces against deprecation and none of the forces for it. If that is the reality, then we could just never deprecate anything because someone, somewhere is bound to be irritated by it. That would certainly simplify the discussion. Going back to specifics, it may be worthwhile to think through the reason we kept the xmllib code but not whrandom. Both were documented, non-buggy, tested, marked as deprecated for a long time, and it was within the realm of possibility that some code still used them. Also, the PEP should discuss the use of the lib-old directory and it should include Barry's link to old documentation. Raymond

Raymond Hettinger wrote:

One other thought: In deciding how long to leave module in, we should consider that Python books are updated infrequently, if at all. It would be a bummer if code in them stopped working as advertised.

Correct. Thanks for reminding me - that is indeed a reasoning that is typically brought up by book authors, so I should have brought it up myself :-) Of course, developers would first see the deprecation warning, which would tell them that their books are outdated. However, if the code is removed, they get an import error, which leaves them guessing what the problem might have been. Regards, Martin

Raymond Hettinger wrote:

The effect of coding this into the PEP is to make deprecation unnecessarily involved.

Can you please explain why you consider this involvement unnecessary? I think it is important that we do not prematurely remove (or even deprecate) modules that are still being actively used.

Also, the list of reasons is insufficiently comprehensive. In the past we've gotten along fine with discussing a situation on python-dev and either reaching a consensus or having Guido give an okay.

I don't think we "got along", but rather "got away". Discussion on python-dev does not cause sufficient user involvement, IMO. And indeed, some people (even on python-dev) think that we *never* should deprecate any modules, as it breaks existing code. They might not be as vocal as supporters of early removal of "ugly" code, but I think their opinion is just as valuable.

We should only talk in terms of versions. Dates are irrelevant and confusing.

Why is a date confusing? I think everybody on Earth understands the notion of a date, there is nothing confusing in this notion. Why are they irrelevant? If we release Python 2.5 next month (and stop developing 2.4), are we entitled to remove all features that were scheduled for removal? I think not: users would have not gotten sufficient advance warning. Users need *time* to react on proposed feature changes.

In the last version of the PEP, it was far from clear the significane of deprecating or removing on date xx-xx-xx. All that matters is the version the warning first appears and the version where the module is removed.

No. It also matters how much time users have had to react.

I do not think the version number messages are useful. When someone imports a file and gets a warning, they know a deprecation is pending and to stop writing code with that module.

Yes, but what about code that currently uses the module? People will not start rewriting (perhaps they aren't the authors in the first place) just because they see a deprecation warning. Instead, they will silence the warning, and go on. If we plan to remove the feature at some point, we need to give these users an indication that they need to act beyond silencing the warning. Such actions could be - have the author of the software update it appropriately - look for a replacement software - ask for an extension of the grace period on python-dev Currently, users do not really have these options, since they do not have an indication that action on their side is really needed if they want their software continue to work with future Python versions.

Further, I want to avoid the previous PEP 4 situation where one nit or another wasn't followed to the letter so we had to keep the module around for another two versions.

Why do you want to avoid that situation? What is the problem with waiting for two more versions? No harm is done in doing so.

It can and should be as simple as having us agree to deprecation, inserting a warning, and then removing it two versions later.

I strongly disagree with that position.

I had thought the purpose of rewording the PEP was just to take out language naming specific modules and to add language on using the warning module. Instead, it looks like an attempt to make it much more difficult and slow to deprecate a module.

I want the PEP to be followed. For that, it needs to be precise, and the procedures need to be agreed upon (or, if we cannot agree, it needs to be Pronounced).

If so, I would like to understand what situation is motivating it. Is there some module that had been deprecated but should not have been?

Deprecation is fine for all the modules so far. Generating the warning is a bit of a problem, removal would be a serious problem, for the following modules: - regex - xmllib - whrandom since I expect that these modules are still widely used, in existing code, by users who are not the authors of that code. This is a problem, since these users cannot do anything about the usage of the deprecated code. They just find that the software that ran happily with the previous Python version first shouts at them in the next Python version, then totally breaks with the subsequent version. They will hate Python for that.

Why the change in procedure?

I think we agree that the previously-documented procedure needs to be changed - it was not followed.

It doesn't affect me personally (I haven't proposed any modules for deprecation and do not have any in mind); however, I think the discussion should take place in the broader context of what we intend to do with the standard library -- do we want regex still there in 2010.

As to the latter question: Definitely. We don't want regex to be used for new code, and we want developers to fully understand that, but we also do want code that currently works correctly with regex continue to do so in 2010. The only exception would be that 2010 no code is left in active use that uses regex, which I doubt.

Also, as proposed, the PEP lists many forces against deprecation and none of the forces for it.

That is not true. Security flaws are giving priority. They can only be undeprecated if the reason for deprecation goes away, which means that somebody fixes the security flaw.

If that is the reality, then we could just never deprecate anything because someone, somewhere is bound to be irritated by it. That would certainly simplify the discussion.

Well, we do want to move developers to the new direction. However, many people use Python code who are *not* developers. We need to take their concerns into account as well.

Going back to specifics, it may be worthwhile to think through the reason we kept the xmllib code but not whrandom. Both were documented, non-buggy, tested, marked as deprecated for a long time, and it was within the realm of possibility that some code still used them.

Indeed. If it is in the realm of possibility that code still uses a module, and that this would not raise security concerns, the module should not be removed.

Also, the PEP should discuss the use of the lib-old directory and it should include Barry's link to old documentation.

I'm happy to adjust details of the procedures - but it seems we disagree on the principles. Regards, Martin

There is one other issue that should get addressed, modules in limbo. I'm concerned about the old email modules. They haven't been deprecated and may stay around for a good while. However, some SF requests have been passed over on the basis that the modules are supplanted and will be phased out. I don't think it is good to leave active modules as orphans. Raymond

Raymond Hettinger wrote:

Instead, there should be a clear decision to deprecate or not. If that entails a comp.lang.python.announce notice and comment period, so be it. Once a decision is made, document it, add a warning, and wait.

Ok, that might be reasonable.

Once a couple versions pass, some useful action needs to take place. If the code is left in-place and the doc index is just re-arranged, then we would have been better off not doing anything at all.

I disagree. The primary purpose (move developers to the alternative approach) is achieved as soon as the warning is added, and the documentation is deleted. Whether the module is actually deleted is of minor importance: it costs nothing to continue including it; disk space is cheap.

The questions of dates was somewhat minor. I was unclear on the implication of, for example, "remove on 15Jan2005". Since Py2.5 won't go out for at least a year, does that mean that the work can't be done now while I have time instead of later when I don't. The only time the removal becomes visible is when a new version goes out the door.

You could remove it now, but if we release Py2.5 in two months, you would have to put it back in. So if you don't have time then, maybe somebody else will. If nobody has time to remove the module, the next release will ship with it, again - no big issue.

Further, if a version is going to have something removed, we should do it at the outset rather than just before a release goes out the door (to allow for early surfacing of any issues).

That is true.

If we really don't care whether it gets done, then we shouldn't bother in the first place.

What do you mean by "bother"? Not put the deprecation warning in? We do want users to move to the new approach of doing something. However, two version are just not enough - it may take 10 or 20 years to remove a widely used feature (e.g. the string module). That it will take so long does not mean we should not start the process *now* - if we start the process in five years, it will *still* take 10 or 20 years. Just be patient. Regards, Martin

One other question occurred to me for "deprecate X in favour of Y" situations - should the deprecation warning added to X point developers towards Y? Also, should the PEP include example wordings for deprecation warnings, rather than being completely freeform? Martin listed some information that should be included, but it seems an example or two would make it easy to get right. E.g.: Unmaintained, with a maintained alternative: "Due to lack of maintenance, this module has been deprecated in favour of module Y and will be removed in Python 2.6 (see PEP 4 for information on the deprecation process)" Security problems, no alternative: "Due to security concerns, this module has been deprecated and will be removed in Python 2.6 (see PEP 4 for information on the deprecation process)" Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

""Martin v. Löwis"" <martin@v.loewis.de> wrote in message news:41B556FE.20504@v.loewis.de... As a (currently) casual user of Python, this is my view of the standard library dilemma and its solution: 1. It is too small: I may someday want to use a module not yet added. 2. It is too big: I cannot keep everything in mind at once and cannot remember, without referring back to the lib manual or pydev, which modules are currently or slated to become deprecated. 3. I do not wish to have other people's code broken,or books made obsolete, without sufficient reason (ie, the code is actively dangerous versus merely broken or superceded). My preferred solution is to apply the 'out of sight, out of mind' principle. 1. Move obsolete modules to a separate directory (lib_old sounds fine) and put that directory in pythonpath. When I ran Python (1.3) from a 20 meg disk, I would have preferred complete removal, but with 60+ gigs, the small extra space is no longer an issue. If someone needs the space or wants to guarantee non-use of old modules, deleting lib_old is easy enough. 2. Remove the docs for obsolete modules from the main part of the current lib reference. Sub-choices for legacy chapters are either complete removal or segregation into a separate document or apppendix to current one. Terry J. Reedy

"Martin v. Löwis" <martin@v.loewis.de> writes:

Anthony Baxter wrote:

...

The library docs for, e.g. xmllib already say deprecated at the top - maybe it should be larger?

I think it would be better to *remove* the xmllib documentation. Existing code which needs the module will continue to work even without documentation, and new code is unlikely to be written for a module that has no documentation, and where importing the module gives a DeprecationWarning.

+1, at least for *hiding* the documentation of xmllib + similar modules. I'm not sure wholesale removal is really a good idea. It's like the new "Non-essential Built-in Functions" section of the lib ref. Cheers, mwh -- <Acapnotic> jemfinch: What's to parse? A numeric code, perhaps a chicken, and some arguments -- from Twisted.Quotes

Carlos Ribeiro wrote:

On Mon, 06 Dec 2004 12:12:48 +0000, Michael Hudson <mwh@python.net> wrote:

...

+1, at least for *hiding* the documentation of xmllib + similar modules. I'm not sure wholesale removal is really a good idea.

1) Move the deprecated documentation into a separate book.

2) Don't include the "Deprecated Modules Reference" in the standard distribution, but make it available as a separate download.

What about the idea of pointing to the last version of the docs that contained the relevant documentation? Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

On Mon, 2004-12-06 at 07:44, Raymond Hettinger wrote:

We are trying to remove clutter, not keep it in perpetuity. Leaving the docs means continuing to have to test it, field bug reports, be aware of its existence, etc.

Do not keep this on a respirator. Let it die in peace.

Old documentation lives on in http://www.python.org/doc/versions.html which seems sufficient for looking up deprecated modules. -Barry

On Sun, 2004-12-05 at 05:40, Raymond Hettinger wrote:

* The number one current python complaint is the state of the standard library: http://www.internetnews.com/dev-news/article.php/3441931 . Some of this may just be the fruits of AMK's highly publicized journal entry; however, I think the concerns about library quality will persist.

I don't know if I'd use one quote from this article to infer that standard library issues are the #1 complaint. But besides that, there's no question IMO that the standard library should be the place where future Python development efforts are focused (not exclusively, but predominately).

* The average quality of the library improves as we take out junk (the tzparse module for example) and put in high quality modules like logging, csv, decimal, etc.

* After working through the tutorial, another huge task for aspiring intermediate Python programmer is to learn about the standard library. That task is made much difficult by the presence of obsolete, incomplete, and/or supplanted modules.

You don't have to remove modules to make this happen. Few developers find modules by doing an 'ls' on Lib -- they look to the library reference docs, books, and existing code. So for example, the re-org I did on the string module docs for 2.4 should help, because now you can't look up a string function such as atoi() without seeing that it's deprecated. A developer writing new code hopefully wouldn't even waste time learning about those functions, and instead hop right to the preferred string methods. Yet, there's no push in the community to actually remove those deprecated functions from string.py.

For example, the high quality, actively maintained email package supplants several older, incomplete, or non-actively maintained modules. The library would be cleaner, more presentable, and easier to use if the old ones were to fall by the wayside.

Yes, but there's still /a lot/ of work to do to update the existing standard library to use the email package. We debated this during 2.4 and decided we can't even add deprecation warnings to things like rfc822.py until we do.

* Ideally, the library will develop in a way that doesn't have a cluttered concept space. This will help get it back to the point where you can hold it in your head all at once and still be able to think about the problem domain. Keeping cruft impedes that goal.

Agreed, but I think it's mostly going to be an organic process over time. -Barry

--On Sonntag, 5. Dezember 2004 10:30 Uhr -0800 Dennis Allison <allison@sumeru.stanford.EDU> wrote:

A report on the zope@zope.org list suggests that Python 2.4 is not fully compatible with Zope 2.7.3. Has any tested against Zope?

To which report do you refer? The only thing I mentioned is that there has not been any audit for Python 2.4 + Zope 2.7 as with Python 2.3 + Zope 2.6/2.7. -aj

* The average quality of the library improves as we take out junk (the tzparse module for example) and put in high quality modules like logging, csv, decimal, etc.

Yes and no. The added modules have to be relevant to what users want to do. While (relatively) minor stuff like csv and decimal are good ideas, of course, logging is kind of an "insider's" module. What many more users want, however, are things like an XML parser, or a CSS parser, or a usable HTTP server, or... The fact that XML 1.1 can't be parsed with already-written Python is a *big* strike against. So removing highly demanded functionality like XML parsing, and adding specialist modules such as csv, do not overall help what users see as the quality of the standard library. Bill

Martin v. Löwis wrote:

As for PEP 4: I don't know whether it needs to be listed there. It appears that the PEP is largely unmaintained (I, personally, do not really maintain it). So one option would be to just stop using PEP 4 for recording deprecations, since we now have the warnings module. If we want to keep PEP 4, we need to follow the procedures it requires (or modify them if we don't like them).

There is a need for a central place where deprecations are posted. You can't tell people to go hunt the standard lib for uses of the warning module in order for them to catch up with the current state of affairs. One thing that we shouldn't forget is that Python users do not necessarily upgrade to new Python releases every time there's a new release. Production users of Python will often skip a release or two depending on their needs, time frames and audit cycles. It is very important for them to know that module or other technology will be deprecated two releases down the road, so that they can accomodate for these changes today. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 13 2004)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2004-12-06: Released eGenix mx Extensions for Python 2.4 ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::