folding cElementTree behind ElementTree in 3.3
Hello, Here's a note from "What's new in Python 3.0": """A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. The pickle / cPickle pair received this treatment. The profile module is on the list for 3.1. The StringIO module has been turned into a class in the io module.""" Is there a good reason why xml.etree.ElementTree / xml.etree.cElementTree did not "receive this treatment"? In the case of this module, it's quite unfortunate because: 1. The accelerated module is much faster and memory efficient (see recent benchmarks here: http://bugs.python.org/issue11379), and XML processing is an area where processing matters 2. The accelerated module implements the same API 3. It's very hard to even find out about the existence of the accelerated module. Its sole mention in the docs is this un-emphasized line in http://docs.python.org/dev/py3k/library/xml.etree.elementtree.html: "A C implementation of this API is available as xml.etree.cElementTree." Even to an experienced user who carefully reads the whole documentation it's not easy to notice. For the typical user who just jumps around to functions/methods he's interested in, it's essentially invisible. Eli
On Wed, Feb 8, 2012 at 1:59 PM, Eli Bendersky <eliben@gmail.com> wrote:
Is there a good reason why xml.etree.ElementTree / xml.etree.cElementTree did not "receive this treatment"?
See PEP 360, which lists "Externally Maintained Packages". In the past we allowed additions to the standard library without requiring that the standard library version become the master version. These days we expect python.org to become the master version, perhaps with backports and experimental features published on PyPI (cf. packaging vs distutils2, unittest vs unittest, contextlib vs contextlib2). ElementTree was one of the last of those externally maintained modules added to the standard library - as documented in the PEP, it's still officially maintained by Fredrik Lundh. Folding the two implementations together in the standard library would mean officially declaring that xml.etree is now an independently maintained fork of Fredrik's version rather than just a "snapshot in time" of a particular version (which is what it has been historically). So the reasons for keeping these two separate to date isn't technical, it's because Fredrik publishes them as separate modules. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 7, 2012 at 22:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
Folding the two implementations together in the standard library would mean officially declaring that xml.etree is now an independently maintained fork of Fredrik's version rather than just a "snapshot in time" of a particular version (which is what it has been historically).
Is ElementTree even still maintained externally? I seem to remember Florent going through headaches to get changes into this area, and I can't find an external repository for this code.
On Wed, Feb 8, 2012 at 06:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 8, 2012 at 1:59 PM, Eli Bendersky <eliben@gmail.com> wrote:
Is there a good reason why xml.etree.ElementTree / xml.etree.cElementTree did not "receive this treatment"?
See PEP 360, which lists "Externally Maintained Packages". In the past we allowed additions to the standard library without requiring that the standard library version become the master version. These days we expect python.org to become the master version, perhaps with backports and experimental features published on PyPI (cf. packaging vs distutils2, unittest vs unittest, contextlib vs contextlib2).
ElementTree was one of the last of those externally maintained modules added to the standard library - as documented in the PEP, it's still officially maintained by Fredrik Lundh. Folding the two implementations together in the standard library would mean officially declaring that xml.etree is now an independently maintained fork of Fredrik's version rather than just a "snapshot in time" of a particular version (which is what it has been historically).
So the reasons for keeping these two separate to date isn't technical, it's because Fredrik publishes them as separate modules.
The idea is to import the C module when xml.etree.ElementTree is imported, falling back to the Python module if that fails for some reason. So this is not modifying the modules, just the Python stdlib facade for them. Besides, in http://mail.python.org/pipermail/python-dev/2011-December/114812.html Stefan Behnel said "[...] Today, ET is *only* being maintained in the stdlib by Florent Xicluna [...]". Is this not true? Eli P.S. Would declaring that xml.etree is now independently maintained by pydev be a bad thing? Why?
On Tue, Feb 7, 2012 at 11:31 PM, Eli Bendersky <eliben@gmail.com> wrote:
Besides, in http://mail.python.org/pipermail/python-dev/2011-December/114812.html Stefan Behnel said "[...] Today, ET is *only* being maintained in the stdlib by Florent Xicluna [...]". Is this not true?
I don't know. I took this to be an observation rather than a declaration of intent by the package owner (Fredrik Lundh).
P.S. Would declaring that xml.etree is now independently maintained by pydev be a bad thing? Why?
So long as Fredrik owns the package, I think forking it for the standard library would be a bad thing, though not for technical reasons. Fredrik provided his libraries for the standard library in good faith, and we still list him as the external maintainer. Until *that* changes, forking would be inappropriate. I'd much rather see a discussion with Fredrik about the future maintenance plan for ElementTree and cElementTree. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens
On Wed, Feb 8, 2012 at 06:41, Fred Drake <fdrake@acm.org> wrote:
On Tue, Feb 7, 2012 at 11:31 PM, Eli Bendersky <eliben@gmail.com> wrote:
Besides, in http://mail.python.org/pipermail/python-dev/2011-December/114812.html Stefan Behnel said "[...] Today, ET is *only* being maintained in the stdlib by Florent Xicluna [...]". Is this not true?
I don't know. I took this to be an observation rather than a declaration of intent by the package owner (Fredrik Lundh).
P.S. Would declaring that xml.etree is now independently maintained by pydev be a bad thing? Why?
So long as Fredrik owns the package, I think forking it for the standard library would be a bad thing, though not for technical reasons. Fredrik provided his libraries for the standard library in good faith, and we still list him as the external maintainer. Until *that* changes, forking would be inappropriate. I'd much rather see a discussion with Fredrik about the future maintenance plan for ElementTree and cElementTree.
Yes, I realize this is a loaded issue and I agree that all steps in this direction should be taken with Fredrik's agreement. However, to re-focus: The initial proposal of changing *the stdlib import facade* for xml.etree.ElementTree to use the C accelerator (_elementtree) by default. Will that somehow harm Fredrik's sovereignty over ET? Are there any other problems hidden here? Because if not, it appears like a change of only a few lines of code could provide a significantly better XML processing experience in 3.3 for a lot of users (and save some keystrokes for the ones who already know to look for cElementTree). Eli
On Tue, Feb 7, 2012 at 11:46 PM, Eli Bendersky <eliben@gmail.com> wrote:
The initial proposal of changing *the stdlib import facade* for xml.etree.ElementTree to use the C accelerator (_elementtree) by default.
I guess this is one source of confusion: what are you referring to an an "import façade"? When I look in Lib/xml/etree/, I see the ElementTree, ElementPath, and ElementInclude modules, and a wrapper for cElementTree's extension module. There isn't any sort of façade for ElementTree; are you proposing to add one, perhaps in xml.etree/__init__.py? -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens
On Wed, Feb 8, 2012 at 07:10, Fred Drake <fdrake@acm.org> wrote:
On Tue, Feb 7, 2012 at 11:46 PM, Eli Bendersky <eliben@gmail.com> wrote:
The initial proposal of changing *the stdlib import facade* for xml.etree.ElementTree to use the C accelerator (_elementtree) by default.
I guess this is one source of confusion: what are you referring to an an "import façade"? When I look in Lib/xml/etree/, I see the ElementTree, ElementPath, and ElementInclude modules, and a wrapper for cElementTree's extension module.
There isn't any sort of façade for ElementTree; are you proposing to add one, perhaps in xml.etree/__init__.py?
AFAICS ElementPath is a helper used by ElementTree, and cElementTree has one of its own. It's not documented for stand-alone use. ElementInclude also isn't documented and doesn't appear to be used anywhere. The facade can be added to xml/etree/ElementTree.py since that's the only documented module. It can attempt to do: from _elementtree import * (which is what cElementTree.py) does, and on failure, just go on doing what it does now. Eli
Eli Bendersky, 08.02.2012 07:07:
On Wed, Feb 8, 2012 at 07:10, Fred Drake wrote:
On Tue, Feb 7, 2012 at 11:46 PM, Eli Bendersky wrote:
The initial proposal of changing *the stdlib import facade* for xml.etree.ElementTree to use the C accelerator (_elementtree) by default.
I guess this is one source of confusion: what are you referring to an an "import façade"? When I look in Lib/xml/etree/, I see the ElementTree, ElementPath, and ElementInclude modules, and a wrapper for cElementTree's extension module.
There isn't any sort of façade for ElementTree; are you proposing to add one, perhaps in xml.etree/__init__.py?
AFAICS ElementPath is a helper used by ElementTree, and cElementTree has one of its own. It's not documented for stand-alone use. ElementInclude also isn't documented and doesn't appear to be used anywhere.
The facade can be added to xml/etree/ElementTree.py since that's the only documented module. It can attempt to do:
from _elementtree import *
(which is what cElementTree.py) does, and on failure, just go on doing what it does now.
Basically, cElementTree (actually the accelerator module) reuses everything from ElementTree that it does not implement itself, e.g. the serialiser or the ElementPath implementation in ElementPath.py (which is not commonly being used by itself anyway). ElementInclude is meant to be independently imported by user code and works with both implementations, although it uses plain ElementTree by default and currently needs explicit configuring for cElementTree. It looks like that need would vanish when ElementTree uses the accelerator module internally. So, ElementTree.py is a superset of cElementTree's C module, and importing that C module into ElementTree.py instead of only importing it into cElementTree.py would just make ElementTree.py faster, that's basically it. Stefan
The facade can be added to xml/etree/ElementTree.py since that's the only documented module. It can attempt to do:
from _elementtree import *
(which is what cElementTree.py) does, and on failure, just go on doing what it does now.
Basically, cElementTree (actually the accelerator module) reuses everything from ElementTree that it does not implement itself, e.g. the serialiser or the ElementPath implementation in ElementPath.py (which is not commonly being used by itself anyway).
ElementInclude is meant to be independently imported by user code and works with both implementations, although it uses plain ElementTree by default and currently needs explicit configuring for cElementTree. It looks like that need would vanish when ElementTree uses the accelerator module internally.
So, ElementTree.py is a superset of cElementTree's C module, and importing that C module into ElementTree.py instead of only importing it into cElementTree.py would just make ElementTree.py faster, that's basically it.
Yep. Any objections from pydev? Stefan, in the other thread (... XML batteries ) you said you will contact Fredrik, did you manage to get hold of him? Eli
Fred Drake, 08.02.2012 05:41:
On Tue, Feb 7, 2012 at 11:31 PM, Eli Bendersky wrote:
Besides, in http://mail.python.org/pipermail/python-dev/2011-December/114812.html Stefan Behnel said "[...] Today, ET is *only* being maintained in the stdlib by Florent Xicluna [...]". Is this not true?
I don't know. I took this to be an observation rather than a declaration of intent by the package owner (Fredrik Lundh).
This observation resulted from the fact that Fredrik hasn't updated the code in his public ElementTree repository(ies) since 2009, i.e. way before the release of Python 2.7 and 3.2 that integrated these changes. https://bitbucket.org/effbot/et-2009-provolone/overview The integration of ElementTree 1.3 into the standard library was almost exclusively done by Florent, with some supporting comments by Fredrik. Note that ElementTree 1.3 has not even been officially released yet, so the only "final" public release of it is in the standard library. Since then, Florent has been actively working on bug tickets, most of which have not received any reaction on the side of Fredrik. That makes me consider it the reality that "today, ET is only being maintained in the stdlib".
P.S. Would declaring that xml.etree is now independently maintained by pydev be a bad thing? Why?
So long as Fredrik owns the package, I think forking it for the standard library would be a bad thing, though not for technical reasons. Fredrik provided his libraries for the standard library in good faith, and we still list him as the external maintainer. Until *that* changes, forking would be inappropriate. I'd much rather see a discussion with Fredrik about the future maintenance plan for ElementTree and cElementTree.
I didn't get a response from him to my e-mails since early 2010. Maybe others have more luck if they try, but I don't have the impression that waiting another two years gets us anywhere interesting. Given that it was two months ago that I started the "Fixing the XML batteries" thread (and years since I brought up the topic for the first time), it seems to be hard enough already to get anyone on python-dev actually do something for Python's XML support, instead of just actively discouraging those who invest time and work into it. Stefan
On Wed, Feb 8, 2012 at 08:37, Stefan Behnel <stefan_ml@behnel.de> wrote:
I didn't get a response from him to my e-mails since early 2010. Maybe others have more luck if they try, but I don't have the impression that waiting another two years gets us anywhere interesting.
Given that it was two months ago that I started the "Fixing the XML batteries" thread (and years since I brought up the topic for the first time), it seems to be hard enough already to get anyone on python-dev actually do something for Python's XML support, instead of just actively discouraging those who invest time and work into it.
I concur. It's important that we consider Fredrik's ownership of the modules, but if he fails to reply to email and doesn't update his repositories, there should be enough cause for python-dev to go on and appropriate the stdlib versions of those modules. Cheers, Dirkjan
On Wed, Feb 8, 2012 at 11:36, Dirkjan Ochtman <dirkjan@ochtman.nl> wrote:
On Wed, Feb 8, 2012 at 08:37, Stefan Behnel <stefan_ml@behnel.de> wrote:
I didn't get a response from him to my e-mails since early 2010. Maybe others have more luck if they try, but I don't have the impression that waiting another two years gets us anywhere interesting.
Given that it was two months ago that I started the "Fixing the XML batteries" thread (and years since I brought up the topic for the first time), it seems to be hard enough already to get anyone on python-dev actually do something for Python's XML support, instead of just actively discouraging those who invest time and work into it.
I concur. It's important that we consider Fredrik's ownership of the modules, but if he fails to reply to email and doesn't update his repositories, there should be enough cause for python-dev to go on and appropriate the stdlib versions of those modules.
+1. That said, I think that the particular change discussed in this thread can be made anyway, since it doesn't really modify ET's APIs or functionality, just the way it gets imported from stdlib. Eli
On 8 February 2012 09:49, Eli Bendersky <eliben@gmail.com> wrote:
I concur. It's important that we consider Fredrik's ownership of the modules, but if he fails to reply to email and doesn't update his repositories, there should be enough cause for python-dev to go on and appropriate the stdlib versions of those modules.
+1.
That said, I think that the particular change discussed in this thread can be made anyway, since it doesn't really modify ET's APIs or functionality, just the way it gets imported from stdlib.
I would suggest that, assuming python-dev want to take ownership of the module, one last-ditch attempt be made to contact Fredrik. We should email him, and copy python-dev (and maybe even python-list) asking for his view, and ideally his blessing on the stdlib version being forked and maintained independently going forward. Put a time limit on responses ("if we don't hear by XXX, we'll assume Fredrik is either uncontactable or not interested, and therefore we can go ahead with maintaining the stdlib version independently"). It's important to respect Fredrik's wishes and ownership, but we can't leave part of the stdlib frozen and abandoned just because he's not available any longer. Paul. PS The only other options I can see are to remove elementtree from the stdlib altogether, or explicitly document it as frozen and no longer maintained.
On Wed, 8 Feb 2012 11:11:07 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
On 8 February 2012 09:49, Eli Bendersky <eliben@gmail.com> wrote:
I concur. It's important that we consider Fredrik's ownership of the modules, but if he fails to reply to email and doesn't update his repositories, there should be enough cause for python-dev to go on and appropriate the stdlib versions of those modules.
+1.
That said, I think that the particular change discussed in this thread can be made anyway, since it doesn't really modify ET's APIs or functionality, just the way it gets imported from stdlib.
I would suggest that, assuming python-dev want to take ownership of the module, one last-ditch attempt be made to contact Fredrik. We should email him, and copy python-dev (and maybe even python-list) asking for his view, and ideally his blessing on the stdlib version being forked and maintained independently going forward. Put a time limit on responses ("if we don't hear by XXX, we'll assume Fredrik is either uncontactable or not interested, and therefore we can go ahead with maintaining the stdlib version independently").
It's important to respect Fredrik's wishes and ownership, but we can't leave part of the stdlib frozen and abandoned just because he's not available any longer.
It's not frozen, it's actually maintained. Regards Antoine.
On Wed, Feb 8, 2012 at 10:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 8 Feb 2012 11:11:07 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
It's important to respect Fredrik's wishes and ownership, but we can't leave part of the stdlib frozen and abandoned just because he's not available any longer.
It's not frozen, it's actually maintained.
Indeed, it sounds like the most appropriate course (if we don't hear otherwise from Fredrik) may be to just update PEP 360 to acknowledge current reality (i.e. the most current release of ElementTree is actually the one maintained by Florent in the stdlib). I'll note that this change isn't *quite* as simple as Eli's description earlier in the thread may suggest, though - the test suite also needs to be updated to ensure that the Python version is still fully exercised without the C acceleration applied. And such an an alteration would definitely be an explicit fork, even though the user facing API doesn't change - we're changing the structure of the code in a way that means some upstream deltas (if they happen to occur) may not apply cleanly. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
It's not frozen, it's actually maintained.
Indeed, it sounds like the most appropriate course (if we don't hear otherwise from Fredrik) may be to just update PEP 360 to acknowledge current reality (i.e. the most current release of ElementTree is actually the one maintained by Florent in the stdlib).
I'll note that this change isn't *quite* as simple as Eli's description earlier in the thread may suggest, though - the test suite also needs to be updated to ensure that the Python version is still fully exercised without the C acceleration applied.
Sure thing. I suppose similar machinery already exists for things like pickle / cPickle. I still maintain that it's a simple change :-)
And such an an alteration would definitely be an explicit fork, even though the user facing API doesn't change - we're changing the structure of the code in a way that means some upstream deltas (if they happen to occur) may not apply cleanly.
This is a very minimal delta, however. I think it can even be made simpler by replacing ElementTree with a facade module that either imports _elementtree or the Python ElementTree. So the delta vs. upstream would only be in file placement. But these are two conflicting discussions - if changes were made in stdlib *already* that were not propagated upstream, what use is a clean delta? Eli
On 8 February 2012 12:21, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Feb 8, 2012 at 10:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 8 Feb 2012 11:11:07 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
It's important to respect Fredrik's wishes and ownership, but we can't leave part of the stdlib frozen and abandoned just because he's not available any longer.
It's not frozen, it's actually maintained.
Indeed, it sounds like the most appropriate course (if we don't hear otherwise from Fredrik) may be to just update PEP 360 to acknowledge current reality (i.e. the most current release of ElementTree is actually the one maintained by Florent in the stdlib).
Ah, OK. My apologies, I had misunderstood the previous discussion. In which case I agree with Nick, lets' update PEP 360 and move forward. On that basis, +1 to Eli's suggestion of making cElementTree a transparent accelerator. Paul
2012/2/8 Nick Coghlan <ncoghlan@gmail.com>
On Wed, Feb 8, 2012 at 10:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:>
It's not frozen, it's actually maintained.
Indeed, it sounds like the most appropriate course (if we don't hear otherwise from Fredrik) may be to just update PEP 360 to acknowledge current reality (i.e. the most current release of ElementTree is actually the one maintained by Florent in the stdlib).
Actually, it was part of my learning curve to the development of Python, as you can see on the thread of the issue http://bugs.python.org/issue6472 . I spent some time between December 2009 and March 2010 to merge the "experimental" 1.3 in the standard library, both for 2.7 and 3.2. Upstream, there were 2 different test suites for the Python and the C implementation, but I merged them in a single test suite, and I've patched the C accelerator to conform to the same behaviour as the Python reference module. With the knowledge I acquired, I chased some other bugs related to ElementTree at the same time. With the feedback and some support coming from Antoine, Fredrik and Stefan we shaped a decent ElementTree 1.3 for the standard library. I am not aware of any effort to maintain the ElementTree package outside of the standard library since I did this merge. So, in the current state, we could consider the standard library package as the most up to date and stable version of ElementTree. I concur with Eli proposal to set the C accelerator as default : the test suite ensures that both implementations behave the same. I cannot commit myself for the long-term maintenance of ElementTree in the standard library, both because I don't have a strong interest in XML parsing, and because I have many other projects which keep me away from core python development for long period of times. However, I think it is a good thing if all the packages which are part of the standard library follow the same rules. We should try to find an agreement with Fredrik, explicit or implicit, which delegates the evolution and the maintenance of ElementTree to the Python community. IIRC, we have other examples in the standard library where the community support helped a lot to refresh a package where the original maintainer did not have enough time to pursue its work. I'll note that this change isn't *quite* as simple as Eli's
description earlier in the thread may suggest, though - the test suite also needs to be updated to ensure that the Python version is still fully exercised without the C acceleration applied. And such an an alteration would definitely be an explicit fork, even though the user facing API doesn't change - we're changing the structure of the code in a way that means some upstream deltas (if they happen to occur) may not apply cleanly.
The test suite is a "de facto" fork of the upstream test suites, since upstream test suites do not guarantee the same behaviour between cElementTree and ElementTree. -- Florent Xicluna
On Wed, Feb 8, 2012 at 10:04 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:>
It's not frozen, it's actually maintained.
Indeed, it sounds like the most appropriate course (if we don't hear otherwise from Fredrik) may be to just update PEP 360 to acknowledge current reality (i.e. the most current release of ElementTree is actually the one maintained by Florent in the stdlib).
Actually, it was part of my learning curve to the development of Python, as you can see on the thread of the issue http://bugs.python.org/issue6472 . I spent some time between December 2009 and March 2010 to merge the "experimental" 1.3 in the standard library, both for 2.7 and 3.2. Upstream, there were 2 different test suites for the Python and the C implementation, but I merged them in a single test suite, and I've patched the C accelerator to conform to the same behaviour as the Python reference module. With the knowledge I acquired, I chased some other bugs related to ElementTree at the same time. With the feedback and some support coming from Antoine, Fredrik and Stefan we shaped a decent ElementTree 1.3 for the standard library.
I am not aware of any effort to maintain the ElementTree package outside of the standard library since I did this merge. So, in the current state, we could consider the standard library package as the most up to date and stable version of ElementTree. I concur with Eli proposal to set the C accelerator as default : the test suite ensures that both implementations behave the same.
I cannot commit myself for the long-term maintenance of ElementTree in the standard library, both because I don't have a strong interest in XML parsing, and because I have many other projects which keep me away from core python development for long period of times.
However, I think it is a good thing if all the packages which are part of the standard library follow the same rules. We should try to find an agreement with Fredrik, explicit or implicit, which delegates the evolution and the maintenance of ElementTree to the Python community. IIRC, we have other examples in the standard library where the community support helped a lot to refresh a package where the original maintainer did not have enough time to pursue its work.
Thanks for the input, Florent. So, to paraphrase, there already are code changes in the stdlib version of ET/cET which are not upstream. You made it explicit about the tests, so the question is only left for the modules themselves. Is that right? Eli
2012/2/10 Eli Bendersky <eliben@gmail.com>
Thanks for the input, Florent. So, to paraphrase, there already are code changes in the stdlib version of ET/cET which are not upstream. You made it explicit about the tests, so the question is only left for the modules themselves. Is that right?
The port of ElementTree to Python 3000 was done in the standard library only. The work was done back in 2006, 2007 and 2008. There was never a public version of ElementTree for Python 3 outside of the standard library. It is already a significant change from the upstream branch (many changes in the C extension code). Then when I enforced the same test suite for both implementation, I have fixed many things in the C extension module too. To my knowledge, these fixes were not included upstream. Since two years, there was regular maintenance of the package in the standard library, but none of the patch were integrated upstream. I hope it answers the question, -- Florent Xicluna
On Fri, Feb 10, 2012 at 10:32, Florent <florent.xicluna@gmail.com> wrote:
2012/2/10 Eli Bendersky <eliben@gmail.com>
Thanks for the input, Florent. So, to paraphrase, there already are code changes in the stdlib version of ET/cET which are not upstream. You made it explicit about the tests, so the question is only left for the modules themselves. Is that right?
The port of ElementTree to Python 3000 was done in the standard library only. The work was done back in 2006, 2007 and 2008. There was never a public version of ElementTree for Python 3 outside of the standard library. It is already a significant change from the upstream branch (many changes in the C extension code).
Then when I enforced the same test suite for both implementation, I have fixed many things in the C extension module too. To my knowledge, these fixes were not included upstream.
Since two years, there was regular maintenance of the package in the standard library, but none of the patch were integrated upstream.
Folks, with this in mind, can we just acknowledge that the stdlib ElementTree is de-facto forked from Fredrik Lundh's official releases and get on with our lives? Note the code review discussion here - http://codereview.appspot.com/207048/show - where Fredrik Lundh more or less acknowledges this fact and shows no real objections to it. By "get on with our lives" I mean keep fixing problems in ElementTree inside stdlib, as well as work on exposing the C implementation behind the ElementTree API by default, falling back on the Python API (and being true to PEP 399). Eli
Eli Bendersky, 10.02.2012 10:06:
On Fri, Feb 10, 2012 at 10:32, Florent wrote:
2012/2/10 Eli Bendersky
Thanks for the input, Florent. So, to paraphrase, there already are code changes in the stdlib version of ET/cET which are not upstream. You made it explicit about the tests, so the question is only left for the modules themselves. Is that right?
The port of ElementTree to Python 3000 was done in the standard library only. The work was done back in 2006, 2007 and 2008. There was never a public version of ElementTree for Python 3 outside of the standard library. It is already a significant change from the upstream branch (many changes in the C extension code).
Then when I enforced the same test suite for both implementation, I have fixed many things in the C extension module too. To my knowledge, these fixes were not included upstream.
Since two years, there was regular maintenance of the package in the standard library, but none of the patch were integrated upstream.
Folks, with this in mind, can we just acknowledge that the stdlib ElementTree is de-facto forked from Fredrik Lundh's official releases and get on with our lives? Note the code review discussion here - http://codereview.appspot.com/207048/show - where Fredrik Lundh more or less acknowledges this fact and shows no real objections to it.
By "get on with our lives" I mean keep fixing problems in ElementTree inside stdlib, as well as work on exposing the C implementation behind the ElementTree API by default, falling back on the Python API (and being true to PEP 399).
+1 None of this would make the situation any worse than it currently is, but provide serious improvements to the user experience. Stefan
Paul Moore wrote:
I would suggest that, assuming python-dev want to take ownership of the module, one last-ditch attempt be made to contact Fredrik. We should email him,
I wouldn't call email to be "last-ditch". I call email "first-ditch". I would expect that a last-ditch attempt would include trying to call him by phone, sending him a dead-tree letter by post, and if important enough, actually driving out to his home or place of work and trying to see him face to face. (All depending on the importance of making contact, naturally.) -- Steven
That said, I think that the particular change discussed in this thread can be made anyway, since it doesn't really modify ET's APIs or functionality, just the way it gets imported from stdlib.
I would suggest that, assuming python-dev want to take ownership of the module, one last-ditch attempt be made to contact Fredrik. We should email him, and copy python-dev (and maybe even python-list) asking for his view, and ideally his blessing on the stdlib version being forked and maintained independently going forward. Put a time limit on responses ("if we don't hear by XXX, we'll assume Fredrik is either uncontactable or not interested, and therefore we can go ahead with maintaining the stdlib version independently").
It's important to respect Fredrik's wishes and ownership, but we can't leave part of the stdlib frozen and abandoned just because he's not available any longer.
IMHO it's no longer a question of "wanting" to take ownership. According to Florent, this has already happened to some extent. Also, given the support history of ET outside stdlib, we can't in the same breath not take ownership and keep recommending this module. Lack of maintenance makes it a dead end, which is a shame given the choice of alternative modules for XML parsing in the stdlib. I don't mind sending Fredrik an email as you detailed. Any suggested things to include in it? Also, the most recent email (from 2009) of him I can find is "fredrik at pythonware.com". If anyone knows of anything more up-to-date, please let me know. Eli
IMHO it's no longer a question of "wanting" to take ownership. According to Florent, this has already happened to some extent.
"Ownership to some extent" is not a useful concept. Either you have ownership, or you don't.
I don't mind sending Fredrik an email as you detailed. Any suggested things to include in it?
I'd ask Fredrik if he wants to yield ownership, to some (specific) other person. What really worries me is the question who that other person is. There is a difference between fixing some issues, and actively taking over ownership, with all its consequences (long-term commitment, willingness to defend difficult decisions even if you are constantly being insulted for that decision, and so on). *Not* having an owner just means that it will be as unmaintained in the future as it appears to be now. Regards, Martin
On Fri, Feb 10, 2012 at 11:43, "Martin v. Löwis" <martin@v.loewis.de> wrote:
IMHO it's no longer a question of "wanting" to take ownership. According to Florent, this has already happened to some extent.
"Ownership to some extent" is not a useful concept. Either you have ownership, or you don't.
I don't mind sending Fredrik an email as you detailed. Any suggested things to include in it?
I'd ask Fredrik if he wants to yield ownership, to some (specific) other person.
What really worries me is the question who that other person is. There is a difference between fixing some issues, and actively taking over ownership, with all its consequences (long-term commitment, willingness to defend difficult decisions even if you are constantly being insulted for that decision, and so on).
*Not* having an owner just means that it will be as unmaintained in the future as it appears to be now.
How does this differ from any other module in stdlib that may not have a single designated owner, but which at the same time *is* being maintained by the core developers as a group? ISTM that requiring a five-year commitment is just going to scare any contributors away - is that what we want? What worries me most is that there seems to be a flow towards status quo on such things because status quo is the easiest to do. But in some cases, status quo is bad. Here we have a quite popular package in stdlib whose maintainer stopped maintaining it about two years ago. Another person stepped up and did some good work to bring the package up to date, fix bugs, and improve the test suite. What happens now? Do we give up on touching it until Fredrik Lundh decides on a come-back or some person who is willing to commit 5 years is found? Or do we just *keep* maintaining it in the stdlib as we do with other modules, fixing bugs, tests, documentation and so on? Eli
How does this differ from any other module in stdlib that may not have a single designated owner, but which at the same time *is* being maintained by the core developers as a group? ISTM that requiring a five-year commitment is just going to scare any contributors away - is that what we want?
I'm not talking about contributors, I'm talking about a maintainer. When we have a maintainer, that can actually attract contributors. Recognizing that something is a long-term commitment is scary, yes. However, a number of contributors has accepted such commitments, e.g. the release managers. Asking that for a subpackage is not asked too much, IMO. Compare this to distutils: if we've had a commitment of the original author to maintain that for a long period, the setuptools, distribute, distutils2, and packaging forks may not have been necessary. In absence of a maintainer, nobody is able to make difficult decisions.
What happens now? Do we give up on touching it until Fredrik Lundh decides on a come-back or some person who is willing to commit 5 years is found? Or do we just *keep* maintaining it in the stdlib as we do with other modules, fixing bugs, tests, documentation and so on?
If we really can't find somebody dedicated to that code base enough, we should consider removing it from the standard library. Regards, Martin
What happens now? Do we give up on touching it until Fredrik Lundh decides on a come-back or some person who is willing to commit 5 years is found? Or do we just *keep* maintaining it in the stdlib as we do with other modules, fixing bugs, tests, documentation and so on?
If we really can't find somebody dedicated to that code base enough, we should consider removing it from the standard library.
Does this imply that each and every package in the stdlib currently has a dedicated maintainer who promised to be dedicated to it? Or otherwise, should those packages that *don't* have a maintainer be removed from the standard library? Isn't that a bit harsh? ElementTree is an overall functional library and AFAIK the preferred stdlib tool for processing XML for many developers. It currently needs some attention to fix a few issues, expose the fast C implementation by default when ElementTree is imported, and improve the documentation. At this point, I'm interested enough to work on these - given that the political issue with Fredrik Lundh is resolved. However, I can't *honestly* say I promise to maintain the package until 2017. So, what's next? Eli
Does this imply that each and every package in the stdlib currently has a dedicated maintainer who promised to be dedicated to it? Or otherwise, should those packages that *don't* have a maintainer be removed from the standard library?
That is my opinion, yes. Some people (including myself) are willing to act as maintainers for large sets of modules, covering even code that they don't ever use themselves.
Isn't that a bit harsh? ElementTree is an overall functional library and AFAIK the preferred stdlib tool for processing XML for many developers. It currently needs some attention to fix a few issues, expose the fast C implementation by default when ElementTree is imported, and improve the documentation. At this point, I'm interested enough to work on these - given that the political issue with Fredrik Lundh is resolved. However, I can't *honestly* say I promise to maintain the package until 2017. So, what's next?
If you feel qualified to make changes, go ahead and make them. Take the praise if they are good changes, take the blame if they fire back. Please do try to stay around until either has happened. It would also good if you would declare "I will maintain the etree package". Regards, Martin
"Martin v. Löwis", 10.02.2012 11:32:
What happens now? Do we give up on touching it until Fredrik Lundh decides on a come-back or some person who is willing to commit 5 years is found? Or do we just *keep* maintaining it in the stdlib as we do with other modules, fixing bugs, tests, documentation and so on?
If we really can't find somebody dedicated to that code base enough, we should consider removing it from the standard library.
Well, that's totally not the current situation, though. There has been a large amount of maintenance going into the ElementTree modules already, so there is evidently a substantial interest in a) having them in the stdlib and b) keeping them working well. The current decisions could easily be taken by the interested parties, of which there seem to be enough involved in the relevant python-dev threads so far. Note that even decisions taken by a maintainer are not guaranteed to pass easily and without opposition. On a related note, it may be worth mentioning that it's generally known for several years now that the MiniDOM library has very serious performance problems, and there doesn't seem to be any maintainer around who has made a visible effort to solve them. Maybe we should remove MiniDOM from the stdlib, because no-one seems to be dedicated to that code base enough to fix it. Stefan
That makes me consider it the reality that "today, ET is only being maintained in the stdlib".
I think different people will have different perceptions of reality here. In my interaction with Fredrik Lundh, I got the impression that he might consider code still maintained even if he didn't touch it for five and more years - as he might get back some time and work on it, and as it may not have significant bugs that need fixing. If someone steps into charge and actually takes over ElementTree maintainance, that would be fine with me (as long as I have some trust that he'll continue to work on it for the next five years). We might have to "write off" contributions from Fredrik Lundh to Python because of that. Notice that the last time something like this came up (bsddb), it actually resulted in a removal of the respective package from the standard library.
Given that it was two months ago that I started the "Fixing the XML batteries" thread (and years since I brought up the topic for the first time), it seems to be hard enough already to get anyone on python-dev actually do something for Python's XML support, instead of just actively discouraging those who invest time and work into it.
It depends on the nature of the changes you want to see done. Just bashing some piece of code is not something that I personally consider a worthwhile thing, so I'll likely continue to discourage changes in a direction that demeans some XML library in favor of some other. Regards, Martin
"Martin v. Löwis", 10.02.2012 10:37:
Given that it was two months ago that I started the "Fixing the XML batteries" thread (and years since I brought up the topic for the first time), it seems to be hard enough already to get anyone on python-dev actually do something for Python's XML support, instead of just actively discouraging those who invest time and work into it.
It depends on the nature of the changes you want to see done. Just bashing some piece of code is not something that I personally consider a worthwhile thing, so I'll likely continue to discourage changes in a direction that demeans some XML library in favor of some other.
This is getting off-topic for this thread, but anyway. What I meant with my paragraph above was that none of the topics I brought up has received any action on the side of those with commit rights yet, regardless of how obvious they were and how much or little dispute there was about them. IMHO, all of this boils down to whether or not we should make it easier for users to efficiently use the stdlib. Backing ElementTree by the accelerator module helps here, and fixing the docs to point (new) users to ElementTree instead of MiniDOM helps as well. I can happily accept that you have a different opinion on the latter topic than I do. What I cannot accept is that, as we speak, this leads to users getting drawn into using the wrong tool for their job, into wasting their time (both for development and runtime) and potentially into getting drawn away from the (IMHO) perfect language for XML processing. I don't think bashing is the right word here. Everyone who, knowing the alternatives, decides to use MiniDOM is welcome to do so. I'm just stating, both from my personal experience and from discussions on c.l.py, that the current documentation makes it easier for new users to take the wrong decision for them than to make this decision in an informed way. MiniDOM *may* be the right thing further down along the way in some cases. It's almost *never* the right thing to start with, simply because if you do, it inherently takes way too much time until you reach the point where the evidence becomes obvious that it actually was the wrong decision. The documentation should allow innocent users to see this risk clearly before they start wasting their time. So, getting back to the topic again, is there any reason why you would oppose backing the ElementTree module in the stdlib by cElementTree's accelerator module? Or can we just consider this part of the discussion settled and start getting work done? Stefan
So, getting back to the topic again, is there any reason why you would oppose backing the ElementTree module in the stdlib by cElementTree's accelerator module? Or can we just consider this part of the discussion settled and start getting work done?
I'd still like to know who is in charge of the etree package now. I know that I'm not, so I just don't have any opinion on the technical question of using the accelerator module (it sounds like a reasonable idea, but it also sounds like something that may break existing code). If the maintainer of the etree package would pronounce that it is ok to make this change, I'd have no objection at all. Lacking a maintainer, I feel responsible for any bad consequences of that change, which makes me feel uneasy about it. Regards, Martin
I'd still like to know who is in charge of the etree package now. I know that I'm not, so I just don't have any opinion on the technical question of using the accelerator module (it sounds like a reasonable idea, but it also sounds like something that may break existing code). If the maintainer of the etree package would pronounce that it is ok to make this change, I'd have no objection at all. Lacking a maintainer, I feel responsible for any bad consequences of that change, which makes me feel uneasy about it.
Martin, as you've seen Fredrik Lundh finally officially ceded the maintenance of the ElementTree code to the Python developers: http://mail.python.org/pipermail/python-dev/2012-February/116389.html The change of backing ElementTree by cElementTree has already been implemented in the default branch (3.3) by Florent Xicluna with careful review from me and others. etree has an extensive (albeit a bit clumsy) set of tests which keep passing successfully after the change. The bots are also happy. In the past couple of years Florent has been the de-facto maintainer of etree in the standard library, although I don't think he ever "committed" to keep maintaining it for years to come. Neither can I make this commitment, however I do declare that I will do my best to keep the library functional, and I also plan to work on improving its documentation and cleaning up some of the accumulated cruft in its implementation. I also have all the intentions to take the blame if something breaks. That said, Florent is probably the one most familiar with the code at this point, and although his help will be most appreciated I can't expect or demand from him to stick around for a few years. We're all volunteers here, after all. Eli
The change of backing ElementTree by cElementTree has already been implemented in the default branch (3.3) by Florent Xicluna with careful review from me and others. etree has an extensive (albeit a bit clumsy) set of tests which keep passing successfully after the change.
I just noticed an incompatible change: xml.etree.ElementTree.Element used to be a type, but is now a function.
In the past couple of years Florent has been the de-facto maintainer of etree in the standard library, although I don't think he ever "committed" to keep maintaining it for years to come. Neither can I make this commitment, however I do declare that I will do my best to keep the library functional, and I also plan to work on improving its documentation and cleaning up some of the accumulated cruft in its implementation. I also have all the intentions to take the blame if something breaks.
Would you mind adding yourself to http://docs.python.org/devguide/experts.html Regards, Martin
On Mon, Feb 20, 2012 at 01:12, "Martin v. Löwis" <martin@v.loewis.de> wrote:
The change of backing ElementTree by cElementTree has already been implemented in the default branch (3.3) by Florent Xicluna with careful review from me and others. etree has an extensive (albeit a bit clumsy) set of tests which keep passing successfully after the change.
I just noticed an incompatible change: xml.etree.ElementTree.Element used to be a type, but is now a function.
Yes, this is a result of an incompatibility between the Python and C implementations of ElementTree. Since these have now been merged by default, one or the other had to be kept and the choice of cElementTree appeared to be sensible since this is what most people are expected to use in 2.7 and 3.2 anyway. I have an issue open for converting some function constructors into classes. Perhaps Element should also have this fate.
In the past couple of years Florent has been the de-facto maintainer of etree in the standard library, although I don't think he ever "committed" to keep maintaining it for years to come. Neither can I make this commitment, however I do declare that I will do my best to keep the library functional, and I also plan to work on improving its documentation and cleaning up some of the accumulated cruft in its implementation. I also have all the intentions to take the blame if something breaks.
Would you mind adding yourself to
Sure, I'll do that. Eli
On 2012-02-20, at 12:36 , Eli Bendersky wrote:
On Mon, Feb 20, 2012 at 01:12, "Martin v. Löwis" <martin@v.loewis.de> wrote:
The change of backing ElementTree by cElementTree has already been implemented in the default branch (3.3) by Florent Xicluna with careful review from me and others. etree has an extensive (albeit a bit clumsy) set of tests which keep passing successfully after the change.
I just noticed an incompatible change: xml.etree.ElementTree.Element used to be a type, but is now a function.
Yes, this is a result of an incompatibility between the Python and C implementations of ElementTree. Since these have now been merged by default, one or the other had to be kept and the choice of cElementTree appeared to be sensible since this is what most people are expected to use in 2.7 and 3.2 anyway. I have an issue open for converting some function constructors into classes. Perhaps Element should also have this fate.
I'm not sure that's much of an issue, Element (and most of the top-level utility "constructors") are documented as being frontend interfaces with no specific type of their own, and indeed they are simply functions in lxml, just as they are in cElementTree. Others will probably disagree, but as far as I am concerned these can stay functions, avoid issues when switching to lxml or between ElementTree and lxml (from one project to the next).
On Mon, Feb 20, 2012 at 9:43 PM, Xavier Morel <python-dev@masklinn.net> wrote:
I'm not sure that's much of an issue, Element (and most of the top-level utility "constructors") are documented as being frontend interfaces with no specific type of their own, and indeed they are simply functions in lxml, just as they are in cElementTree.
Others will probably disagree, but as far as I am concerned these can stay functions, avoid issues when switching to lxml or between ElementTree and lxml (from one project to the next).
For a similar situation (only the other way around), we're probably going to add a pure Python variant of functools.partial as a staticmethod, while the C version is a callable extension type. (see http://bugs.python.org/issue12428) Basically, if something is just documented as being callable without subclassing or instance checks being mentioned as supported in the docs, it can be implemented as either a type or an ordinary function, or pretty much any other kind of callable without being deemed an API change (obviously that's not a free pass to go make such implementation changes without a compelling reason, but C vs Python often *is* such a reason). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Basically, if something is just documented as being callable without subclassing or instance checks being mentioned as supported in the docs, it can be implemented as either a type or an ordinary function, or pretty much any other kind of callable without being deemed an API change
So what would be your evaluation of http://docs.python.org/library/xml.etree.elementtree.html#xml.etree.ElementT... in that respect? Regards, Martin
On Tue, Feb 21, 2012 at 1:55 AM, <martin@v.loewis.de> wrote:
Basically, if something is just documented as being callable without subclassing or instance checks being mentioned as supported in the docs, it can be implemented as either a type or an ordinary function, or pretty much any other kind of callable without being deemed an API change
So what would be your evaluation of
http://docs.python.org/library/xml.etree.elementtree.html#xml.etree.ElementT...
in that respect?
Completely different from the functools.partial case - with that, the docs are very careful to *never* call functools.partial a class (instead saying "returns a callable object"). The ElementTree docs unambiguously call Element a class (several times), so a conforming implementation must provide it as a class (i.e. supporting use in isinstance() checks. inheritance, etc) rather than as just a callable. A factory function is not a backwards compatible replacement (sorry Eli - given those docs, I'm definitely with Martin on this one). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 21, 2012 at 00:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Feb 21, 2012 at 1:55 AM, <martin@v.loewis.de> wrote:
Basically, if something is just documented as being callable without subclassing or instance checks being mentioned as supported in the docs, it can be implemented as either a type or an ordinary function, or pretty much any other kind of callable without being deemed an API change
So what would be your evaluation of
http://docs.python.org/library/xml.etree.elementtree.html#xml.etree.ElementT...
in that respect?
Completely different from the functools.partial case - with that, the docs are very careful to *never* call functools.partial a class (instead saying "returns a callable object").
The ElementTree docs unambiguously call Element a class (several times), so a conforming implementation must provide it as a class (i.e. supporting use in isinstance() checks. inheritance, etc) rather than as just a callable. A factory function is not a backwards compatible replacement (sorry Eli - given those docs, I'm definitely with Martin on this one).
No need to be sorry :-) I don't think my view differs from Martin's here, by the way. My point is just that this isn't a regression, since "use cElementTree" is ubiquitous advice, and the C implementation has Element as a factory function and not a class, so the documentation wasn't correct to begin with. So the documentation isn't correct for previous versions any way you look at it. There's a conflict in that it says Element is a class and also that cElementTree implements the same API. So the two choices here are either change the documentation or the C implementation to actually make Element a class. The first is of course simpler. However, someone somewhere may have written code that knowingly forces the Python implementation to be used and subclasses Element. Such code will break in 3.3, so it probably makes sense to invest in making Element a class in the C implementation as well. Eli
On Tue, Feb 21, 2012 at 11:39 AM, Eli Bendersky <eliben@gmail.com> wrote:
So the two choices here are either change the documentation or the C implementation to actually make Element a class. The first is of course simpler. However, someone somewhere may have written code that knowingly forces the Python implementation to be used and subclasses Element. Such code will break in 3.3, so it probably makes sense to invest in making Element a class in the C implementation as well.
Yeah, that's my take as well (especially since, in 3.2 and earlier, "forcing" use of the pure Python version was just a matter of importing ElementTree instead of cElementTree). While Xavier's point about lxml following cElementTree's lead and using a factory function is an interesting one, I think in this case the documented behaviour + pure Python implementation win out over the C accelerator behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Feb 21, 2012 at 03:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Feb 21, 2012 at 11:39 AM, Eli Bendersky <eliben@gmail.com> wrote:
So the two choices here are either change the documentation or the C implementation to actually make Element a class. The first is of course simpler. However, someone somewhere may have written code that knowingly forces the Python implementation to be used and subclasses Element. Such code will break in 3.3, so it probably makes sense to invest in making Element a class in the C implementation as well.
Yeah, that's my take as well (especially since, in 3.2 and earlier, "forcing" use of the pure Python version was just a matter of importing ElementTree instead of cElementTree).
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve? Eli
Am 21.02.2012 11:41, schrieb Eli Bendersky:
On Tue, Feb 21, 2012 at 03:59, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
On Tue, Feb 21, 2012 at 11:39 AM, Eli Bendersky <eliben@gmail.com <mailto:eliben@gmail.com>> wrote: > So the two choices here are either change the documentation or the C > implementation to actually make Element a class. The first is of course > simpler. However, someone somewhere may have written code that knowingly > forces the Python implementation to be used and subclasses Element. Such > code will break in 3.3, so it probably makes sense to invest in making > Element a class in the C implementation as well.
Yeah, that's my take as well (especially since, in 3.2 and earlier, "forcing" use of the pure Python version was just a matter of importing ElementTree instead of cElementTree).
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve?
Statements like this make me *extremely* worried. Please try to adopt a position of much higher caution, accepting that a change is "incompatible" if there is a remote possibility that someone might actually rely on the original behavior. Otherwise, I predict that you will get flooded with complaints that you broke ET for no good reason. In the specific case, I tried to write a script that determines the memory usage of ET. As Element is lacking gc.get_referents support, I tried isinstance(o, Element), which failed badly. Feel free to dismiss this application as irrelevant, but I do wish that somebody was in charge of ET who was taking backwards compatibility as serious as Fredrik Lundh. Regards, Martin
On Tue, Feb 21, 2012 at 10:47:43PM +0100, "Martin v. Löwis" wrote:
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve?
Statements like this make me *extremely* worried. Please try to adopt a position of much higher caution, accepting that a change is "incompatible" if there is a remote possibility that someone might actually rely on the original behavior. Otherwise, I predict that you will get flooded with complaints that you broke ET for no good reason.
I'm happy to stand up as an example of someone who uses a custom Element class. My specific use case is loading the project Gutenberg database, which is a 210MB XML file. I created a custom Element class which I use for the top-level element (a custom element_factory passed to TreeBuilder distinguishes between the top-level element and all others). The custom Element class doesn't add children, so it keeps ElementTree from storing all of the elements its seen so far. On a system with 1 GB of RAM, there was no other way to get the file to load. So, I would be one of those people who would flood in the complaints. :) -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Wed, Feb 22, 2012 at 8:16 AM, Andrew McNabb <amcnabb@mcnabbs.org> wrote:
So, I would be one of those people who would flood in the complaints. :)
As another "you don't know what you're going to break" war story: In Python 2.5, using "-m" with a package appeared to work, but actually slightly corrupted the import state (mostly in a benign way, but if it ever bit you it would lead to some very confusing behaviour). Since I'd never intended to allow that to happen (as I knew about the state corruption problem), for 2.6 I added back the "this doesn't work properly" guard that had been present in the earlier versions of 2.5, but had been lost when some duplicate code in pkgutil and runpy was merged into a single version. Doing that broke things: http://bugs.python.org/issue4195 The basic rule is, if it's documented to work a certain way and the current implementation does work that way, then, someone, somewhere is relying on it working as documented. If it *doesn't* actually work that way (or the behaviour isn't explicitly documented at all), then we have some leeway to decide whether to bring the docs in line with the actual behaviour or vice-versa. For the Element case though, there's no such discrepancy - the docs and implementation have been consistent for years, so we need to maintain the current behaviour if the C acceleration is going to be used implicitly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I'm happy to stand up as an example of someone who uses a custom Element class. My specific use case is loading the project Gutenberg database, which is a 210MB XML file. I created a custom Element class which I use for the top-level element (a custom element_factory passed to TreeBuilder distinguishes between the top-level element and all others). The custom Element class doesn't add children, so it keeps ElementTree from storing all of the elements its seen so far. On a system with 1 GB of RAM, there was no other way to get the file to load.
So, I would be one of those people who would flood in the complaints. :)
Andrew, could you elaborate on your use case? Are you using cElementTree to do the parsing, or ElementTree (the Python implementation). Can you show a short code sample? Thanks in advance, Eli
On Wed, Feb 22, 2012 at 04:24:38AM +0200, Eli Bendersky wrote:
Andrew, could you elaborate on your use case? Are you using cElementTree to do the parsing, or ElementTree (the Python implementation). Can you show a short code sample?
I'm mostly using ElementTree because several classes/functions that I need are not in cElementTree or behave differently. Specifically, the program loads TreeBuilder, XMLParser, iterparse from ElementTree; the only class from cElementTree that works is Element. A shortened version of the program is available here: http://aml.cs.byu.edu/~amcnabb/gutenberg-short.py By the way, this code is just one example of how one might rely on the documented extensibility of ElementTree. There are probably many other examples out there that look nothing like mine. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Wed, Feb 22, 2012 at 7:47 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Am 21.02.2012 11:41, schrieb Eli Bendersky:
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve?
Statements like this make me *extremely* worried. Please try to adopt a position of much higher caution, accepting that a change is "incompatible" if there is a remote possibility that someone might actually rely on the original behavior. Otherwise, I predict that you will get flooded with complaints that you broke ET for no good reason.
Indeed. It's a *major* PITA at times (and has definitely led to some ugly workarounds), but we have to take documented API compatibility very seriously. We're often even reluctant to change long-standing *de facto* behaviour, let alone things that are written up in the docs as being explicitly supported. In Python 3, merely saying "this class" or "this type" is as good as saying "this instance of the type metaclass" as far as API guarantees go. That's the reason for the awkward phrasing in the functools docs - specifically to *avoid* saying that functools.partial is a class, as we want to allow closure-based implementations as well. The key thing to remember is that the web-style "eh, just change it, people can fix their code to cope" mentality is a tiny *minority* in the global software development community. There's a huge amount of Python code out there, and a lot of it is hidden behind corporate firewalls. Our attention to backward compatibility concerns is one of the reasons why Python's reach extends into so many different areas. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve?
Statements like this make me *extremely* worried. Please try to adopt a position of much higher caution, accepting that a change is "incompatible" if there is a remote possibility that someone might actually rely on the original behavior. Otherwise, I predict that you will get flooded with complaints that you broke ET for no good reason.
No need to be worried, Martin. If you read back in this thread you'll see that I agree that backwards compatibility should be preserved, by making the Element exposed from _elementtree also a type. I was simply trying to have a discussion to better understand the use cases and implications. I hope that's alright.
In the specific case, I tried to write a script that determines the memory usage of ET. As Element is lacking gc.get_referents support, I tried isinstance(o, Element), which failed badly.
Thanks for describing the use case. By the way, when working with ET I also wanted to track the memory usage of the package a couple of times, which made me lament that there's no useful memory profiler in the stdlib. Eli
Thanks for describing the use case. By the way, when working with ET I also wanted to track the memory usage of the package a couple of times, which made me lament that there's no useful memory profiler in the stdlib.
A memory profiler can be a ten-line Python function which, however, does need to be tuned to the application. So I'm not sure it can be provided by the stdlib in a reasonable fashion beyond what's already there, but it may not be necessary to have it in the stdlib, either. Regards, Martin
On Tue, 21 Feb 2012 12:41:17 +0200 Eli Bendersky <eliben@gmail.com> wrote:
On Tue, Feb 21, 2012 at 03:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Feb 21, 2012 at 11:39 AM, Eli Bendersky <eliben@gmail.com> wrote:
So the two choices here are either change the documentation or the C implementation to actually make Element a class. The first is of course simpler. However, someone somewhere may have written code that knowingly forces the Python implementation to be used and subclasses Element. Such code will break in 3.3, so it probably makes sense to invest in making Element a class in the C implementation as well.
Yeah, that's my take as well (especially since, in 3.2 and earlier, "forcing" use of the pure Python version was just a matter of importing ElementTree instead of cElementTree).
I can't fathom why someone would do it though, since bar tiny differences (like this one) cET is just a faster ET and it's available practically everywhere with CPython. I mean, is it really important to be able to subclass ET.Element? What goal does it serve?
It probably wouldn't be very difficult to make element_new() the tp_new of Element_Type, and expose that type as "Element". That would settle the issue nicely and avoid compatibility concerns :) Regards Antoine.
It probably wouldn't be very difficult to make element_new() the tp_new of Element_Type, and expose that type as "Element". That would settle the issue nicely and avoid compatibility concerns :)
Regards
I guess it's not as simple as that. element_new doesn't quite have the signature required for tp_new. Besides, a constructor would also be needed (since a subclass may be interested in calling Element.__init__) and there's no natural function to serve as the constructor. I've opened issue 14128 to track this. I plan to implement a standard tp_new and tp_init functions for Element to expose it as a class from the module. element_new also happens to be used internally - I'll try to refactor to avoid code duplication as much as possible. Eli
On Fri, Feb 10, 2012 at 7:37 PM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
Notice that the last time something like this came up (bsddb), it actually resulted in a removal of the respective package from the standard library.
bsddb was a *very* different case - it was actively causing buildbot stability problems and various reports on the tracker due to changes in the external Berkeley DB API. Once we had sqlite3 in the standard lib as an alternate DB-API backend, it was hard to justify the ongoing maintenance hassles *despite* Jesus Cea stepping up as the maintainer (and he still maintains the pybsddb version - that was actually a big factor in *letting* us drop it, since we could just direct current users towards the PyPI version). Most orphan modules in the stdlib aren't like that - yes, their APIs stagnate (because nobody feels they have the authority and/or expertise to make potentially controversial decisions), but for many of them, that's not a particularly bad thing. For others, the world has moved on around them and they becomes traps for the unwary, but still, taking the modules out is unwarranted, since we'd be breaking code without giving affected users a good alternative (for orphan modules, nobody is likely to take the time to maintain them on PyPI if they weren't willing to do so in the stdlib - this actually stands in stark *contrast* to the bsddb case, which was decidedly *not* an orphan module when it was removed). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2012/2/10 Nick Coghlan <ncoghlan@gmail.com>:
Most orphan modules in the stdlib aren't like that - yes, their APIs stagnate (because nobody feels they have the authority and/or expertise to make potentially controversial decisions), but for many of them, that's not a particularly bad thing.
You're right, and sometimes a contributor steps in and propose a PEP to move things forward for a so-called orphan module. If I'm not wrong, it was the case for StringIO, pickle, distutils, wsgiref and optparse even if each of these packages has its own story. -- Florent Xicluna
<snip>
"""A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. The pickle / cPickle pair received this treatment. The profile module is on the list for 3.1. The StringIO module has been turned into a class in the io module."""
Is there a good reason why xml.etree.ElementTree / xml.etree.cElementTree did not "receive this treatment"?
<snip> Since there appeared to be an overall positive response for making this change in Python 3.3, and since there isn't longer any doubt about the ownership of the package *in Python's stdlib* (see http://mail.python.org/pipermail/python-dev/2012-February/116389.html), I've opened issue 13988 on the bug tracker to follow the implementation. Eli
Since there appeared to be an overall positive response for making this change in Python 3.3, and since there isn't longer any doubt about the ownership of the package *in Python's stdlib* (see http://mail.python.org/pipermail/python-dev/2012-February/116389.html), I've opened issue 13988 on the bug tracker to follow the implementation.
The change was committed to the default branch. In 3.3, "import xml.etree.ElementTree" will automatically use the _elementtree accelerator, if available, and will fall back to a Python implementation otherwise. The documentation of ElementTree has also been updated to reflect this fact. Thanks a lot to Florent Xicluna for the great co-operation, and all the others who submitted opinions in the issue. For more details see http://bugs.python.org/issue13988 Eli
Eli Bendersky, 13.02.2012 12:35:
Since there appeared to be an overall positive response for making this change in Python 3.3, and since there isn't longer any doubt about the ownership of the package *in Python's stdlib* (see http://mail.python.org/pipermail/python-dev/2012-February/116389.html), I've opened issue 13988 on the bug tracker to follow the implementation.
The change was committed to the default branch. In 3.3, "import xml.etree.ElementTree" will automatically use the _elementtree accelerator, if available, and will fall back to a Python implementation otherwise. The documentation of ElementTree has also been updated to reflect this fact.
Thanks! Stefan
The change was committed to the default branch. In 3.3, "import xml.etree.ElementTree" will automatically use the _elementtree accelerator, if available, and will fall back to a Python implementation otherwise. The documentation of ElementTree has also been updated to reflect this fact.
An open question remains on whether to deprecate cElementTree, now that this change is in place. Currently in 3.3 the whole cElementTree module is: # Deprecated alias for xml.etree.ElementTree from xml.etree.ElementTree import * Would it be alright to issue a DeprecationWarning if this module is imported? Then hopefully a couple of releases after 3.3 we can just dump it. Eli
On Tue, Feb 14, 2012 at 1:42 PM, Eli Bendersky <eliben@gmail.com> wrote:
An open question remains on whether to deprecate cElementTree, now that this change is in place.
Currently in 3.3 the whole cElementTree module is:
# Deprecated alias for xml.etree.ElementTree
from xml.etree.ElementTree import *
Would it be alright to issue a DeprecationWarning if this module is imported? Then hopefully a couple of releases after 3.3 we can just dump it.
What do we really gain by dumping it, though? Just add a CPython specific test that ensures: for key, value in xml.etree.ElementTree.__dict__.items(): self.assertIs(getattr(xml.etree.cElementTree, key), value) and then ignore it for the next decade or so. Programmatic deprecation is a significant imposition on third party developers and should really be reserved for APIs that actively encourage writing broken code (e.g. contextlib.nested) or are seriously problematic for python-dev to maintain. For cleanup stuff, documented deprecation is sufficient. Something that might be worth doing (although it would likely scare the peanut gallery) is to create a PEP 4000 to record the various cleanup tasks (like dropping cElementTree) that are being deliberately excluded from the 3.x series. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Currently in 3.3 the whole cElementTree module is:
# Deprecated alias for xml.etree.ElementTree
from xml.etree.ElementTree import *
Would it be alright to issue a DeprecationWarning if this module is imported? Then hopefully a couple of releases after 3.3 we can just dump
it.
What do we really gain by dumping it, though? Just add a CPython specific test that ensures:
for key, value in xml.etree.ElementTree.__dict__.items(): self.assertIs(getattr(xml.etree.cElementTree, key), value)
and then ignore it for the next decade or so.
With the deprecation warning being silent, is there much to lose, though? Cleanups help lower the clutter and mental burden on maintainers in the long run. If nothing is ever cleaned up don't we end up with PHP :-) ?
Programmatic deprecation is a significant imposition on third party developers and should really be reserved for APIs that actively encourage writing broken code (e.g. contextlib.nested) or are seriously problematic for python-dev to maintain. For cleanup stuff, documented deprecation is sufficient.
A quick search of the sources for DeprecationWarning show that it's being used much more liberally than solely for stuff that encourages writing broken code. Has there been a recent policy change with regards to what's considered deprecated? Eli
On Tue, Feb 14, 2012 at 2:25 PM, Eli Bendersky <eliben@gmail.com> wrote:
With the deprecation warning being silent, is there much to lose, though?
Yes, it creates problems for anyone that deliberately converts all warnings to errors when running their test suites. This forces them to spend time switching over to a Python version dependent import of either cElementTree or ElementTree that could have been spent doing something actually productive instead of mere busywork. And, of course, even people that *don't* convert warnings to errors when running tests will have to make the same switch when the module is eventually removed.
Cleanups help lower the clutter and mental burden on maintainers in the long run. If nothing is ever cleaned up don't we end up with PHP :-) ?
It's a balancing act, sure. But when the maintenance burden for us is low and the cost to third parties clear, documented deprecation for eventual removal in the next release series is the better choice.
Programmatic deprecation is a significant imposition on third party developers and should really be reserved for APIs that actively encourage writing broken code (e.g. contextlib.nested) or are seriously problematic for python-dev to maintain. For cleanup stuff, documented deprecation is sufficient.
A quick search of the sources for DeprecationWarning show that it's being used much more liberally than solely for stuff that encourages writing broken code. Has there been a recent policy change with regards to what's considered deprecated?
It's always been judged on a case-by-case basis, but yes, there's been a deliberate push in favour of purely documented deprecations in recent years (initially mostly from Raymond Hettinger, more recently from me as well as I came to appreciate the merit of Raymond's point of view). It mainly started with the decision to leave optparse alone (aside from a deprecation note in the docs) even after argparse was added to the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan, 14.02.2012 05:44:
On Tue, Feb 14, 2012 at 2:25 PM, Eli Bendersky wrote:
With the deprecation warning being silent, is there much to lose, though?
Yes, it creates problems for anyone that deliberately converts all warnings to errors when running their test suites. This forces them to spend time switching over to a Python version dependent import of either cElementTree or ElementTree that could have been spent doing something actually productive instead of mere busywork.
And, of course, even people that *don't* convert warnings to errors when running tests will have to make the same switch when the module is eventually removed.
I'm -1 on emitting a deprecation warning just because cElementTree is being replaced by a bare import. That's an implementation detail, just like cElementTree should have been an implementation detail in the first place. In all currently maintained CPython releases, importing cElementTree is the right thing to do for users. These days, other Python implementations already provide the cElementTree module as a bare alias for ElementTree.py anyway, without emitting any warnings. Why should CPython be the only one that shouts at users for importing it? Stefan
On 2012-02-14, at 08:58 , Stefan Behnel wrote:
These days, other Python implementations already provide the cElementTree module as a bare alias for ElementTree.py anyway, without emitting any warnings. Why should CPython be the only one that shouts at users for importing it?
Since all warnings are now silent by default (including DeprecationWarning), it's less of a shout and more of an eyebrow-frown and a tut-tuting really.
Le 14/02/2012 08:58, Stefan Behnel a écrit :
I'm -1 on emitting a deprecation warning just because cElementTree is being replaced by a bare import. That's an implementation detail, just like cElementTree should have been an implementation detail in the first place. In all currently maintained CPython releases, importing cElementTree is the right thing to do for users.
+1!
On 14/02/2012 9.58, Stefan Behnel wrote:
Nick Coghlan, 14.02.2012 05:44:
On Tue, Feb 14, 2012 at 2:25 PM, Eli Bendersky wrote:
With the deprecation warning being silent, is there much to lose, though? Yes, it creates problems for anyone that deliberately converts all warnings to errors when running their test suites. This forces them to spend time switching over to a Python version dependent import of either cElementTree or ElementTree that could have been spent doing something actually productive instead of mere busywork.
If I'm writing code that imports cElementTree on 3.3+, and I explicitly turn on DeprecationWarnings (that would otherwise be silenced) to check if I'm doing something wrong, I would like Python to tell me "You don't need to import that anymore, just use ElementTree.". If I'm also converting all the warnings to errors, it's probably because I really want my code to do the right thing and spending 1 minute to add/change two line of code to fix this won't probably bother me too much. Regular users won't even notice the warning, unless they stumble upon the note in the doc or enable the warnings (and eventually when the module is removed).
And, of course, even people that *don't* convert warnings to errors when running tests will have to make the same switch when the module is eventually removed.
When the module is eventually removed and you didn't warn them in advance, the situation is going to turn much worse, because their code will suddenly stop working once they upgrade to the newer version. I don't mind keeping the module and the warning around for a few versions and give enough time for everyone to update their imports, but if eventually the module is removed I don't want all these developers to come and say "why you removed cElementTree without saying anything and broke all my code?".
I'm -1 on emitting a deprecation warning just because cElementTree is being replaced by a bare import. That's an implementation detail, just like cElementTree should have been an implementation detail in the first place. In all currently maintained CPython releases, importing cElementTree is the right thing to do for users.
From 3.3 the right thing will be importing ElementTree, and at some point in the future that will be the only way to do it.
These days, other Python implementations already provide the cElementTree module as a bare alias for ElementTree.py anyway, without emitting any warnings. Why should CPython be the only one that shouts at users for importing it?
I would watch this from the opposite point of view. Why should the other Python implementation have a to keep around a dummy module due to a CPython implementation detail? If we all go through a deprecation process we will eventually be able to get rid of this. Best Regards, Ezio Melotti
Stefan
On Thu, 16 Feb 2012 19:32:24 +0200 Ezio Melotti <ezio.melotti@gmail.com> wrote:
If I'm writing code that imports cElementTree on 3.3+, and I explicitly turn on DeprecationWarnings (that would otherwise be silenced) to check if I'm doing something wrong, I would like Python to tell me "You don't need to import that anymore, just use ElementTree.". If I'm also converting all the warnings to errors, it's probably because I really want my code to do the right thing and spending 1 minute to add/change two line of code to fix this won't probably bother me too much.
But then you're going from a cumbersome situation (where you have to import cElementTree and then fallback on regular ElementTree) to an even more cumbersome one (where you have to first check the Python version, then conditionally import cElementTree, then fallback on regular ElementTree).
And, of course, even people that *don't* convert warnings to errors when running tests will have to make the same switch when the module is eventually removed.
When the module is eventually removed and you didn't warn them in advance, the situation is going to turn much worse, because their code will suddenly stop working once they upgrade to the newer version.
Why would we remove the module? It seems "supporting" it should be mostly trivial (it's an alias).
I would watch this from the opposite point of view. Why should the other Python implementation have a to keep around a dummy module due to a CPython implementation detail?
I don't know, but they already have this module, and it certainly costs them nothing to keep it. Regards Antoine.
I personally don't see any reason to drop a module that isn't terminally broken or unmaintainable, apart from scaring users away by making them think that we don't care about backward compatibility.
On 16/02/2012 19.55, Antoine Pitrou wrote:
On Thu, 16 Feb 2012 19:32:24 +0200 Ezio Melotti<ezio.melotti@gmail.com> wrote:
If I'm writing code that imports cElementTree on 3.3+, and I explicitly turn on DeprecationWarnings (that would otherwise be silenced) to check if I'm doing something wrong, I would like Python to tell me "You don't need to import that anymore, just use ElementTree.". If I'm also converting all the warnings to errors, it's probably because I really want my code to do the right thing and spending 1 minute to add/change two line of code to fix this won't probably bother me too much. But then you're going from a cumbersome situation (where you have to import cElementTree and then fallback on regular ElementTree) to an even more cumbersome one (where you have to first check the Python version, then conditionally import cElementTree, then fallback on regular ElementTree).
This is true if you need to support Python <=3.2, but on the long run this won't be needed anymore and a plain "import ElementTree" will be enough.
When the module is eventually removed and you didn't warn them in advance, the situation is going to turn much worse, because their code will suddenly stop working once they upgrade to the newer version. Why would we remove the module? It seems "supporting" it should be mostly trivial (it's an alias).
I'm assuming that eventually the module will be removed (maybe for Python 4?), and I don't expect nor want to seen it removed in the near future. If something gets removed it should be deprecated first, and it's usually better to deprecate it sooner so that the developers have more time to update their code. As I proposed on the tracker though, we could even delay the deprecation to 3.4 (by that time they might not need to support 3.2 anymore).
I would watch this from the opposite point of view. Why should the other Python implementation have a to keep around a dummy module due to a CPython implementation detail? I don't know, but they already have this module, and it certainly costs them nothing to keep it.
There will also be a cost if people keep importing cElementTree and fall back on ElementTree on failure even when this won't be necessary anymore. This also means that more people will have to fix their code if/when the module will be removed if they kept using cElementTree. They can also find cElementTree in old code/tutorial and figure out that it's better to use the C one because is faster and keep doing so because the only warning that would stop them is hidden in the doc. I think the problem with the DeprecationWarnings being too noisy was fixed by silencing them; if they are still too noisy then we need a better mechanism to warn people who care (and going to check the doc every once in a while to see if some new doc warning has been added doesn't strike me as a valid solution). Best Regards, Ezio Melotti
On Fri, Feb 17, 2012 at 4:29 AM, Ezio Melotti <ezio.melotti@gmail.com> wrote:
I'm assuming that eventually the module will be removed (maybe for Python 4?), and I don't expect nor want to seen it removed in the near future. If something gets removed it should be deprecated first, and it's usually better to deprecate it sooner so that the developers have more time to update their code.
Not really - as soon as we programmatically deprecate something, it means anyone with a strict warnings policy (or with customers that have such a policy) has to update their code *now*. (Previously it was even worse than that, which is why deprecation warnings are no longer displayed by default). For things that we have no intention of deprecating in 3.x, but will likely ditch in a hypothetical future Python 4000, we'll almost certainly do exactly what we did with Pyk: later in the 3.x series, add a "-4" command line switch and a sys.py4kwarning flag to trigger conditional deprecation warnings. So, assuming things continue as they have for the first couple of decades of Python's existence, we can probably start worrying about it some time around 2020 :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 18/02/2012 0.04, Nick Coghlan wrote:
On Fri, Feb 17, 2012 at 4:29 AM, Ezio Melotti<ezio.melotti@gmail.com> wrote:
I'm assuming that eventually the module will be removed (maybe for Python 4?), and I don't expect nor want to seen it removed in the near future. If something gets removed it should be deprecated first, and it's usually better to deprecate it sooner so that the developers have more time to update their code. Not really - as soon as we programmatically deprecate something, it means anyone with a strict warnings policy (or with customers that have such a policy) has to update their code *now*. (Previously it was even worse than that, which is why deprecation warnings are no longer displayed by default).
The ones with a strict warning policy should be ready to deal with this situation. A possible solution (that I already proposed a while ago) would be to reuse the 2to3 framework to provide fixers that could be used for these "mechanical" updates between 3.x releases. For example I wrote a 2to3 fixer to replace all the deprecate unittest methods (fail*, some assert*) with the correct ones, but this can't be used to fix them while moving from 3.1 to 3.2.
For things that we have no intention of deprecating in 3.x, but will likely ditch in a hypothetical future Python 4000, we'll almost certainly do exactly what we did with Pyk: later in the 3.x series, add a "-4" command line switch and a sys.py4kwarning flag to trigger conditional deprecation warnings.
I think Guido mentioned somewhere that this hypothetical Python 4000 will most likely be backward compatible, so we would still need a regular deprecation period.
So, assuming things continue as they have for the first couple of decades of Python's existence, we can probably start worrying about it some time around 2020 :)
What bothers me most is that a valid mechanism to warn users who cares about things that will be removed is being hindered in several ways. DeprecationWarnings were first silenced (and this is fine as long as the developers are educated to enable warnings while testing), now discouraged (because people are still able to make them visible and also to turn them into errors), and on the tracker there's even a discussion about making the deprecation notes in the doc less visible (because the red boxes are too "scary"). See also http://mail.python.org/pipermail/python-dev/2011-October/114199.html Best Regards, Ezio Melotti
Cheers, Nick.
On 17 February 2012 04:55, Antoine Pitrou <solipsis@pitrou.net> wrote:
But then you're going from a cumbersome situation (where you have to import cElementTree and then fallback on regular ElementTree) to an even more cumbersome one (where you have to first check the Python version, then conditionally import cElementTree, then fallback on regular ElementTree).
Well, you can reverse the import so you're not relying on version numbers: import xml.etree.ElementTree as ElementTree try: import xml.etree.cElementTree as ElementTree except ImportError: pass There is a slight cost compared to previously (always importing the python version) and you'll still be using cElementTree directly until it's removed, but if/when it is removed you won't notice it. Tim Delaney
On Feb 13, 2012, at 8:44 PM, Nick Coghlan wrote:
On Tue, Feb 14, 2012 at 2:25 PM, Eli Bendersky <eliben@gmail.com> wrote:
With the deprecation warning being silent, is there much to lose, though?
Yes, it creates problems for anyone that deliberately converts all warnings to errors when running their test suites. This forces them to spend time switching over to a Python version dependent import of either cElementTree or ElementTree that could have been spent doing something actually productive instead of mere busywork.
And, of course, even people that *don't* convert warnings to errors when running tests will have to make the same switch when the module is eventually removed.
What about a PendingDeprecationWarning? I think you're usually only going to convert DeprecationWarnings to errors (with python -W error::DeprecationWarning or warnings.simplefilter('error', DeprecationWarning)) -- Philip Jenvey
On Thu, Feb 16, 2012 at 1:40 PM, Philip Jenvey <pjenvey@underboss.org> wrote:
What about a PendingDeprecationWarning? I think you're usually only going to convert DeprecationWarnings to errors (with python -W error::DeprecationWarning or warnings.simplefilter('error', DeprecationWarning))
Think "-Wall" for strict testing regimes :) If you trawl back in the archives a few years, you'll find I've historically been on the *other* side of this kind of argument. I've since come to recognise that programmatic deprecation really is a big hammer that hits the wider Python community - it needs to be treated with appropriate respect. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Feb 13, 2012 at 23:16, Nick Coghlan <ncoghlan@gmail.com> wrote:
An open question remains on whether to deprecate cElementTree, now that
On Tue, Feb 14, 2012 at 1:42 PM, Eli Bendersky <eliben@gmail.com> wrote: this
change is in place.
Currently in 3.3 the whole cElementTree module is:
# Deprecated alias for xml.etree.ElementTree
from xml.etree.ElementTree import *
Would it be alright to issue a DeprecationWarning if this module is imported? Then hopefully a couple of releases after 3.3 we can just dump it.
What do we really gain by dumping it, though? Just add a CPython specific test that ensures:
for key, value in xml.etree.ElementTree.__dict__.items(): self.assertIs(getattr(xml.etree.cElementTree, key), value)
and then ignore it for the next decade or so.
Programmatic deprecation is a significant imposition on third party developers and should really be reserved for APIs that actively encourage writing broken code (e.g. contextlib.nested) or are seriously problematic for python-dev to maintain. For cleanup stuff, documented deprecation is sufficient.
Something that might be worth doing (although it would likely scare the peanut gallery) is to create a PEP 4000 to record the various cleanup tasks (like dropping cElementTree) that are being deliberately excluded from the 3.x series.
I honestly think a PEP 4000 is a good idea simply to document stuff that we are allowing to exist in Python 3 but don't think people should necessarily be using in order to follow best practices (e.g. this, ditching optparse, no more % string formatting, etc.).
participants (20)
-
"Martin v. Löwis" -
Andrew McNabb -
Antoine Pitrou -
Brett Cannon -
Brian Curtin -
Charles-François Natali -
Dirkjan Ochtman -
Eli Bendersky -
Ezio Melotti -
Florent -
Fred Drake -
martin＠v.loewis.de
-
Nick Coghlan -
Paul Moore -
Philip Jenvey -
Stefan Behnel -
Steven D'Aprano -
Tim Delaney -
Xavier Morel -
Éric Araujo