Le Thu, 18 Feb 2010 22:46:41 +0100, Martin v. Löwis a écrit :

...

It's time to comment and review.

Unfortunately, it's not. I strongly object to any substantial change to the code base without explicit approval by Fredrik Lundh.

Which most probably puts elementtree in bugfix-only mode. I don't necessarily disagree with such a decision, but it must be quite clear. Regards Antoine.

On Fri, Feb 19, 2010 at 7:40 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

Which most probably puts elementtree in bugfix-only mode. I don't necessarily disagree with such a decision, but it must be quite clear.

The current situation is even worse than bugfix-only mode. Even bugfixes struggle to make it in.

My point is that the decision as already made when ElementTree was incorporated into the standard library; it's the same policy for most code that Fredrik Lundh has contributed (and which he still maintains outside the standard library as well). He has made it fairly clear on several occasions that this is how he expects things to work, and unless we want to truly fork the code, we should comply.

We need someone to maintain the copy of ElementTree in the Python repository. Ideally this means pulling upgrades and bugfixes from Fredrik's repository every now and then. If the goals of Python ElementTree and Fredrik ElementTree diverge I don't see a problem with an amicable fork. Fredrik and Python ElementTree do have rather different constraints (for example, Python ElementTree has fewer opportunities for breaking backwards compatibility). Schiavo Simon

All, I hope that Fredrik himself has time to chime in at least briefly, but he told me off-line that he sees nothing controversial in the currently proposed set of changes. On Fri, Feb 19, 2010 at 5:04 AM, Jeroen Ruigrok van der Werven <asmodai@in-nomine.org> wrote:

-On [20100219 08:37], Simon Cross (hodgestar+pythondev@gmail.com) wrote:

...

We need someone to maintain the copy of ElementTree in the Python repository. Ideally this means pulling upgrades and bugfixes from Fredrik's repository every now and then.

Which will give you nothing as that tree hasn't been touched in over three years.

I can understand giving special consideration to maintainers, but that would imply they actually maintain something, no?

-- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Contentment that derives from knowing when to be content is eternal contentment... _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

We need someone to maintain the copy of ElementTree in the Python repository.

We have one: Fredrik Lundh.

Ideally this means pulling upgrades and bugfixes from Fredrik's repository every now and then. If the goals of Python ElementTree and Fredrik ElementTree diverge I don't see a problem with an amicable fork.

I see one: Fredrik will not consider such a fork amicable. Of course, if you could make him state in public that he is fine with a procedure that you propose, go ahead. He had stated in public that he is fine with the procedure I'm defending right now, that's why I'm defending it: no substantial changes without his explicit approval (breakage due to language changes is about the only exception - not even bug fixes are allowed). Regards, Martin

Florent Xicluna, 18.02.2010 10:21:

For this purpose, I grew the test suite from 300 lines to 1800 lines, using both the tests from upstream and the tests proposed by Neil Muller on issue #6232.

Just a comment on this. While the new tests may work with ElementTree as is, there are a couple of problem with them. They become apparent when running the test suite against lxml.etree. Some of the tests depend on specifics of the serialiser that may not be guaranteed, such as the order of attributes or namespace declarations in a tag, or whitespace before the self-closing tag terminator (" />"). ET 1.3 has a mostly redesigned serialiser, and it may receive another couple of improvements or changes before it comes out. None of theses features is really required to hold for anything but the current as-is implementation. Other tests rely on non-XML being serialised, such as 'text<subtag>subtext</subtag>' This is achieved by setting "root.tag" to None - I'm not sure this is a feature of ElementTree, and I'd be surprised if it was documented anywhere. Several of the tests also use invalid namespace URIs like "uri" or non well-formed attribute names like "123". That's bad style at best. There are also some tests for implementation details, such as the "_cache" in ElementPath or the parser version (requiring expat), or even this test: element.getiterator == element.iter which doesn't apply to lxml.etree, as its element.getiterator() complies with the ET 1.2 interface for compatibility, whereas the new element.iter() obeys the ET 1.3 interface that defines it. Asserting both to be equal doesn't make much sense in the context of their specification. Another example is check_method(element.findall("*").next) In lxml.etree, this produces an AttributeError: 'list' object has no attribute 'next' because element.findall() is specified in the official ET documentation to return "a list or iterator", i.e. not necessarily an iterator but always an iterable. There is an iterfind() that would do the above, which matches ET 1.3. So my impression is that many of the tests try to provide guarantees where they cannot or should not exist, and even where the output is clearly non-conforming with respect to standards. I don't think it makes sense to put these into a regression test suite. That said, I should add that lxml's test suite includes about 250 unit tests that work with (and adapt to) lxml.etree, ElementTree and cElementTree, in Py2.3+ and Py3.x, and with ET 1.2 and ET 1.3. Although certainly not a copy&run replacement, those should be much better suited to accompany the existing stdlib tests. Stefan

Florent Xicluna, 20.02.2010 11:53:

Stefan Behnel writes:

...

None of theses features is really required to hold for anything but the current as-is implementation.

I agree.

...

So my impression is that many of the tests try to provide guarantees where they cannot or should not exist, and even where the output is clearly non-conforming with respect to standards. I don't think it makes sense to put these into a regression test suite.

The test suite in the stdlib should try to cover every piece of code, even implementation details and edge cases.

It shouldn't test for un(der)defined or non-guaranteed behaviour, bugs or the /absence/ of certain features, though. Apart from that, I agree with the following:

It guarantees that the implementation details do not change between minor releases. And it helps identify regression or evolution of the behavior when the library is updated.

Additionally, these tests ensure that the C implementation can be used as a drop-in replacement of the Python implementation. It is a request expressed by many users of the "xml.etree" package.

That's certainly a worthy goal, but it's orthogonal to the interest of changing/improving/evolving ElementTree itself. The goal here is to make cElementTree compatible with ET, without any impact on ET. I agree with Fredrik that there are a number of additional features in ET 1.3 (and lxml 2.x) that can be easily added to the existing API, e.g. the element.extend() method. Other new features (and certainly the incompatible changes) are a lot more controversial, though.

...

That said, I should add that lxml's test suite includes about 250 unit tests that work with (and adapt to) lxml.etree, ElementTree and cElementTree, in Py2.3+ and Py3.x, and with ET 1.2 and ET 1.3. Although certainly not a copy&run replacement, those should be much better suited to accompany the existing stdlib tests.

Interesting. I may add these tests to the test_suite, if they are not completely redundant.

They certainly were not redundant with the original tests that shipped with ET, and they test all sorts of funny cases. The main files are https://codespeak.net/viewvc/lxml/trunk/src/lxml/tests/test_elementtree.py?v... https://codespeak.net/viewvc/lxml/trunk/src/lxml/tests/test_io.py?view=marku... There are also a number of ET API tests in other parts of the test suite (mostly test_etree.py, also test_unicode.py). Some of them would work with ET but produce different results due to various reasons, including the fact that lxml.etree behaves "more correct" in some cases. The latter are the kind of tests that I would prefer not to see in the stdlib test suite. Stefan

The last commits by Fredrik to ElementTree in Python SVN that I can see are dated 2006-08-16. The last commits I can see to ElementTree at http://svn.effbot.python-hosting.com/ are dated 2006-07-05.

And?

To paraphrase Antoine's comment [1] on Rietveld -- we need a process that results in bug fixes for users of the copy of ElementTree in Python.

[1] http://codereview.appspot.com/207048/show (most direct link I could find)

To quote Fredrik Lundh from the same reviews: # You do realize that you're merging in an experimental release, right? # I'm a bit worried that the result of this effort will be plenty of # incompatibilities with the upstream library (and there are also signs # on bugs.python.org that some people involved don't understand the # difference between specification of a portable API and artifacts of a # certain implementation of the same API), but I'm travelling right now, # and have no bandwidth to deal with this. Just be careful. # Since you've effectively hijacked the library, and have created your # own fork that's not fully compatible with any formal release of the # upstream library, and am not contributing any patches back to # upstream, I suggest renaming it instead. This may be politely phrased, but it seems that he is quite upset about these proposed changes. I'd rather drop ElementTree from the standard library than fork it. Regards, Martin

Le Sat, 20 Feb 2010 13:08:39 +0100, Martin v. Löwis a écrit :

Please be EXTREMELY careful. I urge you not to act on this until mid-March (which is the earliest time at which Fredrik has said he may have time to look into this).

Ok, so let's wait until then before we make a decision. cheers Antoine.

-On [20100220 13:04], "Martin v. Löwis" (martin@v.loewis.de) wrote:

...

The last commits by Fredrik to ElementTree in Python SVN that I can see are dated 2006-08-16. The last commits I can see to ElementTree at http://svn.effbot.python-hosting.com/ are dated 2006-07-05.

And?

[snip]

# Since you've effectively hijacked the library, and have created your # own fork that's not fully compatible with any formal release of the # upstream library, and am not contributing any patches back to # upstream, I suggest renaming it instead.

This may be politely phrased, but it seems that he is quite upset about these proposed changes.

I'd rather drop ElementTree from the standard library than fork it.

Maybe I am fully misunderstanding something here and I am also known for just bluntly stating things but: Isn't inclusion into the standard library under the assumption that maintenance will be performed on the code? With all due respect to Frederik, but if you add such a module to the base distribution and then ignore it for 3-4 years I personally have a hard time feeling your 'outrage' being justified for someone who is trying to fix outstanding issues in ElementTree. I also do not find your idea of dropping the module productive either Martin. Just dropping it for no other reason because someone cannot be bothered to act as a responsible maintainer just seems not useful for Python users at all. Especially since patches *are* available. If Frederik has problems with that he should have put a bit more effort into maintaining it in the first place. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B In this short time of promise, you're a memory...

-On [20100220 22:47], "Martin v. Löwis" (martin@v.loewis.de) wrote:

In general, that's the assumption, and Guido has stated that he dislikes exceptions. However, Fredrik's code was included only under the exception. ElementTree wouldn't be part of the standard library if an exception had not been made.

I was not fully aware of that bit of history, my thanks for enlightening me on it.

If users and co-developers think that these issues absolutely must be resolved now (rather than waiting some more), I see only two options: a) ElementTree is removed from the library b) we declare that we fork ElementTree, and designate a maintainer.

Just fixing the bugs without designating a maintainer is *not* an option, because we absolutely need somebody to pronounce on changes. It will not be Guido, and if it is not Fredrik, somebody else must step forward. I would then ask that person, as the first thing, to rename the package when making incompatible changes.

Call me a sceptic or pragmatist, but I don't see the situation change suddenly from what it has been for the past couple of years. I vaguely remember running into problems or limitations myself with ElementTree and switching to lxml at one point. It sort of has to escalate now in order to get the maintainer to look at it and I doubt that's how we want to keep operating in the future? So the choice of removal or forking may actually be quite imminent, but of course, that's my interpretation of things.

Well, I promised that we will stick to the procedure when integrating ElementTree. I'm not willing to break this promise.

Honourable and I can understand that. Although it doesn't make it flexible to work on. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B The fragrance always stays in the hand that gives the rose...

2010/2/28 Stefan Behnel <stefan_ml@behnel.de>

I would actually encourage Florent to do the opposite: act now and prepare a patch against the latest official ET 1.2 and cET releases (or their SVN version respectively) that integrates everything that is considered safe, i.e. everything that makes cET compatible with ET and everything that seems clearly stable in ET 1.3 and does not break compatibility for existing code that uses ET 1.2. If you send that to Fredrik, I expect little opposition to making that the base for a 1.2.8 release, which can then be folded back into the stdlib.

I exchanged some e-mails with Fredrik last week. Not sure if it will be 1.2.8 or 1.3, but now he is positive on the goals of the patch. I've commited all the changes and external fixes to a branch of the Mercurial repo owned by Fredrik. I'm expecting an answer soon. Branch based on the official etree repository (Mercurial): http://bitbucket.org/flox/et-2009-provolone/ Patch based on this branch: http://codereview.appspot.com/207048 (patch set 7 almost identical to the tip of the Mercurial repo) -- Florent