On 2013-03-06 14:18, Amaury Forgeot d'Arc wrote:

Hi,

2013/3/6 Matěj Cepl mailto:mcepl@redhat.com>

On 2013-02-26, 16:25 GMT, Terry Reedy wrote: > On 2/21/2013 4:22 PM, Matej Cepl wrote: >> as my method to commemorate Aaron Swartz, I have decided to port his >> html2text to work fully with the latest python 3.3. After some time >> dealing with various bugs, I have now in my repo >> https://github.com/mcepl/html2text (branch python3) working solution >> which works all the way to python 3.2 (inclusive; >> https://travis-ci.org/mcepl/html2text). However, the last problem >> remains. This >> >> <li>Run this command: >> <pre>ls -l *.html</pre></li> >> <li>?</li> >> >> should lead to >> >> * Run this command: >> >> ls -l *.html >> >> * ? >> >> but it doesn’t. It leads to this (with python 3.3 only) >> >> * Run this command: >> ls -l *.html >> >> * ? >> >> Does anybody know about something which changed in modules re or >> http://docs.python.org/3.3/whatsnew/changelog.html between 3.2 and >> 3.3, which could influence this script? > > Search the changelob or 3.3 misc/News for items affecting those two > modules. There are at least 4. > http://docs.python.org/3.3/whatsnew/changelog.html > > It is faintly possible that the switch from narrow/wide builds to > unified builds somehow affected that. Have you tested with 2.7/3.2 on > both narrow and wide unicode builds?

So, in the end, I have went the long way and bisected cpython to find the commit which broke my tests, and it seems that the culprit is http://hg.python.org/cpython/rev/123f2dc08b3e so it is clearly something Unicode related.

Unfortunately, it really doesn't tell me what exactly is broken (is it a known regression) and if there is known workaround. Could anybody suggest a way how to find bugs on http://bugs.python.org related to some particular commit (plain search for 123f2dc0 didn’t find anything).

I strongly suspect an incorrect usage of the "is" operator: https://github.com/mcepl/html2text/blob/master/html2text.py#L95 Identity of strings is not guaranteed...

Does it change something if you use "==" instead?

That function looks a little odd to me. Maybe I just don't understand what it's doing! :-)

On 2013-03-06, 18:34 GMT, Victor Stinner wrote:

In short, Unicode was rewritten in Python 3.3 for the PEP 393. It's not surprising that minor details like singleton differ. You should not use "is" to compare strings in Python, or your program will fail on other Python implementations (like PyPy, IronPython, or Jython) or even on a different CPython version.

I am sorry, I don't understand what you are saying. Even though this has been changed to https://github.com/mcepl/html2text/blob/fix_tests/html2text.py#L90 the tests still fail. But, Amaury is right: the function doesn't make much sense. However, ... when I have “fixed it” from https://github.com/mcepl/html2text/blob/master/html2text.py#L95 def onlywhite(line): """Return true if the line does only consist of whitespace characters.""" for c in line: if c is not ' ' and c is not ' ': return c is ' ' return line to https://github.com/mcepl/html2text/blob/fix_tests/html2text.py#L90 def onlywhite(line): """Return true if the line does only consist of whitespace characters.""" for c in line: if c != ' ' and c != ' ': return c == ' ' return line tests on ALL versions of Python are suddenly failing ... https://travis-ci.org/mcepl/html2text/builds/5288190 Curiouser and curiouser! At least, I seem to have the point, where things are breaking, but I have to admit that condition really doesn’t make any sense to me.

Anyway, you spotted a missed optimization: it's now "fixed" in Python 3.3 and 3.4 by the following commits.

Well, whatever is the problem, it is not fixed in python 3.3.0 (as you can see in https://travis-ci.org/mcepl/html2text/builds/4969045) as I can see on my computer. Actually, good news is that it seems to be fixed in the master branch of cpython (or the tip, as they say in the Mercurial world). Any thoughts? Matěj

You should try to write a simple test not using your library (just copy/paste code) reproducing the issue. If you can do that, please fill an issue on bugs.python.org. Victor 2013/3/7 Matej Cepl :

On 2013-03-06, 18:34 GMT, Victor Stinner wrote:

...

In short, Unicode was rewritten in Python 3.3 for the PEP 393. It's not surprising that minor details like singleton differ. You should not use "is" to compare strings in Python, or your program will fail on other Python implementations (like PyPy, IronPython, or Jython) or even on a different CPython version.

I am sorry, I don't understand what you are saying. Even though this has been changed to https://github.com/mcepl/html2text/blob/fix_tests/html2text.py#L90 the tests still fail.

But, Amaury is right: the function doesn't make much sense. However, ...

when I have “fixed it” from https://github.com/mcepl/html2text/blob/master/html2text.py#L95

def onlywhite(line): """Return true if the line does only consist of whitespace characters.""" for c in line: if c is not ' ' and c is not ' ': return c is ' ' return line

to https://github.com/mcepl/html2text/blob/fix_tests/html2text.py#L90

def onlywhite(line): """Return true if the line does only consist of whitespace characters.""" for c in line: if c != ' ' and c != ' ': return c == ' ' return line

tests on ALL versions of Python are suddenly failing ... https://travis-ci.org/mcepl/html2text/builds/5288190

Curiouser and curiouser! At least, I seem to have the point, where things are breaking, but I have to admit that condition really doesn’t make any sense to me.

...

Anyway, you spotted a missed optimization: it's now "fixed" in Python 3.3 and 3.4 by the following commits.

Well, whatever is the problem, it is not fixed in python 3.3.0 (as you can see in https://travis-ci.org/mcepl/html2text/builds/4969045) as I can see on my computer. Actually, good news is that it seems to be fixed in the master branch of cpython (or the tip, as they say in the Mercurial world).

Any thoughts?

Matěj _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

Hi Matej, On Thu, Mar 7, 2013 at 11:08 AM, Matej Cepl wrote:

if c is not ' ' and c is not ' ': if c != ' ' and c != ' ':

Sorry for the delay in answering, but I just noticed what is wrong in this "fix": it compares c with the same single-character ' ' twice, whereas the original compared it with ' ' and with the two-character ' '. A bientôt, Armin.

On 05/05/2013 23:01, Matej Cepl wrote:

----- Original Message -----

...

From: "Armin Rigo" To: "Matej Cepl" Cc: python-dev@python.org Sent: Saturday, May 4, 2013 11:59:42 AM Subject: Re: [Python-Dev] Difference in RE between 3.2 and 3.3 (or Aaron Swartz memorial)

Hi Matej,

On Thu, Mar 7, 2013 at 11:08 AM, Matej Cepl wrote:

...

if c is not ' ' and c is not ' ': if c != ' ' and c != ' ':

Sorry for the delay in answering, but I just noticed what is wrong in this "fix": it compares c with the same single-character ' ' twice, whereas the original compared it with ' ' and with the two-character '

Comments on https://github.com/mcepl/html2text/commit/f511f3c78e60d7734d677f8945580f52ef... (perhaps in https://github.com/aaronsw/html2text/pull/77) are more than welcome. When using

SPACE_RE = re.compile(r'\s\+')

That will match a whitespace character followed by a '+'.

for checking, whole onlywhite function is not needed anymore (and it still made me wonder what Aaron meant when he wrote it). Why line.isspace() doesn't work is weird though.

What do you mean by "doesn't work"?