[Python-Dev] status of development documentation

[Samuele Pedroni]

Tim Peters wrote:
> [Neal]
> 
>>Hmmm, I thought others were running the tests on Windows too.  There
>>was one report on Nov 22 about running Purify on Windows 2k (subject:
>>ast status, memory leaks, etc).  He had problems with a stack overflow
>>in test_compile.  He was going to disable the test and re-run.  I
>>never heard back though.  Based on that info, I would guess that
>>test_builtin was working on Win 2k on Nov 22.
> 
> 
> I wouldn't assume that.  My "nobody" was wrt the universe of Python
> developers, not users; folks like Trent and MarkH and you and me. 
> Without "normal" baseline test experience, users don't understand what
> they're seeing, and so their reports can be highly misleading.  You
> can trust that while I don't understand what I'm seeing here either,
> at least I told the absolute truth about it and didn't hold back
> critical details ;-)
> 
> That said, I was hoping to do some Python work over Thanksgiving week,
> but was mortally discouraged on the Nov 19-20 weekend by all the test
> failures I saw.  So I'm pretty sure (but not certain) that
> test_builtin was failing then.
> 
> 
>>>(A parenthentical question:  is there a reason you don't pass -uall to
>>>regrtest.py?)
> 
> 
>>It's calling make test.  I thought about calling regrtest.py instead
>>and doing as you suggest.  Is there a benefit to running make test?
> 
> 
> You're asking a Windows guy about make:  bad career move ;-)
> 
> 
>>I know it runs with and without -O.  I guess it's only machine time, I
>>could run make test and regrtest.py -uall.
> 
> 
> -uall is helpful in finding bugs.  One thing in particular here is
> that test_compiler runs only a tiny subset of its full test unless an
> appropriate -u flag is given.
> 
> 
>>>On WinXP Pro SP2 today, passing -uall, and after fixing all the MS
>>>compiler warnings that have crept in:
>>>
>>>251 tests OK.
>>>12 tests failed:
>>>    test_builtin test_coding test_compiler test_pep263
>>>    test_univnewlines test_urllib test_urllib2 test_urllibnet
>>>    test_userlist test_wave test_whichdb test_zipfile
>>>1 skip unexpected on win32:
>>>    test_xml_etree_c
> 
> 
>>Ouch!  I'm might be to blame for at least some of those. :-(
> 
> 
> I'm sure it's not as bad as it looks.  For example, test_coding and
> (the -uall) test_compiler fail for exactly the same reason.  For
> another, when a test fails on Windows, it sometimes leaves a "@test"
> file or directory behind, which causes a cascade of bogus failures
> later.  For example, test_userlist, test_wave, test_whichdb, and
> test_zipfile all pass when run alone here.  Others probably do too.
> 
> ...
> 
>>Do you know if the tests were broken before the AST merge (about Oct
>>22 I think)?
> 
> 
> I don't know.  I'm getting more evidence that most (if not all) of the
> failures are due to compile-time parsing screwups, so the AST merge is
> a prime suspect.
> 
> Is it possible that generated parse tables (whatever) are out-of-date
> on a Windows box?  There are no makefiles here, and the Windows build
> has always relied on Unix-heads to check in correct generated parser
> files.
> 
> 
>>>The code up to the first failure is short:
>>>
>>>        bom = '\xef\xbb\xbf'
>>>        compile(bom + 'print 1\n', '', 'exec')
>>>
>>>Curiously, that sequence doesn't blow up under the released Windows
>>>Python 2.4.2, so somebody broke something here since then ...
> 
> 
>>There were a bunch of changes to Parser/tokenizer.c to handle error
>>conditions.  Those go back to Oct 1.  I don't *think* those would
>>cause these, but I'm not sure.
>>
>>Sorry, I don't know any more.  I guess you might have to binary search
>>by date to try and find the problem.
> 
> 
> That's just silly ;-)  What I need is someone who understands what
> this code is _supposed_ to do, so we can fix it; finding the checkin
> that caused it isn't nearly as interesting.  Windows has an excellent
> debug-build debugger and I can easily step through the code.  But I
> have no idea why compiling a string starting with  '\xef\xbb\xbf'
> should _not_ be a syntax error -- it looks like a syntax error to me.
> 
> And when I step thru the code, it looks like a syntax error to the
> parser too.  It peels off the first character (\xef), and says "syntax
> error" at that point:
> 
> Py_CompileStringFlags ->
> PyParser_ASTFromString ->
> PyParser_ParseStringFlagsFilename ->
> parsetok ->
> PyTokenizer_Get
> 
> That sets `a` to point at the start of the string, `b` to point at the
> second character, and returns type==51.  Then `len` is set to 1, 
> `str` is malloc'ed to hold 2 bytes, and `str` is filled in with
> "\xef\x00" (the first byte of the input, as a NUL-terminated C
> string).
> 
> PyParser_AddToken then calls classify(), which falls off the end of
> its last loop and returns -1:  syntax error.
> 
> So how it gets there is really quite straightfoward.  The problem for
> me is that I have no idea why it _should_ do something other than
> that, let alone what that may be.

PEP263:

"""
     To aid with platforms such as Windows, which add Unicode BOM marks
     to the beginning of Unicode files, the UTF-8 signature
     '\xef\xbb\xbf' will be interpreted as 'utf-8' encoding as well
     (even if no magic encoding comment is given).
"""







> This needs someone who knows
> something :-)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/pedronis%40strakt.com