Detecting line endings

[Arthur]

Arthur wrote:
> Alex Martelli wrote:
> 
> I just got flummoxed by this issue, working with a (pre-alpha) package 
> by very experienced Python programmers who sent file.readline to 
> tokenizer.py without universal newline support.  Went on a long (and 
> educational) journey trying to figure out why my file was not being 
> processed as expected.

For example, the widely used MoinMoin source code colorizer sends files 
to tokenizer without universal newline support:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52298

Is my premise that tokenizer needs universal newline support to be 
reliable correct?

What else could put it out of sync with the complier?

Art