As for the universal new lines— it seems that either converting when the file is read (default behavior) or a simple replace of “\r\n” first is a simple solution.
I’m still confused about the use case though. It seems it involves large amounts of text, where you need to access individual lines, but not where a list-of-lines makes sense. I can see that. But I’m having trouble imagining a use case where you need that, and performance is critical for building the data structure.
But if all those are requirements, I’d think a custom data structure would be ideal — one that was both a large single string, and a sequence (and iterable) of lines.
Which reminds me of a ragged array, which I have implemented as an extension to numpy. (Both in pure python and Cython).
Perhaps a C-implemented (or accelerated) class that does all this would be a nice third party package. And if proven useful, stdlib in the future.
I imagine you wouldn’t want the dependency, but it would be interesting to benchmark a numpy solution. Numpy isn’t very memory efficient for strings (UCS-4), but it should be fast.
Final note: it seems the regex solution for a single char is performant, but overly complicated. I’ve been very happy that I can do most anything with string methods, and rarely need to reach for regex. For something this simple, it would be nice to have a string method.
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython