Thanks all for all the responses! That's quite a bit to think about.
A couple of thoughts:
1. First, I do support a transition to UTF-8, so I understand we don't want to add more methods that deal with character offsets. (I'm familiar with how strings work in Rust.) However, does that mean we won't be using/exposing any offset at all, or will it become possible to slice using byte offsets?
2.
The commercial application I mentioned where this is critical is
actually using bytes instead of str. Sorry for not mentioning earlier.
We were doing the following:
list(accumulate(chain([0], map(len, text.splitlines(True)))))where
text is a bytes object. This is significantly faster than a binary regex for finding all universal line endings. This application is an asyncio web
app that streams Cisco show-tech files (often several gigabytes) from a
file server over HTTP; stores them chunk by chunk into a local cache
file on disk; and builds a index of byte offsets in the meantime by running the
above expression over every chunk. That way the client web app can
quickly load the lines from disk as the user scrolls through the file. A
very niche application indeed, so use of Cython would be acceptable in this
particular case. I published the relevant snippet here to be studied:
https://gist.github.com/jonathanslenders/59ddf8fe2a0954c7f1865fba3b151868 It does handle an interesting edge case regarding UTF-16.
3. The code in prompt_toolkit can be found here:
https://github.com/prompt-toolkit/python-prompt-toolkit/blob/master/src/prompt_toolkit/document.py#L209
(It's not yet using 'accumulate' there, but for the rest it's the same.) Also here, universal line
endings support is important, because the editing buffer can in theory
contain a mix of line endings. It has to be performant, because it executes on every key stroke. In this case, a more complex data structure could probably solve performance issues here, but it's really not worth the complexity that it introduces in every text manipulation (like every key binding). Also try using the "re" library to search over a list of lines or anything that's not a simple string.
4. I tested on 3.11.0b3. Using the splitlines() approach is still 2.5 times faster than re. Imagine if splitlines() doesn't have to do the work to actually create the substrings, but only has to return the offsets, that should be even much faster and not require so much memory. (I have an benchmark that does it one chunk at a time, to prevent using too much memory:
https://gist.github.com/jonathanslenders/bfca8e4f318ca64e718b4085a737accf )
So talking about bytes. Would
it be acceptable to have a `bytes.line_offsets()` method instead? Or `bytes.splitlines(return_offsets=True)`? Because byte
offsets are okay, or not? `str.splitlines(return_offsets=True)` would be very nice, but I understand the concerns.
It's somewhat frustrating here knowing that for `splitlines()`, the information is there, already computed, just not immediately accessible. (without having Python do lots of unnecessary work.)
Jonathan