How to read from a file to an arbitrary delimiter efficiently?
oscar.j.benjamin at gmail.com
Sun Feb 28 15:28:29 EST 2016
On 25 February 2016 at 06:50, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> I have a need to read to an arbitrary delimiter, which might be any of a
> (small) set of characters. For the sake of the exercise, lets say it is
> either ! or ? (for example).
> I want to read from files reasonably efficiently. I don't mind if there is a
> little overhead, but my first attempt is 100 times slower than the built-in
> "read to the end of the line" method.
You can get something much faster using mmap and searching for a
def readuntil(m, delim):
start = m.tell()
index = m.find(delim, start)
if index == -1:
return m.read(index - start)
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
chunk = readuntil(m, b'!') # Note byte-string
if not chunk:
# Do stuff with chunk
My timing makes that ~7x slower than iterating over the lines of the
file but still around 100x faster than reading individual characters.
I'm not sure how to generalise it to looking for multiple delimiters
without dropping back to reading individual characters though.
More information about the Python-list