[Patches] [ python-Patches-580331 ] xreadlines caching, file iterator
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 06 Aug 2002 08:56:40 -0700
Patches item #580331, was opened at 2002-07-11 17:45
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=580331&group_id=5470
Category: Core (C code)
Group: Python 2.3
>Status: Closed
Resolution: Accepted
Priority: 5
Submitted By: Oren Tirosh (orenti)
Assigned to: Guido van Rossum (gvanrossum)
Summary: xreadlines caching, file iterator
Initial Comment:
Calling f.xreadlines() multiple times returns the same
xreadlines object.
A file is an iterator - __iter__() returns self and next() calls
the cached xreadlines object's next method.
----------------------------------------------------------------------
>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-06 11:56
Message:
Logged In: YES
user_id=6380
Thanks! Checked in.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-06 11:30
Message:
Logged In: YES
user_id=6380
Hm, test_file fails on a technicality. I'll take it from
here. Thanks!
----------------------------------------------------------------------
Comment By: Oren Tirosh (orenti)
Date: 2002-08-06 01:03
Message:
Logged In: YES
user_id=562624
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-05 16:07
Message:
Logged In: YES
user_id=6380
Thanks!
Making xreadlines an alias for __iter__ sounds about right,
for backwards compatibility.
Then we should probably deprecate xreadlines, despite the
fact that it could be useful for other file-like objects;
it's just not a pretty enough interface.
----------------------------------------------------------------------
Comment By: Oren Tirosh (orenti)
Date: 2002-08-05 16:01
Message:
Logged In: YES
user_id=562624
Updated patch.
What to do about the xreadlines method? The patch doesn't
touch it but It could be made an alias to __iter__ and the
dependency of file objects on the xreadlines module will be
eliminated.
On my linux machine the highest performance is achieved for
buffer sizes somewhere around 4096-8192. Higher or lower
values are significantly slower.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2002-08-05 12:14
Message:
Logged In: YES
user_id=31435
Just FYI, in apps that do "read + process" in a loop, a small
buffer is often faster because the data has a decent shot at
staying in L1 cache. Make the buffer very large (100s of
Kb), and it won't even stay in L2 cache.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-05 11:29
Message:
Logged In: YES
user_id=6380
OK, I'll await a new patch.
----------------------------------------------------------------------
Comment By: Oren Tirosh (orenti)
Date: 2002-08-05 11:22
Message:
Logged In: YES
user_id=562624
> What's a normal text file? One with a million bytes? :-)
I meant 100kBYTE lines... Some apps actually use such long
lines.
Yes, it works just fine with universal newlines.
Ok, the #ifdefs will go.
Strange, a bigger buffer seems to actually slow it down... I'll
have to investigate this further.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-08-05 10:52
Message:
Logged In: YES
user_id=6380
This begins to look good.
What's a normal text file? One with a million bytes? :-)
Have you made sure this works as expected in Universal
newline mode?
I'd like a patch that doesn't use #define WITH_READAHEAD_BUFFER.
You might also experiment with larger buffer sizes (I
predict that a larger buffer doesn't make much difference,
since it didn't for xreadlines, but it would be nice to
verify that and then add a comment; at least once a year
someone asks whether the buffer shouldn't be much larger).
----------------------------------------------------------------------
Comment By: Oren Tirosh (orenti)
Date: 2002-08-05 02:27
Message:
Logged In: YES
user_id=562624
The version of the patch still makes a file an iterator but it no
longer depends on xreadlines - it implements the readahead
buffering inside the file object.
It is about 19% faster than xreadlines for normal text files and
about 40% faster for files with 100k lines.
The methods readline and read do not use this readahead
mechanism because it skews the current file position (just like
xreadlines does).
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-07-17 13:50
Message:
Logged In: YES
user_id=6380
Alas, there's a fatal flaw. The file object and the
xreadlines object now both have pointers to each other,
creating an unbreakable cycle (since neither participates in
GC). Weak refs can't be used to resolve this dilemma. I
personally think that's enough to just stick with the status
quo (I was never more than +0 on the idea of making the file
an interator anyway). But I'll leave it to Oren to come up
with another hack (please use this same SF patch).
Oren, if you'd like to give up, please say so and I'll close
the item in a jiffy. In fact, I positively encourage you to
give up. But I don't expect you to take this offer. :-)
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-07-16 21:33
Message:
Logged In: YES
user_id=6380
I'm reviewing this and will check it in, or something like
it (probably).
----------------------------------------------------------------------
Comment By: Oren Tirosh (orenti)
Date: 2002-07-16 01:26
Message:
Logged In: YES
user_id=562624
Now invalidates cache on a seek.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-07-15 10:38
Message:
Logged In: YES
user_id=6380
I posted some comments to python-dev.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=580331&group_id=5470