[Python-Dev] xreadlines : readlines :: xrange : range

[Tim Peters]

[Tim]
>> Well, it would be easy to fiddle the HAVE_GETC_UNLOCKED method
>> to keep the file locked until the line was complete, and I
>> wouldn't be opposed to making life saner on platforms that allow it.

[Guido]
> Hm...  That would be possible, except for one unfortunate detail:
> _PyString_Resize() may call PyErr_BadInternalCall() which touches
> thread state.

FLOCKFILE/FUNLOCKFILE are independent of Python's notion of thread state.
IOW, do FLOCKFILE once before the for(;;), and FUNLOCKFILE once on every
*exit* path thereafter.  We can block/unblock Python threads as often as
desired between those *file*-locking brackets.  The only thing the repeated
FLOCKFILE/FUNLOCKFILE calls do to my eyes now is to create the *possibility*
for multiple readers to get partial lines of the file.

> ...
> NO, NO NO!  Mixing reads and writes on the same stream wasn't what we
> are locking against at all.  (As you've found out, it doesn't even
> work.)

On Windows, yes, but that still seems to me to be a bug in MS's code.  If
anyone had reported a core dump on any other platform, I'd be more tractable
<wink> on this point.

> We're only trying to protect against concurrent *reads*.

As above, I believe that we could do a better job of that, then, on
platforms that HAVE_GETC_UNLOCKED, by protecting not only against core dumps
but also against .readline() not delivering an intact line from the file.

>> But since FLOCKFILE is in effect, other threads *trying* to write
>> to the stream we're reading will get blocked anyway.  Seems to give us
>> potential for deadlocks.

> Only if tyeh are holding other locks at the same time.

I'm not being clear, then.  Thread X does f.readline(), on a
HAVE_GETC_UNLOCKED platform.  get_line allows other threads to run and
invokes FLOCKFILE on f->f_fp.  get_line's GETC in thread X eventually hits
the end of the stdio buffer, and does its platform's version of _filbuf.
_filbuf may wait (depending on the nature of the stream) for more input to
show up.  Simultaneously, thread Y attempts to write some data to f.  But
the *FLOCKFILE* lock prevents it from doing anything with f.  So X is
waiting for Y to write data inside platform _filbuf, but Y is waiting for X
to release the platform stream lock inside some platform stream-output
routine (if I'm being clear now, Python locks have nothing to do with this
scenario:  it's the platform stream lock).

I think this is purely the user's fault if it happens.  Just pointing it out
as another insecurity we're probably not able to protect users from.

> ...
> Yeah.  But this is insane use -- see my comments on SF.  It's only
> worth fixing because it could be used to intentionally crash Python --
> but there are easier ways...

If it's unique to MS (as I suspect), I see no reason to even consider trying
to fix it in Python.  Unless the Perl Mongers use it to crash Zope <wink>.