[Python-Dev] xreadlines : readlines :: xrange : range

[Tim Peters]

[Tim]
> Perl appears to ignore the issue of thread safety here (on Windows and
> everywhere else).

[Paul Prescod]
> If you can create a sample program that demonstrates the unsafety
> I'll anonymously submit it as a bug on our internal system

I don't want to spend time on that, as I *assume* it's already well-known
within the Perl thread community.  Besides, the last version of Perl I got
from ActiveState <wink> complains:

     No threads in this perl at temp.pl line 14

if I try to use Perl threads.  That's:

> \perl\bin\perl -v

This is perl, v5.6.0 built for MSWin32-x86-multi-thread
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2000, Larry Wall

Binary build 620 provided by ActiveState Tool Corp.
http://www.ActiveState.com
Built 18:31:05 Oct 31 2000

...

If I can repair that by downloading a more recent release, let me know.

> and ensure that the next version of Perl is as slow as Python. :)

I don't want to slow them down!  To the contrary, now I've got a solid
reason for why I keep using Perl for simple high-volume text-crunching jobs
<wink>.

> Seriously: If someone comes at me with Perl-IO-is-way-faster-than-
> Python-IO, I'd like to know what concretely they've given up in order
> to achieve that performance.

My line-at-a-time test case used (rounding to nearest whole integers) 30
seconds in Python and 6 in Perl.  The result of testing many changes to
Python's implementation was that the excess 24 seconds broke down like so:

    17   spent inside internal MS threadsafe getc() lock/unlock
             routines
     5   uncertain, but evidence suggests much of it due to MS
             malloc/realloc (Perl does its own memory mgmt)
     2   for not copying directly out of the platform FILE*
             implementation struct in a highly optimized loop (like
             Perl does)

My last checkin to fileobject.c reclaimed 17 seconds on Win98SE while
remaining threadsafe, via a combination of locking per line instead of per
character, and invoking realloc much less often (only for lines exceeding
200 chars).  (BTW, I'm still curious to know how that compares to the
getc_unlocked hack on a platform other than Windows!)

> And even just for my own interest I'd like to understand the cost/
> benefit of stream thread safety.

If you're not *using* threads, or not using them to muck with the same
stream at the same time, the ratio is infinite.  And that's usually the
case.

> For instance would it make sense to just write a thread-safe
> wrapper for streams used from multiple threads?

Alas, on Windows you can't pick and choose:  you get the threadsafe libc, or
you don't.  So long as anyone may want to use threads for any reason
whatsoever, we must link with threadsafe libraries.  But, as above, on
Windows we're not paying much for that anymore in this case (unless maybe
the threadsafe MS malloc family is also outrageously slower than its
careless counterpart ...).  It does prevent me from persuing the "optimized
inner loop" business, because MS doesn't expose its locking primitives (so I
can't do in C everything I would need to do to optimize the inner loop while
remaining threadsafe).

there-are-damn-few-pieces-of-libc-we-wouldn't-be-better-off-
    writing-ourselves-but-then-we'd-have-a-much-harder-time-
    playing-with-others'-code-ly y'rs  - tim