[Python-Dev] End of the line

[Tim Peters]

[Tim, notes that Perl line-at-a-time text mode input runs 3x faster than
 Python's on his platform]

And much to my surprise, it turns out Perl reads lines a character at a time
too!  And they do not reimplement stdio.  But they do cheat.

Perl's internals are written on top of an abstract IO API, with "PerlIO *"
instead of "FILE *", "PerlIO_tell(PerlIO *)" instead of "ftell(FILE*)", and
so on.  Nothing surprising in the details, except maybe that stdin is
modeled as a function "PerlIO *PerlIO_stdin(void)" instead of as global data
(& ditto for stdout/stderr).

The usual *implementation* of these guys is as straight macro substitution
to the corresponding C stdio call.  It's possible to implement them some
other way, but I don't see anything in the source that suggests anyone has
done so, except possibly to build it all on AT&T's SFIO lib.

So where's the cheating?  In these API functions:

    int     PerlIO_has_base(PerlIO *);
    int     PerlIO_has_cntptr(PerlIO *);
    int     PerlIO_canset_cnt(PerlIO *);

    char   *PerlIO_get_ptr(PerlIO *);
    int     PerlIO_get_cnt(PerlIO *);
    void    PerlIO_set_cnt(PerlIO *,int);
    void    PerlIO_set_ptrcnt(PerlIO *,char *,int);
    char   *PerlIO_get_base(PerlIO *);
    int     PerlIO_get_bufsiz(PerlIO *);

In almost all platform stdio implementations, the C FILE struct has members
that may vary in name but serve the same purpose:  an internal buffer, and
some way (pointer or offset) to get at "the next" buffer character.  The
guys above are usually just (after layers & layers of config stuff sets it
up) macros that expand into the platform's internal way of spelling these
things.  For example, the count member is spelled under Windows as fp->_cnt
under VC, or as fp->level under Borland.

The payoff is in Perl's sv_gets function, in file sv.c.  This is long and
very complicated, but at its core has a fast inner loop that copies
characters (provided the PerlIO_has/canXXX functions say it's possible)
directly from the stdio buffer into a Perl string variable -- in the way a
platform fgets function *would* do it if it bothered to optimize fgets.  In
my experience, platforms usually settle for the same kind of
fgetc/EOF?/newline? loop Python uses, as if fgets were a stdio client rather
than a stdio primitive.  Perl's keeps everything in registers inside the
loop, updates the FILE struct members only at the boundaries, and doesn't
check for EOF except at the boundaries (so long as the buffer has unread
stuff in it, you can't be at EOF).

If the stdio buffer is exhausted before the input terminator is seen (Perl
has "input record separator" and "paragraph mode" gimmicks, so it's hairier
than just looking for \n), it calls PerlIO_getc once to force the platform
to refill the buffer, and goes back to the screaming loop.

Major hackery, but major payoff (on most platforms) too.  The abstract I/O
layer is a fine idea regardless.  The sad thing is that the real reason Perl
is so fast here is that platform fgets is so needlessly slow.

perl-input-is-faster-than-c-input-ly y'rs  - tim