[Python-Dev] IO module improvements

[Antoine Pitrou]

Pascal Chambon <chambon.pascal <at> gmail.com> writes:
> 
> The problem is, doing that forwarding is quite complicated.

Hmm, why is it complicated? I agree it can be tedious (especially in C), but it
doesn't seem complicated in itself.

> My own FileIO class alas needs locking, because for example, on windows
> truncating a file means seeking + setting end of file + restoring
> pointer. 

That's assuming you need FileIO to be thread-safe at all. If you always wrap it
in a Buffered object, the Buffered object will ensure thread-safety using its
own lock.
(I suppose use cases for unbuffered file IO *in Python* must be pretty rare, so
most of the time you shouln't use an unwrapped FileIO anyway)

> And I TextIOWrapper seems to deserve locks. Maybe excerpts like this
> one really are thread-safe, but a long study would be required to
> ensure it.
> [snip]

Actually, TextIOWrapper is simply not thread-safe for most of its operations. I
think we did the work for simple writing, though, since it's better for
multi-threaded use of print().

> There are chances that my approach is slower, but the gains are so high
> in terms of maintainability and use of use, that I would definitely
> advocate it.

I agree that optimizations must always be balanced with maintainability and
simplicity. In this case, though, the IO system is really a critical part and
I'm not sure users would like us to pessimize the implementation.

> Typically, the micro-optimizations you speak about can please heavy
> programs, but they make code a mined land (maybe that's why they
> haven't been put into _pyio :p). 

Well, there's no point in micro-optimizing _pyio since it's dramatically slower
than the C version :) It's there more as a reference implementation.

> Maybe I should take the latest _pyio version, and make a fork offering
> high level flexibility and security, for those who don't care about so
> high performances ?

You can, but be aware that _pyio is *really* slow... I'm not sure it would be a
service to many users.

cheers

Antoine.