[Python-Dev] IO module improvements
solipsis at pitrou.net
Sat Feb 6 22:15:15 CET 2010
Pascal Chambon <chambon.pascal <at> gmail.com> writes:
> The problem is, doing that forwarding is quite complicated.
Hmm, why is it complicated? I agree it can be tedious (especially in C), but it
doesn't seem complicated in itself.
> My own FileIO class alas needs locking, because for example, on windows
> truncating a file means seeking + setting end of file + restoring
That's assuming you need FileIO to be thread-safe at all. If you always wrap it
in a Buffered object, the Buffered object will ensure thread-safety using its
(I suppose use cases for unbuffered file IO *in Python* must be pretty rare, so
most of the time you shouln't use an unwrapped FileIO anyway)
> And I TextIOWrapper seems to deserve locks. Maybe excerpts like this
> one really are thread-safe, but a long study would be required to
> ensure it.
Actually, TextIOWrapper is simply not thread-safe for most of its operations. I
think we did the work for simple writing, though, since it's better for
multi-threaded use of print().
> There are chances that my approach is slower, but the gains are so high
> in terms of maintainability and use of use, that I would definitely
> advocate it.
I agree that optimizations must always be balanced with maintainability and
simplicity. In this case, though, the IO system is really a critical part and
I'm not sure users would like us to pessimize the implementation.
> Typically, the micro-optimizations you speak about can please heavy
> programs, but they make code a mined land (maybe that's why they
> haven't been put into _pyio :p).
Well, there's no point in micro-optimizing _pyio since it's dramatically slower
than the C version :) It's there more as a reference implementation.
> Maybe I should take the latest _pyio version, and make a fork offering
> high level flexibility and security, for those who don't care about so
> high performances ?
You can, but be aware that _pyio is *really* slow... I'm not sure it would be a
service to many users.
More information about the Python-Dev