Addition to I/O stack: send()
Hello, currently, Python has no builtin way to "copy a file object into another", an operation commonly referred to as "send". I think this would be a good addition to the I/O stack. To make things clearer, this is pseudo-code of what I meant: def send(self, out): while 1: data = self.read(4096) if not data: break out.write(data) I'll notice that the implementation can be tweaked in different ways: * Some operating systems do support this operation natively (through syscalls like "sendfile"), which is faster because the data does not roundtrips to user space. This is useful eg. for webservers serving files to network. * In the user-space implementation, the buffer size could match whatever buffer already exists for a buffered file reader (eg: BufferedIOBase and derivates). Because of these different details, I think it would be a good addition to the I/O stack. Though, I don't fully understand it yet as to provide a more concrete proposals as where to put each different possible implementation. Also, I'm unsure how this could couple with different kind of files (pipes, sockets, etc.). As far as I can tell, the new I/O stack is not used by the standard socket library yet, for instance. -- Giovanni Bajo Develer S.r.l. http://www.develer.com
Hi Giovanni,
I think it all sounds like premature optimization. If you have any workload where the cost of user/kernel switching or of copying the data measuredly affects the overall program speed (rather than e.g. the Python interpretation overhead), it would be interesting to hear about it. (I'm not saying that copying stuff is free; actually, the 3.1 I/O stack somewhat tries to minimize copying of data between the various layers, by using e.g. memoryviews and readinto(). But I don't think it is worth going to extremes just to avoid all memory copies. The cost of a single non-optimized method call is likely to be higher than memcpy'ing, say, 1024 bytes...) Right now, it is true there is no way to do this kind of things. There are two distinct things you are asking for really: 1/ direct access to the buffered data of a BufferedIOBase object without copying things around 2/ automagical optimization of file descriptor-to-file descriptor copying through something like sendfile(), when supported by the OS Also, please notice that 1 and 2 are exclusive :-) In 1 you are doing buffered I/O, while in 2 you are doing raw I/O. If you really need über-fast copy between two file descriptors, the solution is probably to whip up a trivial C extension that simply wraps around the sendfile() system call. You could propose it for inclusion in the stdlib, in the os module, by the way. Regards Antoine.
Hello again,
We could add two methods to BufferedIOBase: - peek_inside(self[, n]): returns a view of up to n bytes on the internal buffer - read_inside(self[, n]): same as peek_inside(), but also advances the file position after the end of the returned view The good thing is that default unoptimized versions of these methods could be provided in the BufferedIOBase ABC, simply by calling peek() (resp. read()) and wrapping the result in a memoryview. Regards Antoine.
Hi Giovanni,
I think it all sounds like premature optimization. If you have any workload where the cost of user/kernel switching or of copying the data measuredly affects the overall program speed (rather than e.g. the Python interpretation overhead), it would be interesting to hear about it. (I'm not saying that copying stuff is free; actually, the 3.1 I/O stack somewhat tries to minimize copying of data between the various layers, by using e.g. memoryviews and readinto(). But I don't think it is worth going to extremes just to avoid all memory copies. The cost of a single non-optimized method call is likely to be higher than memcpy'ing, say, 1024 bytes...) Right now, it is true there is no way to do this kind of things. There are two distinct things you are asking for really: 1/ direct access to the buffered data of a BufferedIOBase object without copying things around 2/ automagical optimization of file descriptor-to-file descriptor copying through something like sendfile(), when supported by the OS Also, please notice that 1 and 2 are exclusive :-) In 1 you are doing buffered I/O, while in 2 you are doing raw I/O. If you really need über-fast copy between two file descriptors, the solution is probably to whip up a trivial C extension that simply wraps around the sendfile() system call. You could propose it for inclusion in the stdlib, in the os module, by the way. Regards Antoine.
Hello again,
We could add two methods to BufferedIOBase: - peek_inside(self[, n]): returns a view of up to n bytes on the internal buffer - read_inside(self[, n]): same as peek_inside(), but also advances the file position after the end of the returned view The good thing is that default unoptimized versions of these methods could be provided in the BufferedIOBase ABC, simply by calling peek() (resp. read()) and wrapping the result in a memoryview. Regards Antoine.
participants (2)
-
Antoine Pitrou -
Giovanni Bajo