[Python-ideas] speeding up shutil.copy*()

[Antoine Pitrou]

On Sun, 3 Mar 2013 19:40:04 +0100
Charles-François Natali
<cf.natali at gmail.com> wrote:
> 
> > sendfile() is a Linux-only syscall. It's also limited to certain kinds
> > of file descriptors. The limitations have been lifted in recent kernel
> > versions.
> 
> No, it's not Linux-only, many BSD also have it, although all don't
> support an arbitrary output file descriptor (Solaris does allow
> regular files too). It would be possible to catch EINVAL/EBADF, and
> fall back to a regular copy loop.
> 
> Note that the above benchmark is really biased by writing the data to
> /dev/null: with a real target file, the zero-copy wouldn't bring such
> a large gain, because the bottleneck will really be the I/O devices
> (also a read()/write() loop is more expensive in Python than in C).

Can you post your benchmark's code? I could time it on a SSD.

> But I see at least two cases where it could be interesting: when
> reading/writing from/to a tmpfs partition, or when the source and
> target files are on different disks.

That's already nice.

Regards

Antoine.