Hello,
I've been working on a patch which speeds up shutil.copy* operations for all 3 major platforms (Linux, Windows, OSX):
Since the speedup is quite consistent I'd love to see this merged in, but considering shutil.copy* is quite crucial I wanted to hear other folk's opinion first. Attached patch attempts to use platform-specific zero-copy syscalls [1] by default and fallbacks on using plain read() / write() variant in case of immediate failure. In theory this should work fine, in practice I haven't tested it on exotic (e.g. network) filesystems. In order to diminish risks of breakage I think if it would make sense to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move())
Thoughts?
[1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)
since the matter is a bit sensitive in terms of potential breakage on exotic / untested (e.g. network) filesystems I want to raise some attention about:
https://bugs.python.org/issue33671Attached patch attempts to use platform-specific zero-copy syscalls by default and fallbacks on using plain read() / write() copy in case of immediate failure. In order to diminish risks I think it would make sense to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move())
Thoughts?
--