[Python-ideas] shutil zero-copy and exotic filesystems

[Giampaolo Rodola']

Hello,
I've been working on a patch which speeds up shutil.copy* operations for
all 3 major platforms (Linux, Windows, OSX):
https://bugs.python.org/issue33671
Since the speedup is quite consistent I'd love to see this merged in, but
considering shutil.copy* is quite crucial I wanted to hear other folk's
opinion first. Attached patch attempts to use platform-specific zero-copy
syscalls [1] by default and fallbacks on using plain read() / write()
variant in case of immediate failure. In theory this should work fine, in
practice I haven't tested it on exotic (e.g. network) filesystems. In order
to diminish risks of breakage I think if it would make sense to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy
(copyfile(). copy(), copy2(), copytree(), move())

Thoughts?

[1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)




since the matter is a bit sensitive in terms of potential breakage on
exotic / untested (e.g. network) filesystems I want to raise some attention
about:
https://bugs.python.org/issue33671
Attached patch attempts to use platform-specific zero-copy syscalls by
default and fallbacks on using plain read() / write() copy in case of
immediate failure.  In order to diminish risks I think it would make sense
to:
- add a global shutil.NO_ZEROCOPY variable defaulting to False
- add a "no_zerocopy" argument to all functions involving a copy
(copyfile(). copy(), copy2(), copytree(), move())

Thoughts?

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180529/3c197b27/attachment.html>