<div dir="ltr">Whops, I hit "send" too soon. Sorry about the messed up message.<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' <span dir="ltr"><<a href="mailto:g.rodola@gmail.com" target="_blank">g.rodola@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello,</div><div>I've been working on a patch which speeds up shutil.copy* operations for all 3 major platforms (Linux, Windows, OSX):</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="https://bugs.python.org/issue33671" target="_blank">https://bugs.python.org/<wbr>issue33671</a></span></div><div>Since the speedup is quite consistent I'd love to see this merged in, but considering shutil.copy* is quite crucial I wanted to hear other folk's opinion first. Attached patch attempts to use platform-specific zero-copy syscalls [1] by default and fallbacks on using plain read() / write() variant in case of immediate failure. In theory this should work fine, in practice I haven't tested it on exotic (e.g. network) filesystems. In order to diminish risks of breakage I think if it would make sense to:</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">- add a global <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">shutil.NO_ZEROCOPY variable defaulting to False</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move())</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Thoughts?<br></div><div><br></div>[1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows)<br class="m_-6589696259362747416gmail-Apple-interchange-newline"><br><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><br></div><div><br></div><div>since the matter is a bit sensitive in terms of potential breakage on exotic / untested (e.g. network) filesystems I want to raise some attention about:</div><a href="https://bugs.python.org/issue33671" target="_blank">https://bugs.python.org/<wbr>issue33671</a><div><div>Attached patch attempts to use platform-specific zero-copy syscalls by default and fallbacks on using plain read() / write() copy in case of immediate failure.  In order to diminish risks I think it would make sense to:</div><div>- add a global <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">shutil.NO_ZEROCOPY variable defaulting to False</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move())</span></div><div><br></div><div>Thoughts?<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div class="m_-6589696259362747416gmail_signature"><div dir="ltr"><div>Giampaolo - <a href="http://grodola.blogspot.com" target="_blank">http://grodola.blogspot.com</a></div><div><br></div><div><br></div></div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Giampaolo - <a href="http://grodola.blogspot.com" target="_blank">http://grodola.blogspot.com</a></div><div><br></div></div></div>
</div></div>