[Python-checkins] (no subject)

[Łukasz Langa]

To: python-checkins at python.org
Subject: bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block
 size (GH-16491) (#16506)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/938c00ca9e4207a2531041edff2e82490b02=
047f
commit: 938c00ca9e4207a2531041edff2e82490b02047f
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.co=
m>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-01T09:55:02+02:00
summary:

bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH=
-16491) (#16506)

(cherry picked from commit 94e165096fd65e8237e60de570fb609604ab94c9)

Co-authored-by: Giampaolo Rodola <g.rodola at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
M Lib/shutil.py
M Lib/socket.py

diff --git a/Lib/shutil.py b/Lib/shutil.py
index 5c1255a6713d..1e89256cc344 100644
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -135,9 +135,13 @@ def _fastcopy_sendfile(fsrc, fdst):
     # should not make any difference, also in case the file content
     # changes while being copied.
     try:
-        blocksize =3D max(os.fstat(infd).st_size, 2 ** 23)  # min 8MB
-    except Exception:
-        blocksize =3D 2 ** 27  # 128MB
+        blocksize =3D max(os.fstat(infd).st_size, 2 ** 23)  # min 8MiB
+    except OSError:
+        blocksize =3D 2 ** 27  # 128MiB
+    # On 32-bit architectures truncate to 1GiB to avoid OverflowError,
+    # see bpo-38319.
+    if sys.maxsize < 2 ** 32:
+        blocksize =3D min(blocksize, 2 ** 30)
=20
     offset =3D 0
     while True:
diff --git a/Lib/socket.py b/Lib/socket.py
index af2ed0e76a49..813f4ef5c3e1 100644
--- a/Lib/socket.py
+++ b/Lib/socket.py
@@ -355,8 +355,8 @@ def _sendfile_use_sendfile(self, file, offset=3D0, count=
=3DNone):
                 raise _GiveupOnSendfile(err)  # not a regular file
             if not fsize:
                 return 0  # empty file
-            blocksize =3D fsize if not count else count
-
+            # Truncate to 1GiB to avoid OverflowError, see bpo-38319.
+            blocksize =3D min(count or fsize, 2 ** 30)
             timeout =3D self.gettimeout()
             if timeout =3D=3D 0:
                 raise ValueError("non-blocking sockets are not supported")
diff --git a/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rs=
t b/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
new file mode 100644
index 000000000000..376a9e459de2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
@@ -0,0 +1,2 @@
+sendfile() used in socket and shutil modules was raising OverflowError for
+files >=3D 2GiB on 32-bit architectures.  (patch by Giampaolo Rodola)