[New-bugs-announce] [issue31651] io.FileIO cannot write more than 2GB (-4096) bytes??? must be documented (if not fixed)

[Yaroslav Halchenko]

New submission from Yaroslav Halchenko <yarikoptic at gmail.com>:

originally detected on python 2.7, but replicated with python 3.5.3 -- apparently io.FileIO, if given a bytestring of 2GB or more, cannot write it all at once -- saves (and returns that size) only 2GB - 4096.

I found no indication for such behavior anywhere in the documentation. And it is surprising to me especially since regular file.write does it just fine!  attached is the code snippet which I list below and which demonstrates it

$> python3 --version; python3 longwrite.py
Python 3.5.3
Written 2147479552 out of 2147483648
4096 bytes were not written
Traceback (most recent call last):
  File "longwrite.py", line 28, in <module>
    assert in_digest == out_digest, "Digests do not match"
AssertionError: Digests do not match
python3 longwrite.py  7.03s user 5.80s system 99% cpu 12.848 total
1 11365 ->1.....................................:Sat 30 Sep 2017 04:56:26 PM EDT:.
smaug:/mnt/btrfs/scrap/tmp
$> cat longwrite.py
# -*- coding: utf-8 -*-
import io
import os
import hashlib

s = u' '*(256**4//2)  #+ u"перфекто"
s=s.encode('utf-8')
#s=' '*(10)

in_digest = hashlib.md5(s).hexdigest()
fname = 'outlong.dat'

if os.path.exists(fname):
    os.unlink(fname)

with io.FileIO(fname, 'wb') as f:
#with open(fname, 'wb') as f:
     n = f.write(s)

#n = os.stat(fname).st_size
print("Written %d out of %d" % (n, len(s)))
if n != len(s):
    print("%d bytes were not written" % (len(s) - n))

# checksum
with open(fname, 'rb') as f:
    out_digest = hashlib.md5(f.read()).hexdigest()
assert in_digest == out_digest, "Digests do not match"
print("all ok")

----------
components: IO
files: longwrite.py
messages: 303429
nosy: Yaroslav.Halchenko
priority: normal
severity: normal
status: open
title: io.FileIO cannot write more than 2GB (-4096) bytes??? must be documented (if not fixed)
type: behavior
versions: Python 2.7, Python 3.5
Added file: https://bugs.python.org/file47182/longwrite.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31651>
_______________________________________