Mailman 3 March 2019 - Python-checkins

bpo-36434: Properly handle writing errors in ZIP files. (GH-12559) (GH-12628)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/4724ba9b57c45ce4bca3c828f2ed8dcff3… commit: 4724ba9b57c45ce4bca3c828f2ed8dcff3800a0c branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2019-03-30T15:52:16+02:00 summary: bpo-36434: Properly handle writing errors in ZIP files. (GH-12559) (GH-12628) Errors during writing no longer prevent to properly close the ZIP file. (cherry picked from … [View More]commit 2524fdefc9bb2a97b99319190aeb23703079ad4c) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: A Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst M Lib/test/test_zipfile.py M Lib/zipfile.py diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index b48366a53263..ac8f64ce22f6 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -402,6 +402,43 @@ def test_per_file_compresslevel(self): self.assertEqual(one_info._compresslevel, 1) self.assertEqual(nine_info._compresslevel, 9) + def test_writing_errors(self): + class BrokenFile(io.BytesIO): + def write(self, data): + nonlocal count + if count is not None: + if count == stop: + raise OSError + count += 1 + super().write(data) + + stop = 0 + while True: + testfile = BrokenFile() + count = None + with zipfile.ZipFile(testfile, 'w', self.compression) as zipfp: + with zipfp.open('file1', 'w') as f: + f.write(b'data1') + count = 0 + try: + with zipfp.open('file2', 'w') as f: + f.write(b'data2') + except OSError: + stop += 1 + else: + break + finally: + count = None + with zipfile.ZipFile(io.BytesIO(testfile.getvalue())) as zipfp: + self.assertEqual(zipfp.namelist(), ['file1']) + self.assertEqual(zipfp.read('file1'), b'data1') + + with zipfile.ZipFile(io.BytesIO(testfile.getvalue())) as zipfp: + self.assertEqual(zipfp.namelist(), ['file1', 'file2']) + self.assertEqual(zipfp.read('file1'), b'data1') + self.assertEqual(zipfp.read('file2'), b'data2') + + def tearDown(self): unlink(TESTFN) unlink(TESTFN2) diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 75dc59d1fc29..e83600e13d5e 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1101,47 +1101,50 @@ def write(self, data): def close(self): if self.closed: return - super().close() - # Flush any data from the compressor, and update header info - if self._compressor: - buf = self._compressor.flush() - self._compress_size += len(buf) - self._fileobj.write(buf) - self._zinfo.compress_size = self._compress_size - else: - self._zinfo.compress_size = self._file_size - self._zinfo.CRC = self._crc - self._zinfo.file_size = self._file_size - - # Write updated header info - if self._zinfo.flag_bits & 0x08: - # Write CRC and file sizes after the file data - fmt = '<LLQQ' if self._zip64 else '<LLLL' - self._fileobj.write(struct.pack(fmt, _DD_SIGNATURE, self._zinfo.CRC, - self._zinfo.compress_size, self._zinfo.file_size)) - self._zipfile.start_dir = self._fileobj.tell() - else: - if not self._zip64: - if self._file_size > ZIP64_LIMIT: - raise RuntimeError('File size unexpectedly exceeded ZIP64 ' - 'limit') - if self._compress_size > ZIP64_LIMIT: - raise RuntimeError('Compressed size unexpectedly exceeded ' - 'ZIP64 limit') - # Seek backwards and write file header (which will now include - # correct CRC and file sizes) - - # Preserve current position in file - self._zipfile.start_dir = self._fileobj.tell() - self._fileobj.seek(self._zinfo.header_offset) - self._fileobj.write(self._zinfo.FileHeader(self._zip64)) - self._fileobj.seek(self._zipfile.start_dir) - - self._zipfile._writing = False - - # Successfully written: Add file to our caches - self._zipfile.filelist.append(self._zinfo) - self._zipfile.NameToInfo[self._zinfo.filename] = self._zinfo + try: + super().close() + # Flush any data from the compressor, and update header info + if self._compressor: + buf = self._compressor.flush() + self._compress_size += len(buf) + self._fileobj.write(buf) + self._zinfo.compress_size = self._compress_size + else: + self._zinfo.compress_size = self._file_size + self._zinfo.CRC = self._crc + self._zinfo.file_size = self._file_size + + # Write updated header info + if self._zinfo.flag_bits & 0x08: + # Write CRC and file sizes after the file data + fmt = '<LLQQ' if self._zip64 else '<LLLL' + self._fileobj.write(struct.pack(fmt, _DD_SIGNATURE, self._zinfo.CRC, + self._zinfo.compress_size, self._zinfo.file_size)) + self._zipfile.start_dir = self._fileobj.tell() + else: + if not self._zip64: + if self._file_size > ZIP64_LIMIT: + raise RuntimeError( + 'File size unexpectedly exceeded ZIP64 limit') + if self._compress_size > ZIP64_LIMIT: + raise RuntimeError( + 'Compressed size unexpectedly exceeded ZIP64 limit') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) + + # Preserve current position in file + self._zipfile.start_dir = self._fileobj.tell() + self._fileobj.seek(self._zinfo.header_offset) + self._fileobj.write(self._zinfo.FileHeader(self._zip64)) + self._fileobj.seek(self._zipfile.start_dir) + + # Successfully written: Add file to our caches + self._zipfile.filelist.append(self._zinfo) + self._zipfile.NameToInfo[self._zinfo.filename] = self._zinfo + finally: + self._zipfile._writing = False + + class ZipFile: """ Class with methods to open, read, write, close, list zip files. diff --git a/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst b/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst new file mode 100644 index 000000000000..6e3e050c89bd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst @@ -0,0 +1 @@ +Errors during writing to a ZIP file no longer prevent to properly close it. [View Less]

1 0

C API docs: Py_IsInitialized is always safe to call (GH-12630)
by Miss Islington (bot) March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/128e40f06f3ddc18e113339af464ffefd0… commit: 128e40f06f3ddc18e113339af464ffefd070eaef branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T04:29:43-07:00 summary: C API docs: Py_IsInitialized is always safe to call (GH-12630) (cherry picked from commit ddbb978e1065dde21d1662386b26ded359f4b16e) Co-authored-by: Nick Coghlan <ncoghlan(a)gmail.… [View More]

1 0

C API docs: Py_IsInitialized is always safe to call (GH-12630)
by Nick Coghlan March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/ddbb978e1065dde21d1662386b26ded359… commit: ddbb978e1065dde21d1662386b26ded359f4b16e branch: master author: Nick Coghlan <ncoghlan(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T21:24:05+10:00 summary: C API docs: Py_IsInitialized is always safe to call (GH-12630) files: M Doc/c-api/init.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 2c6d21fa9da1..b87e999fe02b 100644 --- a/Doc/c-api/init.rst +++ b/… [View More]

1 0

bpo-22831: Use "with" to avoid possible fd leaks in tools (part 2). (GH-10927)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/172bb39452ae8b3ccdf5d1f23ead46f442… commit: 172bb39452ae8b3ccdf5d1f23ead46f44200cd49 branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T08:33:02+02:00 summary: bpo-22831: Use "with" to avoid possible fd leaks in tools (part 2). (GH-10927) files: M Tools/demo/markov.py M Tools/demo/rpython.py M Tools/demo/rpythond.py M Tools/freeze/checkextensions_win32.py M Tools/freeze/… [View More]freeze.py M Tools/i18n/pygettext.py M Tools/scripts/cleanfuture.py M Tools/scripts/combinerefs.py M Tools/scripts/dutree.py M Tools/scripts/eptags.py M Tools/scripts/finddiv.py M Tools/scripts/fixnotice.py M Tools/scripts/fixps.py M Tools/scripts/get-remote-certificate.py M Tools/scripts/h2py.py M Tools/scripts/ifdef.py M Tools/scripts/md5sum.py M Tools/scripts/mkreal.py M Tools/scripts/nm2def.py M Tools/scripts/objgraph.py M Tools/scripts/parseentities.py M Tools/scripts/pathfix.py M Tools/scripts/pdeps.py M Tools/scripts/ptags.py M Tools/scripts/rgrep.py M Tools/unicode/gencjkcodecs.py M Tools/unicode/gencodec.py diff --git a/Tools/demo/markov.py b/Tools/demo/markov.py index 7a0720fa7c34..9729f3820fa0 100755 --- a/Tools/demo/markov.py +++ b/Tools/demo/markov.py @@ -78,9 +78,9 @@ def test(): continue else: f = open(filename, 'r') - if debug: print('processing', filename, '...') - text = f.read() - f.close() + with f: + if debug: print('processing', filename, '...') + text = f.read() paralist = text.split('\n\n') for para in paralist: if debug > 1: print('feeding ...') diff --git a/Tools/demo/rpython.py b/Tools/demo/rpython.py index 5e7bc0a27d11..8d7e2747636c 100755 --- a/Tools/demo/rpython.py +++ b/Tools/demo/rpython.py @@ -22,17 +22,16 @@ def main(): port = int(port[i+1:]) host = host[:i] command = ' '.join(sys.argv[2:]) - s = socket(AF_INET, SOCK_STREAM) - s.connect((host, port)) - s.send(command.encode()) - s.shutdown(SHUT_WR) - reply = b'' - while True: - data = s.recv(BUFSIZE) - if not data: - break - reply += data - print(reply.decode(), end=' ') - s.close() + with socket(AF_INET, SOCK_STREAM) as s: + s.connect((host, port)) + s.send(command.encode()) + s.shutdown(SHUT_WR) + reply = b'' + while True: + data = s.recv(BUFSIZE) + if not data: + break + reply += data + print(reply.decode(), end=' ') main() diff --git a/Tools/demo/rpythond.py b/Tools/demo/rpythond.py index 9ffe13ab4fee..a885b3e946a1 100755 --- a/Tools/demo/rpythond.py +++ b/Tools/demo/rpythond.py @@ -26,16 +26,16 @@ def main(): s.listen(1) while True: conn, (remotehost, remoteport) = s.accept() - print('connection from', remotehost, remoteport) - request = b'' - while 1: - data = conn.recv(BUFSIZE) - if not data: - break - request += data - reply = execute(request.decode()) - conn.send(reply.encode()) - conn.close() + with conn: + print('connection from', remotehost, remoteport) + request = b'' + while 1: + data = conn.recv(BUFSIZE) + if not data: + break + request += data + reply = execute(request.decode()) + conn.send(reply.encode()) def execute(request): stdout = sys.stdout diff --git a/Tools/freeze/checkextensions_win32.py b/Tools/freeze/checkextensions_win32.py index c9cb576ae520..64350df2b84f 100644 --- a/Tools/freeze/checkextensions_win32.py +++ b/Tools/freeze/checkextensions_win32.py @@ -130,7 +130,8 @@ def parse_dsp(dsp): ret = [] dsp_path, dsp_name = os.path.split(dsp) try: - lines = open(dsp, "r").readlines() + with open(dsp, "r") as fp: + lines = fp.readlines() except IOError as msg: sys.stderr.write("%s: %s\n" % (dsp, msg)) return None diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py index 3ab56fd0fe1d..83aa508a46a9 100755 --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -142,7 +142,8 @@ def main(): # last option can not be "-i", so this ensures "pos+1" is in range! if sys.argv[pos] == '-i': try: - options = open(sys.argv[pos+1]).read().split() + with open(sys.argv[pos+1]) as infp: + options = infp.read().split() except IOError as why: usage("File name '%s' specified with the -i option " "can not be read - %s" % (sys.argv[pos+1], why) ) diff --git a/Tools/i18n/pygettext.py b/Tools/i18n/pygettext.py index b46dd339736f..b1d281d793bd 100755 --- a/Tools/i18n/pygettext.py +++ b/Tools/i18n/pygettext.py @@ -561,9 +561,8 @@ class Options: # initialize list of strings to exclude if options.excludefilename: try: - fp = open(options.excludefilename) - options.toexclude = fp.readlines() - fp.close() + with open(options.excludefilename) as fp: + options.toexclude = fp.readlines() except IOError: print(_( "Can't read --exclude-file: %s") % options.excludefilename, file=sys.stderr) diff --git a/Tools/scripts/cleanfuture.py b/Tools/scripts/cleanfuture.py index b48ab60dd65b..94f691263215 100755 --- a/Tools/scripts/cleanfuture.py +++ b/Tools/scripts/cleanfuture.py @@ -96,11 +96,11 @@ def check(file): errprint("%r: I/O Error: %s" % (file, str(msg))) return - ff = FutureFinder(f, file) - changed = ff.run() - if changed: - ff.gettherest() - f.close() + with f: + ff = FutureFinder(f, file) + changed = ff.run() + if changed: + ff.gettherest() if changed: if verbose: print("changed.") @@ -122,9 +122,8 @@ def check(file): os.rename(file, bak) if verbose: print("renamed", file, "to", bak) - g = open(file, "w") - ff.write(g) - g.close() + with open(file, "w") as g: + ff.write(g) if verbose: print("wrote new", file) else: diff --git a/Tools/scripts/combinerefs.py b/Tools/scripts/combinerefs.py index 7ca95267c93d..49ccca73909f 100755 --- a/Tools/scripts/combinerefs.py +++ b/Tools/scripts/combinerefs.py @@ -85,9 +85,7 @@ def read(fileiter, pat, whilematch): else: break -def combine(fname): - f = open(fname) - +def combinefile(f): fi = iter(f) for line in read(fi, re.compile(r'^Remaining objects:$'), False): @@ -121,8 +119,11 @@ def combine(fname): print('[%s->%s]' % (addr2rc[addr], rc), end=' ') print(guts, addr2guts[addr]) - f.close() print("%d objects before, %d after" % (before, after)) +def combine(fname): + with open(fname) as f: + combinefile(f) + if __name__ == '__main__': combine(sys.argv[1]) diff --git a/Tools/scripts/dutree.py b/Tools/scripts/dutree.py index 6b4361ac6138..d25cf72b707e 100755 --- a/Tools/scripts/dutree.py +++ b/Tools/scripts/dutree.py @@ -4,18 +4,18 @@ import os, sys, errno def main(): - p = os.popen('du ' + ' '.join(sys.argv[1:]), 'r') total, d = None, {} - for line in p.readlines(): - i = 0 - while line[i] in '0123456789': i = i+1 - size = eval(line[:i]) - while line[i] in ' \t': i = i+1 - filename = line[i:-1] - comps = filename.split('/') - if comps[0] == '': comps[0] = '/' - if comps[len(comps)-1] == '': del comps[len(comps)-1] - total, d = store(size, comps, total, d) + with os.popen('du ' + ' '.join(sys.argv[1:])) as p: + for line in p: + i = 0 + while line[i] in '0123456789': i = i+1 + size = eval(line[:i]) + while line[i] in ' \t': i = i+1 + filename = line[i:-1] + comps = filename.split('/') + if comps[0] == '': comps[0] = '/' + if comps[len(comps)-1] == '': del comps[len(comps)-1] + total, d = store(size, comps, total, d) try: display(total, d) except IOError as e: diff --git a/Tools/scripts/eptags.py b/Tools/scripts/eptags.py index 401ac7e29cd2..7f8059ba71ad 100755 --- a/Tools/scripts/eptags.py +++ b/Tools/scripts/eptags.py @@ -28,29 +28,30 @@ def treat_file(filename, outfp): except OSError: sys.stderr.write('Cannot open %s\n'%filename) return - charno = 0 - lineno = 0 - tags = [] - size = 0 - while 1: - line = fp.readline() - if not line: - break - lineno = lineno + 1 - m = matcher.search(line) - if m: - tag = m.group(0) + '\177%d,%d\n' % (lineno, charno) - tags.append(tag) - size = size + len(tag) - charno = charno + len(line) + with fp: + charno = 0 + lineno = 0 + tags = [] + size = 0 + while 1: + line = fp.readline() + if not line: + break + lineno = lineno + 1 + m = matcher.search(line) + if m: + tag = m.group(0) + '\177%d,%d\n' % (lineno, charno) + tags.append(tag) + size = size + len(tag) + charno = charno + len(line) outfp.write('\f\n%s,%d\n' % (filename,size)) for tag in tags: outfp.write(tag) def main(): - outfp = open('TAGS', 'w') - for filename in sys.argv[1:]: - treat_file(filename, outfp) + with open('TAGS', 'w') as outfp: + for filename in sys.argv[1:]: + treat_file(filename, outfp) if __name__=="__main__": main() diff --git a/Tools/scripts/finddiv.py b/Tools/scripts/finddiv.py index a705f562036e..d21253cf1c8a 100755 --- a/Tools/scripts/finddiv.py +++ b/Tools/scripts/finddiv.py @@ -55,17 +55,17 @@ def process(filename, listnames): except IOError as msg: sys.stderr.write("Can't open: %s\n" % msg) return 1 - g = tokenize.generate_tokens(fp.readline) - lastrow = None - for type, token, (row, col), end, line in g: - if token in ("/", "/="): - if listnames: - print(filename) - break - if row != lastrow: - lastrow = row - print("%s:%d:%s" % (filename, row, line), end=' ') - fp.close() + with fp: + g = tokenize.generate_tokens(fp.readline) + lastrow = None + for type, token, (row, col), end, line in g: + if token in ("/", "/="): + if listnames: + print(filename) + break + if row != lastrow: + lastrow = row + print("%s:%d:%s" % (filename, row, line), end=' ') def processdir(dir, listnames): try: diff --git a/Tools/scripts/fixnotice.py b/Tools/scripts/fixnotice.py index ad967f94781c..317051dd82f3 100755 --- a/Tools/scripts/fixnotice.py +++ b/Tools/scripts/fixnotice.py @@ -73,22 +73,19 @@ def main(): elif opt == '--dry-run': DRYRUN = 1 elif opt == '--oldnotice': - fp = open(arg) - OLD_NOTICE = fp.read() - fp.close() + with open(arg) as fp: + OLD_NOTICE = fp.read() elif opt == '--newnotice': - fp = open(arg) - NEW_NOTICE = fp.read() - fp.close() + with open(arg) as fp: + NEW_NOTICE = fp.read() for arg in args: process(arg) def process(file): - f = open(file) - data = f.read() - f.close() + with open(file) as f: + data = f.read() i = data.find(OLD_NOTICE) if i < 0: if VERBOSE: @@ -102,9 +99,8 @@ def process(file): data = data[:i] + NEW_NOTICE + data[i+len(OLD_NOTICE):] new = file + ".new" backup = file + ".bak" - f = open(new, "w") - f.write(data) - f.close() + with open(new, "w") as f: + f.write(data) os.rename(file, backup) os.rename(new, file) diff --git a/Tools/scripts/fixps.py b/Tools/scripts/fixps.py index b00226120684..725300e56a27 100755 --- a/Tools/scripts/fixps.py +++ b/Tools/scripts/fixps.py @@ -14,20 +14,18 @@ def main(): except IOError as msg: print(filename, ': can\'t open :', msg) continue - line = f.readline() - if not re.match('^#! */usr/local/bin/python', line): - print(filename, ': not a /usr/local/bin/python script') - f.close() - continue - rest = f.read() - f.close() + with f: + line = f.readline() + if not re.match('^#! */usr/local/bin/python', line): + print(filename, ': not a /usr/local/bin/python script') + continue + rest = f.read() line = re.sub('/usr/local/bin/python', '/usr/bin/env python', line) print(filename, ':', repr(line)) - f = open(filename, "w") - f.write(line) - f.write(rest) - f.close() + with open(filename, "w") as f: + f.write(line) + f.write(rest) if __name__ == '__main__': main() diff --git a/Tools/scripts/get-remote-certificate.py b/Tools/scripts/get-remote-certificate.py index 5811f202eda9..38901286e19a 100755 --- a/Tools/scripts/get-remote-certificate.py +++ b/Tools/scripts/get-remote-certificate.py @@ -29,9 +29,8 @@ def strip_to_x509_cert(certfile_contents, outfile=None): return None else: tn = tempfile.mktemp() - fp = open(tn, "wb") - fp.write(m.group(1) + b"\n") - fp.close() + with open(tn, "wb") as fp: + fp.write(m.group(1) + b"\n") try: tn2 = (outfile or tempfile.mktemp()) status, output = subproc(r'openssl x509 -in "%s" -out "%s"' % @@ -39,9 +38,8 @@ def strip_to_x509_cert(certfile_contents, outfile=None): if status != 0: raise RuntimeError('OpenSSL x509 failed with status %s and ' 'output: %r' % (status, output)) - fp = open(tn2, 'rb') - data = fp.read() - fp.close() + with open(tn2, 'rb') as fp: + data = fp.read() os.unlink(tn2) return data finally: @@ -49,9 +47,8 @@ def strip_to_x509_cert(certfile_contents, outfile=None): if sys.platform.startswith("win"): tfile = tempfile.mktemp() - fp = open(tfile, "w") - fp.write("quit\n") - fp.close() + with open(tfile, "w") as fp: + fp.write("quit\n") try: status, output = subproc( 'openssl s_client -connect "%s:%s" -showcerts < "%s"' % diff --git a/Tools/scripts/h2py.py b/Tools/scripts/h2py.py index 4363c0cf7354..ea37c04d4c56 100755 --- a/Tools/scripts/h2py.py +++ b/Tools/scripts/h2py.py @@ -69,23 +69,21 @@ def main(): sys.stdout.write('# Generated by h2py from stdin\n') process(sys.stdin, sys.stdout) else: - fp = open(filename, 'r') - outfile = os.path.basename(filename) - i = outfile.rfind('.') - if i > 0: outfile = outfile[:i] - modname = outfile.upper() - outfile = modname + '.py' - outfp = open(outfile, 'w') - outfp.write('# Generated by h2py from %s\n' % filename) - filedict = {} - for dir in searchdirs: - if filename[:len(dir)] == dir: - filedict[filename[len(dir)+1:]] = None # no '/' trailing - importable[filename[len(dir)+1:]] = modname - break - process(fp, outfp) - outfp.close() - fp.close() + with open(filename) as fp: + outfile = os.path.basename(filename) + i = outfile.rfind('.') + if i > 0: outfile = outfile[:i] + modname = outfile.upper() + outfile = modname + '.py' + with open(outfile, 'w') as outfp: + outfp.write('# Generated by h2py from %s\n' % filename) + filedict = {} + for dir in searchdirs: + if filename[:len(dir)] == dir: + filedict[filename[len(dir)+1:]] = None # no '/' trailing + importable[filename[len(dir)+1:]] = modname + break + process(fp, outfp) def pytify(body): # replace ignored patterns by spaces @@ -161,9 +159,10 @@ def process(fp, outfp, env = {}): except IOError: pass if inclfp: - outfp.write( - '\n# Included from %s\n' % filename) - process(inclfp, outfp, env) + with inclfp: + outfp.write( + '\n# Included from %s\n' % filename) + process(inclfp, outfp, env) else: sys.stderr.write('Warning - could not find file %s\n' % filename) diff --git a/Tools/scripts/ifdef.py b/Tools/scripts/ifdef.py index b1711ce5c16e..22249b2d0af5 100755 --- a/Tools/scripts/ifdef.py +++ b/Tools/scripts/ifdef.py @@ -45,9 +45,8 @@ def main(): if filename == '-': process(sys.stdin, sys.stdout) else: - f = open(filename, 'r') - process(f, sys.stdout) - f.close() + with open(filename) as f: + process(f, sys.stdout) def process(fpi, fpo): keywords = ('if', 'ifdef', 'ifndef', 'else', 'endif') diff --git a/Tools/scripts/md5sum.py b/Tools/scripts/md5sum.py index 9cf4bdc9c691..f910576377aa 100755 --- a/Tools/scripts/md5sum.py +++ b/Tools/scripts/md5sum.py @@ -47,10 +47,10 @@ def printsum(filename, out=sys.stdout): except IOError as msg: sys.stderr.write('%s: Can\'t open: %s\n' % (filename, msg)) return 1 - if fnfilter: - filename = fnfilter(filename) - sts = printsumfp(fp, filename, out) - fp.close() + with fp: + if fnfilter: + filename = fnfilter(filename) + sts = printsumfp(fp, filename, out) return sts def printsumfp(fp, filename, out=sys.stdout): diff --git a/Tools/scripts/mkreal.py b/Tools/scripts/mkreal.py index b21909e691d6..f169da43fe11 100755 --- a/Tools/scripts/mkreal.py +++ b/Tools/scripts/mkreal.py @@ -18,14 +18,13 @@ def mkrealfile(name): st = os.stat(name) # Get the mode mode = S_IMODE(st[ST_MODE]) linkto = os.readlink(name) # Make sure again it's a symlink - f_in = open(name, 'r') # This ensures it's a file - os.unlink(name) - f_out = open(name, 'w') - while 1: - buf = f_in.read(BUFSIZE) - if not buf: break - f_out.write(buf) - del f_out # Flush data to disk before changing mode + with open(name, 'rb') as f_in: # This ensures it's a file + os.unlink(name) + with open(name, 'wb') as f_out: + while 1: + buf = f_in.read(BUFSIZE) + if not buf: break + f_out.write(buf) os.chmod(name, mode) def mkrealdir(name): diff --git a/Tools/scripts/nm2def.py b/Tools/scripts/nm2def.py index 83bbcd749f4d..a885ebd6fecc 100755 --- a/Tools/scripts/nm2def.py +++ b/Tools/scripts/nm2def.py @@ -42,7 +42,8 @@ def symbols(lib=PYTHONLIB,types=('T','C','D')): - lines = os.popen(NM % lib).readlines() + with os.popen(NM % lib) as pipe: + lines = pipe.readlines() lines = [s.strip() for s in lines] symbols = {} for line in lines: @@ -97,7 +98,7 @@ def main(): exports = export_list(s) f = sys.stdout # open('PC/python_nt.def','w') f.write(DEF_TEMPLATE % (exports)) - f.close() + # f.close() if __name__ == '__main__': main() diff --git a/Tools/scripts/objgraph.py b/Tools/scripts/objgraph.py index 3bb1712a9dcf..add41e692c03 100755 --- a/Tools/scripts/objgraph.py +++ b/Tools/scripts/objgraph.py @@ -180,7 +180,8 @@ def main(): if filename == '-': readinput(sys.stdin) else: - readinput(open(filename, 'r')) + with open(filename) as f: + readinput(f) # warndups() # diff --git a/Tools/scripts/parseentities.py b/Tools/scripts/parseentities.py index c686b0241a71..0229d3af86ba 100755 --- a/Tools/scripts/parseentities.py +++ b/Tools/scripts/parseentities.py @@ -50,13 +50,15 @@ def writefile(f,defs): if __name__ == '__main__': if len(sys.argv) > 1: - infile = open(sys.argv[1]) + with open(sys.argv[1]) as infile: + text = infile.read() else: - infile = sys.stdin + text = sys.stdin.read() + + defs = parse(text) + if len(sys.argv) > 2: - outfile = open(sys.argv[2],'w') + with open(sys.argv[2],'w') as outfile: + writefile(outfile, defs) else: - outfile = sys.stdout - text = infile.read() - defs = parse(text) - writefile(outfile,defs) + writefile(sys.stdout, defs) diff --git a/Tools/scripts/pathfix.py b/Tools/scripts/pathfix.py index c5bf984306a3..1a0cf1c9e69c 100755 --- a/Tools/scripts/pathfix.py +++ b/Tools/scripts/pathfix.py @@ -103,29 +103,27 @@ def fix(filename): except IOError as msg: err('%s: cannot open: %r\n' % (filename, msg)) return 1 - line = f.readline() - fixed = fixline(line) - if line == fixed: - rep(filename+': no change\n') - f.close() - return - head, tail = os.path.split(filename) - tempname = os.path.join(head, '@' + tail) - try: - g = open(tempname, 'wb') - except IOError as msg: - f.close() - err('%s: cannot create: %r\n' % (tempname, msg)) - return 1 - rep(filename + ': updating\n') - g.write(fixed) - BUFSIZE = 8*1024 - while 1: - buf = f.read(BUFSIZE) - if not buf: break - g.write(buf) - g.close() - f.close() + with f: + line = f.readline() + fixed = fixline(line) + if line == fixed: + rep(filename+': no change\n') + return + head, tail = os.path.split(filename) + tempname = os.path.join(head, '@' + tail) + try: + g = open(tempname, 'wb') + except IOError as msg: + err('%s: cannot create: %r\n' % (tempname, msg)) + return 1 + with g: + rep(filename + ': updating\n') + g.write(fixed) + BUFSIZE = 8*1024 + while 1: + buf = f.read(BUFSIZE) + if not buf: break + g.write(buf) # Finishing touch -- move files diff --git a/Tools/scripts/pdeps.py b/Tools/scripts/pdeps.py index f8218ac5243d..4e8e930948f1 100755 --- a/Tools/scripts/pdeps.py +++ b/Tools/scripts/pdeps.py @@ -64,29 +64,28 @@ def main(): # Collect data from one file # def process(filename, table): - fp = open(filename, 'r') - mod = os.path.basename(filename) - if mod[-3:] == '.py': - mod = mod[:-3] - table[mod] = list = [] - while 1: - line = fp.readline() - if not line: break - while line[-1:] == '\\': - nextline = fp.readline() - if not nextline: break - line = line[:-1] + nextline - m_found = m_import.match(line) or m_from.match(line) - if m_found: - (a, b), (a1, b1) = m_found.regs[:2] - else: continue - words = line[a1:b1].split(',') - # print '#', line, words - for word in words: - word = word.strip() - if word not in list: - list.append(word) - fp.close() + with open(filename) as fp: + mod = os.path.basename(filename) + if mod[-3:] == '.py': + mod = mod[:-3] + table[mod] = list = [] + while 1: + line = fp.readline() + if not line: break + while line[-1:] == '\\': + nextline = fp.readline() + if not nextline: break + line = line[:-1] + nextline + m_found = m_import.match(line) or m_from.match(line) + if m_found: + (a, b), (a1, b1) = m_found.regs[:2] + else: continue + words = line[a1:b1].split(',') + # print '#', line, words + for word in words: + word = word.strip() + if word not in list: + list.append(word) # Compute closure (this is in fact totally general) diff --git a/Tools/scripts/ptags.py b/Tools/scripts/ptags.py index 396cbd07ea4d..eedd411702c1 100755 --- a/Tools/scripts/ptags.py +++ b/Tools/scripts/ptags.py @@ -19,9 +19,9 @@ def main(): for filename in args: treat_file(filename) if tags: - fp = open('tags', 'w') - tags.sort() - for s in tags: fp.write(s) + with open('tags', 'w') as fp: + tags.sort() + for s in tags: fp.write(s) expr = r'^[ \t]*(def|class)[ \t]+([a-zA-Z0-9_]+)[ \t]*[:\(]' @@ -33,21 +33,22 @@ def treat_file(filename): except: sys.stderr.write('Cannot open %s\n' % filename) return - base = os.path.basename(filename) - if base[-3:] == '.py': - base = base[:-3] - s = base + '\t' + filename + '\t' + '1\n' - tags.append(s) - while 1: - line = fp.readline() - if not line: - break - m = matcher.match(line) - if m: - content = m.group(0) - name = m.group(2) - s = name + '\t' + filename + '\t/^' + content + '/\n' - tags.append(s) + with fp: + base = os.path.basename(filename) + if base[-3:] == '.py': + base = base[:-3] + s = base + '\t' + filename + '\t' + '1\n' + tags.append(s) + while 1: + line = fp.readline() + if not line: + break + m = matcher.match(line) + if m: + content = m.group(0) + name = m.group(2) + s = name + '\t' + filename + '\t/^' + content + '/\n' + tags.append(s) if __name__ == '__main__': main() diff --git a/Tools/scripts/rgrep.py b/Tools/scripts/rgrep.py index 1917e05e4941..c39bf93aad3b 100755 --- a/Tools/scripts/rgrep.py +++ b/Tools/scripts/rgrep.py @@ -30,29 +30,30 @@ def main(): f = open(filename) except IOError as msg: usage("can't open %r: %s" % (filename, msg), 1) - f.seek(0, 2) - pos = f.tell() - leftover = None - while pos > 0: - size = min(pos, bufsize) - pos = pos - size - f.seek(pos) - buffer = f.read(size) - lines = buffer.split("\n") - del buffer - if leftover is None: - if not lines[-1]: - del lines[-1] - else: - lines[-1] = lines[-1] + leftover - if pos > 0: - leftover = lines[0] - del lines[0] - else: - leftover = None - for line in reversed(lines): - if prog.search(line): - print(line) + with f: + f.seek(0, 2) + pos = f.tell() + leftover = None + while pos > 0: + size = min(pos, bufsize) + pos = pos - size + f.seek(pos) + buffer = f.read(size) + lines = buffer.split("\n") + del buffer + if leftover is None: + if not lines[-1]: + del lines[-1] + else: + lines[-1] = lines[-1] + leftover + if pos > 0: + leftover = lines[0] + del lines[0] + else: + leftover = None + for line in reversed(lines): + if prog.search(line): + print(line) def usage(msg, code=2): diff --git a/Tools/unicode/gencjkcodecs.py b/Tools/unicode/gencjkcodecs.py index ebccfc7f96ba..45866bf2f610 100644 --- a/Tools/unicode/gencjkcodecs.py +++ b/Tools/unicode/gencjkcodecs.py @@ -61,7 +61,8 @@ def gencodecs(prefix): encoding=enc.lower(), owner=loc) codecpath = os.path.join(prefix, enc + '.py') - open(codecpath, 'w').write(code) + with open(codecpath, 'w') as f: + f.write(code) if __name__ == '__main__': import sys diff --git a/Tools/unicode/gencodec.py b/Tools/unicode/gencodec.py index 31f0112150d6..1e5aced63aa0 100644 --- a/Tools/unicode/gencodec.py +++ b/Tools/unicode/gencodec.py @@ -72,9 +72,8 @@ def parsecodes(codes, len=len, range=range): def readmap(filename): - f = open(filename,'r') - lines = f.readlines() - f.close() + with open(filename) as f: + lines = f.readlines() enc2uni = {} identity = [] unmapped = list(range(256)) @@ -359,18 +358,16 @@ def getregentry(): def pymap(name,map,pyfile,encodingname,comments=1): code = codegen(name,map,encodingname,comments) - f = open(pyfile,'w') - f.write(code) - f.close() + with open(pyfile,'w') as f: + f.write(code) def marshalmap(name,map,marshalfile): d = {} for e,(u,c) in map.items(): d[e] = (u,c) - f = open(marshalfile,'wb') - marshal.dump(d,f) - f.close() + with open(marshalfile,'wb') as f: + marshal.dump(d,f) def convertdir(dir, dirprefix='', nameprefix='', comments=1): @@ -411,8 +408,8 @@ def rewritepythondir(dir, dirprefix='', comments=1): print('converting %s to %s' % (mapname, dirprefix + codefile)) try: - map = marshal.load(open(os.path.join(dir,mapname), - 'rb')) + with open(os.path.join(dir, mapname), 'rb') as f: + map = marshal.load(f) if not map: print('* map is empty; skipping') else: [View Less]

1 0

bpo-22831: Use "with" to avoid possible fd leaks in tools (part 1). (GH-10926)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/afbb7a371fb44edc731344eab5b474ad8f… commit: afbb7a371fb44edc731344eab5b474ad8f7b57d7 branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T08:32:18+02:00 summary: bpo-22831: Use "with" to avoid possible fd leaks in tools (part 1). (GH-10926) files: M Tools/scripts/fixcid.py M Tools/scripts/fixdiv.py M Tools/scripts/fixheader.py M Tools/scripts/gprof2html.py M Tools/scripts/… [View More]texi2html.py diff --git a/Tools/scripts/fixcid.py b/Tools/scripts/fixcid.py index c66a0ac093a7..8f35eaeeb4f6 100755 --- a/Tools/scripts/fixcid.py +++ b/Tools/scripts/fixcid.py @@ -281,36 +281,36 @@ def addsubst(substfile): except IOError as msg: err(substfile + ': cannot read substfile: ' + str(msg) + '\n') sys.exit(1) - lineno = 0 - while 1: - line = fp.readline() - if not line: break - lineno = lineno + 1 - try: - i = line.index('#') - except ValueError: - i = -1 # Happens to delete trailing \n - words = line[:i].split() - if not words: continue - if len(words) == 3 and words[0] == 'struct': - words[:2] = [words[0] + ' ' + words[1]] - elif len(words) != 2: - err(substfile + '%s:%r: warning: bad line: %r' % (substfile, lineno, line)) - continue - if Reverse: - [value, key] = words - else: - [key, value] = words - if value[0] == '*': - value = value[1:] - if key[0] == '*': - key = key[1:] - NotInComment[key] = value - if key in Dict: - err('%s:%r: warning: overriding: %r %r\n' % (substfile, lineno, key, value)) - err('%s:%r: warning: previous: %r\n' % (substfile, lineno, Dict[key])) - Dict[key] = value - fp.close() + with fp: + lineno = 0 + while 1: + line = fp.readline() + if not line: break + lineno = lineno + 1 + try: + i = line.index('#') + except ValueError: + i = -1 # Happens to delete trailing \n + words = line[:i].split() + if not words: continue + if len(words) == 3 and words[0] == 'struct': + words[:2] = [words[0] + ' ' + words[1]] + elif len(words) != 2: + err(substfile + '%s:%r: warning: bad line: %r' % (substfile, lineno, line)) + continue + if Reverse: + [value, key] = words + else: + [key, value] = words + if value[0] == '*': + value = value[1:] + if key[0] == '*': + key = key[1:] + NotInComment[key] = value + if key in Dict: + err('%s:%r: warning: overriding: %r %r\n' % (substfile, lineno, key, value)) + err('%s:%r: warning: previous: %r\n' % (substfile, lineno, Dict[key])) + Dict[key] = value if __name__ == '__main__': main() diff --git a/Tools/scripts/fixdiv.py b/Tools/scripts/fixdiv.py index 1213a4e39760..df7c481aa228 100755 --- a/Tools/scripts/fixdiv.py +++ b/Tools/scripts/fixdiv.py @@ -179,27 +179,27 @@ def usage(msg): def readwarnings(warningsfile): prog = re.compile(PATTERN) + warnings = {} try: f = open(warningsfile) except IOError as msg: sys.stderr.write("can't open: %s\n" % msg) return - warnings = {} - while 1: - line = f.readline() - if not line: - break - m = prog.match(line) - if not m: - if line.find("division") >= 0: - sys.stderr.write("Warning: ignored input " + line) - continue - filename, lineno, what = m.groups() - list = warnings.get(filename) - if list is None: - warnings[filename] = list = [] - list.append((int(lineno), sys.intern(what))) - f.close() + with f: + while 1: + line = f.readline() + if not line: + break + m = prog.match(line) + if not m: + if line.find("division") >= 0: + sys.stderr.write("Warning: ignored input " + line) + continue + filename, lineno, what = m.groups() + list = warnings.get(filename) + if list is None: + warnings[filename] = list = [] + list.append((int(lineno), sys.intern(what))) return warnings def process(filename, list): @@ -210,84 +210,84 @@ def process(filename, list): except IOError as msg: sys.stderr.write("can't open: %s\n" % msg) return 1 - print("Index:", filename) - f = FileContext(fp) - list.sort() - index = 0 # list[:index] has been processed, list[index:] is still to do - g = tokenize.generate_tokens(f.readline) - while 1: - startlineno, endlineno, slashes = lineinfo = scanline(g) - if startlineno is None: - break - assert startlineno <= endlineno is not None - orphans = [] - while index < len(list) and list[index][0] < startlineno: - orphans.append(list[index]) - index += 1 - if orphans: - reportphantomwarnings(orphans, f) - warnings = [] - while index < len(list) and list[index][0] <= endlineno: - warnings.append(list[index]) - index += 1 - if not slashes and not warnings: - pass - elif slashes and not warnings: - report(slashes, "No conclusive evidence") - elif warnings and not slashes: - reportphantomwarnings(warnings, f) - else: - if len(slashes) > 1: - if not multi_ok: - rows = [] - lastrow = None - for (row, col), line in slashes: - if row == lastrow: - continue - rows.append(row) - lastrow = row - assert rows - if len(rows) == 1: - print("*** More than one / operator in line", rows[0]) + with fp: + print("Index:", filename) + f = FileContext(fp) + list.sort() + index = 0 # list[:index] has been processed, list[index:] is still to do + g = tokenize.generate_tokens(f.readline) + while 1: + startlineno, endlineno, slashes = lineinfo = scanline(g) + if startlineno is None: + break + assert startlineno <= endlineno is not None + orphans = [] + while index < len(list) and list[index][0] < startlineno: + orphans.append(list[index]) + index += 1 + if orphans: + reportphantomwarnings(orphans, f) + warnings = [] + while index < len(list) and list[index][0] <= endlineno: + warnings.append(list[index]) + index += 1 + if not slashes and not warnings: + pass + elif slashes and not warnings: + report(slashes, "No conclusive evidence") + elif warnings and not slashes: + reportphantomwarnings(warnings, f) + else: + if len(slashes) > 1: + if not multi_ok: + rows = [] + lastrow = None + for (row, col), line in slashes: + if row == lastrow: + continue + rows.append(row) + lastrow = row + assert rows + if len(rows) == 1: + print("*** More than one / operator in line", rows[0]) + else: + print("*** More than one / operator per statement", end=' ') + print("in lines %d-%d" % (rows[0], rows[-1])) + intlong = [] + floatcomplex = [] + bad = [] + for lineno, what in warnings: + if what in ("int", "long"): + intlong.append(what) + elif what in ("float", "complex"): + floatcomplex.append(what) else: - print("*** More than one / operator per statement", end=' ') - print("in lines %d-%d" % (rows[0], rows[-1])) - intlong = [] - floatcomplex = [] - bad = [] - for lineno, what in warnings: - if what in ("int", "long"): - intlong.append(what) - elif what in ("float", "complex"): - floatcomplex.append(what) - else: - bad.append(what) - lastrow = None - for (row, col), line in slashes: - if row == lastrow: - continue - lastrow = row - line = chop(line) - if line[col:col+1] != "/": - print("*** Can't find the / operator in line %d:" % row) - print("*", line) - continue - if bad: - print("*** Bad warning for line %d:" % row, bad) - print("*", line) - elif intlong and not floatcomplex: - print("%dc%d" % (row, row)) - print("<", line) - print("---") - print(">", line[:col] + "/" + line[col:]) - elif floatcomplex and not intlong: - print("True division / operator at line %d:" % row) - print("=", line) - elif intlong and floatcomplex: - print("*** Ambiguous / operator (%s, %s) at line %d:" % ( - "|".join(intlong), "|".join(floatcomplex), row)) - print("?", line) - fp.close() + bad.append(what) + lastrow = None + for (row, col), line in slashes: + if row == lastrow: + continue + lastrow = row + line = chop(line) + if line[col:col+1] != "/": + print("*** Can't find the / operator in line %d:" % row) + print("*", line) + continue + if bad: + print("*** Bad warning for line %d:" % row, bad) + print("*", line) + elif intlong and not floatcomplex: + print("%dc%d" % (row, row)) + print("<", line) + print("---") + print(">", line[:col] + "/" + line[col:]) + elif floatcomplex and not intlong: + print("True division / operator at line %d:" % row) + print("=", line) + elif intlong and floatcomplex: + print("*** Ambiguous / operator (%s, %s) at line %d:" % + ("|".join(intlong), "|".join(floatcomplex), row)) + print("?", line) def reportphantomwarnings(warnings, f): blocks = [] diff --git a/Tools/scripts/fixheader.py b/Tools/scripts/fixheader.py index ec840575b205..c834eec1e20e 100755 --- a/Tools/scripts/fixheader.py +++ b/Tools/scripts/fixheader.py @@ -15,8 +15,8 @@ def process(filename): except IOError as msg: sys.stderr.write('%s: can\'t open: %s\n' % (filename, str(msg))) return - data = f.read() - f.close() + with f: + data = f.read() if data[:2] != '/*': sys.stderr.write('%s does not begin with C comment\n' % filename) return @@ -25,25 +25,25 @@ def process(filename): except IOError as msg: sys.stderr.write('%s: can\'t write: %s\n' % (filename, str(msg))) return - sys.stderr.write('Processing %s ...\n' % filename) - magic = 'Py_' - for c in filename: - if ord(c)<=0x80 and c.isalnum(): - magic = magic + c.upper() - else: magic = magic + '_' - sys.stdout = f - print('#ifndef', magic) - print('#define', magic) - print('#ifdef __cplusplus') - print('extern "C" {') - print('#endif') - print() - f.write(data) - print() - print('#ifdef __cplusplus') - print('}') - print('#endif') - print('#endif /*', '!'+magic, '*/') + with f: + sys.stderr.write('Processing %s ...\n' % filename) + magic = 'Py_' + for c in filename: + if ord(c)<=0x80 and c.isalnum(): + magic = magic + c.upper() + else: magic = magic + '_' + print('#ifndef', magic, file=f) + print('#define', magic, file=f) + print('#ifdef __cplusplus', file=f) + print('extern "C" {', file=f) + print('#endif', file=f) + print(file=f) + f.write(data) + print(file=f) + print('#ifdef __cplusplus', file=f) + print('}', file=f) + print('#endif', file=f) + print('#endif /*', '!'+magic, '*/', file=f) if __name__ == '__main__': main() diff --git a/Tools/scripts/gprof2html.py b/Tools/scripts/gprof2html.py index 4ca705c3c61c..b14def4ef848 100755 --- a/Tools/scripts/gprof2html.py +++ b/Tools/scripts/gprof2html.py @@ -28,14 +28,7 @@ def add_escapes(filename): for line in fp: yield html.escape(line) - -def main(): - filename = "gprof.out" - if sys.argv[1:]: - filename = sys.argv[1] - outputfilename = filename + ".html" - input = add_escapes(filename) - output = open(outputfilename, "w") +def gprof2html(input, output, filename): output.write(header % filename) for line in input: output.write(line) @@ -78,7 +71,16 @@ def main(): part = '<a href="#call:%s">%s</a>' % (part, part) output.write(part) output.write(trailer) - output.close() + + +def main(): + filename = "gprof.out" + if sys.argv[1:]: + filename = sys.argv[1] + outputfilename = filename + ".html" + input = add_escapes(filename) + with open(outputfilename, "w") as output: + gprof2html(input, output, filename) webbrowser.open("file:" + os.path.abspath(outputfilename)) if __name__ == '__main__': diff --git a/Tools/scripts/texi2html.py b/Tools/scripts/texi2html.py index 5565c210dabb..c06d812ab3fb 100755 --- a/Tools/scripts/texi2html.py +++ b/Tools/scripts/texi2html.py @@ -118,11 +118,10 @@ def write(self, *lines): self.lines.append(line) def flush(self): - fp = open(self.dirname + '/' + makefile(self.name), 'w') - fp.write(self.prologue) - fp.write(self.text) - fp.write(self.epilogue) - fp.close() + with open(self.dirname + '/' + makefile(self.name), 'w') as fp: + fp.write(self.prologue) + fp.write(self.text) + fp.write(self.epilogue) def link(self, label, nodename, rel=None, rev=None): if nodename: @@ -558,14 +557,14 @@ def do_include(self, args): except IOError as msg: print('*** Can\'t open include file', repr(file)) return - print('!'*self.debugging, '--> file', repr(file)) - save_done = self.done - save_skip = self.skip - save_stack = self.stack - self.includedepth = self.includedepth + 1 - self.parserest(fp, 0) - self.includedepth = self.includedepth - 1 - fp.close() + with fp: + print('!'*self.debugging, '--> file', repr(file)) + save_done = self.done + save_skip = self.skip + save_stack = self.stack + self.includedepth = self.includedepth + 1 + self.parserest(fp, 0) + self.includedepth = self.includedepth - 1 self.done = save_done self.skip = save_skip self.stack = save_stack @@ -1770,78 +1769,75 @@ def finalize(self): # PROJECT FILE try: - fp = open(projectfile,'w') - print('[OPTIONS]', file=fp) - print('Auto Index=Yes', file=fp) - print('Binary TOC=No', file=fp) - print('Binary Index=Yes', file=fp) - print('Compatibility=1.1', file=fp) - print('Compiled file=' + resultfile + '', file=fp) - print('Contents file=' + contentfile + '', file=fp) - print('Default topic=' + defaulttopic + '', file=fp) - print('Error log file=ErrorLog.log', file=fp) - print('Index file=' + indexfile + '', file=fp) - print('Title=' + title + '', file=fp) - print('Display compile progress=Yes', file=fp) - print('Full-text search=Yes', file=fp) - print('Default window=main', file=fp) - print('', file=fp) - print('[WINDOWS]', file=fp) - print('main=,"' + contentfile + '","' + indexfile - + '","","",,,,,0x23520,222,0x1046,[10,10,780,560],' - '0xB0000,,,,,,0', file=fp) - print('', file=fp) - print('[FILES]', file=fp) - print('', file=fp) - self.dumpfiles(fp) - fp.close() + with open(projectfile, 'w') as fp: + print('[OPTIONS]', file=fp) + print('Auto Index=Yes', file=fp) + print('Binary TOC=No', file=fp) + print('Binary Index=Yes', file=fp) + print('Compatibility=1.1', file=fp) + print('Compiled file=' + resultfile + '', file=fp) + print('Contents file=' + contentfile + '', file=fp) + print('Default topic=' + defaulttopic + '', file=fp) + print('Error log file=ErrorLog.log', file=fp) + print('Index file=' + indexfile + '', file=fp) + print('Title=' + title + '', file=fp) + print('Display compile progress=Yes', file=fp) + print('Full-text search=Yes', file=fp) + print('Default window=main', file=fp) + print('', file=fp) + print('[WINDOWS]', file=fp) + print('main=,"' + contentfile + '","' + indexfile + + '","","",,,,,0x23520,222,0x1046,[10,10,780,560],' + '0xB0000,,,,,,0', file=fp) + print('', file=fp) + print('[FILES]', file=fp) + print('', file=fp) + self.dumpfiles(fp) except IOError as msg: print(projectfile, ':', msg) sys.exit(1) # CONTENT FILE try: - fp = open(contentfile,'w') - print('<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">', file=fp) - print('', file=fp) - print('<HTML>', file=fp) - print('<HEAD>', file=fp) - print('<meta name="GENERATOR" ' - 'content="Microsoft® HTML Help Workshop 4.1">', file=fp) - print('', file=fp) - print('</HEAD>', file=fp) - print('<BODY>', file=fp) - print(' <OBJECT type="text/site properties">', file=fp) - print(' <param name="Window Styles" value="0x800025">', file=fp) - print(' <param name="comment" value="title:">', file=fp) - print(' <param name="comment" value="base:">', file=fp) - print(' </OBJECT>', file=fp) - self.dumpnodes(fp) - print('</BODY>', file=fp) - print('</HTML>', file=fp) - fp.close() + with open(contentfile, 'w') as fp: + print('<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">', file=fp) + print('', file=fp) + print('<HTML>', file=fp) + print('<HEAD>', file=fp) + print('<meta name="GENERATOR" ' + 'content="Microsoft® HTML Help Workshop 4.1">', file=fp) + print('', file=fp) + print('</HEAD>', file=fp) + print('<BODY>', file=fp) + print(' <OBJECT type="text/site properties">', file=fp) + print(' <param name="Window Styles" value="0x800025">', file=fp) + print(' <param name="comment" value="title:">', file=fp) + print(' <param name="comment" value="base:">', file=fp) + print(' </OBJECT>', file=fp) + self.dumpnodes(fp) + print('</BODY>', file=fp) + print('</HTML>', file=fp) except IOError as msg: print(contentfile, ':', msg) sys.exit(1) # INDEX FILE try: - fp = open(indexfile ,'w') - print('<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">', file=fp) - print('', file=fp) - print('<HTML>', file=fp) - print('<HEAD>', file=fp) - print('<meta name="GENERATOR" ' - 'content="Microsoft® HTML Help Workshop 4.1">', file=fp) - print('', file=fp) - print('</HEAD>', file=fp) - print('<BODY>', file=fp) - print('<OBJECT type="text/site properties">', file=fp) - print('</OBJECT>', file=fp) - self.dumpindex(fp) - print('</BODY>', file=fp) - print('</HTML>', file=fp) - fp.close() + with open(indexfile, 'w') as fp: + print('<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">', file=fp) + print('', file=fp) + print('<HTML>', file=fp) + print('<HEAD>', file=fp) + print('<meta name="GENERATOR" ' + 'content="Microsoft® HTML Help Workshop 4.1">', file=fp) + print('', file=fp) + print('</HEAD>', file=fp) + print('<BODY>', file=fp) + print('<OBJECT type="text/site properties">', file=fp) + print('</OBJECT>', file=fp) + self.dumpindex(fp) + print('</BODY>', file=fp) + print('</HTML>', file=fp) except IOError as msg: print(indexfile , ':', msg) sys.exit(1) @@ -2064,8 +2060,8 @@ def test(): print(file, ':', msg) sys.exit(1) - parser.parse(fp) - fp.close() + with fp: + parser.parse(fp) parser.report() htmlhelp.finalize() [View Less]

1 0

bpo-36434: Properly handle writing errors in ZIP files. (GH-12559)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/2524fdefc9bb2a97b99319190aeb237030… commit: 2524fdefc9bb2a97b99319190aeb23703079ad4c branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T08:25:19+02:00 summary: bpo-36434: Properly handle writing errors in ZIP files. (GH-12559) Errors during writing no longer prevent to properly close the ZIP file. files: A Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.… [View More]rst M Lib/test/test_zipfile.py M Lib/zipfile.py diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 7b8922f4e054..14e1e08c5bfd 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -402,6 +402,43 @@ def test_per_file_compresslevel(self): self.assertEqual(one_info._compresslevel, 1) self.assertEqual(nine_info._compresslevel, 9) + def test_writing_errors(self): + class BrokenFile(io.BytesIO): + def write(self, data): + nonlocal count + if count is not None: + if count == stop: + raise OSError + count += 1 + super().write(data) + + stop = 0 + while True: + testfile = BrokenFile() + count = None + with zipfile.ZipFile(testfile, 'w', self.compression) as zipfp: + with zipfp.open('file1', 'w') as f: + f.write(b'data1') + count = 0 + try: + with zipfp.open('file2', 'w') as f: + f.write(b'data2') + except OSError: + stop += 1 + else: + break + finally: + count = None + with zipfile.ZipFile(io.BytesIO(testfile.getvalue())) as zipfp: + self.assertEqual(zipfp.namelist(), ['file1']) + self.assertEqual(zipfp.read('file1'), b'data1') + + with zipfile.ZipFile(io.BytesIO(testfile.getvalue())) as zipfp: + self.assertEqual(zipfp.namelist(), ['file1', 'file2']) + self.assertEqual(zipfp.read('file1'), b'data1') + self.assertEqual(zipfp.read('file2'), b'data2') + + def tearDown(self): unlink(TESTFN) unlink(TESTFN2) diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 61cd929f614f..2dc016472117 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1105,47 +1105,50 @@ def write(self, data): def close(self): if self.closed: return - super().close() - # Flush any data from the compressor, and update header info - if self._compressor: - buf = self._compressor.flush() - self._compress_size += len(buf) - self._fileobj.write(buf) - self._zinfo.compress_size = self._compress_size - else: - self._zinfo.compress_size = self._file_size - self._zinfo.CRC = self._crc - self._zinfo.file_size = self._file_size - - # Write updated header info - if self._zinfo.flag_bits & 0x08: - # Write CRC and file sizes after the file data - fmt = '<LLQQ' if self._zip64 else '<LLLL' - self._fileobj.write(struct.pack(fmt, _DD_SIGNATURE, self._zinfo.CRC, - self._zinfo.compress_size, self._zinfo.file_size)) - self._zipfile.start_dir = self._fileobj.tell() - else: - if not self._zip64: - if self._file_size > ZIP64_LIMIT: - raise RuntimeError('File size unexpectedly exceeded ZIP64 ' - 'limit') - if self._compress_size > ZIP64_LIMIT: - raise RuntimeError('Compressed size unexpectedly exceeded ' - 'ZIP64 limit') - # Seek backwards and write file header (which will now include - # correct CRC and file sizes) - - # Preserve current position in file - self._zipfile.start_dir = self._fileobj.tell() - self._fileobj.seek(self._zinfo.header_offset) - self._fileobj.write(self._zinfo.FileHeader(self._zip64)) - self._fileobj.seek(self._zipfile.start_dir) - - self._zipfile._writing = False - - # Successfully written: Add file to our caches - self._zipfile.filelist.append(self._zinfo) - self._zipfile.NameToInfo[self._zinfo.filename] = self._zinfo + try: + super().close() + # Flush any data from the compressor, and update header info + if self._compressor: + buf = self._compressor.flush() + self._compress_size += len(buf) + self._fileobj.write(buf) + self._zinfo.compress_size = self._compress_size + else: + self._zinfo.compress_size = self._file_size + self._zinfo.CRC = self._crc + self._zinfo.file_size = self._file_size + + # Write updated header info + if self._zinfo.flag_bits & 0x08: + # Write CRC and file sizes after the file data + fmt = '<LLQQ' if self._zip64 else '<LLLL' + self._fileobj.write(struct.pack(fmt, _DD_SIGNATURE, self._zinfo.CRC, + self._zinfo.compress_size, self._zinfo.file_size)) + self._zipfile.start_dir = self._fileobj.tell() + else: + if not self._zip64: + if self._file_size > ZIP64_LIMIT: + raise RuntimeError( + 'File size unexpectedly exceeded ZIP64 limit') + if self._compress_size > ZIP64_LIMIT: + raise RuntimeError( + 'Compressed size unexpectedly exceeded ZIP64 limit') + # Seek backwards and write file header (which will now include + # correct CRC and file sizes) + + # Preserve current position in file + self._zipfile.start_dir = self._fileobj.tell() + self._fileobj.seek(self._zinfo.header_offset) + self._fileobj.write(self._zinfo.FileHeader(self._zip64)) + self._fileobj.seek(self._zipfile.start_dir) + + # Successfully written: Add file to our caches + self._zipfile.filelist.append(self._zinfo) + self._zipfile.NameToInfo[self._zinfo.filename] = self._zinfo + finally: + self._zipfile._writing = False + + class ZipFile: """ Class with methods to open, read, write, close, list zip files. diff --git a/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst b/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst new file mode 100644 index 000000000000..6e3e050c89bd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-03-26-14-20-59.bpo-36434.PTdidw.rst @@ -0,0 +1 @@ +Errors during writing to a ZIP file no longer prevent to properly close it. [View Less]

1 0

bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603)
by Serhiy Storchaka March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/7a465cb5ee7e298cae626ace1fc3e7d97d… commit: 7a465cb5ee7e298cae626ace1fc3e7d97df79f2e branch: master author: Serhiy Storchaka <storchaka(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T08:23:38+02:00 summary: bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603) The bug occurred when the encoded surrogate character is passed to the incremental decoder in two chunks. files: A Misc/NEWS.d/next/Core and Builtins/… [View More]

1 0

bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602)
by Miss Islington (bot) March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/5b80cb5584a72044424f2d82d0ae79c720… commit: 5b80cb5584a72044424f2d82d0ae79c720f24c47 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-03-29T22:38:14-07:00 summary: bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602) (cherry picked from commit 38f4e468d4b55551e135c67337c18ae142193ba8) Co-authored-by: Inada Naoki <songofacandy(a)… [View More]

1 0

bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602)
by Inada Naoki March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/38f4e468d4b55551e135c67337c18ae142… commit: 38f4e468d4b55551e135c67337c18ae142193ba8 branch: master author: Inada Naoki <songofacandy(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T14:32:08+09:00 summary: bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602) files: M Doc/library/sys.rst diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 0fa5bd462294..52026f6a2bce 100644 --- a/Doc/library/sys.… [View More]

1 0

github: assign @methane to dict owner (GH-12617)
by Inada Naoki March 30, 2019

March 30, 2019

https://github.com/python/cpython/commit/7444daada1270c94501d5516ab1e553b5e… commit: 7444daada1270c94501d5516ab1e553b5e2b9586 branch: master author: Inada Naoki <songofacandy(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-03-30T11:08:17+09:00 summary: github: assign @methane to dict owner (GH-12617) files: M .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 03e36cfcedd4..40d2cc127690 100644 --- a/.github/CODEOWNERS +++ b/.github/… [View More]

1 0