, argc=3, argv=0x7fffffffddf8, init=, fini=, rtld_fini=, stack_end=0x7fffffffdde8) at ../csu/libc-start.c:291 #20 0x00000000004933e9 in _start () ~The reason I thought that this may be a bug is because while researching the problem, I came across this other bug (https://bugzilla.redhat.com/show_bug.cgi?id=674206) where the user specifies a similar issue for RedHat. The difference, though, is that in that case it could be fixed by specifying the argtypes, while in my case it's already specified. I tested this on an Ubuntu 16.04 and ArchLinux machine and got the same results. ---------- components: ctypes messages: 329367 nosy: execve priority: normal severity: normal status: open title: Ctypes segfault or TypeError tested for python2.7 and 3 versions: Python 2.7, Python 3.5 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 6 17:29:09 2018 From: report at bugs.python.org (Julien Palard) Date: Tue, 06 Nov 2018 22:29:09 +0000 Subject: [New-bugs-announce] [issue35181] Doc: Namespace Packages: Inconsistent documentation of __loader__ being None Message-ID: <1541543349.68.0.788709270274.issue35181@psf.upfronthosting.co.za> New submission from Julien Palard : The documentation states that a __loader__ of a namespace package should be None: - [1] "For namespace packages this should be set to None." - [2] "To indicate to the import machinery that the spec represents a namespace portion. the path entry finder sets ?loader? on the spec to None". But this looks wrong [3], looks like it has been changed in [4]/[5]. I think one should rely on __file__ being None on namespace packages (which make sense as they span over multiple directories) instead of __loader__ being None (a side effect of the import machinery ?). [1]: https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec.loader [2]: https://docs.python.org/3/reference/import.html#path-entry-finder-protocol [3]: https://stackoverflow.com/questions/52869541/namespace-package-spec-loader-and-loader-attributes-not-set-to-none [4]: https://bugs.python.org/issue32303 [5]: https://github.com/python/cpython/pull/5481/files#diff-a6592cec2ebc8dba9bbf7d396370b138L319 ---------- assignee: docs at python components: Documentation messages: 329392 nosy: docs at python, mdk priority: normal severity: normal status: open title: Doc: Namespace Packages: Inconsistent documentation of __loader__ being None versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 7 05:23:30 2018 From: report at bugs.python.org (Andriy Maletsky) Date: Wed, 07 Nov 2018 10:23:30 +0000 Subject: [New-bugs-announce] [issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits Message-ID: <1541586210.62.0.788709270274.issue35182@psf.upfronthosting.co.za> New submission from Andriy Maletsky : When communicate() is called in a loop, it crashes when the child process has already closed any piped standard stream, but still continues to be running. How this happens: 1) the parent waits for the child events inside communicate() call 2) the child closes its side of any attached pipes long before exiting (in my case there is some complex c++ application which had messed with its termination) 3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE (for stdin) or EOF (for stdout), decides to close corresponding file descriptors from its side 4) communicate() waits for the death of the child, but a timeout is fired 5) parent handles timeout exception and calls communicate() again 6) an exception is raised when communicate() tries to register closed file in epoll I think there may be a simple solution: before registering file descriptors in epoll, we may check whether any of them is already closed, and don't register it in that case. Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64: import subprocess child = subprocess.Popen( ['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1), time.sleep(30)'], stdout=subprocess.PIPE, ) while True: try: child.communicate(timeout=3) break except subprocess.TimeoutExpired: # do something useful here pass Here is a stacktrace: Traceback (most recent call last): File "test.py", line 10, in child.communicate(timeout=3) File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate selector.register(self.stdout, selectors.EVENT_READ) File "/usr/local/lib/python3.7/selectors.py", line 352, in register key = super().register(fileobj, events, data) File "/usr/local/lib/python3.7/selectors.py", line 238, in register key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data) File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup return _fileobj_to_fd(fileobj) File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd "{!r}".format(fileobj)) from None ValueError: Invalid file object: <_io.BufferedReader name=3> ---------- messages: 329412 nosy: and800 priority: normal severity: normal status: open title: Popen.communicate() breaks when child closes its side of pipe but not exits type: crash versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 7 06:49:13 2018 From: report at bugs.python.org (Shaun Griffith) Date: Wed, 07 Nov 2018 11:49:13 +0000 Subject: [New-bugs-announce] [issue35183] os.path.splitext documentation needs typical example Message-ID: <1541591353.01.0.788709270274.issue35183@psf.upfronthosting.co.za> New submission from Shaun Griffith : As with many entries on the os.path doc page, splitext needs a typical example. Not grokking the bare minimum text, I had to actually try it in the interpreter to see what it did. The one example that *is* there is an edge case, and does nothing to explain the normal behavior, or why this is the correct behavior for the edge case. Here is where I tripped up: Split the pathname path into a pair (root, ext) such that root + ext == path, and ext is empty or begins with a period and contains at most one period. One interpretation of this is that ext is either empty, or has a period, _and nothing else_. Here are 2 examples for typical use: >>> splitext('readme.txt') ('readme', '.txt') >>> splitext('/some/long/pathname/warble.csv') ('/some/long/pathname/warble', '.csv') ---------- assignee: docs at python components: Documentation messages: 329414 nosy: docs at python, shaungriffith priority: normal severity: normal status: open title: os.path.splitext documentation needs typical example type: enhancement versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 7 11:47:20 2018 From: report at bugs.python.org (=?utf-8?q?Marta_G=C3=B3mez?=) Date: Wed, 07 Nov 2018 16:47:20 +0000 Subject: [New-bugs-announce] [issue35184] Makefile is not correctly generated when compiling pyextat with DXML_POOR_ENTROPY=1 Message-ID: <1541609240.08.0.788709270274.issue35184@psf.upfronthosting.co.za> New submission from Marta G?mez : Hello, I was getting this same error (https://bugs.python.org/issue35139) when compiling pyextat so I added these changes (https://github.com/python/cpython/pull/10291/files) to my Setup.dist. After running ./configure, the generated Makefile throws the following error: $ make Makefile:264: *** missing separator. Stop. I have tried this in both Python 3.7 and 3.6 and I get the error in both (the line numbers are different but the error is the same). I have tested this in both Ubuntu 16/18 and CentOS 7. Any hints in how could I fix this? Best regards, Marta ---------- components: Build messages: 329424 nosy: mgmacias95 priority: normal severity: normal status: open title: Makefile is not correctly generated when compiling pyextat with DXML_POOR_ENTROPY=1 type: compile error versions: Python 3.6, Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 7 12:43:57 2018 From: report at bugs.python.org (Ben Spiller) Date: Wed, 07 Nov 2018 17:43:57 +0000 Subject: [New-bugs-announce] [issue35185] Logger race condition - loses lines if removeHandler called from another thread while logging Message-ID: <1541612637.38.0.788709270274.issue35185@psf.upfronthosting.co.za> New submission from Ben Spiller : I just came across a fairly serious thread-safety / race condition bug in the logging.Loggers class, which causes random log lines to be lost i.e. not get passed to some of the registered handlers, if (other, unrelated) handlers are being added/removed using add/removeHandler from another thread during logging. This potentially affects all log handler classes, though for timing reasons I've found it easiest to reproduce with the logging.FileHandler. See attached test program that reproduces this. I did some debugging and looks like although add/removeHandler are protected by _acquireLock(), they modify the self.handlers list in-place, and the callHandlers method iterates over self.handlers with no locking - so if you're unlucky you can end up with some of your handlers not being called. A trivial way to fix the bug is by editing callHandlers and copying the list before iterating over it: - for hdlr in c.handlers: + for hdlr in list(c.handlers): However since that could affect the performance of routine log statements a better fix is probably to change the implementation of add/removeHandler to avoid in-place modification of self.handlers so that (as a result of the GIL) it'll be safe to iterate over the list in callHandlers, e.g. change removeHandler like this: - self.handlers.remove(hdlr) + newhandlers = list(self.handlers) + newhandlers.remove(hdlr) + self.handlers = hdlr (and the equivalent in addHandler) ---------- components: Library (Lib) files: logger-race.py messages: 329429 nosy: benspiller priority: normal severity: normal status: open title: Logger race condition - loses lines if removeHandler called from another thread while logging type: behavior versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7 Added file: https://bugs.python.org/file47914/logger-race.py _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 7 20:56:11 2018 From: report at bugs.python.org (Paul Ganssle) Date: Thu, 08 Nov 2018 01:56:11 +0000 Subject: [New-bugs-announce] [issue35186] distutils.command.upload uses deprecated platform.dist with bdist_rpm Message-ID: <1541642171.78.0.788709270274.issue35186@psf.upfronthosting.co.za> New submission from Paul Ganssle : It seems that in issue #1322, `platform.dist` was deprecated, and is slated for removal in Python 3.8, but it is currently used in distutils.command.upload.upload_file: https://github.com/python/cpython/blob/6843ffe4533e9f2cde036296fd932fef6f059687/Lib/distutils/command/upload.py#L126 As far as I can tell, this line is hit only when you run `python setup.py bdist_rpm upload`, as far as I can tell. Using the `upload` command is *itself* very deprecated, and is dangerously broken in many ways. See, for example, this setuptools issue: https://github.com/pypa/setuptools/issues/1381 So I see three possible options (two realistic): 1. Remove the whole "comment" field from the upload, both for bdist_rpm and bdist_dumb. I'm not sure if PyPI uses this for anything, but I highly doubt it. 2. Have both bdist_rpm and bdist_dumb send a comment based on `platform.platform(terse=1)` 3. Remove the upload command functionality entirely for Python 3.8 with no deprecation warning. Obviously #3 is the unrealistic one, but it's very tempting. That said, I think we should go with #1 and maybe open a separate issue to actually deprecate the `upload` command in distutils (though to be honest we'll probably monkey-patch it out of existence in setuptools within a year and everyone uses `setuptools` anyway, so maybe #3 is less unrealistic than it seems). ---------- components: Distutils messages: 329453 nosy: dstufft, eric.araujo, p-ganssle priority: normal severity: normal status: open title: distutils.command.upload uses deprecated platform.dist with bdist_rpm versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 02:57:46 2018 From: report at bugs.python.org (xiyunlong) Date: Thu, 08 Nov 2018 07:57:46 +0000 Subject: [New-bugs-announce] [issue35187] a bug about np.arrange Message-ID: <1541663866.8.0.788709270274.issue35187@psf.upfronthosting.co.za> New submission from xiyunlong : when I use np.arange as follows: np.arange(1,2.2,0.2) I got the array as [1. , 1.2, 1.4, 1.6, 1.8, 2. , 2.2],we know, the maxvalue 2.2 of the array should not be included in the array. Is this a bug? ---------- components: Library (Lib) messages: 329460 nosy: xyl123 priority: normal severity: normal status: open title: a bug about np.arrange versions: Python 3.5 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 07:05:53 2018 From: report at bugs.python.org (xiyunlong) Date: Thu, 08 Nov 2018 12:05:53 +0000 Subject: [New-bugs-announce] [issue35188] something confused about numpy.arange Message-ID: <1541678753.34.0.788709270274.issue35188@psf.upfronthosting.co.za> New submission from xiyunlong : when I use:print(np.arange(1,2.2,0.2)) I got: 1.0 1.2 1.4 1.5999999999999999 1.7999999999999998 1.9999999999999998 2.1999999999999997 could any one tell me why it's not 1.0,1.2,1.4,1.6,1.8,2.0? Many thanks! ---------- components: Library (Lib) messages: 329466 nosy: xyl123 priority: normal severity: normal status: open title: something confused about numpy.arange versions: Python 3.5 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 07:45:30 2018 From: report at bugs.python.org (Aapo Samuli Keskimolo) Date: Thu, 08 Nov 2018 12:45:30 +0000 Subject: [New-bugs-announce] [issue35189] EINTR is not being retried Message-ID: <1541681130.38.0.788709270274.issue35189@psf.upfronthosting.co.za> New submission from Aapo Samuli Keskimolo : According to https://www.python.org/dev/peps/pep-0475/ the EINTR interruption should be retried automatically, but somehow it does not work and the exception is raised: 2018-11-05 05:21:35,257 ERROR:storage(23491): Remote storage operation failed (request: '{ 'excludeSubModules': None, 'storageLocation': 'qt/qtdatavis3d/68faa5b00f73096eb096c6acdfce76b052ca20b9/LinuxUbuntu_18_04x86_64LinuxQEMUarm64GCCqtci-linux-Ubuntu-18.04-x86_64-a6 c9f7Release/ac4280d182ec320eaf0e68efaeeeb6be14b9689f/test_1542834179', 'type': 3}') Traceback (most recent call last): File "src/storage.py", line 507, in handle self.handle_upload_artifact(message) File "src/storage.py", line 437, in handle_upload_artifact log.info("upload of %s to %s", uploadType, message.storageLocation) File "/usr/lib/python3.6/logging/__init__.py", line 1306, in info self._log(INFO, msg, args, **kwargs) File "/usr/lib/python3.6/logging/__init__.py", line 1442, in _log self.handle(record) File "/usr/lib/python3.6/logging/__init__.py", line 1452, in handle self.callHandlers(record) File "/usr/lib/python3.6/logging/__init__.py", line 1514, in callHandlers hdlr.handle(record) File "/usr/lib/python3.6/logging/__init__.py", line 861, in handle self.acquire() File "/home/vmbuilder/qt-ci/src/application.py", line 151, in acquire fcntl.lockf(self._lock_fd, fcntl.LOCK_EX) InterruptedError: [Errno 4] Interrupted system call ---------- components: Library (Lib) messages: 329469 nosy: akeskimo priority: normal severity: normal status: open title: EINTR is not being retried type: crash versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 09:41:26 2018 From: report at bugs.python.org (=?utf-8?q?Miro_Hron=C4=8Dok?=) Date: Thu, 08 Nov 2018 14:41:26 +0000 Subject: [New-bugs-announce] [issue35190] collections.abc.Sequence cannot be used to test whether a class provides a particular interface Message-ID: <1541688086.42.0.788709270274.issue35190@psf.upfronthosting.co.za> New submission from Miro Hron?ok : The collections.abc ? Abstract Base Classes for Containers documentation says: > This module provides abstract base classes that can be used to test whether a class provides a particular interface; for example, whether it is hashable or whether it is a mapping. https://docs.python.org/3/library/collections.abc.html However this is not true for Sequence. When I implement a class that provides a particular interface (defined in the Collections Abstract Base Classes table in that very page), I cannot check whether it implements a Sequence. See an example: from collections import abc class Box: def __init__(self, wrapped): self._w = wrapped def __len__(self): return len(self._w) def __iter__(self): yield from self._w def __getitem__(self, i): return self._w[i] def __reversed__(self): yield from reversed(self._w) def __contains__(self, i): return i in self._w def index(self, value, start=0, stop=None): return self._w.index(value, start, stop) def count(self, value): return self._w.count(value) b = Box([1, 2, 3]) for t in 'Sized', 'Iterable', 'Reversible', 'Container', 'Collection', 'Sequence': print(f'{t}: {isinstance(b, getattr(abc, t))}') My class is Reversible. My class is a Collection (as it is a Sized Iterable Container). It implements __getitem__, __len__, __contains__, __iter__, __reversed__, index, and count. Yet my class instance is not an instance of Sequence. I suppose this behavior might be intentional, as discussed in issue16728 - or it might as well not be. The main concern was that dict also provides these methods, but is not considered a Sequence, however dict does not provide index() or count(). Regardless whether this is right or wrong behavior, as documented this should be a Sequence. See also https://stackoverflow.com/questions/34927949/issubclass-of-abstract-base-class-sequence As I see it, either: collections.abc.Sequence needs a __subclasshook__ so it can be used as the documentation implies. Or: the documentation should not say that "abstract base classes (from abc module) can be used to test whether a class provides a particular interface" if it doesn't generally apply Or: the Sequence documentation should say: "this particular abstract base class cannot be used to test whether a class provides a particular interface because reasons" (yet I don't really get those reasons) ---------- assignee: docs at python components: Documentation, Library (Lib) messages: 329475 nosy: docs at python, hroncok, vstinner priority: normal severity: normal status: open title: collections.abc.Sequence cannot be used to test whether a class provides a particular interface versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 10:43:43 2018 From: report at bugs.python.org (Alexey Izbyshev) Date: Thu, 08 Nov 2018 15:43:43 +0000 Subject: [New-bugs-announce] [issue35191] socket.setblocking(x) treats multiples of 2**32 as False Message-ID: <1541691823.13.0.788709270274.issue35191@psf.upfronthosting.co.za> New submission from Alexey Izbyshev : UBSAN with -fsanitize=implicit-integer-truncation reported a suspicious case: testSetBlocking_overflow (test.test_socket.NonBlockingTCPTests) ... /scratch2/izbyshev/cpython/Modules/socketmodule.c:2688:33: runtime error: implicit conversion from type 'long' of value 4294967296 (64-bit, signed) to type 'int' changed the value to 0 (32-bit, signed) It turned out that sock_setblocking() converts its (logically boolean) argument to long, but then passes it to internal_setblocking() which accepts int (https://github.com/python/cpython/blob/fd512d76456b65c529a5bc58d8cfe73e4a10de7a/Modules/socketmodule.c#L2688). This results in unexpected truncation on platforms with 64-bit long. testSetBlocking_overflow() which is supposed to check this doesn't work because it only checks socket timeout which is updated correctly. However, the actual state of socket descriptor is changed to the opposite value (non-blocking) in this case. ---------- components: Extension Modules messages: 329478 nosy: izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: socket.setblocking(x) treats multiples of 2**32 as False type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 15:08:28 2018 From: report at bugs.python.org (Adam Dunlap) Date: Thu, 08 Nov 2018 20:08:28 +0000 Subject: [New-bugs-announce] [issue35192] pathlib mkdir throws FileExistsError when not supposed to Message-ID: <1541707708.76.0.788709270274.issue35192@psf.upfronthosting.co.za> New submission from Adam Dunlap : I have 2 processes (one C, one python) that are started at the same time and both need a certain directory tree to exist to continue, so both processes create the directory tree while ignoring errors about it already existing. In the python process, I'm using pathlib's mkdir function (https://docs.python.org/3/library/pathlib.html#pathlib.Path.mkdir) with parents=True, exist_ok=True, but this is sometimes throwing a FileExistsError. I don't expect this because the documentation says "If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file." The last component is never a non-directory file. I believe what is happening is that mkdir realizes that the parent doesn't exist, so it recursively tries to make the parent directory. However, when it recurses, it uses the default exists_ok value of False. Before the recursive call can make the parent directory, the other process makes the directory. This causes the inner call to throw a FileExistsError. I believe the correct thing to do is to always pass True for exists_ok in the recursive call. ---------- components: Library (Lib) messages: 329484 nosy: Adam Dunlap priority: normal severity: normal status: open title: pathlib mkdir throws FileExistsError when not supposed to type: behavior versions: Python 3.5 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 16:08:35 2018 From: report at bugs.python.org (Gregory P. Smith) Date: Thu, 08 Nov 2018 21:08:35 +0000 Subject: [New-bugs-announce] [issue35193] Off by one error in peephole call to find_op on case RETURN_VALUE Message-ID: <1541711315.03.0.788709270274.issue35193@psf.upfronthosting.co.za> New submission from Gregory P. Smith : An off by one error was introduced to peephole.c by the "off by one error fix" in https://bugs.python.org/issue28517. Clang's memory sanitizer detects it (msan). find_op is ultimately called with h == codelen so it accesses one byte out of bounds. I have a fix, PR coming. ---------- assignee: gregory.p.smith components: Interpreter Core messages: 329485 nosy: gregory.p.smith, serhiy.storchaka priority: normal severity: normal status: open title: Off by one error in peephole call to find_op on case RETURN_VALUE type: crash versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 17:55:24 2018 From: report at bugs.python.org (Alexey Izbyshev) Date: Thu, 08 Nov 2018 22:55:24 +0000 Subject: [New-bugs-announce] [issue35194] A typo in a constant in cp932 codec Message-ID: <1541717724.61.0.788709270274.issue35194@psf.upfronthosting.co.za> New submission from Alexey Izbyshev : UBSan with -fsanitize=implicit-integer-truncation found a suspicious one: /scratch2/izbyshev/cpython/Modules/cjkcodecs/_codecs_jp.c:43:17: runtime error: implicit conversion from type 'unsigned int' of value 4294966013 (32-bit, unsigned) to type 'unsigned char' changed the value to 253 (8-bit, unsigned) Indeed, the wrong constant was used (the correct one is used in corresponding decoder code at https://github.com/python/cpython/blob/fd512d76456b65c529a5bc58d8cfe73e4a10de7a/Modules/cjkcodecs/_codecs_jp.c#L105). In this case the truncation was harmless because only the lowest byte of the wrong result was used, and it was correct. But it probably makes sense to fix it if only to reduce noise from UBSan. All Python versions are affected, but I've marked 3.8 only since I'm not sure what the policy for backporting such changes is. ---------- components: Extension Modules messages: 329489 nosy: izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: A typo in a constant in cp932 codec type: behavior versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 8 17:56:21 2018 From: report at bugs.python.org (Dragoljub) Date: Thu, 08 Nov 2018 22:56:21 +0000 Subject: [New-bugs-announce] [issue35195] Pandas read_csv() is 3.5X Slower on Python 3.7.1 vs Python 3.6.7 & 3.5.2 On Windows 10 Message-ID: <1541717781.13.0.788709270274.issue35195@psf.upfronthosting.co.za> New submission from Dragoljub : xref: https://github.com/pandas-dev/pandas/issues/23516 Example: import io import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(1000000, 10), columns=('COL{}'.format(i) for i in range(10))) csv = io.StringIO(df.to_csv(index=False)) df2 = pd.read_csv(csv) #3.5X slower on Python 3.7.1 pd.read_csv() reads data at 30MB/sec on Python 3.7.1 while at 100MB/sec on Python 3.6.7. This issue seems to be only present on Windows 10 Builds both x86 & x64. Possibly some IO changes in Python 3.7 could have contributed to this slowdown on Windows but not on Linux? ---------- components: IO messages: 329490 nosy: Dragoljub priority: normal severity: normal status: open title: Pandas read_csv() is 3.5X Slower on Python 3.7.1 vs Python 3.6.7 & 3.5.2 On Windows 10 type: performance versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 05:00:52 2018 From: report at bugs.python.org (Raymond Hettinger) Date: Fri, 09 Nov 2018 10:00:52 +0000 Subject: [New-bugs-announce] [issue35196] IDLE text squeezer is too aggressive and is slow Message-ID: <1541757652.98.0.788709270274.issue35196@psf.upfronthosting.co.za> New submission from Raymond Hettinger : The squeezed text is impairing IDLE's usability for teaching purposes. Typing help() on any built-in type such as str immediately results in a squeeze-button rather than displaying help. The same is true for showing lines from a file read or from a URL. I recommend showing the first 50 to 100 lines and then squeezing the remainder. Also, I think this may be the logic that is slowing down successive print calls in a loop. Try running: for i in range(500): print(i, sep=' ') or even: for i in range(500): print(i, i**2) The output has noticeably slow delays between successive print() calls. ---------- assignee: terry.reedy components: IDLE messages: 329507 nosy: rhettinger, terry.reedy priority: normal severity: normal status: open title: IDLE text squeezer is too aggressive and is slow type: behavior versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 09:06:44 2018 From: report at bugs.python.org (STINNER Victor) Date: Fri, 09 Nov 2018 14:06:44 +0000 Subject: [New-bugs-announce] [issue35197] graminit.h defines very generic names like 'stmt' or 'test' Message-ID: <1541772404.82.0.788709270274.issue35197@psf.upfronthosting.co.za> New submission from STINNER Victor : graminit.h is an header file associating Grammar entities to their values, example: #define stmt 269 Problem: defines symbols have no prefix and cause compilation issues on regular C code depending where graminit.h is included. Example with ast.c: static int validate_stmt(stmt_ty stmt) { ... } "#define stmt 269" causes a compilation issue, the preprocessor replaces the code with: static int validate_stmt(stmt_ty 269) { ... } ... which is invalid. Another example: return validate_expr(exp->v.IfExp.test, Load) && validate_expr(exp->v.IfExp.body, Load) && validate_expr(exp->v.IfExp.orelse, Load); The preprocessor replaces "exp->v.IfExp.test" with "exp->v.IfExp.305" which is invalid... The compile.h header file works around the issue by redefining 3 constants but using "Py_" prefix: /* These definitions must match corresponding definitions in graminit.h. There's code in compile.c that checks that they are the same. */ #define Py_single_input 256 #define Py_file_input 257 #define Py_eval_input 258 For comparison, graminit.h uses: #define single_input 256 #define file_input 257 #define eval_input 258 There are different solutions: * Do nothing: require to include graminit.h at the right place * Add a prefix to all symbols, ex: test => Py_test, grammar_test, etc. * Rename problematic names like 'test' and 'stmt' * Fix C code to avoid problematic name: 'test' is an an attribute name of a node struct... Note: "static const int single_input = 305;" cause a complation error on "case single_input": "case label does not reduce to an integer constant". IMHO adding a prefix would be a nice enhancement and a good compromise. It only impacts the four C files which include graminit.h: future.c, ast.c, parsetok.c and parsermodule.c. ---------- components: Interpreter Core messages: 329517 nosy: vstinner priority: normal severity: normal status: open title: graminit.h defines very generic names like 'stmt' or 'test' versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 10:20:45 2018 From: report at bugs.python.org (Ayappan) Date: Fri, 09 Nov 2018 15:20:45 +0000 Subject: [New-bugs-announce] [issue35198] Build issue while compiling cpp files in AIX Message-ID: <1541776845.13.0.788709270274.issue35198@psf.upfronthosting.co.za> New submission from Ayappan : I am trying to build pandas-0.23.4 using python3 in AIX. Everything goes fine till it encounters a cpp file. Below is the compile error output. building 'pandas._libs.window' extension gcc -fPIC -maix64 -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/freeware/lib64/python3.6/site-packages/numpy/core/include -I/opt/freeware/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.aix-7.2-3.6/pandas/_libs/window.o -Wno-unused-function In file included from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821, from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18, from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4, from pandas/_libs/window.cpp:587: /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp] #warning "Using deprecated NumPy API, disable it by " \ ^~~~~~~ g++ gcc -maix64 -pthread -bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib build/temp.aix-7.2-3.6/pandas/_libs/window.o -L/opt/freeware/lib64 -o build/lib.aix-7.2-3.6/pandas/_libs/window.so g++: error: gcc: No such file or directory g++: error: unrecognized command line option '-bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp' error: command 'g++' failed with exit status 1 Seems like somewhere setting up the proper compile options for compiling cpp files is not proper. Any hint will be really useful. ---------- components: Build messages: 329522 nosy: Ayappan priority: normal severity: normal status: open title: Build issue while compiling cpp files in AIX type: compile error versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 10:32:02 2018 From: report at bugs.python.org (STINNER Victor) Date: Fri, 09 Nov 2018 15:32:02 +0000 Subject: [New-bugs-announce] [issue35199] Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode Message-ID: <1541777522.86.0.788709270274.issue35199@psf.upfronthosting.co.za> New submission from STINNER Victor : Currently, even when Python is compiled in debug mode, PyTuple_GET_ITEM() doesn't check that the first argument is a tuple objet and that the second argument is valid index. It can lead to a crash and Python doesn't help debugging. I propose to convert the macro to a function call and use regular assertions to abort Python if the C API is misused. I propose to use a function call rather than abusing the preprocessor syntax like (assert(...),expr) syntax used in unicodeobject.h, because I the preprocessor causes complex bugs (difficult to understand and to fix/work around) and because later I would like to experiment to be able to compile C extensions to always use function calls, but get a different implementation depending on the "Python runtime". I elaborated this idea on this website: * https://pythoncapi.readthedocs.io/runtimes.html#debug-build * https://pythoncapi.readthedocs.io/ I am working on an implementation. ---------- components: Interpreter Core messages: 329523 nosy: vstinner priority: normal severity: normal status: open title: Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 11:25:51 2018 From: report at bugs.python.org (Julien Palard) Date: Fri, 09 Nov 2018 16:25:51 +0000 Subject: [New-bugs-announce] [issue35200] Range repr could be better Message-ID: <1541780751.51.0.788709270274.issue35200@psf.upfronthosting.co.za> New submission from Julien Palard : This morning I was teaching Python (again and again), and again I was thinking we could do better about the representation of ranges. Typically in the current repr of ranges we do not see that the end is excluded: >>> range(10) range(0, 10) However it has the (little?) benefit of respecting the "repr gives valid Python". I propose to change it to: >>> range(10) ---------- messages: 329531 nosy: mdk priority: normal severity: normal status: open title: Range repr could be better type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 11:54:51 2018 From: report at bugs.python.org (Daniel Israel) Date: Fri, 09 Nov 2018 16:54:51 +0000 Subject: [New-bugs-announce] [issue35201] Recursive '**' matches non-existent directories. Message-ID: <1541782491.74.0.788709270274.issue35201@psf.upfronthosting.co.za> New submission from Daniel Israel : In the following case, when there is no file or directory 'a', glob will still return it: >>> glob.glob("a/**", recursive=True) [ 'a/' ] Note that this is inconsistent with the '*' pattern: >>> glob.glob("a/*", recursive=True) [] ---------- components: Library (Lib) messages: 329537 nosy: daniel priority: normal severity: normal status: open title: Recursive '**' matches non-existent directories. type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 14:09:12 2018 From: report at bugs.python.org (=?utf-8?b?U3Jpbml2YXMgIFJlZGR5IFRoYXRpcGFydGh5KOCwtuCxjeCwsOCxgOCwqA==?= =?utf-8?b?4LC/4LC14LC+4LC44LGNIOCwsOCxhuCwoeCxjeCwoeCwvyDgsKTgsL7gsJ8=?= =?utf-8?b?4LC/4LCq4LCw4LGN4LCk4LC/KQ==?=) Date: Fri, 09 Nov 2018 19:09:12 +0000 Subject: [New-bugs-announce] [issue35202] Remove unused imports in standard library Message-ID: <1541790552.02.0.788709270274.issue35202@psf.upfronthosting.co.za> New submission from Srinivas Reddy Thatiparthy(?????????? ?????? ?????????) : Please find the modules where we need to remove. codecs.py:1105:5: F401 'encodings' imported but unused code.py:298:13: F401 'readline' imported but unused platform.py:118:1: F401 'warnings' imported but unused platform.py:509:9: F401 'java.lang' imported but unused pstats.py:562:9: F401 'readline' imported but unused shutil.py:13:1: F401 'io' imported but unused site.py:409:13: F401 'rlcompleter' imported but unused site.py:511:13: F401 'sitecustomize' imported but unused site.py:531:13: F401 'usercustomize' imported but unused trace.py:54:1: F401 're' imported but unused typing.py:21:1: F401 'abc' imported but unused I was just wondering that as a community, do we recommend this practice of cleaning up Or just we keep as is inorder to NOT to break accidental imports? ---------- messages: 329548 nosy: thatiparthy priority: normal severity: normal status: open title: Remove unused imports in standard library _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 14:27:37 2018 From: report at bugs.python.org (Dexter) Date: Fri, 09 Nov 2018 19:27:37 +0000 Subject: [New-bugs-announce] [issue35203] Windows Installer Ignores Launcher Installer Options Where The Python Launcher Is Already Present Message-ID: <1541791657.21.0.788709270274.issue35203@psf.upfronthosting.co.za> New submission from Dexter : When installing Python 3.7 from the Windows Installer in a restricted environment the installer appears to ignore the InstallLauncherAllUsers=0 and Include_launcher=0 flags if the launcher is already present from a previous install. I am on a machine where the system python is centrally controlled and cannot be modified by me. In previous versions I have installed personal copies using the InstallLauncherAllUsers=0 InstallAllUsers=0 as exemplified in the docs or just unticking in the UI. However in 3.7 the tick-box for the launcher is ticked but greyed out in the UI. I am unable to de-select it and, as I do not have permission to modify the system wide launcher, I cannot install Python 3.7. My suspicion is that the box is greyed out if someone has installed the site wide version to ensure the site wide version is in-line with the installer but: a) What if this is an older version and I do not want it replaced. b) What if a user like me can't upgrade that script and is happy to have a version that will not be understood by the launcher for other purposes. Assuming there is no current workaround. I propose that either we remove this restriction or at least allow an override. e.g. ForceExistingLauncherUpgrade=1 by default to preserve current behaviour and if 0 then allow the other launcher exclusions to apply. ---------- components: Windows messages: 329550 nosy: gr-dexterl, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Windows Installer Ignores Launcher Installer Options Where The Python Launcher Is Already Present type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 17:33:27 2018 From: report at bugs.python.org (Alexey Izbyshev) Date: Fri, 09 Nov 2018 22:33:27 +0000 Subject: [New-bugs-announce] [issue35204] Disable thread and memory sanitizers for address_in_range() Message-ID: <1541802807.41.0.788709270274.issue35204@psf.upfronthosting.co.za> New submission from Alexey Izbyshev : address_in_range() in Objects/obmalloc.c may access memory that is mapped but is considered free by the underlying libc allocator. In #18596, address sanitizing was disabled for this function. But thread and memory sanitizers similarly trip on this function: WARNING: ThreadSanitizer: heap-use-after-free (pid=24361) Read of size 4 at 0x7b7c00000020 by main thread: #0 address_in_range /scratch2/izbyshev/cpython/Objects/obmalloc.c:1312:23 (python+0x59e912) ==3515==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x9713f8 in address_in_range /scratch2/izbyshev/cpython/Objects/obmalloc.c:1313:35 I suggest to disable them for this function as well. ---------- components: Interpreter Core messages: 329561 nosy: benjamin.peterson, izbyshev, serhiy.storchaka, vstinner priority: normal severity: normal status: open title: Disable thread and memory sanitizers for address_in_range() type: enhancement versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 19:15:57 2018 From: report at bugs.python.org (wenjun.o) Date: Sat, 10 Nov 2018 00:15:57 +0000 Subject: [New-bugs-announce] [issue35205] os.path.realpath preserves the trailing backslash on Windows in python 3.6.7 and python 3.7.1 Message-ID: <1541808957.5.0.788709270274.issue35205@psf.upfronthosting.co.za> New submission from wenjun.o : The statement "os.path.realpath('C:\\Users\\')" returns different results between 3.6.6 and 3.6.7 (and between 3.7.0 and 3.7.1) on Windows. With python 3.6.6 and 3.7.0 I got 'C:\\Users'. With python 3.6.7 and 3.7.1 I got 'C:\\Users\\'. Note the extra trailing backslash with python 3.6.7 and 3.7.1. On Linux the behavior is consistent where the trailing slash in the input will be remove from the output: os.path.realpath('/home/') --> '/home'. I think we should keep the behavior of removing trailing backslash on Windows. Thanks. ---------- components: Library (Lib) messages: 329574 nosy: wouyang priority: normal severity: normal status: open title: os.path.realpath preserves the trailing backslash on Windows in python 3.6.7 and python 3.7.1 type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 20:37:33 2018 From: report at bugs.python.org (STINNER Victor) Date: Sat, 10 Nov 2018 01:37:33 +0000 Subject: [New-bugs-announce] [issue35206] Add new experimental _Py_CAPI2 API Message-ID: <1541813853.84.0.788709270274.issue35206@psf.upfronthosting.co.za> New submission from STINNER Victor : Add a new experimental "_Py_CAPI2" API: opt-in which doesn't leak implementation details and replace macros with function calls. Attached PR modified PyTuple_GET_ITEM() if _Py_CAPI2 is defined. With this API, PyTuple_GET_ITEM() macro becomes a function call and the implementation uses assertions to check if the first argument is a tuple and that the index is valid. It should help to investigate bugs when Python is compiled in debug mode. This issue implements the idea that I proposed on python-dev: https://mail.python.org/pipermail/python-dev/2018-November/155702.html ---------- components: Interpreter Core messages: 329583 nosy: vstinner priority: normal severity: normal status: open title: Add new experimental _Py_CAPI2 API versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 9 23:13:18 2018 From: report at bugs.python.org (Pablo Galindo Salgado) Date: Sat, 10 Nov 2018 04:13:18 +0000 Subject: [New-bugs-announce] [issue35207] Disallow expressions like (a) = 42 Message-ID: <1541823198.59.0.788709270274.issue35207@psf.upfronthosting.co.za> New submission from Pablo Galindo Salgado : As commented on Issue33878, the fact that expressions like (x) = 42 are accepted is, in reality, an implementation detail and they should be disallowed. ---------- assignee: pablogsal components: Interpreter Core messages: 329590 nosy: pablogsal priority: normal severity: normal status: open title: Disallow expressions like (a) = 42 versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 10 02:18:21 2018 From: report at bugs.python.org (Tal Einat) Date: Sat, 10 Nov 2018 07:18:21 +0000 Subject: [New-bugs-announce] [issue35208] IDLE: Squeezed lines count ignores window width Message-ID: <1541834301.3.0.788709270274.issue35208@psf.upfronthosting.co.za> New submission from Tal Einat : Squeezing a single long line with a newline, e.g. 'a'*200, results in "Squeezed text (1 lines)". Also, changing the width of the window (even to very small widths) doesn't affect the number of lines reported when squeezing (when squeezing *after* the width change). ---------- assignee: terry.reedy components: IDLE messages: 329600 nosy: taleinat, terry.reedy priority: normal severity: normal status: open title: IDLE: Squeezed lines count ignores window width type: behavior versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 10 05:56:38 2018 From: report at bugs.python.org (remi bertholet) Date: Sat, 10 Nov 2018 10:56:38 +0000 Subject: [New-bugs-announce] [issue35209] Crash with tkinter text on osx Message-ID: <1541847398.17.0.788709270274.issue35209@psf.upfronthosting.co.za> New submission from remi bertholet : Hello, If I press the key "Alt-Shift-F" with script it crash all times. I reproduces the same crash with Python 2.7. To reproduce it, on terminal osx 10.13.6 (17G3025) : - run "python3 ScrollFrame.py" - Click one time on tk window (it show on foreground) - Press "Ctrl-Alt-F" (all other alt produce the same crash) - And you obtain the crash next : The crash on python 3.7 : 2018-11-10 11:50:30.702 Python[3095:292595] *** Terminating app due to uncaught exception 'NSRangeException', reason: '-[__NSCFConstantString characterAtIndex:]: Range or index out of bounds' *** First throw call stack: ( 0 CoreFoundation 0x00007fff2b04123b __exceptionPreprocess + 171 1 libobjc.A.dylib 0x00007fff522d2c76 objc_exception_throw + 48 2 CoreFoundation 0x00007fff2b0d2d5d +[NSException raise:format:] + 205 3 CoreFoundation 0x00007fff2afbd236 -[__NSCFString characterAtIndex:] + 102 4 Tk 0x00007fff37721343 TkpInitKeymapInfo + 731 5 Tk 0x00007fff377271a1 Tk_MacOSXSetupTkNotifier + 798 6 Tcl 0x00007fff37621a50 Tcl_DoOneEvent + 300 7 _tkinter.cpython-37m-darwin.so 0x000000010dd957ac _tkinter_tkapp_mainloop + 256 8 Python 0x000000010d3d4f42 _PyMethodDef_RawFastCallKeywords + 525 9 Python 0x000000010d3d981e _PyMethodDescr_FastCallKeywords + 82 10 Python 0x000000010d46b1a2 call_function + 615 11 Python 0x000000010d4622ec _PyEval_EvalFrameDefault + 2579 12 Python 0x000000010d46bad1 _PyEval_EvalCodeWithName + 1837 13 Python 0x000000010d3d4474 _PyFunction_FastCallKeywords + 225 14 Python 0x000000010d46b17a call_function + 575 15 Python 0x000000010d462305 _PyEval_EvalFrameDefault + 2604 16 Python 0x000000010d46bad1 _PyEval_EvalCodeWithName + 1837 17 Python 0x000000010d461851 PyEval_EvalCode + 42 18 Python 0x000000010d490a5f run_mod + 54 19 Python 0x000000010d48fa7a PyRun_FileExFlags + 164 20 Python 0x000000010d48f159 PyRun_SimpleFileExFlags + 283 21 Python 0x000000010d4a6842 pymain_main + 5114 22 Python 0x000000010d4a6fd4 _Py_UnixMain + 104 23 libdyld.dylib 0x00007fff52eec015 start + 1 ) libc++abi.dylib: terminating with uncaught exception of type NSException Abort trap: 6 The crash on python 2.7 : 2018-11-10 11:51:34.728 Python[3104:294757] *** Terminating app due to uncaught exception 'NSRangeException', reason: '-[__NSCFConstantString characterAtIndex:]: Range or index out of bounds' *** First throw call stack: ( 0 CoreFoundation 0x00007fff2b04123b __exceptionPreprocess + 171 1 libobjc.A.dylib 0x00007fff522d2c76 objc_exception_throw + 48 2 CoreFoundation 0x00007fff2b0d2d5d +[NSException raise:format:] + 205 3 CoreFoundation 0x00007fff2afbd236 -[__NSCFString characterAtIndex:] + 102 4 Tk 0x00007fff37721343 TkpInitKeymapInfo + 731 5 Tk 0x00007fff377271a1 Tk_MacOSXSetupTkNotifier + 798 6 Tcl 0x00007fff37621a50 Tcl_DoOneEvent + 300 7 _tkinter.so 0x00000001026cca77 Tkapp_MainLoop + 354 8 Python 0x000000010236a357 PyEval_EvalFrameEx + 19822 9 Python 0x00000001023653d2 PyEval_EvalCodeEx + 1562 10 Python 0x000000010236e4d7 fast_function + 290 11 Python 0x000000010236a25a PyEval_EvalFrameEx + 19569 12 Python 0x00000001023653d2 PyEval_EvalCodeEx + 1562 13 Python 0x0000000102364db2 PyEval_EvalCode + 32 14 Python 0x000000010238679b run_mod + 49 15 Python 0x0000000102386842 PyRun_FileExFlags + 130 16 Python 0x00000001023863c4 PyRun_SimpleFileExFlags + 706 17 Python 0x0000000102397f44 Py_Main + 3136 18 libdyld.dylib 0x00007fff52eec015 start + 1 19 ??? 0x0000000000000002 0x0 + 2 ) libc++abi.dylib: terminating with uncaught exception of type NSException Abort trap: 6 ---------- components: Tkinter files: ScrollFrame.py messages: 329606 nosy: remi_bertholet priority: normal severity: normal status: open title: Crash with tkinter text on osx versions: Python 3.7 Added file: https://bugs.python.org/file47918/ScrollFrame.py _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 10 12:23:52 2018 From: report at bugs.python.org (tzickel) Date: Sat, 10 Nov 2018 17:23:52 +0000 Subject: [New-bugs-announce] [issue35210] Use bytes + memoryview + resize instead of bytesarray + array in io.RawIOBase.read Message-ID: <1541870632.41.0.788709270274.issue35210@psf.upfronthosting.co.za> New submission from tzickel : There was a TODO in the code about this: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Modules/_io/iobase.c#L909 ---------- components: IO messages: 329629 nosy: tzickel priority: normal severity: normal status: open title: Use bytes + memoryview + resize instead of bytesarray + array in io.RawIOBase.read type: performance versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 10 14:39:52 2018 From: report at bugs.python.org (bob moth) Date: Sat, 10 Nov 2018 19:39:52 +0000 Subject: [New-bugs-announce] [issue35211] Turtle Shutdown Problem(s) Message-ID: <1541878792.83.0.788709270274.issue35211@psf.upfronthosting.co.za> New submission from bob moth : Running life.py from the Free Python Games requires alt-Q twice, and the GUI keeps running after System.exit. Running life2.py, (which is my clone, and I will happily share it if you want a great stress tester that gobbles 100% of a CPU core for as long as you want,) doesn't respond to alt-Q and requires a force quit. ---------- components: Tkinter, macOS files: turtle shutdown problem.txt messages: 329640 nosy: bmoth, ned.deily, ronaldoussoren priority: normal severity: normal status: open title: Turtle Shutdown Problem(s) type: behavior versions: Python 3.7 Added file: https://bugs.python.org/file47920/turtle shutdown problem.txt _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sun Nov 11 07:34:00 2018 From: report at bugs.python.org (Arminius) Date: Sun, 11 Nov 2018 12:34:00 +0000 Subject: [New-bugs-announce] [issue35212] Expressions with format specifiers in f-strings give wrong code position in AST Message-ID: <1541939640.9.0.788709270274.issue35212@psf.upfronthosting.co.za> New submission from Arminius : ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier. Compare the trees of f'{a}' and f'{a:b}' : >>> ast.parse("f'{a}'") Module( body=[ Expr( lineno=1, col_offset=0, value=JoinedStr( lineno=1, col_offset=0, values=[ FormattedValue( lineno=1, col_offset=0, value=Name(lineno=1, col_offset=3, id='a', ctx=Load()), conversion=-1, format_spec=None, ), ], ), ), ], ) >>> ast.parse("f'{a:b}'") Module( body=[ Expr( lineno=1, col_offset=0, value=JoinedStr( col_offset=0, lineno=1, col_offset=0, values=[ FormattedValue( lineno=1, col_offset=0, value=Name(lineno=1, col_offset=1, id='a', ctx=Load()), conversion=-1, format_spec=JoinedStr( lineno=1, col_offset=0, values=[Str(lineno=1, col_offset=0, s='b')], ), ), ], ), ), ], ) In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets, Name(lineno=1, col_offset=3, id='a', ctx=Load()) and Name(lineno=1, col_offset=1, id='a', ctx=Load()) . Expected behavior: col_offset=3 for name "a" in both f-strings. (In my specific use case, this breaks a semantic highlighting plugin for vim.) ---------- components: Interpreter Core messages: 329672 nosy: arminius priority: normal severity: normal status: open title: Expressions with format specifiers in f-strings give wrong code position in AST type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 00:06:14 2018 From: report at bugs.python.org (Terry J. Reedy) Date: Mon, 12 Nov 2018 05:06:14 +0000 Subject: [New-bugs-announce] [issue35213] IDLE: use 'macOS' where appropriate. Message-ID: <1541999174.82.0.788709270274.issue35213@psf.upfronthosting.co.za> New submission from Terry J. Reedy : 'maxOS' is Apple's current name for the Mac operating system. See https://en.wikipedia.org/wiki/MacOS, Contents, Release History. The module name macosx.py is currently left as is. Within that, 'Mac OS X' is correct for 10.6 for 10.8. ---------- assignee: terry.reedy components: IDLE messages: 329713 nosy: terry.reedy priority: normal severity: normal stage: needs patch status: open title: IDLE: use 'macOS' where appropriate. type: enhancement versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 02:26:35 2018 From: report at bugs.python.org (Gregory P. Smith) Date: Mon, 12 Nov 2018 07:26:35 +0000 Subject: [New-bugs-announce] [issue35214] Get the test suite passing with clang Memory Sanitizer enabled Message-ID: <1542007595.0.0.788709270274.issue35214@psf.upfronthosting.co.za> New submission from Gregory P. Smith : clang's memory sanitizer (-fsanitize=memory) turns up useful problems in code. I'm working on getting a CPython buildbot running it setup but would like our build to be cleaner to start with before I run that. These are the initial fixes required for most of CPython to pass in an msan build. We've been using these with our interpreters at Google. (PR coming) ---------- assignee: gregory.p.smith components: Build, Extension Modules, Interpreter Core, Tests messages: 329723 nosy: gregory.p.smith, twouters priority: normal severity: normal stage: patch review status: open title: Get the test suite passing with clang Memory Sanitizer enabled type: security versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 03:50:08 2018 From: report at bugs.python.org (paul) Date: Mon, 12 Nov 2018 08:50:08 +0000 Subject: [New-bugs-announce] [issue35215] Replacing CPython memory allocation Message-ID: <1542012608.34.0.788709270274.issue35215@psf.upfronthosting.co.za> New submission from paul : Hi all, I am trying to replace the version of malloc/free...etc with my own function suit. I am have issues with loading the initial library setup. I am looking for wisdom from the community as to why this may be the case. Facts: - i just grabbed the latest cpython repo - my memory suit seem to be working from independent testing on other code - i am working on linux - i went into obmalloc.c and replaced the malloc, free, realloc, calloc with my own functions. - i changed the mmap/unmap to use my malloc and free in obmalloc.c - my allocated produces aligned allocations. - i dump the exceptions text being generated to see what is happening: EXCEPTION:: module 'sys' has no attribute '__file__' EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN' EXCEPTION:: module 'sys' has no attribute '__cached__' EXCEPTION:: module 'sys' has no attribute '__path__' EXCEPTION:: module 'builtins' has no attribute '__file__' EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN' EXCEPTION:: module 'builtins' has no attribute '__cached__' EXCEPTION:: module 'builtins' has no attribute '__path__' EXCEPTION:: module '_frozen_importlib' has no attribute '__file__' EXCEPTION:: type object 'FrozenImporter' has no attribute '_ORIGIN' EXCEPTION:: module '_frozen_importlib' has no attribute '__cached__' EXCEPTION:: module '_frozen_importlib' has no attribute '__path__' EXCEPTION:: module '_imp' has no attribute '__file__' EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN' EXCEPTION:: module '_imp' has no attribute '__cached__' EXCEPTION:: module '_imp' has no attribute '__path__' EXCEPTION:: name '_bootstrap' is not defined EXCEPTION:: name '_bootstrap' is not defined EXCEPTION:: name '_bootstrap' is not defined EXCEPTION:: name '_bootstrap' is not defined Fatal Python error: initfsencoding: failed to get the Python codec of the filesystem encoding Traceback (most recent call last): File "/home/paul/fresh_cpython/debug/../Lib/encodings/__init__.py", line 31, in File "", line 989, in _find_and_load File "", line 973, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 773, in exec_module File "", line 909, in get_code File "", line 966, in get_data OSError: [Errno 14] Bad address Aborted ---------- components: Library (Lib) messages: 329725 nosy: paul.harvey at rakuten.com priority: normal severity: normal status: open title: Replacing CPython memory allocation type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 07:19:11 2018 From: report at bugs.python.org (=?utf-8?q?C=C3=A9dric_Van_Rompay?=) Date: Mon, 12 Nov 2018 12:19:11 +0000 Subject: [New-bugs-announce] [issue35216] misleading error message from shutil.copy() Message-ID: <1542025151.44.0.788709270274.issue35216@psf.upfronthosting.co.za> New submission from C?dric Van Rompay : When calling `shutil.copy('file.txt', 'not_here/')`, if directory `not_here/` does not exist, the raised error is: IsADirectoryError: [Errno 21] Is a directory: 'not_here/' If the intent of the user was to copy a file in a directory but the user forgot to create the destination directory, this can be very misleading, as the error tends to indicate that the directory exists. It happened to me and I was thinking "yes it's a directory, then what? that's exactly what I want,copy to this directory!" when the problem was that I forgot to create the destination directory. I would suggest to catch the `IsADirectoryError` in shutil.copyfile() (at `open(dst, 'wb')`) and raise instead an error saying something like "destination is a directory AND this directory does not exists". ---------- components: Library (Lib) messages: 329728 nosy: cedricvanrompay priority: normal severity: normal status: open title: misleading error message from shutil.copy() type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 08:23:21 2018 From: report at bugs.python.org (=?utf-8?b?6Zu25qy454m5?=) Date: Mon, 12 Nov 2018 13:23:21 +0000 Subject: [New-bugs-announce] [issue35217] REPL history is broken when python is invoked with cmd /c Message-ID: <1542029001.03.0.788709270274.issue35217@psf.upfronthosting.co.za> New submission from ??? : Windows 7 x64 Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] Steps to reproduce: 1. Create a script: ``` from subprocess import run run(["cmd.exe", "/c", "python"]) run(["python"]) run("python", shell=True) ``` 2. Run the script. Actual result: The script will invoke Python REPL 3 times. The first and the third REPL don't save the command history. Pressing up/down arrows would clear the entire line. Pressing F7 has no effect. The second REPL works fine. Expected result: Command history should work in all instances. ---------- components: Windows messages: 329729 nosy: paul.moore, steve.dower, tim.golden, zach.ware, ??? priority: normal severity: normal status: open title: REPL history is broken when python is invoked with cmd /c type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 09:38:21 2018 From: report at bugs.python.org (keeely) Date: Mon, 12 Nov 2018 14:38:21 +0000 Subject: [New-bugs-announce] [issue35218] Impossible to round-trip decompress -> compress zipfiles with Python 3 due to losing flag_bits Message-ID: <1542033501.3.0.788709270274.issue35218@psf.upfronthosting.co.za> New submission from keeely : Python 2.7 allows one to capture from one file a list of ZipInfo objects, and then recreate a new file using those ZipInfo objects. The same thing appears to be impossible with Python 3 without resorting to monkey-patches because of a line of code in the writestr() code path ignoring the passed in value of ZipInfo and overwritting it with zeros. See: https://github.com/python/cpython/blob/master/Lib/zipfile.py#L1567 See also a possible solution at: https://stackoverflow.com/questions/53254622/zipfile-header-language-encoding-bit-set-differently-between-python2-and-python3 ---------- components: Library (Lib) messages: 329733 nosy: keeely priority: normal severity: normal status: open title: Impossible to round-trip decompress -> compress zipfiles with Python 3 due to losing flag_bits type: behavior versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 12:57:32 2018 From: report at bugs.python.org (Barry A. Warsaw) Date: Mon, 12 Nov 2018 17:57:32 +0000 Subject: [New-bugs-announce] [issue35219] macOS 10.14 High Sierra crashes in multiprocessing Message-ID: <1542045452.64.0.788709270274.issue35219@psf.upfronthosting.co.za> New submission from Barry A. Warsaw : As we're beginning to roll out macOS 10.14 High Sierra, we're seeing an increase in core files inside /cores. This can consume a ton of disk space if you have coredumps enabled. I don't have a short reproducer yet, but we believe this is related to changes in macOS behavior around fork(). Here are some clues: https://github.com/ansible/ansible/issues/32499#issuecomment-341578864 https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr/52230415#52230415 http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html ---------- messages: 329747 nosy: barry priority: normal severity: normal status: open title: macOS 10.14 High Sierra crashes in multiprocessing versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 15:45:18 2018 From: report at bugs.python.org (Mathieu Dupuy) Date: Mon, 12 Nov 2018 20:45:18 +0000 Subject: [New-bugs-announce] [issue35220] delete "how do I emulate os.kill" section in Windows FAQ Message-ID: <1542055518.86.0.788709270274.issue35220@psf.upfronthosting.co.za> New submission from Mathieu Dupuy : That section is a tip on how to kill process on Windows for Python prior to 2.7 and 3.2. 3.1 end of support was April 2012 and 2.6 was October 2013, so that hasn't been needed for supported versions of Python for more than 5 years. Beside not being needed anymore for a long time, when I read it with the eyes of a Python profane, it makes Python looks bad, like a language from the pasts with warts you need to circumvent. Let's delete that. ---------- assignee: docs at python components: Documentation messages: 329751 nosy: deronnax, docs at python priority: normal pull_requests: 9750 severity: normal status: open title: delete "how do I emulate os.kill" section in Windows FAQ type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 16:57:54 2018 From: report at bugs.python.org (Julien Palard) Date: Mon, 12 Nov 2018 21:57:54 +0000 Subject: [New-bugs-announce] [issue35221] Enhance venv activate commands readability Message-ID: <1542059874.69.0.788709270274.issue35221@psf.upfronthosting.co.za> New submission from Julien Palard : I'd like to enhance the readability of the "venv activation commands" in library/venv.html, my students are often confused by it, sometimes they copy the angle brackets, sometime even the prompt. I don't think we can fix everything, but maybe enhance a bit, I ran a few tests. Before modification: https://screenshotscdn.firefoxusercontent.com/images/77cf6612-1934-4a3b-9376-87a012a76bc3.png First try with a :file:, not better: https://screenshotscdn.firefoxusercontent.com/images/db4e1191-a740-4828-b53a-ccc4a8d33dae.png Second try, simply double backquotes: https://screenshotscdn.firefoxusercontent.com/images/d6be92a1-e4af-40f5-9281-d9c09d8a0908.png Third try using bold but it hurt the eyes: https://screenshotscdn.firefoxusercontent.com/images/6fb1026f-064b-41dd-822b-3b258e37a85e.png 4th try using simply capslock, maybe better: https://screenshotscdn.firefoxusercontent.com/images/9e9c3f7b-639c-4f26-abcc-20948e15a65d.png 5th try using capslock and italics, I like it: https://screenshotscdn.firefoxusercontent.com/images/b5ae5582-1706-4ea9-a702-910daf2c69bc.png Someone on #python-fr proposed https://framapic.org/XnBQZcJVRlZw/F05D7I8nSKd0.png (rendered from markdown in github). I like it but it would require a specific role to render bold-italic in code block. ---------- assignee: mdk components: Documentation messages: 329762 nosy: mdk priority: normal severity: normal status: open title: Enhance venv activate commands readability _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 17:09:52 2018 From: report at bugs.python.org (skreft) Date: Mon, 12 Nov 2018 22:09:52 +0000 Subject: [New-bugs-announce] [issue35222] email.utils.formataddr is not exactly the reverse of email.utils.parseaddr Message-ID: <1542060592.48.0.788709270274.issue35222@psf.upfronthosting.co.za> New submission from skreft : The docs (https://docs.python.org/3/library/email.util.html#email.utils.formataddr) say that formataddr is the inverse of parseaddr, however non-ascii email addresses are treated differently in both methods. parseaddr will return non-ascci addresses, whereas formataddr will raise a UnicodeError. Below is an example: In [1]: import email.utils as u In [2]: u.parseaddr('skreft+?and?@sudoai.com') Out[2]: ('', 'skreft+?and?@sudoai.com') In [3]: u.formataddr(u.parseaddr('skreft+?and?@sudoai.com')) --------------------------------------------------------------------------- UnicodeEncodeError Traceback (most recent call last) in () ----> 1 u.formataddr(u.parseaddr('skreft+?and?@sudoai.com')) /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/email/utils.py in formataddr(pair, charset) 89 name, address = pair 90 # The address MUST (per RFC) be ascii, so raise a UnicodeError if it isn't. ---> 91 address.encode('ascii') 92 if name: 93 try: UnicodeEncodeError: 'ascii' codec can't encode character '\xf1' in position 7: ordinal not in range(128) ---------- components: email messages: 329765 nosy: barry, r.david.murray, skreft priority: normal severity: normal status: open title: email.utils.formataddr is not exactly the reverse of email.utils.parseaddr versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 19:06:16 2018 From: report at bugs.python.org (Roffild) Date: Tue, 13 Nov 2018 00:06:16 +0000 Subject: [New-bugs-announce] [issue35223] Pathlib incorrectly merges strings. Message-ID: <1542067576.58.0.788709270274.issue35223@psf.upfronthosting.co.za> New submission from Roffild : Code: import os print(os.path.join("C:/123\\345", "\\", "folder///filename.bin")) import pathlib print(pathlib.PureWindowsPath("C:/123\\345", "\\", "folder///filename.bin")) Result: C:\folder///filename.bin C:\folder\filename.bin Expected result for Windows: C:\123\345\folder\filename.bin The number of slashes should be controlled by the library. Replacing / on \ should also depend on the OS. ---------- components: IO, Library (Lib), Windows messages: 329776 nosy: Roffild, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Pathlib incorrectly merges strings. type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 19:52:17 2018 From: report at bugs.python.org (Emily Morehouse) Date: Tue, 13 Nov 2018 00:52:17 +0000 Subject: [New-bugs-announce] [issue35224] PEP 572: Assignment Expressions Message-ID: <1542070337.94.0.788709270274.issue35224@psf.upfronthosting.co.za> New submission from Emily Morehouse : This issue will serve to track development and PRs for the implementation of PEP 572: Assignment Expressions. ---------- assignee: emilyemorehouse components: Interpreter Core messages: 329781 nosy: emilyemorehouse, gvanrossum, tim.peters priority: normal severity: normal status: open title: PEP 572: Assignment Expressions type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 12 22:04:30 2018 From: report at bugs.python.org (Benjamin Peterson) Date: Tue, 13 Nov 2018 03:04:30 +0000 Subject: [New-bugs-announce] [issue35225] test_faulthandler fails under ubsan Message-ID: <1542078270.77.0.788709270274.issue35225@psf.upfronthosting.co.za> New submission from Benjamin Peterson : The UBsan buildbot is failing test_faulthandler. We should either spread some __attribute__((no_sanitize_undefined)) around or make the tests skip. ====================================================================== FAIL: test_enable_fd (test.test_faulthandler.FaultHandlerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 274, in test_enable_fd self.check_fatal_error(""" File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 115, in check_fatal_error self.check_error(code, line_number, fatal_error, **kw) File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 108, in check_error output, exitcode = self.get_output(code, filename=filename, fd=fd) File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 73, in get_output self.assertEqual(output, '') AssertionError: 'UndefinedBehaviorSanitizer:DEADLYSIGNAL\n[2619 chars]TING' != '' Diff is 2693 characters long. Set self.maxDiff to None to see it. ====================================================================== FAIL: test_enable_file (test.test_faulthandler.FaultHandlerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 258, in test_enable_file self.check_fatal_error(""" File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 115, in check_fatal_error self.check_error(code, line_number, fatal_error, **kw) File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 108, in check_error output, exitcode = self.get_output(code, filename=filename, fd=fd) File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 68, in get_output self.assertEqual(output, '') AssertionError: 'UndefinedBehaviorSanitizer:DEADLYSIGNAL\n[2619 chars]TING' != '' Diff is 2693 characters long. Set self.maxDiff to None to see it. ====================================================================== FAIL: test_sigfpe (test.test_faulthandler.FaultHandlerTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 176, in test_sigfpe self.check_fatal_error(""" File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 115, in check_fatal_error self.check_error(code, line_number, fatal_error, **kw) File "/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Lib/test/test_faulthandler.py", line 110, in check_error self.assertRegex(output, regex) AssertionError: Regex didn't match: '^Fatal Python error: Floating point exception\n\nCurrent thread 0x[0-9a-f]+ \$most recent call first\$:\n File "", line 3 in ' not found in 'Modules/faulthandler.c:1059:11: runtime error: division by zero\nFatal Python error: Floating point exception\n\nCurrent thread 0x00007f14c5e81740 (most recent call first):\n File "", line 3 in \nUndefinedBehaviorSanitizer:DEADLYSIGNAL\n==5027==ERROR: UndefinedBehaviorSanitizer: FPE on unknown address 0x006d000013a3 (pc 0x7f14c5355f9f bp 0x000002f485a0 sp 0x000002f698e8 T5027)\n #0 0x7f14c5355f9e in gsignal (/lib/x86_64-linux-gnu/libpthread.so.0+0x10f9e)\n #1 0x7f14c53560bf (/lib/x86_64-linux-gnu/libpthread.so.0+0x110bf)\n #2 0x829906 in faulthandler_sigfpe /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/./Modules/faulthandler.c:1059:11\n #3 0x4707c3 in _PyMethodDef_RawFastCallKeywords /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Objects/call.c\n #4 0x46e5d1 in _PyCFunction_FastCallKeywords /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Objects/call.c:730:14\n #5 0x656805 in call_function /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/ceval.c:4570:9\n #6 0x64b0d6 in _PyEval_EvalFrameDefault /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/ceval.c:3164:23\n #7 0x6596be in _PyEval_EvalCodeWithName /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/ceval.c:3946:14\n #8 0x6384ee in PyEval_EvalCodeEx /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/ceval.c:3975:12\n #9 0x6384ee in PyEval_EvalCode /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/ceval.c:508\n #10 0x6e816e in run_mod /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/pythonrun.c:1030:9\n #11 0x6e816e in PyRun_StringFlags /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/pythonrun.c:954\n #12 0x6e80bc in PyRun_SimpleStringFlags /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Python/pythonrun.c:450:9\n #13 0x44db44 in pymain_run_command /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Modules/main.c:293:11\n #14 0x44db44 in pymain_run_python /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Modules/main.c:1600\n #15 0x44db44 in pymain_main /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Modules/main.c:1758\n #16 0x44f29b in _Py_UnixMain /var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/Modules/main.c:1795:12\n #17 0x7f14c4ba72e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)\n #18 0x425499 in _start (/var/lib/buildbot/clang-ubsan/3.x.gps-clang-ubsan/build/python+0x425499)\n\nUndefinedBehaviorSanitizer can not provide additional info.\n==5027==ABORTING' ---------------------------------------------------------------------- ---------- components: Build messages: 329797 nosy: benjamin.peterson priority: normal severity: normal status: open title: test_faulthandler fails under ubsan versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 03:05:32 2018 From: report at bugs.python.org (Chris Withers) Date: Tue, 13 Nov 2018 08:05:32 +0000 Subject: [New-bugs-announce] [issue35226] mock.call equality surprisingly broken Message-ID: <1542096332.14.0.788709270274.issue35226@psf.upfronthosting.co.za> New submission from Chris Withers : $ python3 Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from unittest.mock import call >>> call(x=1) == call(x=2) False Good so far? >>> call(x=1).foo == call(x=2).foo True Eep, I think a lot of people might find that quite surprising! ---------- assignee: michael.foord components: Tests messages: 329813 nosy: cjw296, michael.foord priority: normal severity: normal status: open title: mock.call equality surprisingly broken type: behavior versions: Python 2.7, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 03:27:56 2018 From: report at bugs.python.org (=?utf-8?b?TWljaGHFgiBHw7Nybnk=?=) Date: Tue, 13 Nov 2018 08:27:56 +0000 Subject: [New-bugs-announce] [issue35227] [RFE] tarfile: support adding file objects without prior known size Message-ID: <1542097676.97.0.788709270274.issue35227@psf.upfronthosting.co.za> New submission from Micha? G?rny : Currently, the tarfile module only supports adding files if their size is known prior to adding. However, I think it'd be helpful to be able to store large dynamically generated streams straight into the (uncompressed) .tar file without being able to precalculate the final size and without having to use a temporary file. I'm not really sure how the API should look like (i.e. whether it should be a new method or extension of addfile()) but the mechanism would be rather simple -- write TarInfo with size of 0, write data until end of stream, write padding appropriately to written data, seek back and update TarInfo. Of course, the use of this API would have to be restricted to cases when underlying file supports seeking back and random writes, i.e. not stream, not compressed. ---------- components: Library (Lib) messages: 329818 nosy: mgorny priority: normal severity: normal status: open title: [RFE] tarfile: support adding file objects without prior known size type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 06:06:57 2018 From: report at bugs.python.org (Christian Ullrich) Date: Tue, 13 Nov 2018 11:06:57 +0000 Subject: [New-bugs-announce] [issue35228] Index search in CHM help crashes viewer Message-ID: <1542107217.87.0.788709270274.issue35228@psf.upfronthosting.co.za> New submission from Christian Ullrich : Typing into the index keyword field frequently crashes the CHM viewer with the first or second character. The Python CHMs are the only ones that I see this crash with; it started around 3.6 or so. >From event log: Faulting application name: hh.exe, version: 10.0.17134.1, time stamp: 0xa0eff942 Faulting module name: hhctrl.ocx, version: 10.0.17134.48, time stamp: 0x412ce02a Exception code: 0xc0000005 Fault offset: 0x00000000000387b2 Faulting process ID: 0x3558 Faulting application start time: 0x01d47b3f8db53eef Faulting application path: C:\Windows\hh.exe Faulting module path: C:\Windows\System32\hhctrl.ocx Report ID: 53f84d38-adb4-4c0b-bd45-9bce02406e6f Faulting package full name: Faulting package-relative application ID: This particular crash involved the file from 3.7.1: sha1(doc\python371.chm) = 1c0e2d861cadc759436bfac0a767c5bb633423f9 ---------- assignee: docs at python components: Documentation, Windows messages: 329824 nosy: chrullrich, docs at python, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Index search in CHM help crashes viewer type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 06:49:14 2018 From: report at bugs.python.org (STINNER Victor) Date: Tue, 13 Nov 2018 11:49:14 +0000 Subject: [New-bugs-announce] [issue35229] Deprecate _PyObject_GC_TRACK() in Python 3.6 Message-ID: <1542109754.28.0.788709270274.issue35229@psf.upfronthosting.co.za> New submission from STINNER Victor : The _PyObject_GC_TRACK() and _PyObject_GC_UNTRACK() macros are documented: * https://docs.python.org/dev/c-api/gcsupport.html#c._PyObject_GC_TRACK * https://docs.python.org/dev/c-api/gcsupport.html#c._PyObject_GC_UNTRACK The documentation says "It should not be used for extension modules." These macros have been broken in Python 3.7 by Eric Snow's work on moving internal APIs to Include/internal/: _PyObject_GC_TRACK() access _PyRuntime.gc.generation0, whereas accessing _PyRuntime requires to include Include/internal/pycore_pystate.h (header previously called Include/internal/pystate.h in Python 3.7) which is not installed by "make install". My PR 10507 moved these macros to Include/internal/ to clarify that they must not be used outside CPython internals. I suggest to deprecate them in Python 3.6: just add a note in Python 3.6 and 3.7 documentation. ---------- messages: 329829 nosy: vstinner priority: normal severity: normal status: open title: Deprecate _PyObject_GC_TRACK() in Python 3.6 versions: Python 3.6, Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 07:08:45 2018 From: report at bugs.python.org (INADA Naoki) Date: Tue, 13 Nov 2018 12:08:45 +0000 Subject: [New-bugs-announce] [issue35230] Remove _Py_REF_DEBUG_COMMA Message-ID: <1542110925.09.0.788709270274.issue35230@psf.upfronthosting.co.za> New submission from INADA Naoki : After 2aaf0c, _Py_REF_DEBUG_COMMA in Include/object.h is used only in dictobject.c. Let's remove it. ---------- messages: 329833 nosy: inada.naoki priority: normal severity: normal status: open title: Remove _Py_REF_DEBUG_COMMA _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 08:56:20 2018 From: report at bugs.python.org (pmpp) Date: Tue, 13 Nov 2018 13:56:20 +0000 Subject: [New-bugs-announce] [issue35231] make install may not call ldconfig on posix Message-ID: <1542117380.5.0.788709270274.issue35231@psf.upfronthosting.co.za> New submission from pmpp : observed on ubuntu trusty x64 with release 3.7.1 at least when switching from "m" abi to "dm" abi sometimes libpython is not resolved in /usr/local/lib so maybe add "ldconfig" as a preventive mesure. ---------- components: Installation messages: 329840 nosy: pmpp priority: normal severity: normal status: open title: make install may not call ldconfig on posix type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 09:30:46 2018 From: report at bugs.python.org (Antony Lee) Date: Tue, 13 Nov 2018 14:30:46 +0000 Subject: [New-bugs-announce] [issue35232] Add `module`/`qualname` arguments to make_dataclass for picklability Message-ID: <1542119446.33.0.788709270274.issue35232@psf.upfronthosting.co.za> New submission from Antony Lee : Currently, dataclasses created by make_dataclass are not picklable, because their __module__ is set to "types". It seems that this would be easily fixed by letting make_dataclass gain a `module` and a `qualname` kwarg, similarly to the Enum functional form (https://docs.python.org/3/library/enum.html#functional-api). ---------- components: Library (Lib) messages: 329847 nosy: Antony.Lee priority: normal severity: normal status: open title: Add `module`/`qualname` arguments to make_dataclass for picklability versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 10:11:01 2018 From: report at bugs.python.org (STINNER Victor) Date: Tue, 13 Nov 2018 15:11:01 +0000 Subject: [New-bugs-announce] [issue35233] _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers Message-ID: <1542121861.06.0.788709270274.issue35233@psf.upfronthosting.co.za> New submission from STINNER Victor : _PyMainInterpreterConfig_Copy() doesn't copy the install_signal_handlers attribute and so config->install_signal_handlers is always -1. Bug spotted by Francis Hart: https://github.com/python/cpython/pull/10516 Attached PR fix the issue. ---------- components: Interpreter Core messages: 329849 nosy: vstinner priority: normal severity: normal status: open title: _PyMainInterpreterConfig_Copy() doesn't copy install_signal_handlers versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 11:06:02 2018 From: report at bugs.python.org (Michael Casadevall) Date: Tue, 13 Nov 2018 16:06:02 +0000 Subject: [New-bugs-announce] [issue35234] ssl module falls over with internationalized domain names Message-ID: <1542125162.5.0.788709270274.issue35234@psf.upfronthosting.co.za> New submission from Michael Casadevall : Test case attached. In Python 3.6, ssl tries to validate the hostname on its own, but fails to convert the SSL certificates hostname from IDNA back to UTF-8 and mismatches. Python 3.7 and master are unaffected since this got fixed by accident when validation was changed to depend on OpenSSL alone and not do it in python though the underlying match_hostname function is still bugged. ---------- assignee: christian.heimes components: SSL files: ssl_test.py messages: 329852 nosy: christian.heimes, mcasadevall priority: normal severity: normal status: open title: ssl module falls over with internationalized domain names type: behavior versions: Python 3.6 Added file: https://bugs.python.org/file47932/ssl_test.py _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 12:46:18 2018 From: report at bugs.python.org (Mark Inderhees) Date: Tue, 13 Nov 2018 17:46:18 +0000 Subject: [New-bugs-announce] [issue35235] Access violation on alloc in Windows x86-64 embeddable python, pymalloc_alloc Message-ID: <1542131178.89.0.788709270274.issue35235@psf.upfronthosting.co.za> New submission from Mark Inderhees : I'm seeing an AV in python allocation logic when running my scripts. This only happens when using embedded python. It does not reproduce when using installed python. This is on Windows x86-64, I'm running Windows 10. I've seen this on 3.6.5 and 3.7.1. Here's the 3.7.1 stack on failure: 0:004> kpn # Child-SP RetAddr Call Site 00 000000d1`7439d650 00007ff8`e8303db6 python37!pymalloc_alloc(void ** ptr_p = 0x000000d1`7439d6c0, unsigned int64 nbytes = )+0x4b [c:\_work\5\s\objects\obmalloc.c @ 1398] 01 000000d1`7439d680 00007ff8`e82fb064 python37!_PyObject_Malloc(void * ctx = 0x00000000`00000000, unsigned int64 nbytes = 0x1c)+0x16 [c:\_work\5\s\objects\obmalloc.c @ 1555] 02 (Inline Function) --------`-------- python37!PyObject_Malloc+0x1c [c:\_work\5\s\objects\obmalloc.c @ 616] 03 000000d1`7439d6b0 00007ff8`e82fb18c python37!_PyLong_New(int64 size = 0n1)+0x40 [c:\_work\5\s\objects\longobject.c @ 210] 04 000000d1`7439d6e0 00007ff8`e83cbf28 python37!PyLong_FromLongLong(int64 ival = )+0x4c [c:\_work\5\s\objects\longobject.c @ 1102] 05 (Inline Function) --------`-------- python37!_PyLong_FromTime_t+0xc [c:\_work\5\s\python\pytime.c @ 88] 06 000000d1`7439d710 00007ff8`e83cb86e python37!tmtotuple(struct tm * p = 0x000000d1`7439d768, char * zone = 0x000000d1`7439d7c0 "Pacific Standard Time", int64 gmtoff = 0n-28800)+0xe0 [c:\_work\5\s\modules\timemodule.c @ 426] 07 000000d1`7439d740 00007ff8`e8307e18 python37!time_localtime(struct _object * self = 0x0000029f`3753bbe0, struct _object * args = )+0x96 [c:\_work\5\s\modules\timemodule.c @ 521] 08 000000d1`7439d850 00007ff8`e8308737 python37!_PyMethodDef_RawFastCallKeywords(struct PyMethodDef * method = 0x00000000`00000000, struct _object * self = 0x0000029f`371b7778, struct _object ** args = 0x0000029f`3754b3c8, int64 nargs = 0n1, struct _object * kwnames = 0x00000000`00000000)+0x3a8 [c:\_work\5\s\objects\call.c @ 694] 09 (Inline Function) --------`-------- python37!_PyCFunction_FastCallKeywords+0x22 [c:\_work\5\s\objects\call.c @ 730] 0a 000000d1`7439d8d0 00007ff8`e83091d3 python37!call_function(struct _object *** pp_stack = 0x000000d1`7439da28, int64 oparg = , struct _object * kwnames = 0x00000000`00000000)+0x3a7 [c:\_work\5\s\python\ceval.c @ 4554] 0b 000000d1`7439d990 00007ff8`e82f1390 python37!_PyEval_EvalFrameDefault(struct _frame * f = 0x0000029f`3754b220, int throwflag = 0n928230480)+0x913 [c:\_work\5\s\python\ceval.c @ 3095] 0c (Inline Function) --------`-------- python37!PyEval_EvalFrameEx+0x17 [c:\_work\5\s\python\ceval.c @ 547] 0d 000000d1`7439dad0 00007ff8`e83086a2 python37!_PyEval_EvalCodeWithName(struct _object * _co = 0x0000029f`37415a50, struct _object * globals = , struct _object * locals = , struct _object ** args = 0xffffffff`ffff2a88, int64 argcount = 0n3, struct _object ** kwnames = 0x00000000`00000000, struct _object ** kwargs = 0x0000029f`3753de28, int64 kwcount = 0n0, int kwstep = 0n1, struct _object ** defs = 0x0000029f`3743fe60, int64 defcount = 0n1, struct _object * kwdefs = 0x00000000`00000000, struct _object * closure = 0x00000000`00000000, struct _object * name = 0x0000029f`3744f470, struct _object * qualname = 0x0000029f`37441930)+0x1a0 [c:\_work\5\s\python\ceval.c @ 3930] 0e (Inline Function) --------`-------- python37!_PyFunction_FastCallKeywords+0x274 [c:\_work\5\s\objects\call.c @ 433] 0f 000000d1`7439db80 00007ff8`e8308d29 python37!call_function(struct _object *** pp_stack = 0x000000d1`7439dcd8, int64 oparg = , struct _object * kwnames = 0x00000000`00000000)+0x312 [c:\_work\5\s\python\ceval.c @ 4607] .... ---------- messages: 329857 nosy: markind priority: normal severity: normal status: open title: Access violation on alloc in Windows x86-64 embeddable python, pymalloc_alloc _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 14:20:41 2018 From: report at bugs.python.org (Ian Liu Rodrigues) Date: Tue, 13 Nov 2018 19:20:41 +0000 Subject: [New-bugs-announce] [issue35236] urllib.request.urlopen throws on some valid FTP files Message-ID: <1542136841.11.0.788709270274.issue35236@psf.upfronthosting.co.za> New submission from Ian Liu Rodrigues : Some FTP clients will not allow changing to a directory if the path does not ends with a slash. For example, try out this in a public FTP: from ftplib import FTP ftp = FTP('ftp.unicamp.br') ftp.login() ftp.cwd('pub/libreoffice') # throws error ftp.cwd('pub/libreoffice/') # OK The problem is urllib.request doesn't include the trailing slash, thus throwing an error. This behavior also happens with the command line ftp client. I think this happens because the libreoffice directory is a symlink, and this can be a FTP server specific behavior. ---------- components: Library (Lib) messages: 329862 nosy: Ian Liu Rodrigues priority: normal severity: normal status: open title: urllib.request.urlopen throws on some valid FTP files type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 15:55:02 2018 From: report at bugs.python.org (Thierry Parmentelat) Date: Tue, 13 Nov 2018 20:55:02 +0000 Subject: [New-bugs-announce] [issue35237] Allow Path instances in sys.path ? Message-ID: <1542142502.59.0.788709270274.issue35237@psf.upfronthosting.co.za> New submission from Thierry Parmentelat : Hi; this is my first entry in this repo I ran today into a rather frustrating experience, that I'd to avoid it for others as far as possible In a nutshell I had a sphinx stub that read: import sys sys.path.append("..") import mymodule and because I was switching from os.path to pathlib.Path, I rather stupidly changed it into this import sys sys.path.append(Path("..").resolve()) import mymodule --- When trying to run this new code, the message I got was ModuleNotFoundError So I tried to troubleshoot the situation, and inserted something like for x in sys.path: print(x) and so the proper path showed up, and that's the point where I started banging my head against the screen Granted, I was doing too many changes at the same time, and it's entirely my fault if it took me ages really to figure it out eventually. --- Still, my proposal would be to *) either accept Path instances in sys.path, or *) somehow send a warning message stating that the Path instance is totally ignored; at least, I haven't found the corresponding piece of code yet, but as far as I am concerned it's truly as if this Path instance had been completely ignored when executing the import statement *) or at the very least add a mention about this possible issue in the import documentation I understand it was a stupid mistake, but with Path becoming THE right way to deal with paths, I think it is safe to predict that I won't be the only one making it. As I said I am a complete newbie with cpython's codebase and devel practices, so I'll warmly welcome all inputs on this one. thanks ---------- components: Library (Lib) messages: 329864 nosy: thierry.parmentelat priority: normal severity: normal status: open title: Allow Path instances in sys.path ? type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 16:29:55 2018 From: report at bugs.python.org (Oscar Esteban) Date: Tue, 13 Nov 2018 21:29:55 +0000 Subject: [New-bugs-announce] [issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver Message-ID: <1542144595.84.0.788709270274.issue35238@psf.upfronthosting.co.za> New submission from Oscar Esteban : ## Context We are developers of nipype (https://github.com/nipype) which is a workflow engine for neuroimaging software. We are experiencing problems that gave rise to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this - https://bugs.python.org/issue20104#msg222570 Our software runs command line subprocesses that can be quite memory-hungry and in some cases, in the order of tens of thousands processes. Therefore, we frequently see the OOM killing some of the processes. ## Status We have successfully leveraged the ``forkserver`` context (in addition to a low number of `maxtasksperchild`) of multiprocessing to ease the load. However, the fork_exec memory allocation is still problematic on systems that do not allow overcommitting virtual memory. Waiting for os.posix_spawn to be rolled out might not be an option for us, as the problem is hitting badly right now. ## Proposed solution I'd like to page experts on Lib/multiprocessing and Lib/subprocess to give their opinions about the following: is it possible to write an extension to `multiprocessing.util.Popen` such that it has the API of `subprocess.Popen` but the fork happens via the forkserver? My naive intuition is that we would need to create a new type of Process, make sure that it then calls os.exec*e() --possibly around here https://github.com/python/cpython/blob/f966e5397ed8f5c42c185223fc9b4d750a678d02/Lib/multiprocessing/popen_forkserver.py#L51--, and finally handle communication with the subprocess. Please let me know if that is even possible. ---------- components: Library (Lib) messages: 329868 nosy: oesteban priority: normal severity: normal status: open title: Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver type: enhancement versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 18:08:32 2018 From: report at bugs.python.org (STINNER Victor) Date: Tue, 13 Nov 2018 23:08:32 +0000 Subject: [New-bugs-announce] [issue35239] _PySys_EndInit() doesn't copy main interpreter configuration Message-ID: <1542150512.39.0.788709270274.issue35239@psf.upfronthosting.co.za> New submission from STINNER Victor : Extract of _PySys_EndInit(): SET_SYS_FROM_STRING_BORROW("path", config->module_search_path); sys.path is initialized from _PyMainInterpreterConfig.module_search_path object: the list of strings is not copied. As a consequence, when sys.path is modified, _PyMainInterpreterConfig is modified as well. For example, "import site" modifies sys.path. I dislike this behavior. I prefer to see _PyMainInterpreterConfig as "constant": a snapshot of the configuration used to startup Python. A side effect is that Py_NewInterpreter() copies the modified configuration, whereas starting from the "original" configuration. For the specific case of Py_NewInterpreter(), I'm not sure of what is the expected behavior... Py_NewInterpreter() imports the "site" module again, so it should reapply the same sys.path change. Note: I found this bug while working on better tests for global, core and main configurations: https://github.com/python/cpython/pull/10524 ---------- components: Interpreter Core messages: 329872 nosy: eric.snow, ncoghlan, vstinner priority: normal severity: normal status: open title: _PySys_EndInit() doesn't copy main interpreter configuration versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 20:19:39 2018 From: report at bugs.python.org (STINNER Victor) Date: Wed, 14 Nov 2018 01:19:39 +0000 Subject: [New-bugs-announce] [issue35240] Travis CI: xvfb-run: error: Xvfb failed to start Message-ID: <1542158379.67.0.788709270274.issue35240@psf.upfronthosting.co.za> New submission from STINNER Victor : Travis CI fails randomly with "xvfb-run: error: Xvfb failed to start". Example: https://travis-ci.org/python/cpython/jobs/454776965 whereas the full test suite succeeded. ---------- messages: 329881 nosy: vstinner priority: normal severity: normal status: open title: Travis CI: xvfb-run: error: Xvfb failed to start _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 13 21:38:37 2018 From: report at bugs.python.org (Pablo Galindo Salgado) Date: Wed, 14 Nov 2018 02:38:37 +0000 Subject: [New-bugs-announce] [issue35241] test_embed fails on MacOS buildbots Message-ID: <1542163117.23.0.788709270274.issue35241@psf.upfronthosting.co.za> New submission from Pablo Galindo Salgado : https://buildbot.python.org/all/#/builders/147/builds/430/steps/4/logs/stdio https://buildbot.python.org/all/#/builders/109/builds/789/steps/4/logs/stdio ====================================================================== FAIL: test_init_default_config (test.test_embed.InitConfigTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 398, in test_init_default_config self.check_config("init_default_config", {}, {}) File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 391, in check_config self.assertEqual(config['global_config'], expected_global) AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 0} != {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 0} {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, 'Py_DontWriteBytecodeFlag': 0, 'Py_FileSystemDefaultEncodeErrors': 'surrogateescape', 'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_FrozenFlag': 0, - 'Py_HasFileSystemDefaultEncoding': 1, ? ^ + 'Py_HasFileSystemDefaultEncoding': 0, ? ^ 'Py_HashRandomizationFlag': 1, 'Py_IgnoreEnvironmentFlag': 0, 'Py_InspectFlag': 0, 'Py_InteractiveFlag': 0, 'Py_IsolatedFlag': 0, 'Py_NoSiteFlag': 0, 'Py_NoUserSiteDirectory': 0, 'Py_OptimizeFlag': 0, 'Py_QuietFlag': 0, 'Py_UTF8Mode': 0, 'Py_UnbufferedStdioFlag': 0, 'Py_VerboseFlag': 0} ====================================================================== FAIL: test_init_dev_mode (test.test_embed.InitConfigTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 479, in test_init_dev_mode self.check_config("init_dev_mode", core_config, {}) File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 391, in check_config self.assertEqual(config['global_config'], expected_global) AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 0} != {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 0} {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, 'Py_DontWriteBytecodeFlag': 0, 'Py_FileSystemDefaultEncodeErrors': 'surrogateescape', 'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_FrozenFlag': 0, - 'Py_HasFileSystemDefaultEncoding': 1, ? ^ + 'Py_HasFileSystemDefaultEncoding': 0, ? ^ 'Py_HashRandomizationFlag': 1, 'Py_IgnoreEnvironmentFlag': 0, 'Py_InspectFlag': 0, 'Py_InteractiveFlag': 0, 'Py_IsolatedFlag': 0, 'Py_NoSiteFlag': 0, 'Py_NoUserSiteDirectory': 0, 'Py_OptimizeFlag': 0, 'Py_QuietFlag': 0, 'Py_UTF8Mode': 0, 'Py_UnbufferedStdioFlag': 0, 'Py_VerboseFlag': 0} ====================================================================== FAIL: test_init_isolated (test.test_embed.InitConfigTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 489, in test_init_isolated self.check_config("init_isolated", core_config, global_config) File "/Users/buildbot/buildarea/3.7.billenstein-sierra/build/Lib/test/test_embed.py", line 391, in check_config self.assertEqual(config['global_config'], expected_global) AssertionError: {'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_[467 chars]': 1} != {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, [467 chars]': 1} {'Py_BytesWarningFlag': 0, 'Py_DebugFlag': 0, 'Py_DontWriteBytecodeFlag': 0, 'Py_FileSystemDefaultEncodeErrors': 'surrogateescape', 'Py_FileSystemDefaultEncoding': 'utf-8', 'Py_FrozenFlag': 0, - 'Py_HasFileSystemDefaultEncoding': 1, ? ^ + 'Py_HasFileSystemDefaultEncoding': 0, ? ^ 'Py_HashRandomizationFlag': 1, 'Py_IgnoreEnvironmentFlag': 1, 'Py_InspectFlag': 0, 'Py_InteractiveFlag': 0, 'Py_IsolatedFlag': 1, 'Py_NoSiteFlag': 0, 'Py_NoUserSiteDirectory': 1, 'Py_OptimizeFlag': 0, 'Py_QuietFlag': 0, 'Py_UTF8Mode': 0, 'Py_UnbufferedStdioFlag': 0, 'Py_VerboseFlag': 0} ---------------------------------------------------------------------- Ran 15 tests in 6.518s FAILED (failures=3) 1 test failed again: test_embed ---------- components: Tests messages: 329887 nosy: pablogsal, vstinner priority: normal severity: normal status: open title: test_embed fails on MacOS buildbots versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 00:26:21 2018 From: report at bugs.python.org (Sergei Zobov) Date: Wed, 14 Nov 2018 05:26:21 +0000 Subject: [New-bugs-announce] [issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished Message-ID: <1542173181.91.0.788709270274.issue35242@psf.upfronthosting.co.za> New submission from Sergei Zobov : In case if the process target function put something unpickable in the `multiprocessing.Queue` and finish before `ForkingPickler`'s exception is handled we get two weird situations: * Internal queue semaphore stay acquired and never will be released, so we get `qsize()` always returns 1(or more if more then one process) even if there is nothing in queue's buffer and `empty()` returns `True`. * An exception's traceback will be silently suppressed. To reproduce this behavior I wrote a test in cpython repo (be aware sometime it pass sometime no): https://github.com/szobov/cpython/commit/95524f10a7b6510e9ab4647223cc67af85ebff86 I suppose it happens because of this: * The function `util._exit_function()` calls every time process target function finished. Inside this function the global (for current process) flag `utils._exiting` marks as `True`: https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py#L300 https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L294-L295 * When exception while pickling object happens in `Queue._feed` function we are ignore it because of the clause `is_exiting()` that checks the flag `utils._exiting` described above. https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L257-L259 So we have unreleased semaphore and ignored exceptions. I think it's quite disappointing for user because they have broken queue and nothing about why it happened. I'm not quite sure that it's a bag but I found nothing in documentation about it. ---------- messages: 329889 nosy: szobov priority: normal severity: normal status: open title: multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 03:08:28 2018 From: report at bugs.python.org (pmpp) Date: Wed, 14 Nov 2018 08:08:28 +0000 Subject: [New-bugs-announce] [issue35243] readline timeout too long for async gfx use Message-ID: <1542182908.62.0.788709270274.issue35243@psf.upfronthosting.co.za> New submission from pmpp : unlike https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame which is fires about each 1/60 second : the callback for gui via PyOS_InputHook is as long as 0.1 second. https://github.com/python/cpython/blob/8e0b05e2f4b9fd703cbe1ae8d058852ef3781f44/Modules/readline.c#L1192 using repl asyncronously with such a timer is a bad experience when using async opengl based gui : asyncio loop steps should be able to be served at vsync speed which is not actually possible because of hardcoded value. 0.008 seconds would not be so bad, best would be plan next call to hit T+ frametime ( with a default to 0.016 ) each pass inside the loop. a use case for python would be panda3d and its various gui. real life example in other mainstream language : the javascript repl in the browser console. ---------- components: Extension Modules messages: 329891 nosy: pmpp priority: normal severity: normal status: open title: readline timeout too long for async gfx use type: enhancement versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 07:08:32 2018 From: report at bugs.python.org (Jaime Torres) Date: Wed, 14 Nov 2018 12:08:32 +0000 Subject: [New-bugs-announce] [issue35244] Allow to setup Clang as default compiler for modules build Message-ID: <1542197312.07.0.788709270274.issue35244@psf.upfronthosting.co.za> New submission from Jaime Torres : Clang is not in the list of allowed compilers when executed: python setup.py build --help-compiler --compiler=bcpp Borland C++ Compiler --compiler=cygwin Cygwin port of GNU C Compiler for Win32 --compiler=mingw32 Mingw32 port of GNU C Compiler for Win32 --compiler=msvc Microsoft Visual C++ --compiler=unix standard UNIX-style compiler I've tried to use the value unix in windows, to see if it will search for environment variables CC and CXX, but without success. ---------- components: Distutils messages: 329902 nosy: Jaime Torres, dstufft, eric.araujo priority: normal severity: normal status: open title: Allow to setup Clang as default compiler for modules build versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 08:24:33 2018 From: report at bugs.python.org (Yechoh) Date: Wed, 14 Nov 2018 13:24:33 +0000 Subject: [New-bugs-announce] [issue35245] list comprehension for flattened or nested list differ too much Message-ID: <1542201873.24.0.788709270274.issue35245@psf.upfronthosting.co.za> New submission from Yechoh : Suppose we have: forest = [[1,2],[3,4]] and we want: l1 = [['1','2'],['3','4']] we could write: l1 = [[str(leaf) for leaf in tree] for tree in forest] Now if we want: l2 = ['1','2','3','4'] What I expect to need to write is: l2 = [str(leaf) for leaf in tree for tree in forest] However, this gives an error: Traceback (most recent call last): File "", line 1, in NameError: name 'tree' is not defined Instead, I should write: l2 = [str(leaf) for tree in forest for leaf in tree] Notice the different order. I would prefer if the first version of constructing l2 over the second, since it follows closer to the reading order. Also, it is closer to the comprehension for a nested list, so changing a nested list comprehension to a flattened list comprehension is easy. ---------- messages: 329906 nosy: Yechoh priority: normal severity: normal status: open title: list comprehension for flattened or nested list differ too much type: enhancement _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 08:50:49 2018 From: report at bugs.python.org (lilydjwg) Date: Wed, 14 Nov 2018 13:50:49 +0000 Subject: [New-bugs-announce] [issue35246] asyncio.create_subprocess_exec doesn't accept pathlib.Path like subprocess does Message-ID: <1542203449.76.0.788709270274.issue35246@psf.upfronthosting.co.za> New submission from lilydjwg : When I pass pathlib.Path to asyncio.create_subprocess_exec I get: TypeError: program arguments must be a bytes or text string, not PosixPath It's not so good when subprocess accepts this kind of arguments without issues. So can you add support for this? ---------- components: asyncio messages: 329907 nosy: asvetlov, lilydjwg, yselivanov priority: normal severity: normal status: open title: asyncio.create_subprocess_exec doesn't accept pathlib.Path like subprocess does type: enhancement versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 11:14:47 2018 From: report at bugs.python.org (markmcclain) Date: Wed, 14 Nov 2018 16:14:47 +0000 Subject: [New-bugs-announce] [issue35247] test.test_socket.RDSTest.testPeek hangs indefinitely Message-ID: <1542212087.25.0.788709270274.issue35247@psf.upfronthosting.co.za> New submission from markmcclain : test.test_socket.RDSTest.testPeek() can hang indefinitely. ThreadableTest attempts to ensure ordering of operations, but allows both the client and server to proceed without guaranteeing that the data is sent prior to the blocking attempt to peek at the socket's contents. Either the test needs to make recvfrom non-blocking with a timeout/retry loop or ensure data is sent prior to allowing the server to proceed. ---------- components: Tests messages: 329914 nosy: markmcclain priority: normal severity: normal status: open title: test.test_socket.RDSTest.testPeek hangs indefinitely type: behavior versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 12:50:30 2018 From: report at bugs.python.org (Mathieu Lamarre) Date: Wed, 14 Nov 2018 17:50:30 +0000 Subject: [New-bugs-announce] [issue35248] RawArray causes FileNotFoundError at cleanup Message-ID: <1542217830.03.0.788709270274.issue35248@psf.upfronthosting.co.za> Change by Mathieu Lamarre : ---------- components: Library (Lib) nosy: Mathieu Lamarre priority: normal severity: normal status: open title: RawArray causes FileNotFoundError at cleanup type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 15:08:26 2018 From: report at bugs.python.org (Jules Lasne) Date: Wed, 14 Nov 2018 20:08:26 +0000 Subject: [New-bugs-announce] [issue35249] Docs Makefile always rebuilds entire doc Message-ID: <1542226106.26.0.788709270274.issue35249@psf.upfronthosting.co.za> New submission from Jules Lasne : When working on the Documentation or when translating it, I often have to rebuild the doc locally to see the changes I made and see how it looks. However, with the current configuration, It takes (on my computer at least) more than 4 minutes to build. After investigating, I've found out that the build options on the Makefile of python-docs-fr and of cpython/Docs both have the `E` and `a` options set, which forces a complete rebuild, even when running locally. https://github.com/python/cpython/blob/9ee1d42f019ac827f73479ce241e95733d050e67/Doc/Makefile#L178-L208 https://github.com/python/python-docs-fr/commit/af8c7a95ab5bc50523ba919c74bb6e6b89d16962 My proposal is to add a `make dev` in the `python-docs-fr` Makefile and add in `cpython/Docs` a mode which will only recompile the needed files. ---------- components: Build messages: 329930 nosy: seluj78 priority: normal severity: normal status: open title: Docs Makefile always rebuilds entire doc type: performance versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 15:40:45 2018 From: report at bugs.python.org (Shane) Date: Wed, 14 Nov 2018 20:40:45 +0000 Subject: [New-bugs-announce] [issue35250] Minor parameter documentation mismatch for turtle Message-ID: <1542228045.84.0.788709270274.issue35250@psf.upfronthosting.co.za> New submission from Shane : I noticed a slight mismatch in the parameter documentation for one of turtle's functions. onclick() accepts the parameters (fun, btn, add), but the documentation describes the parameters (fun, num, add). A minor issue, to be sure, but I wanted to point it out since I just noticed it. I'm using Python 3.7.1 but I suspect it's not isolated to that. >>> help(turtle.Turtle().onclick) Help on method onclick in module turtle: onclick(fun, btn=1, add=None) method of turtle.Turtle instance Bind fun to mouse-click event on this turtle on canvas. Arguments: fun -- a function with two arguments, to which will be assigned the coordinates of the clicked point on the canvas. num -- number of the mouse-button defaults to 1 (left mouse button). add -- True or False. If True, new binding will be added, otherwise it will replace a former binding. ---------- assignee: docs at python components: Documentation messages: 329932 nosy: Shane Smith, docs at python priority: normal severity: normal status: open title: Minor parameter documentation mismatch for turtle _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 17:06:27 2018 From: report at bugs.python.org (Lysandros Nikolaou) Date: Wed, 14 Nov 2018 22:06:27 +0000 Subject: [New-bugs-announce] [issue35251] FTPHandler.ftp_open documentation error Message-ID: <1542233187.5.0.788709270274.issue35251@psf.upfronthosting.co.za> New submission from Lysandros Nikolaou : On https://docs.python.org/3/library/urllib.request.html?highlight=request#ftphandler-objects there is the sentence "The login is always done with empty username and password.", but in the (not so rare) case I am not missing something here, the code does parse the username and password and passes it on to connect_ftp, which then uses them. Should we update the docs? ---------- components: email messages: 329936 nosy: barry, lys.nikolaou, r.david.murray priority: normal severity: normal status: open title: FTPHandler.ftp_open documentation error versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 20:14:23 2018 From: report at bugs.python.org (Lysandros Nikolaou) Date: Thu, 15 Nov 2018 01:14:23 +0000 Subject: [New-bugs-announce] [issue35252] test_functools dead code after FIXME Message-ID: <1542244463.09.0.788709270274.issue35252@psf.upfronthosting.co.za> New submission from Lysandros Nikolaou : In test_functools.TestSingleDispatch.test_invalid_registrations (https://github.com/python/cpython/blob/4c596d54aa6a55e9d2a3db78891e656ebbfb63c8/Lib/test/test_functools.py#L2299) there is a FIXME with an immediate return afterwards that says that the code after the return should only be allowed to run after PEP 560 is implemented. Now that it is implemented the dead code should work fine, so the return has to be deleted. ---------- components: Tests messages: 329937 nosy: lys.nikolaou priority: normal severity: normal status: open title: test_functools dead code after FIXME versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 20:50:46 2018 From: report at bugs.python.org (Jorge Ramos) Date: Thu, 15 Nov 2018 01:50:46 +0000 Subject: [New-bugs-announce] [issue35253] Linker warning LNK4281 Message-ID: <1542246646.48.0.788709270274.issue35253@psf.upfronthosting.co.za> New submission from Jorge Ramos : When building python 3.6.7 x64 from source I get multiple warnings: -------------------------------------------------------------------- LINK : LNK4281 - "undesirable base address 0x1D110000 for x64 image; set base address above 4GB for best ASLR optimization". -------------------------------------------------------------------- after that, the build gives the offending module: -------------------------------------- [..\PCbuild\pythoncore.vcxproj] --------------------------------------- (e.g., there are many others) the .. means whatever parent directory the build files are on. I use the following batch file: ------------------------------------------------------------ Tools\msi\buildrelease.bat -x64 --pgo '-m test -q --pgo' ------------------------------------------------------------- Is this a problem? ---------- components: Build messages: 329938 nosy: neyuru priority: normal severity: normal status: open title: Linker warning LNK4281 type: compile error versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 14 23:59:51 2018 From: report at bugs.python.org (mengdexiaolian) Date: Thu, 15 Nov 2018 04:59:51 +0000 Subject: [New-bugs-announce] [issue35254] Process finished with exit code -1073741795 (0xC000001D) Message-ID: <1542257991.0.0.788709270274.issue35254@psf.upfronthosting.co.za> New submission from mengdexiaolian : when i run the following script with face_recognition module, there is a unnormal exception occurs: Process finished with exit code -1073741795 (0xC000001D) import face_recognition encodings = [] filename1 = "catherine.jpg" image1 = face_recognition.load_image_file(filename1) print(image1) encoding1 = face_recognition.face_encodings(image1)[0] print("bobo",encoding1) encodings.append(encoding1) print(encodings) filename2 = "william.jpg" image2 = face_recognition.load_image_file(filename2) encoding2 = face_recognition.face_encodings(image2)[0] encodings.append(encoding2) print(encodings) ---------- components: Extension Modules messages: 329942 nosy: mengdexiaolian priority: normal severity: normal status: open title: Process finished with exit code -1073741795 (0xC000001D) type: compile error versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 06:31:18 2018 From: report at bugs.python.org (Mathieu Dupuy) Date: Thu, 15 Nov 2018 11:31:18 +0000 Subject: [New-bugs-announce] [issue35255] delete "How do I extract the downloaded documentation" section in Windows FAQ Message-ID: <1542281478.85.0.788709270274.issue35255@psf.upfronthosting.co.za> New submission from Mathieu Dupuy : I think it's been a long time since Windows/IE no longer mess up with file extensions, so long I can't recall anymore. I just tried on a Windows 7 + IE 11 (released in 2013) to download the .tar.bz2 archive from docs.python.org and it worked alright. To me, that entry makes Python looks bad ("the archive format we use does not work correctly on the most common, main desktop platform"), the problem seems to not exist anymore, and IMHO, is not really Python related. I think that entry should be removed. ---------- messages: 329947 nosy: deronnax priority: normal severity: normal status: open title: delete "How do I extract the downloaded documentation" section in Windows FAQ _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 08:30:01 2018 From: report at bugs.python.org (Iuliana Netoi) Date: Thu, 15 Nov 2018 13:30:01 +0000 Subject: [New-bugs-announce] [issue35256] The Console of Python 3.7.0. Message-ID: <1542288601.52.0.788709270274.issue35256@psf.upfronthosting.co.za> New submission from Iuliana Netoi : While running a function that uses a list to which I append the element "z", the Console modifies my code by adding "z" to the list each time I consequently run the function. Is that a bug? My function written in the Editor of Python 3.7.0. is the following: #%% my_list = ["this","is","my","very","first","list"] def problem2_2(my_list): my_list.append("z") print(my_list) #%% I click the "Run" icon in the Menu. Then I type problem2_2(my_list) in the CONSOLE. At the first execution it is ok. But when executing right again, the console adds another element "z" to my list. And so on, for each next execution - as if the Console modifies my code. (Anyway when reverting to the Editor, click inside the cell, and then "Run" icon in the Menu, only after the first execution it is printed the right number of elements)<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< There is the Console executions: mmy_list = ["this","is","my","very","first","list"] def problem2_2(my_list): my_list.append("z") print(my_list) problem2_2(my_list) ['this', 'is', 'my', 'very', 'first', 'list', 'z'] problem2_2(my_list) ['this', 'is', 'my', 'very', 'first', 'list', 'z', 'z'] problem2_2(my_list) ['this', 'is', 'my', 'very', 'first', 'list', 'z', 'z', 'z'] problem2_2(my_list) ['this', 'is', 'my', 'very', 'first', 'list', 'z', 'z', 'z', 'z'] ---------- components: Tests messages: 329949 nosy: iuliananet priority: normal severity: normal status: open title: The Console of Python 3.7.0. versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 11:14:01 2018 From: report at bugs.python.org (Charalampos Stratakis) Date: Thu, 15 Nov 2018 16:14:01 +0000 Subject: [New-bugs-announce] [issue35257] Add LDFLAGS_NODIST for the LDFLAGS not intended for propagation to C extensions. Message-ID: <1542298441.4.0.788709270274.issue35257@psf.upfronthosting.co.za> New submission from Charalampos Stratakis : Through acb8c5234302f8057b331abaafb2cc8697daf58f the CFLAGS_NODIST variable was created, in order to place there compiler flags used by the interpreter, but not intended to be propagated to C extensions. I saw a similar issue when working on backporting 67e997bcfdac55191033d57a16d1408aL1313 on python 3.6, where the -flto flag should be passed to CFLAGS_NODIST instead of BASECFLAGS, however even if that is fixed, the LDFLAGS will still be propagated to C extensions. Thus in order to provide more flexibility in that regard, I propose to add the LDFLAGS_NODIST variable, which in a similar vein as CFLAGS_NODIST, will hold the LDFLAGS intended to be used only by the interpreter. Thoughts or comments on this approach? ---------- components: Build, Distutils, Extension Modules messages: 329951 nosy: cstratak, dstufft, eric.araujo priority: normal severity: normal status: open title: Add LDFLAGS_NODIST for the LDFLAGS not intended for propagation to C extensions. versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 12:33:43 2018 From: report at bugs.python.org (Alexey Izbyshev) Date: Thu, 15 Nov 2018 17:33:43 +0000 Subject: [New-bugs-announce] [issue35258] Consider enabling -Wmissing-prototypes Message-ID: <1542303223.2.0.788709270274.issue35258@psf.upfronthosting.co.za> New submission from Alexey Izbyshev : This issue is a follow-up of msg329608 of #35081. GCC and Clang have -Wmissing-prototypes[1] diagnostic that is reported if a global function is defined without a previous declaration containing a prototype. The reasons may be the following: 1) The header where it is declared is not included before the definition. 2) There is a declaration before the definition, but it doesn't contain a prototype (e.g., "int foo()"). 3) There is no separate declaration of the function in the project at all. (1) is undesirable because the compiler can't check that signatures of the declaration and the definition match. If they don't, subtle issues may occur when such function is called using a wrong declaration. (2) is undesirable too because there is usually no reason to use declarations without prototypes. Often "int foo()" was meant to be "int foo(void)" -- these two have different meaning in C. (3) may mean that either the function is unused (which is probably undesirable) or it was intentionally defined without a declaration. One case when the latter makes sense is plugin-like files which define initialization functions or similar external entry points. Those may be called via pointers obtained with dlsym() or have calls generated at compile time, like Modules/config.c in CPython. It would be good to enable -Wmissing-prototypes to catch (1) and (2) -- such issues exist as of time of this report. However, (3) is a problem because of module initialization functions which would cause unwanted diagnostics. There is no function attribute that could be put in PyMODINIT_FUNC macro to suppress the diagnostic for a single function. The general diagnostic suppression machinery (#pragma GCC diagnostic[2]) affects everything on the following lines, and while the state can be saved and restored via "#pragma GCC diagnostic push/pop", we'd either need another macro to restore the state after the end of module initializer definitions or require that initializers always come last in the file. A workaround is to declare the initializer inside PyMODINIT_FUNC (or a new macro). For example (omitting platform-specific parts of PyMODINIT_FUNC): #define PyMODINIT_FUNC(name) \ extern PyObject *name(void); \ PyObject *name(void) PyMODINIT_FUNC(PyInit_time) { ... } I've also noticed that PyMODINIT_FUNC is used for functions with a different signature in Modules/_testmultiphase.c, so either another macro would be needed for that, or a general macro accepting a prototype as its variadic parameter could be added. Even if it's deemed infeasible to enable -Wmissing-prototypes by default, developers may find it useful to ensure that the diagnostic is reported only for files in Modules directory. [1] https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html [2] https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html ---------- components: Build messages: 329954 nosy: izbyshev, vstinner priority: normal severity: normal status: open title: Consider enabling -Wmissing-prototypes type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 13:24:51 2018 From: report at bugs.python.org (Arthur Neufeld) Date: Thu, 15 Nov 2018 18:24:51 +0000 Subject: [New-bugs-announce] [issue35259] Py_FinalizeEx unconditionally exists in Py_LIMITED_API Message-ID: <1542306291.56.0.788709270274.issue35259@psf.upfronthosting.co.za> New submission from Arthur Neufeld : In application compiled with #define Py_LIMITED_API 0x03040000 this method was used Py_FinalizeEx() Tested application with: SET PATH=...;C:\Program Files\Python35 APPLICATION.EXE Result: Entry Point Not Found (X) The procedure entry point Py_FinalizeEx could not be located in the dynamic link library C:\PATH\TO\APPLICATION.EXE Expected Result: Application runs Rebuilding, after replacing Py_FinalizeEx(void) with Py_Finalize(void): Application runs when pointing to 3.5 DLLs. Suggested resolution: Declaration should be restricted to ABI 3.6+: #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000 PyAPI_FUNC(int) Py_FinalizeEx(void); #endif ---------- components: Interpreter Core messages: 329959 nosy: AJNeufeld priority: normal severity: normal status: open title: Py_FinalizeEx unconditionally exists in Py_LIMITED_API type: crash versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 16:09:43 2018 From: report at bugs.python.org (batterystaple456) Date: Thu, 15 Nov 2018 21:09:43 +0000 Subject: [New-bugs-announce] [issue35260] 2to3 Parse Error on Python 3 print() with arguments Message-ID: <1542316183.46.0.788709270274.issue35260@psf.upfronthosting.co.za> New submission from batterystaple456 : File a.py: ------------------------------------ print("foo", end='bar') # ParseError ------------------------------------ I think the error is caused by arguments in the Python 3 print() function (e.g. end, sep) that are not ignored by 2to3, and (possibly) treated like Python 2 code. Console stuff: --------------------------------------------------- $ pip install 2to3 Requirement already satisfied: 2to3 in .\python37\lib\site-packages (1.0) $ 2to3 .\a.py ... RefactoringTool: Can't parse .\a.py: ParseError: bad input: type=22, value='=', context=('', (1, 16)) ... --------------------------------------------------- ---------- components: 2to3 (2.x to 3.x conversion tool) messages: 329966 nosy: batterystaple456 priority: normal severity: normal status: open title: 2to3 Parse Error on Python 3 print() with arguments type: compile error versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 16:56:03 2018 From: report at bugs.python.org (pmpp) Date: Thu, 15 Nov 2018 21:56:03 +0000 Subject: [New-bugs-announce] [issue35261] readline.c: PyOS_InputHook not protected against SIGWINCH Message-ID: <1542318963.39.0.788709270274.issue35261@psf.upfronthosting.co.za> New submission from pmpp : when using PyOS_InputHook from within readline module two signals are to be handled SIGINT and SIGWINCH SIGINT is really usefull in case hook has a problem though in some case it should be nice to prevent it too (async loop in repl background) but SIGWINCH is an annoyance and often lead to interpreter crash. sample file: linux crash test under ubuntu bionic python 3.7.1 gcc7 more elaborated test case https://github.com/pmp-p/aioprompt ---------- components: Extension Modules files: pih.py messages: 329969 nosy: pmpp priority: normal severity: normal status: open title: readline.c: PyOS_InputHook not protected against SIGWINCH type: crash versions: Python 3.7 Added file: https://bugs.python.org/file47935/pih.py _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 15 20:44:22 2018 From: report at bugs.python.org (Abram Clark) Date: Fri, 16 Nov 2018 01:44:22 +0000 Subject: [New-bugs-announce] [issue35262] There should be a list.get just like dict.get Message-ID: <1542332662.89.0.788709270274.issue35262@psf.upfronthosting.co.za> New submission from Abram Clark : Just like with dictionaries, it is convenient to index lists without catching exceptions. Adding an import for this most basic functionality is slightly tedious, and I can't imagine it would break anything to add a list.get method. ---------- messages: 329978 nosy: Abram Clark priority: normal severity: normal status: open title: There should be a list.get just like dict.get type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 16 01:26:06 2018 From: report at bugs.python.org (Raymond Hettinger) Date: Fri, 16 Nov 2018 06:26:06 +0000 Subject: [New-bugs-announce] [issue35263] Add None handling for get_saved() in IDLE Message-ID: <1542349566.92.0.788709270274.issue35263@psf.upfronthosting.co.za> New submission from Raymond Hettinger : Exception in Tkinter callback Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tkinter/__init__.py", line 1705, in __call__ return self.func(*args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/multicall.py", line 176, in handler r = l[i](event) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/filelist.py", line 54, in close_all_callback reply = edit.close() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/pyshell.py", line 1014, in close return EditorWindow.close(self) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/editor.py", line 1015, in close reply = self.maybesave() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/outwin.py", line 93, in maybesave return 'yes' if self.get_saved() else 'no' File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/idlelib/editor.py", line 948, in get_saved return self.undo.get_saved() AttributeError: 'NoneType' object has no attribute 'get_saved' ---------- assignee: terry.reedy components: IDLE messages: 329982 nosy: rhettinger, terry.reedy priority: normal severity: normal status: open title: Add None handling for get_saved() in IDLE versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 16 08:10:44 2018 From: report at bugs.python.org (Alexandru Ardelean) Date: Fri, 16 Nov 2018 13:10:44 +0000 Subject: [New-bugs-announce] [issue35264] SSL Module build fails with OpenSSL 1.1.0 for Python 2.7 Message-ID: <1542373844.15.0.788709270274.issue35264@psf.upfronthosting.co.za> New submission from Alexandru Ardelean : The Modules/_ssl.c build fails. Error is: ``` building '_ssl' extension arm-openwrt-linux-muslgnueabi-gcc -fPIC -fno-strict-aliasing -Os -pipe -mcpu=xscale -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=soft -iremap/build_dir/target-arm_xscale_musl_eabi/Python-2.7.15:Python-2.7.15 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -DNDEBUG -fno-inline -DNDEBUG -Os -pipe -mcpu=xscale -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=soft -iremap/build_dir/target-arm_xscale_musl_eabi/Python-2.7.15:Python-2.7.15 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I. -IInclude -I./Include -I/staging_dir/target-arm_xscale_musl_eabi/include -I/staging_dir/toolchain-arm_xscale_gcc-7.3.0_musl_eabi/usr/include -I/staging_dir/toolchain-arm_xscale_gcc-7.3.0_musl_eabi/include/fortify -I/staging_dir/toolchain-arm_xscale_gcc-7.3.0_musl_eabi/include -I/staging_dir/target-arm_xscale_musl_eabi/usr/include -I/staging_dir/hostpkg/bin/Include -I/staging_dir/hostpkg/bin -c /build_dir/target-arm_xscale_musl_eabi/Python-2.7.15/Modules/_ssl.c -o build/temp.linux2-2.7//build_dir/target-arm_xscale_musl_eabi/Python-2.7.15/Modules/_ssl.o cc1: note: someone does not honour COPTS correctly, passed 2 times /build_dir/target-arm_xscale_musl_eabi/Python-2.7.15/Modules/_ssl.c:2117:44: error: 'PySSL_selected_npn_protocol' undeclared here (not in a function); did you mean 'PySSL_selected_alpn_protocol'? {"selected_npn_protocol", (PyCFunction)PySSL_selected_npn_protocol, METH_NOARGS}, ^~~~~~~~~~~~~~~~~~~~~~~~~~~ PySSL_selected_alpn_protocol ``` This is on OpenWrt. ---------- assignee: christian.heimes components: Build, SSL messages: 329995 nosy: Alexandru Ardelean, christian.heimes priority: normal severity: normal status: open title: SSL Module build fails with OpenSSL 1.1.0 for Python 2.7 versions: Python 2.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 16 09:25:45 2018 From: report at bugs.python.org (STINNER Victor) Date: Fri, 16 Nov 2018 14:25:45 +0000 Subject: [New-bugs-announce] [issue35265] Internal C API: pass the memory allocator in a context Message-ID: <1542378345.74.0.788709270274.issue35265@psf.upfronthosting.co.za> New submission from STINNER Victor : Currently, the Python initialization code base is full of: static void pymain_clear_cmdline(_PyMain *pymain, _PyCmdline *cmdline) { PyMemAllocatorEx old_alloc; _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc); ... PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc); } It's a hack that I wrote when I worked on the Python initialization code, to make sure that we use the same memory allocator to allocate and release memory. The problem is that Python allocates memory at startup, then the memory allocator can be changed in Py_Initialize() (ex: by PYTHONMALLOC environment variable), but at exit, we need to reuse the same memory allocator that we used to allocate memory early. Instead of "forcing" the memory allocator to "default allocator", I propose to add a new "context" structure which contains the memory allocator (PyMemAllocatorEx structure). Example: typedef struct { /* Enable UTF-8 mode? Set by -X utf8 command line option and PYTHONUTF8 environment variable. If set to -1 (default), inherit Py_UTF8Mode value. */ int utf8_mode; PyMemAllocatorEx raw_alloc; } _PyConfigCtx; The context should be passed to any function allocating directly or indirectly memory. Py_Main(), "core config" and "path config" are good target for these changes. These functions are used during Python initialization. I know that Eric Snow would like to pass a "context" to functions of the C API. I don't want to make a giant change of the C API. I only want to modify the internal C API, and only modifiy Python internal. The context that I propose is designed for early Python initialization, and fix the practical problem of memory allocator. Later we can discussed how _PyRuntime and the pointer to the current interpreter can be added into this context, or if a new context should be created. Attached PR is a proof-of-concept of my idea, but the giant PR might be splitted into smaller changes to more incremental changes. ---------- components: Interpreter Core messages: 330001 nosy: eric.snow, ncoghlan, vstinner priority: normal severity: normal status: open title: Internal C API: pass the memory allocator in a context type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 16 09:44:06 2018 From: report at bugs.python.org (STINNER Victor) Date: Fri, 16 Nov 2018 14:44:06 +0000 Subject: [New-bugs-announce] [issue35266] Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpreterConfig Message-ID: <1542379446.44.0.788709270274.issue35266@psf.upfronthosting.co.za> New submission from STINNER Victor : The C code of Python initialization uses _PyCoreConfig and _PyMainInterpreterConfig which co-exist and more or less redundant. For example, both structures have "argv": wchar_** for _PyCoreConfig, PyObject* (list of str) for _PyMainInterpreterConfig. I propose to put _PyCoreConfig inside _PyMainInterpreterConfig and move wchar_t* strings into a new _PyPreConfig structure. The main idea is that _PyPreConfig and wchar_* type are only used for early Python initialization, but once Python is initialization, the configuration should be provided as Python objects. I didn't check all configuration variables to see if it's doable or not, but I wrote a proof-of-concept pull request to show my idea with concrete (and working!) code. See attached PR. ---------- components: Interpreter Core messages: 330005 nosy: eric.snow, ncoghlan, vstinner priority: normal severity: normal status: open title: Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpreterConfig versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 16 20:06:39 2018 From: report at bugs.python.org (dzhu) Date: Sat, 17 Nov 2018 01:06:39 +0000 Subject: [New-bugs-announce] [issue35267] reproducible deadlock with multiprocessing.Pool Message-ID: <1542416799.14.0.788709270274.issue35267@psf.upfronthosting.co.za> New submission from dzhu : The attached snippet causes a deadlock just about every time it's run (tested with 3.6.7/Ubuntu, 3.7.1/Arch, 3.6.7/OSX, and 3.7.1/OSX -- deadlock seems to be less frequent on the last, but still common). The issue appears to be something like the following sequence of events: 1. The main thread calls pool.__exit__, eventually entering Pool._terminate_pool. 2. result_handler's state is set to TERMINATE, causing it to stop reading from outqueue. 3. The main thread, in _terminate_pool, joins on worker_handler, which is (usually) in the middle of sleeping for 0.1 seconds, opening a window for the next two steps to occur. 4. The worker process finishes its task and acquires the shared outqueue._wlock. 5. The worker attempts to put the result into outqueue, but its pickled form is too big to fit into the buffer of os.pipe, and it blocks here with the lock held. 6. worker_handler wakes up and exits, freeing _terminate_pool to continue. 7. _terminate_pool terminates the worker. 8. task_handler tries to put None into outqueue, but blocks, since the lock was acquired by the terminated worker. 9. _terminate_pool joins on task_handler, and everything is deadlocked. ---------- components: Library (Lib) files: lock.py messages: 330017 nosy: dzhu priority: normal severity: normal status: open title: reproducible deadlock with multiprocessing.Pool type: behavior versions: Python 3.6, Python 3.7 Added file: https://bugs.python.org/file47937/lock.py _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 17 06:52:54 2018 From: report at bugs.python.org (Cezary Wagner) Date: Sat, 17 Nov 2018 11:52:54 +0000 Subject: [New-bugs-announce] [issue35268] Windows asyncio reading continously stdin and stdout Stockfish Message-ID: <1542455574.0.0.788709270274.issue35268@psf.upfronthosting.co.za> New submission from Cezary Wagner : I have some problems with asyncio on Windows - it block where it should go. Documentation shows that it should work: https://docs.python.org/3/library/asyncio-stream.html -> StreamReader should return something. See 1st example: import asyncio import sys async def run_stockfish(): STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish 9\stockfish_9_x64_bmi2.exe' stockfish = await asyncio.subprocess.create_subprocess_exec( STOCKFISH_PATH, stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) stockfish.stdin.write('uci'.encode()) while True: # BUG? - blocks at this line - no output line = await stockfish.stdout.read() print(line) await stockfish.wait() if sys.platform == "win32": asyncio.set_event_loop_policy( asyncio.WindowsProactorEventLoopPolicy()) asyncio.run(run_stockfish(), debug=True) 1st output (nothing): Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32 runfile('C:/Users/Cezary Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy/s01_async_stockfish.py', wdir='C:/Users/Cezary Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy') 2nd example: import asyncio import sys async def run_stockfish(): STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish 9\stockfish_9_x64_bmi2.exe' stockfish = await asyncio.subprocess.create_subprocess_exec( STOCKFISH_PATH, stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) stockfish.stdin.write('uci'.encode()) while True: # BUG? - blocks at this line line = await stockfish.stdout.readline() print(line) await stockfish.wait() if sys.platform == "win32": asyncio.set_event_loop_policy( asyncio.WindowsProactorEventLoopPolicy()) asyncio.run(run_stockfish(), debug=True) 2nd output is little better (first line is read): Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32 runfile('C:/Users/Cezary Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy/s01_async_stockfish.py', wdir='C:/Users/Cezary Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy') b'Stockfish 9 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott\r\n' What should be done in both case (or maybe done): Stockfish 9 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott uci id name Stockfish 9 64 BMI2 id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott option name Debug Log File type string default option name Contempt type spin default 20 min -100 max 100 option name Threads type spin default 1 min 1 max 512 option name Hash type spin default 16 min 1 max 131072 option name Clear Hash type button option name Ponder type check default false option name MultiPV type spin default 1 min 1 max 500 option name Skill Level type spin default 20 min 0 max 20 option name Move Overhead type spin default 30 min 0 max 5000 option name Minimum Thinking Time type spin default 20 min 0 max 5000 option name Slow Mover type spin default 89 min 10 max 1000 option name nodestime type spin default 0 min 0 max 10000 option name UCI_Chess960 type check default false option name SyzygyPath type string default option name SyzygyProbeDepth type spin default 1 min 1 max 100 option name Syzygy50MoveRule type check default true option name SyzygyProbeLimit type spin default 6 min 0 max 6 uciok ---------- components: Windows, asyncio messages: 330028 nosy: Cezary.Wagner, asvetlov, paul.moore, steve.dower, tim.golden, yselivanov, zach.ware priority: normal severity: normal status: open title: Windows asyncio reading continously stdin and stdout Stockfish versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 17 12:15:56 2018 From: report at bugs.python.org (Zackery Spytz) Date: Sat, 17 Nov 2018 17:15:56 +0000 Subject: [New-bugs-announce] [issue35269] A possible segfault involving a newly-created coroutine Message-ID: <1542474956.13.0.788709270274.issue35269@psf.upfronthosting.co.za> New submission from Zackery Spytz : If compute_cr_origin() fails in PyCoro_New(), coro->cr_origin won't be initialized. This will cause a crash during the coroutine's deallocation. I'll create a PR for this issue. ---------- components: Interpreter Core, asyncio messages: 330031 nosy: ZackerySpytz, asvetlov, yselivanov priority: normal severity: normal status: open title: A possible segfault involving a newly-created coroutine type: crash versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sat Nov 17 18:06:01 2018 From: report at bugs.python.org (daniel hahler) Date: Sat, 17 Nov 2018 23:06:01 +0000 Subject: [New-bugs-announce] [issue35270] Cmd.complete does not handle cmd=None Message-ID: <1542495961.77.0.788709270274.issue35270@psf.upfronthosting.co.za> New submission from daniel hahler : When `parseline` returns `None` for `cmd` (like pdb++ might cause it when changing a cmd "disable" to "!disable"), the following will cause a TypeError: > compfunc = getattr(self, 'complete_' + cmd) "None" should be also forwarded to "completedefault", just like cmd="". ---------- components: Library (Lib) messages: 330035 nosy: blueyed priority: normal severity: normal status: open title: Cmd.complete does not handle cmd=None type: crash versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sun Nov 18 07:08:07 2018 From: report at bugs.python.org (wvxvw) Date: Sun, 18 Nov 2018 12:08:07 +0000 Subject: [New-bugs-announce] [issue35271] venv creates pyvenv.cfg with wrong home Message-ID: <1542542887.08.0.788709270274.issue35271@psf.upfronthosting.co.za> New submission from wvxvw : I must have some environment variable set, which affects this behavior, but I don't know which one that would be. What happens is as follows: No matter in what directory I execute this command: python3 -m venv .venv or python3 -m venv $(pwd)/.venv no matter if the path is relative or absolute, the value of "home" in generated pyvenv.cfg will point to some other virtual environment I created earlier. I would expect that at the minimum, if you aren't going to use the argument given on command line, then, at the minimum, alert the user about it. Why even bother given the argument in this case? PS. I discovered this, perhaps, due to using Emacs + pyvenv-* commands which probably set some environment variables. ---------- components: Library (Lib) messages: 330044 nosy: wvxvw priority: normal severity: normal status: open title: venv creates pyvenv.cfg with wrong home type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Sun Nov 18 08:36:06 2018 From: report at bugs.python.org (Muhammed Alkan) Date: Sun, 18 Nov 2018 13:36:06 +0000 Subject: [New-bugs-announce] [issue35272] sqlite3 get the connected database url Message-ID: <1542548166.85.0.788709270274.issue35272@psf.upfronthosting.co.za> New submission from Muhammed Alkan : A feature for to get the connected database path or URL from connection object (sqlite3.Connection). ---------- messages: 330047 nosy: berker.peksag, ghaering, midnio priority: normal severity: normal status: open title: sqlite3 get the connected database url type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 01:24:19 2018 From: report at bugs.python.org (yesheng) Date: Mon, 19 Nov 2018 06:24:19 +0000 Subject: [New-bugs-announce] [issue35273] 'eval' in generator expression behave different in dict from list Message-ID: <1542608659.0.0.788709270274.issue35273@psf.upfronthosting.co.za> New submission from yesheng <13611358285 at 139.com>: def yyy(): a, b = 'abc', 'abd' print([eval(i) for i in ('a', 'b')]) def zzz(): a, b = 'abc', 'abd' print({i: eval(i) for i in ('a', 'b')}) yyy() # ok zzz() # NameError: name 'a' is not defined, however in yyy() it is ok ---------- messages: 330073 nosy: yesheng priority: normal severity: normal status: open title: 'eval' in generator expression behave different in dict from list type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 02:45:59 2018 From: report at bugs.python.org (otst) Date: Mon, 19 Nov 2018 07:45:59 +0000 Subject: [New-bugs-announce] [issue35274] Running print("\x98") then freeze in Interpreter Message-ID: <1542613559.05.0.788709270274.issue35274@psf.upfronthosting.co.za> New submission from otst : My environment OS:Ubuntu18.04(x64) Python:3.6.6 Run print("\x98") in Python3 interpreter then freeze or slow responsed. Not problem run print '\x98' in Python 2.7.15rc1. Also no problem for python3 -c "print('\x98');" and run .py file. ---------- components: Interpreter Core messages: 330076 nosy: otst priority: normal severity: normal status: open title: Running print("\x98") then freeze in Interpreter type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 05:38:14 2018 From: report at bugs.python.org (Thomas Guettler) Date: Mon, 19 Nov 2018 10:38:14 +0000 Subject: [New-bugs-announce] [issue35275] Reading umask (thread-safe) Message-ID: <1542623894.75.0.788709270274.issue35275@psf.upfronthosting.co.za> New submission from Thomas Guettler : Up to now there is no thread way to read the umask in Python https://stackoverflow.com/questions/53227072/reading-umask-thread-safe You can use this pattern: current_umask = os.umask(0) # line1 os.umask(current_umask) # line2 return current_umask A thread which executes between line1 and line2 will have a different umask. I would be great, if the python standard library would provide correspondig thread safe method. Related question at stackoverflow: https://stackoverflow.com/questions/53227072/reading-umask-thread-safe ---------- messages: 330083 nosy: guettli priority: normal severity: normal status: open title: Reading umask (thread-safe) _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 06:50:08 2018 From: report at bugs.python.org (STINNER Victor) Date: Mon, 19 Nov 2018 11:50:08 +0000 Subject: [New-bugs-announce] [issue35276] Document thread safety Message-ID: <1542628208.58.0.788709270274.issue35276@psf.upfronthosting.co.za> New submission from STINNER Victor : Many developers only discover that a Python function/module is not thread safe because they have a bug in production... Some examples: * bpo-7672: ssl * bpo-8865: select.poll is not thread safe * bpo-539175, bpo-21216: socket.gethostbyname() * bpo-7980: time.strptime() * bpo-6647: warnings.catch_warnings() * bpo-11077, bpo-33479: Tkinter * bpo-1336, bpo-19809: subprocess on Python 2 * bpo-15329: deque * bpo-35275: os.umask() Hopefully, sometimes it was possible to fix it: * bpo-3139: bytearray, buffer protocol * bpo-28969: @functools.lru_cache * bpo-21291: subprocess.Popen.wait() In the asyncio documentation, I explicitly documented that, by design, most classes are not thread-safe. For example, asyncio.Lock() is *NOT* thread-safe: https://docs.python.org/dev/library/asyncio-sync.html#asyncio.Lock Maybe we should start to use a standard way to describe "thread safety". See "POSIX Safety Concepts" of the GNU libc: https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html#POSIX-Safety-Concepts Example with setlocale, "MT-Unsafe": https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html -- My own (incomplete) list of "process-wide states": https://vstinner.readthedocs.io/threads.html#process-wide ---------- assignee: docs at python components: Documentation messages: 330093 nosy: docs at python, vstinner priority: normal severity: normal status: open title: Document thread safety versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 07:21:44 2018 From: report at bugs.python.org (Donald Stufft) Date: Mon, 19 Nov 2018 12:21:44 +0000 Subject: [New-bugs-announce] [issue35277] Upgrade bundled pip/setuptools Message-ID: <1542630104.24.0.788709270274.issue35277@psf.upfronthosting.co.za> New submission from Donald Stufft : Upgrade the bundled pip/setuptools to the latest version. Making this issue because blurb gets mad at me if I try to add a news entry without a bpo issue. ---------- assignee: dstufft components: Library (Lib) messages: 330096 nosy: dstufft priority: normal severity: normal status: open title: Upgrade bundled pip/setuptools versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 07:46:03 2018 From: report at bugs.python.org (Yusuke Endoh) Date: Mon, 19 Nov 2018 12:46:03 +0000 Subject: [New-bugs-announce] [issue35278] directory traversal in tempfile prefix Message-ID: <1542631563.19.0.788709270274.issue35278@psf.upfronthosting.co.za> New submission from Yusuke Endoh : Hello, The tempfile library does not check the prefix argument, which can be exploited to create files outside tmpdir by using directory traversal. ``` >>> import tempfile >>> tempfile.gettempprefix() 'tmp' >>> f = tempfile.NamedTemporaryFile(prefix="/home/mame/cracked") >>> f.name '/home/mame/crackedlt3y_ddm' ``` The same issue was found and treated as a vulnerability in PHP (CVE-2006-1494) and Ruby (CVE-2018-6914). I first reported this issue to security at python.org at July 2018. Some people kindly discussed it, and finally I was told to create a ticket here. ---------- components: Library (Lib) messages: 330097 nosy: Yusuke Endoh priority: normal severity: normal status: open title: directory traversal in tempfile prefix type: security versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 09:26:55 2018 From: report at bugs.python.org (=?utf-8?b?Vm9qdMSbY2ggQm/EjWVr?=) Date: Mon, 19 Nov 2018 14:26:55 +0000 Subject: [New-bugs-announce] [issue35279] asyncio uses too many threads by default Message-ID: <1542637615.61.0.788709270274.issue35279@psf.upfronthosting.co.za> New submission from Vojt?ch Bo?ek : By default, asyncio spawns as many as os.cpu_count() * 5 threads to run I/O on. When combined with beefy machines (e.g. kubernetes servers) with, says, 56 cores, it results in very high memory usage. This is amplified by the fact that the `concurrent.futures.ThreadPoolExecutor` threads are never killed, and are not re-used until `max_workers` threads are spawned. Workaround: loop.set_default_executor(concurrent.futures.ThreadPoolExecutor(max_workers=8)) This is still not ideal as the program might not need max_workers threads, but they are still spawned anyway. I've hit this issue when running asyncio program in kubernetes. It created 260 idle threads and then ran out of memory. I think the default max_workers should be limited to some max value and ThreadPoolExecutor should not spawn new threads unless necessary. ---------- components: asyncio messages: 330101 nosy: Vojt?ch Bo?ek, asvetlov, yselivanov priority: normal severity: normal status: open title: asyncio uses too many threads by default type: resource usage versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 10:20:30 2018 From: report at bugs.python.org (dingens) Date: Mon, 19 Nov 2018 15:20:30 +0000 Subject: [New-bugs-announce] [issue35280] Interactive shell overwrites history Message-ID: <1542640830.02.0.788709270274.issue35280@psf.upfronthosting.co.za> New submission from dingens : When opening two shells at the same time, history from the one that is closed first gets lost. Steps to reproduce: - let's call the contents of the history before this experiment `z` - open two interactive shells called A and B - execute commands in them, say `a` and `b`, respectively - close shell A - close shell B Actual behavior: - history contains `z` then `b` Expected behavior: - history contains `z` then `a` then `b` possibly related: issue22940 ---------- messages: 330103 nosy: dingens priority: normal severity: normal status: open title: Interactive shell overwrites history type: behavior versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Mon Nov 19 14:25:49 2018 From: report at bugs.python.org (Lihu) Date: Mon, 19 Nov 2018 19:25:49 +0000 Subject: [New-bugs-announce] [issue35281] Allow access to unittest.TestSuite tests Message-ID: <1542655549.43.0.788709270274.issue35281@psf.upfronthosting.co.za> New submission from Lihu : I would like to have access to the list of tests in a TestSuite object. Currently there are two ways of doing this: I can access the protected attribute `_tests`, or I can iterate over the TestSuite and re-create data that I already have. The former option isn't part of the public API and makes mypy mad, and the latter seems redundant. Could we get a public property or getter for the internal test list (or a more appropriate implementation if necessary)? ---------- components: Library (Lib) messages: 330111 nosy: lbenezriravin priority: normal severity: normal status: open title: Allow access to unittest.TestSuite tests type: enhancement versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 20 14:34:11 2018 From: report at bugs.python.org (Martin DeMello) Date: Tue, 20 Nov 2018 19:34:11 +0000 Subject: [New-bugs-announce] [issue35282] Add a return value to lib2to3.refactor.refactor_file and refactor_dir Message-ID: <1542742451.18.0.788709270274.issue35282@psf.upfronthosting.co.za> New submission from Martin DeMello : It would be useful for lib2to3.refactor to return which files were actually changed. Right now that information is logged but not returned. ---------- components: 2to3 (2.x to 3.x conversion tool) messages: 330149 nosy: martindemello priority: normal severity: normal status: open title: Add a return value to lib2to3.refactor.refactor_file and refactor_dir type: enhancement versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 20 17:16:29 2018 From: report at bugs.python.org (Dieter Maurer) Date: Tue, 20 Nov 2018 22:16:29 +0000 Subject: [New-bugs-announce] [issue35283] "threading._DummyThread" redefines "is_alive" but forgets "isAlive" Message-ID: <1542752189.25.0.788709270274.issue35283@psf.upfronthosting.co.za> New submission from Dieter Maurer : In module "threading", class "Thread" defines "is_alive" and defines "isAlive = is_alive". The derived class "_DummyThread" redefines "is_alive" but forgets to update the "isAlive" alias. As a consequence, calling "_DummyThread.isAlive" leads to an "AssertionErrror". The "isAlive" method is mentioned in the docstring of "Thread.join". ---------- components: Library (Lib) messages: 330158 nosy: dmaurer priority: normal severity: normal status: open title: "threading._DummyThread" redefines "is_alive" but forgets "isAlive" type: crash _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 20 17:38:01 2018 From: report at bugs.python.org (Zackery Spytz) Date: Tue, 20 Nov 2018 22:38:01 +0000 Subject: [New-bugs-announce] [issue35284] Incomplete error handling in the compiler's compiler_call() Message-ID: <1542753481.5.0.788709270274.issue35284@psf.upfronthosting.co.za> New submission from Zackery Spytz : compiler_call() needs to check if an error occurred during the maybe_optimize_method_call() call. ---------- components: Interpreter Core messages: 330159 nosy: ZackerySpytz priority: normal severity: normal status: open title: Incomplete error handling in the compiler's compiler_call() type: behavior versions: Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Tue Nov 20 17:46:27 2018 From: report at bugs.python.org (=?utf-8?b?SWduYXMgQnJhxaFpxaFraXM=?=) Date: Tue, 20 Nov 2018 22:46:27 +0000 Subject: [New-bugs-announce] [issue35285] Make Proactor api extensible for reasonably any file handle Message-ID: <1542753987.17.0.788709270274.issue35285@psf.upfronthosting.co.za> New submission from Ignas Bra?i?kis : There was old issue (https://bugs.python.org/issue30539) requesting extension ability at proactor loop for third party libraries. It was rejected as it was asking to broad functionality and not particular api. This issue purpose is to discuss more concrete api needed to bring proactor functionally at least on par with selector. While loop covers most valid cases it still has no ability to run any third party libraries with file descriptors. As a consequence there is workarounds with unofficial api and handle wrappers to expose and call private proactor methods (https://github.com/m-labs/asyncserial/blob/master/asyncserial/asyncserial.py#L172). Selector basically has the following api: loop.add_reader(fd, callback, *args) loop.remove_reader(fd) loop.add_writer(fd, callback, *args) loop.remove_writer(fd) Nature of selector makes it easy for user to provide those operations on descriptor: make read from when it is available or make write from somewhere when you can also cancel operation. Proactor on the other hand need to be told what and where to copy, and run callback when it's done (probably with error, also buffer passed to callback ???). It also needs to have ability to cancel operations. One possibility of such api could look like this: loop.start_read_operation(fd, buffer, done_callback, *args) where done_callback = function(buffer, bytes_read, error) loop.cancel_read_operation(fd, operation) loop.start_write_operation(fd, buffer, done_callback, *args) where done_callback = function(bytes_written, error) loop.cancel_write_operation(fd, operation) There probably can be a plenty refinement here to the naming buffer, or callback workings, but in general there should be start and cancel operations. This api might even be implemented for selectors (selectors might imitate behaviour with custom writer), thought probably it this usage might be discouraged in favour of add_writer/reader logic. As examples of this api besides IOCP (which is implementation in question here) there is linux aio (http://man7.org/linux/man-pages/man2/io_submit.2.html) api which uses similar proactor concepts. This api obviously can be wrapped to higher level api with futures, but this probably should remain low level functionality where it is responsibility of library implementer to do right thing. Usage is pretty limited here: mainly writing file descriptors and devices/ttys asynchronously on Win32 platform. Python itself is not exactly systems programming language, but it convenient for accessing some OS features. Sockets and pipes are covered by standard library. But still if could prevent plenty of hacks and give library writers and odd few users place to hang on. ---------- components: asyncio messages: 330160 nosy: Ignas Bra?i?kis, asvetlov, yselivanov priority: normal severity: normal status: open title: Make Proactor api extensible for reasonably any file handle type: enhancement _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 02:15:29 2018 From: report at bugs.python.org (Boris Yang) Date: Wed, 21 Nov 2018 07:15:29 +0000 Subject: [New-bugs-announce] [issue35289] wrong result for difflib.SequenceMatcher Message-ID: <1542784529.35.0.788709270274.issue35289@psf.upfronthosting.co.za> New submission from Boris Yang : How to repeat: # -*- coding: UTF-8 -*- from difflib import SequenceMatcher seqMatcher = SequenceMatcher(None, "????", "????") seqMatcher.get_matching_blocks() Expect Result: [Match(a=0, b=3, size=2), Match(a=2, b=0, size=2), Match(a=5, b=5, size=0)] Current Result: [Match(a=0, b=3, size=2), Match(a=5, b=5, size=0)] ---------- messages: 330175 nosy: Boris Yang priority: normal severity: normal status: open title: wrong result for difflib.SequenceMatcher type: behavior versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 05:49:52 2018 From: report at bugs.python.org (STINNER Victor) Date: Wed, 21 Nov 2018 10:49:52 +0000 Subject: [New-bugs-announce] [issue35290] [FreeBSD] test_c_locale_coercion: fsencoding is ASCII instead of UTF-8 on FreeBSD CURRENT buildbot Message-ID: <1542797392.41.0.788709270274.issue35290@psf.upfronthosting.co.za> New submission from STINNER Victor : It seems like the commit 02e6bf7f2025cddcbde6432f6b6396198ab313f4 (bpo-28604) broke test_c_locale_coercion on FreeBSD CURRENT Shared 3.x: https://buildbot.python.org/all/#/builders/168/builds/247 0:19:33 load avg: 7.69 [401/419/2] test_c_locale_coercion failed -- running: test_decimal (40 sec 626 ms), test_lib2to3 (2 min 3 sec) test_external_target_locale_configuration (test.test_c_locale_coercion.LocaleConfigurationTests) ... ok test_LC_ALL_set_to_C (test.test_c_locale_coercion.LocaleCoercionTests) ... ok test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... ok ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE=None) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE=None) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE=None) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE=None) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE=None) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='1') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='1') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='1') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='1') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='1') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='true') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='true') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='true') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='true') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='true') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='false') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='false') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='false') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='false') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='false') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='warn') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion self._check_child_encoding_details(base_var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='warn') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'c', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='warn') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='warn') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': 'invalid.ascii', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ====================================================================== FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='warn') ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion self._check_child_encoding_details(var_dict, File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details self.assertEqual(encoding_details, expected_details) AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} - {'fsencoding': 'ascii', ? ^^^^^ + {'fsencoding': 'utf-8', ? ^^^^^ 'lang': '', 'lc_all': '', 'lc_ctype': 'c.utf-8', 'stderr_info': 'utf-8:backslashreplace', 'stdin_info': 'utf-8:surrogateescape', 'stdout_info': 'utf-8:surrogateescape'} ---------------------------------------------------------------------- Ran 6 tests in 6.205s FAILED (failures=30) test test_c_locale_coercion failed ---------- components: Interpreter Core messages: 330189 nosy: vstinner priority: normal severity: normal status: open title: [FreeBSD] test_c_locale_coercion: fsencoding is ASCII instead of UTF-8 on FreeBSD CURRENT buildbot versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 10:12:32 2018 From: report at bugs.python.org (Jason Madden) Date: Wed, 21 Nov 2018 15:12:32 +0000 Subject: [New-bugs-announce] [issue35291] duplicate of memoryview from io.BufferedWriter leaks Message-ID: <1542813152.98.0.788709270274.issue35291@psf.upfronthosting.co.za> New submission from Jason Madden : Using Python 2.7.15, if a BufferedWriter wraps an IO object that duplicates the memoryview passed to the IO object's `write` method, that memoryview leaks. This script demonstrates the problem by leaking a memoryview for each iteration of the loop (if the `flush` is skipped, the leaks are less frequent but still occur): ``` from __future__ import print_function import io import gc def count_memoryview(): result = 0 for x in gc.get_objects(): if type(x) is memoryview: result += 1 return result class FileLike(object): closed = False def writable(self): return True def write(self, data): memoryview(data) # XXX: This causes the problem return len(data) bf = io.BufferedWriter(FileLike()) i = 0 memoryview_count = 0 while True: if i == 0 or i % 100 == 0: # This reports 100 new memoryview objects each time old = memoryview_count new = count_memoryview() print(i, "memoryview", new, "+%s" % (new - old)) memoryview_count = new bf.write(b"test") bf.flush() i += 1 ``` The leak can also be observed using the operating system's memory monitoring tools for the process (seen on both Fedora and macOS). Commenting out the line in `FileLike.write` that makes a duplicate memoryview of the given buffer solves the leak. Deleting the BufferedWriter doesn't appear to reclaim the leaked memoryviews. I can't duplicate this in Python 3.4 or above. Originally reported to gevent in https://github.com/gevent/gevent/issues/1318 Possibly related to Issue 26720 and Issue 15994 ---------- components: IO messages: 330206 nosy: jmadden priority: normal severity: normal status: open title: duplicate of memoryview from io.BufferedWriter leaks versions: Python 2.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 12:41:40 2018 From: report at bugs.python.org (Steve Dower) Date: Wed, 21 Nov 2018 17:41:40 +0000 Subject: [New-bugs-announce] [issue35292] Make SimpleHTTPRequestHandler load mimetypes lazily Message-ID: <1542822100.42.0.788709270274.issue35292@psf.upfronthosting.co.za> New submission from Steve Dower : Importing http.server triggers mimetypes.init(), which can be fairly expensive on Windows (and potentially other platforms?) due to having to enumerate a lot of registry keys. Instead, SimpleHTTPRequestHandler.extensions_map can be a dict with just its default values in lib/http/server.py and the first call to guess_type() can initialize mimetypes if necessary. We should check whether people read from SimpleHTTPRequestHandler.extensions_map directly before calling guess_type(), and decide how quickly we can make the change. My gut feeling is we will be okay to make this in the next release but probably shouldn't backport it. ---------- components: Library (Lib), Windows keywords: easy messages: 330212 nosy: paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal stage: needs patch status: open title: Make SimpleHTTPRequestHandler load mimetypes lazily type: performance versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 16:27:11 2018 From: report at bugs.python.org (STINNER Victor) Date: Wed, 21 Nov 2018 21:27:11 +0000 Subject: [New-bugs-announce] [issue35293] make doctest (Sphinx) emits a lot of warnings Message-ID: <1542835631.23.0.788709270274.issue35293@psf.upfronthosting.co.za> New submission from STINNER Victor : The following commands emits a lot of warnings: make PYTHON=../python SPHINXOPTS="-q -W -j4" -C Doc/ venv doctest # adapt PYTHON to point to Python 3.8 compiled in debug mmode Travis CI logs: https://travis-ci.org/python/cpython/jobs/458140641 make: Entering directory `/home/travis/build/python/cpython/Doc' ../python -m venv ./venv (...) The venv has been created in the ./venv directory make[1]: Entering directory `/home/travis/build/python/cpython/Doc' mkdir -p build Building NEWS from Misc/NEWS.d with blurb PATH=./venv/bin:$PATH sphinx-build -b doctest -d build/doctrees -D latex_elements.papersize= -q -W -j4 -W . build/doctest /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/babel/localedata.py:17: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import MutableMapping /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Mapping /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/jinja2/sandbox.py:82: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import MutableSet, MutableMapping, MutableSequence /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/sphinx/environment/__init__.py:340: RemovedInSphinx30Warning: env.versionchanges() is deprecated. Please use ChangeSetDomain instead. for version, changes in other.versionchanges.items(): /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/sphinx/environment/__init__.py:341: RemovedInSphinx30Warning: env.versionchanges() is deprecated. Please use ChangeSetDomain instead. self.versionchanges.setdefault(version, []).extend( /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/venv/lib/python3.8/site-packages/docutils/parsers/rst/states.py:2130: RemovedInSphinx40Warning: highlightlang directive is deprecated. Please use highlight directive instead. result = directive_instance.run() /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) /home/travis/build/python/cpython/Doc/tools/extensions/pyspecific.py:274: RemovedInSphinx30Warning: env.note_versionchange() is deprecated. Please use ChangeSetDomain.note_changeset() instead. env.note_versionchange('deprecated', version[0], node, self.lineno) :1: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly formatargspec(*getfullargspec(f)) :1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp obj dead or exiting /home/travis/build/python/cpython/Lib/socket.py:660: ResourceWarning: unclosed self._sock = None ResourceWarning: Enable tracemalloc to get the object allocation traceback make[1]: Leaving directory `/home/travis/build/python/cpython/Doc' make: Leaving directory `/home/travis/build/python/cpython/Doc' The command "if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then XVFB_RUN=xvfb-run; fi $XVFB_RUN make buildbottest TESTOPTS="-j4 -uall,-cpu" if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then $XVFB_RUN make PYTHON=../python SPHINXOPTS="-q -W -j4" -C Doc/ venv doctest fi " exited with 0. ---------- components: Build messages: 330217 nosy: vstinner priority: normal severity: normal status: open title: make doctest (Sphinx) emits a lot of warnings versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Wed Nov 21 20:02:32 2018 From: report at bugs.python.org (Ruslan Dautkhanov) Date: Thu, 22 Nov 2018 01:02:32 +0000 Subject: [New-bugs-announce] [issue35294] Race condition involving SocketServer.TCPServer Message-ID: <1542848552.12.0.788709270274.issue35294@psf.upfronthosting.co.za> New submission from Ruslan Dautkhanov : We had a long discussion in https://issues.apache.org/jira/browse/SPARK-26019 regarding a race condition that happens around https://github.com/apache/spark/blob/master/python/pyspark/accumulators.py#L289 The fix I created here - https://github.com/apache/spark/pull/23113/files basically changes bind_and_activate to False in SocketServer.TCPServer.__init__'s call and manually doing self.server_bind() self.server_activate() self.serve_forever() in that thread instead of main thread. We have a fix, but we're not sure if this is a bug in Python with multithreading / handling SocketServer.TCPServer ? Thank you. ---------- components: IO, Library (Lib) messages: 330228 nosy: Ruslan Dautkhanov priority: normal severity: normal status: open title: Race condition involving SocketServer.TCPServer versions: Python 2.7, Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 22 07:12:44 2018 From: report at bugs.python.org (Marcin Kowalczyk) Date: Thu, 22 Nov 2018 12:12:44 +0000 Subject: [New-bugs-announce] [issue35295] Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred Message-ID: <1542888764.29.0.788709270274.issue35295@psf.upfronthosting.co.za> New submission from Marcin Kowalczyk : The documentation is silent whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred. We are under the assumption that both are acceptable for the given caller, i.e. the caller wants to access just the sequence of UTF-8 code units (e.g. for calling a C++ function which takes std::string_view or std::string as a parameter), and the caller either will copy the UTF-8 code units immediately or is willing to own a temporary object to ensure a lifetime of the UTF-8 code units. File comments in unicodeobject.h about PyUnicode_AsUTF8AndSize() have a warning: *** This API is for interpreter INTERNAL USE ONLY and will likely *** be removed or changed in the future. *** If you need to access the Unicode object as UTF-8 bytes string, *** please use PyUnicode_AsUTF8String() instead. The discrepancy between these comments and the documentation should be fixed. Either the documentation is correct and the comment is outdated, or the comment is correct and the documentation is lacking guidance. It is not even clear which function is better technically: - PyUnicode_AsUTF8String() always allocates the string. PyUnicode_AsUTF8AndSize() does not allocate the string if the unicode object is ASCII-only (this is common) or if PyUnicode_AsUTF8AndSize() was already called before. - If conversion must be performed, then PyUnicode_AsUTF8String() makes a single allocation, while PyUnicode_AsUTF8AndSize() first calls PyUnicode_AsUTF8String() and then copies the string. - If the string is converted multiple times, then PyUnicode_AsUTF8AndSize() caches the result - faster. If the string is converted once, then the result persists as long as the string persists - wastes memory. I see the following possible resolutions: 1a. Declare both functions equally acceptable. Remove comments claiming that PyUnicode_AsUTF8AndSize() should be avoided. 1b. 1a, and change the implementation of PyUnicode_AsUTF8AndSize() to avoid allocating the string twice if it needs to be materialized, so that PyUnicode_AsUTF8AndSize() is never significantly slower than PyUnicode_AsUTF8String(). 2a. Declare PyUnicode_AsUTF8String() preferred. Indicate this in the documentation. 2b. 2a, and provide a public interface to check and access UTF-8 code units without allocating a new string in case this is possible (I think PyUnicode_READY() + PyUnicode_IS_ASCII() + PyUnicode_DATA() + PyUnicode_GET_LENGTH() would work, but they are not documented; or possibly also check if the string has a cached UTF-8 representation without populating that cached representation), so that a combination of the check with PyUnicode_AsUTF8String() is rarely or never significantly slower than PyUnicode_AsUTF8AndSize(). ---------- assignee: docs at python components: Documentation, Interpreter Core, Unicode messages: 330249 nosy: Marcin Kowalczyk, docs at python, ezio.melotti, vstinner priority: normal severity: normal status: open title: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_AsUTF8String() is preferred type: performance versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 22 12:33:21 2018 From: report at bugs.python.org (STINNER Victor) Date: Thu, 22 Nov 2018 17:33:21 +0000 Subject: [New-bugs-announce] [issue35296] Install Include/internal/ header files Message-ID: <1542908001.51.0.788709270274.issue35296@psf.upfronthosting.co.za> New submission from STINNER Victor : For some very specific use cases (ex: debuggers), it can be interesting to access the internal API of CPython. I propose to install the internal API: Include/internal/*.h header files. Attached PR implements this idea. ---------- components: Interpreter Core messages: 330271 nosy: eric.snow, nascheme, ncoghlan, vstinner priority: normal severity: normal status: open title: Install Include/internal/ header files versions: Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 22 16:25:02 2018 From: report at bugs.python.org (Zsolt Cserna) Date: Thu, 22 Nov 2018 21:25:02 +0000 Subject: [New-bugs-announce] [issue35297] untokenize documentation is not correct Message-ID: <1542921902.22.0.788709270274.issue35297@psf.upfronthosting.co.za> New submission from Zsolt Cserna : untokenize documentation (https://docs.python.org/3/library/tokenize.html#tokenize.untokenize) states the following: """ Converts tokens back into Python source code. The iterable must return sequences with at least two elements, the token type and the token string. Any additional sequence elements are ignored. """ This last sentence is clearly not true because here: https://github.com/python/cpython/blob/master/Lib/tokenize.py#L242 The code checks for the length of the input token there, and the code behaves differently, in terms of whitespace, when an iterator of 2-tuples are given and when there are more elements in the tuple. When there are more elements in the tuple, the function renders whitespaces as the same as they were present in the original source. So this code: tokenize.untokenize(tokenize.tokenize(source.readline)) And this: tokenize.untokenize([x[:2] for x in tokenize.tokenize(source.readline)]) Have different results. I don't know that it is a documentation issue or a bug in the module itself, so I created this bugreport to seek for assistance in this regard. ---------- assignee: docs at python components: Documentation messages: 330281 nosy: csernazs, docs at python priority: normal severity: normal status: open title: untokenize documentation is not correct versions: Python 3.7 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Thu Nov 22 19:14:54 2018 From: report at bugs.python.org (gilado) Date: Fri, 23 Nov 2018 00:14:54 +0000 Subject: [New-bugs-announce] [issue35298] Segfault in _PyObject_GenericGetAttrWithDict Message-ID: <1542932094.03.0.788709270274.issue35298@psf.upfronthosting.co.za> New submission from gilado : Running pulseeffects, it crashes occasionally. I have yet to find a cause, it just happens. I decided to run it under gdb to catch the crash. It worked for a few days before crashing. $ GDK_DPI_SCALE=0.6 gdb -args /usr/bin/python3 /usr/bin/pulseeffects gdb: Symbol `acs_map' has different size in shared object, consider re-linking GNU gdb (GDB) 7.8-1pclos2014 (PCLinuxOS release 2014) ... ... 13:57:26.125 - PulseEffects - CRITICAL - PM - context operation failed! 13:57:29.779 - PulseEffects - CRITICAL - PM - context operation failed! 13:57:30.391 - PulseEffects - CRITICAL - PM - context operation failed! [New LWP 27696] Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 27696] 0x00007ffff73bcae6 in ?? () from /lib64/libc.so.6 (gdb) bt #0 0x00007ffff73bcae6 in () at /lib64/libc.so.6 #1 0x00007ffff29f8e08 in () at /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so #2 0x00007ffff79baf1e in _PyObject_GenericGetAttrWithDict () at /usr/lib64/libpython3.6m.so.1.0 #3 0x00007ffff7a2b6b0 in _PyEval_EvalFrameDefault () at /usr/lib64/libpython3.6m.so.1.0 #4 0x00007ffff7a271df in () at /usr/lib64/libpython3.6m.so.1.0 #5 0x00007ffff7a2fce0 in _PyFunction_FastCallDict () at /usr/lib64/libpython3.6m.so.1.0 #6 0x00007ffff79719fe in _PyObject_FastCallDict () at /usr/lib64/libpython3.6m.so.1.0 #7 0x00007ffff7971c0d in _PyObject_Call_Prepend () at /usr/lib64/libpython3.6m.so.1.0 #8 0x00007ffff79717c8 in PyObject_Call () at /usr/lib64/libpython3.6m.so.1.0 #9 0x00007ffff29f433d in () at /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so #10 0x00007ffff3f73dcf in ffi_closure_unix64_inner () at /usr/lib64/libffi.so.6 #11 0x00007ffff3f74148 in ffi_closure_unix64 () at /usr/lib64/libffi.so.6 #12 0x00007fffde2abaf5 in () at /usr/lib64/libpulse.so.0 #13 0x00007fffddf27ce1 in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #14 0x00007fffddf28033 in pa_pdispatch_run () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #15 0x00007fffde2aaa9e in () at /usr/lib64/libpulse.so.0 #16 0x00007fffddf2ab4f in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #17 0x00007fffddf2d5db in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #18 0x00007fffddf2d979 in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #19 0x00007fffddf2e1df in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #20 0x00007fffde2bea58 in pa_mainloop_dispatch () at /usr/lib64/libpulse.so.0 #21 0x00007fffde2bee1e in pa_mainloop_iterate () at /usr/lib64/libpulse.so.0 #22 0x00007fffde2beea0 in pa_mainloop_run () at /usr/lib64/libpulse.so.0 #23 0x00007fffde2cc786 in () at /usr/lib64/libpulse.so.0 #24 0x00007fffddf3db48 in () at /usr/lib64/pulseaudio/libpulsecommon-12.2.so #25 0x00007ffff76ed62e in () at /lib64/libpthread.so.0 #26 0x00007ffff7421def in clone () at /lib64/libc.so.6 (gdb) Seems to be a python bug not validating a data pointer. A similar bug was fixed here https://bugs.python.org/issue12149 reviewing the code https://github.com/python/cpython/blob/master/Objects/object.c I think line 981 mentioned in the bug description is now line 1235 f = descr->ob_type->tp_descr_get; Which now is protected inside an if (descr != NULL). So they fixed it here. Further just below that both 'f' and 'descr' are used like this if (f != NULL && PyDescr_IsData(descr)) { res = f(descr, obj, (PyObject *)obj->ob_type); The problem now seem to be further below (line 1279) where the code is if (f != NULL) { res = f(descr, obj, (PyObject *)Py_TYPE(obj)); The problem is that they forget to check that descr is valid. ---------- components: Interpreter Core messages: 330289 nosy: gilado priority: normal severity: normal status: open title: Segfault in _PyObject_GenericGetAttrWithDict type: crash versions: Python 3.6 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 23 01:31:25 2018 From: report at bugs.python.org (Jorge Ramos) Date: Fri, 23 Nov 2018 06:31:25 +0000 Subject: [New-bugs-announce] [issue35299] LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found Message-ID: <1542954685.4.0.788709270274.issue35299@psf.upfronthosting.co.za> New submission from Jorge Ramos : Build of Python 3.6.7 fails with error "LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found" I am using Windows 10 64bit. Steps to reproduce this error: 1) Install Visual Studio 2017 (With the python development and native dev tools) (I have also installed NVIDIA CUDA Toolkit but I don't think this matters) 2) Download source code from Python to folder E:\RepoGit\3.6 (this is the download folder) 3) Open a command prompt, navigate to "E:\RepoGit" and run "PCBuild\build.bat -p x64" 4) Close the command prompt 5) Copy file pyconfig.h file from E:\RepoGit\3.6\PC and paste it to E:\RepoGit\3.6\include. Also, create an environment variable called "LIB" that contains the folder "E:\RepoGiT\3.6\PCbuild\amd64" (this folder was created at step 3) 6) Open a command prompt and run "Tools\msi\buildrelease.bat -x64" NOTE1: if step 5 is skipped, the overall build does not fail, but the PGInstrument x64 Interpreter fails in distutils with error "E:\RepoGiT\3.6\include\Python.h(8): fatal error C1083: Cannot open include file: 'pyconfig.h': No such file or directory", as shown in message https://bugs.python.org/msg329247 of issue https://bugs.python.org/issue35157 NOTE2: if pyconfig.h in step 5 is copied but the environment variable is not created, the PGInstrument x64 Interpreter fails in distutils with error "LINK : fatal error LNK1104: cannot open file 'python36.lib'", and the overall build fails with the same error shown in the title of this Issue. NOTE3: If all the steps are run, the PGInstrument x64 Interpreter succeeds in its tests but the overall build fails with the error shown in the title of this issue. All the verbose of this compilation is included in the file attached. ---------- components: Build, Windows files: LIBenRepogiT.txt messages: 330299 nosy: neyuru, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found type: compile error versions: Python 3.6 Added file: https://bugs.python.org/file47941/LIBenRepogiT.txt _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 23 06:50:20 2018 From: report at bugs.python.org (bolorsociedad) Date: Fri, 23 Nov 2018 11:50:20 +0000 Subject: [New-bugs-announce] [issue35300] Bug with memoization and mutable objects Message-ID: <1542973820.71.0.788709270274.issue35300@psf.upfronthosting.co.za> Change by bolorsociedad : ---------- components: Library (Lib) nosy: bolorsociedad priority: normal severity: normal status: open title: Bug with memoization and mutable objects type: behavior versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 23 09:21:13 2018 From: report at bugs.python.org (Jonathan) Date: Fri, 23 Nov 2018 14:21:13 +0000 Subject: [New-bugs-announce] [issue35301] python.exe crashes - lzma? Message-ID: <1542982873.09.0.788709270274.issue35301@psf.upfronthosting.co.za> New submission from Jonathan : Python 3.6.3 on Windows 7 64 bit. I keep getting intermittent crashes of Python.exe with my project. The error is below and as best I can see seems to be indicating the issue with the LZMA module. All I'm doing is using it to compress web-pages (typically XML ones). Problem is, I can't replicate it. When I try and run the process again, the page that previously crashed it now works. I've had this happen three times so far (out of about 150,000 pages downloaded and compressed). My use of LZMA is quite simply: from lzma import compress compress(page_content, preset=5) The pages it has crashed on so far (but which work fine now): http://www.geoportal.rlp.de/mapbender/php/wms.php?inspire=1&layer_id=35833&request=GetCapabilities&service=WMS&version=1.3.0 https://maps.runnymede.gov.uk/geoserver/planning/lp_ancient_woodland/wms?request=GetCapabilities&service=WMS&version=1.3.0 http://serviziogc.regione.fvg.it/geoserver/GEOLOGIA/wms?request=GetCapabilities&service=WMS&version=1.3.0 I took a mini dump (attached) and full dump using Process Explorer. The full dump is 100MB compressed (1GB uncompressed). ---- Problem signature: Problem Event Name: APPCRASH Application Name: python.exe Application Version: 3.6.3150.1013 Application Timestamp: 59d3d3a3 Fault Module Name: _lzma.pyd Fault Module Version: 3.6.3150.1013 Fault Module Timestamp: 59d3d343 Exception Code: c0000005 Exception Offset: 000000000002e368 OS Version: 6.1.7601.2.1.0.768.3 Locale ID: 2057 Additional Information 1: d00b Additional Information 2: d00be1cd6ce69f4b081e66c649a14f90 Additional Information 3: 8c17 Additional Information 4: 8c17b9c4b6a39b7e31bf71b3b1374f1b ---------- files: python-mini.7z messages: 330327 nosy: jonathan-lp priority: normal severity: normal status: open title: python.exe crashes - lzma? type: crash versions: Python 3.6 Added file: https://bugs.python.org/file47944/python-mini.7z _______________________________________ Python tracker _______________________________________ From report at bugs.python.org Fri Nov 23 13:30:53 2018 From: report at bugs.python.org (Neil Booth) Date: Fri, 23 Nov 2018 18:30:53 +0000 Subject: [New-bugs-announce] [issue35302] create_connection with local_addr misses valid socket bindings Message-ID: <1542997853.2.0.788709270274.issue35302@psf.upfronthosting.co.za> New submission from Neil Booth : I run a machine with IPv4 and IPv6 interfaces on MacOSX Mojave. I try to loop.create_connection() to a remote machine whose domain resolves to an IPv6 address only, with a local_addr domain name argument that resolves to two local addresses: an IPv4 one first and an IPv6 one second. The loop https://github.com/python/cpython/blob/master/Lib/asyncio/base_events.py#L927-L943 that loops through the local addresses, IPv4 then IPv6, successfully binds the IPv4 laddr_info to the IPv6 socket successfully (something I find surprising) and then the connection attempt fails with OSError(65, 'No route to host'), at which point the sockets is closed and the IPv6 laddr_info is never tried and the connection attempt fails. If I reverse the order of the loop so that the IPv6 laddr_info is tried first then the connection succeeds. I suggest either all laddr_info bindings should be tried for each outer loop of infos, rather than just 1, or that those of a different socket "type" (IPv4 vs IPv6) should be skipped in the inner loop before attempting a binding. ---------- components: asyncio messages: 330354 nosy: asvetlov, kyuupichan, yselivanov priority: normal severity: normal status: open title: create_connection with local_addr misses valid socket bindings type: behavior versions: Python 3.5, Python 3.6, Python 3.7 _______________________________________ Python tracker