Mailman 3 November 2018 - Python-Dev

Summary of Python tracker Issues
by Python tracker Nov. 30, 2018

Nov. 30, 2018

ACTIVITY SUMMARY (2018-11-23 - 2018-11-30) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6880 (+15) closed 40234 (+47) total 47114 (+62) Open issues with patches: 2736 Issues opened (41) ================== #33015: Fix function cast warning in thread_pthread.h https://bugs.python.org/issue33015 reopened by vstinner #35302: create_connection with local_addr misses valid socket bindings https://bugs.python.org/issue35302 opened by kyuupichan #35305: subprocess.Popen(['/sbin/ldconfig', '-p'], stdin=PIPE) itself https://bugs.python.org/issue35305 opened by Henrik Bengtsson #35306: OSError [WinError 123] when testing if pathlib.Path('*') (aste https://bugs.python.org/issue35306 opened by jimbo1qaz_ #35307: Command line help example is missing "--prompt" option https://bugs.python.org/issue35307 opened by daftwullie #35310: select which was interrupted by EINTR isn't re-run if the time https://bugs.python.org/issue35310 opened by Brian Maissy #35311: exception unpickling error causes `multiprocessing.Pool` to ha https://bugs.python.org/issue35311 opened by Anthony Sottile #35312: lib2to3.pgen2.parse.ParseError is not roundtrip pickleable https://bugs.python.org/issue35312 opened by Anthony Sottile #35316: test_eintr fails randomly on macOS https://bugs.python.org/issue35316 opened by vstinner #35318: Check accuracy of str() doc string for its encoding argument https://bugs.python.org/issue35318 opened by rhettinger #35319: pkgutil.get_data() is a wrapper for a deprecated class https://bugs.python.org/issue35319 opened by Kevin.Norris #35321: None _frozen_importlib.__spec__.origin attribute https://bugs.python.org/issue35321 opened by maggyero #35324: ssl: FileNotFoundError when do handshake https://bugs.python.org/issue35324 opened by josephshi(a)yahoo.com #35325: imp.find_module() return value documentation discrepancy https://bugs.python.org/issue35325 opened by StefanBauerTT #35327: Using skipTest with subTest gives a misleading UI. https://bugs.python.org/issue35327 opened by p-ganssle #35328: Set a environment variable for venv prompt https://bugs.python.org/issue35328 opened by batisteo #35329: Documentation - capitalization issue https://bugs.python.org/issue35329 opened by Strijker #35330: When using mock to wrap an existing object, side_effect requir https://bugs.python.org/issue35330 opened by noamraph #35331: Incorrect __module__ attribute for _struct.Struct and perhaps https://bugs.python.org/issue35331 opened by bup #35332: shutil.rmtree(..., ignore_errors=True) doesn't ignore errors f https://bugs.python.org/issue35332 opened by rabraham #35333: Rename private structs to use names closer to types https://bugs.python.org/issue35333 opened by vstinner #35335: msgfmt should be able to merge more than one po file https://bugs.python.org/issue35335 opened by s-ball #35337: Check index in PyTuple_GET_ITEM/PyTuple_SET_ITEM in debug mode https://bugs.python.org/issue35337 opened by vstinner #35338: set union/intersection/difference could accept zero arguments https://bugs.python.org/issue35338 opened by carandraug #35341: Add generic version of OrderedDict to typing module https://bugs.python.org/issue35341 opened by itoijala #35342: email "default" policy raises exception iterating over unparse https://bugs.python.org/issue35342 opened by rptb1 #35344: platform: get macOS version rather than darwin version? https://bugs.python.org/issue35344 opened by vstinner #35346: Modernize Lib/platform.py code https://bugs.python.org/issue35346 opened by vstinner #35348: Problems with handling the file command output in platform.arc https://bugs.python.org/issue35348 opened by serhiy.storchaka #35350: importing "ctypes" immediately causes a segmentation fault https://bugs.python.org/issue35350 opened by n0s69z #35351: LTOFLAGS are passed to BASECFLAGS when using LTO https://bugs.python.org/issue35351 opened by cstratak #35352: test_asyncio fails on RHEL8 https://bugs.python.org/issue35352 opened by cstratak #35353: Add frame command to pdb https://bugs.python.org/issue35353 opened by cnklein #35354: Generator functions stack overflow https://bugs.python.org/issue35354 opened by asdwqii #35357: unittest.mock.call can't represent calls to a method called 'p https://bugs.python.org/issue35357 opened by cjw296 #35358: avoid '-' in importlib.import_module and builtins.__import__ https://bugs.python.org/issue35358 opened by matrixise #35359: [2.7][Windows] Define _CRT_SECURE_NO_WARNINGS to build Modules https://bugs.python.org/issue35359 opened by vstinner #35360: [Windows] Update SQLite dependency https://bugs.python.org/issue35360 opened by vstinner #35361: Update libffi dependency to 3.2.1? https://bugs.python.org/issue35361 opened by vstinner #35362: list inheritor with abstract content does not raise https://bugs.python.org/issue35362 opened by vaslem #35363: test_eintr: test_open() hangs randomly on x86-64 El Capitan 3. https://bugs.python.org/issue35363 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #35362: list inheritor with abstract content does not raise https://bugs.python.org/issue35362 #35361: Update libffi dependency to 3.2.1? https://bugs.python.org/issue35361 #35359: [2.7][Windows] Define _CRT_SECURE_NO_WARNINGS to build Modules https://bugs.python.org/issue35359 #35357: unittest.mock.call can't represent calls to a method called 'p https://bugs.python.org/issue35357 #35353: Add frame command to pdb https://bugs.python.org/issue35353 #35351: LTOFLAGS are passed to BASECFLAGS when using LTO https://bugs.python.org/issue35351 #35332: shutil.rmtree(..., ignore_errors=True) doesn't ignore errors f https://bugs.python.org/issue35332 #35331: Incorrect __module__ attribute for _struct.Struct and perhaps https://bugs.python.org/issue35331 #35329: Documentation - capitalization issue https://bugs.python.org/issue35329 #35324: ssl: FileNotFoundError when do handshake https://bugs.python.org/issue35324 #35302: create_connection with local_addr misses valid socket bindings https://bugs.python.org/issue35302 #35298: Segfault in _PyObject_GenericGetAttrWithDict https://bugs.python.org/issue35298 #35297: untokenize documentation is not correct https://bugs.python.org/issue35297 #35295: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_ https://bugs.python.org/issue35295 #35294: Race condition involving SocketServer.TCPServer https://bugs.python.org/issue35294 Most recent 15 issues waiting for review (15) ============================================= #35359: [2.7][Windows] Define _CRT_SECURE_NO_WARNINGS to build Modules https://bugs.python.org/issue35359 #35352: test_asyncio fails on RHEL8 https://bugs.python.org/issue35352 #35351: LTOFLAGS are passed to BASECFLAGS when using LTO https://bugs.python.org/issue35351 #35346: Modernize Lib/platform.py code https://bugs.python.org/issue35346 #35344: platform: get macOS version rather than darwin version? https://bugs.python.org/issue35344 #35337: Check index in PyTuple_GET_ITEM/PyTuple_SET_ITEM in debug mode https://bugs.python.org/issue35337 #35328: Set a environment variable for venv prompt https://bugs.python.org/issue35328 #35316: test_eintr fails randomly on macOS https://bugs.python.org/issue35316 #35312: lib2to3.pgen2.parse.ParseError is not roundtrip pickleable https://bugs.python.org/issue35312 #35310: select which was interrupted by EINTR isn't re-run if the time https://bugs.python.org/issue35310 #35307: Command line help example is missing "--prompt" option https://bugs.python.org/issue35307 #35284: Incomplete error handling in Python/compile.c:compiler_call() https://bugs.python.org/issue35284 #35283: "threading._DummyThread" redefines "is_alive" but forgets "isA https://bugs.python.org/issue35283 #35282: Add a return value to lib2to3.refactor.refactor_file and refac https://bugs.python.org/issue35282 #35278: [security] directory traversal in tempfile prefix https://bugs.python.org/issue35278 Top 10 most discussed issues (10) ================================= #35134: Add a new Include/cpython/ subdirectory for the "CPython API" https://bugs.python.org/issue35134 11 msgs #33930: Segfault with deep recursion into object().__dir__ https://bugs.python.org/issue33930 8 msgs #35338: set union/intersection/difference could accept zero arguments https://bugs.python.org/issue35338 8 msgs #35352: test_asyncio fails on RHEL8 https://bugs.python.org/issue35352 8 msgs #33015: Fix function cast warning in thread_pthread.h https://bugs.python.org/issue33015 7 msgs #35316: test_eintr fails randomly on macOS https://bugs.python.org/issue35316 7 msgs #35346: Modernize Lib/platform.py code https://bugs.python.org/issue35346 7 msgs #35344: platform: get macOS version rather than darwin version? https://bugs.python.org/issue35344 6 msgs #35327: Using skipTest with subTest gives a misleading UI. https://bugs.python.org/issue35327 5 msgs #35337: Check index in PyTuple_GET_ITEM/PyTuple_SET_ITEM in debug mode https://bugs.python.org/issue35337 5 msgs Issues closed (46) ================== #27903: Avoid ResourceWarnings from platform._dist_try_harder https://bugs.python.org/issue27903 closed by xtreak #29877: compileall hangs when accessing urandom even if number of work https://bugs.python.org/issue29877 closed by inada.naoki #30167: site.main() does not work on Python 3.6 and superior if PYTHON https://bugs.python.org/issue30167 closed by inada.naoki #31241: ast col_offset wrong for list comprehensions, generators and t https://bugs.python.org/issue31241 closed by serhiy.storchaka #31628: test_emails failure on FreeBSD https://bugs.python.org/issue31628 closed by vstinner #31900: localeconv() should decode numeric fields from LC_NUMERIC enco https://bugs.python.org/issue31900 closed by vstinner #32035: Documentation of zipfile.ZipFile().writestr() fails to mention https://bugs.python.org/issue32035 closed by serhiy.storchaka #33012: Invalid function cast warnings with gcc 8 for METH_NOARGS https://bugs.python.org/issue33012 closed by serhiy.storchaka #33029: Invalid function cast warnings with gcc 8 for getter and sette https://bugs.python.org/issue33029 closed by serhiy.storchaka #33954: float.__format__('n') fails with _PyUnicode_CheckConsistency a https://bugs.python.org/issue33954 closed by vstinner #34021: [2.7] test_regrtest: test_env_changed() fails on x86 Windows X https://bugs.python.org/issue34021 closed by vstinner #34022: 6 tests fail using SOURCE_DATE_EPOCH env var https://bugs.python.org/issue34022 closed by vstinner #34100: Same constants in tuples are not merged while compile() https://bugs.python.org/issue34100 closed by inada.naoki #34212: Cygwin link failure with builtin modules since issue30860 https://bugs.python.org/issue34212 closed by benjamin.peterson #34279: RFC: issue a warning in regrtest when no tests have been execu https://bugs.python.org/issue34279 closed by vstinner #34812: [Security] support.args_from_interpreter_flags() doesn't inher https://bugs.python.org/issue34812 closed by vstinner #34893: Add 2to3 fixer to change send and recv methods of socket objec https://bugs.python.org/issue34893 closed by benjamin.peterson #34978: check type of object in fix_dict.py in 2to3 https://bugs.python.org/issue34978 closed by benjamin.peterson #35189: PEP 475: fnctl functions are not retried if interrupted by a s https://bugs.python.org/issue35189 closed by vstinner #35217: REPL history is broken when python is invoked with cmd /c https://bugs.python.org/issue35217 closed by eryksun #35240: Travis CI: xvfb-run: error: Xvfb failed to start https://bugs.python.org/issue35240 closed by vstinner #35255: delete "How do I extract the downloaded documentation" section https://bugs.python.org/issue35255 closed by mdk #35281: Allow access to unittest.TestSuite tests https://bugs.python.org/issue35281 closed by brett.cannon #35300: Document what functions are suitable for use with the lru_cach https://bugs.python.org/issue35300 closed by rhettinger #35303: A reference leak in _operator.c's methodcaller_repr() https://bugs.python.org/issue35303 closed by serhiy.storchaka #35304: The return of truncate(size=None) (file io) is unexpected https://bugs.python.org/issue35304 closed by martin.panter #35308: webbrowser regression: BROWSER env var not respected https://bugs.python.org/issue35308 closed by serhiy.storchaka #35309: Typo in urllib.request warning: cpath → capath https://bugs.python.org/issue35309 closed by benjamin.peterson #35313: test_embed fails in travis CI when tests are executed from a v https://bugs.python.org/issue35313 closed by vstinner #35314: fnmatch failed with leading caret (^) https://bugs.python.org/issue35314 closed by serhiy.storchaka #35315: Error (Segmentation fault - core dumped) while installing Pyth https://bugs.python.org/issue35315 closed by christian.heimes #35317: test_email: test_localtime_daylight_false_dst_true() fails dep https://bugs.python.org/issue35317 closed by vstinner #35320: Writable __spec__.has_location attribute https://bugs.python.org/issue35320 closed by brett.cannon #35322: test_datetime leaks memory on Windows https://bugs.python.org/issue35322 closed by vstinner #35323: Windows x86 executable installer can't install https://bugs.python.org/issue35323 closed by outofthink #35326: ctypes cast and from_address cause crash on Windows 10 https://bugs.python.org/issue35326 closed by eryksun #35334: urllib3 fails with type error exception, when cannot reach PyP https://bugs.python.org/issue35334 closed by christian.heimes #35336: Bug in C locale coercion with PYTHONCOERCECLOCALE=1 https://bugs.python.org/issue35336 closed by vstinner #35339: Populating instances of class automatically https://bugs.python.org/issue35339 closed by steven.daprano #35340: global symbol "freegrammar" should be made static ore renamed https://bugs.python.org/issue35340 closed by benjamin.peterson #35343: Importlib.reload has different behaviour in script and interac https://bugs.python.org/issue35343 closed by brett.cannon #35345: Remove platform.popen(), deprecated since Python 3.3 https://bugs.python.org/issue35345 closed by vstinner #35347: test_socket: NonBlockingTCPTests.testRecv() uses a weak 100 ms https://bugs.python.org/issue35347 closed by vstinner #35349: collections.abc.Sequence isn't an abstract base class for arra https://bugs.python.org/issue35349 closed by serhiy.storchaka #35355: Some Errors involving formatted strings aren't reported in the https://bugs.python.org/issue35355 closed by terry.reedy #35356: A possible reference leak in the nis module https://bugs.python.org/issue35356 closed by serhiy.storchaka

1 0

Ping on PR #8712
by E. Madison Bray Nov. 26, 2018

Nov. 26, 2018

Hi folks, I've had a PR open for nearly 3 months now with no review at: https://github.com/python/cpython/pull/8712 I know everyone is overextended so normally I wouldn't fuss about it. But I would still like to remain committed to providing better Cygwin (and to a lesser extent, personally, MinGW) support in CPython. I have had a buildbot chugging along rather uselessly due to the blocker issue that the above PR fixes: https://buildbot.python.org/all/#/builders/164 Only when the above issue is fixed will it be possible to get some semi-useful builds and test runs on this buildbot. The issue that is fixed is really a general bug, it just happens to only affect builds on those platforms that implement a POSIX layer on top of Windows. Specifically, modules that are built into the libpython DLL are not linked properly. This is a regression that was introduced by https://bugs.python.org/issue30860 The fix I've proposed is simple and undisruptive on unaffected platforms. Thanks for having a look!

1 0

Usage of tafile.copyfileobj
by Rémi Lapeyre Nov. 24, 2018

Nov. 24, 2018

Hi, I’m working on the tarfile module to add support for file objects whose size is not know beforehand (https://bugs.python.org/issue35227). In doing so, I need to adapt `tarfile.copyfileobj` to return the length of the file after it has been copied. Calling this function with `length=None` currently leads to data being copied but without adding the necessary padding. This seems weird to me, I do not understand why this would be needed and this behaviour is currently not used. This function is not documented in Python documentation so nobody is probably using it. Can I safely change `tarfile.copyfileobj` to make it write the padding when `length=None`? Thanks, Rémi

1 0

C API changes
by Stefan Krah Nov. 23, 2018

Nov. 23, 2018

Armin Rigo wrote: > The C API would change a lot, so it's not reasonable to do that in the > CPython repo. But it could be a third-party project, attempting to > define an API like this and implement it well on top of both CPython > and PyPy. IMHO this might be a better idea than just changing the API > of functions defined long ago to make them more regular (e.g. stop > returning borrowed references); by now this would mostly mean creating > more work for the PyPy team to track and adapt to the changes, with no > real benefits. I like this idea. For example, when writing two versions of a C module, one that uses CPython internals indiscriminately and another that uses a "clean" API, such a third-party project would help. I'd also be more motivated to write two versions if I know that the project is supported by PyPy devs. Do you think that such an API might be faster than CFFI on PyPy? Stefan Krah

1 0

Summary of Python tracker Issues
by Python tracker Nov. 23, 2018

Nov. 23, 2018

ACTIVITY SUMMARY (2018-11-16 - 2018-11-23) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6865 ( +2) closed 40187 (+33) total 47052 (+35) Open issues with patches: 2735 Issues opened (27) ================== #35148: cannot activate a venv environment on a Swiss German windows https://bugs.python.org/issue35148 reopened by vinay.sajip #35267: reproducible deadlock with multiprocessing.Pool https://bugs.python.org/issue35267 opened by dzhu #35268: Windows asyncio reading continously stdin and stdout Stockfish https://bugs.python.org/issue35268 opened by Cezary.Wagner #35270: Cmd.complete does not handle cmd=None https://bugs.python.org/issue35270 opened by blueyed #35271: venv creates pyvenv.cfg with wrong home https://bugs.python.org/issue35271 opened by wvxvw #35272: sqlite3 get the connected database url https://bugs.python.org/issue35272 opened by midnio #35276: Document thread safety https://bugs.python.org/issue35276 opened by vstinner #35277: Upgrade bundled pip/setuptools https://bugs.python.org/issue35277 opened by dstufft #35278: [security] directory traversal in tempfile prefix https://bugs.python.org/issue35278 opened by Yusuke Endoh #35279: asyncio uses too many threads by default https://bugs.python.org/issue35279 opened by Vojtěch Boček #35280: Interactive shell overwrites history https://bugs.python.org/issue35280 opened by dingens #35281: Allow access to unittest.TestSuite tests https://bugs.python.org/issue35281 opened by lbenezriravin #35282: Add a return value to lib2to3.refactor.refactor_file and refac https://bugs.python.org/issue35282 opened by martindemello #35283: "threading._DummyThread" redefines "is_alive" but forgets "isA https://bugs.python.org/issue35283 opened by dmaurer #35284: Incomplete error handling in Python/compile.c:compiler_call() https://bugs.python.org/issue35284 opened by ZackerySpytz #35285: Make Proactor api extensible for reasonably any file handle https://bugs.python.org/issue35285 opened by Ignas Brašiškis #35286: wrong result for difflib.SequenceMatcher https://bugs.python.org/issue35286 opened by Boris Yang #35291: duplicate of memoryview from io.BufferedWriter leaks https://bugs.python.org/issue35291 opened by jmadden #35292: Make SimpleHTTPRequestHandler load mimetypes lazily https://bugs.python.org/issue35292 opened by steve.dower #35293: make doctest (Sphinx) emits a lot of warnings https://bugs.python.org/issue35293 opened by vstinner #35294: Race condition involving SocketServer.TCPServer https://bugs.python.org/issue35294 opened by Ruslan Dautkhanov #35295: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_ https://bugs.python.org/issue35295 opened by Marcin Kowalczyk #35297: untokenize documentation is not correct https://bugs.python.org/issue35297 opened by csernazs #35298: Segfault in _PyObject_GenericGetAttrWithDict https://bugs.python.org/issue35298 opened by gilado #35299: LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found https://bugs.python.org/issue35299 opened by neyuru #35300: Bug with memoization and mutable objects https://bugs.python.org/issue35300 opened by bolorsociedad #35301: python.exe crashes - lzma? https://bugs.python.org/issue35301 opened by jonathan-lp Most recent 15 issues with no replies (15) ========================================== #35301: python.exe crashes - lzma? https://bugs.python.org/issue35301 #35299: LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found https://bugs.python.org/issue35299 #35298: Segfault in _PyObject_GenericGetAttrWithDict https://bugs.python.org/issue35298 #35297: untokenize documentation is not correct https://bugs.python.org/issue35297 #35295: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_ https://bugs.python.org/issue35295 #35294: Race condition involving SocketServer.TCPServer https://bugs.python.org/issue35294 #35291: duplicate of memoryview from io.BufferedWriter leaks https://bugs.python.org/issue35291 #35285: Make Proactor api extensible for reasonably any file handle https://bugs.python.org/issue35285 #35284: Incomplete error handling in Python/compile.c:compiler_call() https://bugs.python.org/issue35284 #35282: Add a return value to lib2to3.refactor.refactor_file and refac https://bugs.python.org/issue35282 #35280: Interactive shell overwrites history https://bugs.python.org/issue35280 #35279: asyncio uses too many threads by default https://bugs.python.org/issue35279 #35270: Cmd.complete does not handle cmd=None https://bugs.python.org/issue35270 #35264: SSL Module build fails with OpenSSL 1.1.0 for Python 2.7 https://bugs.python.org/issue35264 #35263: Add None handling for get_saved() in IDLE https://bugs.python.org/issue35263 Most recent 15 issues waiting for review (15) ============================================= #35284: Incomplete error handling in Python/compile.c:compiler_call() https://bugs.python.org/issue35284 #35283: "threading._DummyThread" redefines "is_alive" but forgets "isA https://bugs.python.org/issue35283 #35282: Add a return value to lib2to3.refactor.refactor_file and refac https://bugs.python.org/issue35282 #35278: [security] directory traversal in tempfile prefix https://bugs.python.org/issue35278 #35277: Upgrade bundled pip/setuptools https://bugs.python.org/issue35277 #35270: Cmd.complete does not handle cmd=None https://bugs.python.org/issue35270 #35266: Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpret https://bugs.python.org/issue35266 #35264: SSL Module build fails with OpenSSL 1.1.0 for Python 2.7 https://bugs.python.org/issue35264 #35263: Add None handling for get_saved() in IDLE https://bugs.python.org/issue35263 #35259: Py_FinalizeEx unconditionally exists in Py_LIMITED_API https://bugs.python.org/issue35259 #35255: delete "How do I extract the downloaded documentation" section https://bugs.python.org/issue35255 #35252: test_functools dead code after FIXME https://bugs.python.org/issue35252 #35250: Minor parameter documentation mismatch for turtle https://bugs.python.org/issue35250 #35236: urllib.request.urlopen throws on some valid FTP files https://bugs.python.org/issue35236 #35226: mock.call equality surprisingly broken https://bugs.python.org/issue35226 Top 10 most discussed issues (10) ================================= #35134: Add a new Include/cpython/ subdirectory for the "CPython API" https://bugs.python.org/issue35134 14 msgs #35283: "threading._DummyThread" redefines "is_alive" but forgets "isA https://bugs.python.org/issue35283 13 msgs #34995: functools.cached_property does not maintain the wrapped method https://bugs.python.org/issue34995 9 msgs #35276: Document thread safety https://bugs.python.org/issue35276 5 msgs #22121: IDLE should start with HOME as the initial working directory https://bugs.python.org/issue22121 4 msgs #35021: Assertion failures in datetimemodule.c. https://bugs.python.org/issue35021 4 msgs #35266: Add _PyPreConfig and rework _PyCoreConfig and _PyMainInterpret https://bugs.python.org/issue35266 4 msgs #24658: open().write() fails on 2 GB+ data (OS X) https://bugs.python.org/issue24658 3 msgs #24746: doctest 'fancy diff' formats incorrectly strip trailing whites https://bugs.python.org/issue24746 3 msgs #33416: Add endline and endcolumn to every AST node https://bugs.python.org/issue33416 3 msgs Issues closed (33) ================== #9842: Document ... used in recursive repr of containers https://bugs.python.org/issue9842 closed by serhiy.storchaka #15245: ast.literal_eval fails on some literals https://bugs.python.org/issue15245 closed by serhiy.storchaka #18407: Fix compiler warnings in pythoncore for Win64 https://bugs.python.org/issue18407 closed by vstinner #18859: README.valgrind should mention --with-valgrind https://bugs.python.org/issue18859 closed by lukasz.langa #25438: document what codec PyMemberDef T_STRING decodes the char * as https://bugs.python.org/issue25438 closed by gregory.p.smith #28401: Don't support the PEP384 stable ABI in pydebug builds https://bugs.python.org/issue28401 closed by brett.cannon #28604: Exception raised by python3.5 when using en_GB locale https://bugs.python.org/issue28604 closed by vstinner #31115: Py3.6 threading/reference counting issues with `numexpr` https://bugs.python.org/issue31115 closed by serhiy.storchaka #31146: Docs: On non-public translations, language picker fallback to https://bugs.python.org/issue31146 closed by mdk #32281: bdist_rpm fails if no rpm command and rpmbuild is not in /usr/ https://bugs.python.org/issue32281 closed by ned.deily #32892: Remove specific constant AST types in favor of ast.Constant https://bugs.python.org/issue32892 closed by serhiy.storchaka #33211: lineno and col_offset are wrong on function definitions with d https://bugs.python.org/issue33211 closed by levkivskyi #33325: Optimize sequences of constants in the compiler https://bugs.python.org/issue33325 closed by serhiy.storchaka #34532: Windows launcher exits with error after listing available vers https://bugs.python.org/issue34532 closed by steve.dower #35035: Documentation for email.utils is named email.util.rst https://bugs.python.org/issue35035 closed by mdk #35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions https://bugs.python.org/issue35059 closed by vstinner #35081: Move internal headers to Include/internal/ https://bugs.python.org/issue35081 closed by vstinner #35169: Improve error messages for assignment https://bugs.python.org/issue35169 closed by serhiy.storchaka #35200: Make the half-open range behaviour easier to teach https://bugs.python.org/issue35200 closed by mdk #35202: Remove unused imports in standard library https://bugs.python.org/issue35202 closed by terry.reedy #35206: [WIP] Add a new experimental _Py_CAPI2 API https://bugs.python.org/issue35206 closed by vstinner #35220: delete "how do I emulate os.kill" section in Windows FAQ https://bugs.python.org/issue35220 closed by deronnax #35221: Enhance venv activate commands readability https://bugs.python.org/issue35221 closed by rhettinger #35265: Internal C API: pass the memory allocator in a context https://bugs.python.org/issue35265 closed by vstinner #35269: A possible segfault involving a newly-created coroutine https://bugs.python.org/issue35269 closed by asvetlov #35273: 'eval' in generator expression behave different in dict from l https://bugs.python.org/issue35273 closed by rhettinger #35274: Running print("\x98") then freeze in Interpreter https://bugs.python.org/issue35274 closed by vstinner #35275: Reading umask (thread-safe) https://bugs.python.org/issue35275 closed by serhiy.storchaka #35287: wrong result for difflib.SequenceMatcher https://bugs.python.org/issue35287 closed by Boris Yang #35288: wrong result for difflib.SequenceMatcher https://bugs.python.org/issue35288 closed by Boris Yang #35289: wrong result for difflib.SequenceMatcher https://bugs.python.org/issue35289 closed by Boris Yang #35290: [FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 l https://bugs.python.org/issue35290 closed by vstinner #35296: Install Include/internal/ header files https://bugs.python.org/issue35296 closed by vstinner

1 0

General concerns about C API changes
by Raymond Hettinger Nov. 23, 2018

Nov. 23, 2018

Overall, I support the efforts to improve the C API, but over the last few weeks have become worried. I don't want to hold up progress with fear, uncertainty, and doubt. Yet, I would like to be more comfortable that we're all aware of what is occurring and what are the potential benefits and risks. * Inline functions are great. They provide true local variables, better separation of concerns, are far less kludgy than text based macro substitution, and will typically generate the same code as the equivalent macro. This is good tech when used with in a single source file where it has predictable results. However, I'm not at all confident about moving these into header files which are included in multiple target .c files which need be compiled into separate .o files and linked to other existing libraries. With a macro, I know for sure that the substitution is taking place. This happens at all levels of optimization and in a debug mode. The effects are 100% predictable and have a well-established track record in our mature battle-tested code base. With cross module function calls, I'm less confident about what is happening, partly because compilers are free to ignore inline directives and partly because the semantics of inlining are less clear when the crossing module boundaries. * Other categories of changes that we make tend to have only a shallow reach. However, these C API changes will likely touch every C extension that has ever been written, some of which is highly tuned but not actively re-examined. If any mistakes are make, they will likely be pervasive. Accordingly, caution is warranted. My expectation was that the changes would be conducted in experimental branches. But extensive changes are already being made (or about to be made) on the 3.8 master. If a year from now, we decide that the changes were destabilizing or that the promised benefits didn't materialize, they will be difficult to undo because there are so many of them and because they will be interleaved with other changes. The original motivation was to achieve a 2x speedup in return for significantly churning the C API. However, the current rearranging of the include files and macro-to-inline-function changes only give us churn. At the very best, they will be performance neutral. At worst, formerly cheap macro calls will become expensive in places that we haven't thought to run timings on. Given that compilers don't have to honor an inline directive, we can't really know for sure -- perhaps today it works out fine, and perhaps tomorrow the compilers opt for a different behavior. Maybe everything that is going on is fine. Maybe it's not. I am not expert enough to know for sure, but we should be careful before green-lighting such an extensive series of changes directly to master. Reasonable questions to ask are: 1) What are the risks to third party modules, 2) Do we really know that the macro-to-inline-function transformations are semantically neutral. 3) If there is no performance benefit (none has been seen so far, nor is any promised in the pending PRs), is it worth it? We do know that PyPy folks have had their share of issues with the C API, but I'm not sure that we can make any of this go away without changing the foundations of the whole ecosystem. It is inconvenient for a full GC environment to interact with the API for a reference counted environment -- I don't think we can make this challenge go away without giving up reference counting. It is inconvenient for a system that manifests objects on demand to interact with an API that assumes that objects have identity and never more once they are created -- I don't think we can make this go away either. It is inconvenient to a system that uses unboxed data to interact with our API where everything is an object that includes a type pointer and reference count -- We have provided an API for boxing and boxing, but the trip back-and-forth is inconveniently expensive -- I don't think we can make that go away either because too much of the ecosystem depends on that API. There are some things that can be mitigated such as challenges with borrowed references but that doesn't seem to have been the focus on any of the PRs. In short, I'm somewhat concerned about the extensive changes that are occurring. I do know they will touch substantially every C module in the entire ecosystem. I don't know whether they are safe or whether they will give any real benefit. FWIW, none of this is a criticism of the work being done. Someone needs to think deeply about the C API or else progress will never be made. That said, it is a high risk project with many PRs going directly into master, so it does warrant having buy in that the churn isn't destabilizing and will actually produce a benefit that is worth it. Raymond

6 9

Experiment an opt-in new C API for Python? (leave current API unchanged)
by Stefan Krah Nov. 22, 2018

Nov. 22, 2018

Victor Stinner wrote: > Moreover, I failed to find anyone who can explain me how the C API is used > in the wild, which functions are important or not, what is the C API, etc. In practice people desperately *have* to use whatever is there, including functions with underscores that are not even officially in the C-API. I have to use _PyFloat_Pack* in order to be compatible with CPython, I need PySlice_Unpack() etc., I need PyUnicode_KIND(), need PyUnicode_AsUTF8AndSize(), I *wish* there were PyUnicode_AsAsciiAndSize(). In general, in daily use of the C-API I wish it were *larger* and not smaller. I often want functions that return C instead of Python values ot functions that take C instead of Python values. The ideal situation for me would be a lower layer library, say libcpython.a that has all those functions like _PyFloat_Pack*. It would be an enormous amount of work though, especially since the status quo kind of works. Stefan Krah

9 15

bpo-34532 status
by Brendan Gerrity Nov. 20, 2018

Nov. 20, 2018

Just wanted to check on bpo-34532/pr#9039. The requested changes were submitted as a commit to the PR. Best, Brendan

3 2

Experiment an opt-in new C API for Python? (leave current API unchanged)
by Victor Stinner Nov. 20, 2018

Nov. 20, 2018

Hi, The current C API of Python is both a strength and a weakness of the Python ecosystem as a whole. It's a strength because it allows to quickly reuse a huge number of existing libraries by writing a glue for them. It made numpy possible and this project is a big sucess! It's a weakness because of its cost on the maintenance, it prevents optimizations, and more generally it prevents to experiment modifying Python internals. For example, CPython cannot use tagged pointers, because the existing C API is heavily based on the ability to dereference a PyObject* object and access directly members of objects (like PyTupleObject). For example, Py_INCREF() modifies *directly* PyObject.ob_refcnt. It's not possible neither to use a Python compiled in debug mode on C extensions (compiled in release mode), because the ABI is different in debug mode. As a consequence, nobody uses the debug mode, whereas it is very helpful to develop C extensions and investigate bugs. I also consider that the C API gives too much work to PyPy (for their "cpyext" module). A better C API (not leaking implementation) details would make PyPy more efficient (and simplify its implementation in the long term, when the support for the old C API can be removed). For example, PyList_GetItem(list, 0) currently converts all items of the list to PyObject* in PyPy, it can waste memory if only the first item of the list is needed. PyPy has much more efficient storage than an array of PyObject* for lists. I wrote a website to explain all these issues with much more details: https://pythoncapi.readthedocs.io/ I identified "bad APIs" like using borrowed references or giving access to PyObject** (ex: PySequence_Fast_ITEMS). I already wrote an (incomplete) implementation of a new C API which doesn't leak implementation details: https://github.com/pythoncapi/pythoncapi It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the name) to get the new API. The current C API is unchanged. Ah, important points. I don't want to touch the current C API nor make it less efficient. And compatibility in both directions (current C API <=> new C API) is very important for me. There is no such plan as "Python 4" which would break the world and *force* everybody to upgrade to the new C API, or stay to Python 3 forever. No. The new C API must be an opt-in option, and current C API remains the default and not be changed. I have different ideas for the compatibility part, but I'm not sure of what are the best options yet. My short term for the new C API would be to ease the experimentation of projects like tagged pointers. Currently, I have to maintain the implementation of a new C API which is not really convenient. -- Today I tried to abuse the Py_DEBUG define for the new C API, but it seems to be a bad idea: https://github.com/python/cpython/pull/10435 A *new* define is needed to opt-in for the new C API. Victor

15 51

Need discussion for a PR about memory and objects
by Stephane Wirtel Nov. 20, 2018

Nov. 20, 2018

In this PR [https://github.com/python/cpython/pull/3382] "Remove reference to address from the docs, as it only causes confusion", opened by Chris Angelico, there is a discussion about the right term to use for the address of an object in memory. If you are interested by the topic, you could comment it. If there is no comments then I think we could close the PR. Thank you Stéphane -- Stéphane Wirtel - https://wirtel.be - @matrixise

10 12