Python-Dev April 2020

python-dev@python.org

87 participants
48 discussions

Do we need port some extension modules to the modules directory?

by Hai Shi

Hi, I notice some modules not in modules directory(for example: _warnings、marshal in python directory). Do we need port those modules to modules directory? Sorry for my noisy info. Bests Wishs, Hai Shi

12 months

Python2 as 𝑣 → 𝑒

by Mike Miller

Unless I've read something wrong, it looks like the final Python 2 release (2.7.18) should approximate the math constant e: >>> import math >>> math.e 2.718281828459045 Aka: Python as 𝑣 → 𝑒 Python ≈ 𝑒 Would be fun to note that in the announcement/docs somehow. -Mike

12 months

Fixing Copy on Writes with Immortal Instances

by Eddie Elizondo

Copy on writes are a big problem in large Python application that rely on multiple processes sharing the same memory. I just sent out a patch introduces Immortal Instances which allows the user to bypass reference counting on specific objects to avoid CoW on forked processes. This has helped in reducing the memory constraints in some of our applications. The PR has a more in-depth explanation about the change. Looking forward to get your thoughts. Github PR: https://github.com/python/cpython/pull/19474 Bug Report: https://bugs.python.org/issue40255 - Eddie E.

1 year

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2020-04-03 - 2020-04-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7433 (+32) closed 44563 (+37) total 51996 (+69) Open issues with patches: 2934 Issues opened (50) ================== #39984: Move pending calls from _PyRuntime to PyInterpreterState https://bugs.python.org/issue39984 reopened by vstinner #40178: Convert the remaining os funtions to Argument Clinic https://bugs.python.org/issue40178 opened by serhiy.storchaka #40179: Argument Clinic incorretly translates #elif https://bugs.python.org/issue40179 opened by serhiy.storchaka #40180: isinstance(cls_with_metaclass, non_type) raises KeyError https://bugs.python.org/issue40180 opened by terry.reedy #40181: IDLE: remove positional-only note from calltips https://bugs.python.org/issue40181 opened by terry.reedy #40183: AC_COMPILE_IFELSE doesn't work in all cases https://bugs.python.org/issue40183 opened by jerome.hamm #40186: test_notify_all hangs forever in sparc64 https://bugs.python.org/issue40186 opened by BTaskaya #40188: Azure Pipelines jobs failing randomly with: Unable to connect https://bugs.python.org/issue40188 opened by vstinner #40191: tempfile.mkstemp() | Documentation Error https://bugs.python.org/issue40191 opened by Howard Waterfall #40192: time.thread_time isn't outputting in nanoseconds in AIX https://bugs.python.org/issue40192 opened by BTaskaya #40195: multiprocessing.Queue.put can fail silently due to pickle erro https://bugs.python.org/issue40195 opened by Sander Land #40197: Add nanoseconds to timing table in What's new python 3.8 https://bugs.python.org/issue40197 opened by mchels #40198: macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH d https://bugs.python.org/issue40198 opened by dgelessus #40199: Invalid escape sequence DeprecationWarnings don't trigger by d https://bugs.python.org/issue40199 opened by Numerlor #40202: Misleading grammatically of ValueError Message? https://bugs.python.org/issue40202 opened by Jacob RR #40203: Warn about invalid PYTHONUSERBASE https://bugs.python.org/issue40203 opened by Volker Weißmann #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 opened by xtreak #40205: Profile 'builtins' parameter documentation missing https://bugs.python.org/issue40205 opened by bar.harel #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 opened by BTaskaya #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 opened by BTaskaya #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 opened by hakancelik #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 opened by Nikolai Ehrhardt #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 opened by BTaskaya #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 opened by BTaskaya #40213: contextlib.aclosing() https://bugs.python.org/issue40213 opened by John Belmonte #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 opened by aeros #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 opened by Óscar García Amor #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 opened by vstinner #40218: sys.executable is a non existing path if python is executed fr https://bugs.python.org/issue40218 opened by Volker Weißmann #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 opened by Stephen Bell #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 opened by vstinner #40222: "Zero cost" exception handling https://bugs.python.org/issue40222 opened by Mark.Shannon #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 opened by skrah #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 opened by brendon-zhang(a)hotmail.com #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 opened by iivanyuk #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 opened by Mark.Shannon #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 opened by Steven Lu #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 opened by Henry Carscadden #40231: Fix pending calls in subinterpreters https://bugs.python.org/issue40231 opened by vstinner #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 opened by vstinner #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 opened by eric.snow #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 opened by dholth #40236: datetime.datetime.strptime get day error https://bugs.python.org/issue40236 opened by zhanying #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 opened by vstinner #40238: os.stat() on Windows succeeds for nonexistent paths with trail https://bugs.python.org/issue40238 opened by laloch #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 opened by Eric Cousineau #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 opened by vstinner #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 opened by wmeehan #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 opened by Michael.Felt #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 opened by 0az Most recent 15 issues with no replies (15) ========================================== #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 Most recent 15 issues waiting for review (15) ============================================= #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 Top 10 most discussed issues (10) ================================= #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 16 msgs #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 16 msgs #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 13 msgs #39481: Implement PEP 585 (Type Hinting Generics In Standard Collectio https://bugs.python.org/issue39481 12 msgs #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 12 msgs #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 11 msgs #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 9 msgs #40060: socket.TCP_NOTSENT_LOWAT is missing in official macOS builds https://bugs.python.org/issue40060 8 msgs #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 7 msgs #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 7 msgs Issues closed (36) ================== #5654: Add C hook in PyDict_SetItem for debuggers https://bugs.python.org/issue5654 closed by vstinner #25780: Add support for CAN_RAW_JOIN_FILTERS https://bugs.python.org/issue25780 closed by pablogsal #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 closed by vstinner #37127: Handling pending calls during runtime finalization may cause p https://bugs.python.org/issue37127 closed by vstinner #38501: multiprocessing.Pool hangs atexit (and garbage collection some https://bugs.python.org/issue38501 closed by pablogsal #38689: IDLE crashes when KeyError is raised during calltip generation https://bugs.python.org/issue38689 closed by terry.reedy #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 closed by BTaskaya #39689: struct and memoryview tests rely on undefined behavior (as rev https://bugs.python.org/issue39689 closed by petr.viktorin #40082: trip_signal() gets NULL tstate on Windows on CTRL+C https://bugs.python.org/issue40082 closed by vstinner #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 closed by ammar2 #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 closed by terry.reedy #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 closed by pablogsal #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 closed by SilentGhost #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 closed by steve.dower #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 closed by benjamin.peterson #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 closed by nergall2 #40177: Python Language Reference Documentation https://bugs.python.org/issue40177 closed by mark.dickinson #40182: Remove the _field_types attribute of NamedTuple https://bugs.python.org/issue40182 closed by serhiy.storchaka #40184: Compiler warnings under sparc64 https://bugs.python.org/issue40184 closed by benjamin.peterson #40185: Refactor typing.NamedTuple https://bugs.python.org/issue40185 closed by serhiy.storchaka #40187: Refactor typing.TypedDict https://bugs.python.org/issue40187 closed by serhiy.storchaka #40189: MultiProcessing Subclass - overrides run method issue https://bugs.python.org/issue40189 closed by Python_dev_IL #40190: Add support for _SC_AIX_REALMEM in posix.sysconf https://bugs.python.org/issue40190 closed by vstinner #40193: thread.get_native_id() support for solaris https://bugs.python.org/issue40193 closed by BTaskaya #40194: Document special meanings of a single underscore https://bugs.python.org/issue40194 closed by SilentGhost #40196: symtable.Symbol.is_local() can be True for global symbols https://bugs.python.org/issue40196 closed by pablogsal #40200: count 0 crash https://bugs.python.org/issue40200 closed by BTaskaya #40201: Last digit count error https://bugs.python.org/issue40201 closed by josh.r #40206: Multiplying 4.6*100 will result in 459.99999999999994 https://bugs.python.org/issue40206 closed by eric.smith #40216: Support --runstatedir in configure https://bugs.python.org/issue40216 closed by vstinner #40220: Modules/gcmodule.c:434: update_refs: Assertion "gc_get_refs(gc https://bugs.python.org/issue40220 closed by gregory.p.smith #40224: delete_me https://bugs.python.org/issue40224 closed by ronaldoussoren #40226: Leak in tstate->interp->ceval.pending https://bugs.python.org/issue40226 closed by vstinner #40233: Awkward to set compression on writeable ZipFile.open() https://bugs.python.org/issue40233 closed by dholth #40239: Add a function for merging sorted iterables https://bugs.python.org/issue40239 closed by serhiy.storchaka #40242: zmq mysql core dump https://bugs.python.org/issue40242 closed by eric.smith

1 year

Need help with test_ctypes failing on Windows (test_load_dll_with_flags)

by Guido van Rossum

I have a large PR (https://github.com/python/cpython/pull/18239, for PEP 585) that's failing in the Azures pipeline on Win32 and Win64 only. My trusty assistant who has a Windows laptop couldn't reproduce the failure. Can I buy a hint from someone? Steve? The relevant failure output is: ====================================================================== ERROR: test_load_dll_with_flags (ctypes.test.test_loading.LoaderTest) [WinDLL('_sqlite3.dll', winmode=0)] ---------------------------------------------------------------------- Traceback (most recent call last): File "d:\a\1\s\lib\ctypes\test\test_loading.py", line 140, in should_pass subprocess.check_output( File "d:\a\1\s\lib\subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "d:\a\1\s\lib\subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['d:\\a\\1\\s\\PCbuild\\win32\\python.exe', '-c', "from ctypes import *; import nt;WinDLL('_sqlite3.dll', winmode=0)"]' returned non-zero exit status 1. ---------------------------------------------------------------------- -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

1 year

Problem with instantiating a C extension class from another class

by Musbur

Hello, (I've asked this question before on python-list but only got scarce and ultimately unhelpful answers. Although this isn't about Python development itself, I'm assuming that there is more CPython knowledge on this list than on the other, so please bear with me.) I've written a C extension module, each defining a class (called Series and CLHist, respectively). I can import both modules in Python and use the classes. Everything works fine, also according to valgrind. Now I want to return a Series instance directly from a CLHist method, and that immediately crashes Python with a segfault. Here's a bit of code and just the _new, _init, and _finalize methods which I've sprinkled with debugging code. When I instantiate a Series object from Python, I see output from all three methods. When I call pyseries_new() from C, none of the three functions are called and it crashes. At some point I thought maybe I should call PyType_Ready() from the C side before instantiating Series. In a "standard" (system) Python installation, it printed the desired debugging output and crashed later. Under a debugging Python installation I built it outputs nothing except an interesting post-mortem "Fatal Python error: UNREF invalid object" (see below). Here's my C code with the boilerplate stuff left out. typedef struct { PyObject_HEAD struct clh_series *series; } Series; static PyObject *Series_new(PyTypeObject *type, PyObject *args, PyObject *kw) { Series *self; self = (Series *) type->tp_alloc(type, 0); fprintf(stderr, "Series_new(%p)\n", self); self->series = NULL; return (PyObject*)self; } static int Series_init(Series *self, PyObject *args, PyObject *kw) { fprintf(stderr, "Series_init(%p)\n", self); self->series = NULL; return 0; } static void Series_finalize(PyObject *self) { fprintf(stderr, "Series_finalize(%p)\n", self); clh_series_free(((Series*)self)->series); } /* To create a new Series object directly from C */ PyObject *pyseries_new(struct clh_series *series) { Series *pyseries; pyseries = PyObject_New(Series, &series_type); PyObject_Init((PyObject *)pyseries, &series_type); pyseries->series = series; return (PyObject *) pyseries; } Here's the debugging output: * ob object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 * op->_ob_prev->_ob_next object : <_Series object at 0x7f1a4e2ee9d0> type : _Series refcount: 1 address : 0x7f1a4e2ee9d0 * op->_ob_next->_ob_prev object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 Fatal Python error: UNREF invalid object

1 year

Question about bytecode stability

by Jonathan Goble

I'm getting ready to get back into an old personal project with some new ideas. Before I do, I need to ask a question regarding the stability of CPython bytecode. Before you say it, I am well aware and fully understand that the opcodes may change in value and semantics between 3.x and 3.(x+1) without notice, and that's acceptable to me. My question is, are the opcodes guaranteed stable across the lifetime of a single 3.x release? In other words, are they guaranteed to not change values or semantics between 3.x.y and 3.x.(y+1)? Reading through the list of opcodes in the dis documentation, it seems that all changes have occurred in 3.x.0, so it seems the answer would be yes, but then the "CPython implementation detail" paragraph at the top doesn't specify that and is a little vague on whether that's true or not. (For context, my personal project is a BASIC implementation in Python that would compile to Python bytecode.)

1 year

[RELEASE] Python 2.7.18 release candidate 1

by Benjamin Peterson

Greetings, 2.7.18 release candidate 1, a testing release for the last release of the Python 2.7 series, is now available for download. The CPython core developers stopped applying routine bugfixes to the 2.7 branch on January 1. 2.7.18 will includes fixes that were made between the release of 2.7.17 and the end of 2019. A final—very final!—release is expected in 2 weeks. Downloads are at: https://www.python.org/downloads/release/python-2718rc1/ The full changelog is at https://raw.githubusercontent.com/python/cpython/v2.7.18rc1/Misc/NEWS.d/2.7… Test it out, and let us know if there are any critical problems at https://bugs.python.org/ (This is the last chance!) All the best, Benjamin

1 year

How to enable tracemalloc for the test suite?

by Skip Montanaro

I've got a memory issue in my modified Python interpreter I'm trying to debug. Output at the end of the problematic unit test looks like this: ... == Tests result: FAILURE then SUCCESS == 1 test OK. 1 re-run test: test_rattlesnake Total duration: 2.9 sec Tests result: FAILURE then SUCCESS Debug memory block at address p=0x55a227969080: API '' 0 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 *** OUCH at p-5: 0x00 *** OUCH at p-4: 0x00 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0x55a227969080 are not all FORBIDDENBYTE (0xfd): at tail+0: 0x00 *** OUCH at tail+1: 0x00 *** OUCH at tail+2: 0x00 *** OUCH at tail+3: 0x00 *** OUCH at tail+4: 0x00 *** OUCH at tail+5: 0x00 *** OUCH at tail+6: 0x00 *** OUCH at tail+7: 0x00 *** OUCH The block was made by call #0 to debug malloc/realloc. Enable tracemalloc to get the memory block allocation traceback Looking at the tracemalloc module docs and trying various command line args (-X tracemalloc=5) or environment variables (PYTHONTRACEMALLOC=5) I'm unable to provoke any different output. The module docs give a tantalizing suggestion: Example of output of the Python test suite: but I see no command line args related to running the test suite with tracemalloc enabled. Pointers appreciated. Skip

1 year

Any thoughts about a control flow optimizer for CPython?

by joannah nanjekye

Hey all, From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. -- Best, Joannah Nanjekye *"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis*

1 year

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev April 2020