Mailman 3 April 2019 - Python-checkins

bpo-36465: Fix test_regrtest on Windows (GH-12945)
by Victor Stinner April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/235e7b2b0d937ba8b4a9172aa72206187e… commit: 235e7b2b0d937ba8b4a9172aa72206187e3e1f54 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-25T01:46:31+02:00 summary: bpo-36465: Fix test_regrtest on Windows (GH-12945) Fix Py_DEBUG constant: check for sys.gettotalrefcount attribute rather than sys.getobjects. Update also SpecialBuilds.txt documentation. files: M Lib/test/test_regrtest.py M Misc/SpecialBuilds.txt diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 61937767ec12..a9febd00eda0 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -21,7 +21,7 @@ from test.libregrtest import utils -Py_DEBUG = hasattr(sys, 'getobjects') +Py_DEBUG = hasattr(sys, 'gettotalrefcount') ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..') ROOT_DIR = os.path.abspath(os.path.normpath(ROOT_DIR)) diff --git a/Misc/SpecialBuilds.txt b/Misc/SpecialBuilds.txt index c188dd707590..193de57b0850 100644 --- a/Misc/SpecialBuilds.txt +++ b/Misc/SpecialBuilds.txt @@ -30,6 +30,8 @@ name "_" holds a reference to the last result displayed! Py_REF_DEBUG also checks after every decref to verify that the refcount hasn't gone negative, and causes an immediate fatal error if it has. +Py_DEBUG implies Py_REF_DEBUG. + Special gimmicks: sys.gettotalrefcount() @@ -39,6 +41,8 @@ sys.gettotalrefcount() Py_TRACE_REFS ------------- +Build option: ``./configure --with-trace-refs``. + Turn on heavy reference debugging. This is major surgery. Every PyObject grows two more pointers, to maintain a doubly-linked list of all live heap-allocated objects. Most built-in type objects are not in this list, as they're statically @@ -49,8 +53,6 @@ has been created. Note that because the fundamental PyObject layout changes, Python modules compiled with Py_TRACE_REFS are incompatible with modules compiled without it. -Py_TRACE_REFS implies Py_REF_DEBUG. - Special gimmicks: sys.getobjects(max[, type]) @@ -138,7 +140,8 @@ look at the object, you're likely to see that it's entirely filled with 0xDB (meaning freed memory is getting used) or 0xCB (meaning uninitialized memory is getting used). -Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. +Note that PYMALLOC_DEBUG requires WITH_PYMALLOC. Py_DEBUG implies +PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled). Special gimmicks: @@ -156,7 +159,7 @@ Py_DEBUG This is what is generally meant by "a debug build" of Python. -Py_DEBUG implies LLTRACE, Py_REF_DEBUG, Py_TRACE_REFS, and PYMALLOC_DEBUG (if +Py_DEBUG implies LLTRACE, Py_REF_DEBUG, and PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled). In addition, C assert()s are enabled (via the C way: by not defining NDEBUG), and some routines do additional sanity checks inside "#ifdef Py_DEBUG" blocks. @@ -223,3 +226,5 @@ the interpreter is doing are sprayed to stdout, such as every opcode and opcode argument and values pushed onto and popped off the value stack. Not useful very often, but very useful when needed. + +Py_DEBUG implies LLTRACE.

1 0

bpo-36465: Make release and debug ABI compatible (GH-12615)
by Victor Stinner April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/f4e4703e746067d6630410408d414b1100… commit: f4e4703e746067d6630410408d414b11003334d6 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-25T00:56:28+02:00 summary: bpo-36465: Make release and debug ABI compatible (GH-12615) Release build and debug build are now ABI compatible: the Py_DEBUG define no longer implies Py_TRACE_REFS define which introduces the only ABI incompatibility. A new "./configure --with-trace-refs" build option is now required to get Py_TRACE_REFS define which adds sys.getobjects() function and PYTHONDUMPREFS environment variable. Changes: * Add ./configure --with-trace-refs * Py_DEBUG no longer implies Py_TRACE_REFS files: A Misc/NEWS.d/next/Build/2019-04-09-18-19-43.bpo-36465.-w6vx6.rst M Doc/using/cmdline.rst M Include/object.h M Lib/test/support/__init__.py M configure M configure.ac M pyconfig.h.in diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 0574336cf354..fd47ce2ab538 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -922,15 +922,18 @@ conflict. Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ -Setting these variables only has an effect in a debug build of Python, that is, -if Python was configured with the ``--with-pydebug`` build option. +Setting these variables only has an effect in a debug build of Python. .. envvar:: PYTHONTHREADDEBUG If set, Python will print threading debug info. + Need Python configured with the ``--with-pydebug`` build option. + .. envvar:: PYTHONDUMPREFS If set, Python will dump objects and reference counts still alive after shutting down the interpreter. + + Need Python configured with the ``--with-trace-refs`` build option. diff --git a/Include/object.h b/Include/object.h index 86cbfc581ed1..13e88a6dc6f0 100644 --- a/Include/object.h +++ b/Include/object.h @@ -54,13 +54,8 @@ A standard interface exists for objects that contain an array of items whose size is determined when the object is allocated. */ -/* Py_DEBUG implies Py_TRACE_REFS. */ -#if defined(Py_DEBUG) && !defined(Py_TRACE_REFS) -#define Py_TRACE_REFS -#endif - -/* Py_TRACE_REFS implies Py_REF_DEBUG. */ -#if defined(Py_TRACE_REFS) && !defined(Py_REF_DEBUG) +/* Py_DEBUG implies Py_REF_DEBUG. */ +#if defined(Py_DEBUG) && !defined(Py_REF_DEBUG) #define Py_REF_DEBUG #endif diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 2bb561b4cee1..dc96318b38f7 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1653,7 +1653,7 @@ def python_is_optimized(): _header = 'nP' _align = '0n' -if hasattr(sys, "gettotalrefcount"): +if hasattr(sys, "getobjects"): _header = '2P' + _header _align = '0P' _vheader = _header + 'n' diff --git a/Misc/NEWS.d/next/Build/2019-04-09-18-19-43.bpo-36465.-w6vx6.rst b/Misc/NEWS.d/next/Build/2019-04-09-18-19-43.bpo-36465.-w6vx6.rst new file mode 100644 index 000000000000..52de1ef44830 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2019-04-09-18-19-43.bpo-36465.-w6vx6.rst @@ -0,0 +1,5 @@ +Release build and debug build are now ABI compatible: the ``Py_DEBUG`` define +no longer implies ``Py_TRACE_REFS`` define which introduces the only ABI +incompatibility. A new ``./configure --with-trace-refs`` build option is now +required to get ``Py_TRACE_REFS`` define which adds :func:`sys.getobjects` +function and ``PYTHONDUMPREFS`` environment variable. diff --git a/configure b/configure index e96c03ad4f59..b02d17c053c6 100755 --- a/configure +++ b/configure @@ -814,6 +814,7 @@ with_suffix enable_shared enable_profiling with_pydebug +with_trace_refs with_assertions enable_optimizations with_lto @@ -1500,6 +1501,7 @@ Optional Packages: compiler --with-suffix=.exe set executable suffix --with-pydebug build with Py_DEBUG defined + --with-trace-refs enable tracing references for debugging purpose --with-assertions build with C assertions enabled --with-lto Enable Link Time Optimization in any build. Disabled by default. @@ -6333,8 +6335,30 @@ $as_echo "no" >&6; } fi -# Check for --with-assertions. Py_DEBUG implies assertions, but also changes -# the ABI. This allows enabling assertions without changing the ABI. +# Check for --with-trace-refs +# --with-trace-refs +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-trace-refs" >&5 +$as_echo_n "checking for --with-trace-refs... " >&6; } + +# Check whether --with-trace-refs was given. +if test "${with_trace_refs+set}" = set; then : + withval=$with_trace_refs; +else + with_trace_refs=no +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_trace_refs" >&5 +$as_echo "$with_trace_refs" >&6; } + +if test "$with_trace_refs" = "yes" +then + +$as_echo "#define Py_TRACE_REFS 1" >>confdefs.h + +fi + +# Check for --with-assertions. +# This allows enabling assertions without Py_DEBUG. assertions='false' { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-assertions" >&5 $as_echo_n "checking for --with-assertions... " >&6; } diff --git a/configure.ac b/configure.ac index 3f378c97345e..65d3f8e69129 100644 --- a/configure.ac +++ b/configure.ac @@ -1228,8 +1228,21 @@ else AC_MSG_RESULT(no); Py_DEBUG='false' fi], [AC_MSG_RESULT(no)]) -# Check for --with-assertions. Py_DEBUG implies assertions, but also changes -# the ABI. This allows enabling assertions without changing the ABI. +# Check for --with-trace-refs +# --with-trace-refs +AC_MSG_CHECKING(for --with-trace-refs) +AC_ARG_WITH(trace-refs, + AS_HELP_STRING([--with-trace-refs],[enable tracing references for debugging purpose]),, + with_trace_refs=no) +AC_MSG_RESULT($with_trace_refs) + +if test "$with_trace_refs" = "yes" +then + AC_DEFINE(Py_TRACE_REFS, 1, [Define if you want to enable tracing references for debugging purpose]) +fi + +# Check for --with-assertions. +# This allows enabling assertions without Py_DEBUG. assertions='false' AC_MSG_CHECKING(for --with-assertions) AC_ARG_WITH(assertions, diff --git a/pyconfig.h.in b/pyconfig.h.in index d41d57932766..562c0271133b 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1374,6 +1374,9 @@ externally defined: 0 */ #undef Py_HASH_ALGORITHM +/* Define if you want to enable tracing references for debugging purpose */ +#undef Py_TRACE_REFS + /* assume C89 semantics that RETSIGTYPE is always void */ #undef RETSIGTYPE

1 0

Add @pablogsal to code owners file for Parser/pgen (GH-12944)
by Pablo Galindo April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/888f37bc2826d9ab2cbec6f153e7f58a34… commit: 888f37bc2826d9ab2cbec6f153e7f58a34785c4a branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2019-04-24T21:25:23+01:00 summary: Add @pablogsal to code owners file for Parser/pgen (GH-12944) Add myself to the codeowners file as I would like to be automatically added as a reviewer for PRs that touch that component. files: M .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2ca555b7fa50..9b7a4032fb43 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -43,6 +43,9 @@ Python/bootstrap_hash.c @python/crypto-team **/*imap* @python/email-team **/*poplib* @python/email-team +# Parser/Pgen +/Parser/pgen/ @pablogsal + # SQLite 3 **/*sqlite* @berkerpeksag

1 0

bpo-36668: FIX reuse semaphore tracker for child processes (#5172)
by Antoine Pitrou April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/004b93ea8947bcbe85b6fa16fe0999bfa7… commit: 004b93ea8947bcbe85b6fa16fe0999bfa712d5c1 branch: master author: Thomas Moreau <thomas.moreau.2010(a)gmail.com> committer: Antoine Pitrou <antoine(a)python.org> date: 2019-04-24T21:45:52+02:00 summary: bpo-36668: FIX reuse semaphore tracker for child processes (#5172) Fix the multiprocessing.semaphore_tracker so it is reused by child processes. files: A Misc/NEWS.d/next/Library/2018-04-06-11-06-23.bpo-31310.eq9ky0.rst M Lib/multiprocessing/semaphore_tracker.py M Lib/test/_test_multiprocessing.py diff --git a/Lib/multiprocessing/semaphore_tracker.py b/Lib/multiprocessing/semaphore_tracker.py index 82833bcf861a..3c2c3ad61aee 100644 --- a/Lib/multiprocessing/semaphore_tracker.py +++ b/Lib/multiprocessing/semaphore_tracker.py @@ -44,20 +44,23 @@ def ensure_running(self): This can be run from any process. Usually a child process will use the semaphore created by its parent.''' with self._lock: - if self._pid is not None: + if self._fd is not None: # semaphore tracker was launched before, is it still running? + if self._check_alive(): + # => still alive + return + # => dead, launch it again + os.close(self._fd) + + # Clean-up to avoid dangling processes. try: - pid, _ = os.waitpid(self._pid, os.WNOHANG) + # _pid can be None if this process is a child from another + # python process, which has started the semaphore_tracker. + if self._pid is not None: + os.waitpid(self._pid, 0) except ChildProcessError: - # The process terminated + # The semaphore_tracker has already been terminated. pass - else: - if not pid: - # => still alive - return - - # => dead, launch it again - os.close(self._fd) self._fd = None self._pid = None @@ -99,6 +102,17 @@ def ensure_running(self): finally: os.close(r) + def _check_alive(self): + '''Check that the pipe has not been closed by sending a probe.''' + try: + # We cannot use send here as it calls ensure_running, creating + # a cycle. + os.write(self._fd, b'PROBE:0\n') + except OSError: + return False + else: + return True + def register(self, name): '''Register name of semaphore with semaphore tracker.''' self._send('REGISTER', name) @@ -150,6 +164,8 @@ def main(fd): cache.add(name) elif cmd == b'UNREGISTER': cache.remove(name) + elif cmd == b'PROBE': + pass else: raise RuntimeError('unrecognized command %r' % cmd) except Exception: diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 553ab8178316..836fde88cd26 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -4891,6 +4891,34 @@ def test_semaphore_tracker_sigkill(self): # Uncatchable signal. self.check_semaphore_tracker_death(signal.SIGKILL, True) + @staticmethod + def _is_semaphore_tracker_reused(conn, pid): + from multiprocessing.semaphore_tracker import _semaphore_tracker + _semaphore_tracker.ensure_running() + # The pid should be None in the child process, expect for the fork + # context. It should not be a new value. + reused = _semaphore_tracker._pid in (None, pid) + reused &= _semaphore_tracker._check_alive() + conn.send(reused) + + def test_semaphore_tracker_reused(self): + from multiprocessing.semaphore_tracker import _semaphore_tracker + _semaphore_tracker.ensure_running() + pid = _semaphore_tracker._pid + + r, w = multiprocessing.Pipe(duplex=False) + p = multiprocessing.Process(target=self._is_semaphore_tracker_reused, + args=(w, pid)) + p.start() + is_semaphore_tracker_reused = r.recv() + + # Clean up + p.join() + w.close() + r.close() + + self.assertTrue(is_semaphore_tracker_reused) + class TestSimpleQueue(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2018-04-06-11-06-23.bpo-31310.eq9ky0.rst b/Misc/NEWS.d/next/Library/2018-04-06-11-06-23.bpo-31310.eq9ky0.rst new file mode 100644 index 000000000000..32ebf4efb74b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-04-06-11-06-23.bpo-31310.eq9ky0.rst @@ -0,0 +1 @@ +Fix the multiprocessing.semaphore_tracker so it is reused by child processes

1 0

Issue 35224: Add PEP 572 (assignment expressions) to What's New (#12941)
by Guido van Rossum April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/09d434caa2c01477ea2ccc3e9b88e9faa0… commit: 09d434caa2c01477ea2ccc3e9b88e9faa0107c61 branch: master author: Guido van Rossum <guido(a)python.org> committer: GitHub <noreply(a)github.com> date: 2019-04-24T11:30:17-07:00 summary: Issue 35224: Add PEP 572 (assignment expressions) to What's New (#12941) This is meant as a stub, during the PyCon sprints we can iterate. files: M Doc/whatsnew/3.8.rst diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index bd7ad3f87cb5..ae8163a1b156 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -67,6 +67,22 @@ Summary -- Release highlights New Features ============ +Assignment expressions +---------------------- + +There is new syntax (the "walrus operator", ``:=``) to assign values +to variables as part of an expression. Example:: + + if (n := len(a)) > 10: + print(f"List is too long ({n} elements, expected <= 10)") + +See :pep:`572` for a full description. + +(Contributed by Emily Morehouse in :issue:`35224`.) + +.. TODO: Emily will sprint on docs at PyCon US 2019. + + Parallel filesystem cache for compiled bytecode files -----------------------------------------------------

1 0

bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
by Victor Stinner April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/43125224d6da5febb34101ebfd36536d79… commit: 43125224d6da5febb34101ebfd36536d791d68cd branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-24T18:23:53+02:00 summary: bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939) Py_InitializeEx() now uses a runtime variable passed to subfunctions, rather than working directly on the global variable _PyRuntime. Add 'runtime' parameter to _PyCoreConfig_Write(), _PySys_Create(), _PySys_InitMain(), _PyGILState_Init(), emit_stderr_warning_for_legacy_locale() and other subfunctions. files: M Include/internal/pycore_coreconfig.h M Include/internal/pycore_pylifecycle.h M Python/coreconfig.c M Python/pylifecycle.c M Python/pystate.c M Python/sysmodule.c diff --git a/Include/internal/pycore_coreconfig.h b/Include/internal/pycore_coreconfig.h index e88a46314c89..ac98e00159ce 100644 --- a/Include/internal/pycore_coreconfig.h +++ b/Include/internal/pycore_coreconfig.h @@ -8,6 +8,8 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif +#include "pycore_pystate.h" /* _PyRuntimeState */ + /* --- _PyWstrList ------------------------------------------------ */ @@ -108,7 +110,8 @@ PyAPI_FUNC(_PyInitError) _PyCoreConfig_SetPathConfig( const _PyCoreConfig *config); PyAPI_FUNC(_PyInitError) _PyCoreConfig_Read(_PyCoreConfig *config, const _PyArgv *args); -PyAPI_FUNC(void) _PyCoreConfig_Write(const _PyCoreConfig *config); +PyAPI_FUNC(void) _PyCoreConfig_Write(const _PyCoreConfig *config, + _PyRuntimeState *runtime); /* --- Function used for testing ---------------------------------- */ diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h index f07bc427bb88..69709b10b8f8 100644 --- a/Include/internal/pycore_pylifecycle.h +++ b/Include/internal/pycore_pylifecycle.h @@ -34,10 +34,13 @@ extern _PyInitError _PyFaulthandler_Init(int enable); extern int _PyTraceMalloc_Init(int enable); extern PyObject * _PyBuiltin_Init(void); extern _PyInitError _PySys_Create( + _PyRuntimeState *runtime, PyInterpreterState *interp, PyObject **sysmod_p); extern _PyInitError _PySys_SetPreliminaryStderr(PyObject *sysdict); -extern int _PySys_InitMain(PyInterpreterState *interp); +extern int _PySys_InitMain( + _PyRuntimeState *runtime, + PyInterpreterState *interp); extern _PyInitError _PyImport_Init(PyInterpreterState *interp); extern _PyInitError _PyExc_Init(void); extern _PyInitError _PyBuiltins_AddExceptions(PyObject * bltinmod); @@ -74,7 +77,10 @@ extern void _PyFaulthandler_Fini(void); extern void _PyHash_Fini(void); extern int _PyTraceMalloc_Fini(void); -extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *); +extern void _PyGILState_Init( + _PyRuntimeState *runtime, + PyInterpreterState *interp, + PyThreadState *tstate); extern void _PyGILState_Fini(_PyRuntimeState *runtime); PyAPI_FUNC(void) _PyGC_DumpShutdownStats(void); diff --git a/Python/coreconfig.c b/Python/coreconfig.c index 7b55b06b7a6e..d05beef2aa55 100644 --- a/Python/coreconfig.c +++ b/Python/coreconfig.c @@ -1605,13 +1605,13 @@ config_init_stdio(const _PyCoreConfig *config) - set Py_xxx global configuration variables - initialize C standard streams (stdin, stdout, stderr) */ void -_PyCoreConfig_Write(const _PyCoreConfig *config) +_PyCoreConfig_Write(const _PyCoreConfig *config, _PyRuntimeState *runtime) { _PyCoreConfig_SetGlobalConfig(config); config_init_stdio(config); /* Write the new pre-configuration into _PyRuntime */ - _PyPreConfig *preconfig = &_PyRuntime.preconfig; + _PyPreConfig *preconfig = &runtime->preconfig; preconfig->isolated = config->isolated; preconfig->use_environment = config->use_environment; preconfig->dev_mode = config->dev_mode; diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index fe4cb97a4740..03486c3b195a 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -286,9 +286,9 @@ static const char *_C_LOCALE_WARNING = "locales is recommended.\n"; static void -_emit_stderr_warning_for_legacy_locale(void) +emit_stderr_warning_for_legacy_locale(_PyRuntimeState *runtime) { - const _PyPreConfig *preconfig = &_PyRuntime.preconfig; + const _PyPreConfig *preconfig = &runtime->preconfig; if (preconfig->coerce_c_locale_warn && _Py_LegacyLocaleDetected()) { PySys_FormatStderr("%s", _C_LOCALE_WARNING); } @@ -468,7 +468,8 @@ _Py_SetLocaleFromEnv(int category) */ static _PyInitError -_Py_Initialize_ReconfigureCore(PyInterpreterState **interp_p, +_Py_Initialize_ReconfigureCore(_PyRuntimeState *runtime, + PyInterpreterState **interp_p, const _PyCoreConfig *core_config) { PyThreadState *tstate = _PyThreadState_GET(); @@ -482,7 +483,7 @@ _Py_Initialize_ReconfigureCore(PyInterpreterState **interp_p, } *interp_p = interp; - _PyCoreConfig_Write(core_config); + _PyCoreConfig_Write(core_config, runtime); if (_PyCoreConfig_Copy(&interp->core_config, core_config) < 0) { return _Py_INIT_NO_MEMORY(); @@ -500,18 +501,14 @@ _Py_Initialize_ReconfigureCore(PyInterpreterState **interp_p, static _PyInitError -pycore_init_runtime(const _PyCoreConfig *core_config) +pycore_init_runtime(_PyRuntimeState *runtime, + const _PyCoreConfig *core_config) { - if (_PyRuntime.initialized) { + if (runtime->initialized) { return _Py_INIT_ERR("main interpreter already initialized"); } - _PyCoreConfig_Write(core_config); - - _PyInitError err = _PyRuntime_Initialize(); - if (_Py_INIT_FAILED(err)) { - return err; - } + _PyCoreConfig_Write(core_config, runtime); /* Py_Finalize leaves _Py_Finalizing set in order to help daemon * threads behave a little more gracefully at interpreter shutdown. @@ -522,14 +519,14 @@ pycore_init_runtime(const _PyCoreConfig *core_config) * threads still hanging around from a previous Py_Initialize/Finalize * pair :( */ - _PyRuntime.finalizing = NULL; + runtime->finalizing = NULL; - err = _Py_HashRandomization_Init(core_config); + _PyInitError err = _Py_HashRandomization_Init(core_config); if (_Py_INIT_FAILED(err)) { return err; } - err = _PyInterpreterState_Enable(&_PyRuntime); + err = _PyInterpreterState_Enable(runtime); if (_Py_INIT_FAILED(err)) { return err; } @@ -538,7 +535,8 @@ pycore_init_runtime(const _PyCoreConfig *core_config) static _PyInitError -pycore_create_interpreter(const _PyCoreConfig *core_config, +pycore_create_interpreter(_PyRuntimeState *runtime, + const _PyCoreConfig *core_config, PyInterpreterState **interp_p) { PyInterpreterState *interp = PyInterpreterState_New(); @@ -565,7 +563,7 @@ pycore_create_interpreter(const _PyCoreConfig *core_config, _PyEval_FiniThreads(); /* Auto-thread-state API */ - _PyGILState_Init(interp, tstate); + _PyGILState_Init(runtime, interp, tstate); /* Create the GIL */ PyEval_InitThreads(); @@ -671,19 +669,20 @@ pycore_init_import_warnings(PyInterpreterState *interp, PyObject *sysmod) static _PyInitError -_Py_InitializeCore_impl(PyInterpreterState **interp_p, +_Py_InitializeCore_impl(_PyRuntimeState *runtime, + PyInterpreterState **interp_p, const _PyCoreConfig *core_config) { PyInterpreterState *interp; - _PyCoreConfig_Write(core_config); + _PyCoreConfig_Write(core_config, runtime); - _PyInitError err = pycore_init_runtime(core_config); + _PyInitError err = pycore_init_runtime(runtime, core_config); if (_Py_INIT_FAILED(err)) { return err; } - err = pycore_create_interpreter(core_config, &interp); + err = pycore_create_interpreter(runtime, core_config, &interp); if (_Py_INIT_FAILED(err)) { return err; } @@ -696,7 +695,7 @@ _Py_InitializeCore_impl(PyInterpreterState **interp_p, } PyObject *sysmod; - err = _PySys_Create(interp, &sysmod); + err = _PySys_Create(runtime, interp, &sysmod); if (_Py_INIT_FAILED(err)) { return err; } @@ -712,7 +711,7 @@ _Py_InitializeCore_impl(PyInterpreterState **interp_p, } /* Only when we get here is the runtime core fully initialized */ - _PyRuntime.core_initialized = 1; + runtime->core_initialized = 1; return _Py_INIT_OK(); } @@ -726,8 +725,9 @@ preinit(const _PyPreConfig *src_config, const _PyArgv *args) if (_Py_INIT_FAILED(err)) { return err; } + _PyRuntimeState *runtime = &_PyRuntime; - if (_PyRuntime.pre_initialized) { + if (runtime->pre_initialized) { /* If it's already configured: ignored the new configuration */ return _Py_INIT_OK(); } @@ -751,7 +751,7 @@ preinit(const _PyPreConfig *src_config, const _PyArgv *args) goto done; } - _PyRuntime.pre_initialized = 1; + runtime->pre_initialized = 1; err = _Py_INIT_OK(); done: @@ -795,7 +795,8 @@ _Py_PreInitializeFromCoreConfig(const _PyCoreConfig *coreconfig) static _PyInitError -pyinit_coreconfig(_PyCoreConfig *config, +pyinit_coreconfig(_PyRuntimeState *runtime, + _PyCoreConfig *config, const _PyCoreConfig *src_config, const _PyArgv *args, PyInterpreterState **interp_p) @@ -811,11 +812,11 @@ pyinit_coreconfig(_PyCoreConfig *config, return err; } - if (!_PyRuntime.core_initialized) { - return _Py_InitializeCore_impl(interp_p, config); + if (!runtime->core_initialized) { + return _Py_InitializeCore_impl(runtime, interp_p, config); } else { - return _Py_Initialize_ReconfigureCore(interp_p, config); + return _Py_Initialize_ReconfigureCore(runtime, interp_p, config); } } @@ -838,7 +839,8 @@ pyinit_coreconfig(_PyCoreConfig *config, * safe to call without calling Py_Initialize first) */ static _PyInitError -_Py_InitializeCore(const _PyCoreConfig *src_config, +_Py_InitializeCore(_PyRuntimeState *runtime, + const _PyCoreConfig *src_config, const _PyArgv *args, PyInterpreterState **interp_p) { @@ -855,7 +857,7 @@ _Py_InitializeCore(const _PyCoreConfig *src_config, } _PyCoreConfig local_config = _PyCoreConfig_INIT; - err = pyinit_coreconfig(&local_config, src_config, args, interp_p); + err = pyinit_coreconfig(runtime, &local_config, src_config, args, interp_p); _PyCoreConfig_Clear(&local_config); return err; } @@ -894,16 +896,17 @@ _Py_ReconfigureMainInterpreter(PyInterpreterState *interp) * non-zero return code. */ static _PyInitError -_Py_InitializeMainInterpreter(PyInterpreterState *interp) +_Py_InitializeMainInterpreter(_PyRuntimeState *runtime, + PyInterpreterState *interp) { - if (!_PyRuntime.core_initialized) { + if (!runtime->core_initialized) { return _Py_INIT_ERR("runtime core not initialized"); } /* Configure the main interpreter */ _PyCoreConfig *core_config = &interp->core_config; - if (_PyRuntime.initialized) { + if (runtime->initialized) { return _Py_ReconfigureMainInterpreter(interp); } @@ -913,7 +916,7 @@ _Py_InitializeMainInterpreter(PyInterpreterState *interp) * This means anything which needs support from extension modules * or pure Python code in the standard library won't work. */ - _PyRuntime.initialized = 1; + runtime->initialized = 1; return _Py_INIT_OK(); } @@ -921,7 +924,7 @@ _Py_InitializeMainInterpreter(PyInterpreterState *interp) return _Py_INIT_ERR("can't initialize time"); } - if (_PySys_InitMain(interp) < 0) { + if (_PySys_InitMain(runtime, interp) < 0) { return _Py_INIT_ERR("can't finish initializing sys"); } @@ -974,7 +977,7 @@ _Py_InitializeMainInterpreter(PyInterpreterState *interp) Py_XDECREF(warnings_module); } - _PyRuntime.initialized = 1; + runtime->initialized = 1; if (core_config->site_import) { err = initsite(); /* Module site */ @@ -984,7 +987,7 @@ _Py_InitializeMainInterpreter(PyInterpreterState *interp) } #ifndef MS_WINDOWS - _emit_stderr_warning_for_legacy_locale(); + emit_stderr_warning_for_legacy_locale(runtime); #endif return _Py_INIT_OK(); @@ -995,16 +998,23 @@ _Py_InitializeMainInterpreter(PyInterpreterState *interp) static _PyInitError init_python(const _PyCoreConfig *config, const _PyArgv *args) { - PyInterpreterState *interp = NULL; _PyInitError err; - err = _Py_InitializeCore(config, args, &interp); + + err = _PyRuntime_Initialize(); + if (_Py_INIT_FAILED(err)) { + return err; + } + _PyRuntimeState *runtime = &_PyRuntime; + + PyInterpreterState *interp = NULL; + err = _Py_InitializeCore(runtime, config, args, &interp); if (_Py_INIT_FAILED(err)) { return err; } config = &interp->core_config; if (config->_init_main) { - err = _Py_InitializeMainInterpreter(interp); + err = _Py_InitializeMainInterpreter(runtime, interp); if (_Py_INIT_FAILED(err)) { return err; } @@ -1040,7 +1050,15 @@ _Py_InitializeFromConfig(const _PyCoreConfig *config) void Py_InitializeEx(int install_sigs) { - if (_PyRuntime.initialized) { + _PyInitError err; + + err = _PyRuntime_Initialize(); + if (_Py_INIT_FAILED(err)) { + _Py_ExitInitError(err); + } + _PyRuntimeState *runtime = &_PyRuntime; + + if (runtime->initialized) { /* bpo-33932: Calling Py_Initialize() twice does nothing. */ return; } @@ -1048,7 +1066,7 @@ Py_InitializeEx(int install_sigs) _PyCoreConfig config = _PyCoreConfig_INIT; config.install_signal_handlers = install_sigs; - _PyInitError err = _Py_InitializeFromConfig(&config); + err = _Py_InitializeFromConfig(&config); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } @@ -1364,12 +1382,15 @@ Py_Finalize(void) static _PyInitError new_interpreter(PyThreadState **tstate_p) { - PyInterpreterState *interp; - PyThreadState *tstate, *save_tstate; - PyObject *bimod, *sysmod; _PyInitError err; - if (!_PyRuntime.initialized) { + err = _PyRuntime_Initialize(); + if (_Py_INIT_FAILED(err)) { + return err; + } + _PyRuntimeState *runtime = &_PyRuntime; + + if (!runtime->initialized) { return _Py_INIT_ERR("Py_Initialize must be called first"); } @@ -1377,20 +1398,20 @@ new_interpreter(PyThreadState **tstate_p) interpreters: disable PyGILState_Check(). */ _PyGILState_check_enabled = 0; - interp = PyInterpreterState_New(); + PyInterpreterState *interp = PyInterpreterState_New(); if (interp == NULL) { *tstate_p = NULL; return _Py_INIT_OK(); } - tstate = PyThreadState_New(interp); + PyThreadState *tstate = PyThreadState_New(interp); if (tstate == NULL) { PyInterpreterState_Delete(interp); *tstate_p = NULL; return _Py_INIT_OK(); } - save_tstate = PyThreadState_Swap(tstate); + PyThreadState *save_tstate = PyThreadState_Swap(tstate); /* Copy the current interpreter config into the new interpreter */ _PyCoreConfig *core_config; @@ -1419,14 +1440,15 @@ new_interpreter(PyThreadState **tstate_p) } interp->modules = modules; - sysmod = _PyImport_FindBuiltin("sys", modules); + PyObject *sysmod = _PyImport_FindBuiltin("sys", modules); if (sysmod != NULL) { interp->sysdict = PyModule_GetDict(sysmod); - if (interp->sysdict == NULL) + if (interp->sysdict == NULL) { goto handle_error; + } Py_INCREF(interp->sysdict); PyDict_SetItemString(interp->sysdict, "modules", modules); - if (_PySys_InitMain(interp) < 0) { + if (_PySys_InitMain(runtime, interp) < 0) { return _Py_INIT_ERR("can't finish initializing sys"); } } @@ -1434,7 +1456,7 @@ new_interpreter(PyThreadState **tstate_p) goto handle_error; } - bimod = _PyImport_FindBuiltin("builtins", modules); + PyObject *bimod = _PyImport_FindBuiltin("builtins", modules); if (bimod != NULL) { interp->builtins = PyModule_GetDict(bimod); if (interp->builtins == NULL) diff --git a/Python/pystate.c b/Python/pystate.c index 498a9546e1b8..a413f9d7ab4c 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1052,13 +1052,13 @@ PyThreadState_IsCurrent(PyThreadState *tstate) Py_Initialize/Py_FinalizeEx */ void -_PyGILState_Init(PyInterpreterState *interp, PyThreadState *tstate) +_PyGILState_Init(_PyRuntimeState *runtime, + PyInterpreterState *interp, PyThreadState *tstate) { /* must init with valid states */ assert(interp != NULL); assert(tstate != NULL); - _PyRuntimeState *runtime = &_PyRuntime; struct _gilstate_runtime_state *gilstate = &runtime->gilstate; if (PyThread_tss_create(&gilstate->autoTSSkey) != 0) { diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 3de94e8468be..0f7af2c69da5 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -2155,12 +2155,12 @@ static PyStructSequence_Desc flags_desc = { }; static PyObject* -make_flags(void) +make_flags(_PyRuntimeState *runtime, PyInterpreterState *interp) { int pos = 0; PyObject *seq; - const _PyPreConfig *preconfig = &_PyRuntime.preconfig; - const _PyCoreConfig *config = &_PyInterpreterState_GET_UNSAFE()->core_config; + const _PyPreConfig *preconfig = &runtime->preconfig; + const _PyCoreConfig *config = &interp->core_config; seq = PyStructSequence_New(&FlagsType); if (seq == NULL) @@ -2375,7 +2375,8 @@ static struct PyModuleDef sysmodule = { } while (0) static _PyInitError -_PySys_InitCore(PyObject *sysdict) +_PySys_InitCore(_PyRuntimeState *runtime, PyInterpreterState *interp, + PyObject *sysdict) { PyObject *version_info; int res; @@ -2465,8 +2466,8 @@ _PySys_InitCore(PyObject *sysdict) goto type_init_failed; } } - /* Set flags to their default values */ - SET_SYS_FROM_STRING("flags", make_flags()); + /* Set flags to their default values (updated by _PySys_InitMain()) */ + SET_SYS_FROM_STRING("flags", make_flags(runtime, interp)); #if defined(MS_WINDOWS) /* getwindowsversion */ @@ -2587,7 +2588,7 @@ sys_create_xoptions_dict(const _PyCoreConfig *config) int -_PySys_InitMain(PyInterpreterState *interp) +_PySys_InitMain(_PyRuntimeState *runtime, PyInterpreterState *interp) { PyObject *sysdict = interp->sysdict; const _PyCoreConfig *config = &interp->core_config; @@ -2641,7 +2642,7 @@ _PySys_InitMain(PyInterpreterState *interp) #undef SET_SYS_FROM_WSTR /* Set flags to their final values */ - SET_SYS_FROM_STRING_INT_RESULT("flags", make_flags()); + SET_SYS_FROM_STRING_INT_RESULT("flags", make_flags(runtime, interp)); /* prevent user from creating new instances */ FlagsType.tp_init = NULL; FlagsType.tp_new = NULL; @@ -2708,7 +2709,8 @@ _PySys_SetPreliminaryStderr(PyObject *sysdict) /* Create sys module without all attributes: _PySys_InitMain() should be called later to add remaining attributes. */ _PyInitError -_PySys_Create(PyInterpreterState *interp, PyObject **sysmod_p) +_PySys_Create(_PyRuntimeState *runtime, PyInterpreterState *interp, + PyObject **sysmod_p) { PyObject *modules = PyDict_New(); if (modules == NULL) { @@ -2737,7 +2739,7 @@ _PySys_Create(PyInterpreterState *interp, PyObject **sysmod_p) return err; } - err = _PySys_InitCore(sysdict); + err = _PySys_InitCore(runtime, interp, sysdict); if (_Py_INIT_FAILED(err)) { return err; }

1 0

bpo-30840: Document relative imports (GH-12831) (GH-12938)
by Nick Coghlan April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/4d0233ec656bc7e7814e5f6f484e79a50a… commit: 4d0233ec656bc7e7814e5f6f484e79a50a0daf91 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Nick Coghlan <ncoghlan(a)gmail.com> date: 2019-04-25T01:44:19+10:00 summary: bpo-30840: Document relative imports (GH-12831) (GH-12938) * Document relative imports (cherry picked from commit 70bf713617e15fad390ed953e48b3c65d9bc90ec) Co-authored-by: Joannah Nanjekye <33177550+nanjekyejoannah(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst M Doc/reference/import.rst M Doc/reference/simple_stmts.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 9a0ab39d3b4a..88290c88bb35 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -921,6 +921,46 @@ it is sufficient to raise :exc:`ModuleNotFoundError` directly from ``None``. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately. +.. _relativeimports: + +Package Relative Imports +======================== + +Relative imports use leading dots. A single leading dot indicates a relative +import, starting with the current package. Two or more leading dots indicate a +relative import to the parent(s) of the current package, one level per dot +after the first. For example, given the following package layout:: + + package/ + __init__.py + subpackage1/ + __init__.py + moduleX.py + moduleY.py + subpackage2/ + __init__.py + moduleZ.py + moduleA.py + +In either ``subpackage1/moduleX.py`` or ``subpackage1/__init__.py``, +the following are valid relative imports:: + + from .moduleY import spam + from .moduleY import spam as ham + from . import moduleY + from ..subpackage1 import moduleY + from ..subpackage2.moduleZ import eggs + from ..moduleA import foo + +Absolute imports may use either the ``import <>`` or ``from <> import <>`` +syntax, but relative imports may only use the second form; the reason +for this is that:: + + import XXX.YYY.ZZZ + +should expose ``XXX.YYY.ZZZ`` as a usable expression, but .moduleY is +not a valid expression. + Special considerations for __main__ =================================== diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 07072d954384..1779832a9fdd 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -829,7 +829,8 @@ exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. -The specification for relative imports is contained within :pep:`328`. +The specification for relative imports is contained in +the :ref:`relativeimports` section. :func:`importlib.import_module` is provided to support applications that determine dynamically the modules to be loaded. diff --git a/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst b/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst new file mode 100644 index 000000000000..210f54f2593e --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst @@ -0,0 +1 @@ +Document relative imports \ No newline at end of file

1 0

bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
by Victor Stinner April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/8e91c246e468515b877690e090c73f4965… commit: 8e91c246e468515b877690e090c73f496552541d branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-24T17:24:01+02:00 summary: bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937) * Add a 'runtime' variable to Py_FinalizeEx() rather than working directly on the global variable _PyRuntime * Add a 'runtime' parameter to _PyGC_Fini(), _PyGILState_Fini() and call_ll_exitfuncs() files: M Include/internal/pycore_pylifecycle.h M Modules/gcmodule.c M Python/pylifecycle.c M Python/pystate.c diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h index bfff24b80a9f..f07bc427bb88 100644 --- a/Include/internal/pycore_pylifecycle.h +++ b/Include/internal/pycore_pylifecycle.h @@ -8,6 +8,8 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif +#include "pycore_pystate.h" /* _PyRuntimeState */ + /* True if the main interpreter thread exited due to an unhandled * KeyboardInterrupt exception, suggesting the user pressed ^C. */ PyAPI_DATA(int) _Py_UnhandledKeyboardInterrupt; @@ -63,7 +65,7 @@ extern void PyAsyncGen_Fini(void); extern void _PyExc_Fini(void); extern void _PyImport_Fini(void); extern void _PyImport_Fini2(void); -extern void _PyGC_Fini(void); +extern void _PyGC_Fini(_PyRuntimeState *runtime); extern void _PyType_Fini(void); extern void _Py_HashRandomization_Fini(void); extern void _PyUnicode_Fini(void); @@ -73,7 +75,7 @@ extern void _PyHash_Fini(void); extern int _PyTraceMalloc_Fini(void); extern void _PyGILState_Init(PyInterpreterState *, PyThreadState *); -extern void _PyGILState_Fini(void); +extern void _PyGILState_Fini(_PyRuntimeState *runtime); PyAPI_FUNC(void) _PyGC_DumpShutdownStats(void); diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c index a75d5fed95f1..f36c7f5d5e40 100644 --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -1865,9 +1865,10 @@ _PyGC_DumpShutdownStats(void) } void -_PyGC_Fini(void) +_PyGC_Fini(_PyRuntimeState *runtime) { - Py_CLEAR(_PyRuntime.gc.callbacks); + struct _gc_runtime_state *gc = &runtime->gc; + Py_CLEAR(gc->callbacks); } /* for debugging */ diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index c7920ef6262d..fe4cb97a4740 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -65,7 +65,7 @@ static _PyInitError init_sys_streams(PyInterpreterState *interp); static _PyInitError initsigs(void); static void call_py_exitfuncs(PyInterpreterState *); static void wait_for_thread_shutdown(void); -static void call_ll_exitfuncs(void); +static void call_ll_exitfuncs(_PyRuntimeState *runtime); int _Py_UnhandledKeyboardInterrupt = 0; _PyRuntimeState _PyRuntime = _PyRuntimeState_INIT; @@ -1131,23 +1131,23 @@ flush_std_files(void) int Py_FinalizeEx(void) { - PyInterpreterState *interp; - PyThreadState *tstate; int status = 0; - if (!_PyRuntime.initialized) + _PyRuntimeState *runtime = &_PyRuntime; + if (!runtime->initialized) { return status; + } // Wrap up existing "threading"-module-created, non-daemon threads. wait_for_thread_shutdown(); - /* Get current thread state and interpreter pointer */ - tstate = _PyThreadState_GET(); - interp = tstate->interp; - // Make any remaining pending calls. _Py_FinishPendingCalls(); + /* Get current thread state and interpreter pointer */ + PyThreadState *tstate = _PyThreadState_GET(); + PyInterpreterState *interp = tstate->interp; + /* The interpreter is still entirely intact at this point, and the * exit funcs may be relying on that. In particular, if some thread * or exit func is still waiting to do an import, the import machinery @@ -1174,9 +1174,9 @@ Py_FinalizeEx(void) /* Remaining threads (e.g. daemon threads) will automatically exit after taking the GIL (in PyEval_RestoreThread()). */ - _PyRuntime.finalizing = tstate; - _PyRuntime.initialized = 0; - _PyRuntime.core_initialized = 0; + runtime->finalizing = tstate; + runtime->initialized = 0; + runtime->core_initialized = 0; /* Flush sys.stdout and sys.stderr */ if (flush_std_files() < 0) { @@ -1294,7 +1294,7 @@ Py_FinalizeEx(void) PyFloat_Fini(); PyDict_Fini(); PySlice_Fini(); - _PyGC_Fini(); + _PyGC_Fini(runtime); _Py_HashRandomization_Fini(); _PyArg_Fini(); PyAsyncGen_Fini(); @@ -1314,7 +1314,7 @@ Py_FinalizeEx(void) PyGrammar_RemoveAccelerators(&_PyParser_Grammar); /* Cleanup auto-thread-state */ - _PyGILState_Fini(); + _PyGILState_Fini(runtime); /* Delete current thread. After this, many C API calls become crashy. */ PyThreadState_Swap(NULL); @@ -1336,7 +1336,7 @@ Py_FinalizeEx(void) } #endif - call_ll_exitfuncs(); + call_ll_exitfuncs(runtime); _PyRuntime_Finalize(); return status; @@ -2223,10 +2223,11 @@ int Py_AtExit(void (*func)(void)) } static void -call_ll_exitfuncs(void) +call_ll_exitfuncs(_PyRuntimeState *runtime) { - while (_PyRuntime.nexitfuncs > 0) - (*_PyRuntime.exitfuncs[--_PyRuntime.nexitfuncs])(); + while (runtime->nexitfuncs > 0) { + (*runtime->exitfuncs[--runtime->nexitfuncs])(); + } fflush(stdout); fflush(stderr); diff --git a/Python/pystate.c b/Python/pystate.c index 6aaf993cfe2c..498a9546e1b8 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1078,10 +1078,11 @@ _PyGILState_GetInterpreterStateUnsafe(void) } void -_PyGILState_Fini(void) +_PyGILState_Fini(_PyRuntimeState *runtime) { - PyThread_tss_delete(&_PyRuntime.gilstate.autoTSSkey); - _PyRuntime.gilstate.autoInterpreterState = NULL; + struct _gilstate_runtime_state *gilstate = &runtime->gilstate; + PyThread_tss_delete(&gilstate->autoTSSkey); + gilstate->autoInterpreterState = NULL; } /* Reset the TSS key - called by PyOS_AfterFork_Child().

1 0

bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
by Victor Stinner April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/b930a2d2b1247bdba560db341ba90a9cbb… commit: b930a2d2b1247bdba560db341ba90a9cbb538eb3 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-24T17:14:33+02:00 summary: bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936) The PyOS_AfterFork_Child() function now pass a 'runtime' parameter to subfunctions. * Fix _PyRuntimeState_ReInitThreads(): use the correct memory allocator * Add runtime parameter to _PyRuntimeState_ReInitThreads(), _PyGILState_Reinit() and _PyInterpreterState_DeleteExceptMain() * Move _PyGILState_Reinit() to the internal C API. files: M Include/cpython/pystate.h M Include/internal/pycore_pystate.h M Modules/posixmodule.c M Python/pystate.c diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index 2341dda3fabd..94331f35e1bd 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -155,7 +155,6 @@ PyAPI_FUNC(PyInterpreterState *) _PyInterpreterState_Get(void); PyAPI_FUNC(int) _PyState_AddModule(PyObject*, struct PyModuleDef*); PyAPI_FUNC(void) _PyState_ClearModules(void); PyAPI_FUNC(PyThreadState *) _PyThreadState_Prealloc(PyInterpreterState *); -PyAPI_FUNC(void) _PyGILState_Reinit(void); /* Similar to PyThreadState_Get(), but don't issue a fatal error * if it is NULL. */ diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index 509ca3634a3a..2c24f679dc02 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -185,9 +185,9 @@ typedef struct pyruntimestate { /* Note: _PyRuntimeState_INIT sets other fields to 0/NULL */ PyAPI_DATA(_PyRuntimeState) _PyRuntime; -PyAPI_FUNC(_PyInitError) _PyRuntimeState_Init(_PyRuntimeState *); -PyAPI_FUNC(void) _PyRuntimeState_Fini(_PyRuntimeState *); -PyAPI_FUNC(void) _PyRuntimeState_ReInitThreads(void); +PyAPI_FUNC(_PyInitError) _PyRuntimeState_Init(_PyRuntimeState *runtime); +PyAPI_FUNC(void) _PyRuntimeState_Fini(_PyRuntimeState *runtime); +PyAPI_FUNC(void) _PyRuntimeState_ReInitThreads(_PyRuntimeState *runtime); /* Initialize _PyRuntimeState. Return NULL on success, or return an error message on failure. */ @@ -236,8 +236,10 @@ PyAPI_FUNC(void) _PyThreadState_Init( PyThreadState *tstate); PyAPI_FUNC(void) _PyThreadState_DeleteExcept(PyThreadState *tstate); -PyAPI_FUNC(_PyInitError) _PyInterpreterState_Enable(_PyRuntimeState *); -PyAPI_FUNC(void) _PyInterpreterState_DeleteExceptMain(void); +PyAPI_FUNC(_PyInitError) _PyInterpreterState_Enable(_PyRuntimeState *runtime); +PyAPI_FUNC(void) _PyInterpreterState_DeleteExceptMain(_PyRuntimeState *runtime); + +PyAPI_FUNC(void) _PyGILState_Reinit(_PyRuntimeState *runtime); #ifdef __cplusplus } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index e8dbdcc94aa7..56ec3ee5a0ee 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -421,12 +421,13 @@ PyOS_AfterFork_Parent(void) void PyOS_AfterFork_Child(void) { - _PyGILState_Reinit(); - _PyInterpreterState_DeleteExceptMain(); + _PyRuntimeState *runtime = &_PyRuntime; + _PyGILState_Reinit(runtime); + _PyInterpreterState_DeleteExceptMain(runtime); PyEval_ReInitThreads(); _PyImport_ReInitLock(); _PySignal_AfterFork(); - _PyRuntimeState_ReInitThreads(); + _PyRuntimeState_ReInitThreads(runtime); run_at_forkers(_PyInterpreterState_Get()->after_forkers_child, 0); } diff --git a/Python/pystate.c b/Python/pystate.c index ef9d79288736..6aaf993cfe2c 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -108,23 +108,31 @@ _PyRuntimeState_Fini(_PyRuntimeState *runtime) */ void -_PyRuntimeState_ReInitThreads(void) +_PyRuntimeState_ReInitThreads(_PyRuntimeState *runtime) { // This was initially set in _PyRuntimeState_Init(). - _PyRuntime.main_thread = PyThread_get_thread_ident(); + runtime->main_thread = PyThread_get_thread_ident(); + + /* Force default allocator, since _PyRuntimeState_Fini() must + use the same allocator than this function. */ + PyMemAllocatorEx old_alloc; + _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc); + + runtime->interpreters.mutex = PyThread_allocate_lock(); + runtime->interpreters.main->id_mutex = PyThread_allocate_lock(); + runtime->xidregistry.mutex = PyThread_allocate_lock(); + + PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc); - _PyRuntime.interpreters.mutex = PyThread_allocate_lock(); - if (_PyRuntime.interpreters.mutex == NULL) { + if (runtime->interpreters.mutex == NULL) { Py_FatalError("Can't initialize lock for runtime interpreters"); } - _PyRuntime.interpreters.main->id_mutex = PyThread_allocate_lock(); - if (_PyRuntime.interpreters.main->id_mutex == NULL) { + if (runtime->interpreters.main->id_mutex == NULL) { Py_FatalError("Can't initialize ID lock for main interpreter"); } - _PyRuntime.xidregistry.mutex = PyThread_allocate_lock(); - if (_PyRuntime.xidregistry.mutex == NULL) { + if (runtime->xidregistry.mutex == NULL) { Py_FatalError("Can't initialize lock for cross-interpreter data registry"); } } @@ -290,20 +298,22 @@ PyInterpreterState_Delete(PyInterpreterState *interp) * is a current interpreter state, it *must* be the main interpreter. */ void -_PyInterpreterState_DeleteExceptMain() +_PyInterpreterState_DeleteExceptMain(_PyRuntimeState *runtime) { + struct pyinterpreters *interpreters = &runtime->interpreters; + PyThreadState *tstate = PyThreadState_Swap(NULL); - if (tstate != NULL && tstate->interp != _PyRuntime.interpreters.main) { + if (tstate != NULL && tstate->interp != interpreters->main) { Py_FatalError("PyInterpreterState_DeleteExceptMain: not main interpreter"); } HEAD_LOCK(); - PyInterpreterState *interp = _PyRuntime.interpreters.head; - _PyRuntime.interpreters.head = NULL; + PyInterpreterState *interp = interpreters->head; + interpreters->head = NULL; while (interp != NULL) { - if (interp == _PyRuntime.interpreters.main) { - _PyRuntime.interpreters.main->next = NULL; - _PyRuntime.interpreters.head = interp; + if (interp == interpreters->main) { + interpreters->main->next = NULL; + interpreters->head = interp; interp = interp->next; continue; } @@ -319,7 +329,7 @@ _PyInterpreterState_DeleteExceptMain() } HEAD_UNLOCK(); - if (_PyRuntime.interpreters.head == NULL) { + if (interpreters->head == NULL) { Py_FatalError("PyInterpreterState_DeleteExceptMain: missing main"); } PyThreadState_Swap(tstate); @@ -1079,31 +1089,20 @@ _PyGILState_Fini(void) * don't reset TSS upon fork(), see issue #10517. */ void -_PyGILState_Reinit(void) +_PyGILState_Reinit(_PyRuntimeState *runtime) { - /* Force default allocator, since _PyRuntimeState_Fini() must - use the same allocator than this function. */ - PyMemAllocatorEx old_alloc; - _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc); - - _PyRuntime.interpreters.mutex = PyThread_allocate_lock(); - - PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc); - - if (_PyRuntime.interpreters.mutex == NULL) { - Py_FatalError("Can't initialize threads for interpreter"); - } - + struct _gilstate_runtime_state *gilstate = &runtime->gilstate; PyThreadState *tstate = PyGILState_GetThisThreadState(); - PyThread_tss_delete(&_PyRuntime.gilstate.autoTSSkey); - if (PyThread_tss_create(&_PyRuntime.gilstate.autoTSSkey) != 0) { + + PyThread_tss_delete(&gilstate->autoTSSkey); + if (PyThread_tss_create(&gilstate->autoTSSkey) != 0) { Py_FatalError("Could not allocate TSS entry"); } /* If the thread had an associated auto thread state, reassociate it with * the new key. */ if (tstate && - PyThread_tss_set(&_PyRuntime.gilstate.autoTSSkey, (void *)tstate) != 0) + PyThread_tss_set(&gilstate->autoTSSkey, (void *)tstate) != 0) { Py_FatalError("Couldn't create autoTSSkey mapping"); }

1 0

bpo-30840: Document relative imports (#12831)
by Nick Coghlan April 24, 2019

April 24, 2019

https://github.com/python/cpython/commit/70bf713617e15fad390ed953e48b3c65d9… commit: 70bf713617e15fad390ed953e48b3c65d9bc90ec branch: master author: Joannah Nanjekye <33177550+nanjekyejoannah(a)users.noreply.github.com> committer: Nick Coghlan <ncoghlan(a)gmail.com> date: 2019-04-25T01:14:44+10:00 summary: bpo-30840: Document relative imports (#12831) * document relative imports * 📜🤖 Added by blurb_it. * fix indentation error * remove indentation * Document relative imports * Document relative imports * remove from ...package * Document relative imports * remove trailing space * Document relative imports * Document relative imports files: A Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst M Doc/reference/import.rst M Doc/reference/simple_stmts.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 9a0ab39d3b4a..88290c88bb35 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -921,6 +921,46 @@ it is sufficient to raise :exc:`ModuleNotFoundError` directly from ``None``. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately. +.. _relativeimports: + +Package Relative Imports +======================== + +Relative imports use leading dots. A single leading dot indicates a relative +import, starting with the current package. Two or more leading dots indicate a +relative import to the parent(s) of the current package, one level per dot +after the first. For example, given the following package layout:: + + package/ + __init__.py + subpackage1/ + __init__.py + moduleX.py + moduleY.py + subpackage2/ + __init__.py + moduleZ.py + moduleA.py + +In either ``subpackage1/moduleX.py`` or ``subpackage1/__init__.py``, +the following are valid relative imports:: + + from .moduleY import spam + from .moduleY import spam as ham + from . import moduleY + from ..subpackage1 import moduleY + from ..subpackage2.moduleZ import eggs + from ..moduleA import foo + +Absolute imports may use either the ``import <>`` or ``from <> import <>`` +syntax, but relative imports may only use the second form; the reason +for this is that:: + + import XXX.YYY.ZZZ + +should expose ``XXX.YYY.ZZZ`` as a usable expression, but .moduleY is +not a valid expression. + Special considerations for __main__ =================================== diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 00964afc6d3d..207057cbc124 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -828,7 +828,8 @@ exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. -The specification for relative imports is contained within :pep:`328`. +The specification for relative imports is contained in +the :ref:`relativeimports` section. :func:`importlib.import_module` is provided to support applications that determine dynamically the modules to be loaded. diff --git a/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst b/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst new file mode 100644 index 000000000000..210f54f2593e --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-04-14-19-46-21.bpo-30840.R-JFzw.rst @@ -0,0 +1 @@ +Document relative imports \ No newline at end of file

1 0