Mailman 3 August 2022 - Python-checkins

gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231)
by miss-islington Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/83ff85bc593fb34a2bd81fd00656d6dc0f… commit: 83ff85bc593fb34a2bd81fd00656d6dc0fee06fa branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2022-08-24T07:20:38-07:00 summary: gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231) (cherry picked from commit 09563a764ebc54f98087c128419f46cf0822b4b7) Co-authored-by: Dong-hee Na <donghee.na(a)python.org> files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index d22cc2812dc..ba5d2cf8033 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -164,6 +164,8 @@ are always available. They are listed here in alphabetical order. :func:`sys.breakpointhook` can be set to some other function and :func:`breakpoint` will automatically call that, allowing you to drop into the debugger of choice. + If :func:`sys.breakpointhook` is not available to be called, this function will + raise :exc:`RuntimeError`. .. audit-event:: builtins.breakpoint breakpointhook breakpoint

1 0

gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231)
by miss-islington Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/bf2728b9f1a9873f22382c72f21adb9804… commit: bf2728b9f1a9873f22382c72f21adb98046f613c branch: 3.11 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2022-08-24T07:20:22-07:00 summary: gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231) (cherry picked from commit 09563a764ebc54f98087c128419f46cf0822b4b7) Co-authored-by: Dong-hee Na <donghee.na(a)python.org> files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index da7de18722f..fbde9129274 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -164,6 +164,8 @@ are always available. They are listed here in alphabetical order. :func:`sys.breakpointhook` can be set to some other function and :func:`breakpoint` will automatically call that, allowing you to drop into the debugger of choice. + If :func:`sys.breakpointhook` is not available to be called, this function will + raise :exc:`RuntimeError`. .. audit-event:: builtins.breakpoint breakpointhook breakpoint

1 0

gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231)
by corona10 Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/09563a764ebc54f98087c128419f46cf08… commit: 09563a764ebc54f98087c128419f46cf0822b4b7 branch: main author: Dong-hee Na <donghee.na(a)python.org> committer: corona10 <donghee.na92(a)gmail.com> date: 2022-08-24T23:03:36+09:00 summary: gh-96197: Define the behavior of breakpoint if sys.breakpointhook is lost (gh-96231) files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index da7de18722f..fbde9129274 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -164,6 +164,8 @@ are always available. They are listed here in alphabetical order. :func:`sys.breakpointhook` can be set to some other function and :func:`breakpoint` will automatically call that, allowing you to drop into the debugger of choice. + If :func:`sys.breakpointhook` is not available to be called, this function will + raise :exc:`RuntimeError`. .. audit-event:: builtins.breakpoint breakpointhook breakpoint

1 0

GH-96177: Move GIL and eval breaker code out of ceval.c into ceval_gil.c. (GH-96204)
by markshannon Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/a4a9f2e879c0c9572e0cecbc702dc1dd31… commit: a4a9f2e879c0c9572e0cecbc702dc1dd31f80221 branch: main author: Mark Shannon <mark(a)hotpy.org> committer: markshannon <mark(a)hotpy.org> date: 2022-08-24T14:21:01+01:00 summary: GH-96177: Move GIL and eval breaker code out of ceval.c into ceval_gil.c. (GH-96204) files: A Python/ceval_gil.c D Python/ceval_gil.h M Include/internal/pycore_ceval.h M Include/internal/pycore_pystate.h M Makefile.pre.in M PCbuild/_freeze_module.vcxproj M PCbuild/_freeze_module.vcxproj.filters M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/ceval.c M Tools/c-analyzer/cpython/_parser.py M Tools/gdb/libpython.py diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h index 1b999301938c..2fcdaad358b0 100644 --- a/Include/internal/pycore_ceval.h +++ b/Include/internal/pycore_ceval.h @@ -133,6 +133,9 @@ extern struct _PyInterpreterFrame* _PyEval_GetFrame(void); extern PyObject* _Py_MakeCoro(PyFunctionObject *func); +extern int _Py_HandlePending(PyThreadState *tstate); + + #ifdef __cplusplus } #endif diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index 51d119c23e73..3d6d400f74dd 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -85,13 +85,14 @@ _PyThreadState_GET(void) return _PyRuntimeState_GetThreadState(&_PyRuntime); } -PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalError_TstateNULL(const char *func); - static inline void _Py_EnsureFuncTstateNotNULL(const char *func, PyThreadState *tstate) { if (tstate == NULL) { - _Py_FatalError_TstateNULL(func); + _Py_FatalErrorFunc(func, + "the function must be called with the GIL held, " + "after Python initialization and before Python finalization, " + "but the GIL is released (the current Python thread state is NULL)"); } } diff --git a/Makefile.pre.in b/Makefile.pre.in index 414e6045b4d6..3491e91f43ab 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -388,6 +388,7 @@ PYTHON_OBJS= \ Python/getcopyright.o \ Python/getplatform.o \ Python/getversion.o \ + Python/ceval_gil.o \ Python/hamt.o \ Python/hashtable.o \ Python/import.o \ @@ -1419,8 +1420,7 @@ regen-opcode-targets: $(srcdir)/Python/opcode_targets.h.new $(UPDATE_FILE) $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/opcode_targets.h.new -Python/ceval.o: $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/ceval_gil.h \ - $(srcdir)/Python/condvar.h +Python/ceval.o: $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/condvar.h Python/frozen.o: $(FROZEN_FILES_OUT) diff --git a/PCbuild/_freeze_module.vcxproj b/PCbuild/_freeze_module.vcxproj index 5821c3d9e4d8..4c0072971123 100644 --- a/PCbuild/_freeze_module.vcxproj +++ b/PCbuild/_freeze_module.vcxproj @@ -199,6 +199,7 @@ <ClCompile Include="..\Python\getopt.c" /> <ClCompile Include="..\Python\getplatform.c" /> <ClCompile Include="..\Python\getversion.c" /> + <ClCompile Include="..\Python\ceval_gil.c" /> <ClCompile Include="..\Python\hamt.c" /> <ClCompile Include="..\Python\hashtable.c" /> <ClCompile Include="..\Python\import.c" /> diff --git a/PCbuild/_freeze_module.vcxproj.filters b/PCbuild/_freeze_module.vcxproj.filters index b657f56e28f2..5c984999c0cd 100644 --- a/PCbuild/_freeze_module.vcxproj.filters +++ b/PCbuild/_freeze_module.vcxproj.filters @@ -184,6 +184,9 @@ <ClCompile Include="..\Python\getversion.c"> <Filter>Source Files</Filter> </ClCompile> + <ClCompile Include="..\Python\ceval_gil.c"> + <Filter>Source Files</Filter> + </ClCompile> <ClCompile Include="..\Python\hamt.c"> <Filter>Source Files</Filter> </ClCompile> diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 3ff4be518724..45e5013e61f6 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -327,7 +327,6 @@ <ClInclude Include="..\Parser\pegen.h" /> <ClInclude Include="..\PC\errmap.h" /> <ClInclude Include="..\PC\pyconfig.h" /> - <ClInclude Include="..\Python\ceval_gil.h" /> <ClInclude Include="..\Python\condvar.h" /> <ClInclude Include="..\Python\importdl.h" /> <ClInclude Include="..\Python\stdlib_module_names.h" /> @@ -502,6 +501,7 @@ <ClCompile Include="..\Python\getopt.c" /> <ClCompile Include="..\Python\getplatform.c" /> <ClCompile Include="..\Python\getversion.c" /> + <ClCompile Include="..\Python\ceval_gil.c" /> <ClCompile Include="..\Python\hamt.c" /> <ClCompile Include="..\Python\hashtable.c" /> <ClCompile Include="..\Python\import.c" /> diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 64d248dfafd8..581ea6e3c58e 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -312,9 +312,6 @@ <ClInclude Include="..\Python\condvar.h"> <Filter>Python</Filter> </ClInclude> - <ClInclude Include="..\Python\ceval_gil.h"> - <Filter>Python</Filter> - </ClInclude> <ClInclude Include="..\Include\pyhash.h"> <Filter>Include</Filter> </ClInclude> @@ -1097,6 +1094,9 @@ <ClCompile Include="..\Python\getversion.c"> <Filter>Python</Filter> </ClCompile> + <ClCompile Include="..\Python\ceval_gil.c"> + <Filter>Python</Filter> + </ClCompile> <ClCompile Include="..\Python\hashtable.c"> <Filter>Modules</Filter> </ClCompile> diff --git a/Python/ceval.c b/Python/ceval.c index 7024addfe626..1ab104c18ed7 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -13,13 +13,11 @@ #include "pycore_ceval.h" // _PyEval_SignalAsyncExc() #include "pycore_code.h" #include "pycore_function.h" -#include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_moduleobject.h" // PyModuleObject #include "pycore_opcode.h" // EXTRA_CASES #include "pycore_pyerrors.h" // _PyErr_Fetch() -#include "pycore_pylifecycle.h" // _PyErr_Print() #include "pycore_pymem.h" // _PyMem_IsPtrFreed() #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_range.h" // _PyRangeIterObject @@ -237,582 +235,9 @@ is_tstate_valid(PyThreadState *tstate) #endif -/* This can set eval_breaker to 0 even though gil_drop_request became - 1. We believe this is all right because the eval loop will release - the GIL eventually anyway. */ -static inline void -COMPUTE_EVAL_BREAKER(PyInterpreterState *interp, - struct _ceval_runtime_state *ceval, - struct _ceval_state *ceval2) -{ - _Py_atomic_store_relaxed(&ceval2->eval_breaker, - _Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request) - | (_Py_atomic_load_relaxed_int32(&ceval->signals_pending) - && _Py_ThreadCanHandleSignals(interp)) - | (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do) - && _Py_ThreadCanHandlePendingCalls()) - | ceval2->pending.async_exc); -} - - -static inline void -SET_GIL_DROP_REQUEST(PyInterpreterState *interp) -{ - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval2->gil_drop_request, 1); - _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); -} - - -static inline void -RESET_GIL_DROP_REQUEST(PyInterpreterState *interp) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval2->gil_drop_request, 0); - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); -} - - -static inline void -SIGNAL_PENDING_CALLS(PyInterpreterState *interp) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval2->pending.calls_to_do, 1); - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); -} - - -static inline void -UNSIGNAL_PENDING_CALLS(PyInterpreterState *interp) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval2->pending.calls_to_do, 0); - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); -} - - -static inline void -SIGNAL_PENDING_SIGNALS(PyInterpreterState *interp, int force) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval->signals_pending, 1); - if (force) { - _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); - } - else { - /* eval_breaker is not set to 1 if thread_can_handle_signals() is false */ - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); - } -} - - -static inline void -UNSIGNAL_PENDING_SIGNALS(PyInterpreterState *interp) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - _Py_atomic_store_relaxed(&ceval->signals_pending, 0); - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); -} - - -static inline void -SIGNAL_ASYNC_EXC(PyInterpreterState *interp) -{ - struct _ceval_state *ceval2 = &interp->ceval; - ceval2->pending.async_exc = 1; - _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); -} - - -static inline void -UNSIGNAL_ASYNC_EXC(PyInterpreterState *interp) -{ - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - ceval2->pending.async_exc = 0; - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); -} - - #ifdef HAVE_ERRNO_H #include <errno.h> #endif -#include "ceval_gil.h" - -void _Py_NO_RETURN -_Py_FatalError_TstateNULL(const char *func) -{ - _Py_FatalErrorFunc(func, - "the function must be called with the GIL held, " - "after Python initialization and before Python finalization, " - "but the GIL is released (the current Python thread state is NULL)"); -} - -int -_PyEval_ThreadsInitialized(_PyRuntimeState *runtime) -{ - return gil_created(&runtime->ceval.gil); -} - -int -PyEval_ThreadsInitialized(void) -{ - _PyRuntimeState *runtime = &_PyRuntime; - return _PyEval_ThreadsInitialized(runtime); -} - -PyStatus -_PyEval_InitGIL(PyThreadState *tstate) -{ - if (!_Py_IsMainInterpreter(tstate->interp)) { - /* Currently, the GIL is shared by all interpreters, - and only the main interpreter is responsible to create - and destroy it. */ - return _PyStatus_OK(); - } - - struct _gil_runtime_state *gil = &tstate->interp->runtime->ceval.gil; - assert(!gil_created(gil)); - - PyThread_init_thread(); - create_gil(gil); - - take_gil(tstate); - - assert(gil_created(gil)); - return _PyStatus_OK(); -} - -void -_PyEval_FiniGIL(PyInterpreterState *interp) -{ - if (!_Py_IsMainInterpreter(interp)) { - /* Currently, the GIL is shared by all interpreters, - and only the main interpreter is responsible to create - and destroy it. */ - return; - } - - struct _gil_runtime_state *gil = &interp->runtime->ceval.gil; - if (!gil_created(gil)) { - /* First Py_InitializeFromConfig() call: the GIL doesn't exist - yet: do nothing. */ - return; - } - - destroy_gil(gil); - assert(!gil_created(gil)); -} - -void -PyEval_InitThreads(void) -{ - /* Do nothing: kept for backward compatibility */ -} - -void -_PyEval_Fini(void) -{ -#ifdef Py_STATS - _Py_PrintSpecializationStats(1); -#endif -} - -void -PyEval_AcquireLock(void) -{ - _PyRuntimeState *runtime = &_PyRuntime; - PyThreadState *tstate = _PyRuntimeState_GetThreadState(runtime); - _Py_EnsureTstateNotNULL(tstate); - - take_gil(tstate); -} - -void -PyEval_ReleaseLock(void) -{ - _PyRuntimeState *runtime = &_PyRuntime; - PyThreadState *tstate = _PyRuntimeState_GetThreadState(runtime); - /* This function must succeed when the current thread state is NULL. - We therefore avoid PyThreadState_Get() which dumps a fatal error - in debug mode. */ - struct _ceval_runtime_state *ceval = &runtime->ceval; - struct _ceval_state *ceval2 = &tstate->interp->ceval; - drop_gil(ceval, ceval2, tstate); -} - -void -_PyEval_ReleaseLock(PyThreadState *tstate) -{ - struct _ceval_runtime_state *ceval = &tstate->interp->runtime->ceval; - struct _ceval_state *ceval2 = &tstate->interp->ceval; - drop_gil(ceval, ceval2, tstate); -} - -void -PyEval_AcquireThread(PyThreadState *tstate) -{ - _Py_EnsureTstateNotNULL(tstate); - - take_gil(tstate); - - struct _gilstate_runtime_state *gilstate = &tstate->interp->runtime->gilstate; - if (_PyThreadState_Swap(gilstate, tstate) != NULL) { - Py_FatalError("non-NULL old thread state"); - } -} - -void -PyEval_ReleaseThread(PyThreadState *tstate) -{ - assert(is_tstate_valid(tstate)); - - _PyRuntimeState *runtime = tstate->interp->runtime; - PyThreadState *new_tstate = _PyThreadState_Swap(&runtime->gilstate, NULL); - if (new_tstate != tstate) { - Py_FatalError("wrong thread state"); - } - struct _ceval_runtime_state *ceval = &runtime->ceval; - struct _ceval_state *ceval2 = &tstate->interp->ceval; - drop_gil(ceval, ceval2, tstate); -} - -#ifdef HAVE_FORK -/* This function is called from PyOS_AfterFork_Child to destroy all threads - which are not running in the child process, and clear internal locks - which might be held by those threads. */ -PyStatus -_PyEval_ReInitThreads(PyThreadState *tstate) -{ - _PyRuntimeState *runtime = tstate->interp->runtime; - - struct _gil_runtime_state *gil = &runtime->ceval.gil; - if (!gil_created(gil)) { - return _PyStatus_OK(); - } - recreate_gil(gil); - - take_gil(tstate); - - struct _pending_calls *pending = &tstate->interp->ceval.pending; - if (_PyThread_at_fork_reinit(&pending->lock) < 0) { - return _PyStatus_ERR("Can't reinitialize pending calls lock"); - } - - /* Destroy all threads except the current one */ - _PyThreadState_DeleteExcept(runtime, tstate); - return _PyStatus_OK(); -} -#endif - -/* This function is used to signal that async exceptions are waiting to be - raised. */ - -void -_PyEval_SignalAsyncExc(PyInterpreterState *interp) -{ - SIGNAL_ASYNC_EXC(interp); -} - -PyThreadState * -PyEval_SaveThread(void) -{ - _PyRuntimeState *runtime = &_PyRuntime; - PyThreadState *tstate = _PyThreadState_Swap(&runtime->gilstate, NULL); - _Py_EnsureTstateNotNULL(tstate); - - struct _ceval_runtime_state *ceval = &runtime->ceval; - struct _ceval_state *ceval2 = &tstate->interp->ceval; - assert(gil_created(&ceval->gil)); - drop_gil(ceval, ceval2, tstate); - return tstate; -} - -void -PyEval_RestoreThread(PyThreadState *tstate) -{ - _Py_EnsureTstateNotNULL(tstate); - - take_gil(tstate); - - struct _gilstate_runtime_state *gilstate = &tstate->interp->runtime->gilstate; - _PyThreadState_Swap(gilstate, tstate); -} - - -/* Mechanism whereby asynchronously executing callbacks (e.g. UNIX - signal handlers or Mac I/O completion routines) can schedule calls - to a function to be called synchronously. - The synchronous function is called with one void* argument. - It should return 0 for success or -1 for failure -- failure should - be accompanied by an exception. - - If registry succeeds, the registry function returns 0; if it fails - (e.g. due to too many pending calls) it returns -1 (without setting - an exception condition). - - Note that because registry may occur from within signal handlers, - or other asynchronous events, calling malloc() is unsafe! - - Any thread can schedule pending calls, but only the main thread - will execute them. - There is no facility to schedule calls to a particular thread, but - that should be easy to change, should that ever be required. In - that case, the static variables here should go into the python - threadstate. -*/ - -void -_PyEval_SignalReceived(PyInterpreterState *interp) -{ -#ifdef MS_WINDOWS - // bpo-42296: On Windows, _PyEval_SignalReceived() is called from a signal - // handler which can run in a thread different than the Python thread, in - // which case _Py_ThreadCanHandleSignals() is wrong. Ignore - // _Py_ThreadCanHandleSignals() and always set eval_breaker to 1. - // - // The next eval_frame_handle_pending() call will call - // _Py_ThreadCanHandleSignals() to recompute eval_breaker. - int force = 1; -#else - int force = 0; -#endif - /* bpo-30703: Function called when the C signal handler of Python gets a - signal. We cannot queue a callback using _PyEval_AddPendingCall() since - that function is not async-signal-safe. */ - SIGNAL_PENDING_SIGNALS(interp, force); -} - -/* Push one item onto the queue while holding the lock. */ -static int -_push_pending_call(struct _pending_calls *pending, - int (*func)(void *), void *arg) -{ - int i = pending->last; - int j = (i + 1) % NPENDINGCALLS; - if (j == pending->first) { - return -1; /* Queue full */ - } - pending->calls[i].func = func; - pending->calls[i].arg = arg; - pending->last = j; - return 0; -} - -/* Pop one item off the queue while holding the lock. */ -static void -_pop_pending_call(struct _pending_calls *pending, - int (**func)(void *), void **arg) -{ - int i = pending->first; - if (i == pending->last) { - return; /* Queue empty */ - } - - *func = pending->calls[i].func; - *arg = pending->calls[i].arg; - pending->first = (i + 1) % NPENDINGCALLS; -} - -/* This implementation is thread-safe. It allows - scheduling to be made from any thread, and even from an executing - callback. - */ - -int -_PyEval_AddPendingCall(PyInterpreterState *interp, - int (*func)(void *), void *arg) -{ - struct _pending_calls *pending = &interp->ceval.pending; - - /* Ensure that _PyEval_InitState() was called - and that _PyEval_FiniState() is not called yet. */ - assert(pending->lock != NULL); - - PyThread_acquire_lock(pending->lock, WAIT_LOCK); - int result = _push_pending_call(pending, func, arg); - PyThread_release_lock(pending->lock); - - /* signal main loop */ - SIGNAL_PENDING_CALLS(interp); - return result; -} - -int -Py_AddPendingCall(int (*func)(void *), void *arg) -{ - /* Best-effort to support subinterpreters and calls with the GIL released. - - First attempt _PyThreadState_GET() since it supports subinterpreters. - - If the GIL is released, _PyThreadState_GET() returns NULL . In this - case, use PyGILState_GetThisThreadState() which works even if the GIL - is released. - - Sadly, PyGILState_GetThisThreadState() doesn't support subinterpreters: - see bpo-10915 and bpo-15751. - - Py_AddPendingCall() doesn't require the caller to hold the GIL. */ - PyThreadState *tstate = _PyThreadState_GET(); - if (tstate == NULL) { - tstate = PyGILState_GetThisThreadState(); - } - - PyInterpreterState *interp; - if (tstate != NULL) { - interp = tstate->interp; - } - else { - /* Last resort: use the main interpreter */ - interp = _PyInterpreterState_Main(); - } - return _PyEval_AddPendingCall(interp, func, arg); -} - -static int -handle_signals(PyThreadState *tstate) -{ - assert(is_tstate_valid(tstate)); - if (!_Py_ThreadCanHandleSignals(tstate->interp)) { - return 0; - } - - UNSIGNAL_PENDING_SIGNALS(tstate->interp); - if (_PyErr_CheckSignalsTstate(tstate) < 0) { - /* On failure, re-schedule a call to handle_signals(). */ - SIGNAL_PENDING_SIGNALS(tstate->interp, 0); - return -1; - } - return 0; -} - -static int -make_pending_calls(PyInterpreterState *interp) -{ - /* only execute pending calls on main thread */ - if (!_Py_ThreadCanHandlePendingCalls()) { - return 0; - } - - /* don't perform recursive pending calls */ - static int busy = 0; - if (busy) { - return 0; - } - busy = 1; - - /* unsignal before starting to call callbacks, so that any callback - added in-between re-signals */ - UNSIGNAL_PENDING_CALLS(interp); - int res = 0; - - /* perform a bounded number of calls, in case of recursion */ - struct _pending_calls *pending = &interp->ceval.pending; - for (int i=0; i<NPENDINGCALLS; i++) { - int (*func)(void *) = NULL; - void *arg = NULL; - - /* pop one item off the queue while holding the lock */ - PyThread_acquire_lock(pending->lock, WAIT_LOCK); - _pop_pending_call(pending, &func, &arg); - PyThread_release_lock(pending->lock); - - /* having released the lock, perform the callback */ - if (func == NULL) { - break; - } - res = func(arg); - if (res) { - goto error; - } - } - - busy = 0; - return res; - -error: - busy = 0; - SIGNAL_PENDING_CALLS(interp); - return res; -} - -void -_Py_FinishPendingCalls(PyThreadState *tstate) -{ - assert(PyGILState_Check()); - assert(is_tstate_valid(tstate)); - - struct _pending_calls *pending = &tstate->interp->ceval.pending; - - if (!_Py_atomic_load_relaxed_int32(&(pending->calls_to_do))) { - return; - } - - if (make_pending_calls(tstate->interp) < 0) { - PyObject *exc, *val, *tb; - _PyErr_Fetch(tstate, &exc, &val, &tb); - PyErr_BadInternalCall(); - _PyErr_ChainExceptions(exc, val, tb); - _PyErr_Print(tstate); - } -} - -/* Py_MakePendingCalls() is a simple wrapper for the sake - of backward-compatibility. */ -int -Py_MakePendingCalls(void) -{ - assert(PyGILState_Check()); - - PyThreadState *tstate = _PyThreadState_GET(); - assert(is_tstate_valid(tstate)); - - /* Python signal handler doesn't really queue a callback: it only signals - that a signal was received, see _PyEval_SignalReceived(). */ - int res = handle_signals(tstate); - if (res != 0) { - return res; - } - - res = make_pending_calls(tstate->interp); - if (res != 0) { - return res; - } - - return 0; -} - -/* The interpreter's recursion limit */ - -void -_PyEval_InitRuntimeState(struct _ceval_runtime_state *ceval) -{ - _gil_initialize(&ceval->gil); -} - -void -_PyEval_InitState(struct _ceval_state *ceval, PyThread_type_lock pending_lock) -{ - struct _pending_calls *pending = &ceval->pending; - assert(pending->lock == NULL); - - pending->lock = pending_lock; -} - -void -_PyEval_FiniState(struct _ceval_state *ceval) -{ - struct _pending_calls *pending = &ceval->pending; - if (pending->lock != NULL) { - PyThread_free_lock(pending->lock); - pending->lock = NULL; - } -} int Py_GetRecursionLimit(void) @@ -1182,71 +607,6 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) } -/* Handle signals, pending calls, GIL drop request - and asynchronous exception */ -static int -eval_frame_handle_pending(PyThreadState *tstate) -{ - _PyRuntimeState * const runtime = &_PyRuntime; - struct _ceval_runtime_state *ceval = &runtime->ceval; - - /* Pending signals */ - if (_Py_atomic_load_relaxed_int32(&ceval->signals_pending)) { - if (handle_signals(tstate) != 0) { - return -1; - } - } - - /* Pending calls */ - struct _ceval_state *ceval2 = &tstate->interp->ceval; - if (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do)) { - if (make_pending_calls(tstate->interp) != 0) { - return -1; - } - } - - /* GIL drop request */ - if (_Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request)) { - /* Give another thread a chance */ - if (_PyThreadState_Swap(&runtime->gilstate, NULL) != tstate) { - Py_FatalError("tstate mix-up"); - } - drop_gil(ceval, ceval2, tstate); - - /* Other threads may run now */ - - take_gil(tstate); - - if (_PyThreadState_Swap(&runtime->gilstate, tstate) != NULL) { - Py_FatalError("orphan tstate"); - } - } - - /* Check for asynchronous exception. */ - if (tstate->async_exc != NULL) { - PyObject *exc = tstate->async_exc; - tstate->async_exc = NULL; - UNSIGNAL_ASYNC_EXC(tstate->interp); - _PyErr_SetNone(tstate, exc); - Py_DECREF(exc); - return -1; - } - -#ifdef MS_WINDOWS - // bpo-42296: On Windows, _PyEval_SignalReceived() can be called in a - // different thread than the Python thread, in which case - // _Py_ThreadCanHandleSignals() is wrong. Recompute eval_breaker in the - // current Python thread with the correct _Py_ThreadCanHandleSignals() - // value. It prevents to interrupt the eval loop at every instruction if - // the current Python thread cannot handle signals (if - // _Py_ThreadCanHandleSignals() is false). - COMPUTE_EVAL_BREAKER(tstate->interp, ceval, ceval2); -#endif - - return 0; -} - - /* Computed GOTOs, or the-optimization-commonly-but-improperly-known-as-"threaded code" using gcc's labels-as-values extension @@ -1750,7 +1110,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int * All loops should include a check of the eval breaker. * We also check on return from any builtin function. */ - if (eval_frame_handle_pending(tstate) != 0) { + if (_Py_HandlePending(tstate) != 0) { goto error; } DISPATCH(); diff --git a/Python/ceval_gil.c b/Python/ceval_gil.c new file mode 100644 index 000000000000..a67908667667 --- /dev/null +++ b/Python/ceval_gil.c @@ -0,0 +1,986 @@ + +#include "Python.h" +#include "pycore_atomic.h" // _Py_atomic_int +#include "pycore_ceval.h" // _PyEval_SignalReceived() +#include "pycore_pyerrors.h" // _PyErr_Fetch() +#include "pycore_pylifecycle.h" // _PyErr_Print() +#include "pycore_initconfig.h" // _PyStatus_OK() +#include "pycore_pymem.h" // _PyMem_IsPtrFreed() + +/* + Notes about the implementation: + + - The GIL is just a boolean variable (locked) whose access is protected + by a mutex (gil_mutex), and whose changes are signalled by a condition + variable (gil_cond). gil_mutex is taken for short periods of time, + and therefore mostly uncontended. + + - In the GIL-holding thread, the main loop (PyEval_EvalFrameEx) must be + able to release the GIL on demand by another thread. A volatile boolean + variable (gil_drop_request) is used for that purpose, which is checked + at every turn of the eval loop. That variable is set after a wait of + `interval` microseconds on `gil_cond` has timed out. + + [Actually, another volatile boolean variable (eval_breaker) is used + which ORs several conditions into one. Volatile booleans are + sufficient as inter-thread signalling means since Python is run + on cache-coherent architectures only.] + + - A thread wanting to take the GIL will first let pass a given amount of + time (`interval` microseconds) before setting gil_drop_request. This + encourages a defined switching period, but doesn't enforce it since + opcodes can take an arbitrary time to execute. + + The `interval` value is available for the user to read and modify + using the Python API `sys.{get,set}switchinterval()`. + + - When a thread releases the GIL and gil_drop_request is set, that thread + ensures that another GIL-awaiting thread gets scheduled. + It does so by waiting on a condition variable (switch_cond) until + the value of last_holder is changed to something else than its + own thread state pointer, indicating that another thread was able to + take the GIL. + + This is meant to prohibit the latency-adverse behaviour on multi-core + machines where one thread would speculatively release the GIL, but still + run and end up being the first to re-acquire it, making the "timeslices" + much longer than expected. + (Note: this mechanism is enabled with FORCE_SWITCHING above) +*/ + +// GH-89279: Force inlining by using a macro. +#if defined(_MSC_VER) && SIZEOF_INT == 4 +#define _Py_atomic_load_relaxed_int32(ATOMIC_VAL) (assert(sizeof((ATOMIC_VAL)->_value) == 4), *((volatile int*)&((ATOMIC_VAL)->_value))) +#else +#define _Py_atomic_load_relaxed_int32(ATOMIC_VAL) _Py_atomic_load_relaxed(ATOMIC_VAL) +#endif + +/* This can set eval_breaker to 0 even though gil_drop_request became + 1. We believe this is all right because the eval loop will release + the GIL eventually anyway. */ +static inline void +COMPUTE_EVAL_BREAKER(PyInterpreterState *interp, + struct _ceval_runtime_state *ceval, + struct _ceval_state *ceval2) +{ + _Py_atomic_store_relaxed(&ceval2->eval_breaker, + _Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request) + | (_Py_atomic_load_relaxed_int32(&ceval->signals_pending) + && _Py_ThreadCanHandleSignals(interp)) + | (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do) + && _Py_ThreadCanHandlePendingCalls()) + | ceval2->pending.async_exc); +} + + +static inline void +SET_GIL_DROP_REQUEST(PyInterpreterState *interp) +{ + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval2->gil_drop_request, 1); + _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); +} + + +static inline void +RESET_GIL_DROP_REQUEST(PyInterpreterState *interp) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval2->gil_drop_request, 0); + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); +} + + +static inline void +SIGNAL_PENDING_CALLS(PyInterpreterState *interp) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval2->pending.calls_to_do, 1); + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); +} + + +static inline void +UNSIGNAL_PENDING_CALLS(PyInterpreterState *interp) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval2->pending.calls_to_do, 0); + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); +} + + +static inline void +SIGNAL_PENDING_SIGNALS(PyInterpreterState *interp, int force) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval->signals_pending, 1); + if (force) { + _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); + } + else { + /* eval_breaker is not set to 1 if thread_can_handle_signals() is false */ + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); + } +} + + +static inline void +UNSIGNAL_PENDING_SIGNALS(PyInterpreterState *interp) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + _Py_atomic_store_relaxed(&ceval->signals_pending, 0); + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); +} + + +static inline void +SIGNAL_ASYNC_EXC(PyInterpreterState *interp) +{ + struct _ceval_state *ceval2 = &interp->ceval; + ceval2->pending.async_exc = 1; + _Py_atomic_store_relaxed(&ceval2->eval_breaker, 1); +} + + +static inline void +UNSIGNAL_ASYNC_EXC(PyInterpreterState *interp) +{ + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + ceval2->pending.async_exc = 0; + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); +} + +#ifndef NDEBUG +/* Ensure that tstate is valid */ +static int +is_tstate_valid(PyThreadState *tstate) +{ + assert(!_PyMem_IsPtrFreed(tstate)); + assert(!_PyMem_IsPtrFreed(tstate->interp)); + return 1; +} +#endif + +/* + * Implementation of the Global Interpreter Lock (GIL). + */ + +#include <stdlib.h> +#include <errno.h> + +#include "pycore_atomic.h" + + +#include "condvar.h" + +#define MUTEX_INIT(mut) \ + if (PyMUTEX_INIT(&(mut))) { \ + Py_FatalError("PyMUTEX_INIT(" #mut ") failed"); }; +#define MUTEX_FINI(mut) \ + if (PyMUTEX_FINI(&(mut))) { \ + Py_FatalError("PyMUTEX_FINI(" #mut ") failed"); }; +#define MUTEX_LOCK(mut) \ + if (PyMUTEX_LOCK(&(mut))) { \ + Py_FatalError("PyMUTEX_LOCK(" #mut ") failed"); }; +#define MUTEX_UNLOCK(mut) \ + if (PyMUTEX_UNLOCK(&(mut))) { \ + Py_FatalError("PyMUTEX_UNLOCK(" #mut ") failed"); }; + +#define COND_INIT(cond) \ + if (PyCOND_INIT(&(cond))) { \ + Py_FatalError("PyCOND_INIT(" #cond ") failed"); }; +#define COND_FINI(cond) \ + if (PyCOND_FINI(&(cond))) { \ + Py_FatalError("PyCOND_FINI(" #cond ") failed"); }; +#define COND_SIGNAL(cond) \ + if (PyCOND_SIGNAL(&(cond))) { \ + Py_FatalError("PyCOND_SIGNAL(" #cond ") failed"); }; +#define COND_WAIT(cond, mut) \ + if (PyCOND_WAIT(&(cond), &(mut))) { \ + Py_FatalError("PyCOND_WAIT(" #cond ") failed"); }; +#define COND_TIMED_WAIT(cond, mut, microseconds, timeout_result) \ + { \ + int r = PyCOND_TIMEDWAIT(&(cond), &(mut), (microseconds)); \ + if (r < 0) \ + Py_FatalError("PyCOND_WAIT(" #cond ") failed"); \ + if (r) /* 1 == timeout, 2 == impl. can't say, so assume timeout */ \ + timeout_result = 1; \ + else \ + timeout_result = 0; \ + } \ + + +#define DEFAULT_INTERVAL 5000 + +static void _gil_initialize(struct _gil_runtime_state *gil) +{ + _Py_atomic_int uninitialized = {-1}; + gil->locked = uninitialized; + gil->interval = DEFAULT_INTERVAL; +} + +static int gil_created(struct _gil_runtime_state *gil) +{ + return (_Py_atomic_load_explicit(&gil->locked, _Py_memory_order_acquire) >= 0); +} + +static void create_gil(struct _gil_runtime_state *gil) +{ + MUTEX_INIT(gil->mutex); +#ifdef FORCE_SWITCHING + MUTEX_INIT(gil->switch_mutex); +#endif + COND_INIT(gil->cond); +#ifdef FORCE_SWITCHING + COND_INIT(gil->switch_cond); +#endif + _Py_atomic_store_relaxed(&gil->last_holder, 0); + _Py_ANNOTATE_RWLOCK_CREATE(&gil->locked); + _Py_atomic_store_explicit(&gil->locked, 0, _Py_memory_order_release); +} + +static void destroy_gil(struct _gil_runtime_state *gil) +{ + /* some pthread-like implementations tie the mutex to the cond + * and must have the cond destroyed first. + */ + COND_FINI(gil->cond); + MUTEX_FINI(gil->mutex); +#ifdef FORCE_SWITCHING + COND_FINI(gil->switch_cond); + MUTEX_FINI(gil->switch_mutex); +#endif + _Py_atomic_store_explicit(&gil->locked, -1, + _Py_memory_order_release); + _Py_ANNOTATE_RWLOCK_DESTROY(&gil->locked); +} + +#ifdef HAVE_FORK +static void recreate_gil(struct _gil_runtime_state *gil) +{ + _Py_ANNOTATE_RWLOCK_DESTROY(&gil->locked); + /* XXX should we destroy the old OS resources here? */ + create_gil(gil); +} +#endif + +static void +drop_gil(struct _ceval_runtime_state *ceval, struct _ceval_state *ceval2, + PyThreadState *tstate) +{ + struct _gil_runtime_state *gil = &ceval->gil; + if (!_Py_atomic_load_relaxed(&gil->locked)) { + Py_FatalError("drop_gil: GIL is not locked"); + } + + /* tstate is allowed to be NULL (early interpreter init) */ + if (tstate != NULL) { + /* Sub-interpreter support: threads might have been switched + under our feet using PyThreadState_Swap(). Fix the GIL last + holder variable so that our heuristics work. */ + _Py_atomic_store_relaxed(&gil->last_holder, (uintptr_t)tstate); + } + + MUTEX_LOCK(gil->mutex); + _Py_ANNOTATE_RWLOCK_RELEASED(&gil->locked, /*is_write=*/1); + _Py_atomic_store_relaxed(&gil->locked, 0); + COND_SIGNAL(gil->cond); + MUTEX_UNLOCK(gil->mutex); + +#ifdef FORCE_SWITCHING + if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request) && tstate != NULL) { + MUTEX_LOCK(gil->switch_mutex); + /* Not switched yet => wait */ + if (((PyThreadState*)_Py_atomic_load_relaxed(&gil->last_holder)) == tstate) + { + assert(is_tstate_valid(tstate)); + RESET_GIL_DROP_REQUEST(tstate->interp); + /* NOTE: if COND_WAIT does not atomically start waiting when + releasing the mutex, another thread can run through, take + the GIL and drop it again, and reset the condition + before we even had a chance to wait for it. */ + COND_WAIT(gil->switch_cond, gil->switch_mutex); + } + MUTEX_UNLOCK(gil->switch_mutex); + } +#endif +} + + +/* Check if a Python thread must exit immediately, rather than taking the GIL + if Py_Finalize() has been called. + + When this function is called by a daemon thread after Py_Finalize() has been + called, the GIL does no longer exist. + + tstate must be non-NULL. */ +static inline int +tstate_must_exit(PyThreadState *tstate) +{ + /* bpo-39877: Access _PyRuntime directly rather than using + tstate->interp->runtime to support calls from Python daemon threads. + After Py_Finalize() has been called, tstate can be a dangling pointer: + point to PyThreadState freed memory. */ + PyThreadState *finalizing = _PyRuntimeState_GetFinalizing(&_PyRuntime); + return (finalizing != NULL && finalizing != tstate); +} + + +/* Take the GIL. + + The function saves errno at entry and restores its value at exit. + + tstate must be non-NULL. */ +static void +take_gil(PyThreadState *tstate) +{ + int err = errno; + + assert(tstate != NULL); + + if (tstate_must_exit(tstate)) { + /* bpo-39877: If Py_Finalize() has been called and tstate is not the + thread which called Py_Finalize(), exit immediately the thread. + + This code path can be reached by a daemon thread after Py_Finalize() + completes. In this case, tstate is a dangling pointer: points to + PyThreadState freed memory. */ + PyThread_exit_thread(); + } + + assert(is_tstate_valid(tstate)); + PyInterpreterState *interp = tstate->interp; + struct _ceval_runtime_state *ceval = &interp->runtime->ceval; + struct _ceval_state *ceval2 = &interp->ceval; + struct _gil_runtime_state *gil = &ceval->gil; + + /* Check that _PyEval_InitThreads() was called to create the lock */ + assert(gil_created(gil)); + + MUTEX_LOCK(gil->mutex); + + if (!_Py_atomic_load_relaxed(&gil->locked)) { + goto _ready; + } + + while (_Py_atomic_load_relaxed(&gil->locked)) { + unsigned long saved_switchnum = gil->switch_number; + + unsigned long interval = (gil->interval >= 1 ? gil->interval : 1); + int timed_out = 0; + COND_TIMED_WAIT(gil->cond, gil->mutex, interval, timed_out); + + /* If we timed out and no switch occurred in the meantime, it is time + to ask the GIL-holding thread to drop it. */ + if (timed_out && + _Py_atomic_load_relaxed(&gil->locked) && + gil->switch_number == saved_switchnum) + { + if (tstate_must_exit(tstate)) { + MUTEX_UNLOCK(gil->mutex); + PyThread_exit_thread(); + } + assert(is_tstate_valid(tstate)); + + SET_GIL_DROP_REQUEST(interp); + } + } + +_ready: +#ifdef FORCE_SWITCHING + /* This mutex must be taken before modifying gil->last_holder: + see drop_gil(). */ + MUTEX_LOCK(gil->switch_mutex); +#endif + /* We now hold the GIL */ + _Py_atomic_store_relaxed(&gil->locked, 1); + _Py_ANNOTATE_RWLOCK_ACQUIRED(&gil->locked, /*is_write=*/1); + + if (tstate != (PyThreadState*)_Py_atomic_load_relaxed(&gil->last_holder)) { + _Py_atomic_store_relaxed(&gil->last_holder, (uintptr_t)tstate); + ++gil->switch_number; + } + +#ifdef FORCE_SWITCHING + COND_SIGNAL(gil->switch_cond); + MUTEX_UNLOCK(gil->switch_mutex); +#endif + + if (tstate_must_exit(tstate)) { + /* bpo-36475: If Py_Finalize() has been called and tstate is not + the thread which called Py_Finalize(), exit immediately the + thread. + + This code path can be reached by a daemon thread which was waiting + in take_gil() while the main thread called + wait_for_thread_shutdown() from Py_Finalize(). */ + MUTEX_UNLOCK(gil->mutex); + drop_gil(ceval, ceval2, tstate); + PyThread_exit_thread(); + } + assert(is_tstate_valid(tstate)); + + if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request)) { + RESET_GIL_DROP_REQUEST(interp); + } + else { + /* bpo-40010: eval_breaker should be recomputed to be set to 1 if there + is a pending signal: signal received by another thread which cannot + handle signals. + + Note: RESET_GIL_DROP_REQUEST() calls COMPUTE_EVAL_BREAKER(). */ + COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); + } + + /* Don't access tstate if the thread must exit */ + if (tstate->async_exc != NULL) { + _PyEval_SignalAsyncExc(tstate->interp); + } + + MUTEX_UNLOCK(gil->mutex); + + errno = err; +} + +void _PyEval_SetSwitchInterval(unsigned long microseconds) +{ + struct _gil_runtime_state *gil = &_PyRuntime.ceval.gil; + gil->interval = microseconds; +} + +unsigned long _PyEval_GetSwitchInterval() +{ + struct _gil_runtime_state *gil = &_PyRuntime.ceval.gil; + return gil->interval; +} + + +int +_PyEval_ThreadsInitialized(_PyRuntimeState *runtime) +{ + return gil_created(&runtime->ceval.gil); +} + +int +PyEval_ThreadsInitialized(void) +{ + _PyRuntimeState *runtime = &_PyRuntime; + return _PyEval_ThreadsInitialized(runtime); +} + +PyStatus +_PyEval_InitGIL(PyThreadState *tstate) +{ + if (!_Py_IsMainInterpreter(tstate->interp)) { + /* Currently, the GIL is shared by all interpreters, + and only the main interpreter is responsible to create + and destroy it. */ + return _PyStatus_OK(); + } + + struct _gil_runtime_state *gil = &tstate->interp->runtime->ceval.gil; + assert(!gil_created(gil)); + + PyThread_init_thread(); + create_gil(gil); + + take_gil(tstate); + + assert(gil_created(gil)); + return _PyStatus_OK(); +} + +void +_PyEval_FiniGIL(PyInterpreterState *interp) +{ + if (!_Py_IsMainInterpreter(interp)) { + /* Currently, the GIL is shared by all interpreters, + and only the main interpreter is responsible to create + and destroy it. */ + return; + } + + struct _gil_runtime_state *gil = &interp->runtime->ceval.gil; + if (!gil_created(gil)) { + /* First Py_InitializeFromConfig() call: the GIL doesn't exist + yet: do nothing. */ + return; + } + + destroy_gil(gil); + assert(!gil_created(gil)); +} + +void +PyEval_InitThreads(void) +{ + /* Do nothing: kept for backward compatibility */ +} + +void +_PyEval_Fini(void) +{ +#ifdef Py_STATS + _Py_PrintSpecializationStats(1); +#endif +} +void +PyEval_AcquireLock(void) +{ + _PyRuntimeState *runtime = &_PyRuntime; + PyThreadState *tstate = _PyRuntimeState_GetThreadState(runtime); + _Py_EnsureTstateNotNULL(tstate); + + take_gil(tstate); +} + +void +PyEval_ReleaseLock(void) +{ + _PyRuntimeState *runtime = &_PyRuntime; + PyThreadState *tstate = _PyRuntimeState_GetThreadState(runtime); + /* This function must succeed when the current thread state is NULL. + We therefore avoid PyThreadState_Get() which dumps a fatal error + in debug mode. */ + struct _ceval_runtime_state *ceval = &runtime->ceval; + struct _ceval_state *ceval2 = &tstate->interp->ceval; + drop_gil(ceval, ceval2, tstate); +} + +void +_PyEval_ReleaseLock(PyThreadState *tstate) +{ + struct _ceval_runtime_state *ceval = &tstate->interp->runtime->ceval; + struct _ceval_state *ceval2 = &tstate->interp->ceval; + drop_gil(ceval, ceval2, tstate); +} + +void +PyEval_AcquireThread(PyThreadState *tstate) +{ + _Py_EnsureTstateNotNULL(tstate); + + take_gil(tstate); + + struct _gilstate_runtime_state *gilstate = &tstate->interp->runtime->gilstate; + if (_PyThreadState_Swap(gilstate, tstate) != NULL) { + Py_FatalError("non-NULL old thread state"); + } +} + +void +PyEval_ReleaseThread(PyThreadState *tstate) +{ + assert(is_tstate_valid(tstate)); + + _PyRuntimeState *runtime = tstate->interp->runtime; + PyThreadState *new_tstate = _PyThreadState_Swap(&runtime->gilstate, NULL); + if (new_tstate != tstate) { + Py_FatalError("wrong thread state"); + } + struct _ceval_runtime_state *ceval = &runtime->ceval; + struct _ceval_state *ceval2 = &tstate->interp->ceval; + drop_gil(ceval, ceval2, tstate); +} + +#ifdef HAVE_FORK +/* This function is called from PyOS_AfterFork_Child to destroy all threads + which are not running in the child process, and clear internal locks + which might be held by those threads. */ +PyStatus +_PyEval_ReInitThreads(PyThreadState *tstate) +{ + _PyRuntimeState *runtime = tstate->interp->runtime; + + struct _gil_runtime_state *gil = &runtime->ceval.gil; + if (!gil_created(gil)) { + return _PyStatus_OK(); + } + recreate_gil(gil); + + take_gil(tstate); + + struct _pending_calls *pending = &tstate->interp->ceval.pending; + if (_PyThread_at_fork_reinit(&pending->lock) < 0) { + return _PyStatus_ERR("Can't reinitialize pending calls lock"); + } + + /* Destroy all threads except the current one */ + _PyThreadState_DeleteExcept(runtime, tstate); + return _PyStatus_OK(); +} +#endif + +/* This function is used to signal that async exceptions are waiting to be + raised. */ + +void +_PyEval_SignalAsyncExc(PyInterpreterState *interp) +{ + SIGNAL_ASYNC_EXC(interp); +} + +PyThreadState * +PyEval_SaveThread(void) +{ + _PyRuntimeState *runtime = &_PyRuntime; + PyThreadState *tstate = _PyThreadState_Swap(&runtime->gilstate, NULL); + _Py_EnsureTstateNotNULL(tstate); + + struct _ceval_runtime_state *ceval = &runtime->ceval; + struct _ceval_state *ceval2 = &tstate->interp->ceval; + assert(gil_created(&ceval->gil)); + drop_gil(ceval, ceval2, tstate); + return tstate; +} + +void +PyEval_RestoreThread(PyThreadState *tstate) +{ + _Py_EnsureTstateNotNULL(tstate); + + take_gil(tstate); + + struct _gilstate_runtime_state *gilstate = &tstate->interp->runtime->gilstate; + _PyThreadState_Swap(gilstate, tstate); +} + + +/* Mechanism whereby asynchronously executing callbacks (e.g. UNIX + signal handlers or Mac I/O completion routines) can schedule calls + to a function to be called synchronously. + The synchronous function is called with one void* argument. + It should return 0 for success or -1 for failure -- failure should + be accompanied by an exception. + + If registry succeeds, the registry function returns 0; if it fails + (e.g. due to too many pending calls) it returns -1 (without setting + an exception condition). + + Note that because registry may occur from within signal handlers, + or other asynchronous events, calling malloc() is unsafe! + + Any thread can schedule pending calls, but only the main thread + will execute them. + There is no facility to schedule calls to a particular thread, but + that should be easy to change, should that ever be required. In + that case, the static variables here should go into the python + threadstate. +*/ + +void +_PyEval_SignalReceived(PyInterpreterState *interp) +{ +#ifdef MS_WINDOWS + // bpo-42296: On Windows, _PyEval_SignalReceived() is called from a signal + // handler which can run in a thread different than the Python thread, in + // which case _Py_ThreadCanHandleSignals() is wrong. Ignore + // _Py_ThreadCanHandleSignals() and always set eval_breaker to 1. + // + // The next eval_frame_handle_pending() call will call + // _Py_ThreadCanHandleSignals() to recompute eval_breaker. + int force = 1; +#else + int force = 0; +#endif + /* bpo-30703: Function called when the C signal handler of Python gets a + signal. We cannot queue a callback using _PyEval_AddPendingCall() since + that function is not async-signal-safe. */ + SIGNAL_PENDING_SIGNALS(interp, force); +} + +/* Push one item onto the queue while holding the lock. */ +static int +_push_pending_call(struct _pending_calls *pending, + int (*func)(void *), void *arg) +{ + int i = pending->last; + int j = (i + 1) % NPENDINGCALLS; + if (j == pending->first) { + return -1; /* Queue full */ + } + pending->calls[i].func = func; + pending->calls[i].arg = arg; + pending->last = j; + return 0; +} + +/* Pop one item off the queue while holding the lock. */ +static void +_pop_pending_call(struct _pending_calls *pending, + int (**func)(void *), void **arg) +{ + int i = pending->first; + if (i == pending->last) { + return; /* Queue empty */ + } + + *func = pending->calls[i].func; + *arg = pending->calls[i].arg; + pending->first = (i + 1) % NPENDINGCALLS; +} + +/* This implementation is thread-safe. It allows + scheduling to be made from any thread, and even from an executing + callback. + */ + +int +_PyEval_AddPendingCall(PyInterpreterState *interp, + int (*func)(void *), void *arg) +{ + struct _pending_calls *pending = &interp->ceval.pending; + /* Ensure that _PyEval_InitState() was called + and that _PyEval_FiniState() is not called yet. */ + assert(pending->lock != NULL); + + PyThread_acquire_lock(pending->lock, WAIT_LOCK); + int result = _push_pending_call(pending, func, arg); + PyThread_release_lock(pending->lock); + + /* signal main loop */ + SIGNAL_PENDING_CALLS(interp); + return result; +} + +int +Py_AddPendingCall(int (*func)(void *), void *arg) +{ + /* Best-effort to support subinterpreters and calls with the GIL released. + + First attempt _PyThreadState_GET() since it supports subinterpreters. + + If the GIL is released, _PyThreadState_GET() returns NULL . In this + case, use PyGILState_GetThisThreadState() which works even if the GIL + is released. + + Sadly, PyGILState_GetThisThreadState() doesn't support subinterpreters: + see bpo-10915 and bpo-15751. + + Py_AddPendingCall() doesn't require the caller to hold the GIL. */ + PyThreadState *tstate = _PyThreadState_GET(); + if (tstate == NULL) { + tstate = PyGILState_GetThisThreadState(); + } + + PyInterpreterState *interp; + if (tstate != NULL) { + interp = tstate->interp; + } + else { + /* Last resort: use the main interpreter */ + interp = _PyInterpreterState_Main(); + } + return _PyEval_AddPendingCall(interp, func, arg); +} + +static int +handle_signals(PyThreadState *tstate) +{ + assert(is_tstate_valid(tstate)); + if (!_Py_ThreadCanHandleSignals(tstate->interp)) { + return 0; + } + + UNSIGNAL_PENDING_SIGNALS(tstate->interp); + if (_PyErr_CheckSignalsTstate(tstate) < 0) { + /* On failure, re-schedule a call to handle_signals(). */ + SIGNAL_PENDING_SIGNALS(tstate->interp, 0); + return -1; + } + return 0; +} + +static int +make_pending_calls(PyInterpreterState *interp) +{ + /* only execute pending calls on main thread */ + if (!_Py_ThreadCanHandlePendingCalls()) { + return 0; + } + + /* don't perform recursive pending calls */ + static int busy = 0; + if (busy) { + return 0; + } + busy = 1; + + /* unsignal before starting to call callbacks, so that any callback + added in-between re-signals */ + UNSIGNAL_PENDING_CALLS(interp); + int res = 0; + + /* perform a bounded number of calls, in case of recursion */ + struct _pending_calls *pending = &interp->ceval.pending; + for (int i=0; i<NPENDINGCALLS; i++) { + int (*func)(void *) = NULL; + void *arg = NULL; + + /* pop one item off the queue while holding the lock */ + PyThread_acquire_lock(pending->lock, WAIT_LOCK); + _pop_pending_call(pending, &func, &arg); + PyThread_release_lock(pending->lock); + + /* having released the lock, perform the callback */ + if (func == NULL) { + break; + } + res = func(arg); + if (res) { + goto error; + } + } + + busy = 0; + return res; + +error: + busy = 0; + SIGNAL_PENDING_CALLS(interp); + return res; +} + +void +_Py_FinishPendingCalls(PyThreadState *tstate) +{ + assert(PyGILState_Check()); + assert(is_tstate_valid(tstate)); + + struct _pending_calls *pending = &tstate->interp->ceval.pending; + + if (!_Py_atomic_load_relaxed_int32(&(pending->calls_to_do))) { + return; + } + + if (make_pending_calls(tstate->interp) < 0) { + PyObject *exc, *val, *tb; + _PyErr_Fetch(tstate, &exc, &val, &tb); + PyErr_BadInternalCall(); + _PyErr_ChainExceptions(exc, val, tb); + _PyErr_Print(tstate); + } +} + +/* Py_MakePendingCalls() is a simple wrapper for the sake + of backward-compatibility. */ +int +Py_MakePendingCalls(void) +{ + assert(PyGILState_Check()); + + PyThreadState *tstate = _PyThreadState_GET(); + assert(is_tstate_valid(tstate)); + + /* Python signal handler doesn't really queue a callback: it only signals + that a signal was received, see _PyEval_SignalReceived(). */ + int res = handle_signals(tstate); + if (res != 0) { + return res; + } + + res = make_pending_calls(tstate->interp); + if (res != 0) { + return res; + } + + return 0; +} + +/* The interpreter's recursion limit */ + +void +_PyEval_InitRuntimeState(struct _ceval_runtime_state *ceval) +{ + _gil_initialize(&ceval->gil); +} + +void +_PyEval_InitState(struct _ceval_state *ceval, PyThread_type_lock pending_lock) +{ + struct _pending_calls *pending = &ceval->pending; + assert(pending->lock == NULL); + + pending->lock = pending_lock; +} + +void +_PyEval_FiniState(struct _ceval_state *ceval) +{ + struct _pending_calls *pending = &ceval->pending; + if (pending->lock != NULL) { + PyThread_free_lock(pending->lock); + pending->lock = NULL; + } +} + +/* Handle signals, pending calls, GIL drop request + and asynchronous exception */ +int +_Py_HandlePending(PyThreadState *tstate) +{ + _PyRuntimeState * const runtime = &_PyRuntime; + struct _ceval_runtime_state *ceval = &runtime->ceval; + + /* Pending signals */ + if (_Py_atomic_load_relaxed_int32(&ceval->signals_pending)) { + if (handle_signals(tstate) != 0) { + return -1; + } + } + + /* Pending calls */ + struct _ceval_state *ceval2 = &tstate->interp->ceval; + if (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do)) { + if (make_pending_calls(tstate->interp) != 0) { + return -1; + } + } + + /* GIL drop request */ + if (_Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request)) { + /* Give another thread a chance */ + if (_PyThreadState_Swap(&runtime->gilstate, NULL) != tstate) { + Py_FatalError("tstate mix-up"); + } + drop_gil(ceval, ceval2, tstate); + + /* Other threads may run now */ + + take_gil(tstate); + + if (_PyThreadState_Swap(&runtime->gilstate, tstate) != NULL) { + Py_FatalError("orphan tstate"); + } + } + + /* Check for asynchronous exception. */ + if (tstate->async_exc != NULL) { + PyObject *exc = tstate->async_exc; + tstate->async_exc = NULL; + UNSIGNAL_ASYNC_EXC(tstate->interp); + _PyErr_SetNone(tstate, exc); + Py_DECREF(exc); + return -1; + } + +#ifdef MS_WINDOWS + // bpo-42296: On Windows, _PyEval_SignalReceived() can be called in a + // different thread than the Python thread, in which case + // _Py_ThreadCanHandleSignals() is wrong. Recompute eval_breaker in the + // current Python thread with the correct _Py_ThreadCanHandleSignals() + // value. It prevents to interrupt the eval loop at every instruction if + // the current Python thread cannot handle signals (if + // _Py_ThreadCanHandleSignals() is false). + COMPUTE_EVAL_BREAKER(tstate->interp, ceval, ceval2); +#endif + + return 0; +} + diff --git a/Python/ceval_gil.h b/Python/ceval_gil.h deleted file mode 100644 index 4c71edd682bf..000000000000 --- a/Python/ceval_gil.h +++ /dev/null @@ -1,333 +0,0 @@ -/* - * Implementation of the Global Interpreter Lock (GIL). - */ - -#include <stdlib.h> -#include <errno.h> - -#include "pycore_atomic.h" - - -/* - Notes about the implementation: - - - The GIL is just a boolean variable (locked) whose access is protected - by a mutex (gil_mutex), and whose changes are signalled by a condition - variable (gil_cond). gil_mutex is taken for short periods of time, - and therefore mostly uncontended. - - - In the GIL-holding thread, the main loop (PyEval_EvalFrameEx) must be - able to release the GIL on demand by another thread. A volatile boolean - variable (gil_drop_request) is used for that purpose, which is checked - at every turn of the eval loop. That variable is set after a wait of - `interval` microseconds on `gil_cond` has timed out. - - [Actually, another volatile boolean variable (eval_breaker) is used - which ORs several conditions into one. Volatile booleans are - sufficient as inter-thread signalling means since Python is run - on cache-coherent architectures only.] - - - A thread wanting to take the GIL will first let pass a given amount of - time (`interval` microseconds) before setting gil_drop_request. This - encourages a defined switching period, but doesn't enforce it since - opcodes can take an arbitrary time to execute. - - The `interval` value is available for the user to read and modify - using the Python API `sys.{get,set}switchinterval()`. - - - When a thread releases the GIL and gil_drop_request is set, that thread - ensures that another GIL-awaiting thread gets scheduled. - It does so by waiting on a condition variable (switch_cond) until - the value of last_holder is changed to something else than its - own thread state pointer, indicating that another thread was able to - take the GIL. - - This is meant to prohibit the latency-adverse behaviour on multi-core - machines where one thread would speculatively release the GIL, but still - run and end up being the first to re-acquire it, making the "timeslices" - much longer than expected. - (Note: this mechanism is enabled with FORCE_SWITCHING above) -*/ - -#include "condvar.h" - -#define MUTEX_INIT(mut) \ - if (PyMUTEX_INIT(&(mut))) { \ - Py_FatalError("PyMUTEX_INIT(" #mut ") failed"); }; -#define MUTEX_FINI(mut) \ - if (PyMUTEX_FINI(&(mut))) { \ - Py_FatalError("PyMUTEX_FINI(" #mut ") failed"); }; -#define MUTEX_LOCK(mut) \ - if (PyMUTEX_LOCK(&(mut))) { \ - Py_FatalError("PyMUTEX_LOCK(" #mut ") failed"); }; -#define MUTEX_UNLOCK(mut) \ - if (PyMUTEX_UNLOCK(&(mut))) { \ - Py_FatalError("PyMUTEX_UNLOCK(" #mut ") failed"); }; - -#define COND_INIT(cond) \ - if (PyCOND_INIT(&(cond))) { \ - Py_FatalError("PyCOND_INIT(" #cond ") failed"); }; -#define COND_FINI(cond) \ - if (PyCOND_FINI(&(cond))) { \ - Py_FatalError("PyCOND_FINI(" #cond ") failed"); }; -#define COND_SIGNAL(cond) \ - if (PyCOND_SIGNAL(&(cond))) { \ - Py_FatalError("PyCOND_SIGNAL(" #cond ") failed"); }; -#define COND_WAIT(cond, mut) \ - if (PyCOND_WAIT(&(cond), &(mut))) { \ - Py_FatalError("PyCOND_WAIT(" #cond ") failed"); }; -#define COND_TIMED_WAIT(cond, mut, microseconds, timeout_result) \ - { \ - int r = PyCOND_TIMEDWAIT(&(cond), &(mut), (microseconds)); \ - if (r < 0) \ - Py_FatalError("PyCOND_WAIT(" #cond ") failed"); \ - if (r) /* 1 == timeout, 2 == impl. can't say, so assume timeout */ \ - timeout_result = 1; \ - else \ - timeout_result = 0; \ - } \ - - -#define DEFAULT_INTERVAL 5000 - -static void _gil_initialize(struct _gil_runtime_state *gil) -{ - _Py_atomic_int uninitialized = {-1}; - gil->locked = uninitialized; - gil->interval = DEFAULT_INTERVAL; -} - -static int gil_created(struct _gil_runtime_state *gil) -{ - return (_Py_atomic_load_explicit(&gil->locked, _Py_memory_order_acquire) >= 0); -} - -static void create_gil(struct _gil_runtime_state *gil) -{ - MUTEX_INIT(gil->mutex); -#ifdef FORCE_SWITCHING - MUTEX_INIT(gil->switch_mutex); -#endif - COND_INIT(gil->cond); -#ifdef FORCE_SWITCHING - COND_INIT(gil->switch_cond); -#endif - _Py_atomic_store_relaxed(&gil->last_holder, 0); - _Py_ANNOTATE_RWLOCK_CREATE(&gil->locked); - _Py_atomic_store_explicit(&gil->locked, 0, _Py_memory_order_release); -} - -static void destroy_gil(struct _gil_runtime_state *gil) -{ - /* some pthread-like implementations tie the mutex to the cond - * and must have the cond destroyed first. - */ - COND_FINI(gil->cond); - MUTEX_FINI(gil->mutex); -#ifdef FORCE_SWITCHING - COND_FINI(gil->switch_cond); - MUTEX_FINI(gil->switch_mutex); -#endif - _Py_atomic_store_explicit(&gil->locked, -1, - _Py_memory_order_release); - _Py_ANNOTATE_RWLOCK_DESTROY(&gil->locked); -} - -#ifdef HAVE_FORK -static void recreate_gil(struct _gil_runtime_state *gil) -{ - _Py_ANNOTATE_RWLOCK_DESTROY(&gil->locked); - /* XXX should we destroy the old OS resources here? */ - create_gil(gil); -} -#endif - -static void -drop_gil(struct _ceval_runtime_state *ceval, struct _ceval_state *ceval2, - PyThreadState *tstate) -{ - struct _gil_runtime_state *gil = &ceval->gil; - if (!_Py_atomic_load_relaxed(&gil->locked)) { - Py_FatalError("drop_gil: GIL is not locked"); - } - - /* tstate is allowed to be NULL (early interpreter init) */ - if (tstate != NULL) { - /* Sub-interpreter support: threads might have been switched - under our feet using PyThreadState_Swap(). Fix the GIL last - holder variable so that our heuristics work. */ - _Py_atomic_store_relaxed(&gil->last_holder, (uintptr_t)tstate); - } - - MUTEX_LOCK(gil->mutex); - _Py_ANNOTATE_RWLOCK_RELEASED(&gil->locked, /*is_write=*/1); - _Py_atomic_store_relaxed(&gil->locked, 0); - COND_SIGNAL(gil->cond); - MUTEX_UNLOCK(gil->mutex); - -#ifdef FORCE_SWITCHING - if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request) && tstate != NULL) { - MUTEX_LOCK(gil->switch_mutex); - /* Not switched yet => wait */ - if (((PyThreadState*)_Py_atomic_load_relaxed(&gil->last_holder)) == tstate) - { - assert(is_tstate_valid(tstate)); - RESET_GIL_DROP_REQUEST(tstate->interp); - /* NOTE: if COND_WAIT does not atomically start waiting when - releasing the mutex, another thread can run through, take - the GIL and drop it again, and reset the condition - before we even had a chance to wait for it. */ - COND_WAIT(gil->switch_cond, gil->switch_mutex); - } - MUTEX_UNLOCK(gil->switch_mutex); - } -#endif -} - - -/* Check if a Python thread must exit immediately, rather than taking the GIL - if Py_Finalize() has been called. - - When this function is called by a daemon thread after Py_Finalize() has been - called, the GIL does no longer exist. - - tstate must be non-NULL. */ -static inline int -tstate_must_exit(PyThreadState *tstate) -{ - /* bpo-39877: Access _PyRuntime directly rather than using - tstate->interp->runtime to support calls from Python daemon threads. - After Py_Finalize() has been called, tstate can be a dangling pointer: - point to PyThreadState freed memory. */ - PyThreadState *finalizing = _PyRuntimeState_GetFinalizing(&_PyRuntime); - return (finalizing != NULL && finalizing != tstate); -} - - -/* Take the GIL. - - The function saves errno at entry and restores its value at exit. - - tstate must be non-NULL. */ -static void -take_gil(PyThreadState *tstate) -{ - int err = errno; - - assert(tstate != NULL); - - if (tstate_must_exit(tstate)) { - /* bpo-39877: If Py_Finalize() has been called and tstate is not the - thread which called Py_Finalize(), exit immediately the thread. - - This code path can be reached by a daemon thread after Py_Finalize() - completes. In this case, tstate is a dangling pointer: points to - PyThreadState freed memory. */ - PyThread_exit_thread(); - } - - assert(is_tstate_valid(tstate)); - PyInterpreterState *interp = tstate->interp; - struct _ceval_runtime_state *ceval = &interp->runtime->ceval; - struct _ceval_state *ceval2 = &interp->ceval; - struct _gil_runtime_state *gil = &ceval->gil; - - /* Check that _PyEval_InitThreads() was called to create the lock */ - assert(gil_created(gil)); - - MUTEX_LOCK(gil->mutex); - - if (!_Py_atomic_load_relaxed(&gil->locked)) { - goto _ready; - } - - while (_Py_atomic_load_relaxed(&gil->locked)) { - unsigned long saved_switchnum = gil->switch_number; - - unsigned long interval = (gil->interval >= 1 ? gil->interval : 1); - int timed_out = 0; - COND_TIMED_WAIT(gil->cond, gil->mutex, interval, timed_out); - - /* If we timed out and no switch occurred in the meantime, it is time - to ask the GIL-holding thread to drop it. */ - if (timed_out && - _Py_atomic_load_relaxed(&gil->locked) && - gil->switch_number == saved_switchnum) - { - if (tstate_must_exit(tstate)) { - MUTEX_UNLOCK(gil->mutex); - PyThread_exit_thread(); - } - assert(is_tstate_valid(tstate)); - - SET_GIL_DROP_REQUEST(interp); - } - } - -_ready: -#ifdef FORCE_SWITCHING - /* This mutex must be taken before modifying gil->last_holder: - see drop_gil(). */ - MUTEX_LOCK(gil->switch_mutex); -#endif - /* We now hold the GIL */ - _Py_atomic_store_relaxed(&gil->locked, 1); - _Py_ANNOTATE_RWLOCK_ACQUIRED(&gil->locked, /*is_write=*/1); - - if (tstate != (PyThreadState*)_Py_atomic_load_relaxed(&gil->last_holder)) { - _Py_atomic_store_relaxed(&gil->last_holder, (uintptr_t)tstate); - ++gil->switch_number; - } - -#ifdef FORCE_SWITCHING - COND_SIGNAL(gil->switch_cond); - MUTEX_UNLOCK(gil->switch_mutex); -#endif - - if (tstate_must_exit(tstate)) { - /* bpo-36475: If Py_Finalize() has been called and tstate is not - the thread which called Py_Finalize(), exit immediately the - thread. - - This code path can be reached by a daemon thread which was waiting - in take_gil() while the main thread called - wait_for_thread_shutdown() from Py_Finalize(). */ - MUTEX_UNLOCK(gil->mutex); - drop_gil(ceval, ceval2, tstate); - PyThread_exit_thread(); - } - assert(is_tstate_valid(tstate)); - - if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request)) { - RESET_GIL_DROP_REQUEST(interp); - } - else { - /* bpo-40010: eval_breaker should be recomputed to be set to 1 if there - is a pending signal: signal received by another thread which cannot - handle signals. - - Note: RESET_GIL_DROP_REQUEST() calls COMPUTE_EVAL_BREAKER(). */ - COMPUTE_EVAL_BREAKER(interp, ceval, ceval2); - } - - /* Don't access tstate if the thread must exit */ - if (tstate->async_exc != NULL) { - _PyEval_SignalAsyncExc(tstate->interp); - } - - MUTEX_UNLOCK(gil->mutex); - - errno = err; -} - -void _PyEval_SetSwitchInterval(unsigned long microseconds) -{ - struct _gil_runtime_state *gil = &_PyRuntime.ceval.gil; - gil->interval = microseconds; -} - -unsigned long _PyEval_GetSwitchInterval() -{ - struct _gil_runtime_state *gil = &_PyRuntime.ceval.gil; - return gil->interval; -} diff --git a/Tools/c-analyzer/cpython/_parser.py b/Tools/c-analyzer/cpython/_parser.py index 992d2e5a7c3d..dc8423bfcbad 100644 --- a/Tools/c-analyzer/cpython/_parser.py +++ b/Tools/c-analyzer/cpython/_parser.py @@ -174,7 +174,6 @@ def clean_lines(text): Objects/stringlib/unicode_format.h Py_BUILD_CORE 1 Parser/string_parser.h Py_BUILD_CORE 1 Parser/pegen.h Py_BUILD_CORE 1 -Python/ceval_gil.h Py_BUILD_CORE 1 Python/condvar.h Py_BUILD_CORE 1 Modules/_json.c Py_BUILD_CORE_BUILTIN 1 diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 899cb6c7dea0..303409cb0077 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -1752,7 +1752,7 @@ def is_other_python_frame(self): def is_waiting_for_gil(self): '''Is this frame waiting on the GIL?''' - # This assumes the _POSIX_THREADS version of Python/ceval_gil.h: + # This assumes the _POSIX_THREADS version of Python/ceval_gil.c: name = self._gdbframe.name() if name: return (name == 'take_gil')

1 0

gh-96021: Explicitly close the IsolatedAsyncioTestCase runner in tests (GH-96135)
by serhiy-storchaka Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/4de06e3cc0a58d73934f9a2759ad9cd2f6… commit: 4de06e3cc0a58d73934f9a2759ad9cd2f6b031b0 branch: main author: Serhiy Storchaka <storchaka(a)gmail.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2022-08-24T15:07:20+03:00 summary: gh-96021: Explicitly close the IsolatedAsyncioTestCase runner in tests (GH-96135) Tests for IsolatedAsyncioTestCase.debug() rely on the runner be closed in __del__. It makes tests depending on the GC an unreliable on other implementations. It is better to close the runner explicitly even if currently there is no a public API for this. files: M Lib/test/test_unittest/test_async_case.py diff --git a/Lib/test/test_unittest/test_async_case.py b/Lib/test/test_unittest/test_async_case.py index f59fc760d381..d7d4dc91316c 100644 --- a/Lib/test/test_unittest/test_async_case.py +++ b/Lib/test/test_unittest/test_async_case.py @@ -43,10 +43,10 @@ async def __aenter__(self): class TestAsyncCase(unittest.TestCase): maxDiff = None - def tearDown(self): + def setUp(self): # Ensure that IsolatedAsyncioTestCase instances are destroyed before # starting a new event loop - support.gc_collect() + self.addCleanup(support.gc_collect) def test_full_cycle(self): class Test(unittest.IsolatedAsyncioTestCase): @@ -151,6 +151,7 @@ async def on_cleanup(self): events = [] test = Test("test_func") + self.addCleanup(test._tearDownAsyncioRunner) try: test.debug() except MyException: @@ -186,6 +187,7 @@ async def on_cleanup(self): events = [] test = Test("test_func") + self.addCleanup(test._tearDownAsyncioRunner) try: test.debug() except MyException: @@ -221,6 +223,7 @@ async def on_cleanup(self): events = [] test = Test("test_func") + self.addCleanup(test._tearDownAsyncioRunner) try: test.debug() except MyException: @@ -262,6 +265,7 @@ async def on_cleanup2(self): events = [] test = Test("test_func") + self.addCleanup(test._tearDownAsyncioRunner) try: test.debug() except MyException: @@ -424,6 +428,7 @@ async def cleanup(self, fut): events = [] test = Test("test_func") + self.addCleanup(test._tearDownAsyncioRunner) try: test.debug() except MyException:

1 0

gh-93678: add _testinternalcapi.optimize_cfg() and test utils for compiler optimization unit tests (GH-96007)
by iritkatriel Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/420f39f457a97a9379f8423a81776bef42… commit: 420f39f457a97a9379f8423a81776bef428d0746 branch: main author: Irit Katriel <1055913+iritkatriel(a)users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel(a)users.noreply.github.com> date: 2022-08-24T11:02:53+01:00 summary: gh-93678: add _testinternalcapi.optimize_cfg() and test utils for compiler optimization unit tests (GH-96007) files: A Misc/NEWS.d/next/Core and Builtins/2022-08-15-20-52-41.gh-issue-93678.X7GuIJ.rst A Modules/clinic/_testinternalcapi.c.h M Include/internal/pycore_compile.h M Include/internal/pycore_global_strings.h M Include/internal/pycore_runtime_init_generated.h M Lib/test/support/bytecode_helper.py M Lib/test/test_peepholer.py M Modules/_testinternalcapi.c M Python/compile.c diff --git a/Include/internal/pycore_compile.h b/Include/internal/pycore_compile.h index 06a6082cddae..1a628a08ca4e 100644 --- a/Include/internal/pycore_compile.h +++ b/Include/internal/pycore_compile.h @@ -38,6 +38,11 @@ extern int _PyAST_Optimize( struct _arena *arena, _PyASTOptimizeState *state); +/* Access compiler internals for unit testing */ +PyAPI_FUNC(PyObject*) _PyCompile_OptimizeCfg( + PyObject *instructions, + PyObject *consts); + #ifdef __cplusplus } #endif diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index aada22039502..c736bfecd077 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -298,6 +298,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(code) STRUCT_FOR_ID(command) STRUCT_FOR_ID(comment_factory) + STRUCT_FOR_ID(consts) STRUCT_FOR_ID(context) STRUCT_FOR_ID(cookie) STRUCT_FOR_ID(copy) @@ -407,6 +408,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(input) STRUCT_FOR_ID(insert_comments) STRUCT_FOR_ID(insert_pis) + STRUCT_FOR_ID(instructions) STRUCT_FOR_ID(intern) STRUCT_FOR_ID(intersection) STRUCT_FOR_ID(isatty) diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h index 09890cd81201..58d9e934b96c 100644 --- a/Include/internal/pycore_runtime_init_generated.h +++ b/Include/internal/pycore_runtime_init_generated.h @@ -807,6 +807,7 @@ extern "C" { INIT_ID(code), \ INIT_ID(command), \ INIT_ID(comment_factory), \ + INIT_ID(consts), \ INIT_ID(context), \ INIT_ID(cookie), \ INIT_ID(copy), \ @@ -916,6 +917,7 @@ extern "C" { INIT_ID(input), \ INIT_ID(insert_comments), \ INIT_ID(insert_pis), \ + INIT_ID(instructions), \ INIT_ID(intern), \ INIT_ID(intersection), \ INIT_ID(isatty), \ @@ -1916,6 +1918,8 @@ _PyUnicode_InitStaticStrings(void) { PyUnicode_InternInPlace(&string); string = &_Py_ID(comment_factory); PyUnicode_InternInPlace(&string); + string = &_Py_ID(consts); + PyUnicode_InternInPlace(&string); string = &_Py_ID(context); PyUnicode_InternInPlace(&string); string = &_Py_ID(cookie); @@ -2134,6 +2138,8 @@ _PyUnicode_InitStaticStrings(void) { PyUnicode_InternInPlace(&string); string = &_Py_ID(insert_pis); PyUnicode_InternInPlace(&string); + string = &_Py_ID(instructions); + PyUnicode_InternInPlace(&string); string = &_Py_ID(intern); PyUnicode_InternInPlace(&string); string = &_Py_ID(intersection); @@ -5755,6 +5761,10 @@ _PyStaticObjects_CheckRefcnt(void) { _PyObject_Dump((PyObject *)&_Py_ID(comment_factory)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); }; + if (Py_REFCNT((PyObject *)&_Py_ID(consts)) < _PyObject_IMMORTAL_REFCNT) { + _PyObject_Dump((PyObject *)&_Py_ID(consts)); + Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); + }; if (Py_REFCNT((PyObject *)&_Py_ID(context)) < _PyObject_IMMORTAL_REFCNT) { _PyObject_Dump((PyObject *)&_Py_ID(context)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); @@ -6191,6 +6201,10 @@ _PyStaticObjects_CheckRefcnt(void) { _PyObject_Dump((PyObject *)&_Py_ID(insert_pis)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); }; + if (Py_REFCNT((PyObject *)&_Py_ID(instructions)) < _PyObject_IMMORTAL_REFCNT) { + _PyObject_Dump((PyObject *)&_Py_ID(instructions)); + Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); + }; if (Py_REFCNT((PyObject *)&_Py_ID(intern)) < _PyObject_IMMORTAL_REFCNT) { _PyObject_Dump((PyObject *)&_Py_ID(intern)); Py_FatalError("immortal object has less refcnt than expected _PyObject_IMMORTAL_REFCNT"); diff --git a/Lib/test/support/bytecode_helper.py b/Lib/test/support/bytecode_helper.py index 471d4a68f915..05b54911e3f2 100644 --- a/Lib/test/support/bytecode_helper.py +++ b/Lib/test/support/bytecode_helper.py @@ -3,6 +3,7 @@ import unittest import dis import io +from _testinternalcapi import optimize_cfg _UNSPECIFIED = object() @@ -40,3 +41,95 @@ def assertNotInBytecode(self, x, opname, argval=_UNSPECIFIED): msg = '(%s,%r) occurs in bytecode:\n%s' msg = msg % (opname, argval, disassembly) self.fail(msg) + + +class CfgOptimizationTestCase(unittest.TestCase): + + HAS_ARG = set(dis.hasarg) + HAS_TARGET = set(dis.hasjrel + dis.hasjabs + dis.hasexc) + HAS_ARG_OR_TARGET = HAS_ARG.union(HAS_TARGET) + + def setUp(self): + self.last_label = 0 + + def Label(self): + self.last_label += 1 + return self.last_label + + def complete_insts_info(self, insts): + # fill in omitted fields in location, and oparg 0 for ops with no arg. + instructions = [] + for item in insts: + if isinstance(item, int): + instructions.append(item) + else: + assert isinstance(item, tuple) + inst = list(reversed(item)) + opcode = dis.opmap[inst.pop()] + oparg = inst.pop() if opcode in self.HAS_ARG_OR_TARGET else 0 + loc = inst + [-1] * (4 - len(inst)) + instructions.append((opcode, oparg, *loc)) + return instructions + + def normalize_insts(self, insts): + """ Map labels to instruction index. + Remove labels which are not used as jump targets. + """ + labels_map = {} + targets = set() + idx = 1 + for item in insts: + assert isinstance(item, (int, tuple)) + if isinstance(item, tuple): + opcode, oparg, *_ = item + if dis.opmap.get(opcode, opcode) in self.HAS_TARGET: + targets.add(oparg) + idx += 1 + elif isinstance(item, int): + assert item not in labels_map, "label reused" + labels_map[item] = idx + + res = [] + for item in insts: + if isinstance(item, int) and item in targets: + if not res or labels_map[item] != res[-1]: + res.append(labels_map[item]) + elif isinstance(item, tuple): + opcode, oparg, *loc = item + opcode = dis.opmap.get(opcode, opcode) + if opcode in self.HAS_TARGET: + arg = labels_map[oparg] + else: + arg = oparg if opcode in self.HAS_TARGET else None + opcode = dis.opname[opcode] + res.append((opcode, arg, *loc)) + return res + + def get_optimized(self, insts, consts): + insts = self.complete_insts_info(insts) + insts = optimize_cfg(insts, consts) + return insts, consts + + def compareInstructions(self, actual_, expected_): + # get two lists where each entry is a label or + # an instruction tuple. Compare them, while mapping + # each actual label to a corresponding expected label + # based on their locations. + + self.assertIsInstance(actual_, list) + self.assertIsInstance(expected_, list) + + actual = self.normalize_insts(actual_) + expected = self.normalize_insts(expected_) + self.assertEqual(len(actual), len(expected)) + + # compare instructions + for act, exp in zip(actual, expected): + if isinstance(act, int): + self.assertEqual(exp, act) + continue + self.assertIsInstance(exp, tuple) + self.assertIsInstance(act, tuple) + # pad exp with -1's (if location info is incomplete) + exp += (-1,) * (len(act) - len(exp)) + self.assertEqual(exp, act) diff --git a/Lib/test/test_peepholer.py b/Lib/test/test_peepholer.py index e03c42c2f823..7ece468363be 100644 --- a/Lib/test/test_peepholer.py +++ b/Lib/test/test_peepholer.py @@ -4,7 +4,7 @@ import textwrap import unittest -from test.support.bytecode_helper import BytecodeTestCase +from test.support.bytecode_helper import BytecodeTestCase, CfgOptimizationTestCase def compile_pattern_with_fast_locals(pattern): @@ -864,5 +864,81 @@ def trace(frame, event, arg): self.assertNotInBytecode(f, "LOAD_FAST_CHECK") +class DirectiCfgOptimizerTests(CfgOptimizationTestCase): + + def cfg_optimization_test(self, insts, expected_insts, + consts=None, expected_consts=None): + if expected_consts is None: + expected_consts = consts + opt_insts, opt_consts = self.get_optimized(insts, consts) + self.compareInstructions(opt_insts, expected_insts) + self.assertEqual(opt_consts, expected_consts) + + def test_conditional_jump_forward_non_const_condition(self): + insts = [ + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_CONST', 2, 13), + lbl, + ('LOAD_CONST', 3, 14), + ] + expected = [ + ('LOAD_NAME', '1', 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_CONST', '2', 13), + lbl, + ('LOAD_CONST', '3', 14) + ] + self.cfg_optimization_test(insts, expected, consts=list(range(5))) + + def test_conditional_jump_forward_const_condition(self): + # The unreachable branch of the jump is removed + + insts = [ + ('LOAD_CONST', 3, 11), + ('POP_JUMP_IF_TRUE', lbl := self.Label(), 12), + ('LOAD_CONST', 2, 13), + lbl, + ('LOAD_CONST', 3, 14), + ] + expected = [ + ('NOP', None, 11), + ('JUMP', lbl := self.Label(), 12), + lbl, + ('LOAD_CONST', '3', 14) + ] + self.cfg_optimization_test(insts, expected, consts=list(range(5))) + + def test_conditional_jump_backward_non_const_condition(self): + insts = [ + lbl1 := self.Label(), + ('LOAD_NAME', 1, 11), + ('POP_JUMP_IF_TRUE', lbl1, 12), + ('LOAD_CONST', 2, 13), + ] + expected = [ + lbl := self.Label(), + ('LOAD_NAME', '1', 11), + ('POP_JUMP_IF_TRUE', lbl, 12), + ('LOAD_CONST', '2', 13) + ] + self.cfg_optimization_test(insts, expected, consts=list(range(5))) + + def test_conditional_jump_backward_const_condition(self): + # The unreachable branch of the jump is removed + insts = [ + lbl1 := self.Label(), + ('LOAD_CONST', 1, 11), + ('POP_JUMP_IF_TRUE', lbl1, 12), + ('LOAD_CONST', 2, 13), + ] + expected = [ + lbl := self.Label(), + ('NOP', None, 11), + ('JUMP', lbl, 12) + ] + self.cfg_optimization_test(insts, expected, consts=list(range(5))) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-08-15-20-52-41.gh-issue-93678.X7GuIJ.rst b/Misc/NEWS.d/next/Core and Builtins/2022-08-15-20-52-41.gh-issue-93678.X7GuIJ.rst new file mode 100644 index 000000000000..9e2b90ba07a4 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-08-15-20-52-41.gh-issue-93678.X7GuIJ.rst @@ -0,0 +1 @@ +Added test a harness for direct unit tests of the compiler's optimization stage. The ``_testinternalcapi.optimize_cfg()`` function runs the optimiser on a sequence of instructions. The ``CfgOptimizationTestCase`` class in ``test.support`` has utilities for invoking the optimizer and checking the output. diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index 238de749fffc..9d92b076387f 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -14,6 +14,7 @@ #include "Python.h" #include "pycore_atomic_funcs.h" // _Py_atomic_int_get() #include "pycore_bitutils.h" // _Py_bswap32() +#include "pycore_compile.h" // _PyCompile_OptimizeCfg() #include "pycore_fileutils.h" // _Py_normpath #include "pycore_frame.h" // _PyInterpreterFrame #include "pycore_gc.h" // PyGC_Head @@ -25,7 +26,12 @@ #include "pycore_pystate.h" // _PyThreadState_GET() #include "osdefs.h" // MAXPATHLEN +#include "clinic/_testinternalcapi.c.h" +/*[clinic input] +module _testinternalcapi +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=7bb583d8c9eb9a78]*/ static PyObject * get_configs(PyObject *self, PyObject *Py_UNUSED(args)) { @@ -525,6 +531,25 @@ set_eval_frame_record(PyObject *self, PyObject *list) } +/*[clinic input] + +_testinternalcapi.optimize_cfg -> object + + instructions: object + consts: object + +Apply compiler optimizations to an instruction list. +[clinic start generated code]*/ + +static PyObject * +_testinternalcapi_optimize_cfg_impl(PyObject *module, PyObject *instructions, + PyObject *consts) +/*[clinic end generated code: output=5412aeafca683c8b input=7e8a3de86ebdd0f9]*/ +{ + return _PyCompile_OptimizeCfg(instructions, consts); +} + + static PyMethodDef TestMethods[] = { {"get_configs", get_configs, METH_NOARGS}, {"get_recursion_depth", get_recursion_depth, METH_NOARGS}, @@ -543,6 +568,7 @@ static PyMethodDef TestMethods[] = { {"DecodeLocaleEx", decode_locale_ex, METH_VARARGS}, {"set_eval_frame_default", set_eval_frame_default, METH_NOARGS, NULL}, {"set_eval_frame_record", set_eval_frame_record, METH_O, NULL}, + _TESTINTERNALCAPI_OPTIMIZE_CFG_METHODDEF {NULL, NULL} /* sentinel */ }; diff --git a/Modules/clinic/_testinternalcapi.c.h b/Modules/clinic/_testinternalcapi.c.h new file mode 100644 index 000000000000..8113fff37997 --- /dev/null +++ b/Modules/clinic/_testinternalcapi.c.h @@ -0,0 +1,68 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) +# include "pycore_gc.h" // PyGC_Head +# include "pycore_runtime.h" // _Py_ID() +#endif + + +PyDoc_STRVAR(_testinternalcapi_optimize_cfg__doc__, +"optimize_cfg($module, /, instructions, consts)\n" +"--\n" +"\n" +"Apply compiler optimizations to an instruction list."); + +#define _TESTINTERNALCAPI_OPTIMIZE_CFG_METHODDEF \ + {"optimize_cfg", _PyCFunction_CAST(_testinternalcapi_optimize_cfg), METH_FASTCALL|METH_KEYWORDS, _testinternalcapi_optimize_cfg__doc__}, + +static PyObject * +_testinternalcapi_optimize_cfg_impl(PyObject *module, PyObject *instructions, + PyObject *consts); + +static PyObject * +_testinternalcapi_optimize_cfg(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + #if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE) + + #define NUM_KEYWORDS 2 + static struct { + PyGC_Head _this_is_not_used; + PyObject_VAR_HEAD + PyObject *ob_item[NUM_KEYWORDS]; + } _kwtuple = { + .ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS) + .ob_item = { &_Py_ID(instructions), &_Py_ID(consts), }, + }; + #undef NUM_KEYWORDS + #define KWTUPLE (&_kwtuple.ob_base.ob_base) + + #else // !Py_BUILD_CORE + # define KWTUPLE NULL + #endif // !Py_BUILD_CORE + + static const char * const _keywords[] = {"instructions", "consts", NULL}; + static _PyArg_Parser _parser = { + .keywords = _keywords, + .fname = "optimize_cfg", + .kwtuple = KWTUPLE, + }; + #undef KWTUPLE + PyObject *argsbuf[2]; + PyObject *instructions; + PyObject *consts; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); + if (!args) { + goto exit; + } + instructions = args[0]; + consts = args[1]; + return_value = _testinternalcapi_optimize_cfg_impl(module, instructions, consts); + +exit: + return return_value; +} +/*[clinic end generated code: output=3b1fd713290f68a9 input=a9049054013a1b77]*/ diff --git a/Python/compile.c b/Python/compile.c index 339e0e792be4..e5ac162ccc0a 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -457,6 +457,7 @@ typedef struct { static int basicblock_next_instr(basicblock *); +static basicblock *cfg_builder_new_block(cfg_builder *g); static int cfg_builder_maybe_start_new_block(cfg_builder *g); static int cfg_builder_addop_i(cfg_builder *g, int opcode, Py_ssize_t oparg, struct location loc); @@ -767,8 +768,20 @@ cfg_builder_check(cfg_builder *g) } } +static int +cfg_builder_init(cfg_builder *g) +{ + g->g_block_list = NULL; + basicblock *block = cfg_builder_new_block(g); + if (block == NULL) + return 0; + g->g_curblock = g->g_entryblock = block; + g->g_current_label = NO_LABEL; + return 1; +} + static void -cfg_builder_free(cfg_builder* g) +cfg_builder_fini(cfg_builder* g) { cfg_builder_check(g); basicblock *b = g->g_block_list; @@ -785,7 +798,7 @@ cfg_builder_free(cfg_builder* g) static void compiler_unit_free(struct compiler_unit *u) { - cfg_builder_free(&u->u_cfg_builder); + cfg_builder_fini(&u->u_cfg_builder); Py_CLEAR(u->u_ste); Py_CLEAR(u->u_name); Py_CLEAR(u->u_qualname); @@ -1708,7 +1721,6 @@ compiler_enter_scope(struct compiler *c, identifier name, int scope_type, void *key, int lineno) { struct compiler_unit *u; - basicblock *block; u = (struct compiler_unit *)PyObject_Calloc(1, sizeof( struct compiler_unit)); @@ -1786,12 +1798,9 @@ compiler_enter_scope(struct compiler *c, identifier name, c->c_nestlevel++; cfg_builder *g = CFG_BUILDER(c); - g->g_block_list = NULL; - block = cfg_builder_new_block(g); - if (block == NULL) + if (!cfg_builder_init(g)) { return 0; - g->g_curblock = g->g_entryblock = block; - g->g_current_label = NO_LABEL; + } if (u->u_scope_type == COMPILER_SCOPE_MODULE) { c->u->u_loc.lineno = 0; @@ -8220,7 +8229,7 @@ dump_instr(struct instr *i) sprintf(arg, "arg: %d ", i->i_oparg); } if (HAS_TARGET(i->i_opcode)) { - sprintf(arg, "target: %p ", i->i_target); + sprintf(arg, "target: %p [%d] ", i->i_target, i->i_oparg); } fprintf(stderr, "line: %d, opcode: %d %s%s%s\n", i->i_loc.lineno, i->i_opcode, arg, jabs, jrel); @@ -8251,7 +8260,7 @@ static int calculate_jump_targets(basicblock *entryblock); static int -optimize_cfg(basicblock *entryblock, PyObject *consts, PyObject *const_cache); +optimize_cfg(cfg_builder *g, PyObject *consts, PyObject *const_cache); static int trim_unused_consts(basicblock *entryblock, PyObject *consts); @@ -8465,18 +8474,18 @@ static void propagate_line_numbers(basicblock *entryblock); static void -eliminate_empty_basic_blocks(basicblock *entryblock); +eliminate_empty_basic_blocks(cfg_builder *g); static int -remove_redundant_jumps(basicblock *entryblock) { +remove_redundant_jumps(cfg_builder *g) { /* If a non-empty block ends with a jump instruction, check if the next * non-empty block reached through normal flow control is the target * of that jump. If it is, then the jump instruction is redundant and * can be deleted. */ int removed = 0; - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (b->b_iused > 0) { struct instr *b_last_instr = &b->b_instr[b->b_iused - 1]; assert(!IS_ASSEMBLER_OPCODE(b_last_instr->i_opcode)); @@ -8495,7 +8504,7 @@ remove_redundant_jumps(basicblock *entryblock) { } } if (removed) { - eliminate_empty_basic_blocks(entryblock); + eliminate_empty_basic_blocks(g); } return 0; } @@ -8545,13 +8554,12 @@ assemble(struct compiler *c, int addNone) } cfg_builder *g = CFG_BUILDER(c); - basicblock *entryblock = g->g_entryblock; - assert(entryblock != NULL); + assert(g->g_entryblock != NULL); /* Set firstlineno if it wasn't explicitly set. */ if (!c->u->u_firstlineno) { - if (entryblock->b_instr && entryblock->b_instr->i_loc.lineno) { - c->u->u_firstlineno = entryblock->b_instr->i_loc.lineno; + if (g->g_entryblock->b_instr && g->g_entryblock->b_instr->i_loc.lineno) { + c->u->u_firstlineno = g->g_entryblock->b_instr->i_loc.lineno; } else { c->u->u_firstlineno = 1; @@ -8559,11 +8567,11 @@ assemble(struct compiler *c, int addNone) } // This must be called before fix_cell_offsets(). - if (insert_prefix_instructions(c, entryblock, cellfixedoffsets, nfreevars, code_flags)) { + if (insert_prefix_instructions(c, g->g_entryblock, cellfixedoffsets, nfreevars, code_flags)) { goto error; } - int numdropped = fix_cell_offsets(c, entryblock, cellfixedoffsets); + int numdropped = fix_cell_offsets(c, g->g_entryblock, cellfixedoffsets); PyMem_Free(cellfixedoffsets); // At this point we're done with it. cellfixedoffsets = NULL; if (numdropped < 0) { @@ -8575,52 +8583,52 @@ assemble(struct compiler *c, int addNone) if (consts == NULL) { goto error; } - if (calculate_jump_targets(entryblock)) { + if (calculate_jump_targets(g->g_entryblock)) { goto error; } - if (optimize_cfg(entryblock, consts, c->c_const_cache)) { + if (optimize_cfg(g, consts, c->c_const_cache)) { goto error; } - if (trim_unused_consts(entryblock, consts)) { + if (trim_unused_consts(g->g_entryblock, consts)) { goto error; } if (duplicate_exits_without_lineno(g)) { return NULL; } - propagate_line_numbers(entryblock); - guarantee_lineno_for_exits(entryblock, c->u->u_firstlineno); + propagate_line_numbers(g->g_entryblock); + guarantee_lineno_for_exits(g->g_entryblock, c->u->u_firstlineno); - int maxdepth = stackdepth(entryblock, code_flags); + int maxdepth = stackdepth(g->g_entryblock, code_flags); if (maxdepth < 0) { goto error; } /* TO DO -- For 3.12, make sure that `maxdepth <= MAX_ALLOWED_STACK_USE` */ - if (label_exception_targets(entryblock)) { + if (label_exception_targets(g->g_entryblock)) { goto error; } - convert_exception_handlers_to_nops(entryblock); + convert_exception_handlers_to_nops(g->g_entryblock); if (push_cold_blocks_to_end(g, code_flags) < 0) { goto error; } - if (remove_redundant_jumps(entryblock) < 0) { + if (remove_redundant_jumps(g) < 0) { goto error; } - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { clean_basic_block(b); } /* Order of basic blocks must have been determined by now */ - normalize_jumps(entryblock); + normalize_jumps(g->g_entryblock); - if (add_checks_for_loads_of_unknown_variables(entryblock, c) < 0) { + if (add_checks_for_loads_of_unknown_variables(g->g_entryblock, c) < 0) { goto error; } /* Can't modify the bytecode after computing jump offsets. */ - assemble_jump_offsets(entryblock); + assemble_jump_offsets(g->g_entryblock); /* Create assembler */ @@ -8628,7 +8636,7 @@ assemble(struct compiler *c, int addNone) goto error; /* Emit code. */ - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { for (int j = 0; j < b->b_iused; j++) if (!assemble_emit(&a, &b->b_instr[j])) goto error; @@ -8636,13 +8644,13 @@ assemble(struct compiler *c, int addNone) /* Emit location info */ a.a_lineno = c->u->u_firstlineno; - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { for (int j = 0; j < b->b_iused; j++) if (!assemble_emit_location(&a, &b->b_instr[j])) goto error; } - if (!assemble_exception_table(&a, entryblock)) { + if (!assemble_exception_table(&a, g->g_entryblock)) { goto error; } if (_PyBytes_Resize(&a.a_except_table, a.a_except_table_off) < 0) { @@ -9352,16 +9360,19 @@ mark_reachable(basicblock *entryblock) { } static void -eliminate_empty_basic_blocks(basicblock *entryblock) { +eliminate_empty_basic_blocks(cfg_builder *g) { /* Eliminate empty blocks */ - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { basicblock *next = b->b_next; while (next && next->b_iused == 0) { next = next->b_next; } b->b_next = next; } - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + while(g->g_entryblock && g->g_entryblock->b_iused == 0) { + g->g_entryblock = g->g_entryblock->b_next; + } + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { assert(b->b_iused > 0); for (int i = 0; i < b->b_iused; i++) { struct instr *instr = &b->b_instr[i]; @@ -9467,42 +9478,42 @@ calculate_jump_targets(basicblock *entryblock) */ static int -optimize_cfg(basicblock *entryblock, PyObject *consts, PyObject *const_cache) +optimize_cfg(cfg_builder *g, PyObject *consts, PyObject *const_cache) { assert(PyDict_CheckExact(const_cache)); - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (normalize_basic_block(b)) { return -1; } } - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (extend_block(b)) { return -1; } } - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (optimize_basic_block(const_cache, b, consts)) { return -1; } clean_basic_block(b); assert(b->b_predecessors == 0); } - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (extend_block(b)) { return -1; } } - if (mark_reachable(entryblock)) { + if (mark_reachable(g->g_entryblock)) { return -1; } /* Delete unreachable instructions */ - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { if (b->b_predecessors == 0) { b->b_iused = 0; } } - eliminate_empty_basic_blocks(entryblock); - for (basicblock *b = entryblock; b != NULL; b = b->b_next) { + eliminate_empty_basic_blocks(g); + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { clean_basic_block(b); } return 0; @@ -9601,6 +9612,157 @@ duplicate_exits_without_lineno(cfg_builder *g) } +/* Access to compiler optimizations for unit tests. + * + * _PyCompile_OptimizeCfg takes an instruction list, constructs + * a CFG, optimizes it and converts back to an instruction list. + * + * An instruction list is a PyList where each item is either + * a tuple describing a single instruction: + * (opcode, oparg, lineno, end_lineno, col, end_col), or + * a jump target label marking the beginning of a basic block. + */ + +static int +instructions_to_cfg(PyObject *instructions, cfg_builder *g) +{ + assert(PyList_Check(instructions)); + + Py_ssize_t instr_size = PyList_GET_SIZE(instructions); + for (Py_ssize_t i = 0; i < instr_size; i++) { + PyObject *item = PyList_GET_ITEM(instructions, i); + if (PyLong_Check(item)) { + int lbl_id = PyLong_AsLong(item); + if (PyErr_Occurred()) { + return -1; + } + if (lbl_id <= 0 || lbl_id > instr_size) { + /* expect label in a reasonable range */ + PyErr_SetString(PyExc_ValueError, "label out of range"); + return -1; + } + jump_target_label lbl = {lbl_id}; + if (cfg_builder_use_label(g, lbl) < 0) { + return -1; + } + } + else { + if (!PyTuple_Check(item) || PyTuple_GET_SIZE(item) != 6) { + PyErr_SetString(PyExc_ValueError, "expected a 6-tuple"); + return -1; + } + int opcode = PyLong_AsLong(PyTuple_GET_ITEM(item, 0)); + if (PyErr_Occurred()) { + return -1; + } + int oparg = PyLong_AsLong(PyTuple_GET_ITEM(item, 1)); + if (PyErr_Occurred()) { + return -1; + } + struct location loc; + loc.lineno = PyLong_AsLong(PyTuple_GET_ITEM(item, 2)); + if (PyErr_Occurred()) { + return -1; + } + loc.end_lineno = PyLong_AsLong(PyTuple_GET_ITEM(item, 3)); + if (PyErr_Occurred()) { + return -1; + } + loc.col_offset = PyLong_AsLong(PyTuple_GET_ITEM(item, 4)); + if (PyErr_Occurred()) { + return -1; + } + loc.end_col_offset = PyLong_AsLong(PyTuple_GET_ITEM(item, 5)); + if (PyErr_Occurred()) { + return -1; + } + if (!cfg_builder_addop(g, opcode, oparg, loc)) { + return -1; + } + } + } + return 0; +} + +static PyObject * +cfg_to_instructions(cfg_builder *g) +{ + PyObject *instructions = PyList_New(0); + if (instructions == NULL) { + return NULL; + } + int lbl = 1; + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { + b->b_label = lbl++; + } + for (basicblock *b = g->g_entryblock; b != NULL; b = b->b_next) { + PyObject *lbl = PyLong_FromLong(b->b_label); + if (lbl == NULL) { + goto error; + } + if (PyList_Append(instructions, lbl) != 0) { + Py_DECREF(lbl); + goto error; + } + Py_DECREF(lbl); + for (int i = 0; i < b->b_iused; i++) { + struct instr *instr = &b->b_instr[i]; + struct location loc = instr->i_loc; + int arg = HAS_TARGET(instr->i_opcode) ? instr->i_target->b_label : instr->i_oparg; + PyObject *inst_tuple = Py_BuildValue( + "(iiiiii)", instr->i_opcode, arg, + loc.lineno, loc.end_lineno, + loc.col_offset, loc.end_col_offset); + if (inst_tuple == NULL) { + goto error; + } + + if (PyList_Append(instructions, inst_tuple) != 0) { + Py_DECREF(inst_tuple); + goto error; + } + Py_DECREF(inst_tuple); + } + } + + return instructions; +error: + Py_DECREF(instructions); + return NULL; +} + + +PyObject * +_PyCompile_OptimizeCfg(PyObject *instructions, PyObject *consts) +{ + PyObject *res = NULL; + PyObject *const_cache = NULL; + cfg_builder g; + memset(&g, 0, sizeof(cfg_builder)); + if (cfg_builder_init(&g) < 0) { + goto error; + } + if (instructions_to_cfg(instructions, &g) < 0) { + goto error; + } + const_cache = PyDict_New(); + if (const_cache == NULL) { + goto error; + } + if (calculate_jump_targets(g.g_entryblock)) { + goto error; + } + if (optimize_cfg(&g, consts, const_cache) < 0) { + goto error; + } + res = cfg_to_instructions(&g); +error: + Py_XDECREF(const_cache); + cfg_builder_fini(&g); + return res; +} + + /* Retained for API compatibility. * Optimization is now done in optimize_cfg */

1 0

[3.10] gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (GH-96136) (#96227)
by erlend-aasland Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/203b598e5113f3662bfbaf3474dd62a258… commit: 203b598e5113f3662bfbaf3474dd62a258b8b6ce branch: 3.10 author: Erlend E. Aasland <erlend.aasland(a)protonmail.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2022-08-24T09:52:16+02:00 summary: [3.10] gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (GH-96136) (#96227) Co-authored-by: Erlend E. Aasland <erlend.aasland(a)protonmail.com> Co-authored-by: Ezio Melotti <ezio.melotti(a)gmail.com>. (cherry picked from commit 6bda5b85b53443f3467865fbf85cbe72932e7cd6) Co-authored-by: C.A.M. Gerlach <CAM.Gerlach(a)Gerlach.CAM> files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index be9a22efad1..94d1e1b0e7a 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1233,6 +1233,38 @@ and you can let the :mod:`!sqlite3` module convert SQLite types to Python types via :ref:`converters <sqlite3-converters>`. +.. _sqlite3-default-converters: + +Default adapters and converters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There are default adapters for the date and datetime types in the datetime +module. They will be sent as ISO dates/ISO timestamps to SQLite. + +The default converters are registered under the name "date" for +:class:`datetime.date` and under the name "timestamp" for +:class:`datetime.datetime`. + +This way, you can use date/timestamps from Python without any additional +fiddling in most cases. The format of the adapters is also compatible with the +experimental SQLite date/time functions. + +The following example demonstrates this. + +.. literalinclude:: ../includes/sqlite3/pysqlite_datetime.py + +If a timestamp stored in SQLite has a fractional part longer than 6 +numbers, its value will be truncated to microsecond precision by the +timestamp converter. + +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. + + .. _sqlite3-howtos: How-to guides @@ -1270,8 +1302,8 @@ both styles: .. _sqlite3-adapters: -Using adapters to store custom Python types in SQLite databases -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to adapt custom Python types to SQLite values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SQLite supports only a limited set of data types natively. To store custom Python types in SQLite databases, *adapt* them to one of the @@ -1288,8 +1320,8 @@ registering custom adapter functions. .. _sqlite3-conform: -Letting your object adapt itself -"""""""""""""""""""""""""""""""" +How to write adaptable objects +"""""""""""""""""""""""""""""" Suppose we have a :class:`!Point` class that represents a pair of coordinates, ``x`` and ``y``, in a Cartesian coordinate system. @@ -1302,8 +1334,8 @@ The object passed to *protocol* will be of type :class:`PrepareProtocol`. .. literalinclude:: ../includes/sqlite3/adapter_point_1.py -Registering an adapter callable -""""""""""""""""""""""""""""""" +How to register adapter callables +""""""""""""""""""""""""""""""""" The other possibility is to create a function that converts the Python object to an SQLite-compatible type. @@ -1314,8 +1346,8 @@ This function can then be registered using :func:`register_adapter`. .. _sqlite3-converters: -Converting SQLite values to custom Python types -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to convert SQLite values to custom Python types +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Writing an adapter lets you convert *from* custom Python types *to* SQLite values. @@ -1354,36 +1386,6 @@ The following example illustrates the implicit and explicit approaches: .. literalinclude:: ../includes/sqlite3/converter_point.py -Default adapters and converters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are default adapters for the date and datetime types in the datetime -module. They will be sent as ISO dates/ISO timestamps to SQLite. - -The default converters are registered under the name "date" for -:class:`datetime.date` and under the name "timestamp" for -:class:`datetime.datetime`. - -This way, you can use date/timestamps from Python without any additional -fiddling in most cases. The format of the adapters is also compatible with the -experimental SQLite date/time functions. - -The following example demonstrates this. - -.. literalinclude:: ../includes/sqlite3/pysqlite_datetime.py - -If a timestamp stored in SQLite has a fractional part longer than 6 -numbers, its value will be truncated to microsecond precision by the -timestamp converter. - -.. note:: - - The default "timestamp" converter ignores UTC offsets in the database and - always returns a naive :class:`datetime.datetime` object. To preserve UTC - offsets in timestamps, either leave converters disabled, or register an - offset-aware converter with :func:`register_converter`. - - .. _sqlite3-adapter-converter-recipes: Adapter and converter recipes @@ -1431,8 +1433,8 @@ This section shows recipes for common adapters and converters. .. _sqlite3-connection-shortcuts: -Using connection shortcut methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to use connection shortcut methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the :meth:`~Connection.execute`, :meth:`~Connection.executemany`, and :meth:`~Connection.executescript` @@ -1448,7 +1450,7 @@ directly using only a single call on the :class:`Connection` object. .. _sqlite3-connection-context-manager: -Using the connection as a context manager +How to use the connection context manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A :class:`Connection` object can be used as a context manager that @@ -1473,8 +1475,8 @@ the context manager is a no-op. .. _sqlite3-uri-tricks: -Working with SQLite URIs -^^^^^^^^^^^^^^^^^^^^^^^^ +How to work with SQLite URIs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some useful URI tricks include:

1 0

[3.11] gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (GH-96136) (#96226)
by erlend-aasland Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/2b8fd745489c0c51f05bba510832687160… commit: 2b8fd745489c0c51f05bba510832687160be7db4 branch: 3.11 author: Erlend E. Aasland <erlend.aasland(a)protonmail.com> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2022-08-24T09:51:46+02:00 summary: [3.11] gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (GH-96136) (#96226) Co-authored-by: Erlend E. Aasland <erlend.aasland(a)protonmail.com> Co-authored-by: Ezio Melotti <ezio.melotti(a)gmail.com>. (cherry picked from commit 6bda5b85b53443f3467865fbf85cbe72932e7cd6) Co-authored-by: C.A.M. Gerlach <CAM.Gerlach(a)Gerlach.CAM> files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index cf496b7f6a6..4cc8b77f381 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1530,6 +1530,38 @@ and you can let the :mod:`!sqlite3` module convert SQLite types to Python types via :ref:`converters <sqlite3-converters>`. +.. _sqlite3-default-converters: + +Default adapters and converters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There are default adapters for the date and datetime types in the datetime +module. They will be sent as ISO dates/ISO timestamps to SQLite. + +The default converters are registered under the name "date" for +:class:`datetime.date` and under the name "timestamp" for +:class:`datetime.datetime`. + +This way, you can use date/timestamps from Python without any additional +fiddling in most cases. The format of the adapters is also compatible with the +experimental SQLite date/time functions. + +The following example demonstrates this. + +.. literalinclude:: ../includes/sqlite3/pysqlite_datetime.py + +If a timestamp stored in SQLite has a fractional part longer than 6 +numbers, its value will be truncated to microsecond precision by the +timestamp converter. + +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. + + .. _sqlite3-howtos: How-to guides @@ -1567,8 +1599,8 @@ both styles: .. _sqlite3-adapters: -Using adapters to store custom Python types in SQLite databases -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to adapt custom Python types to SQLite values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SQLite supports only a limited set of data types natively. To store custom Python types in SQLite databases, *adapt* them to one of the @@ -1585,8 +1617,8 @@ registering custom adapter functions. .. _sqlite3-conform: -Letting your object adapt itself -"""""""""""""""""""""""""""""""" +How to write adaptable objects +"""""""""""""""""""""""""""""" Suppose we have a :class:`!Point` class that represents a pair of coordinates, ``x`` and ``y``, in a Cartesian coordinate system. @@ -1599,8 +1631,8 @@ The object passed to *protocol* will be of type :class:`PrepareProtocol`. .. literalinclude:: ../includes/sqlite3/adapter_point_1.py -Registering an adapter callable -""""""""""""""""""""""""""""""" +How to register adapter callables +""""""""""""""""""""""""""""""""" The other possibility is to create a function that converts the Python object to an SQLite-compatible type. @@ -1611,8 +1643,8 @@ This function can then be registered using :func:`register_adapter`. .. _sqlite3-converters: -Converting SQLite values to custom Python types -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to convert SQLite values to custom Python types +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Writing an adapter lets you convert *from* custom Python types *to* SQLite values. @@ -1651,36 +1683,6 @@ The following example illustrates the implicit and explicit approaches: .. literalinclude:: ../includes/sqlite3/converter_point.py -Default adapters and converters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are default adapters for the date and datetime types in the datetime -module. They will be sent as ISO dates/ISO timestamps to SQLite. - -The default converters are registered under the name "date" for -:class:`datetime.date` and under the name "timestamp" for -:class:`datetime.datetime`. - -This way, you can use date/timestamps from Python without any additional -fiddling in most cases. The format of the adapters is also compatible with the -experimental SQLite date/time functions. - -The following example demonstrates this. - -.. literalinclude:: ../includes/sqlite3/pysqlite_datetime.py - -If a timestamp stored in SQLite has a fractional part longer than 6 -numbers, its value will be truncated to microsecond precision by the -timestamp converter. - -.. note:: - - The default "timestamp" converter ignores UTC offsets in the database and - always returns a naive :class:`datetime.datetime` object. To preserve UTC - offsets in timestamps, either leave converters disabled, or register an - offset-aware converter with :func:`register_converter`. - - .. _sqlite3-adapter-converter-recipes: Adapter and converter recipes @@ -1728,8 +1730,8 @@ This section shows recipes for common adapters and converters. .. _sqlite3-connection-shortcuts: -Using connection shortcut methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to use connection shortcut methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the :meth:`~Connection.execute`, :meth:`~Connection.executemany`, and :meth:`~Connection.executescript` @@ -1745,7 +1747,7 @@ directly using only a single call on the :class:`Connection` object. .. _sqlite3-connection-context-manager: -Using the connection as a context manager +How to use the connection context manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A :class:`Connection` object can be used as a context manager that @@ -1770,8 +1772,8 @@ the context manager is a no-op. .. _sqlite3-uri-tricks: -Working with SQLite URIs -^^^^^^^^^^^^^^^^^^^^^^^^ +How to work with SQLite URIs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some useful URI tricks include:

1 0

gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (#96136)
by erlend-aasland Aug. 24, 2022

Aug. 24, 2022

https://github.com/python/cpython/commit/6bda5b85b53443f3467865fbf85cbe7293… commit: 6bda5b85b53443f3467865fbf85cbe72932e7cd6 branch: main author: C.A.M. Gerlach <CAM.Gerlach(a)Gerlach.CAM> committer: erlend-aasland <erlend.aasland(a)protonmail.com> date: 2022-08-24T09:14:14+02:00 summary: gh-94635: Frame sqlite3 how-to headings as such & move default adapters to reference (#96136) Co-authored-by: Erlend E. Aasland <erlend.aasland(a)protonmail.com> Co-authored-by: Ezio Melotti <ezio.melotti(a)gmail.com> files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 798c2d795b1..1e54463c11f 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1543,6 +1543,41 @@ and you can let the :mod:`!sqlite3` module convert SQLite types to Python types via :ref:`converters <sqlite3-converters>`. +.. _sqlite3-default-converters: + +Default adapters and converters (deprecated) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. note:: + + The default adapters and converters are deprecated as of Python 3.12. + Instead, use the :ref:`sqlite3-adapter-converter-recipes` + and tailor them to your needs. + +The deprecated default adapters and converters consist of: + +* An adapter for :class:`datetime.date` objects to :class:`strings <str>` in + `ISO 8601`_ format. +* An adapter for :class:`datetime.datetime` objects to strings in + ISO 8601 format. +* A converter for :ref:`declared <sqlite3-converters>` "date" types to + :class:`datetime.date` objects. +* A converter for declared "timestamp" types to + :class:`datetime.datetime` objects. + Fractional parts will be truncated to 6 digits (microsecond precision). + +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. + +.. deprecated:: 3.12 + +.. _ISO 8601: https://en.wikipedia.org/wiki/ISO_8601 + + .. _sqlite3-cli: Command-line interface @@ -1602,8 +1637,8 @@ both styles: .. _sqlite3-adapters: -Using adapters to store custom Python types in SQLite databases -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to adapt custom Python types to SQLite values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SQLite supports only a limited set of data types natively. To store custom Python types in SQLite databases, *adapt* them to one of the @@ -1620,8 +1655,8 @@ registering custom adapter functions. .. _sqlite3-conform: -Letting your object adapt itself -"""""""""""""""""""""""""""""""" +How to write adaptable objects +"""""""""""""""""""""""""""""" Suppose we have a :class:`!Point` class that represents a pair of coordinates, ``x`` and ``y``, in a Cartesian coordinate system. @@ -1634,8 +1669,8 @@ The object passed to *protocol* will be of type :class:`PrepareProtocol`. .. literalinclude:: ../includes/sqlite3/adapter_point_1.py -Registering an adapter callable -""""""""""""""""""""""""""""""" +How to register adapter callables +""""""""""""""""""""""""""""""""" The other possibility is to create a function that converts the Python object to an SQLite-compatible type. @@ -1646,8 +1681,8 @@ This function can then be registered using :func:`register_adapter`. .. _sqlite3-converters: -Converting SQLite values to custom Python types -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to convert SQLite values to custom Python types +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Writing an adapter lets you convert *from* custom Python types *to* SQLite values. @@ -1686,41 +1721,6 @@ The following example illustrates the implicit and explicit approaches: .. literalinclude:: ../includes/sqlite3/converter_point.py -.. _sqlite3-default-converters: - -Default adapters and converters (deprecated) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. note:: - - The default adapters and converters are deprecated as of Python 3.12. - Instead, use the :ref:`sqlite3-adapter-converter-recipes` - and tailor them to your needs. - -The deprecated default adapters and converters consist of: - -* An adapter for :class:`datetime.date` objects to :class:`strings <str>` in - `ISO 8601`_ format. -* An adapter for :class:`datetime.datetime` objects to strings in - ISO 8601 format. -* A converter for :ref:`declared <sqlite3-converters>` "date" types to - :class:`datetime.date` objects. -* A converter for declared "timestamp" types to - :class:`datetime.datetime` objects. - Fractional parts will be truncated to 6 digits (microsecond precision). - -.. note:: - - The default "timestamp" converter ignores UTC offsets in the database and - always returns a naive :class:`datetime.datetime` object. To preserve UTC - offsets in timestamps, either leave converters disabled, or register an - offset-aware converter with :func:`register_converter`. - -.. deprecated:: 3.12 - -.. _ISO 8601: https://en.wikipedia.org/wiki/ISO_8601 - - .. _sqlite3-adapter-converter-recipes: Adapter and converter recipes @@ -1768,8 +1768,8 @@ This section shows recipes for common adapters and converters. .. _sqlite3-connection-shortcuts: -Using connection shortcut methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +How to use connection shortcut methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Using the :meth:`~Connection.execute`, :meth:`~Connection.executemany`, and :meth:`~Connection.executescript` @@ -1785,7 +1785,7 @@ directly using only a single call on the :class:`Connection` object. .. _sqlite3-connection-context-manager: -Using the connection as a context manager +How to use the connection context manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A :class:`Connection` object can be used as a context manager that @@ -1810,8 +1810,8 @@ the context manager is a no-op. .. _sqlite3-uri-tricks: -Working with SQLite URIs -^^^^^^^^^^^^^^^^^^^^^^^^ +How to work with SQLite URIs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some useful URI tricks include:

1 0

[3.10] GH-96071: add regression test for GH-96071 (GH-96137) (#96205)
by pablogsal Aug. 23, 2022

Aug. 23, 2022

https://github.com/python/cpython/commit/e3c4a5b8ede2f239c8a3de5e0f0a755b67… commit: e3c4a5b8ede2f239c8a3de5e0f0a755b67f6c324 branch: 3.10 author: Kumar Aditya <59607654+kumaraditya303(a)users.noreply.github.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2022-08-23T23:28:54+01:00 summary: [3.10] GH-96071: add regression test for GH-96071 (GH-96137) (#96205) Co-authored-by: Kumar Aditya <59607654+kumaraditya303(a)users.noreply.github.com> files: M Lib/test/test_capi.py diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 7b208c90063..0adb689beb8 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -825,6 +825,20 @@ def callback(): t.start() t.join() + @threading_helper.reap_threads + def test_gilstate_ensure_no_deadlock(self): + # See https://github.com/python/cpython/issues/96071 + code = textwrap.dedent(f""" + import _testcapi + + def callback(): + print('callback called') + + _testcapi._test_thread_state(callback) + """) + ret = assert_python_ok('-X', 'tracemalloc', '-c', code) + self.assertIn(b'callback called', ret.out) + class Test_testcapi(unittest.TestCase): locals().update((name, getattr(_testcapi, name))

1 0