Issue #23903 - stable API is incomplete
For those who aren't aware, the stable API (PEP 384) is broken on Windows because the exports from python3.dll have not been kept up to date. Over at http://bugs.python.org/issue23903 we're trying to address this by automatically generating the DLL based on the headers. This has shown that many more functions and data items are in the stable ABI than expected. If it's left entirely to me, I'm planning to add all the public APIs into python3.dll, which will commit them to the stable API for good, and remove the _private APIs that have been added since we last updated the DLL. However, if you've added an API recently that you didn't mean to be in the stable API, here is your chance to remove it. The full list of APIs that have never been available on Windows in the stable ABI but are in the headers are below. If it should not be considered long-term stable, then it needs "#ifndef Py_LIMITED_API" around it. Note that anything NOT on this list has already been released, and so it cannot be removed from the stable API at this time. I want to resolve this for 3.5.3 (that is, release all of these as stable and then it cannot be undone), which is coming up very soon, so if any of the public APIs should NOT be stable, please fix them, and if any of the private APIs SHOULD be stable, they'll probably need version-specific guards (see moduleobject.h). Cheers, Steve Full list of APIs to be added to python3.dll: PyAST_FromNode PyAST_FromNodeObject PyAST_Validate PyCmpWrapper_Type PyCodec_NameReplaceErrors PyErr_GetExcInfo PyErr_ResourceWarning PyErr_SetExcFromWindowsErr PyErr_SetExcFromWindowsErrWithFilename PyErr_SetExcFromWindowsErrWithFilenameObject PyErr_SetExcFromWindowsErrWithFilenameObjects PyErr_SetExcInfo PyErr_SetExcWithArgsKwargs PyErr_SetFromErrnoWithFilenameObjects PyErr_SetFromWindowsErr PyErr_SetFromWindowsErrWithFilename PyErr_SetImportError PyErr_SetImportErrorSubclass PyErr_SyntaxLocationEx PyExc_BlockingIOError PyExc_BrokenPipeError PyExc_ChildProcessError PyExc_ConnectionAbortedError PyExc_ConnectionError PyExc_ConnectionRefusedError PyExc_ConnectionResetError PyExc_FileExistsError PyExc_FileNotFoundError PyExc_InterruptedError PyExc_IsADirectoryError PyExc_ModuleNotFoundError PyExc_NotADirectoryError PyExc_PermissionError PyExc_ProcessLookupError PyExc_RecursionError PyExc_ResourceWarning PyExc_StopAsyncIteration PyExc_TimeoutError PyExc_WindowsError PyImport_AddModuleObject PyImport_ExecCodeModuleObject PyImport_ImportFrozenModuleObject PyImport_ImportModuleLevelObject PyMarshal_ReadObjectFromString PyMarshal_WriteLongToFile PyMarshal_WriteObjectToFile PyMarshal_WriteObjectToString PyMem_Calloc PyMember_GetOne PyMember_SetOne PyMemoryView_FromMemory PyModuleDef_Init PyModuleDef_Type PyModule_AddFunctions PyModule_ExecDef PyModule_FromDefAndSpec2 PyModule_GetNameObject PyModule_NewObject PyModule_SetDocString PyNode_AddChild PyNode_Free PyNode_ListTree PyNode_New PyNumber_InPlaceMatrixMultiply PyNumber_MatrixMultiply PyOS_CheckStack PyOS_FSPath PyObject_Calloc PyObject_GenericSetDict PyParser_SimpleParseStringFlagsFilename PySys_AddXOption PySys_GetXOptions PyThread_GetInfo PyThread_ReInitTLS PyThread_acquire_lock PyThread_acquire_lock_timed PyThread_allocate_lock PyThread_create_key PyThread_delete_key PyThread_delete_key_value PyThread_exit_thread PyThread_free_lock PyThread_get_key_value PyThread_get_stacksize PyThread_get_thread_ident PyThread_init_thread PyThread_release_lock PyThread_set_key_value PyThread_set_stacksize PyThread_start_new_thread PyUnicode_AsMBCSString PyUnicode_AsUCS4 PyUnicode_AsUCS4Copy PyUnicode_AsWideCharString PyUnicode_DecodeCodePageStateful PyUnicode_DecodeLocale PyUnicode_DecodeLocaleAndSize PyUnicode_DecodeMBCS PyUnicode_DecodeMBCSStateful PyUnicode_EncodeCodePage PyUnicode_EncodeLocale PyUnicode_FindChar PyUnicode_GetLength PyUnicode_ReadChar PyUnicode_Substring PyUnicode_WriteChar Py_DecodeLocale Py_EncodeLocale Py_FileSystemDefaultEncodeErrors Py_SetPath Py_hexdigits _PyBytes_DecodeEscape _PyDebug_PrintTotalRefs _PyThreadState_Current _PyTrash_thread_deposit_object _PyTrash_thread_destroy_chain _PyUnicode_DecodeUnicodeEscape _Py_AddToAllObjects _Py_ForgetReference _Py_GetRefTotal _Py_HashSecret_Initialized _Py_NegativeRefcount _Py_NewReference _Py_PrintReferenceAddresses _Py_PrintReferences _Py_RefTotal
I should also point out that when 3.6.0 releases, all of these will already be in the stable API for other platforms. I'm not going to take a stance on whether we can break it there between 3.6.0 and 3.6.1, but it may already be too late to remove any.
Top-posted from my Windows Phone
-----Original Message-----
From: "Steve Dower"
2016-12-21 2:52 GMT+01:00 Steve Dower
_PyBytes_DecodeEscape _PyDebug_PrintTotalRefs _PyThreadState_Current _PyTrash_thread_deposit_object _PyTrash_thread_destroy_chain _PyUnicode_DecodeUnicodeEscape _Py_AddToAllObjects _Py_ForgetReference _Py_GetRefTotal _Py_HashSecret_Initialized _Py_NegativeRefcount _Py_NewReference _Py_PrintReferenceAddresses _Py_PrintReferences _Py_RefTotal
These functions are private. Would it be possible to not export them? Victor
On 21.12.16 11:50, Victor Stinner wrote:
2016-12-21 2:52 GMT+01:00 Steve Dower
: _PyBytes_DecodeEscape _PyDebug_PrintTotalRefs _PyThreadState_Current _PyTrash_thread_deposit_object _PyTrash_thread_destroy_chain _PyUnicode_DecodeUnicodeEscape _Py_AddToAllObjects _Py_ForgetReference _Py_GetRefTotal _Py_HashSecret_Initialized _Py_NegativeRefcount _Py_NewReference _Py_PrintReferenceAddresses _Py_PrintReferences _Py_RefTotal
These functions are private. Would it be possible to not export them?
Private functions used in public macros (like _Py_NewReference) should be exported.
2016-12-21 14:06 GMT+01:00 Serhiy Storchaka
These functions are private. Would it be possible to not export them?
Private functions used in public macros (like _Py_NewReference) should be exported.
Ah, _Py_NewReference is used in the PyObject_INIT(op, typeobj) *macro*, right. IMO it's an issue with our public API: for the stable ABI, we should replace such macro with a function which hides implementation details. Example from pystate.h: #ifdef Py_BUILD_CORE PyAPI_DATA(_Py_atomic_address) _PyThreadState_Current; # define PyThreadState_GET() \ ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current)) #else # define PyThreadState_GET() PyThreadState_Get() #endif Ok, now why should _Py_PrintReferences() function be exported? This private function is only called from Py_FinalizeEx(). It is not used in a macro. Victor
"Ok, now why should _Py_PrintReferences() function be exported?"
It probably shouldn't, but it needs an #ifndef Py_LIMITED_API check so it is excluded from the headers (my list was automatically generated).
And ideally, private functions that are deliberately exported would have comments, or if they're new, a version check on Py_LIMITED_API.
Top-posted from my Windows Phone
-----Original Message-----
From: "Victor Stinner"
These functions are private. Would it be possible to not export them?
Private functions used in public macros (like _Py_NewReference) should be exported.
Ah, _Py_NewReference is used in the PyObject_INIT(op, typeobj) *macro*, right. IMO it's an issue with our public API: for the stable ABI, we should replace such macro with a function which hides implementation details. Example from pystate.h: #ifdef Py_BUILD_CORE PyAPI_DATA(_Py_atomic_address) _PyThreadState_Current; # define PyThreadState_GET() \ ((PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current)) #else # define PyThreadState_GET() PyThreadState_Get() #endif Ok, now why should _Py_PrintReferences() function be exported? This private function is only called from Py_FinalizeEx(). It is not used in a macro. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
On 21.12.16 17:41, Steve Dower wrote:
"Ok, now why should _Py_PrintReferences() function be exported?"
It probably shouldn't, but it needs an #ifndef Py_LIMITED_API check so it is excluded from the headers (my list was automatically generated).
And ideally, private functions that are deliberately exported would have comments, or if they're new, a version check on Py_LIMITED_API.
Seconded.
On Dec 21, 2016 7:43 AM, "Steve Dower"
2016-12-21 17:21 GMT+01:00 Nathaniel Smith
It sounds like the opt-out approach isn't working very well, and maybe an opt-in approach should be considered instead? I recognize that the way C headers work makes this difficult, but it seems like something needs to change.
I proposed something different: "Python 3.7: remove all private C functions from the Python C API?" https://mail.python.org/pipermail/python-dev/2016-September/146386.html Create subdirectories in Include/ to define private functions in different files. Victor
There's a difference between "private", "stable for 3.x" and "stable for all 3" though. It's the third category that's getting too many functions added without due consideration.
Top-posted from my Windows Phone
-----Original Message-----
From: "Victor Stinner"
It sounds like the opt-out approach isn't working very well, and maybe an opt-in approach should be considered instead? I recognize that the way C headers work makes this difficult, but it seems like something needs to change.
I proposed something different: "Python 3.7: remove all private C functions from the Python C API?" https://mail.python.org/pipermail/python-dev/2016-September/146386.html Create subdirectories in Include/ to define private functions in different files. Victor
"maybe the test suite should error out if any unexpected symbols appear in the stable ABI?"
This or on build (personally I prefer this sort of validation at build time, but I know others would prefer to defer it).
We have a script now that can extract all the right functions, though I think it'll only work in the source tree as it relies on clinic. But that should make it fairly straightforward to spit out a list and compare it to a checked in list.
At the same time, we have a problem in the current release, which is the functions I listed earlier. I would really like to fix that without blocking on getting the right long-term fix (since the immediate fix only affects one file in the Windows distribution, though it has implications for supportability).
Top-posted from my Windows Phone
-----Original Message-----
From: "Nathaniel Smith"
participants (4)
-
Nathaniel Smith
-
Serhiy Storchaka
-
Steve Dower
-
Victor Stinner