Hello all,
Victor Stinner recently added the function PyObject_CallNoArgs() for
calling an object without any arguments, see
https://docs.python.org/3.9/c-api/object.html#c.PyObject_CallNoArgs
The next obvious question is: should we have PyObject_CallOneArg(),
PyObject_CallTwoArgs()? Of course, this cannot continue forever, so I
suggest adding the 1 and 2 argument variants which would cover most use
cases. Cython has something like that and it becomes very natural to use
these functions.
The main advantage is that we can implement these variants much more
efficiently than the existing PyObject_CallFunction() or
PyObject_CallFunctionObjArgs().
I am trying to use the PEP384 style type-construction with PyType_FromSpec.
I couldn't find much documentation on this, so first: is this the expected
way to make types going forward? I have always used static objects, but
this API is actually very convenient for programatically generated types.
As for my actual question: while nested structs like tp_as_number and
tp_as_mapping have been broken up such that you just pass Py_nb_add etc.,
tp_methods still just wants a null-terminated array of PyMethodDef objects.
This would be fine, but the resulting PyCFunction objects hold pointers
into this array, meaning I must ensure that the array outlives the type. I
could put the array in a capsule and assign it to the type, but then if a
user deletes the attribute from the type, it will free the memory calling
those methods will be a use after free. I could make a metaclass that has
room for the array on the type object itself, but it appears that the type
object is always allocated with PyType_GenericAlloc(&PyType_type, nmembers)
where nmembers is the size of the Py_tp_members slot. Also, even if I could
get the space, this would leak an implementation detail into the class
hierarchy, which is probably fine but seems messy.
One idea I have is to change PyCFunctionObject to hold a PyMethodDef
*value* instead of a pointer. I don't think the extra foot print size will
affect much, and will lower the total memory usage of a PyCFunctionObject
by one pointer. If anything, it avoids a second indirection to find the
function's implementation. I wanted to poll this list before working on
this to see if this has already been discussed or if I am just doing
something wrong. PyCFunctionObject is not part of the limited ABI, so this
should be a stable ABI compatible change. We can make this same change in
PyType_Ready for statically allocated types.
In https://bugs.python.org/issue36710 Victor wants to move away from
the _PyRuntime C global, instead passing the _PyRuntimeState around
explicitly. I'm in favor of the general idea, but not on a
case-by-case basis like this. I left a comment on that issue
(https://bugs.python.org/msg340945) that explains my position in more
detail. In retrospect, I should have just posted here. :) So I've
copied that comment below, as-is.
FYI, my intention is not to refuse Victor's objective in the issue.
Rather, I want to make sure we have consensus on a valid broader
objective on which to focus. This seemed like a perfect opportunity
to start a discussion about it.
-eric
--------------------------
Status Quo
============
For simplicity sake, let's say nearly all the code operates relative
to the 3 levels of runtime state:
* global - _PyRuntimeState
* interpreter - PyInterpreterState
* thread - PyThreadState
Furthermore, there are 3 groups of functions in the C-API:
* context-sensitive - operate relative to the current Python thread
* runtime-dependent - operate relative to some part of the runtime
state, regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state
Most of the C-API is context-sensitive. A small portion is
runtime-dependent. A handful of functions are runtime-independent
(effectively otherwise stateless helper functions that only happen to
be part of the C-API).
Each context-sensitive function relies on there being a "runtime
context" it can use relative to the current OS thread. That context
consists of the current (i.e. active) PyThreadState, the corresponding
PyInterpreterState, and the global _PyRuntimeState. That context is
derived from data in TSS (see caveats below). This group includes
most of the C-API.
Each runtime-dependent function operates against one or more runtime
state target, regardless of the current thread context (or even if
there isn't one). The target state (e.g. PyInterpreterState) is
always passed explicitly. Again, this is only a small portion of the
C-API.
Caveats:
* for context-sensitive functions, we get the global runtime state
from the global C variable (_PyRuntime) rather than via the implicit
thread context
* for some of the runtime-dependent functions that target
_PyRuntimeState, we rely on the global C variable
All of this is the pattern we use currently. Using TSS to identify
the implicit runtime context has certain benefits and costs:
benefits:
* sticking with the status quo means no backward incompatibility for
existing C-extension code
* easier to distinguish the context-sensitive functions from the
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument
costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)
Alternative
=============
For every context-sensitive function we could add a new first
parameter, "context", that provides the runtime context to use. That
would be something like this:
struct {
PyThreadState *tstate;
...
} PyRuntimeContext;
The interpreter state and global runtime state would still be
accessible via the same indirection we have now.
Taking this alternative would eliminate the previous costs. Having a
consistent "PyRuntimeContext *context" first parameter would maintain
the easy distinction from runtime-dependent functions. Asking callers
to pass in the context explicitly is probably better regardless. As
to backward compatibility, we could maintain a shim to bridge between
the old way and the new.
About the C-global _PyRuntime
==============================
Currently the global runtime state (_PyRuntimeState) is stored in a
static global C variable, _PyRuntime. I added it at the time I
consolidated many of the existing C globals into a single struct.
Having a C global makes it easy to do the wrong thing, so it may be
good to do something else.
That would mean allocating a _PyRuntimeState on the heap early in
startup and pass that around where needed. I expect that would not
have any meaningful performance penalty. It would probably also
simplify some of the code we currently use to manage _PyRuntime
correctly.
As a bonus, this would be important if we decided that
multiple-runtimes-per-process were a desirable thing. That's a neat
idea, though I don't see a need currently. So on its own it's not
really a justification for dropping a static _PyRuntime. :) However,
I think the other reasons are enough.
Conclusions
====================
This issue has a specific objective that I think is premature. We
have an existing pattern and we should stick with that until we decide
to change to a new pattern. That said, a few things should get
corrected and we should investigate alternative patterns for the
context-sensitive C-API.
As to getting rid of the _PyRuntime global variable in favor of
putting it on the heap, I'm not opposed. However, doing so should
probably be handled in a separate issue.
Here are my thoughts on actionable items:
1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the
parent _PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime state, get
it from the current PyInterpreterState
6. for runtime-dependent C-API that targets the global runtime state,
ensure the _PyRuntimeState is always an explicit parameter
7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap
during startup to pass around
Hi,
I'm working on my PEP 587 "Python Initialization Configuration":
https://www.python.org/dev/peps/pep-0587/
I just proposed to add a new "configure_locale" configuration
parameter to leave the LC_CTYPE locale unchanged:
https://bugs.python.org/issue36945
Right now, Python always calls setlocale(LC_CTYPE, "") on all
platforms to set the LC_CTYPE locale to the user preferred locale.
Is it a problem when Python is embedded in an application? Or am I
trying to fix an artificial issue? :-)
By the way, Nick Coghlan and me decided to disable LC_CTYPE locale
coercion (PEP 538) and UTF-8 Mode (PEP 540) by default in Python 3.8.
It should reduce the risk of mojibake.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
Dear C API lovers,
I have a question about the usual args/kwargs call protocol (as used by
PyObject_Call and tp_call): is the "args" tuple supposed to be of type
exactly tuple or are tuple subclasses allowed? Same question for
"kwargs" and dict.
The documentation is not clear about this.
I do know that calls from Python are done with an exact tuple and exact
dict, but in C this is not so clear. For example, the implementation of
_PyErr_CreateException uses PyTuple_Check(), not PyTuple_CheckExact():
static PyObject*
_PyErr_CreateException(PyObject *exception, PyObject *value)
{
if (value == NULL || value == Py_None) {
return _PyObject_CallNoArg(exception);
}
else if (PyTuple_Check(value)) {
return PyObject_Call(exception, value, NULL);
}
else {
return PyObject_CallFunctionObjArgs(exception, value, NULL);
}
}
Also, the implementation of the call protocol has assertions of both
forms: you see both assert(PyTuple_Check(args)) and
assert(PyTuple_CheckExact(args)) and similar for dict. It seems pretty
random whether or not an exact type check is done.
I'm pretty sure that there are code paths that would result in an
assertion failure because of this.
So basically my question is: what is the right behavior? Given the
implementation of the Python bytecode interpreter, I would be inclined
to say that type checks should be exact.
I'm noticing this because I'm working on an implementation of PEP 590.
Jeroen.
Hi,
Last months, I was busy to fill https://pythoncapi.readthedocs.io/
website with random notes. Many discussions occurred on this list and
python-dev, but I was only able to make the most simple and least
controversal changes in Python upstream. I didn't write a PEP because
CPython had a governance crisis. Since a new Steering Committee has
been elected, it's time to see how concrete PEP can be written.
IMHO we need to split the giant "C API" problem into multiple PEPs. I
propose 4 PEPs:
PEP A: Ecosystem of C extensions in 2019
PEP B: Define good and bad C APIs
PEP C: Plan to enhance the existing C API
PEP D: Completely new C API
There is also an ongoing discussion about embedded Python and Python
initialization API, but I'm scared by this topic so I don't even
propose to write a new PEP which would supersed PEP 432 :-)
https://bugs.python.org/issue22213
== PEP A: Ecosystem of C extensions in 2019 ==
Discuss cffi, Cython, PyQt usage of the stable ABI, CPython C API,
etc. The goal is not to solve any problem, mostly to list existing
options.
It sounds like an unsual PEP, but I think that a PEP is needed since
the same discussions happened multiple times.
This PEP can describe what are the kind of "C extensions" and maybe
suggest which tools are the best depending on the kind. cffi doesn't
cover all cases, the C API isn't always the right answer, etc.
== PEP B: Define good and bad C APIs ==
https://pythoncapi.readthedocs.io/bad_api.html can be used as a
starting point. It should be an informal PEP which evolves as PEP 7
and PEP 8 are evolving.
== PEP C: Plan to enhance the existing C API ==
This one sounds like the most controversial PEP :-) I see different things:
* Plan to deprecate and remove bad APIs
* Plan to help C extensions maintainers to move away from these bad APIs
* Plan to test the stability of the API
* Plan to test the stability of the ABI
The even more controversial idea: provide multiple Python runtimes for
CPython, not only one:
https://pythoncapi.readthedocs.io/runtimes.html
== PEP D: Completely new C API ==
Well, that's the obvious alternative to PEP C.
Armin Rigo's PyHandle idea may be a good start?
https://pythoncapi.readthedocs.io/pyhandle.html
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
It seems to me that all the new proposals, whether just hiding existing
implementation details or moving to some new object id or tagged
pointer system, will require extension module writers to replace
PyObject *
with
PyHandle
And extension writers should also not be accessing struct fields in
objects or type records directly, so we also want to replace eg
type->tp_str
with some kind of macro or inline function.
So as a first step, we should add a typedef for PyHandle to Python.h
(well, most likely within object.h) and macros defined for all or the most
common object / type struct members.
We can do this now without breaking any existing code. (Uh, right?)
Then interested people can experiment with new ideas and APIs and
we all get some idea of how hard a final changeover would be.
My other suggestion is that once the new typedefs and macros are
in Python.h, announce that Python 4.0 won't guarantee source code
compatibility for existing extension code.
--
cheers,
Hugh Fisher
[Adding back the list - I assume dropping it was an accidenty]
On 04Mar2019 1234, Neil Schemenauer wrote:
> On 2019-03-03, Steve Dower wrote:
>> In my opinion, you can dislike many of the Windows-specific
>> "enhancements" around COM (like DCOM, etc., and I do dislike them), but
>> the core concepts are very well proven, including being used from
>> JavaScript and .NET (fully GC languages). Perhaps moreso than JNI?
>
> Thanks Steve. You are correct, we should learn from COM too. Do
> you have any suggested references? I poked around some in the
> CoreCLR repo on github. E.g. this was pretty interesting:
>
> https://github.com/dotnet/coreclr/blob/master/Documentation/botr/exceptio...
>
> Are there documents in the open source "dotnet" that would be
> relevant to the COM design implementation? I think a challenge with
> COM is that it is more comprehensive and therefore more complicated.
> So, for outsiders, it could be more difficult to understand how it
> works.
I'd suggest looking at the design notes here instead:
https://github.com/Microsoft/xlang
This is the cross-platform implementation of the "core" of COM
(basically, the cross-language ABI part without necessarily including
the magic cross-process/machine and proxy/marshalling support that is
considered part of COM on Windows).
There is also a tool in that repo for generating Python C extensions to
project objects defined in xlang/COM into Python. Since most of the new
Windows API (from Win8 onwards) is defined like this, that's their first
examples, but it's not at all tied to Windows.
Cheers,
Steve
Hi,
has anyone considered mixing the "opaque handle" idea with tagged pointers?
I could imagine (on 64 bits) to reserve, say, a 'signed' 24 bits for a
refcount and the rest for an index into an array of object/vtable pointer
structs. All negative refcounts would have special meanings, such as
- this is the immortal None
- this is actually a tagged integer and not an object index
- this is a tagged float with an exact 32bit (or integer) value
- refcount has overflown and object has become immortal :)
Things like these, can't say which are reasonable and/or fast enough. I
could also imagine reserving a few more bits for a builtin type ID to speed
up type checks, especially for int/float/list/tuple/dict/fast-callable,
maybe also things like flat tuples, where the items are directly stored
consecutively in the object array following the tuple index.
Other runtimes could then implement the handles differently, as really
opaque values and with or without tagged pointers, but CPython could use
its own macros to give meaning to the tags. 32bit architectures would
probably require different macros also for CPython, but since the whole
design would need to work with and without tags, that should be doable.
Does this seem like something worth discussing?
Stefan
