Hello. I'm trying to embed a Py3.8 interpreter into a C++ application to
use as the "glue" for all the other components. We've come across some
complexities -- revolving around the use of frozen modules and multiple
interpreter states -- and I'm wondering if this is the correct list to seek
some assistance?
We can get a single interpreter running. We can add our stream objects and
capture the output, we can load our modules (giving the API access) into it
and generally everything seems fairly happy in single interp mode.
When we try and run a Py_NewInterpreter() to get us more interps to get
encapsulation between multiple scripts, it fails to launch (fails to get
filesystem's codec). The problem seems to be that the frozen modules added
to the launched interpreter are not cached after use. So the cloned
interpreter fails to start (it wants either an encoder search path which is
null for no filesystem OR to have the encodings module available). The
modules don't seem to be being cached because when the import runs them
with PyEval_EvalCode, they return nothing.
I suspect this is something I've missed about how to configure the original
interpreter, but documentation & use-examples for the new split-phase
initialisation are rather hard to find.
Cheers for any help you can provide,
Katie
Hi,
As one of the capi-sig list owners, I got many emails like:
"xxxx <xxx@xxx>'s subscription has been disabled on
capi-sig(a)python.org due to their bounce score exceeding the mailing
list's bounce_score_threshold."
I guess that many people subscribed long time ago when the list was
still run by Mailman 2 and Mailman 2 didn't have this feature. The
list migrated to Mailman 3 something like 1 year ago. I guess that
Mailman 3 started to disable many subscriptions because it has
stricter default policy on bouncing.
Anyway, if you don't get new emails (like this one), check your subscription at:
https://mail.python.org/mailman3/lists/capi-sig.python.org/
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
Hi,
I pushed a few changes to hide even more implementation details from
the limited C API of CPython. See "Changes in the limited C API"
documentation at:
https://docs.python.org/dev/whatsnew/3.9.html#build-and-c-api-changes
I'm interesting to get your feedback on such changes.
I know that the PyQt project uses the limited C API. Apart from that,
I don't know other C extension using it.
One idea would be to consume it directly in CPython for a bunch of C
extensions. It would help to check if the API is complete enough for a
non-trivial extension. It should also help to detect issues earlier.
Currently, CPython only uses the limited C API the "xxlimited" which
is basically a test module only written to show how this API can be
used.
By the way, I just opened "Make PyObject an opaque structure in the
limited C API" issue:
https://bugs.python.org/issue39573
It's a follow-up on Neil Schemenauer's proof-of-concept of tagged
pointer in CPython that he wrote in September 2018.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
Hi all,
and thank you to the authors of this well thought-out PEP.
https://www.python.org/dev/peps/pep-0573/
I read through the latest version and found it in a very good and
convincing shape. It has been resting for more than a year now, and some
things happened along the way, most notably PEP 590, to which I think it
should be adapted. Here are my comments.
I find it a bit sad that state lookups in slot methods are disadvantaged
compared to normal methods, given that their entire point is to provide a
performance _benefit_. However, for the time being, we do not know how
relevant and visible this will become in practice, so it is good that the
PEP does not propose a new/real/complex solution to it for now. Instead, it
resorts to the MRO lookup (which can always be done), and leaves any larger
change open for the future, when we know better if there really is a
problem to solve. Check from my side, this is the right trade-off.
I like the idea of immutable exception types.
I'm not entirely happy with the word "Prepare" vs. "New" in the function
name "PyErr_PrepareImmutableException", which is described to be similar to
"PyErr_NewExceptionWithDoc". I understand that it does not create a new
type everytime you call it, but then, it also only prepares something the
first time you call it. Can't say if there is a good way to express this
behaviour, but "prepare" seems worse to me than "new" or "create" IMHO, not
better.
Section [1] is the one on extension type methods, which IMHO should be
adapted to PEP 590. I think it should be based on the vectorcall/fastcall
calling conventions and not take tuple and dict but C arrays as arguments.
Alternatively, we could have two new signatures (old-style + new-style),
but that feels like overkill. the fastcall signatures are easy enough to
handle in C code, and in most cases easier than tuple+dict. The signature
of PyCMethod should become
(PyObject *obj, PyTypeObject *defining_class,
PyObject *const *args, size_t nargs,
PyObject *keywords)
The descriptor special casing [2] in PEP 590 is also worth mentioning in
the context of this PEP. As far as I can see it, the C method type should
implement the vectorcall protocol, and then pass the call through to the
underlying C function, adding the "defining_class" parameter along the way.
I've written up a PR [3] to change the PEP in that direction.
The "_PyMethodDef_RawFastCall*" functions may no longer need to be modified
after this, I think.
BTW, if there is "a new type PyCMethodObject", then there is probably also
a PyTypeObject for it, right? That should be mentioned somewhere.
The section on static/immutable exceptions [4] says "On deinitialization of
the exception type, *exc will be set to NULL". What sets them to NULL?
Under "Helpers", it says "PyType_GetModuleState … returns pointer to state
of the module". I assume that this refers to the PEP-3121 module state, but
an explicit reference would make it clearer.
It also says in the same section that "When a type without a module is
passed in, SystemError is set and NULL returned". Is that necessarily an
exception case? Why not simply return NULL and let the user deal with it? I
think the question is: is there a use case for getting the module state of
an arbitrary type? If so, passing a type that does not have a module
reference is fine and should not raise an exception. If that's not a use
case, then it _might_ be worth informing the user about their misuse, or
not. I don't know. What was the intention here?
Finally, the reference link to the patch set of the implementation does not
work for me. Maybe it should be this?
https://github.com/Dormouse759/cpython/compare/master...Dormouse759:pep-c
As you can see, I do not have any objections or major concerns about this
PEP. Next, I'll look through the implementation, and would also like to see
at least an attempt to implement it in Cython, to get an idea about how its
usage would look in the general case. But apart from that, I can see that a
lot of thinking has already gone into this PEP.
Stefan
[1]
https://www.python.org/dev/peps/pep-0573/#passing-the-defining-class-to-ext…
[2] https://www.python.org/dev/peps/pep-0590/#descriptor-behavior
[3] https://github.com/python/peps/pull/1159
[4] https://www.python.org/dev/peps/pep-0573/#static-exceptions
Hello,
Currently it's not necessary to call PyState_AddModule in a module
PyInit_* function. For single-phase init modules, Python calls it
automatically.
However, documentation of PyState_FindModule sugests that you need to
call PyState_AddModule.
I wrote what I think is the expected behavior in [PR 16101], but I'm not
sure that's actually it.
If anyone who knows the original intent is reading this, could you chime
in? :)
[PR 16101]: https://github.com/python/cpython/pull/16101/files
Hello all,
Victor Stinner recently added the function PyObject_CallNoArgs() for
calling an object without any arguments, see
https://docs.python.org/3.9/c-api/object.html#c.PyObject_CallNoArgs
The next obvious question is: should we have PyObject_CallOneArg(),
PyObject_CallTwoArgs()? Of course, this cannot continue forever, so I
suggest adding the 1 and 2 argument variants which would cover most use
cases. Cython has something like that and it becomes very natural to use
these functions.
The main advantage is that we can implement these variants much more
efficiently than the existing PyObject_CallFunction() or
PyObject_CallFunctionObjArgs().
I am trying to use the PEP384 style type-construction with PyType_FromSpec.
I couldn't find much documentation on this, so first: is this the expected
way to make types going forward? I have always used static objects, but
this API is actually very convenient for programatically generated types.
As for my actual question: while nested structs like tp_as_number and
tp_as_mapping have been broken up such that you just pass Py_nb_add etc.,
tp_methods still just wants a null-terminated array of PyMethodDef objects.
This would be fine, but the resulting PyCFunction objects hold pointers
into this array, meaning I must ensure that the array outlives the type. I
could put the array in a capsule and assign it to the type, but then if a
user deletes the attribute from the type, it will free the memory calling
those methods will be a use after free. I could make a metaclass that has
room for the array on the type object itself, but it appears that the type
object is always allocated with PyType_GenericAlloc(&PyType_type, nmembers)
where nmembers is the size of the Py_tp_members slot. Also, even if I could
get the space, this would leak an implementation detail into the class
hierarchy, which is probably fine but seems messy.
One idea I have is to change PyCFunctionObject to hold a PyMethodDef
*value* instead of a pointer. I don't think the extra foot print size will
affect much, and will lower the total memory usage of a PyCFunctionObject
by one pointer. If anything, it avoids a second indirection to find the
function's implementation. I wanted to poll this list before working on
this to see if this has already been discussed or if I am just doing
something wrong. PyCFunctionObject is not part of the limited ABI, so this
should be a stable ABI compatible change. We can make this same change in
PyType_Ready for statically allocated types.
In https://bugs.python.org/issue36710 Victor wants to move away from
the _PyRuntime C global, instead passing the _PyRuntimeState around
explicitly. I'm in favor of the general idea, but not on a
case-by-case basis like this. I left a comment on that issue
(https://bugs.python.org/msg340945) that explains my position in more
detail. In retrospect, I should have just posted here. :) So I've
copied that comment below, as-is.
FYI, my intention is not to refuse Victor's objective in the issue.
Rather, I want to make sure we have consensus on a valid broader
objective on which to focus. This seemed like a perfect opportunity
to start a discussion about it.
-eric
--------------------------
Status Quo
============
For simplicity sake, let's say nearly all the code operates relative
to the 3 levels of runtime state:
* global - _PyRuntimeState
* interpreter - PyInterpreterState
* thread - PyThreadState
Furthermore, there are 3 groups of functions in the C-API:
* context-sensitive - operate relative to the current Python thread
* runtime-dependent - operate relative to some part of the runtime
state, regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state
Most of the C-API is context-sensitive. A small portion is
runtime-dependent. A handful of functions are runtime-independent
(effectively otherwise stateless helper functions that only happen to
be part of the C-API).
Each context-sensitive function relies on there being a "runtime
context" it can use relative to the current OS thread. That context
consists of the current (i.e. active) PyThreadState, the corresponding
PyInterpreterState, and the global _PyRuntimeState. That context is
derived from data in TSS (see caveats below). This group includes
most of the C-API.
Each runtime-dependent function operates against one or more runtime
state target, regardless of the current thread context (or even if
there isn't one). The target state (e.g. PyInterpreterState) is
always passed explicitly. Again, this is only a small portion of the
C-API.
Caveats:
* for context-sensitive functions, we get the global runtime state
from the global C variable (_PyRuntime) rather than via the implicit
thread context
* for some of the runtime-dependent functions that target
_PyRuntimeState, we rely on the global C variable
All of this is the pattern we use currently. Using TSS to identify
the implicit runtime context has certain benefits and costs:
benefits:
* sticking with the status quo means no backward incompatibility for
existing C-extension code
* easier to distinguish the context-sensitive functions from the
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument
costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)
Alternative
=============
For every context-sensitive function we could add a new first
parameter, "context", that provides the runtime context to use. That
would be something like this:
struct {
PyThreadState *tstate;
...
} PyRuntimeContext;
The interpreter state and global runtime state would still be
accessible via the same indirection we have now.
Taking this alternative would eliminate the previous costs. Having a
consistent "PyRuntimeContext *context" first parameter would maintain
the easy distinction from runtime-dependent functions. Asking callers
to pass in the context explicitly is probably better regardless. As
to backward compatibility, we could maintain a shim to bridge between
the old way and the new.
About the C-global _PyRuntime
==============================
Currently the global runtime state (_PyRuntimeState) is stored in a
static global C variable, _PyRuntime. I added it at the time I
consolidated many of the existing C globals into a single struct.
Having a C global makes it easy to do the wrong thing, so it may be
good to do something else.
That would mean allocating a _PyRuntimeState on the heap early in
startup and pass that around where needed. I expect that would not
have any meaningful performance penalty. It would probably also
simplify some of the code we currently use to manage _PyRuntime
correctly.
As a bonus, this would be important if we decided that
multiple-runtimes-per-process were a desirable thing. That's a neat
idea, though I don't see a need currently. So on its own it's not
really a justification for dropping a static _PyRuntime. :) However,
I think the other reasons are enough.
Conclusions
====================
This issue has a specific objective that I think is premature. We
have an existing pattern and we should stick with that until we decide
to change to a new pattern. That said, a few things should get
corrected and we should investigate alternative patterns for the
context-sensitive C-API.
As to getting rid of the _PyRuntime global variable in favor of
putting it on the heap, I'm not opposed. However, doing so should
probably be handled in a separate issue.
Here are my thoughts on actionable items:
1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the
parent _PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime state, get
it from the current PyInterpreterState
6. for runtime-dependent C-API that targets the global runtime state,
ensure the _PyRuntimeState is always an explicit parameter
7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap
during startup to pass around
Hi,
I'm working on my PEP 587 "Python Initialization Configuration":
https://www.python.org/dev/peps/pep-0587/
I just proposed to add a new "configure_locale" configuration
parameter to leave the LC_CTYPE locale unchanged:
https://bugs.python.org/issue36945
Right now, Python always calls setlocale(LC_CTYPE, "") on all
platforms to set the LC_CTYPE locale to the user preferred locale.
Is it a problem when Python is embedded in an application? Or am I
trying to fix an artificial issue? :-)
By the way, Nick Coghlan and me decided to disable LC_CTYPE locale
coercion (PEP 538) and UTF-8 Mode (PEP 540) by default in Python 3.8.
It should reduce the risk of mojibake.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
