Nick Coghlan, 31.08.2013 18:49:
On 25 Aug 2013 21:56, "Stefan Behnel" wrote:
One key point to note is that it *doesn't* call _PyImport_FixupExtensionObject, which is the API that handles all the PEP 3121 per-module state stuff. Instead, the idea will be for modules that don't need additional C level state to just implement PyImportExec_NAME, while those that *do* need C level state implement PyImportCreate_NAME and return a custom object (which may or may not be a module subtype).
Is it really a common case for an extension module not to need any C level state at all? I mean, this might work for very simple accelerator modules with only a few stand-alone functions. But anything non-trivial will almost certainly have some kind of global state, cache, external library, etc., and that state is best stored at the C level for safety reasons.
In my experience, most extension authors aren't writing high performance C accelerators, they're exposing an existing C API to Python. It's the cffi use case rather than the Cython use case.
Interesting. I can't really remember a case where I could afford the runtime overhead of implementing a wrapper in Python and going through something like ctypes or cffi. I mean, testing C libraries with Python tools would be one, but then, you wouldn't want to write an extension module for that and instead want to call it directly from the test code as directly as possible. I'm certainly aware that that use case exists, though, and also the case of just wanting to get things done as quickly and easily as possible.
Mutable module global state is always a recipe for obscure bugs, and not something I will ever let through code review without a really good rationale. Hidden process global state is never good, just sometimes a necessary evil.
I'm not necessarily talking about mutable state. Rather about things like pre-initialised data or imported functionality. For example, I often have a bound method of a compiled regex lying around somewhere in my Python modules as a utility function. And the same kind of stuff exists in C code, some may be local to a class, but other things can well be module global. And given that we are talking about module internals here I'd always keep them at the C level rather than exposing them through the module dict. The module dict involves a much higher access overhead, in addition to the reduced safety due to user accessibility. Exported C-APIs are also a use case. You'd import the C-API of another module at init time and from that point on only go through function pointers etc. Those are (sub-)interpreter specific, i.e. they are module global state that is specific to the currently loaded module instances.
However, keep in mind my patch is currently just the part I can implement without PEP 451 module spec objects.
Understood.
Note that even global functions usually hold state, be it in the form of globally imported modules, global caches, constants, ...
If they can be shared safely across multiple instances of the module (e.g. immutable constants), then these can be shared at the C level. Otherwise, a custom Python type will be needed to make them instance specific.
I assume you meant a custom module (extension) type here. Just to be clear, the "module state at the C-level" is meant to be stored in the object struct fields of the extension type that implements the module, at least for modules that want to support reloading and sub-interpreters. Obviously, nothing should be stored in static (global) variables etc.
We also need the create/exec split to properly support reloading. Reload *must* reinitialize the object already in sys.modules instead of inserting a different object or it completely misses the point of reloading modules over deleting and reimporting them (i.e. implicitly affecting the references from other modules that imported the original object).
Interesting. I never thought of it that way.
I'm not sure this can be done in general. What if the module has threads running that access the global state? In that case, reinitialising the module object itself would almost certainly lead to a crash.
My current proposal on import-sig is to make the first hook "prepare_module", and pass in the existing object in the reload case. For the extension loader, this would be reflected in the signature of the C level hook as well, so the module could decide for itself if it supported reloading.
I really don't like the idea of reloading by replacing module state. It would be much simpler if the module itself would be replaced, then the original module could stay alive and could still be used by those who hold a reference to it or parts of its contents. Especially the from-import case would benefit from this. Obviously, you could still run into obscure bugs where a function you call rejects the input because it expects an older version of a type, for example. But I can't see that being worse (or even just different) from the reload-by-refilling-dict case. You seemed to be ok with my idea of making the loader return a wrapped extension module instead of the module itself. We should actually try that.
This is actually my primary motivation for trying to improve the "can this be reloaded or not?" aspects of the loader API in PEP 451.
I assume you mean that the extension module would be able to clearly signal that it can't be reloaded, right? I agree that that's helpful. If you're wrapping a C library, then the way that library is implemented might simply force you to prevent any attempts at reloading the wrapper module. But if reloading is possible at all, it would be even more helpful if we could make it really easy to properly support it.
(keep in mind existing extension modules using the existing API will still never be reloaded)
Sure, that's the cool thing. We can really design this totally from scratch without looking back.
Take a look at the current example - everything gets stored in the module dict for the simple case with no C level global state.
Well, you're storing types there. And those types are your module API. I understand that it's just an example, but I don't think it matches a common case. As far as I can see, the types are not even interacting with each other, let alone doing any C-level access of each other. We should try to focus on the normal case that needs C-level state and C-level field access of extension types. Once that's solved, we can still think about how to make the really simple cases simpler, if it turns out that they are not simple enough.
Our experience is very different - my perspective is that the normal case either eschews C level global state in the extension module, because it causes so many problems, or else just completely ignores subinterpreter support and proper module cleanup.
As soon as you have more than one extension type in your module, and they interact with each other, they will almost certainly have to do type checks against each other to make sure users haven't passed them rubbish before they access any C struct fields of the object. Doing a type check means that at least one type has a pointer to the other, meaning that it holds global module state. I really think that having some kind of global module state is the exceedingly common case for an extension module.
I didn't know about PyType_FromSpec(), BTW. It looks like a nice addition for manually written code (although useless for Cython).
This is the only way to create custom types when using the stable ABI. Can I take your observation to mean that Cython doesn't currently offer the option of limiting itself to the stable ABI?
Correct. I've taken a bird's view at it back then, and keep stumbling over "wow - I couldn't even use that?" kind of declarations in the header files. I don't think it makes sense for Cython. Existing CPython versions are easy to support because they don't change anymore, and new major releases most likely need adaptations anyway, if only to adapt to new features and performance changes. Cython actually knows quite a lot about the inner workings of CPython and its various releases. Going only through the stable ABI parts of the C-API would make the code horribly slow in comparison, so there are huge drawbacks for the benefit it might give. The Cython way of doing it is more like: you want your code to run on a new CPython version, then use a recent Cython release to compile it. It may still work with older ones, but what you actually want is the newest anyway, and you also want to compile the C code for the specific CPython version at hand to get the most out of it. It's the C code that adapts, not the runtime code (or Cython itself). We run continuous integration tests with all of CPython's development branches since 2.4, so we usually support new CPython releases long before they are out. And new releases of CPython rarely affect Cython user code. Stefan