[Python-Dev] Pre-PEP: Redesigning extension modules

[Stefan Behnel]

Nick Coghlan, 01.09.2013 03:28:
> On 1 Sep 2013 05:18, "Stefan Behnel" wrote:
>> I can't really remember a case where I could afford the
>> runtime overhead of implementing a wrapper in Python and going through
>> something like ctypes or cffi. I mean, testing C libraries with Python
>> tools would be one, but then, you wouldn't want to write an extension
>> module for that and instead want to call it directly from the test code as
>> directly as possible.
>>
>> I'm certainly aware that that use case exists, though, and also the case
>> of just wanting to get things done as quickly and easily as possible.
> 
> Keep in mind I first came to Python as a tool for test automation of custom
> C++ hardware APIs that could be written to be SWIG friendly.

Interesting again. Would you still do it that way? I recently had a
discussion with Holger Krekel of py.test fame about testing C code with
Cython, and we quickly agreed that wrapping the code in an extension module
was both too cumbersome and too inflexible for testing purposes.
Specifically, neither of Cython's top selling points fits here, not speed,
not clarity, not API design. It's most likely different for SWIG, which
involves less (not no, just less) manual work and gives you API-wise more
of less exactly what you put in. However, cffi is almost certainly the
better way to do it, because it gives you all sorts of flexibility for your
test code without having to think about the wrapper design all the time.

The situation is also different for C++ where you have less options for
wrapping it. I can imagine SWIG still being the tool of choice on that
front when it comes to bare and direct testing of large code bases.


> I now work for an OS vendor where the 3 common languages for system
> utilities are C, C++ and Python.
> 
> For those use cases, dropping a bunch of standard Python objects in a
> module dict is often going to be a quick and easy solution that avoids a
> lot of nasty pointer lifecycle issues at the C level.

That's yet another use case, BTW. When you control the whole application,
then safety doesn't really matter at these points and keeping a bunch of
stuff in a dict will usually work just fine. I'm mainly used to writing
libraries for (sometimes tons of) other people, in which case the
requirements are so diverse on user side that safety is a top thing to care
about. Anything you can keep inside of C code should stay there.
(Especially when dealing with libxml2&friends in lxml which continuously
present their 'interesting' usability characteristics.)


> * PEP 3121 with a size of "0". As above, but avoids the module state APIs
> in order to support reloading. All module state (including type
> cross-references) is stored in hidden state (e.g. an instance of a custom
> type not exposed to Python, with a reference stored on each custom type
> object defined in the module, and any module level "functions" actually
> being methods of a hidden object).

Thanks for elaborating. I had completely failed to make the mental link
that you could simply stick bound methods as functions into the module
dict, i.e. that they don't even have to be methods of the module itself.
That's something that Cython could already use in older CPythons, even as a
preparation for any future import protocol changes. The object that they
are methods of would then eventually become the module instance.

You'd still suffer a slight performance hit from going from a static global
C variable to a pointer indirection - for everything: string constants,
cached Python objects, all user defined global C variables would have to go
there as Cython cannot know if they are module instance specific state or
not (they usually will be, I guess). But that has to be done anyway if the
goal is to get rid of static state to enable sub-interpreters. I can't wait
seeing lxml run threaded in mod_wsgi... ;-)


>> You seemed to be ok with my idea of making the loader return a wrapped
>> extension module instead of the module itself. We should actually try
>> that.
> 
> Sure, that's just a variant of the "hidden state object" idea I described
> above. It should actually work today with the PEP 3121 custom storage size
> set to zero.

True. The only difference is whether you leave it to the extension type
itself or make it a part of the loader architecture.

Anyway, I promise I'll give it a try in Cython. Will be some work, though,
to rewrite Cython's use of global variables, create a module state type,
migrate everything to heap types, ... I had wanted to do that for a couple
of years, but it's clearly not something for a happy afternoon or two.

Plus, it would even have to be optional in the compiler to avoid
performance regressions for modules that want to continue using fast static
globals simply because they cannot support multiple instances anyway (e.g.
due to external C library dependencies). Let's see if we can solve that at
C compilation time by throwing in a couple of macros. That would at least
help keeping the Cython compiler itself simple in that regard... (I guess
it would also help with testing as we could just duplicate the test suite
runs for both design modes)


>> As soon as you have more than one extension type in your module, and they
>> interact with each other, they will almost certainly have to do type
>> checks
>> against each other to make sure users haven't passed them rubbish before
>> they access any C struct fields of the object. Doing a type check means
>> that at least one type has a pointer to the other, meaning that it holds
>> global module state.
> 
> Sure, but you can use the CPython API rather than writing normal C code. We
> do this fairly often in CPython when we're dealing with things stored in
> modules that can be manipulated from Python.
> 
> It incurs CPython's dynamic dispatch overhead, but sometimes that's worth
> it to avoid needing to deal with C level lifecycle issues.

Not so much of a problem in Cython, because all you usually have to do to
get fast C level access to something is to change a "def" into a "cdef"
somewhere, or add a decorator, or an assignment to a known extension type
variable. Once the module global state is 'virtualised', this will also be
a safe thing to do in the face of multiple module instances, and still be
much faster than going through Python calls.


>> I really think that having some kind of global module state is the
>> exceedingly common case for an extension module.
> 
> I wouldn't be willing to make the call about which of stateless vs stateful
> is more common without a lot more research :)
> 
> They're both common enough that I think they should both be well supported,
> and making the "no custom C level state" case as simple as possible.

Agreed.


>>>> I didn't know about PyType_FromSpec(), BTW. It looks like a nice
>>>> addition for manually written code (although useless for Cython).
>>>
>>> This is the only way to create custom types when using the stable ABI.

I actually think I recall reading about it in the PEP back when it was
designed, decided that it made sense in the given context, and then forgot
about it as I didn't consider it relevant.


> The main advantage of the stable ABI is being able to publish cross-version
> binary extension modules. I guess if Cython already supports generating
> binaries for each new version of CPython before we release it, that
> capability is indeed less useful than it is for those that are maintaining
> extension modules by hand.

I consider it mostly interesting for Linux distributions and closed source
module vendors as it reduces the build/support overhead. But compiling the
generated C code for the specific CPython version at hand really has some
major advantages.

Stefan