Mateusz Loskot, 10.05.2012 14:37:

On 10 May 2012 13:00, Stefan Behnel wrote:

...

Mateusz Loskot, 10.05.2012 13:47:

...

On 10 May 2012 12:28, Stefan Behnel wrote:

...

Mateusz Loskot, 10.05.2012 13:24:

...

I'm writing fairly complex structure of Python extensions directly using Python C API (version 3+ only). I define some type objects with statically, according to the canonical Noddy example presented in the docs. Some type objects have to be defined/composed dynamically in run-time.

Do you really need a type or is a Python class enough? The letter would be much easier to create.

I'm not sure I understand. Aren't terms "class" and "type" names of the same concept, since Python 2.2?

With "type" I meant "extension type", i.e. the one you define using C structs.

So, I probably misused "object type" as a shortcut referring to the pair of struct + PyTypeObject instance:

typedef struct { PyObject_HEAD } noddy_NoddyObject;

static PyTypeObject noddy_NoddyType = { ... }

...

The alternative would be a simple Python class as you would get with

class MyClass(object): pass

This is clear to me. I simply was confused because AFAIK there is no class while using Python C API. As mentioned before, I follow Eli Bendersky's:

" Note that when we create new types using the C API of CPython, there’s no "class" mentioned – we create a new "type", not a new "class". "

I think the emphasis is on "when we create new types". The sentence above sounds like a tautology to me (we obviously create a type and not a class when we create a type).

...

...

What is the noddy_NoddyType in the "Defining New Types" example?

That's an extension type.

Hmm, in the Python manual http://docs.python.org/release/3.2.2/extending/newtypes.html

I'm reading this:

"Moving on, we come to the crunch — the type object.

static PyTypeObject noddy_NoddyType = { ... } "

What makes me interpret it as noddy_NoddyType is the type object.

Simply, I understand that the pair of the struct noddy_NoddyObject and noddy_NoddyType together state the Python extension type,

Am I still confusing the terminology?

I don't care all that much about terminology myself.

...

...

In my system, I'm embedding Python and I add custom Python extension too (let's call it 'emb') So, users of my embedded Python have access to 'emb' module. The 'emb' module defines number of types, some are defined statically, as the noddy_NoddyType, so users can instantiate it

n = emb.Noddy() n.bar() # method defined statically in methods table of noddy_NoddyType

Now, I'd like to add some types which are generated in run-time, way before the 'emb' module is appended to inittab and embedded Python is initialised.

Why would you want to do that before initialising the Python runtime?

The 'emb' must be added to inittab before Python is initialised (that's what the manual says, isn't it.) So, the scheme is this:

/* 1. Dynamically generate emb.GeneratedNoddy */ ... /* Trying to figure out how */

/* 2. Register 'emb' module as built-in */ PyImport_AppendInittab("emb", &PyInit_emb);

/* 3. Initialise Python */ Py_Initialize();

/* 4. we're ready to use */ ...

/* 5. Clean-up */ Py_Finalize();

Does it make sense?

No. You don't need to declare all types before hand. You can add a type (or function, or name, or whatever) to the module dict at any time, just as you can in Python. Whether that's a C implemented type of a normal Python class (or something else) doesn't matter at that point.

The inittab mechanism is just there for convenience when everything *is* statically defined at compile time (or at least module init time).

...

...

And, I'd like to enable users to instantiate them in the same way as Noddy above:

d = emb.GeneratedNoddy()

or allow users to use and access

d = emb.foo() d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition

where:

type(d)

I hope it makes my intentions clear.

Not clear enough. The question is what your dynamically created types should do and provide. Would they need to be implemented in C (not just their methods but the types themselves!) or would a Python implementation suffice, potentially with methods implemented in C?

OK, I think I see where is the gap in my explanation.

0. The generated objects will define wrappers for existing C/C++ API.

1. I have defined contained object which needs extra steps to initialise within C/C++ Namely, I need to initialise members like pointer_to_some_wrapped_api_element, it may be some arbitrary data or PyCapsule, etc.

/* statically typedef struct { PyObject_HEAD /* members initialised PyObject* pointer_to_some_wrapped_api_element; } GeneratedNoddyObject;

2. The corresponding GeneratedNoddyType is generated dynamically, with added set of methods, etc.

3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy.

AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to compose the 'emb' module.

Not at all. It's much easier to use a Python class than to do everything in lengthy C-API declarations.

Alternatively, I can generate Python classes (script in text form), but there is one inconvenience. The only way to expose such classes I know is to compile and import as *separate* module using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the resulting structure would be:

emb emb.Noddy emb._privemb emb._privemb.GeneratedNoddy

Wrong again. You can execute any Python code from your C code. Look for the PyRun_*() functions.

Note that this is even easier in Cython, where you write Python code anyway (instead of C code).

Given that complication of loading classes from textual form through intermediate module, I thought using Python C API to generate extension types is better. (I don't mine dealing with C code verbosity and complexity.)

You should. The simpler the code, the easier it is to maintain it. Why else would you want to embed Python instead of using C for everything?

...

You may get away with using normal Python classes that inherit from an extension type, for example. In that case, I'd also advise you to take a look at Cython, because that makes both the implementation of extension types and the creation of Python classes inheriting from them as easy as Py.

Sounds interesting, I will take a look at it.

However, I'd really like to learn canonical means of generating extension types dynamically using plain Python C API. Long story short, I assume I'm looking for Python C API equivalent of using type() function.

Then call type().

Stefan