Mailman 3 May 2012 - capi-sig

Re: [capi-sig] Creating type object dynamically in run-time
by Campbell Barton May 10, 2012

May 10, 2012

If speed of class creation is important (as it is for us). suggest to call type directly rather then evaluating a string every time and executing it - this is one way. /* always use O not N when calling, N causes refcount errors */ newclass = PyObject_CallFunction((PyObject *)&PyType_Type, (char *)"s(O) {sss()}", idname, py_base_class, "__module__", "bpy.types", "__slots__"); which is like doing this in python... type(idname, (py_base_class, ), … [View More]dict(__module__="bpy.types", __slots__=())) On Thu, May 10, 2012 at 6:17 PM, Mateusz Loskot <mateusz(a)loskot.net> wrote: > On 10 May 2012 14:55, Mateusz Loskot <mateusz(a)loskot.net> wrote: >> On 10 May 2012 14:01, Stefan Behnel <python_capi(a)behnel.de> wrote: >>> >>> You can execute any Python code from your C code. Look for the >>> PyRun_*() functions. >> >> Do you mean something similar to this approach? >> >> /* dynamically generated lengthy class definition */ >> const char* c = "class A(object): pass"; >> >> PyObject* class_a = PyRun_StringFlags(c, ...); >> PyObject_SetAttrString(module, "A", class_a) > > The pseudo-code above is incorrect. > I have come up with the following example of dynamically > generating new Python class, indirectly through script using class keyword. > Such dynamically created class is added to module dictionary: > > /* error checks removed for brevity */ > > static PyModuleDef embmodule = { ... }; > > PyInit_emb(void) > { > PyObject* m = PyModule_Create(&embmodule); > PyObject* d = PyModule_GetDict(m); > > /* Required to allow 'class' use in context of module which is > not yet complete and ready. Otherwise, error is thrown: > ImportError: __build_class__ not found > */ > PyObject* builtins = PyEval_GetBuiltins(); > PyDict_SetItemString(d, "__builtins__", builtins); > > /* Python class is dynamically generated */ > const char* c = "class A(object):\n\tpass"; /* sample class */ > > /* Create object for A class, automatically added to the module > dictionary as noddy.A */ > PyRun_StringFlags(c, Py_file_input, d, d, NULL); > > return m; > } > > This approach works well. > If anyone noticed a problem or there is better way to do the same, > please let me know. > > Best regards, > -- > Mateusz Loskot, http://mateusz.loskot.net > _______________________________________________ > capi-sig mailing list > capi-sig(a)python.org > http://mail.python.org/mailman/listinfo/capi-sig -- - Campbell [View Less]

1 0

Re: [capi-sig] Creating type object dynamically in run-time
by Mateusz Loskot May 10, 2012

May 10, 2012

On 10 May 2012 14:55, Mateusz Loskot <mateusz(a)loskot.net> wrote: > On 10 May 2012 14:01, Stefan Behnel <python_capi(a)behnel.de> wrote: >> >> You can execute any Python code from your C code. Look for the >> PyRun_*() functions. > > Do you mean something similar to this approach? > > /* dynamically generated lengthy class definition */ > const char* c = "class A(object): pass"; > > PyObject* class_a = PyRun_StringFlags(c, ...); > … [View More]

1 0

Re: [capi-sig] Creating type object dynamically in run-time
by Stefan Behnel May 10, 2012

May 10, 2012

Mateusz Loskot, 10.05.2012 14:37: > On 10 May 2012 13:00, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 13:47: >>> On 10 May 2012 12:28, Stefan Behnel wrote: >>>> Mateusz Loskot, 10.05.2012 13:24: >>>>> >>>>> I'm writing fairly complex structure of Python extensions directly using >>>>> Python C API (version 3+ only). I define some type objects with statically, >>>>> according to the canonical Noddy … [View More]example presented in the docs. >>>>> Some type objects have to be defined/composed dynamically in run-time. >>>> >>>> Do you really need a type or is a Python class enough? The letter would be >>>> much easier to create. >>> >>> I'm not sure I understand. >>> Aren't terms "class" and "type" names of the same concept, since Python 2.2? >> >> With "type" I meant "extension type", i.e. the one you define using C structs. > > So, I probably misused "object type" as a shortcut referring > to the pair of struct + PyTypeObject instance: > > typedef struct { > PyObject_HEAD > } noddy_NoddyObject; > > static PyTypeObject noddy_NoddyType = { ... } > >> The alternative would be a simple Python class as you would get with >> >> class MyClass(object): pass > > This is clear to me. I simply was confused because AFAIK there is no > class while using Python C API. As mentioned before, I follow Eli Bendersky's: > > " > Note that when we create new types using the C API of CPython, > there’s no "class" mentioned – we create a new "type", not a new "class". > " I think the emphasis is on "when we create new types". The sentence above sounds like a tautology to me (we obviously create a type and not a class when we create a type). >>> What is the noddy_NoddyType in the "Defining New Types" example? >> >> That's an extension type. > > Hmm, in the Python manual > http://docs.python.org/release/3.2.2/extending/newtypes.html > > I'm reading this: > > "Moving on, we come to the crunch — the type object. > > static PyTypeObject noddy_NoddyType = { > ... > } > " > > What makes me interpret it as noddy_NoddyType is the type object. > > Simply, I understand that the pair of the > struct noddy_NoddyObject and noddy_NoddyType > together state the Python extension type, > > Am I still confusing the terminology? I don't care all that much about terminology myself. >>> In my system, I'm embedding Python and I add custom Python extension >>> too (let's call it 'emb') >>> So, users of my embedded Python have access to 'emb' module. >>> The 'emb' module defines number of types, some are defined statically, >>> as the noddy_NoddyType, so users can instantiate it >>> >>> n = emb.Noddy() >>> n.bar() # method defined statically in methods table of noddy_NoddyType >>> >>> Now, I'd like to add some types which are generated in run-time, way >>> before the 'emb' module is appended to inittab and embedded Python is >>> initialised. >> >> Why would you want to do that before initialising the Python runtime? > > The 'emb' must be added to inittab before Python is initialised (that's what the > manual says, isn't it.) So, the scheme is this: > > /* 1. Dynamically generate emb.GeneratedNoddy */ > ... /* Trying to figure out how */ > > /* 2. Register 'emb' module as built-in */ > PyImport_AppendInittab("emb", &PyInit_emb); > > /* 3. Initialise Python */ > Py_Initialize(); > > /* 4. we're ready to use */ > ... > > /* 5. Clean-up */ > Py_Finalize(); > > > Does it make sense? No. You don't need to declare all types before hand. You can add a type (or function, or name, or whatever) to the module dict at any time, just as you can in Python. Whether that's a C implemented type of a normal Python class (or something else) doesn't matter at that point. The inittab mechanism is just there for convenience when everything *is* statically defined at compile time (or at least module init time). >>> And, I'd like to enable users to instantiate them in the same way as >>> Noddy above: >>> >>> d = emb.GeneratedNoddy() >>> >>> or allow users to use and access >>> >>> d = emb.foo() >>> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition >>> >>> where: >>> >>> type(d) >>> <class 'emb.GeneratedNoddy'> >>> >>> I hope it makes my intentions clear. >> >> Not clear enough. The question is what your dynamically created types >> should do and provide. Would they need to be implemented in C (not just >> their methods but the types themselves!) or would a Python implementation >> suffice, potentially with methods implemented in C? > > OK, I think I see where is the gap in my explanation. > > 0. The generated objects will define wrappers for existing C/C++ API. > > 1. I have defined contained object which needs extra steps to > initialise within C/C++ > Namely, I need to initialise members like pointer_to_some_wrapped_api_element, > it may be some arbitrary data or PyCapsule, etc. > > /* statically > typedef struct { > PyObject_HEAD > /* members initialised > PyObject* pointer_to_some_wrapped_api_element; > } GeneratedNoddyObject; > > 2. The corresponding GeneratedNoddyType is generated dynamically, > with added set of methods, etc. > > 3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy. > > AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to > compose the 'emb' module. Not at all. It's much easier to use a Python class than to do everything in lengthy C-API declarations. > Alternatively, I can generate Python classes (script in text form), > but there is one inconvenience. > The only way to expose such classes I know is to compile and import as > *separate* module > using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the > resulting structure would be: > > emb > emb.Noddy > emb._privemb > emb._privemb.GeneratedNoddy Wrong again. You can execute any Python code from your C code. Look for the PyRun_*() functions. Note that this is even easier in Cython, where you write Python code anyway (instead of C code). > Given that complication of loading classes from textual form through > intermediate module, > I thought using Python C API to generate extension types is better. > (I don't mine dealing with C code verbosity and complexity.) You should. The simpler the code, the easier it is to maintain it. Why else would you want to embed Python instead of using C for everything? >> You may get away with using normal Python classes that inherit from an >> extension type, for example. In that case, I'd also advise you to take a >> look at Cython, because that makes both the implementation of extension >> types and the creation of Python classes inheriting from them as easy as Py. > > Sounds interesting, I will take a look at it. > > However, I'd really like to learn canonical means of generating extension types > dynamically using plain Python C API. > Long story short, I assume I'm looking for Python C API equivalent of > using type() function. Then call type(). Stefan [View Less]

2 1

Re: [capi-sig] Creating type object dynamically in run-time
by Mateusz Loskot May 10, 2012

May 10, 2012

On 10 May 2012 13:37, Mateusz Loskot <mateusz(a)loskot.net> wrote: > On 10 May 2012 13:00, Stefan Behnel <python_capi(a)behnel.de> wrote: >> Mateusz Loskot, 10.05.2012 13:47: >>> >>> Now, I'd like to add some types which are generated in run-time, way >>> before the 'emb' module is appended to inittab and embedded Python is >>> initialised. >> >> Why would you want to do that before initialising the Python runtime? > > … [View More]

1 0

Re: [capi-sig] Creating type object dynamically in run-time
by Stefan Behnel May 10, 2012

May 10, 2012

Mateusz Loskot, 10.05.2012 13:47: > On 10 May 2012 12:28, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 13:24: >>> This is my first post to this list, hello everyone! >>> I hope this post is not off-topic here. >>> >>> I'm writing fairly complex structure of Python extensions directly using >>> Python C API (version 3+ only). I define some type objects with statically, >>> according to the canonical Noddy example presented in … [View More]the docs. >>> Some type objects have to be defined/composed dynamically in run-time. >> >> Do you really need a type or is a Python class enough? The letter would be >> much easier to create. > > I'm not sure I understand. > Aren't terms "class" and "type" names of the same concept, since Python 2.2? With "type" I meant "extension type", i.e. the one you define using C structs. The alternative would be a simple Python class as you would get with class MyClass(object): pass > What is the noddy_NoddyType in the "Defining New Types" example? That's an extension type. > In my system, I'm embedding Python and I add custom Python extension > too (let's call it 'emb') > So, users of my embedded Python have access to 'emb' module. > The 'emb' module defines number of types, some are defined statically, > as the noddy_NoddyType, so users can instantiate it > > n = emb.Noddy() > n.bar() # method defined statically in methods table of noddy_NoddyType > > Now, I'd like to add some types which are generated in run-time, way > before the 'emb' module is appended to inittab and embedded Python is > initialised. Why would you want to do that before initialising the Python runtime? > And, I'd like to enable users to instantiate them in the same way as > Noddy above: > > d = emb.GeneratedNoddy() > > or allow users to use and access > > d = emb.foo() > d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition > > where: > > type(d) > <class 'emb.GeneratedNoddy'> > > I hope it makes my intentions clear. Not clear enough. The question is what your dynamically created types should do and provide. Would they need to be implemented in C (not just their methods but the types themselves!) or would a Python implementation suffice, potentially with methods implemented in C? You may get away with using normal Python classes that inherit from an extension type, for example. In that case, I'd also advise you to take a look at Cython, because that makes both the implementation of extension types and the creation of Python classes inheriting from them as easy as Py. Stefan [View Less]

2 1

Re: [capi-sig] Creating type object dynamically in run-time
by Stefan Behnel May 10, 2012

May 10, 2012

Mateusz Loskot, 10.05.2012 13:24: > This is my first post to this list, hello everyone! > I hope this post is not off-topic here. > > I'm writing fairly complex structure of Python extensions directly using > Python C API (version 3+ only). I define some type objects with statically, > according to the canonical Noddy example presented in the docs. > Some type objects have to be defined/composed dynamically in run-time. Do you really need a type or is a Python class … [View More]

2 1

Creating type object dynamically in run-time
by Mateusz Loskot May 10, 2012

May 10, 2012

Hi, This is my first post to this list, hello everyone! I hope this post is not off-topic here. I'm writing fairly complex structure of Python extensions directly using Python C API (version 3+ only). I define some type objects with statically, according to the canonical Noddy example presented in the docs. Some type objects have to be defined/composed dynamically in run-time. Possible 'brute force' approach is to 1) generate Python script in textual form, 2) compile it using … [View More]Py_CompileStringExFlags and 3) import as a module using PyImport_ExecCodeModule. However, it seems there is a better way to do it, using C API only and without intermediate textual representation imported as a separate module. Now, I've been looking for correct, canonical or recommended way to generate dynamically type object definitions, add methods, add attributes, etc. Here is what I found three approaches discussed on the Web: A) SO: How to dynamically create a derived type in the Python C-API http://stackoverflow.com/questions/8066438/how-to-dynamically-create-a-deri… where PyType_Type.tp_alloc is used to allocate type, then tp_* members are filled and PyType_Ready is called. I've copied the example here for easier discussion: PyTypeObject *BrownNoddyType = (PyTypeObject *)PyType_Type.tp_alloc(&PyType_Type, 0); BrownNoddyType->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; BrownNoddyType->tp_name = "noddy.BrownNoddy"; BrownNoddyType->tp_doc = "BrownNoddy objects"; BrownNoddyType->tp_base = &NoddyType; BrownNoddyType->tp_new = BrownNoddy_new; PyType_Ready(BrownNoddyType); This looks easy, but it is not clear to me a) How to add methods and members, getters and setters? Shall I simply call PyDescr_NewMethod, and other PyDescr_* functions? The manual is quite short here: http://docs.python.org/release/3.2.2/c-api/descriptor.html b) How shall I perform clean-up of BrownNoddyType? Is PyModuleDef .m_free function the right way to do it? Or, I simply set Py_TPFLAGS_HEAPTYPE and the dynamically object types will be reference-counted and cleaned automatically as discussed here http://thread.gmane.org/gmane.comp.python.devel/105648 B) SO: Python and Dynamically Extending C++ Classes http://stackoverflow.com/questions/7759827/python-and-dynamically-extending… This approach is similar to 1), but it uses base type object. New sub-classed type objects are 1) created dynamically using PyObject_New 2) and extended with new methods using PyCFunction_New and PyInstanceMethod_New functions. Here come questions: a) Do I need to perform any clean-up? Where? b) Is it possible/sensible to replace PyCFunction_New and PyInstanceMethod_New with PyDescr_NewMethod? C) Following what's discussed in "How do I add method dynamically to module using C API?" http://www.velocityreviews.com/forums/t727676-how-do-i-add-method-dynamical… (Complete C/C++ code example is presented in the last post at the bottom.) It seems to be similar to the B) approach. This thread is about adding methods to modules and it uses undocumented PyCFunction_NewEx. But, I sense this technique should work for type objects too, once they are created. The big question is, which of these approaches is recommended for Python 3.x, or is there any other/better way to create dynamic type objects not discussed above? I'd appreciate any suggestions, hints or pointers in the Python docs which discuss this (perhaps I've missed it). Best regards, -- Mateusz Loskot, http://mateusz.loskot.net [View Less]

1 0