From mateusz at loskot.net Thu May 10 13:24:01 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 12:24:01 +0100 Subject: [capi-sig] Creating type object dynamically in run-time Message-ID: Hi, This is my first post to this list, hello everyone! I hope this post is not off-topic here. I'm writing fairly complex structure of Python extensions directly using Python C API (version 3+ only). I define some type objects with statically, according to the canonical Noddy example presented in the docs. Some type objects have to be defined/composed dynamically in run-time. Possible 'brute force' approach is to 1) generate Python script in textual form, 2) compile it using Py_CompileStringExFlags and 3) import as a module using PyImport_ExecCodeModule. However, it seems there is a better way to do it, using C API only and without intermediate textual representation imported as a separate module. Now, I've been looking for correct, canonical or recommended way to generate dynamically type object definitions, add methods, add attributes, etc. Here is what I found three approaches discussed on the Web: A) SO: How to dynamically create a derived type in the Python C-API http://stackoverflow.com/questions/8066438/how-to-dynamically-create-a-derived-type-in-the-python-c-api where PyType_Type.tp_alloc is used to allocate type, then tp_* members are filled and PyType_Ready is called. I've copied the example here for easier discussion: PyTypeObject *BrownNoddyType = (PyTypeObject *)PyType_Type.tp_alloc(&PyType_Type, 0); BrownNoddyType->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE; BrownNoddyType->tp_name = "noddy.BrownNoddy"; BrownNoddyType->tp_doc = "BrownNoddy objects"; BrownNoddyType->tp_base = &NoddyType; BrownNoddyType->tp_new = BrownNoddy_new; PyType_Ready(BrownNoddyType); This looks easy, but it is not clear to me a) How to add methods and members, getters and setters? Shall I simply call PyDescr_NewMethod, and other PyDescr_* functions? The manual is quite short here: http://docs.python.org/release/3.2.2/c-api/descriptor.html b) How shall I perform clean-up of BrownNoddyType? Is PyModuleDef .m_free function the right way to do it? Or, I simply set Py_TPFLAGS_HEAPTYPE and the dynamically object types will be reference-counted and cleaned automatically as discussed here http://thread.gmane.org/gmane.comp.python.devel/105648 B) SO: Python and Dynamically Extending C++ Classes http://stackoverflow.com/questions/7759827/python-and-dynamically-extending-c-classes This approach is similar to 1), but it uses base type object. New sub-classed type objects are 1) created dynamically using PyObject_New 2) and extended with new methods using PyCFunction_New and PyInstanceMethod_New functions. Here come questions: a) Do I need to perform any clean-up? Where? b) Is it possible/sensible to replace PyCFunction_New and PyInstanceMethod_New with PyDescr_NewMethod? C) Following what's discussed in "How do I add method dynamically to module using C API?" http://www.velocityreviews.com/forums/t727676-how-do-i-add-method-dynamically-to-module-using-c-api.html (Complete C/C++ code example is presented in the last post at the bottom.) It seems to be similar to the B) approach. This thread is about adding methods to modules and it uses undocumented PyCFunction_NewEx. But, I sense this technique should work for type objects too, once they are created. The big question is, which of these approaches is recommended for Python 3.x, or is there any other/better way to create dynamic type objects not discussed above? I'd appreciate any suggestions, hints or pointers in the Python docs which discuss this (perhaps I've missed it). Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Thu May 10 13:28:09 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 13:28:09 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: Message-ID: <4FABA649.3010808@behnel.de> Mateusz Loskot, 10.05.2012 13:24: > This is my first post to this list, hello everyone! > I hope this post is not off-topic here. > > I'm writing fairly complex structure of Python extensions directly using > Python C API (version 3+ only). I define some type objects with statically, > according to the canonical Noddy example presented in the docs. > Some type objects have to be defined/composed dynamically in run-time. Do you really need a type or is a Python class enough? The letter would be much easier to create. Stefan From mateusz at loskot.net Thu May 10 13:47:02 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 12:47:02 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FABA649.3010808@behnel.de> References: <4FABA649.3010808@behnel.de> Message-ID: On 10 May 2012 12:28, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 13:24: >> This is my first post to this list, hello everyone! >> I hope this post is not off-topic here. >> >> I'm writing fairly complex structure of Python extensions directly using >> Python C API (version 3+ only). I define some type objects with statically, >> according to the canonical Noddy example presented in the docs. >> Some type objects have to be defined/composed dynamically in run-time. > > Do you really need a type or is a Python class enough? The letter would be > much easier to create. Stefan, I'm not sure I understand. Aren't terms "class" and "type" names of the same concept, since Python 2.2? What is the noddy_NoddyType in the "Defining New Types" example? http://docs.python.org/release/3.2.2/extending/newtypes.html I'm sorry if I'm lacking of basics, but I follow the terminology as explained by Eli Bendersky in http://eli.thegreenplace.net/2012/03/30/python-objects-types-classes-and-instances-a-glossary/ In my system, I'm embedding Python and I add custom Python extension too (let's call it 'emb') So, users of my embedded Python have access to 'emb' module. The 'emb' module defines number of types, some are defined statically, as the noddy_NoddyType, so users can instantiate it n = emb.Noddy() n.bar() # method defined statically in methods table of noddy_NoddyType Now, I'd like to add some types which are generated in run-time, way before the 'emb' module is appended to inittab and embedded Python is initialised. And, I'd like to enable users to instantiate them in the same way as Noddy above: d = emb.GeneratedNoddy() or allow users to use and access d = emb.foo() d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition where: type(d) I hope it makes my intentions clear. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Thu May 10 14:00:36 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 14:00:36 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> Message-ID: <4FABADE4.4020401@behnel.de> Mateusz Loskot, 10.05.2012 13:47: > On 10 May 2012 12:28, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 13:24: >>> This is my first post to this list, hello everyone! >>> I hope this post is not off-topic here. >>> >>> I'm writing fairly complex structure of Python extensions directly using >>> Python C API (version 3+ only). I define some type objects with statically, >>> according to the canonical Noddy example presented in the docs. >>> Some type objects have to be defined/composed dynamically in run-time. >> >> Do you really need a type or is a Python class enough? The letter would be >> much easier to create. > > I'm not sure I understand. > Aren't terms "class" and "type" names of the same concept, since Python 2.2? With "type" I meant "extension type", i.e. the one you define using C structs. The alternative would be a simple Python class as you would get with class MyClass(object): pass > What is the noddy_NoddyType in the "Defining New Types" example? That's an extension type. > In my system, I'm embedding Python and I add custom Python extension > too (let's call it 'emb') > So, users of my embedded Python have access to 'emb' module. > The 'emb' module defines number of types, some are defined statically, > as the noddy_NoddyType, so users can instantiate it > > n = emb.Noddy() > n.bar() # method defined statically in methods table of noddy_NoddyType > > Now, I'd like to add some types which are generated in run-time, way > before the 'emb' module is appended to inittab and embedded Python is > initialised. Why would you want to do that before initialising the Python runtime? > And, I'd like to enable users to instantiate them in the same way as > Noddy above: > > d = emb.GeneratedNoddy() > > or allow users to use and access > > d = emb.foo() > d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition > > where: > > type(d) > > > I hope it makes my intentions clear. Not clear enough. The question is what your dynamically created types should do and provide. Would they need to be implemented in C (not just their methods but the types themselves!) or would a Python implementation suffice, potentially with methods implemented in C? You may get away with using normal Python classes that inherit from an extension type, for example. In that case, I'd also advise you to take a look at Cython, because that makes both the implementation of extension types and the creation of Python classes inheriting from them as easy as Py. Stefan From mateusz at loskot.net Thu May 10 14:37:20 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 13:37:20 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FABADE4.4020401@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> Message-ID: On 10 May 2012 13:00, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 13:47: >> On 10 May 2012 12:28, Stefan Behnel wrote: >>> Mateusz Loskot, 10.05.2012 13:24: >>>> >>>> I'm writing fairly complex structure of Python extensions directly using >>>> Python C API (version 3+ only). I define some type objects with statically, >>>> according to the canonical Noddy example presented in the docs. >>>> Some type objects have to be defined/composed dynamically in run-time. >>> >>> Do you really need a type or is a Python class enough? The letter would be >>> much easier to create. >> >> I'm not sure I understand. >> Aren't terms "class" and "type" names of the same concept, since Python 2.2? > > With "type" I meant "extension type", i.e. the one you define using C structs. So, I probably misused "object type" as a shortcut referring to the pair of struct + PyTypeObject instance: typedef struct { PyObject_HEAD } noddy_NoddyObject; static PyTypeObject noddy_NoddyType = { ... } > The alternative would be a simple Python class as you would get with > > ? class MyClass(object): pass This is clear to me. I simply was confused because AFAIK there is no class while using Python C API. As mentioned before, I follow Eli Bendersky's: " Note that when we create new types using the C API of CPython, there?s no "class" mentioned ? we create a new "type", not a new "class". " >> What is the noddy_NoddyType in the "Defining New Types" example? > > That's an extension type. Hmm, in the Python manual http://docs.python.org/release/3.2.2/extending/newtypes.html I'm reading this: "Moving on, we come to the crunch ? the type object. static PyTypeObject noddy_NoddyType = { ... } " What makes me interpret it as noddy_NoddyType is the type object. Simply, I understand that the pair of the struct noddy_NoddyObject and noddy_NoddyType together state the Python extension type, Am I still confusing the terminology? >> In my system, I'm embedding Python and I add custom Python extension >> too (let's call it 'emb') >> So, users of my embedded Python have access to 'emb' module. >> The 'emb' module defines number of types, some are defined statically, >> as the noddy_NoddyType, so users can instantiate it >> >> n = emb.Noddy() >> n.bar() # method defined statically in methods table of noddy_NoddyType >> >> Now, I'd like to add some types which are generated in run-time, way >> before the 'emb' module is appended to inittab and embedded Python is >> initialised. > > Why would you want to do that before initialising the Python runtime? The 'emb' must be added to inittab before Python is initialised (that's what the manual says, isn't it.) So, the scheme is this: /* 1. Dynamically generate emb.GeneratedNoddy */ ... /* Trying to figure out how */ /* 2. Register 'emb' module as built-in */ PyImport_AppendInittab("emb", &PyInit_emb); /* 3. Initialise Python */ Py_Initialize(); /* 4. we're ready to use */ ... /* 5. Clean-up */ Py_Finalize(); Does it make sense? >> And, I'd like to enable users to instantiate them in the same way as >> Noddy above: >> >> d = emb.GeneratedNoddy() >> >> or allow users to use and access >> >> d ?= emb.foo() >> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition >> >> where: >> >> type(d) >> >> >> I hope it makes my intentions clear. > > Not clear enough. The question is what your dynamically created types > should do and provide. Would they need to be implemented in C (not just > their methods but the types themselves!) or would a Python implementation > suffice, potentially with methods implemented in C? OK, I think I see where is the gap in my explanation. 0. The generated objects will define wrappers for existing C/C++ API. 1. I have defined contained object which needs extra steps to initialise within C/C++ Namely, I need to initialise members like pointer_to_some_wrapped_api_element, it may be some arbitrary data or PyCapsule, etc. /* statically typedef struct { PyObject_HEAD /* members initialised PyObject* pointer_to_some_wrapped_api_element; } GeneratedNoddyObject; 2. The corresponding GeneratedNoddyType is generated dynamically, with added set of methods, etc. 3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy. AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to compose the 'emb' module. Alternatively, I can generate Python classes (script in text form), but there is one inconvenience. The only way to expose such classes I know is to compile and import as *separate* module using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the resulting structure would be: emb emb.Noddy emb._privemb emb._privemb.GeneratedNoddy But, I'd like to achieve flat structure: emb emb.Noddy emb.GeneratedNoddy So, presumably I would have to walk the dictionary of emb._privemb and forward generated classes as attributes of 'emb'. Am I correct? Given that complication of loading classes from textual form through intermediate module, I thought using Python C API to generate extension types is better. (I don't mine dealing with C code verbosity and complexity.) > You may get away with using normal Python classes that inherit from an > extension type, for example. In that case, I'd also advise you to take a > look at Cython, because that makes both the implementation of extension > types and the creation of Python classes inheriting from them as easy as Py. Sounds interesting, I will take a look at it. However, I'd really like to learn canonical means of generating extension types dynamically using plain Python C API. Long story short, I assume I'm looking for Python C API equivalent of using type() function. Thank you Stefan! Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From mateusz at loskot.net Thu May 10 14:52:30 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 13:52:30 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> Message-ID: On 10 May 2012 13:37, Mateusz Loskot wrote: > On 10 May 2012 13:00, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 13:47: >>> >>> Now, I'd like to add some types which are generated in run-time, way >>> before the 'emb' module is appended to inittab and embedded Python is >>> initialised. >> >> Why would you want to do that before initialising the Python runtime? > > The 'emb' must be added to inittab before Python is initialised (that's what the > manual says, isn't it.) So, the scheme is this: > > /* 1. Dynamically generate emb.GeneratedNoddy */ > ... /* Trying to figure out how */ > > /* 2. Register 'emb' module as built-in */ > PyImport_AppendInittab("emb", &PyInit_emb); > > /* 3. Initialise Python */ > Py_Initialize(); > > /* 4. we're ready to use */ > ... > > /* 5. Clean-up */ > Py_Finalize(); > > Does it make sense? I recall the explanation above. It is not necessary to place extension types construction before the PyImport_AppendInittab and Py_Initialize, of course. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Thu May 10 15:01:56 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 15:01:56 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> Message-ID: <4FABBC44.6040901@behnel.de> Mateusz Loskot, 10.05.2012 14:37: > On 10 May 2012 13:00, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 13:47: >>> On 10 May 2012 12:28, Stefan Behnel wrote: >>>> Mateusz Loskot, 10.05.2012 13:24: >>>>> >>>>> I'm writing fairly complex structure of Python extensions directly using >>>>> Python C API (version 3+ only). I define some type objects with statically, >>>>> according to the canonical Noddy example presented in the docs. >>>>> Some type objects have to be defined/composed dynamically in run-time. >>>> >>>> Do you really need a type or is a Python class enough? The letter would be >>>> much easier to create. >>> >>> I'm not sure I understand. >>> Aren't terms "class" and "type" names of the same concept, since Python 2.2? >> >> With "type" I meant "extension type", i.e. the one you define using C structs. > > So, I probably misused "object type" as a shortcut referring > to the pair of struct + PyTypeObject instance: > > typedef struct { > PyObject_HEAD > } noddy_NoddyObject; > > static PyTypeObject noddy_NoddyType = { ... } > >> The alternative would be a simple Python class as you would get with >> >> class MyClass(object): pass > > This is clear to me. I simply was confused because AFAIK there is no > class while using Python C API. As mentioned before, I follow Eli Bendersky's: > > " > Note that when we create new types using the C API of CPython, > there?s no "class" mentioned ? we create a new "type", not a new "class". > " I think the emphasis is on "when we create new types". The sentence above sounds like a tautology to me (we obviously create a type and not a class when we create a type). >>> What is the noddy_NoddyType in the "Defining New Types" example? >> >> That's an extension type. > > Hmm, in the Python manual > http://docs.python.org/release/3.2.2/extending/newtypes.html > > I'm reading this: > > "Moving on, we come to the crunch ? the type object. > > static PyTypeObject noddy_NoddyType = { > ... > } > " > > What makes me interpret it as noddy_NoddyType is the type object. > > Simply, I understand that the pair of the > struct noddy_NoddyObject and noddy_NoddyType > together state the Python extension type, > > Am I still confusing the terminology? I don't care all that much about terminology myself. >>> In my system, I'm embedding Python and I add custom Python extension >>> too (let's call it 'emb') >>> So, users of my embedded Python have access to 'emb' module. >>> The 'emb' module defines number of types, some are defined statically, >>> as the noddy_NoddyType, so users can instantiate it >>> >>> n = emb.Noddy() >>> n.bar() # method defined statically in methods table of noddy_NoddyType >>> >>> Now, I'd like to add some types which are generated in run-time, way >>> before the 'emb' module is appended to inittab and embedded Python is >>> initialised. >> >> Why would you want to do that before initialising the Python runtime? > > The 'emb' must be added to inittab before Python is initialised (that's what the > manual says, isn't it.) So, the scheme is this: > > /* 1. Dynamically generate emb.GeneratedNoddy */ > ... /* Trying to figure out how */ > > /* 2. Register 'emb' module as built-in */ > PyImport_AppendInittab("emb", &PyInit_emb); > > /* 3. Initialise Python */ > Py_Initialize(); > > /* 4. we're ready to use */ > ... > > /* 5. Clean-up */ > Py_Finalize(); > > > Does it make sense? No. You don't need to declare all types before hand. You can add a type (or function, or name, or whatever) to the module dict at any time, just as you can in Python. Whether that's a C implemented type of a normal Python class (or something else) doesn't matter at that point. The inittab mechanism is just there for convenience when everything *is* statically defined at compile time (or at least module init time). >>> And, I'd like to enable users to instantiate them in the same way as >>> Noddy above: >>> >>> d = emb.GeneratedNoddy() >>> >>> or allow users to use and access >>> >>> d = emb.foo() >>> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition >>> >>> where: >>> >>> type(d) >>> >>> >>> I hope it makes my intentions clear. >> >> Not clear enough. The question is what your dynamically created types >> should do and provide. Would they need to be implemented in C (not just >> their methods but the types themselves!) or would a Python implementation >> suffice, potentially with methods implemented in C? > > OK, I think I see where is the gap in my explanation. > > 0. The generated objects will define wrappers for existing C/C++ API. > > 1. I have defined contained object which needs extra steps to > initialise within C/C++ > Namely, I need to initialise members like pointer_to_some_wrapped_api_element, > it may be some arbitrary data or PyCapsule, etc. > > /* statically > typedef struct { > PyObject_HEAD > /* members initialised > PyObject* pointer_to_some_wrapped_api_element; > } GeneratedNoddyObject; > > 2. The corresponding GeneratedNoddyType is generated dynamically, > with added set of methods, etc. > > 3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy. > > AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to > compose the 'emb' module. Not at all. It's much easier to use a Python class than to do everything in lengthy C-API declarations. > Alternatively, I can generate Python classes (script in text form), > but there is one inconvenience. > The only way to expose such classes I know is to compile and import as > *separate* module > using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the > resulting structure would be: > > emb > emb.Noddy > emb._privemb > emb._privemb.GeneratedNoddy Wrong again. You can execute any Python code from your C code. Look for the PyRun_*() functions. Note that this is even easier in Cython, where you write Python code anyway (instead of C code). > Given that complication of loading classes from textual form through > intermediate module, > I thought using Python C API to generate extension types is better. > (I don't mine dealing with C code verbosity and complexity.) You should. The simpler the code, the easier it is to maintain it. Why else would you want to embed Python instead of using C for everything? >> You may get away with using normal Python classes that inherit from an >> extension type, for example. In that case, I'd also advise you to take a >> look at Cython, because that makes both the implementation of extension >> types and the creation of Python classes inheriting from them as easy as Py. > > Sounds interesting, I will take a look at it. > > However, I'd really like to learn canonical means of generating extension types > dynamically using plain Python C API. > Long story short, I assume I'm looking for Python C API equivalent of > using type() function. Then call type(). Stefan From mateusz at loskot.net Thu May 10 15:55:27 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 14:55:27 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FABBC44.6040901@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: On 10 May 2012 14:01, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 14:37: >> On 10 May 2012 13:00, Stefan Behnel wrote: >>> Mateusz Loskot, 10.05.2012 13:47: >>>> On 10 May 2012 12:28, Stefan Behnel wrote: >>>>> Mateusz Loskot, 10.05.2012 13:24: >>>>>> >>>>>> I'm writing fairly complex structure of Python extensions directly using >>>>>> Python C API (version 3+ only). I define some type objects with statically, >>>>>> according to the canonical Noddy example presented in the docs. >>>>>> Some type objects have to be defined/composed dynamically in run-time. >>>>> >>>>> Do you really need a type or is a Python class enough? The letter would be >>>>> much easier to create. >>>> >>>> I'm not sure I understand. >>>> Aren't terms "class" and "type" names of the same concept, since Python 2.2? >>> >>> With "type" I meant "extension type", i.e. the one you define using C structs. >> >> So, I probably misused "object type" as a shortcut referring >> to the pair of struct + PyTypeObject instance: >> >> typedef struct { >> ? ? PyObject_HEAD >> } noddy_NoddyObject; >> >> static PyTypeObject noddy_NoddyType = { ... } >> >>> The alternative would be a simple Python class as you would get with >>> >>> ? class MyClass(object): pass >> >> This is clear to me. I simply was confused because AFAIK there is no >> class while using Python C API. As mentioned before, I follow Eli Bendersky's: >> >> " >> Note that when we create new types using the C API of CPython, >> there?s no "class" mentioned ? we create a new "type", not a new "class". >> " > > I think the emphasis is on "when we create new types". The sentence above > sounds like a tautology to me (we obviously create a type and not a class > when we create a type). Thanks for clarification. >>>> In my system, I'm embedding Python and I add custom Python extension >>>> too (let's call it 'emb') >>>> So, users of my embedded Python have access to 'emb' module. >>>> The 'emb' module defines number of types, some are defined statically, >>>> as the noddy_NoddyType, so users can instantiate it >>>> >>>> n = emb.Noddy() >>>> n.bar() # method defined statically in methods table of noddy_NoddyType >>>> >>>> Now, I'd like to add some types which are generated in run-time, way >>>> before the 'emb' module is appended to inittab and embedded Python is >>>> initialised. >>> >>> Why would you want to do that before initialising the Python runtime? >> >> The 'emb' must be added to inittab before Python is initialised (that's what the >> manual says, isn't it.) So, the scheme is this: >> >> /* 1. Dynamically generate emb.GeneratedNoddy */ >> ... /* Trying to figure out how */ >> >> /* 2. Register 'emb' module as built-in */ >> PyImport_AppendInittab("emb", &PyInit_emb); >> >> /* 3. Initialise Python */ >> Py_Initialize(); >> >> /* 4. we're ready to use */ >> ... >> >> /* 5. Clean-up */ >> Py_Finalize(); >> >> >> Does it make sense? > > No. You don't need to declare all types before hand. You can add a type (or > function, or name, or whatever) to the module dict at any time, just as you > can in Python. Whether that's a C implemented type of a normal Python class > (or something else) doesn't matter at that point. > > The inittab mechanism is just there for convenience when everything *is* > statically defined at compile time (or at least module init time). Yes, I'd realised that my assumption was incorrect, so I followed up with response to my own post. >>>> And, I'd like to enable users to instantiate them in the same way as >>>> Noddy above: >>>> >>>> d = emb.GeneratedNoddy() >>>> >>>> or allow users to use and access >>>> >>>> d ?= emb.foo() >>>> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition >>>> >>>> where: >>>> >>>> type(d) >>>> >>>> >>>> I hope it makes my intentions clear. >>> >>> Not clear enough. The question is what your dynamically created types >>> should do and provide. Would they need to be implemented in C (not just >>> their methods but the types themselves!) or would a Python implementation >>> suffice, potentially with methods implemented in C? >> >> OK, I think I see where is the gap in my explanation. >> >> 0. The generated objects will define wrappers for existing C/C++ API. >> >> 1. I have defined contained object which needs extra steps to >> initialise within C/C++ >> Namely, I need to initialise members like pointer_to_some_wrapped_api_element, >> it may be some arbitrary data or PyCapsule, etc. >> >> /* statically >> typedef struct { >> ? ? PyObject_HEAD >> ? ? /* members initialised >> ? ? PyObject* pointer_to_some_wrapped_api_element; >> } GeneratedNoddyObject; >> >> 2. The corresponding GeneratedNoddyType is generated dynamically, >> with added set of methods, etc. >> >> 3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy. >> >> AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to >> compose the 'emb' module. > > Not at all. It's much easier to use a Python class than to do everything in > lengthy C-API declarations. > > >> Alternatively, I can generate Python classes (script in text form), >> but there is one inconvenience. >> The only way to expose such classes I know is to compile and import as >> *separate* module >> using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the >> resulting structure would be: >> >> emb >> emb.Noddy >> emb._privemb >> emb._privemb.GeneratedNoddy > > Wrong again. You can execute any Python code from your C code. Look for the > PyRun_*() functions. Do you mean something similar to this approach? /* dynamically generated lengthy class definition */ const char* c = "class A(object): pass"; PyObject* class_a = PyRun_StringFlags(c, ...); PyObject_SetAttrString(module, "A", class_a) > Note that this is even easier in Cython, where you write Python code anyway > (instead of C code). Yes, I looked at the Cython docs. Unfortunately, I can't use it. I have to stick to Python 3.2 dist. >> Given that complication of loading classes from textual form through >> intermediate module, >> I thought using Python C API to generate extension types is better. >> (I don't mine dealing with C code verbosity and complexity.) > > You should. The simpler the code, the easier it is to maintain it. Why else > would you want to embed Python instead of using C for everything? You are right of course. It's clear to me too, but there seem to be many ways to achieve same/similar results, so I'm trying to survey which one is most recommended in my case. >> However, I'd really like to learn canonical means of generating extension types >> dynamically using plain Python C API. >> Long story short, I assume I'm looking for Python C API equivalent of >> using type() function. > > Then call type(). It's possible, certainly. But, is there any plain C equivalent, perhaps using PyType_Type and the Descriptor Objects? (Why purpose and usage of the Descriptor Objects are not documented?.) I'm seeking for confirmation, if such approach would be close to the type() function. To summary: 1) It seems to be that generating script with Python classes in textual form and then using Python C API like PyRun_StringFlags, etc. is the recommended approach 2) There is approach based on direct call to type() function using Python C API. 3) Is there third option possible which replaces the 2nd's type() call with chain of plain Python C API calls? I know there are pros/cons related to amount of code, maintenance hassle, code complexity, and such. but it is not related to my question really. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From mateusz at loskot.net Thu May 10 18:17:52 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 17:17:52 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: On 10 May 2012 14:55, Mateusz Loskot wrote: > On 10 May 2012 14:01, Stefan Behnel wrote: >> >> You can execute any Python code from your C code. Look for the >> PyRun_*() functions. > > Do you mean something similar to this approach? > > ?/* dynamically generated lengthy class definition */ > const char* c = "class A(object): pass"; > > PyObject* class_a = PyRun_StringFlags(c, ...); > PyObject_SetAttrString(module, "A", class_a) The pseudo-code above is incorrect. I have come up with the following example of dynamically generating new Python class, indirectly through script using class keyword. Such dynamically created class is added to module dictionary: /* error checks removed for brevity */ static PyModuleDef embmodule = { ... }; PyInit_emb(void) { PyObject* m = PyModule_Create(&embmodule); PyObject* d = PyModule_GetDict(m); /* Required to allow 'class' use in context of module which is not yet complete and ready. Otherwise, error is thrown: ImportError: __build_class__ not found */ PyObject* builtins = PyEval_GetBuiltins(); PyDict_SetItemString(d, "__builtins__", builtins); /* Python class is dynamically generated */ const char* c = "class A(object):\n\tpass"; /* sample class */ /* Create object for A class, automatically added to the module dictionary as noddy.A */ PyRun_StringFlags(c, Py_file_input, d, d, NULL); return m; } This approach works well. If anyone noticed a problem or there is better way to do the same, please let me know. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Thu May 10 18:32:04 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 18:32:04 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: <4FABED84.9020001@behnel.de> Mateusz Loskot, 10.05.2012 15:55: > On 10 May 2012 14:01, Stefan Behnel wrote: >> Note that this is even easier in Cython, where you write Python code anyway >> (instead of C code). > > Yes, I looked at the Cython docs. > Unfortunately, I can't use it. I have to stick to Python 3.2 dist. Cython works perfectly with Py3.2, even with the latest 3.3-pre and Python 2.4 and later as well. Helps you write portable code. >>> However, I'd really like to learn canonical means of generating extension types >>> dynamically using plain Python C API. >>> Long story short, I assume I'm looking for Python C API equivalent of >>> using type() function. >> >> Then call type(). > > It's possible, certainly. > But, is there any plain C equivalent, perhaps using PyType_Type > and the Descriptor Objects? > (Why purpose and usage of the Descriptor Objects are not documented?.) > I'm seeking for confirmation, if such approach would be close to the > type() function. > > To summary: > > 1) > It seems to be that generating script with Python classes in textual form > and then using Python C API like PyRun_StringFlags, etc. > is the recommended approach It's certainly much easier than doing everything in C. > 2) > There is approach based on direct call to type() function > using Python C API. > > 3) > Is there third option possible which replaces the 2nd's type() call > with chain of plain Python C API calls? > I know there are pros/cons related to amount of code, maintenance hassle, > code complexity, and such. but it is not related to my question really. I don't know what the best way is here. However, I really wonder what you are trying to achieve. If I was to implement an interface for an embedded Python runtime, I'd much rather focus on the design of the user level API that I'm exposing than the details of the low-level interface implementation. (But then again, I'd leave those mostly to Cython, so if you prefer doing them yourself, you'll have to jump through some more hoops, obviously.) Stefan From ideasman42 at gmail.com Thu May 10 18:38:31 2012 From: ideasman42 at gmail.com (Campbell Barton) Date: Thu, 10 May 2012 18:38:31 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: If speed of class creation is important (as it is for us). suggest to call type directly rather then evaluating a string every time and executing it - this is one way. /* always use O not N when calling, N causes refcount errors */ newclass = PyObject_CallFunction((PyObject *)&PyType_Type, (char *)"s(O) {sss()}", idname, py_base_class, "__module__", "bpy.types", "__slots__"); which is like doing this in python... type(idname, (py_base_class, ), dict(__module__="bpy.types", __slots__=())) On Thu, May 10, 2012 at 6:17 PM, Mateusz Loskot wrote: > On 10 May 2012 14:55, Mateusz Loskot wrote: >> On 10 May 2012 14:01, Stefan Behnel wrote: >>> >>> You can execute any Python code from your C code. Look for the >>> PyRun_*() functions. >> >> Do you mean something similar to this approach? >> >> ?/* dynamically generated lengthy class definition */ >> const char* c = "class A(object): pass"; >> >> PyObject* class_a = PyRun_StringFlags(c, ...); >> PyObject_SetAttrString(module, "A", class_a) > > The pseudo-code above is incorrect. > I have come up with the following example of dynamically > generating new Python class, indirectly through script using class keyword. > Such dynamically created class is added to module dictionary: > > /* error checks removed for brevity */ > > static PyModuleDef embmodule = { ... }; > > PyInit_emb(void) > { > ? ?PyObject* m = PyModule_Create(&embmodule); > ? ?PyObject* d = PyModule_GetDict(m); > > ? ?/* Required to allow 'class' use in context of module which is > ? ? ? not yet complete and ready. Otherwise, error is thrown: > ? ? ? ImportError: __build_class__ not found > ? ?*/ > ? ?PyObject* builtins = PyEval_GetBuiltins(); > ? ?PyDict_SetItemString(d, "__builtins__", builtins); > > ? ?/* Python class is dynamically generated */ > ? ?const char* c = "class A(object):\n\tpass"; /* sample class */ > > ? ?/* Create object for A class, automatically added to the module > dictionary as noddy.A */ > ? ?PyRun_StringFlags(c, Py_file_input, d, d, NULL); > > ? ?return m; > } > > This approach works well. > If anyone noticed a problem or there is better way to do the same, > please let me know. > > Best regards, > -- > Mateusz Loskot, http://mateusz.loskot.net > _______________________________________________ > capi-sig mailing list > capi-sig at python.org > http://mail.python.org/mailman/listinfo/capi-sig -- - Campbell From python_capi at behnel.de Thu May 10 18:39:44 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 18:39:44 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: <4FABEF50.1010704@behnel.de> Mateusz Loskot, 10.05.2012 18:17: > On 10 May 2012 14:55, Mateusz Loskot wrote: >> On 10 May 2012 14:01, Stefan Behnel wrote: >>> You can execute any Python code from your C code. Look for the >>> PyRun_*() functions. >> >> Do you mean something similar to this approach? >> >> /* dynamically generated lengthy class definition */ >> const char* c = "class A(object): pass"; >> >> PyObject* class_a = PyRun_StringFlags(c, ...); >> PyObject_SetAttrString(module, "A", class_a) > > The pseudo-code above is incorrect. > I have come up with the following example of dynamically > generating new Python class, indirectly through script using class keyword. > Such dynamically created class is added to module dictionary: > > /* error checks removed for brevity */ > > static PyModuleDef embmodule = { ... }; > > PyInit_emb(void) > { > PyObject* m = PyModule_Create(&embmodule); > PyObject* d = PyModule_GetDict(m); > > /* Required to allow 'class' use in context of module which is > not yet complete and ready. Otherwise, error is thrown: > ImportError: __build_class__ not found > */ > PyObject* builtins = PyEval_GetBuiltins(); > PyDict_SetItemString(d, "__builtins__", builtins); > > /* Python class is dynamically generated */ > const char* c = "class A(object):\n\tpass"; /* sample class */ > > /* Create object for A class, automatically added to the module > dictionary as noddy.A */ > PyRun_StringFlags(c, Py_file_input, d, d, NULL); You want to take the result here, if only to decref it. What you'd do next is to initialise your types in the module (or let CPython do it for you through the inittab) and then let the Python classes inherit from them, stick C methods into them, etc. > return m; > } > > This approach works well. > If anyone noticed a problem or there is better way to do the same, > please let me know. Looks ok to me. Here's the equivalent code in Cython: class A(object): pass (except that it executes faster because it doesn't run through the Python parser and interpreter at runtime, but I don't think that matters much at module init time). Stefan From mateusz at loskot.net Thu May 10 18:58:24 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 17:58:24 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FABED84.9020001@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABED84.9020001@behnel.de> Message-ID: On 10 May 2012 17:32, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 15:55: >> >> 3) >> Is there third option possible which replaces the 2nd's type() call >> with chain of plain Python C API calls? >> I know there are pros/cons related to amount of code, maintenance hassle, >> code complexity, and such. but it is not related to my question really. > > I don't know what the best way is here. However, I really wonder what you > are trying to achieve. Long story short, I have a software system implemented in C/C++ which provides public API. The API is not fixed, may change, so scripting engines use provided means of reflection to inspect this API and generate bindings on-fly. I embedded Python in this system and now I'm using the mentioned reflection mechanisms to discover API and generate Python extension modules and types dynamically during run-time. I simply use the reflection mechanisms provided to iterate over definitions (structures, functions, constants, etc.) of the public API and I compose Python extension types grouping methods, etc. I can not use Cython, nor I can use tools like SWIG, Boost.Python. The (C)Python is the only tool I can use. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From mateusz at loskot.net Thu May 10 19:01:09 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 18:01:09 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> Message-ID: On 10 May 2012 17:38, Campbell Barton wrote: > If speed of class creation is important (as it is for us). suggest to > call type directly rather then evaluating a string every time and > executing it - this is one way. > > > ? ? ? ? ? ? ? ?/* always use O not N when calling, N causes refcount errors */ > ? ? ? ? ? ? ? ?newclass = PyObject_CallFunction((PyObject *)&PyType_Type, (char > *)"s(O) {sss()}", > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? idname, py_base_class, > "__module__", "bpy.types", "__slots__"); > > which is like doing this in python... > > ?type(idname, (py_base_class, ), dict(__module__="bpy.types", __slots__=())) Campbell, Thanks for the hint. It's good to know how the type() should be called from C. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From mateusz at loskot.net Thu May 10 19:13:14 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Thu, 10 May 2012 18:13:14 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FABEF50.1010704@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABEF50.1010704@behnel.de> Message-ID: On 10 May 2012 17:39, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 18:17: >> On 10 May 2012 14:55, Mateusz Loskot wrote: >>> On 10 May 2012 14:01, Stefan Behnel wrote: >>>> You can execute any Python code from your C code. Look for the >>>> PyRun_*() functions. >>> >>> Do you mean something similar to this approach? >>> >>> ?/* dynamically generated lengthy class definition */ >>> const char* c = "class A(object): pass"; >>> >>> PyObject* class_a = PyRun_StringFlags(c, ...); >>> PyObject_SetAttrString(module, "A", class_a) >> >> The pseudo-code above is incorrect. >> I have come up with the following example of dynamically >> generating new Python class, indirectly through script using class keyword. >> Such dynamically created class is added to module dictionary: >> >> /* error checks removed for brevity */ >> >> static PyModuleDef embmodule = { ... }; >> >> PyInit_emb(void) >> { >> ? ? PyObject* m = PyModule_Create(&embmodule); >> ? ? PyObject* d = PyModule_GetDict(m); >> >> ? ? /* Required to allow 'class' use in context of module which is >> ? ? ? ?not yet complete and ready. Otherwise, error is thrown: >> ? ? ? ?ImportError: __build_class__ not found >> ? ? */ >> ? ? PyObject* builtins = PyEval_GetBuiltins(); >> ? ? PyDict_SetItemString(d, "__builtins__", builtins); >> >> ? ? /* Python class is dynamically generated */ >> ? ? const char* c = "class A(object):\n\tpass"; /* sample class */ >> >> ? ? /* Create object for A class, automatically added to the module >> dictionary as emb.A */ >> ? ? PyRun_StringFlags(c, Py_file_input, d, d, NULL); > > You want to take the result here, if only to decref it. Right, I missed it out from the example. > What you'd do next is to initialise your types in the module (or let > CPython do it for you through the inittab) I'm not sure I completely understand it. I assume that executing these two lines, in context of the module: const char* c = "class A(object):\n\tpass"; PyRun_StringFlags(c, Py_file_input, d, d, NULL); creates complete and initialised A class within this module. Doesn't it? > and then let the Python classes > inherit from them, stick C methods into them, etc. Right. Thanks for the patience :) >> This approach works well. >> If anyone noticed a problem or there is better way to do the same, >> please let me know. > > Looks ok to me. Great! > Here's the equivalent code in Cython: > > ? ?class A(object): pass > > (except that it executes faster because it doesn't run through the Python > parser and interpreter at runtime, but I don't think that matters much at > module init time). It looks really helpful and simple, I'll remember, and chances are I will use it in next project. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Thu May 10 19:52:27 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 19:52:27 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABEF50.1010704@behnel.de> Message-ID: <4FAC005B.30906@behnel.de> Mateusz Loskot, 10.05.2012 19:13: > On 10 May 2012 17:39, Stefan Behnel wrote: >> What you'd do next is to initialise your types in the module (or let >> CPython do it for you through the inittab) > > I'm not sure I completely understand it. > I assume that executing these two lines, > in context of the module: > > const char* c = "class A(object):\n\tpass"; > PyRun_StringFlags(c, Py_file_input, d, d, NULL); > > creates complete and initialised A class within this module. > Doesn't it? Yes, it does. What I meant what: if you want to let the dynamically generated Python classes inherit from your extension types, then initialise the extension types before hand so that they turn up in the module dict and your Python code can use them normally as base types. Stefan From python_capi at behnel.de Thu May 10 19:58:51 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Thu, 10 May 2012 19:58:51 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABED84.9020001@behnel.de> Message-ID: <4FAC01DB.9040209@behnel.de> Mateusz Loskot, 10.05.2012 18:58: > Long story short, I have a software system implemented in C/C++ which > provides public API. The API is not fixed, may change, so scripting > engines use provided means of reflection to inspect this API and > generate bindings on-fly. > I embedded Python in this system and now I'm using the mentioned > reflection mechanisms to discover API and generate Python extension > modules and types dynamically during run-time. > > I simply use the reflection mechanisms provided to iterate over > definitions (structures, functions, constants, etc.) of the public API > and I compose Python extension types grouping methods, etc. Interesting. So you're reflecting on a C/C++ API to dynamically export it to Python space? Then I guess that your internal binding code is also rather generic, right? It won't be able to call the application code directly, for example, if it doesn't know the API at compile time. > I can not use Cython, nor I can use tools like SWIG, Boost.Python. > The (C)Python is the only tool I can use. Also interesting. I wonder who would notice with which tool you have written your source code once it's compiled... Stefan From mateusz at loskot.net Fri May 11 11:22:37 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Fri, 11 May 2012 10:22:37 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FAC005B.30906@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABEF50.1010704@behnel.de> <4FAC005B.30906@behnel.de> Message-ID: On 10 May 2012 18:52, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 19:13: >> On 10 May 2012 17:39, Stefan Behnel wrote: >>> What you'd do next is to initialise your types in the module (or let >>> CPython do it for you through the inittab) >> >> I'm not sure I completely understand it. >> I assume that executing these two lines, >> in context of the module: >> >> const char* c = "class A(object):\n\tpass"; >> PyRun_StringFlags(c, Py_file_input, d, d, NULL); >> >> creates complete and initialised A class within this module. >> Doesn't it? > > Yes, it does. What I meant what: if you want to let the dynamically > generated Python classes inherit from your extension types, then initialise > the extension types before hand so that they turn up in the module dict and > your Python code can use them normally as base types. It makes perfect sense to me, thanks! Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From mateusz at loskot.net Fri May 11 11:38:35 2012 From: mateusz at loskot.net (Mateusz Loskot) Date: Fri, 11 May 2012 10:38:35 +0100 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: <4FAC01DB.9040209@behnel.de> References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABED84.9020001@behnel.de> <4FAC01DB.9040209@behnel.de> Message-ID: On 10 May 2012 18:58, Stefan Behnel wrote: > Mateusz Loskot, 10.05.2012 18:58: >> Long story short, I have a software system implemented in C/C++ which >> provides public API. The API is not fixed, may change, so scripting >> engines use provided means of reflection to inspect this API and >> generate bindings on-fly. >> I embedded Python in this system and now I'm using the mentioned >> reflection mechanisms to discover API and generate Python extension >> modules and types dynamically during run-time. >> >> I simply use the reflection mechanisms provided to iterate over >> definitions (structures, functions, constants, etc.) of the public API >> and I compose Python extension types grouping methods, etc. > > Interesting. So you're reflecting on a C/C++ API to dynamically export it > to Python space? Yes, roughly, that's the idea. > Then I guess that your internal binding code is also > rather generic, right? Let's describe it in simplified way: There are intermediate Python extensions defined which bridge communication between the C++ world and dynamically generated Python classes. For instance, there is a car and car engine. Now, there is 'base car engine' which defines common protocol for possible behaviours. The 'base car engine" is specified and known at compile time, Next, there is variety of kinds of engines where each provide subset of the well-specified behaviours. Polymorphism, indeed. So, those specific engines are dynamically generated in run-time and they communicate with car through the statically defined 'base car engine' protocol. > It won't be able to call the application code > directly, for example, if it doesn't know the API at compile time. Yes, so there are 'base' extension types defined statically. >> I can not use Cython, nor I can use tools like SWIG, Boost.Python. >> The (C)Python is the only tool I can use. > > Also interesting. I wonder who would notice with which tool you have > written your source code once it's compiled... I guess nobody. It's just I had not known Cython before I started. Now, I have most of my implementation ready. I also wouldn't want to introduce new dependencies and as long as Cython is not guaranteed to be 100% compatible drop-in replacement for Python 3.2+, then I'm afraid I can't use it. The FAQ does not confirm it is http://wiki.cython.org/FAQ#IsCythonaPythonimplementation.3F But, I will keep an eye on Cython, it looks & feels a clever and useful idea. and even for "he's stuck his head in C++" hacker :D Stefan, I appreciate your help. I wish I found CAPI-SIG list earlier! Best regards, -- Mateusz Loskot, http://mateusz.loskot.net From python_capi at behnel.de Fri May 11 11:52:35 2012 From: python_capi at behnel.de (Stefan Behnel) Date: Fri, 11 May 2012 11:52:35 +0200 Subject: [capi-sig] Creating type object dynamically in run-time In-Reply-To: References: <4FABA649.3010808@behnel.de> <4FABADE4.4020401@behnel.de> <4FABBC44.6040901@behnel.de> <4FABED84.9020001@behnel.de> <4FAC01DB.9040209@behnel.de> Message-ID: <4FACE163.2070704@behnel.de> Mateusz Loskot, 11.05.2012 11:38: > On 10 May 2012 18:58, Stefan Behnel wrote: >> Mateusz Loskot, 10.05.2012 18:58: >>> Long story short, I have a software system implemented in C/C++ which >>> provides public API. The API is not fixed, may change, so scripting >>> engines use provided means of reflection to inspect this API and >>> generate bindings on-fly. >>> I embedded Python in this system and now I'm using the mentioned >>> reflection mechanisms to discover API and generate Python extension >>> modules and types dynamically during run-time. >>> >>> I simply use the reflection mechanisms provided to iterate over >>> definitions (structures, functions, constants, etc.) of the public API >>> and I compose Python extension types grouping methods, etc. >> >> Interesting. So you're reflecting on a C/C++ API to dynamically export it >> to Python space? > > Yes, roughly, that's the idea. > >> Then I guess that your internal binding code is also >> rather generic, right? > > Let's describe it in simplified way: > There are intermediate Python extensions defined which bridge > communication between the C++ world and dynamically generated Python classes. > For instance, there is a car and car engine. > Now, there is 'base car engine' which defines common protocol for > possible behaviours. > The 'base car engine" is specified and known at compile time, > Next, there is variety of kinds of engines where each provide > subset of the well-specified behaviours. Polymorphism, indeed. > > So, those specific engines are dynamically generated in run-time > and they communicate with car through the statically defined 'base car > engine' protocol. Hmm, that sounds more like you'd want to use a common compile time interface for them statically instead, but I guess I'm just underestimating the requirements here. >> It won't be able to call the application code >> directly, for example, if it doesn't know the API at compile time. > > Yes, so there are 'base' extension types defined statically. Makes sense. >>> I can not use Cython, nor I can use tools like SWIG, Boost.Python. >>> The (C)Python is the only tool I can use. >> >> Also interesting. I wonder who would notice with which tool you have >> written your source code once it's compiled... > > I guess nobody. It's just I had not known Cython before I started. > Now, I have most of my implementation ready. Sure, fair enough. (Although you wouldn't be the first one who ends up rewriting C code in Cython.) > I also wouldn't want to introduce new dependencies > and as long as Cython is not guaranteed to be 100% compatible > drop-in replacement for Python 3.2+, then I'm afraid I can't use it. > The FAQ does not confirm it is > http://wiki.cython.org/FAQ#IsCythonaPythonimplementation.3F Ah, that's the misunderstanding then. It's not meant as a replacement of CPython, only as a replacement for writing C/C++ code using CPython's C-API. It generates C code that runs on top of CPython and thus integrates natively with both C/C++ code and Python code. So, basically, you get the best of both worlds (C and Python) without having to write the C part of it. Stefan