Hello all,
I'd like to announce a new project for producing python extensions, called PyBindGen. It follows a radical new path for Python bindings generation: it is itself written in (gasp!) Python! And it does not even use C++ templates at all! :-)
The project is young but has been slowly developed for a few months now, so it has already reached a point where it is useful for many tasks. For now, it has only a Python interface. No interface description file or header scanning supported yet. It also has been focused more on C++ code for now, although it was designed to also support pure C from the start.
More information in: https://launchpad.net/pybindgen/
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
I'd like to announce a new project for producing python extensions
Launchpad lists lots of (non-existing?) releases and shows a Bazaar history of four months. May I ask in what sense this project is "new"?
The front page also says:
""" Generation is controlled exclusively by a Python API, no limited command-line interface or yet another interface definition file format """
There doesn't seem to be any documentation. Could you give a simple example for that?
More information in: https://launchpad.net/pybindgen/
You should create a PyPI page for it. People are more likely to look for it there.
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
It is new in the sense that it has never been officially announced :)
The front page also says:
http://codebrowse.launchpad.net/~gjc/pybindgen/devel/files/gjc%40inescporto....
(sorry for the long URL)
Basically the examples/ directory contains a sample module. Incidentally it evolved partially into a set of unit tests in addition to being examples.
Thanks for the tip; I'll try.
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
As are the wrapper generators, as it seems:
http://codebrowse.launchpad.net/~gjc/pybindgen/devel/annotate/gjc%40inescpor...
Do you really have to write all of this by hand to 'generate' a wrapper? How do you make it functional? (or does it require an exact API mapping?)
I can't help the feeling that Pyrex/Cython code is quite a bit more readable - and definitely less verbose. Any chance you could motivate your design a little?
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
I don't understand what "exact API mapping" means.
But yes, so far you have to write all this by hand. Although would love to have automatic header file scanning, one day... unfortunately I have not had enough time for that, and would rather focus on the code generation for now. Lots of neat things could be done later on top of the pybindgen python module interface...
I can't help the feeling that Pyrex/Cython code is quite a bit more readable
I don't like having to learn a new language for this. Just like Boost.Python users are comfortable with C++ and avoid having to learn a new language, PyBindGen users are comfortable with Python and don't have to learn a new language.
Besides, pyrex/cython C++ support is poor or nonexistent. Although I acknowledge that if you wrap pure C libraries then this point is moot.
Finally, have you ever looked at the code generated by pyrex? It is rather scary...
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
What I meant was: is the C(++) API mapped directly to Python or can you write abstraction code in between?
But that's what most wrapper generators are there for: move the work from the developer into a generator. If you have to write code for every function/method/class you wrap, I think you're better of with Pyrex/Cython, as Pyrex code is readable and meaningful Python, not just a wrapper writer script.
Pyrex is *almost* Python, no new language to learn. And Cython is even closer, as it features a couple of Python 2.5 and Python 3k features.
Besides, pyrex/cython C++ support is poor or nonexistent. Although I acknowledge that if you wrap pure C libraries then this point is moot.
There are a couple of patches that improve the support. Admittedly, Pyrex does not target C++ libraries, so OO support is mostly missing. But nothing keeps you from implementing an object interface against C++ classes. I bet the Cython project would be happy to include it.
Finally, have you ever looked at the code generated by pyrex?
Yes, definitely, it's pretty readable, well optimised (with Cython, that is) and straight forward (ok, loops are ugly and over-optimised - but *very* fast). And I totally like the Cython feature of keeping the surrounding Cython code inside a C comment. That way, you immediately know what you are looking at.
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
Generally it requires a direct mapping. Why would you want to write code in the middle? Although recently I added support for adding "user functions" to make them appear as methods of objects.
Anyway that's not the point. The PyBindGen design is not set in stone...
Come on, I wouldn't say this interface is unreadable. It's stuff like this:
mod = Module('foo')
Foo = CppClass('Foo', automatic_type_narrowing=True)
mod.add_class(Foo)
Foo.add_static_attribute(ReturnValue.new('int'), 'instance_count')
Foo.add_constructor(
CppConstructor([Parameter.new('std::string', 'datum')]))
Foo.add_constructor(CppConstructor([]))
Foo.add_method(CppMethod(ReturnValue.new('std::string'), 'get_datum',[]))
Not that hard to write this, is it?
For instance, I don't find this very readable:
static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/ static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self, PyObject *__pyx_args, PyObject *__pyx_kwds) { PyObject *__pyx_r; PyObject *__pyx_1 = 0; static char *__pyx_argnames[] = {0}; if (unlikely(!PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "", __pyx_argnames))) return 0; Py_INCREF(__pyx_v_self);
/* "/Users/robert/sage/pyrex/cython-0.9.6.3/Demos/spam.pyx":16 *
- def get_amount(self):
return self.amount # <<<<<<<<<<<<<<- def set_amount(self, new_amount): */ __pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam *)__pyx_v_self)->amount); if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 16; goto __pyx_L1;} __pyx_r = __pyx_1; __pyx_1 = 0; goto __pyx_L0;
__pyx_r = Py_None; Py_INCREF(Py_None); goto __pyx_L0; __pyx_L1:; Py_XDECREF(__pyx_1); __Pyx_AddTraceback("spam.Spam.get_amount"); __pyx_r = 0; __pyx_L0:; Py_DECREF(__pyx_v_self); return __pyx_r; }
Compared with this code generated by pybindgen:
static PyObject * _wrap_fooinvoke_some_object_get_prefix() { PyObject *py_retval; std::string retval;
retval = invoke_some_object_get_prefix();
py_retval = Py_BuildValue("s#", retval.c_str(), retval.size());
return py_retval;}
Pybindgen generated code is almost as clean as the code you would write by hand. To me that counts a lot. I know some people don't care what happens underneath the tools they use as long as it works. I am not that kind of person.
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
I didn't say it's unreadable and I didn't say it was hard to write. I'm just saying that you shouldn't have to write that at all if all you get is a straight mapping between a C(++) API and a Python API. That's what wrapper generators are there for: parse the interface description and generate the wrapper with as little human interaction as possible. And, that's what the wrapper languages (that you don't want to require people to learn) are good at.
So far for the signature.
That's for argument parsing - no need to look at that.
Here we go, this is the code that was translated into the C code below.
The first line is so long to keep you from looking at it, so if you ignore the error handling, what you see is:
__pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam *)__pyx_v_self)->amount);
__pyx_r = __pyx_1;
Relatively straight assignment code, split into code for getting the value and code for setting the value.
Again, ignore the clean up code.
Admittedly, it's far from hand-optimised, aesthetic code, but it's definitely readable.
Ok, but that's almost exactly the code you wrote by hand, step by step through your Python API. So the question is: what's the advantage compared to writing the code *directly* by hand, i.e. in plain C?
Oh, and: you're not passing any arguments here. If you were, your function would look a bit more bloated, just like the Pyrex code.
I do care a lot, that's why I keep submitting patches to Cython. The better the code gets that Cython generates, the less I have to care about the generated C code when I write Cython code.
I think *that* is the essence of a code generator tool.
Stefan
Gustavo Carneiro wrote:
I'd like to announce a new project for producing python extensions
Launchpad lists lots of (non-existing?) releases and shows a Bazaar history of four months. May I ask in what sense this project is "new"?
The front page also says:
""" Generation is controlled exclusively by a Python API, no limited command-line interface or yet another interface definition file format """
There doesn't seem to be any documentation. Could you give a simple example for that?
More information in: https://launchpad.net/pybindgen/
You should create a PyPI page for it. People are more likely to look for it there.
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
It is new in the sense that it has never been officially announced :)
The front page also says:
http://codebrowse.launchpad.net/~gjc/pybindgen/devel/files/gjc%40inescporto....
(sorry for the long URL)
Basically the examples/ directory contains a sample module. Incidentally it evolved partially into a set of unit tests in addition to being examples.
Thanks for the tip; I'll try.
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
As are the wrapper generators, as it seems:
http://codebrowse.launchpad.net/~gjc/pybindgen/devel/annotate/gjc%40inescpor...
Do you really have to write all of this by hand to 'generate' a wrapper? How do you make it functional? (or does it require an exact API mapping?)
I can't help the feeling that Pyrex/Cython code is quite a bit more readable - and definitely less verbose. Any chance you could motivate your design a little?
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
I don't understand what "exact API mapping" means.
But yes, so far you have to write all this by hand. Although would love to have automatic header file scanning, one day... unfortunately I have not had enough time for that, and would rather focus on the code generation for now. Lots of neat things could be done later on top of the pybindgen python module interface...
I can't help the feeling that Pyrex/Cython code is quite a bit more readable
I don't like having to learn a new language for this. Just like Boost.Python users are comfortable with C++ and avoid having to learn a new language, PyBindGen users are comfortable with Python and don't have to learn a new language.
Besides, pyrex/cython C++ support is poor or nonexistent. Although I acknowledge that if you wrap pure C libraries then this point is moot.
Finally, have you ever looked at the code generated by pyrex? It is rather scary...
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
What I meant was: is the C(++) API mapped directly to Python or can you write abstraction code in between?
But that's what most wrapper generators are there for: move the work from the developer into a generator. If you have to write code for every function/method/class you wrap, I think you're better of with Pyrex/Cython, as Pyrex code is readable and meaningful Python, not just a wrapper writer script.
Pyrex is *almost* Python, no new language to learn. And Cython is even closer, as it features a couple of Python 2.5 and Python 3k features.
Besides, pyrex/cython C++ support is poor or nonexistent. Although I acknowledge that if you wrap pure C libraries then this point is moot.
There are a couple of patches that improve the support. Admittedly, Pyrex does not target C++ libraries, so OO support is mostly missing. But nothing keeps you from implementing an object interface against C++ classes. I bet the Cython project would be happy to include it.
Finally, have you ever looked at the code generated by pyrex?
Yes, definitely, it's pretty readable, well optimised (with Cython, that is) and straight forward (ok, loops are ugly and over-optimised - but *very* fast). And I totally like the Cython feature of keeping the surrounding Cython code inside a C comment. That way, you immediately know what you are looking at.
Stefan
On 25/09/2007, Stefan Behnel <python_capi@behnel.de> wrote:
Generally it requires a direct mapping. Why would you want to write code in the middle? Although recently I added support for adding "user functions" to make them appear as methods of objects.
Anyway that's not the point. The PyBindGen design is not set in stone...
Come on, I wouldn't say this interface is unreadable. It's stuff like this:
mod = Module('foo')
Foo = CppClass('Foo', automatic_type_narrowing=True)
mod.add_class(Foo)
Foo.add_static_attribute(ReturnValue.new('int'), 'instance_count')
Foo.add_constructor(
CppConstructor([Parameter.new('std::string', 'datum')]))
Foo.add_constructor(CppConstructor([]))
Foo.add_method(CppMethod(ReturnValue.new('std::string'), 'get_datum',[]))
Not that hard to write this, is it?
For instance, I don't find this very readable:
static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/ static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self, PyObject *__pyx_args, PyObject *__pyx_kwds) { PyObject *__pyx_r; PyObject *__pyx_1 = 0; static char *__pyx_argnames[] = {0}; if (unlikely(!PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "", __pyx_argnames))) return 0; Py_INCREF(__pyx_v_self);
/* "/Users/robert/sage/pyrex/cython-0.9.6.3/Demos/spam.pyx":16 *
- def get_amount(self):
return self.amount # <<<<<<<<<<<<<<- def set_amount(self, new_amount): */ __pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam *)__pyx_v_self)->amount); if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 16; goto __pyx_L1;} __pyx_r = __pyx_1; __pyx_1 = 0; goto __pyx_L0;
__pyx_r = Py_None; Py_INCREF(Py_None); goto __pyx_L0; __pyx_L1:; Py_XDECREF(__pyx_1); __Pyx_AddTraceback("spam.Spam.get_amount"); __pyx_r = 0; __pyx_L0:; Py_DECREF(__pyx_v_self); return __pyx_r; }
Compared with this code generated by pybindgen:
static PyObject * _wrap_fooinvoke_some_object_get_prefix() { PyObject *py_retval; std::string retval;
retval = invoke_some_object_get_prefix();
py_retval = Py_BuildValue("s#", retval.c_str(), retval.size());
return py_retval;}
Pybindgen generated code is almost as clean as the code you would write by hand. To me that counts a lot. I know some people don't care what happens underneath the tools they use as long as it works. I am not that kind of person.
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
I didn't say it's unreadable and I didn't say it was hard to write. I'm just saying that you shouldn't have to write that at all if all you get is a straight mapping between a C(++) API and a Python API. That's what wrapper generators are there for: parse the interface description and generate the wrapper with as little human interaction as possible. And, that's what the wrapper languages (that you don't want to require people to learn) are good at.
So far for the signature.
That's for argument parsing - no need to look at that.
Here we go, this is the code that was translated into the C code below.
The first line is so long to keep you from looking at it, so if you ignore the error handling, what you see is:
__pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam *)__pyx_v_self)->amount);
__pyx_r = __pyx_1;
Relatively straight assignment code, split into code for getting the value and code for setting the value.
Again, ignore the clean up code.
Admittedly, it's far from hand-optimised, aesthetic code, but it's definitely readable.
Ok, but that's almost exactly the code you wrote by hand, step by step through your Python API. So the question is: what's the advantage compared to writing the code *directly* by hand, i.e. in plain C?
Oh, and: you're not passing any arguments here. If you were, your function would look a bit more bloated, just like the Pyrex code.
I do care a lot, that's why I keep submitting patches to Cython. The better the code gets that Cython generates, the less I have to care about the generated C code when I write Cython code.
I think *that* is the essence of a code generator tool.
Stefan
participants (2)
-
Gustavo Carneiro -
Stefan Behnel