Refactoring, Compilation Speed, Pyste, Lua/Ruby/JavaScript...
A number of you (who have my sympathy!) have begun to dig in to Boost.Python's implementation to some degree, usually so you can accomplish something that isn't exposed in the documented interface. I'm sure most of you have noticed that it's not always easy to navigate the internal structure of the library. I'd like to address some of that and at the same time give us the tools to solve some other problems, which I'll discuss at the end. I propose to divide the library's implementation into several namespace layers with corresponding subdirectories of boost/python. These are just rough divisions and I would welcome suggestions for finer-grained layering, or better names, or... These layers are generally ordered from dependencies to dependents. core (for lack of a better name) - This is a Python-independent layer which contains the framework of the type-converter registry, inheritance.hpp which manages base<->derived class conversions, the exception translator framework, boost/python/type_id.hpp, and possibly a few components from the current boost/python/detail. converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities function - Wrapping of (member) (function) pointers into Python callable objects. Maybe this should be called "callable". callback - Invocation of Python callable objects from C++, e.g. call_method<...>, call<...> api - various namespace-scope functions such as del(), getattr(), etc. objects - object, str, dict, tuple, list, long_ ... classes - specific support for class wrapping, including instance_holders, support for smart pointers, etc. I see great potential in this reorganization. One thing I'd like to see happen in the near term is that Pyste might be altered to use some lower-level components of Boost.Python directly to improve compile times. For example, instead of relying on Boost.Python to generate wrappers for functions and member functions, Pyste might instead generate function wrappers directly using Boost.Python's type converters. I have no doubt that Python executes faster than the template engine in most C++ implementations! A longer-term goal is to move enough of the code into the core that we could easily re-use it to support other interpreted languages besides Python. Comments? -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
I propose to divide the library's implementation into several namespace layers with corresponding subdirectories of boost/python. These are just rough divisions and I would welcome suggestions for finer-grained layering, or better names, or... These layers are generally ordered from dependencies to dependents.
core (for lack of a better name) - This is a Python-independent layer which contains the framework of the type-converter registry, inheritance.hpp which manages base<->derived class conversions, the exception translator framework, boost/python/type_id.hpp, and possibly a few components from the current boost/python/detail.
converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities
function - Wrapping of (member) (function) pointers into Python callable objects. Maybe this should be called "callable".
callback - Invocation of Python callable objects from C++, e.g. call_method<...>, call<...>
api - various namespace-scope functions such as del(), getattr(), etc.
objects - object, str, dict, tuple, list, long_ ...
classes - specific support for class wrapping, including instance_holders, support for smart pointers, etc.
Looks great, but perhaps inheritance.hpp should be included in the classes namespace?
I see great potential in this reorganization. One thing I'd like to see happen in the near term is that Pyste might be altered to use some lower-level components of Boost.Python directly to improve compile times. For example, instead of relying on Boost.Python to generate wrappers for functions and member functions, Pyste might instead generate function wrappers directly using Boost.Python's type converters. I have no doubt that Python executes faster than the template engine in most C++ implementations!
I will of course be glad to implement any modification in Pyste in that direction, with Dave's help. 8)
A longer-term goal is to move enough of the code into the core that we could easily re-use it to support other interpreted languages besides Python.
Yeah, but what would we call the library then? Boost::Interpreted? ;) Regards, Nicodemus.
Nicodemus: Another thing I've noticed, that for static member functions the code generated defines them in class scope instead of global scope, causing errors. Basically, you need to change the 'PointerDeclaration' of Method class to look more like this: # If a method is static, don't need the class specifier if self.static: scope = '*' else: scope = '%s::*' % self.class_ if self.const: const = 'const' return '(%s (%s)(%s) %s)&%s' %\ (result, scope, params, const, self.FullName()) That fixes the problem. I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python. Thanks! Roman __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Roman Sulzhyk <roman_sulzhyk@yahoo.com> writes:
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
That's "overridable". The only thing about 'final' which makes me nervous is that IIUC it usually implies enforcement: "thou shalt not try to declare this name in a derived class or the system will slap you on the wrist". But I don't pretend to know Java; maybe I'm wrong? -- Dave Abrahams Boost Consulting www.boost-consulting.com
Roman Sulzhyk wrote:
Nicodemus:
Another thing I've noticed, that for static member functions the code generated defines them in class scope instead of global scope, causing errors.
Basically, you need to change the 'PointerDeclaration' of Method class to look more like this:
# If a method is static, don't need the class specifier if self.static: scope = '*' else: scope = '%s::*' % self.class_
if self.const: const = 'const' return '(%s (%s)(%s) %s)&%s' %\ (result, scope, params, const, self.FullName())
That fixes the problem.
You do not seem to be using the latest CVS, Pyste uses staticmethod to declare static member functions: struct C { static int foo() { return 0; } }; // Module ====================================================================== BOOST_PYTHON_MODULE(test) { class_< C >("C", init< >()) .def(init< const C & >()) .def("foo", &C::foo) .staticmethod("foo") ; } If you are not using the latest CVS, then I might be misunderstanding the problem, but I *stronly* suggest that you use the latest CVS from both Boost.Python and Pyste... Pyste specifically has had lots of bug fixes and some new features since the 1.30.0 release of Boost. ;)
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
I considered it too, but I thought that "no_override" was more clear. What you guys think? Regards, Nicodemus.
Nicodemus <nicodemus@globalite.com.br> writes:
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
I considered it too, but I thought that "no_override" was more clear. What you guys think?
WWJD - What Would Java Do? I think it depends on whether "final" on a Java method means you can write a new (non-virtually-dispatched) one of that name or not. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
I considered it too, but I thought that "no_override" was more clear. What you guys think?
WWJD - What Would Java Do?
I think it depends on whether "final" on a Java method means you can write a new (non-virtually-dispatched) one of that name or not.
I believe that in this context, "final" has the same meaning as in Java... I do not like it much because the meaning is not obvious from the word alone, but perhaps using a familiar term to some other programmers might be better than coming up with a new one?
Nicodemus <nicodemus@globalite.com.br> writes:
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
I considered it too, but I thought that "no_override" was more clear. What you guys think?
WWJD - What Would Java Do?
I think it depends on whether "final" on a Java method means you can write a new (non-virtually-dispatched) one of that name or not.
I believe that in this context, "final" has the same meaning as in Java... I do not like it much because the meaning is not obvious from the word alone, but perhaps using a familiar term to some other programmers might be better than coming up with a new one?
Absolutely. If that's right, I think we should go with "final". It's a sensible semantics, too. There's no reason, once that name has been "sealed off" from the overriding mechanism, not to allow people to reuse it. BTW, it seems likely that people will eventually want to do things like: # finalize any functions X inherited from base classes for b in X.bases: for f in b.member_functions: final(f) Pyste probably ought to expose a slick programmatic interface to the XML info underneath it all... or does it do that already? -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
I believe that in this context, "final" has the same meaning as in Java... I do not like it much because the meaning is not obvious from the word alone, but perhaps using a familiar term to some other programmers might be better than coming up with a new one?
Absolutely. If that's right, I think we should go with "final". It's a sensible semantics, too. There's no reason, once that name has been "sealed off" from the overriding mechanism, not to allow people to reuse it.
From "Java Programmer's SourceBook: Thinking in Java": There are two reasons for *final* methods. The first is to put a “lock” on the method to prevent any inheriting class from changing its meaning. This is done for design reasons when you want to make sure that a method’s behavior is retained during inheritance and cannot be overridden. The second reason for *final* methods is efficiency. The first definition seems to be what we intend by the mechanism. I wil change it in CVS.
BTW, it seems likely that people will eventually want to do things like:
# finalize any functions X inherited from base classes for b in X.bases: for f in b.member_functions: final(f)
Pyste probably ought to expose a slick programmatic interface to the XML info underneath it all... or does it do that already?
It does not. Currently, the headers are parsed after the user scripts have been executed, so what the user is manipulating in the scripts (ie, pyste files) is not gccxml information at all. I agree with you, this is indeed *very* nice to have, and I will put it in my todo list.
<snip>
BTW, it seems likely that people will eventually want to do things like:
# finalize any functions X inherited from base classes for b in X.bases: for f in b.member_functions: final(f)
Pyste probably ought to expose a slick programmatic interface to the XML info underneath it all... or does it do that already?
It does not. Currently, the headers are parsed after the user scripts
have been executed, so what the user is manipulating in the scripts (ie, pyste files) is not gccxml information at all. I agree with you, this is indeed *very* nice to have, and I will put it in my todo list.
Yep, I arrived to similar conclusions also - currently the mechanism is somewhat raw, it would be *very* nice to expose the parsed declarations in the scope of the pyste script, to allow people to mutate them. Talking about todo lists, another useful thing would be to be able to add a command line option to take XML file already pre-generated - that'll simplify pyste script development some, because with G++ 3.x series it takes considerable amounts of time to generate XML from C++ and hence making iterative changes is complex. Roman
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Roman Sulzhyk wrote:
Talking about todo lists, another useful thing would be to be able to add a command line option to take XML file already pre-generated - that'll simplify pyste script development some, because with G++ 3.x series it takes considerable amounts of time to generate XML from C++ and hence making iterative changes is complex.
That is a good idea. But passing individual filenames in the command line does not seem pratical, because you have to specify a xml file *per header file* that will be parsed. Perhaps a flag like "--xml-dir" where you indicate where the xml files will be? That way, before Pyste calls gccxml in the file "test.h", it checks if "test.xml" is present in the xml-dir, and use that if present, or parses it otherwise. What do you think?
Nicodemus <nicodemus@globalite.com.br> writes:
Roman Sulzhyk wrote:
Talking about todo lists, another useful thing would be to be able to add a command line option to take XML file already pre-generated - that'll simplify pyste script development some, because with G++ 3.x series it takes considerable amounts of time to generate XML from C++ and hence making iterative changes is complex.
That is a good idea. But passing individual filenames in the command line does not seem pratical, because you have to specify a xml file *per header file* that will be parsed. Perhaps a flag like "--xml-dir" where you indicate where the xml files will be? That way, before Pyste calls gccxml in the file "test.h", it checks if "test.xml" is present in the xml-dir, and use that if present, or parses it otherwise. What do you think?
What about reading the XML and producing a pickled representation, then re-reading from the XML whenever it's outdated? Then we could easily integrate it with a build system. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
Roman Sulzhyk wrote:
Talking about todo lists, another useful thing would be to be able to add a command line option to take XML file already pre-generated - that'll simplify pyste script development some, because with G++ 3.x series it takes considerable amounts of time to generate XML from C++ and hence making iterative changes is complex.
That is a good idea. But passing individual filenames in the command line does not seem pratical, because you have to specify a xml file *per header file* that will be parsed. Perhaps a flag like "--xml-dir" where you indicate where the xml files will be? That way, before Pyste calls gccxml in the file "test.h", it checks if "test.xml" is present in the xml-dir, and use that if present, or parses it otherwise. What do you think?
What about reading the XML and producing a pickled representation, then re-reading from the XML whenever it's outdated? Then we could easily integrate it with a build system.
Unforunately it is not that simple, because of header dependecies: B.h includes A.h. Class B from B.h is exported, so B.xml is generated. User adds a new method to A, and expects it to reflect in the wrapper for B, but with a simplistic approach Pyste would not be able to note that B.h is outdated. I rather let this problem to build systems already out there, like SCons: the user can easily extend it to generate gccxml files from the headers, with dependency analysis built-in.
--- Nicodemus <nicodemus@globalite.com.br> wrote:
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
Roman Sulzhyk wrote:
Talking about todo lists, another useful thing would be to be able to add a command line option to take XML file already pre-generated - that'll simplify pyste script development some, because with G++ 3.x series it takes considerable amounts of time to generate XML from C++ and hence making iterative changes is complex.
That is a good idea. But passing individual filenames in the command line does not seem pratical, because you have to specify a xml file *per header file* that will be parsed. Perhaps a flag like "--xml-dir" where you indicate where the xml files will be? That way, before Pyste calls gccxml in the file "test.h", it checks if "test.xml" is present in the xml-dir, and use that if present, or parses it otherwise. What do you think?
What about reading the XML and producing a pickled representation, then re-reading from the XML whenever it's outdated? Then we could easily integrate it with a build system.
Unforunately it is not that simple, because of header dependecies: B.h includes A.h. Class B from B.h is exported, so B.xml is generated. User adds a new method to A, and expects it to reflect in the wrapper for B, but with a simplistic approach Pyste would not be able to note that B.h is outdated. I rather let this problem to build systems already out there, like SCons: the user can easily extend it to generate gccxml files from the headers, with dependency analysis built-in.
I see, it's not that simple to make it automagic... I do think that the parsing step needs to be optionally exposed, so people can use it in their makefiles to generate xml explicitely, and pass it in on the command line, e.g. pyste --cache="{foo.h:foo.xml, faa.h:faa.xml}" or something like that. Let's make it least intrusive to pyste for now and move the responsibility to the build system.
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Roman Sulzhyk wrote:
I see, it's not that simple to make it automagic... I do think that the parsing step needs to be optionally exposed, so people can use it in their makefiles to generate xml explicitely, and pass it in on the command line, e.g.
pyste --cache="{foo.h:foo.xml, faa.h:faa.xml}"
or something like that.
What about my suggestion about --xml-dir?
--- Nicodemus <nicodemus@globalite.com.br> wrote:
Roman Sulzhyk wrote:
I see, it's not that simple to make it automagic... I do think that the parsing step needs to be optionally exposed, so people can use it in their makefiles to generate xml explicitely, and pass it in on the command line, e.g.
pyste --cache="{foo.h:foo.xml, faa.h:faa.xml}"
or something like that.
What about my suggestion about --xml-dir?
Well, directory is fine also, however in my example passing and mapping of pre-generated files to header files is explicit, hence build system is responsible for checking appropriate expirations and re-generating files as required. If pyste is to look them up implicitely in the xml-cache directory, it's harder to communicate when they become outdated. I basically approached it from the perspective that build system knows better about dependencies between files and when something needs to be refreshed. However, maybe it's an overcomplication :) Either way is fine with me.
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Roman Sulzhyk wrote:
--- Nicodemus <nicodemus@globalite.com.br> wrote:
What about my suggestion about --xml-dir?
Well, directory is fine also, however in my example passing and mapping of pre-generated files to header files is explicit, hence build system is responsible for checking appropriate expirations and re-generating files as required. If pyste is to look them up implicitely in the xml-cache directory, it's harder to communicate when they become outdated. I basically approached it from the perspective that build system knows better about dependencies between files and when something needs to be refreshed.
From my experience with SCons, it would actually simpler the other way. You make your build system generate the gccxml files and the pyste files. Whenever a header changes, the related gccxml file will be rebuilt, and consequently the pyste file will be rebuilt also. With --cache, you would have to make your build system generate the command line with the dictionary-like syntax, which I believe would be more complicated than a static command line "python pyste.py --module=foo --xml-dir=xml-cache bar.pyste bah.pyste"?
However, maybe it's an overcomplication :) Either way is fine with me.
I think --xml-dir is a better solution, unless I am missing something, in which case I would be thankful if you could enlighten me. 8)
I think --xml-dir is a better solution, unless I am missing something, in which case I would be thankful if you could enlighten me. 8)
I agree, let's do --xml-dir.
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
--- Nicodemus <nicodemus@globalite.com.br> wrote:
From my experience with SCons, it would actually simpler the other way. You make your build system generate the gccxml files and the pyste files. Whenever a header changes, the related gccxml file will be rebuilt, and consequently the pyste file will be rebuilt also. With --cache, you would have to make your build system generate the command line with the dictionary-like syntax, which I believe would be more complicated than a static command line "python pyste.py --module=foo --xml-dir=xml-cache bar.pyste bah.pyste"?
Sorry if this is a stupid suggestion (I have not used pyste ever although I find it very exciting): it seems to me what you really need a .pyste scanner, analog to the dependency scanner for .cpp files. The scanner would recursively search all files that a .pyste file depends on. Then you define an action that determines what to do if any of the dependencies has changed. The user will never see these details, but simply specify: BoostPythonExtension(target="foo", sources=["bar.pyste", "bah.pyste"]) I am guessing one could mix .pyste and manually coded .cpp files without having to do anything special: BoostPythonExtension(target="foo", sources=["bar.pyste", "bah.pyste", "custom.cpp"]) The intermediate xml files would automatically stay around just like .o files until the user runs scons --clean. Optionally combine this with Scons' Repository() feature to keep the source code trees free of derived files at all times. Ralf __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
--- Nicodemus <nicodemus@globalite.com.br> wrote:
From my experience with SCons, it would actually simpler the other way. You make your build system generate the gccxml files and the pyste files. Whenever a header changes, the related gccxml file will be rebuilt, and consequently the pyste file will be rebuilt also. With --cache, you would have to make your build system generate the command line with the dictionary-like syntax, which I believe would be more complicated than a static command line "python pyste.py --module=foo --xml-dir=xml-cache bar.pyste bah.pyste"?
Sorry if this is a stupid suggestion (I have not used pyste ever although I find it very exciting): it seems to me what you really need a .pyste scanner, analog to the dependency scanner for .cpp files. The scanner would recursively search all files that a .pyste file depends on. Then you define an action
----- Original Message ----- From: "Ralf W. Grosse-Kunstleve" <rwgk@yahoo.com> To: <c++-sig@python.org> Sent: Wednesday, June 18, 2003 1:29 AM Subject: Re: [C++-sig] Re: Pyste bug - static member functions... that
determines what to do if any of the dependencies has changed. The user will never see these details, but simply specify:
BoostPythonExtension(target="foo", sources=["bar.pyste", "bah.pyste"])
I am guessing one could mix .pyste and manually coded .cpp files without having to do anything special:
BoostPythonExtension(target="foo", sources=["bar.pyste", "bah.pyste", "custom.cpp"])
The intermediate xml files would automatically stay around just like .o files until the user runs scons --clean. Optionally combine this with Scons' Repository() feature to keep the source code trees free of derived files at all times.
Yes, you are right Ralf, thanks for the remainder! I have not used bare SCons (we use a wrapper around it at work) in some time, so this detail escaped my mind.
Nicodemus, Dave: Either one, no_override or final seem fine, doesn't matter so much. Nicodemus, you're right, I am using stock 1.3.0, so I guess the bug I've mentioned is irrelevant. Do you guys know, is there a way to get a CVS snapshot without actually using CVS, i.e. are nightly snapshots available for download somewhere, I couldn't find it on sourceforge... Thanks! Roman --- Nicodemus <nicodemus@globalite.com.br> wrote:
David Abrahams wrote:
Nicodemus <nicodemus@globalite.com.br> writes:
I also like Dave's suggestion to use the term 'final' to denote virtual functions that are not expected to be overloadable from Python.
I considered it too, but I thought that "no_override" was more clear. What you guys think?
WWJD - What Would Java Do?
I think it depends on whether "final" on a Java method means you can write a new (non-virtually-dispatched) one of that name or not.
I believe that in this context, "final" has the same meaning as in Java... I do not like it much because the meaning is not obvious from
the word alone, but perhaps using a familiar term to some other programmers might be better than coming up with a new one?
_______________________________________________ C++-sig mailing list C++-sig@python.org http://mail.python.org/mailman/listinfo/c++-sig
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
--- Roman Sulzhyk <roman_sulzhyk@yahoo.com> wrote:
Do you guys know, is there a way to get a CVS snapshot without actually using CVS, i.e. are nightly snapshots available for download somewhere, I couldn't find it on sourceforge...
In case it is useful: a full, untainted snapshot of the boost CVS tree from a few hours ago is in this file: http://cci.lbl.gov/cctbx_build/results/2003_06_17_0136/cctbx_bundle.tar.gz Subdirectory cctbx_sources/boost. Simply delete the rest. Ralf __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Nicodemus <nicodemus@globalite.com.br> writes:
I propose to divide the library's implementation into several namespace layers with corresponding subdirectories of boost/python. These are just rough divisions and I would welcome suggestions for finer-grained layering, or better names, or... These layers are generally ordered from dependencies to dependents.
core (for lack of a better name) - This is a Python-independent layer which contains the framework of the type-converter registry, inheritance.hpp which manages base<->derived class conversions, the exception translator framework, boost/python/type_id.hpp, and possibly a few components from the current boost/python/detail.
converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities
function - Wrapping of (member) (function) pointers into Python callable objects. Maybe this should be called "callable".
callback - Invocation of Python callable objects from C++, e.g. call_method<...>, call<...>
api - various namespace-scope functions such as del(), getattr(), etc.
objects - object, str, dict, tuple, list, long_ ...
classes - specific support for class wrapping, including instance_holders, support for smart pointers, etc.
Looks great, but perhaps inheritance.hpp should be included in the classes namespace?
Well, it's an interesting issue. The stuff in inheritance.[ch]pp is entirely independent of Python, so it should really be in a layer that's language-independent. Maybe the core needs to be further subdivided? -- Dave Abrahams Boost Consulting www.boost-consulting.com
converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities
As somebody who has tried to go through the library, this bit is the part I'd really like to understand. I'm sure the reorganisation would help, but I'd really love to read a short doc on how this actually all hangs together. The registry, how the templates automates the construction of the PyObject and Type, how conversions are looked up (what's that graph stuff in there?). Though brilliant, the combination of templates and preprocessor makes it really (really, really) hard going. I've read through a fair bit of c/c++ code before, and I can say that this stuff is the hardest I've ever tried to decipher - and I know it is well written, not like most of the other stuff I've looked at. Even stepping through it in the debugger is not that enlightening. I think I could use the library better and assist more in extending it if I could put all the bits together in my head.(I have in mind something like the big text file that Joel wrote for Phoenix.) Just putting in a bid for your time Dave... Cheers, Brett
Brett Calcott wrote:
converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities
As somebody who has tried to go through the library, this bit is the part I'd really like to understand. I'm sure the reorganisation would help, but I'd really love to read a short doc on how this actually all hangs together. The registry, how the templates automates the construction of the PyObject and Type, how conversions are looked up (what's that graph stuff in there?).
Though brilliant, the combination of templates and preprocessor makes it really (really, really) hard going. I've read through a fair bit of c/c++ code before, and I can say that this stuff is the hardest I've ever tried to decipher - and I know it is well written, not like most of the other stuff I've looked at. Even stepping through it in the debugger is not that enlightening.
I think I could use the library better and assist more in extending it if I could put all the bits together in my head.(I have in mind something like the big text file that Joel wrote for Phoenix.)
Which text file was that? Indeed, the preprocessor really gets in the way. Unfortunately, I had to bite the bullet too and the next Phoenix/LL code will have to be preprocessor driven.
Just putting in a bid for your time Dave...
Some implementation notes would really help. Anyway, I promised Dave to help with the reorganization. I'm not at all quite familiar with the code but I sure could help in shuffling things around into modules, to begin with. Cheers, -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net
I think I could use the library better and assist more in extending it if I could put all the bits together in my head.(I have in mind something like the big text file that Joel wrote for Phoenix.)
Which text file was that? Indeed, the preprocessor really gets in the way. Unfortunately, I had to bite the bullet too and the next Phoenix/LL code will have to be preprocessor driven.
Maybe my memory serves me badly. Before it made it to boost, there was a single text files documenting the different layers and how they fitted together. I thought of it as it stepped piece by piece through things, and why they were build that way (well, that is what I remember anyway). Maybe I misremembered the whole thing. What year is it again..? Brett
Brett Calcott wrote:
I think I could use the library better and assist more in extending it if I could put all the bits together in my head.(I have in mind something like the big text file that Joel wrote for Phoenix.)
Which text file was that? Indeed, the preprocessor really gets in the way. Unfortunately, I had to bite the bullet too and the next Phoenix/LL code will have to be preprocessor driven.
Maybe my memory serves me badly. Before it made it to boost, there was a single text files documenting the different layers and how they fitted together. I thought of it as it stepped piece by piece through things, and why they were build that way (well, that is what I remember anyway).
Maybe I misremembered the whole thing. What year is it again..?
2003? Oh yeah, I remember now. Yeh, you have a good memory! -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net
"Brett Calcott" <brett.calcott@paradise.net.nz> writes:
converter - This stuff which handles Python-specific conversion mechanics is mostly already segregated in boost/python/converter, but it could be better organized. Almost everything else in the library is built upon these capabilities
As somebody who has tried to go through the library, this bit is the part I'd really like to understand. I'm sure the reorganisation would help, but I'd really love to read a short doc on how this actually all hangs together. The registry, how the templates automates the construction of the PyObject and Type, how conversions are looked up (what's that graph stuff in there?).
The graph stuff is not in this layer, it's in the core. It implements something like this: dst_ptr = dynamic_cast(src_void_ptr, src_type_id, dst_type_id)
Though brilliant, the combination of templates and preprocessor makes it really (really, really) hard going. I've read through a fair bit of c/c++ code before, and I can say that this stuff is the hardest I've ever tried to decipher - and I know it is well written, not like most of the other stuff I've looked at. Even stepping through it in the debugger is not that enlightening.
I think I could use the library better and assist more in extending it if I could put all the bits together in my head.(I have in mind something like the big text file that Joel wrote for Phoenix.)
Just putting in a bid for your time Dave...
Well, I'd be happy to collaborate with you on an implementation document. It's not something I can afford to do by myself, in part because I don't know what questions to answer. If you would agree to have a conversation with me about it here, and from that create a RestructuredText document which we'll include in the CVS, I'd be more than happy to type out details at length. I think you might have to dig into the code a little, too. Deal? -- Dave Abrahams Boost Consulting www.boost-consulting.com
As somebody who has tried to go through the library, this bit is the part I'd really like to understand. I'm sure the reorganisation would help, but I'd really love to read a short doc on how this actually all hangs together. The registry, how the templates automates the construction of the PyObject and Type, how conversions are looked up (what's that graph stuff in there?).
The graph stuff is not in this layer, it's in the core. It implements something like this:
dst_ptr = dynamic_cast(src_void_ptr, src_type_id, dst_type_id)
I think I could use the library better and assist more in extending it
if I
could put all the bits together in my head.(I have in mind something
Ok. So I think I need to know about the core as well then. like
the big text file that Joel wrote for Phoenix.)
Just putting in a bid for your time Dave...
Well, I'd be happy to collaborate with you on an implementation document. It's not something I can afford to do by myself, in part because I don't know what questions to answer. If you would agree to have a conversation with me about it here, and from that create a RestructuredText document which we'll include in the CVS, I'd be more than happy to type out details at length. I think you might have to dig into the code a little, too. Deal?
Deal. It be a bit slow at first as I am relocating to another country in the next 10 days (NZ -> Aussie), so my internet connection will be sporadic, and I'll be busy finding a new home and settling in. I'd like to approach it as though we are trying to write a simplified version of boost.python, ignoring compiler workarounds, and not using the preproc. We'll show a coded python C extension, an equivalent in boost.python syntax, then go through the structures needed for the boost.python code to generate the equivalently operating C extension code that we initially showed. We could work through examples like the ones on this page : http://starship.python.net/crew/arcege/extwriting/pyext.html ie. python extension: ================= PyObject *python_add(self, args).... ... ... .. PyMethodDef methods[] = { { "add", MyCommand, METH_VARARGS}, {NULL, NULL}, }; void initexample() { (void)Py_InitModule("add", methods); } equivalent in boost ================== ... def("add", python_add); Explain how the boost example generated equivalent code to above using very cut-down techniques that capture the 'essence' of the boost.python implementation, but not the detail. ================= 1. The module def 2. how is the methods array created (or simulated) 3. How does argument parsing happen.. ...etc.. Does this make sense? Comments (from anyone)? Cheers, Brett
Please know that I take my own response with a grain of salt; if potential contributors really think they want the presentation you describe, who am I to argue? "Brett Calcott" <brett.calcott@paradise.net.nz> writes:
Well, I'd be happy to collaborate with you on an implementation document. It's not something I can afford to do by myself, in part because I don't know what questions to answer. If you would agree to have a conversation with me about it here, and from that create a RestructuredText document which we'll include in the CVS, I'd be more than happy to type out details at length. I think you might have to dig into the code a little, too. Deal?
Deal. It be a bit slow at first as I am relocating to another country in the next 10 days (NZ -> Aussie)
Interesting change. How come?
so my internet connection will be sporadic, and I'll be busy finding a new home and settling in.
OK, I'm patient.
I'd like to approach it as though we are trying to write a simplified version of boost.python, ignoring compiler workarounds, and not using the preproc. We'll show a coded python C extension, an equivalent in boost.python syntax, then go through the structures needed for the boost.python code to generate the equivalently operating C extension code that we initially showed.
I think that sounds like it will cover a lot of details that nobody needs to see, and in particular there are many things supported by Boost.Python (derived<->base conversions, overloading, even simple stuff like classes and static method support) that we don't know how to do in 'C' without replicating large swaths of Boost.Python functionality so the document would end up being as much about 'C' extension writing as about Boost.Python internals. I'm hoping that this document will be useful to other people who want to be involved with the project (and *me* as well), and I worry that by starting from the top level, what you're describing will fail to illuminate the architecture of Boost.Python, which IMO is the real obstacle to understanding the implementation.
We could work through examples like the ones on this page :
http://starship.python.net/crew/arcege/extwriting/pyext.html
ie.
python extension: ================= PyObject *python_add(self, args).... ... ... ..
PyMethodDef methods[] = { { "add", MyCommand, METH_VARARGS}, {NULL, NULL}, };
void initexample() { (void)Py_InitModule("add", methods); }
equivalent in boost ================== ... def("add", python_add);
Explain how the boost example generated equivalent code to above using very cut-down techniques that capture the 'essence' of the boost.python implementation, but not the detail.
a. I think you need to understand much of the detail. b. The code generated by Boost.Python isn't equivalent to the above in any trivial sense. Almost everything in traditional 'C' extension building is geared towards initializing arrays of (function) pointers and constants which the Python core treats almost as a program, interpreting elements of the array and generating Python objects (e.g. callables) from them. Boost.Python doesn't/can't take advantage of these mechanisms, so it builds the Python objects directly. I guess what I'm saying is that the mapping is not very direct so I don't see a comparison with the 'C' API extension is going to help.
================= 1. The module def 2. how is the methods array created (or simulated) 3. How does argument parsing happen.. ...etc..
Does this make sense? Comments (from anyone)?
If you are convinced this is the right way to go, I'm still happy to help and answer questions. I would prefer to do a bottom-up description, starting with the core and working outward... especially because it would help the luabind people who are interested in integration with Boost.Python. A top-down description will always be describing new things in terms of their components which are concepts nobody understands yet. I'm not going to argue too strongly with anyone who's volunteering, though! -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
b. The code generated by Boost.Python isn't equivalent to the above in any trivial sense. Almost everything in traditional 'C' extension building is geared towards initializing arrays of (function) pointers and constants which the Python core treats almost as a program, interpreting elements of the array and generating Python objects (e.g. callables) from them. Boost.Python doesn't/can't take advantage of these mechanisms, so it builds the Python objects directly. I guess what I'm saying is that the mapping is not very direct so I don't see a comparison with the 'C' API extension is going to help.
Could just be me, but I think the above is exactly the kind of thing we'd want in such a document. :) I'd be happy to pitch in with the internal documentation after my exams btw. I'll have to dig deeper into Boost.Python to examine the Py_Finalize bug anyway. Regards, Dirk Gerrits
Dirk Gerrits <dirk@gerrits.homeip.net> writes:
David Abrahams wrote:
b. The code generated by Boost.Python isn't equivalent to the above in any trivial sense. Almost everything in traditional 'C' extension building is geared towards initializing arrays of (function) pointers and constants which the Python core treats almost as a program, interpreting elements of the array and generating Python objects (e.g. callables) from them. Boost.Python doesn't/can't take advantage of these mechanisms, so it builds the Python objects directly. I guess what I'm saying is that the mapping is not very direct so I don't see a comparison with the 'C' API extension is going to help.
Could just be me, but I think the above is exactly the kind of thing we'd want in such a document. :)
I'd be happy to pitch in with the internal documentation after my exams btw. I'll have to dig deeper into Boost.Python to examine the Py_Finalize bug anyway.
I'll happily be argued down on this point by any number of volunteer contributors ;-) -- Dave Abrahams Boost Consulting www.boost-consulting.com
"David Abrahams" <dave@boost-consulting.com> wrote :
Please know that I take my own response with a grain of salt; if potential contributors really think they want the presentation you describe, who am I to argue?
Hmm. maybe finding the 'right way' might take time. You say you don't know the right questions to ask, but I don't think I do yet either.
Deal. It be a bit slow at first as I am relocating to another country in the next 10 days (NZ -> Aussie)
Interesting change. How come?
I'm giving up my paying job (programming) to go and do a PhD in Philosophy at the Australian National University. I'll be using boost.python to do some stuff for my thesis :)
I'd like to approach it as though we are trying to write a simplified version of boost.python, ignoring compiler workarounds, and not using the preproc. We'll show a coded python C extension, an equivalent in boost.python syntax, then go through the structures needed for the boost.python code to generate the equivalently operating C extension code that we initially showed.
I think that sounds like it will cover a lot of details that nobody needs to see, and in particular there are many things supported by Boost.Python (derived<->base conversions, overloading, even simple stuff like classes and static method support) that we don't know how to do in 'C' without replicating large swaths of Boost.Python functionality so the document would end up being as much about 'C' extension writing as about Boost.Python internals.
Right, but we can simply not do this stuff (at first anyway).
I'm hoping that this document will be useful to other people who want to be involved with the project (and *me* as well), and I worry that by starting from the top level, what you're describing will fail to illuminate the architecture of Boost.Python, which IMO is the real obstacle to understanding the implementation.
a. I think you need to understand much of the detail.
b. The code generated by Boost.Python isn't equivalent to the above in any trivial sense. Almost everything in traditional 'C' extension building is geared towards initializing arrays of (function) pointers and constants which the Python core treats almost as a program, interpreting elements of the array and generating Python objects (e.g. callables) from them. Boost.Python doesn't/can't take advantage of these mechanisms, so it builds the Python objects directly. I guess what I'm saying is that the mapping is not very direct so I don't see a comparison with the 'C' API extension is going to help.
Well ok. But looking at class.cpp I can see the standard python structs that you see in a C extension. So you must start with these, right?
If you are convinced this is the right way to go, I'm still happy to help and answer questions. I would prefer to do a bottom-up description, starting with the core and working outward... especially because it would help the luabind people who are interested in integration with Boost.Python. A top-down description will always be describing new things in terms of their components which are concepts nobody understands yet. I'm not going to argue too strongly with anyone who's volunteering, though!
And the advantage to starting top-down is that it gives a motivation to what you are trying to achieve. The architecture of Boost.Python comes from the motivation to use a simple syntax to generate a complex interface. 'How' makes a lot more sense when you know 'why'. Perhaps we should just start in a corner and work our way around in the most interesting direction. Why not try the following: We start a conversation here directed at exposing the architecture (rather than getting my code compiling) - I copy abstracts from the conversation into Structured Text onto a Wiki page where they can be edited and added to. Using a Wiki means we can head off on tangents, leave terms or whole sections to be defined later on, and easily include active links to code, Python C extensions, and template tutorials. All of which will be needed I think. Where is a Wiki we can use? Where shall we start :) Cheers, Brett
"Brett Calcott" <brett.calcott@paradise.net.nz> writes:
"David Abrahams" <dave@boost-consulting.com> wrote :
Please know that I take my own response with a grain of salt; if potential contributors really think they want the presentation you describe, who am I to argue?
Hmm. maybe finding the 'right way' might take time. You say you don't know the right questions to ask, but I don't think I do yet either.
I'm willing.
Deal. It be a bit slow at first as I am relocating to another country in the next 10 days (NZ -> Aussie)
Interesting change. How come?
I'm giving up my paying job (programming) to go and do a PhD in Philosophy at the Australian National University. I'll be using boost.python to do some stuff for my thesis :)
Nice!
I'd like to approach it as though we are trying to write a simplified version of boost.python, ignoring compiler workarounds, and not using the preproc. We'll show a coded python C extension, an equivalent in boost.python syntax, then go through the structures needed for the boost.python code to generate the equivalently operating C extension code that we initially showed.
I think that sounds like it will cover a lot of details that nobody needs to see, and in particular there are many things supported by Boost.Python (derived<->base conversions, overloading, even simple stuff like classes and static method support) that we don't know how to do in 'C' without replicating large swaths of Boost.Python functionality so the document would end up being as much about 'C' extension writing as about Boost.Python internals.
Right, but we can simply not do this stuff (at first anyway).
It's up to you, I guess. As long as I don't have to write working 'C' API code ;-)
Well ok. But looking at class.cpp I can see the standard python structs that you see in a C extension. So you must start with these, right?
Yep. That's a bottom-up view, though ;-)
If you are convinced this is the right way to go, I'm still happy to help and answer questions. I would prefer to do a bottom-up description, starting with the core and working outward... especially because it would help the luabind people who are interested in integration with Boost.Python. A top-down description will always be describing new things in terms of their components which are concepts nobody understands yet. I'm not going to argue too strongly with anyone who's volunteering, though!
And the advantage to starting top-down is that it gives a motivation to what you are trying to achieve. The architecture of Boost.Python comes from the motivation to use a simple syntax to generate a complex interface. 'How' makes a lot more sense when you know 'why'.
I think the tutorial shows 'why' already, but as I said, I'm willing to leave it up to you.
Perhaps we should just start in a corner and work our way around in the most interesting direction. Why not try the following: We start a conversation here directed at exposing the architecture (rather than getting my code compiling) - I copy abstracts from the conversation into Structured Text onto a Wiki page where they can be edited and added to. Using a Wiki means we can head off on tangents, leave terms or whole sections to be defined later on, and easily include active links to code, Python C extensions, and template tutorials. All of which will be needed I think.
Where is a Wiki we can use?
There is (http://www.python.org/cgi-bin/moinmoin/boost_2epython), but note that it doesn't do ReST and I much prefer CVS for this sort of thing. I hate editing in a webpage, and I like to be able to apply all the usual revision control tools. I'd happily give you Boost CVS access if you wanted to do it that way. If you have a strong preference for Wiki it's no problem, since you'll be doing most of the interaction anyway.
Where shall we start :)
What are you interested in? -- Dave Abrahams Boost Consulting www.boost-consulting.com
There is (http://www.python.org/cgi-bin/moinmoin/boost_2epython), but note that it doesn't do ReST and I much prefer CVS for this sort of thing. I hate editing in a webpage, and I like to be able to apply all the usual revision control tools. I'd happily give you Boost CVS access if you wanted to do it that way. If you have a strong preference for Wiki it's no problem, since you'll be doing most of the interaction anyway.
You're right. Editing in a webpage has all the power of windows notepad. I'll cut and paste into a document, and we can sort out CVS access.
Where shall we start :)
What are you interested in?
Let me have a think, and I'll start a new thread. I'm in the middle of packing, so it might be a few days away. I look forward to you filling my brain up :) Cheers, Brett
"Brett Calcott" <brett.calcott@paradise.net.nz> writes:
There is (http://www.python.org/cgi-bin/moinmoin/boost_2epython), but note that it doesn't do ReST and I much prefer CVS for this sort of thing. I hate editing in a webpage, and I like to be able to apply all the usual revision control tools. I'd happily give you Boost CVS access if you wanted to do it that way. If you have a strong preference for Wiki it's no problem, since you'll be doing most of the interaction anyway.
You're right. Editing in a webpage has all the power of windows notepad.
Only better: you can't resize the editor pane!
I'll cut and paste into a document, and we can sort out CVS access.
Where shall we start :)
What are you interested in?
Let me have a think, and I'll start a new thread. I'm in the middle of packing, so it might be a few days away.
I look forward to you filling my brain up :)
I think you can get a lot of good material from the luabind discussion. -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (7)
-
Brett Calcott -
David Abrahams -
Dirk Gerrits -
Joel de Guzman -
Nicodemus -
Ralf W. Grosse-Kunstleve -
Roman Sulzhyk