New Major-Release Boost.Python Development
I'd like to start work on a new major release of Boost.Python. While the library is currently well-maintained in terms of bugfixes, I get the sense that neither the original developers nor the current maintainer have the time or inclination to work on new features. I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost. To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review. Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet). I am willing to take the lead on this project; I have a number of features that exist as extensions in the boost sandbox already that would work better if they could be more fully integrated into the Boost.Python core, and I think I have the necessary understanding of the full code base to coordinate things. I'd like to save a full discussion of what features a new version would include for another thread, but I am hoping other people on the list might volunteer some time to work on aspects they have coded up elsewhere - I know many such extensions exist. So I have a few questions for anyone who's paying attention: - For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this? I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have. - For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute? Thanks! Jim Bosch
On 08/25/2011 04:59 PM, Jim Bosch wrote:
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review. Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet).
Jim, this is an interesting idea. There has been lots of general (dare I say generic ?) discussion concerning process improvements (which unfortunately most of the time diverted into tool discussions). Among the fundamental issues is a modularization of boost. I think it would be great if boost.python could follow through on its own, by becoming a separate entity. Thus, I'm fully supportive of such a move.
- For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this? I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have.
I think branching off and moving to a separate site seems a fair choice. It would be great to "come back" under the "Boost.org" umbrella once that will be possible, i.e. once the Boost.org structure allows for that. (In the sake of progress I'm refraining from voicing my personal preferences for tools or hosting sites. I'm sure I can learn and adapt to whatever gets agreed on.)
- For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute?
I'd definitely like to help. I have a wishlist of my own for improvements that I'd like to see, and which I'd be happy to share and work on. Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Hi Jim, CC to Dave. This is great news. My main interests have been stability and not increasing the memory footprint of boost.python extensions. I'm not in a position to further develop boost.python. Troy and Ravi have done a significant amount of work. I hope they will comment for themselves. I'd prefer if developments stayed under the boost umbrella, e.g. as boost/python/v3, but I don't feel very strongly about this. Ralf On Thu, Aug 25, 2011 at 1:59 PM, Jim Bosch <talljimbo@gmail.com> wrote:
I'd like to start work on a new major release of Boost.Python. While the library is currently well-maintained in terms of bugfixes, I get the sense that neither the original developers nor the current maintainer have the time or inclination to work on new features. I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review. Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet).
I am willing to take the lead on this project; I have a number of features that exist as extensions in the boost sandbox already that would work better if they could be more fully integrated into the Boost.Python core, and I think I have the necessary understanding of the full code base to coordinate things. I'd like to save a full discussion of what features a new version would include for another thread, but I am hoping other people on the list might volunteer some time to work on aspects they have coded up elsewhere - I know many such extensions exist.
So I have a few questions for anyone who's paying attention:
- For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this? I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have.
- For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute?
Thanks!
Jim Bosch
______________________________**_________________ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/**mailman/listinfo/cplusplus-sig<http://mail.python.org/mailman/listinfo/cplusplus-sig>
On 08/25/2011 04:26 PM, Ralf Grosse-Kunstleve wrote:
Hi Jim,
CC to Dave.
This is great news. My main interests have been stability and not increasing the memory footprint of boost.python extensions. I'm not in a position to further develop boost.python. Troy and Ravi have done a significant amount of work. I hope they will comment for themselves. I'd prefer if developments stayed under the boost umbrella, e.g. as boost/python/v3, but I don't feel very strongly about this.
Thanks! I'd have no problem at all calling it boost/python/v3 (in fact I'd hope to). Essentially, my concern is that v3 would fall into a muddy category between an accepted Boost library and a proposed Boost library, and I don't have a good example of how that ought to work, with regard to whether it should exist in the Boost main repository, or even whether it can even use the Boost label. And I would like to have both v2 and v3 available simultaneously as distinct libraries (the way Python 2 and Python 3 are, for instance), at least while v3 is undergoing lots of changes. I'm not sure how to fit that model into the Boost umbrella - I'm happy to do it, but I guess I'm hoping for someone to tell me how it should fit; I don't want to presume that v3 is automatically a Boost library without permission, so it seemed safer to move it outside until it could officially win back its Boost status through review. Jim
On Thu, Aug 25, 2011 at 1:59 PM, Jim Bosch <talljimbo@gmail.com <mailto:talljimbo@gmail.com>> wrote:
I'd like to start work on a new major release of Boost.Python. While the library is currently well-maintained in terms of bugfixes, I get the sense that neither the original developers nor the current maintainer have the time or inclination to work on new features. I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review. Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet).
I am willing to take the lead on this project; I have a number of features that exist as extensions in the boost sandbox already that would work better if they could be more fully integrated into the Boost.Python core, and I think I have the necessary understanding of the full code base to coordinate things. I'd like to save a full discussion of what features a new version would include for another thread, but I am hoping other people on the list might volunteer some time to work on aspects they have coded up elsewhere - I know many such extensions exist.
So I have a few questions for anyone who's paying attention:
- For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this? I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have.
- For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute?
Thanks!
Jim Bosch
_________________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org <mailto:Cplusplus-sig@python.org> http://mail.python.org/__mailman/listinfo/cplusplus-sig <http://mail.python.org/mailman/listinfo/cplusplus-sig>
_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org http://mail.python.org/mailman/listinfo/cplusplus-sig
On 08/26/2011 07:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
Two things on my list that are likely going to be somewhat disruptive are: * Support for subclassing boost.python's own metaclass. * A per-module type registry, to avoid conflicting converters in multi-module projects. Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 08/26/2011 04:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
My list includes: - Propagating constness to Python (essentially already done as an extension, but it could have a much nicer interface if I could mess with class_ itself). - Make custom registry and template-based conversions more accessible. The former may just need more documentation, but the rvalue converters in particular don't seem to have been intended as part of the public API originally, and I think they're an important part of the library. Template-based conversions are even more buried in the details - you have to specialize five or six classes to get it working. I'd like to make it possible to create a template-based conversion by (partial) specializing just one traits class. - Automatic conversions for newer boost libraries (Fusion, Pointer Container, and Filesystem are at the top of my list), and more for the STL and iostreams standard libraries. I'd like to integrate the indexing suite (v2) into Boost.Python proper. - Allow Boost.Python wrapped classes to inherit from Python classes. - An actual "boost.python" Python module to make exceptions and other types used in wrappers easily accessible from Python. - Some limited degree of priority-based overload matching. Not sure how best to approach this one yet, though. Jim
The top of my list is improved interface to numpy. I know there is work going on in the form of ndarray, which seems promising.
On 08/26/2011 04:47 PM, Neal Becker wrote:
The top of my list is improved interface to numpy. I know there is work going on in the form of ndarray, which seems promising.
I'm still hesitant to consider ndarray part of Boost.Python; it's really a separate library, and I think providing a full multidimensional array class to be outside the scope of what Boost.Python should be. But I do intend to keep developing ndarray in parallel with Boost.Python, and some code may well flow from ndarray to Boost.Python. This summer, Stefan Seefeld has been mentoring Ankit Daftery on a GSOC project to clean up the low-level Boost.Python Numpy interface in the Boost sandbox (which began life as part of ndarray). While it may not be an official part of Boost.Python or Boost (it may be submitted as a separate library) immediately, that will provide an improved interface to Numpy on a much shorter timescale than the larger Boost.Python upgrades I'm thinking of for v3. Ultimately I would like to incorporate this low-level Numpy support as an optional component in the mainline Boost.Python v3, but I don't want to tie Boost.Python users to a particular C++ array class, even my own. Jim
On 08/26/2011 08:21 PM, Jim Bosch wrote:
On 08/26/2011 04:47 PM, Neal Becker wrote:
The top of my list is improved interface to numpy. I know there is work going on in the form of ndarray, which seems promising.
I'm still hesitant to consider ndarray part of Boost.Python; it's really a separate library, and I think providing a full multidimensional array class to be outside the scope of what Boost.Python should be. But I do intend to keep developing ndarray in parallel with Boost.Python, and some code may well flow from ndarray to Boost.Python.
This summer, Stefan Seefeld has been mentoring Ankit Daftery on a GSOC project to clean up the low-level Boost.Python Numpy interface in the Boost sandbox (which began life as part of ndarray). While it may not be an official part of Boost.Python or Boost (it may be submitted as a separate library) immediately, that will provide an improved interface to Numpy on a much shorter timescale than the larger Boost.Python upgrades I'm thinking of for v3.
Ultimately I would like to incorporate this low-level Numpy support as an optional component in the mainline Boost.Python v3, but I don't want to tie Boost.Python users to a particular C++ array class, even my own.
I really think that this should be a separate module (right now we call it tentatively "boost.numpy"). It introduces additional dependencies which not all boost.python users may be interested in. And yes, I had planned that we could address enough of the remaining issues to be able to submit boost.numpy for formal review within the next couple of months, i.e. ideally even before the end of this year. FWIW, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 25 Aug 2011 at 13:59, Jim Bosch wrote:
- For other Boost.Python experts on this list: do you have existing code or development time you'd like to contribute?
Firstly, I must commend you as you're a better man than I for initiating this. I mostly chase money these past few years, and I don't like myself for it. About a year ago I promised someone in here I'd refactor the BPL interface to native code to support calling a list of functors every time BPL goes in or out of native code. This would allow the Python GIL to be released or reacquired, or indeed many other useful things e.g. benchmarking. At the time I had a free window of about a month in the near future, so I thought my promise worth giving. Unfortunately, the project I was working on at the time had unexpected problems, and the month got gobbled, so I broke my promise. I won't make that same mistake this time, not least because I have a packed schedule for the upcoming year. I might also add that the number of people willing to fund development in BPL appears to have dropped off a cliff since 2008, so that doesn't help focus minds either. Maybe the release of C++1x might reinvigorate things hopefully. Nevertheless, I'd like to keep informed. If you put it on github I'll stick a watch on it :) BTW, how has the Python3k port worked out? I'm not sure if it's been mainlined yet has it? Best of luck! Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.
On 08/26/2011 01:28 PM, Niall Douglas wrote:
BTW, how has the Python3k port worked out? I'm not sure if it's been mainlined yet has it? Best of luck! Niall
I'm not using Python 3k myself, so I can't comment, but the P3K port most definitely went into trunk and has been part of the last couple of releases. Stefan -- ...ich hab' noch einen Koffer in Berlin...
Trying to catch up here, so responding to everything all at once. on Thu Aug 25 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote: Just how tall are you, Jimbo?
I'd like to start work on a new major release of Boost.Python.
That certainly is welcome news.
While the library is currently well-maintained in terms of bugfixes, I get the sense that neither the original developers nor the current maintainer have the time or inclination to work on new features.
Well, speaking for myself, mostly time. I'd be inclined to do a rewrite along the lines of the langbinding ideas if I had time. Another thing I think should be examined and refreshed is the documentation style.
I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.
There's no need to do this "outside of Boost." A branch in the Boost repository is a perfect place for exploratory development that will eventually be released as part of Boost.
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review.
If you want to go through another review process, that's up to you. Getting review feedback definitely has its advantages. Please, though, use the sandbox or some other area of the Boost svn repository at least until we get my hoped-for Git transition completed.
Perhaps the external site would continue on as the home of more fine-grained releases, or maybe we would fully reintegrate with Boost at that point (especially if Boost addresses some of its own project management and release control issues by that point, which I know is being discussed but to my knowledge doesn't really have a timeline yet).
I am willing to take the lead on this project;
Someone willing to take the lead is the most important part.
I have a number of features that exist as extensions in the boost sandbox already that would work better if they could be more fully integrated into the Boost.Python core,
Good. Although, if I were you I would also carefully re-examine the core to see if it has the best design.
and I think I have the necessary understanding of the full code base to coordinate things.
Awesome.
I'd like to save a full discussion of what features a new version would include for another thread, but I am hoping other people on the list might volunteer some time to work on aspects they have coded up elsewhere - I know many such extensions exist.
So I have a few questions for anyone who's paying attention:
- For the original Boost.Python developers and current maintainers, and other people familiar with developing Boost libraries: do you have any preference on how to approach this? I don't want to step on any toes, especially toes attached to people who are responsible for the excellent library we already have.
See above. No toe-stepping concerns on my end. I have a preference that you build on the ideas we came up with for Boost.Langbinding, but certainly wouldn't insist on it. on Thu Aug 25 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:
Jim,
this is an interesting idea. There has been lots of general (dare I say generic ?) discussion concerning process improvements (which unfortunately most of the time diverted into tool discussions). Among the fundamental issues is a modularization of boost. I think it would be great if boost.python could follow through on its own, by becoming a separate entity.
Separate from Boost? I guess that's a possibility but I'm not sure I see the advantage. on Thu Aug 25 2011, Ralf Grosse-Kunstleve <rwgrosse-kunstleve-AT-lbl.gov> wrote:
Hi Jim,
CC to Dave.
This is great news. My main interests have been stability and not increasing the memory footprint of boost.python extensions. I'm not in a position to further develop boost.python. Troy and Ravi have done a significant amount of work.
Yes, and hopefully integrating that could be part of any next steps.
I hope they will comment for themselves.
I'd prefer if developments stayed under the boost umbrella, e.g. as boost/python/v3, but I don't feel very strongly about this.
Me too. We managed a transition from v1 to v2 within Boost, and I think we could do the same for v3.
On 08/26/2011 07:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
Two things on my list that are likely going to be somewhat disruptive are:
* Support for subclassing boost.python's own metaclass.
Cool.
* A per-module type registry, to avoid conflicting converters in multi-module projects.
Interesting idea. How does sharing types across multiple modules work in that scenario? on Fri Aug 26 2011, "Niall Douglas" <s_sourceforge-AT-nedprod.com> wrote:
About a year ago I promised someone in here I'd refactor the BPL interface to native code to support calling a list of functors every time BPL goes in or out of native code. This would allow the Python GIL to be released or reacquired, or indeed many other useful things e.g. benchmarking.
Sounds useful. on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
On 08/26/2011 04:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
My list includes:
- Propagating constness to Python (essentially already done as an extension, but it could have a much nicer interface if I could mess with class_ itself).
Oooh, now that one is tricky. I'd like to see the design you have in mind. Python doesn't have constness; it has immutability, which is subtly different.
- Make custom registry and template-based conversions more accessible.
+1
The former may just need more documentation, but the rvalue converters in particular don't seem to have been intended as part of the public API originally, and I think they're an important part of the library. Template-based conversions are even more buried in the details - you have to specialize five or six classes to get it working. I'd like to make it possible to create a template-based conversion by (partial) specializing just one traits class.
Hmm, well, IIRC anything you do by pure partial specialization does not go in the registry and can't participate in cross-module communication, as it's entirely static. I guess we need a more formal separation of the two basic techniques for conversion, not to mention documentation, so people know to what they're opting in.
- Automatic conversions for newer boost libraries (Fusion, Pointer Container, and Filesystem are at the top of my list), and more for the STL and iostreams standard libraries. I'd like to integrate the indexing suite (v2) into Boost.Python proper.
Interesting and useful improvements.
- Allow Boost.Python wrapped classes to inherit from Python classes.
+1
- An actual "boost.python" Python module to make exceptions and other types used in wrappers easily accessible from Python.
Nice.
- Some limited degree of priority-based overload matching. Not sure how best to approach this one yet, though.
+1 This is a solved problem... just not in Boost.Python. Daniel Wallin worked it out for luabind and we were going to incorporate it into langbinding. Happy to discuss it further. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
In the interest of keeping this discussion easy-to-follow, I'm going to reply to Dave's email twice, with new subjects - I'll stick to questions about logistics in this email, and talk about features and scope in another. In summary, I'm getting the sense that a branch in the mainline (not sandbox) Boost SVN is the way to go. I imagine most communication would just happen on the C++-sig list, maybe with the [Boost.Python v3] subtitle I've used in this email. On 08/26/2011 01:09 PM, Dave Abrahams wrote:
Trying to catch up here, so responding to everything all at once.
on Thu Aug 25 2011, Jim Bosch<talljimbo-AT-gmail.com> wrote:
Just how tall are you, Jimbo?
6'4". Not that tall, in the grand scheme of things, but it was a teenage internet moniker that stuck with me.
I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.
There's no need to do this "outside of Boost." A branch in the Boost repository is a perfect place for exploratory development that will eventually be released as part of Boost.
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review.
If you want to go through another review process, that's up to you. Getting review feedback definitely has its advantages. Please, though, use the sandbox or some other area of the Boost svn repository at least until we get my hoped-for Git transition completed.
I'd love to see a Git transition too, but I'm actually more familiar with SVN at the moment, and I can certainly see the advantages of working in the same repository as the previous version.
on Thu Aug 25 2011, Stefan Seefeld<stefan-AT-seefeld.name> wrote:
Jim,
this is an interesting idea. There has been lots of general (dare I say generic ?) discussion concerning process improvements (which unfortunately most of the time diverted into tool discussions). Among the fundamental issues is a modularization of boost. I think it would be great if boost.python could follow through on its own, by becoming a separate entity.
Separate from Boost? I guess that's a possibility but I'm not sure I see the advantage.
From my perspective, the advantages are mostly just the same reasons Boost has been talking about increased modularity, with regard to having stable and development versions for some packages and not for others, and to allow users to be able to install some components without installing all of them. Boost.Python (right now, at least) depends on a very small core part of Boost, and to my knowledge no other Boost libraries have real dependencies on Boost.Python (not counting optional dependencies, like the Boost.Graph python bindings). If/when Boost as a whole goes more modular, I think any advantage of being separate would disappear, and the ideal case for Boost.Python would be for that to happen.
on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov> wrote:
Troy and Ravi have done a significant amount of work.
Yes, and hopefully integrating that could be part of any next steps.
I hope they will comment for themselves.
I'd prefer if developments stayed under the boost umbrella, e.g. as boost/python/v3, but I don't feel very strongly about this.
Me too. We managed a transition from v1 to v2 within Boost, and I think we could do the same for v3.
Good to hear. I wasn't using Boost.Python back when that happened, but it's nice to know there's a precedent. Thanks! Jim
on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
In the interest of keeping this discussion easy-to-follow, I'm going to reply to Dave's email twice, with new subjects - I'll stick to questions about logistics in this email, and talk about features and scope in another.
In summary, I'm getting the sense that a branch in the mainline (not sandbox) Boost SVN is the way to go.
It's all the same repository. Working in the sandbox is essentially equivalent to working in a branch of trunk; it's just a matter of where the code lives in the SVN directory hierarchy.
I imagine most communication would just happen on the C++-sig list, maybe with the [Boost.Python v3] subtitle I've used in this email.
Sure.
On 08/26/2011 01:09 PM, Dave Abrahams wrote:
Trying to catch up here, so responding to everything all at once.
on Thu Aug 25 2011, Jim Bosch<talljimbo-AT-gmail.com> wrote:
Just how tall are you, Jimbo?
6'4". Not that tall, in the grand scheme of things, but it was a teenage internet moniker that stuck with me.
Oh, then I should change my email to bigfatdave@somewhere.com
I'd also like to propose some changes that are slightly backwards-incompatible, as well as some that mess with the internals to an extent that I'd feel better about doing it outside Boost itself, to make it easier for adventurous users to play with the new version without affecting people who depend on having an extremely stable library in Boost.
There's no need to do this "outside of Boost." A branch in the Boost repository is a perfect place for exploratory development that will eventually be released as part of Boost.
To that end, I'm inclined to copy the library to somewhere else (possibly the boost sandbox, but more likely a separate site), work on it, produce some minor releases, and re-submit it to Boost for review.
If you want to go through another review process, that's up to you. Getting review feedback definitely has its advantages. Please, though, use the sandbox or some other area of the Boost svn repository at least until we get my hoped-for Git transition completed.
I'd love to see a Git transition too, but I'm actually more familiar with SVN at the moment, and I can certainly see the advantages of working in the same repository as the previous version.
on Thu Aug 25 2011, Stefan Seefeld<stefan-AT-seefeld.name> wrote:
Jim,
this is an interesting idea. There has been lots of general (dare I say generic ?) discussion concerning process improvements (which unfortunately most of the time diverted into tool discussions). Among the fundamental issues is a modularization of boost. I think it would be great if boost.python could follow through on its own, by becoming a separate entity.
Separate from Boost? I guess that's a possibility but I'm not sure I see the advantage.
From my perspective, the advantages are mostly just the same reasons Boost has been talking about increased modularity, with regard to having stable and development versions for some packages and not for others, and to allow users to be able to install some components without installing all of them. Boost.Python (right now, at least) depends on a very small core part of Boost, and to my knowledge no other Boost libraries have real dependencies on Boost.Python (not counting optional dependencies, like the Boost.Graph python bindings). If/when Boost as a whole goes more modular, I think any advantage of being separate would disappear, and the ideal case for Boost.Python would be for that to happen.
In that case, if I were you, I would actually start using Git with the modularized / CMake-ified Boost at http://github.com/boost-lib. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 27 Aug 2011 at 12:29, Dave Abrahams wrote:
In that case, if I were you, I would actually start using Git with the modularized / CMake-ified Boost at http://github.com/boost-lib.
If you do go for git, I have found repo embedded per-branch issue tracking (e.g. http://bugseverywhere.org/) to be a god send for productivity because you can raise issues with your own code without bothering the mainline issue tracker about branch specific (and indeed often personal) issues. It has made as much difference to my productivity as adopting git did because I no longer need to keep (and often misplace) post it notes reminding me of things to do. I even coded up a GUI for it for the TortoiseXXX family of revision tracking GUIs which you can find at http://www.nedprod.com/programs/Win32/BEurtle/. This lets you mark off BE issue fixes with GIT/whatever commits. HTH, Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.
on Sun Aug 28 2011, "Niall Douglas" <s_sourceforge-AT-nedprod.com> wrote:
On 27 Aug 2011 at 12:29, Dave Abrahams wrote:
In that case, if I were you, I would actually start using Git with the modularized / CMake-ified Boost at http://github.com/boost-lib.
If you do go for git, I have found repo embedded per-branch issue tracking (e.g. http://bugseverywhere.org/) to be a god send for productivity because you can raise issues with your own code without bothering the mainline issue tracker about branch specific (and indeed often personal) issues. It has made as much difference to my productivity as adopting git did because I no longer need to keep (and often misplace) post it notes reminding me of things to do.
Last time I looked at BE it was not really ready for primetime. You're motivating me to check it out again. *checks it out* Hmm... not sure this will get beyond the personal, though: it lacks so much of what people have come to expect, like hyperlinking, and especially an *interactive* web interface for bug creation.
I even coded up a GUI for it for the TortoiseXXX family of revision tracking GUIs which you can find at http://www.nedprod.com/programs/Win32/BEurtle/. This lets you mark off BE issue fixes with GIT/whatever commits.
Cool! Maybe with your GUI it would be enough to get people to use it. Still needs a web interface for bug creation. I'm going to have to keep a closer eye on BE. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 3 Oct 2011 at 8:34, Dave Abrahams wrote:
Last time I looked at BE it was not really ready for primetime. You're motivating me to check it out again.
*checks it out*
Hmm... not sure this will get beyond the personal, though: it lacks so much of what people have come to expect, like hyperlinking, and especially an *interactive* web interface for bug creation.
I agree it isn't fit to go beyond the personal. I'd even say, actually, that when you go beyond bug tracking within a (very) small group it's time for Redmine or Trac. I'd also say personal bug trackers are much more of a design/development tool than a deployment/support tool. They're really a type of post-it note tied into the repository. As I know you know Dave, you also often get emails containing good ideas for some software project of yours, but right now you don't have the time to really look into them. Here BE also excels - you dump out the email and attach it to a BE feature issue. That prevents you forgetting about them, or much later on wasting time searching your email not quite remembering the correct search terms.
I even coded up a GUI for it for the TortoiseXXX family of revision tracking GUIs which you can find at http://www.nedprod.com/programs/Win32/BEurtle/. This lets you mark off BE issue fixes with GIT/whatever commits.
Cool! Maybe with your GUI it would be enough to get people to use it. Still needs a web interface for bug creation. I'm going to have to keep a closer eye on BE.
I develop for Windows first and POSIX later, so something which integrates into the Tortoise family of SCM GUIs is much more important to me than a web interface. I really, really wish KDE would integrate proper multi-SCM support into itself too. I agree though that a JSON-RPC interface to BE would be a very useful addition, not just to aid BEurtle's performance, and of course from that a dynamic web interface becomes much easier. My thread regarding the topic (http://void.printf.net/pipermail/be-devel/2011- September/thread.html) went nowhere though. Again, I might add the support myself for the next BEurtle release and upload it as a fork to pypi as BE's author is unwilling to permit BE to go onto pypi. [BTW to anyone interested, my GUI needs a new release as several show stopping bugs have since been found. Hopefully I'll get a chance before Christmas, meanwhile GIT HEAD has already fixed the worst of them] Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.
on Mon Oct 03 2011, "Niall Douglas" <s_sourceforge-AT-nedprod.com> wrote:
On 3 Oct 2011 at 8:34, Dave Abrahams wrote:
Last time I looked at BE it was not really ready for primetime. You're motivating me to check it out again.
*checks it out*
Hmm... not sure this will get beyond the personal, though: it lacks so much of what people have come to expect, like hyperlinking, and especially an *interactive* web interface for bug creation.
I agree it isn't fit to go beyond the personal. I'd even say, actually, that when you go beyond bug tracking within a (very) small group it's time for Redmine or Trac.
I'd also say personal bug trackers are much more of a design/development tool than a deployment/support tool. They're really a type of post-it note tied into the repository.
I'd like to try it. But now, suppose you have a project on github and you want to make it easy for people to report bugs. Clearly you need to enable the GitHub issue tracker, right? How do you integrate that with your be workflow?
As I know you know Dave, you also often get emails containing good ideas for some software project of yours, but right now you don't have the time to really look into them. Here BE also excels - you dump out the email and attach it to a BE feature issue. That prevents you forgetting about them, or much later on wasting time searching your email not quite remembering the correct search terms.
Yeah, I can see the appeal.
I even coded up a GUI for it for the TortoiseXXX family of revision tracking GUIs which you can find at http://www.nedprod.com/programs/Win32/BEurtle/. This lets you mark off BE issue fixes with GIT/whatever commits.
Cool! Maybe with your GUI it would be enough to get people to use it. Still needs a web interface for bug creation. I'm going to have to keep a closer eye on BE.
I develop for Windows first and POSIX later, so something which integrates into the Tortoise family of SCM GUIs is much more important to me than a web interface. I really, really wish KDE would integrate proper multi-SCM support into itself too.
I agree though that a JSON-RPC interface to BE would be a very useful addition, not just to aid BEurtle's performance, and of course from that a dynamic web interface becomes much easier. My thread regarding the topic (http://void.printf.net/pipermail/be-devel/2011- September/thread.html) went nowhere though. Again, I might add the support myself for the next BEurtle release and upload it as a fork to pypi as BE's author is unwilling to permit BE to go onto pypi.
Seriously? That's not a very good sign. Why not? I found installing BE to be a bit of a struggle. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 3 Oct 2011 at 11:13, Dave Abrahams wrote:
I'd also say personal bug trackers are much more of a design/development tool than a deployment/support tool. They're really a type of post-it note tied into the repository.
I'd like to try it. But now, suppose you have a project on github and you want to make it easy for people to report bugs. Clearly you need to enable the GitHub issue tracker, right? How do you integrate that with your be workflow?
That problem, for me personally at least, comes up infrequently enough that copying & pasting is sufficient for now. I absolutely agree that it would be simply great that should say a github bug be assigned to dabrahams on github then it automagically gets pushed into your personal branch. You'd probably need a custom tracker plugin for github though given github's design (github is weird and doesn't appear to export its issues as RSS). Redmine and Trac are much more accommodating. A post-fetch hook in git could pull the RSS bug feed, scan it for things referencing you and push them into your personal BE easily enough once it had a JSON-RPC interface.
I agree though that a JSON-RPC interface to BE would be a very useful addition, not just to aid BEurtle's performance, and of course from that a dynamic web interface becomes much easier. My thread regarding the topic (http://void.printf.net/pipermail/be-devel/2011- September/thread.html) went nowhere though. Again, I might add the support myself for the next BEurtle release and upload it as a fork to pypi as BE's author is unwilling to permit BE to go onto pypi.
Seriously? That's not a very good sign. Why not? I found installing BE to be a bit of a struggle.
I agree :) not helped by unspecified dependencies. BE's author feels that BE ought to be supplied as a vendor distro package e.g. .deb for debian, .rpm for Fedora etc. He feels that anything which parallels that is unhelpful. He did mention that once BE is ported to Python3k he'd support using distribute.py. In http://void.printf.net/pipermail/be-devel/2011- September/000839.html I pointed out that Linux still occupies a small minority of developer eyeballs. Windows still is the overwhelming majority, and pypi is the best you have there. I argued for a pypi based windows distro especially as it makes dealing with Windows Installer easier (I hope you have never had to deal with Windows Installer. It sucks, even with WiX to hand). In http://void.printf.net/pipermail/be-devel/2011- September/000845.html I asked for permission to upload a windows fork of BE fixing several show stopping Windows-specific bugs to pypi but there was no answer. I'll give it a while longer out of grace. I might make a new BEurtle release a Christmas holiday personal project :) I originally wrote nedmalloc that way too. Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.
To summarize how I'm approaching this now: - I think the cmake/github Boost repositories look like the right place to put this. - I'm going to spend some time digesting the langbinding and luabind code, and try to put together a skeleton package; when it gets mature enough for others to find it useful to look at I'll upload it to github. - In the meantime, I'll be re-starting some discussions on the list to flesh out some aspects that I don't have a clear vision for yet. Jim
On 08/26/2011 01:09 PM, Dave Abrahams wrote:
Well, speaking for myself, mostly time. I'd be inclined to do a rewrite along the lines of the langbinding ideas if I had time.
I had only been vaguely aware of langbinding until I followed up on your last email. It's a very nice separation, though after bad experiences using SWIG I am a little wary about trying to build a one-size-fits-all front-end for different languages. It seems like a reasonable way to proceed would be to try to convert Boost.Python to the langbinding interface, but not be obsessive about avoiding Python-specific hacks in the frontend right now. Once we're more feature-complete in Python, we can worry about finding language-agnostic ways to do things that weren't anticipated in the original langbinding design.
Another thing I think should be examined and refreshed is the documentation style.
Agreed. And I wouldn't just limit it to the official documentation; there are a lot of little tidbits of useful but often very old Boost.Python knowledge scattered around the internet (often on Python-affiliated sites or wikis). It'd be nice to unify a lot of that, or at least update the obsolete stuff and add links to a more complete set of official documentation.
Good. Although, if I were you I would also carefully re-examine the core to see if it has the best design.
I wasn't originally planning on doing a full re-evaluation, but after looking over the langbinding proposal and hearing some of the other ideas, I think that will probably be necessary.
on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov> wrote: <snip>
Troy and Ravi have done a significant amount of work.
Yes, and hopefully integrating that could be part of any next steps.
I hope they will comment for themselves.
I was under the impression their work had already been integrated into Boost.Python v2. Is there a large repository of additional Boost.Python work elsewhere that I should be aware of? I am aware of Py++ and the extensions associated with it, and some of that could definitely go into Boost.Python proper (and I think I've heard Roman state that he wouldn't have a problem with someone else doing the work to make it so, and he just didn't have time to do it himself).
* Support for subclassing boost.python's own metaclass.
Cool.
* A per-module type registry, to avoid conflicting converters in multi-module projects.
Interesting idea. How does sharing types across multiple modules work in that scenario?
Hmm. I'm guessing the global type registry would still be the default, and per-module registries would override these when available? It sounds like Stefan has a clear use case in mind, and I'd be curious to know what it is.
on Fri Aug 26 2011, "Niall Douglas"<s_sourceforge-AT-nedprod.com> wrote:
About a year ago I promised someone in here I'd refactor the BPL interface to native code to support calling a list of functors every time BPL goes in or out of native code. This would allow the Python GIL to be released or reacquired, or indeed many other useful things e.g. benchmarking.
Sounds useful.
This is one of the areas - GIL, threading, etc. - where I'm less knowledgeable, especially when multiple platforms are involved. It sounds like a very useful feature for a lot of people, but I have to admit I probably wouldn't dive into this without a lot of backup.
on Fri Aug 26 2011, Jim Bosch<talljimbo-AT-gmail.com> wrote:
On 08/26/2011 04:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
My list includes:
- Propagating constness to Python (essentially already done as an extension, but it could have a much nicer interface if I could mess with class_ itself).
Oooh, now that one is tricky. I'd like to see the design you have in mind. Python doesn't have constness; it has immutability, which is subtly different.
Essentially, C++ objects returned as const references, pointers, or smart pointers get converted into a Python proxy object with methods that forward to the real wrapped object, but only if those methods are marked as const. The proxy objects have rvalue converters but no lvalue converters. I essentially wrote a wrapper around class_ to check for constness of member functions and make the proxy class. When you use class_ directly, there's no const-correctness, but you can do: const_aware(class_<T>("class_name")) .def(...) etc., to get const-correct wrappers. It doesn't work at all well with visitors, though, and I think I could find a better syntax with the ability to mess with class_ itself. Most of the current design can be found in the boost sandbox, in the python_extensions package.
- Make custom registry and template-based conversions more accessible.
+1
The former may just need more documentation, but the rvalue converters in particular don't seem to have been intended as part of the public API originally, and I think they're an important part of the library. Template-based conversions are even more buried in the details - you have to specialize five or six classes to get it working. I'd like to make it possible to create a template-based conversion by (partial) specializing just one traits class.
Hmm, well, IIRC anything you do by pure partial specialization does not go in the registry and can't participate in cross-module communication, as it's entirely static. I guess we need a more formal separation of the two basic techniques for conversion, not to mention documentation, so people know to what they're opting in.
Yeah, that's exactly right. To get it to work cross-module you have to include a header with the specializations in all modules (and worse, in all wrapper source files in those modules). It makes things a little too magical for my taste, but I don't see a way around that, and template-based converters are really helpful in wrapping array-heavy numerical code. Documentation is indeed a big part of this.
- Automatic conversions for newer boost libraries (Fusion, Pointer Container, and Filesystem are at the top of my list), and more for the STL and iostreams standard libraries. I'd like to integrate the indexing suite (v2) into Boost.Python proper.
Interesting and useful improvements.
- Allow Boost.Python wrapped classes to inherit from Python classes.
+1
- An actual "boost.python" Python module to make exceptions and other types used in wrappers easily accessible from Python.
Nice.
- Some limited degree of priority-based overload matching. Not sure how best to approach this one yet, though.
+1 This is a solved problem... just not in Boost.Python. Daniel Wallin worked it out for luabind and we were going to incorporate it into langbinding. Happy to discuss it further.
That's great to hear. I'll have to spend some time looking at luabind. I'm sure I'll have more questions later. Thanks! Jim
on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:
On 08/26/2011 01:09 PM, Dave Abrahams wrote:
Well, speaking for myself, mostly time. I'd be inclined to do a rewrite along the lines of the langbinding ideas if I had time.
I had only been vaguely aware of langbinding until I followed up on your last email. It's a very nice separation, though after bad experiences using SWIG I am a little wary about trying to build a one-size-fits-all front-end for different languages.
It seems like a reasonable way to proceed would be to try to convert Boost.Python to the langbinding interface, but not be obsessive about avoiding Python-specific hacks in the frontend right now.
Absolutely perfect; that's what I was thinking.
Once we're more feature-complete in Python, we can worry about finding language-agnostic ways to do things that weren't anticipated in the original langbinding design.
Another thing I think should be examined and refreshed is the documentation style.
Agreed. And I wouldn't just limit it to the official documentation; there are a lot of little tidbits of useful but often very old Boost.Python knowledge scattered around the internet (often on Python-affiliated sites or wikis). It'd be nice to unify a lot of that, or at least update the obsolete stuff and add links to a more complete set of official documentation.
Sure.
Good. Although, if I were you I would also carefully re-examine the core to see if it has the best design.
I wasn't originally planning on doing a full re-evaluation, but after looking over the langbinding proposal and hearing some of the other ideas, I think that will probably be necessary.
Happy to discuss with you anywhere you get stuck. By the way, there's actually a bunch of langbinding code checked into the SVN repo somewhere. Ah, here: http://svn.boost.org/svn/boost/sandbox/langbinding/
on Thu Aug 25 2011, Ralf Grosse-Kunstleve<rwgrosse-kunstleve-AT-lbl.gov> wrote: <snip>
Troy and Ravi have done a significant amount of work.
Yes, and hopefully integrating that could be part of any next steps.
I hope they will comment for themselves.
I was under the impression their work had already been integrated into Boost.Python v2. Is there a large repository of additional Boost.Python work elsewhere that I should be aware of?
It's my impression that the integration was stalled due to Ralf's concerns about object code size. But I could be wrong.
Interesting idea. How does sharing types across multiple modules work in that scenario?
Hmm. I'm guessing the global type registry would still be the default, and per-module registries would override these when available? It sounds like Stefan has a clear use case in mind, and I'd be curious to know what it is.
Me too.
on Fri Aug 26 2011, Jim Bosch<talljimbo-AT-gmail.com> wrote:
On 08/26/2011 04:17 AM, Neal Becker wrote:
What sort of improvements did you have in mind?
My list includes:
- Propagating constness to Python (essentially already done as an extension, but it could have a much nicer interface if I could mess with class_ itself).
Oooh, now that one is tricky. I'd like to see the design you have in mind. Python doesn't have constness; it has immutability, which is subtly different.
Essentially, C++ objects returned as const references, pointers, or smart pointers get converted into a Python proxy object with methods that forward to the real wrapped object, but only if those methods are marked as const. The proxy objects have rvalue converters but no lvalue converters.
I like it! -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 08/27/2011 04:40 PM, Dave Abrahams wrote:
Hmm. I'm guessing the global type registry would still be the default, and per-module registries would override these when available? It sounds like Stefan has a clear use case in mind, and I'd be curious to know what it is. Me too.
I believe what we were discussing at the time was a situation where an extension module would not only define converters for its own types, but also common types that may are required by the API. This could in particular include common types from libstdc++. If multiple extension modules do this, than a Python program that happens to use them in the same application may end up with undefined behavior (does this constitute an ODR violation under the hood ?) To make this work, the common type converters need to be factored out and shared. This in itself is impractical (since the original authors may not be aware of this need). Furthermore, a converter may require module-specific behavior, i.e. converting a std::string in the context of one module may be different from the desired conversion in another. Binding converters to particular modules (and requiring to explicitly import / exporting converters) seems like a solution to the above. Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 08/27/2011 02:08 PM, Stefan Seefeld wrote:
On 08/27/2011 04:40 PM, Dave Abrahams wrote:
Hmm. I'm guessing the global type registry would still be the default, and per-module registries would override these when available? It sounds like Stefan has a clear use case in mind, and I'd be curious to know what it is. Me too.
I believe what we were discussing at the time was a situation where an extension module would not only define converters for its own types, but also common types that may are required by the API. This could in particular include common types from libstdc++.
If multiple extension modules do this, than a Python program that happens to use them in the same application may end up with undefined behavior (does this constitute an ODR violation under the hood ?)
To make this work, the common type converters need to be factored out and shared. This in itself is impractical (since the original authors may not be aware of this need). Furthermore, a converter may require module-specific behavior, i.e. converting a std::string in the context of one module may be different from the desired conversion in another.
Binding converters to particular modules (and requiring to explicitly import / exporting converters) seems like a solution to the above.
That's definitely a problem that needs to be addressed. I've encountered it too. I hope to provide more libstd++ conversions in Boost.Python itself, which should alleviate the need for this a bit, but it does need a proper solution. To solve it, I think you'd want anything registered by a specific module to appear both in that module's registry and the global registry, with the module's registry taking precedence. Are there any cases where you'd want something only to appear in the module-specific registry? Jim
On 08/28/2011 02:14 PM, Jim Bosch wrote:
To solve it, I think you'd want anything registered by a specific module to appear both in that module's registry and the global registry, with the module's registry taking precedence. Are there any cases where you'd want something only to appear in the module-specific registry?
Anything that gets injected into the global registry is prone to violate the ODR. Of course, also adding it into a local registry and letting that have precedence may mask the ODR violation issue. But in that case, it isn't clear why we'd add it to the global registry at all. It may make sense to define a policy that types exported explicitly by an extension module may be added to the global repository. In contrast, prerequisite types that are only by way of dependency added (e.g., common libstdc++ types) should be stored in a local / private registry. Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 08/29/2011 09:41 AM, Stefan Seefeld wrote:
On 08/28/2011 02:14 PM, Jim Bosch wrote:
To solve it, I think you'd want anything registered by a specific module to appear both in that module's registry and the global registry, with the module's registry taking precedence. Are there any cases where you'd want something only to appear in the module-specific registry?
Anything that gets injected into the global registry is prone to violate the ODR. Of course, also adding it into a local registry and letting that have precedence may mask the ODR violation issue. But in that case, it isn't clear why we'd add it to the global registry at all.
The reason to add it to the global registry is so if you know one of the modules you depend on registered a converter, you don't have to do it yourself.
It may make sense to define a policy that types exported explicitly by an extension module may be added to the global repository. In contrast, prerequisite types that are only by way of dependency added (e.g., common libstdc++ types) should be stored in a local / private registry.
I don't see how having a global registry makes things any worse from an ODR perspective. It seems like it's mostly just the same "where do templates get instantiated" problem that compilers and linkers always have to deal with. In other words, instantiating something like: class_< std::vector<int> > in two shared libraries doesn't seem any different from instantiating something like: std::vector<int> in two shared libraries, and we basically always have to leave it up to the compiler/linker to solve the latter. Is the problem the fact that the global registry stores function pointers to template instantiations? I can see how that would appear to make the multiple (template) definitions more problematic, but it also seems like that's a problem compilers and linkers would have to have already solved. Of course, this may just be wishful thinking on my part; I admit I'm not very familiar with how these problems are handled in practice. Jim
Am Dienstag, 30. August 2011, 08:42:53 schrieb Jim Bosch:
I don't see how having a global registry makes things any worse from an ODR perspective. It seems like it's mostly just the same "where do templates get instantiated" problem that compilers and linkers always have to deal with.
There is a big difference in the case that two separate extensions (want to) use different converters, though.
In other words, instantiating something like:
class_< std::vector<int> >
in two shared libraries doesn't seem any different from instantiating something like:
std::vector<int>
in two shared libraries, and we basically always have to leave it up to the compiler/linker to solve the latter.
Here you assume that class_< std::vector<int> >(…) will generate the same code in both cases. For std::vector<int>, this holds, but for class_< std::vector<int> >(…), the (…) part—which you left out—is crucial. In practice, it happened to me with two extensions which did not even agree on the /type/ of converter (class_ vs. rvalue) for the same class (vigra::TinyVector, a small fixed-size vector). Although it would be nice to fix/allow this (local registries, which can still be imported/reused as a dependency), it /may/ be an even better fix to change the extensions and make them agree on the "optimal" converter. (The obvious problem is of course the definition of "optimal" which may be different depending on the application.) The problem with this is the very reason for the variety of std::vector converters out there (e.g. focusing on conversion to native Python types, conversion speed, or in-place writability/modifications). HTH, Hans
On 08/30/2011 01:26 AM, Hans Meine wrote:
Am Dienstag, 30. August 2011, 08:42:53 schrieb Jim Bosch:
I don't see how having a global registry makes things any worse from an ODR perspective. It seems like it's mostly just the same "where do templates get instantiated" problem that compilers and linkers always have to deal with.
There is a big difference in the case that two separate extensions (want to) use different converters, though.
In other words, instantiating something like:
class_< std::vector<int> >
in two shared libraries doesn't seem any different from instantiating something like:
std::vector<int>
in two shared libraries, and we basically always have to leave it up to the compiler/linker to solve the latter.
Here you assume that class_< std::vector<int> >(…) will generate the same code in both cases. For std::vector<int>, this holds, but for class_< std::vector<int> >(…), the (…) part—which you left out—is crucial.
They aren't different as far as the One Definition Rule goes. You're just passing different arguments to the constructor, not providing a different definition. And I think that's true in both cases; sure, the class_ code is much more complex, but it's still perfectly legal C++, even across shared library boundaries.
In practice, it happened to me with two extensions which did not even agree on the /type/ of converter (class_ vs. rvalue) for the same class (vigra::TinyVector, a small fixed-size vector).
Although it would be nice to fix/allow this (local registries, which can still be imported/reused as a dependency), it /may/ be an even better fix to change the extensions and make them agree on the "optimal" converter. (The obvious problem is of course the definition of "optimal" which may be different depending on the application.) The problem with this is the very reason for the variety of std::vector converters out there (e.g. focusing on conversion to native Python types, conversion speed, or in-place writability/modifications).
I agree with all of the above, and these could all be solved by my proposal of having per-module registrations take precedence over gloal registrations. Having a single optimal converter is clearly the best solution when such a thing exists, and we can anticipate the type, so adding more built-in converters too Boost.Python is part of the solution. But this doesn't have anything to do with the One Definition Rule, and I still don't see that we're having any more problems in that regard than template libraries usually do. Jim
On 30 Aug 2011 at 10:42, Jim Bosch wrote:
I agree with all of the above, and these could all be solved by my proposal of having per-module registrations take precedence over gloal registrations. Having a single optimal converter is clearly the best solution when such a thing exists, and we can anticipate the type, so adding more built-in converters too Boost.Python is part of the solution.
But this doesn't have anything to do with the One Definition Rule, and I still don't see that we're having any more problems in that regard than template libraries usually do.
What I have done in my own libraries is to have per-process type registries keep their data on the basis of shared object or DLL. The "current" SO/DLL can be determined by having an inlined thunk template function pass in the address of some guaranteed per SO/DLL data e.g. a static char * does just fine. One then looks up the pointer address in a list of loaded SO/DLLs so one knows in which context a particular type resolution is being performed. This sounds like a lot of complexity, and at the start getting the data storage right does take time, but in the long run it saves huge amounts of complexity. Once it's working it's a god send because third parties libraries - which often break ODR whether intentionally or unintentionally - work as expected. I have a whole load of code implementing this in open source if consulting an implementation would be useful to you, and of course I'm always here by email. Long time members of this mailing list will know what I'm referring to. HTH, Niall -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.
On 08/30/2011 02:42 AM, Jim Bosch wrote:
On 08/29/2011 09:41 AM, Stefan Seefeld wrote:
On 08/28/2011 02:14 PM, Jim Bosch wrote:
To solve it, I think you'd want anything registered by a specific module to appear both in that module's registry and the global registry, with the module's registry taking precedence. Are there any cases where you'd want something only to appear in the module-specific registry?
Anything that gets injected into the global registry is prone to violate the ODR. Of course, also adding it into a local registry and letting that have precedence may mask the ODR violation issue. But in that case, it isn't clear why we'd add it to the global registry at all.
The reason to add it to the global registry is so if you know one of the modules you depend on registered a converter, you don't have to do it yourself.
As I suggested in reply to Dave, I think it would be better to require modules that depend on external converters to explicitly import them. Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 08/30/2011 04:04 AM, Stefan Seefeld wrote:
On 08/30/2011 02:42 AM, Jim Bosch wrote:
On 08/29/2011 09:41 AM, Stefan Seefeld wrote:
On 08/28/2011 02:14 PM, Jim Bosch wrote:
To solve it, I think you'd want anything registered by a specific module to appear both in that module's registry and the global registry, with the module's registry taking precedence. Are there any cases where you'd want something only to appear in the module-specific registry?
Anything that gets injected into the global registry is prone to violate the ODR. Of course, also adding it into a local registry and letting that have precedence may mask the ODR violation issue. But in that case, it isn't clear why we'd add it to the global registry at all.
The reason to add it to the global registry is so if you know one of the modules you depend on registered a converter, you don't have to do it yourself.
As I suggested in reply to Dave, I think it would be better to require modules that depend on external converters to explicitly import them.
I can understand the motivation for requiring manual imports of conversions, as a sort of "explicit is better than implicit" argument. But I don't think it saves us anything in terms of ODR violations. If I'm understanding the ODR issue (or lack thereof) correctly, we can make everyone mostly happy by providing an option whether conversion registrations should go into a per-module registry, a global one, or both. We'd then allow modules to import the per-module registry of another module into their own if they like. In all cases, we'd look up things in the per-module registry first, and then go to the global registry (where things registered by class_ would typically go). I don't like the idea of Python users being able to tell all modules to use a different set of conversions than the ones their authors intended. That seems fraught with peril, and unnecessary if module builders can pull converters from modules they depend on. Jim
on Sat Aug 27 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:
On 08/27/2011 04:40 PM, Dave Abrahams wrote:
Hmm. I'm guessing the global type registry would still be the default, and per-module registries would override these when available? It sounds like Stefan has a clear use case in mind, and I'd be curious to know what it is. Me too.
I believe what we were discussing at the time was a situation where an extension module would not only define converters for its own types, but also common types that may are required by the API.
That's currently supported by the global registry.
This could in particular include common types from libstdc++.
If multiple extension modules do this, than a Python program that happens to use them in the same application may end up with undefined behavior (does this constitute an ODR violation under the hood ?)
Essentially, yes.
To make this work, the common type converters need to be factored out and shared. This in itself is impractical (since the original authors may not be aware of this need). Furthermore, a converter may require module-specific behavior, i.e. converting a std::string in the context of one module may be different from the desired conversion in another.
Binding converters to particular modules (and requiring to explicitly import / exporting converters) seems like a solution to the above.
It might be, but your description of what you're actually proposing is pretty vague, still, so it's hard to tell. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 08/28/2011 02:39 PM, Dave Abrahams wrote:
on Sat Aug 27 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:
Binding converters to particular modules (and requiring to explicitly import / exporting converters) seems like a solution to the above. It might be, but your description of what you're actually proposing is pretty vague, still, so it's hard to tell.
I haven't fully thought this through myself yet. In terms of UI I imagine this to work like this: import ext_mod would still keep the converters private to ext_mod by default, so only functions and methods from ext_mod itself would have access to it. To make those converters accessible globally would require an extra step, for example: from ext_mod import __converters__ Thus each python module can decide itself whether or not it wants to access them. Implementation-wise this would imply that a single global registry would be replaced by a linked list of per-module registries. The obvious disadvantage is that converter lookup will be relatively slow. There could still be ways to keep the current behavior, both for backward compatibility but also as an optimization, if the author knows that there is no danger of symbol collision. Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Sat, Aug 27, 2011 at 2:39 AM, Jim Bosch <talljimbo@gmail.com> wrote:
I am aware of Py++ and the extensions associated with it, and some of that could definitely go into Boost.Python proper (and I think I've heard Roman state that he wouldn't have a problem with someone else doing the work to make it so, and he just didn't have time to do it himself).
If you will find them useful, I'll be glad to contribute and help with the integration. Py++ also contains the modified version of indexing suite v2 (more containers, bug fixes and additional methods for existing containers). Regards, Roman
On Friday 26 August 2011 16:39:11 Jim Bosch wrote:
Well, speaking for myself, mostly time. I'd be inclined to do a rewrite along the lines of the langbinding ideas if I had time.
I had only been vaguely aware of langbinding until I followed up on your last email. It's a very nice separation, though after bad experiences using SWIG I am a little wary about trying to build a one-size-fits-all front-end for different languages.
Is code for boost.langbinding available anywhere? Or is it currently only a design description? Regards, Ravi
on Sat Sep 17 2011, Ravi <lists_ravi-AT-lavabit.com> wrote:
On Friday 26 August 2011 16:39:11 Jim Bosch wrote:
Well, speaking for myself, mostly time. I'd be inclined to do a rewrite
along the lines of the langbinding ideas if I had time.
I had only been vaguely aware of langbinding until I followed up on your last email. It's a very nice separation, though after bad experiences using SWIG I am a little wary about trying to build a one-size-fits-all front-end for different languages.
Is code for boost.langbinding available anywhere? Or is it currently only a design description?
Yes, there is some code, and there are tests, in http://svn.boost.org/svn/boost/sandbox/langbinding/ HTH, -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 08/26/2011 04:09 PM, Dave Abrahams wrote:
on Thu Aug 25 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:
Jim,
this is an interesting idea. There has been lots of general (dare I say generic ?) discussion concerning process improvements (which unfortunately most of the time diverted into tool discussions). Among the fundamental issues is a modularization of boost. I think it would be great if boost.python could follow through on its own, by becoming a separate entity. Separate from Boost? I guess that's a possibility but I'm not sure I see the advantage.
Jim already followed up on that, and I fully agree with that. If things can happen within Boost, all the better.
* A per-module type registry, to avoid conflicting converters in multi-module projects. Interesting idea. How does sharing types across multiple modules work in that scenario?
That's a good question. I don't have an answer to that. In fact, the idea of having per-module type registries grew out of discussions I had with Troy quite a while ago, where we considered all the bad things that could happen if multiple modules tried to export the same types. As always: explicit is better than implicit.
- Some limited degree of priority-based overload matching. Not sure how best to approach this one yet, though. +1 This is a solved problem... just not in Boost.Python. Daniel Wallin worked it out for luabind and we were going to incorporate it into langbinding. Happy to discuss it further.
I'm happy to see some discussion on langbinding in this context. I also agree with Jim's pragmatic approach to this he is proposing in another mail. Stefan -- ...ich hab' noch einen Koffer in Berlin...
participants (9)
-
Dave Abrahams -
Hans Meine -
Jim Bosch -
Neal Becker -
Niall Douglas -
Ralf Grosse-Kunstleve -
Ravi -
Roman Yakovenko -
Stefan Seefeld