Bugs in the Boost.Python tutorial example
Hi, I had a hard time building the example/tutorial project. Some of this is, I believe, due to bugs in the distribution, and some is probably my lack of understanding. Some of this I posted to the Boost interest group before I could join this (Python C++) SIG, hoping it would get forwarded on here, so there may be a duplication. 1. Calling bjam in the example/tutorial project failed to even start up the build system. This was because the example/bootstrap.jam pointed to the wrong path for the build system root. Instead of ../../../tools/build/v2, it should be ../../../tools/build/src/kernel. When I changed this, bjam now got past the build system startup. 2. Building the project not only compiles hello.cpp, but it also builds a private copy of the Boost/Python library. So I needed to supply the properties needed to correctly build this library (i.e., link=shared, address-mode=64). Of course, I needed to supply those same properties anyway as part of creating the extension. There's probably a way to change something so that the extension uses the library built in the Boost.Python's own project, or if I have obtained the libraries without having to build them, it would use these. I don't know if you intended to have the tutorial example make its own copies, but it seems a waste of resources to do so. 3. The link for debug mode failed, saying that the .pdb file was in the wrong format (LNK1207). This is a bug, due to an option in the link command '/IMPORTLIB:...hello_ext.pdb'. So the linker is creating the .pdb file as an import library, then complaining that it's not a valid pdb file. I changed '.pdb' to '.lib'. I could also have removed this option entirely, since hello_ext.pyd doesn't have anything to export anyway. 4. Before figuring out that the link was the problem, I changed the /Z7 argument to /Zi in the compile command for hello.cpp. I don't know if this was necessary, or if it was necessary to leave it in place. For now, I just wanted to get it to build. Without /Z7, the debug symbols go into example/tutorial/vc120.pdb. I don't know if the linker found these or not. When I try stepping into the extension, I'll know for sure. Microsoft prefers that .pdb files be used for debug symbols rather than embedding them in the .obj files, so this might be the only real reason to make the change. 4. The link for both release mode failed with two undefined symbols, __imp_DecodePointer and __imp_EncodePointer, which are in kernel32.lib. I tried adding kernel32.lib to the link's inputs. But then it warned that there could be static constructors/destructors not getting called. After much research on this topic, I found that the source of the problem was the /NOENTRY argument in the link command, which has the consequence that the automatic CRT initialization of the DLL doesn't occur. So I remove the /NOENTRY and got not warnings, and I didn't need to add kernerl32.lib either. 5. A minor point. The MACHINE:X64 is redundant. The linker knows it's X64 because of the architecture of the input files and libraries. Nothing wrong with it being there, but it's just clutter in the code. 6. Now bjam was successful in building whatever it wanted to build. It said that hello.test passed. Sounds great, I thought. But I then went into Python and tried 'import hello_ext' and that failed. So I have an issue with the test program passing, while the extension didn't actually work. 7. The problem was that bjam didn't put the hello_ext.pyd file in my Python's lib/site-packages folder. It built the .pyd and .pdb files in the example/tutorial/bin/... staging area, and copied (ONLY) the .pyd file to example/tutorial. So not only did the .pyd file get put in the wrong place, but the .pdb was left out. If I am going to debug my extension (such as with Visual Studio's Python Tools), the .pdb file also needs to be in the lib/site-packages folder. Without the .pdb file, the PTVS debugger will not set breakpoints or step into the extension code. To summarize, the fixes I made in order to build the example and use it in Python were: * Change the build system path in examples/bootstrap.jam. * Change the /IMPORTLIB:....pdb to ....lib in the link command. I could have removed it altogether. * Remove /NOENTRY from the link command. * Manually copy the example/tutorial/bin/.../hello_ext.(pyd,pdb) files to Python's lib/site-packages. This cost me about a whole day's time, and I'm pretty resourceful. And I actually want to build my own extensions in VS instead of bjam, linking to the Boost.Python DLL. I started of by trying this, and my extension crashed. So that's why I went about building the example with bjam, to at least see what a successful deployment looked like. But I'll still have to figure out what went wrong with my original attempt; there may be additional bugs to report. I suppose that a lot of people would have given up building the tutorial example and then given up on Boost.Python altogether. Michael Rolle (408) 313-8149
Hi Michael, I had written a tutorial on Compiling Boost.Python using Visual Studio. Please check. http://www.sigverse.org/wiki/en/index.php?Compiling%20Boost.Python%20for%20P... I know I am not directly answering the questions you raised here, but I guess it might help you since this method is working for me and several others. My aim was to embed Python inside the C++ code, so that I can call Python functions. I guess your aim is to call C++ functions from Python. Nevertheless, the process of building the example, is not so different. Please let me know if it helps. I am running on a deadline, else I would have tried to find a concrete solution to your issues. Sorry again. Cheers, Raghav On Mon, Nov 17, 2014 at 1:10 PM, Michael Rolle <m@rolle.name> wrote:
Hi,
I had a hard time building the example/tutorial project. Some of this is, I believe, due to bugs in the distribution, and some is probably my lack of understanding.
Some of this I posted to the Boost interest group before I could join this (Python C++) SIG, hoping it would get forwarded on here, so there may be a duplication.
1. Calling bjam in the example/tutorial project failed to even start up the build system. This was because the example/bootstrap.jam pointed to the wrong path for the build system root. Instead of ../../../tools/build/v2, it should be ../../../tools/build/src/kernel. When I changed this, bjam now got past the build system startup.
2. Building the project not only compiles hello.cpp, but it also builds a private copy of the Boost/Python library. So I needed to supply the properties needed to correctly build this library (i.e., link=shared, address-mode=64). Of course, I needed to supply those same properties anyway as part of creating the extension.
There's probably a way to change something so that the extension uses the library built in the Boost.Python's own project, or if I have obtained the libraries without having to build them, it would use these. I don't know if you intended to have the tutorial example make its own copies, but it seems a waste of resources to do so.
3. The link for debug mode failed, saying that the .pdb file was in the wrong format (LNK1207). This is a bug, due to an option in the link command '/IMPORTLIB:...hello_ext.pdb'. So the linker is creating the .pdb file as an import library, then complaining that it's not a valid pdb file. I changed '.pdb' to '.lib'. I could also have removed this option entirely, since hello_ext.pyd doesn't have anything to export anyway.
4. Before figuring out that the link was the problem, I changed the /Z7 argument to /Zi in the compile command for hello.cpp. I don't know if this was necessary, or if it was necessary to leave it in place. For now, I just wanted to get it to build. Without /Z7, the debug symbols go into example/tutorial/vc120.pdb. I don't know if the linker found these or not. When I try stepping into the extension, I'll know for sure. Microsoft prefers that .pdb files be used for debug symbols rather than embedding them in the .obj files, so this might be the only real reason to make the change.
4. The link for both release mode failed with two undefined symbols, __imp_DecodePointer and __imp_EncodePointer, which are in kernel32.lib. I tried adding kernel32.lib to the link's inputs. But then it warned that there could be static constructors/destructors not getting called. After much research on this topic, I found that the source of the problem was the /NOENTRY argument in the link command, which has the consequence that the automatic CRT initialization of the DLL doesn't occur. So I remove the /NOENTRY and got not warnings, and I didn't need to add kernerl32.lib either.
5. A minor point. The MACHINE:X64 is redundant. The linker knows it's X64 because of the architecture of the input files and libraries. Nothing wrong with it being there, but it's just clutter in the code.
6. Now bjam was successful in building whatever it wanted to build. It said that hello.test passed. Sounds great, I thought. But I then went into Python and tried 'import hello_ext' and that failed. So I have an issue with the test program passing, while the extension didn't actually work.
7. The problem was that bjam didn't put the hello_ext.pyd file in my Python's lib/site-packages folder. It built the .pyd and .pdb files in the example/tutorial/bin/... staging area, and copied (ONLY) the .pyd file to example/tutorial. So not only did the .pyd file get put in the wrong place, but the .pdb was left out. If I am going to debug my extension (such as with Visual Studio's Python Tools), the .pdb file also needs to be in the lib/site-packages folder. Without the .pdb file, the PTVS debugger will not set breakpoints or step into the extension code.
To summarize, the fixes I made in order to build the example and use it in Python were: * Change the build system path in examples/bootstrap.jam. * Change the /IMPORTLIB:....pdb to ....lib in the link command. I could have removed it altogether. * Remove /NOENTRY from the link command. * Manually copy the example/tutorial/bin/.../hello_ext.(pyd,pdb) files to Python's lib/site-packages.
This cost me about a whole day's time, and I'm pretty resourceful. And I actually want to build my own extensions in VS instead of bjam, linking to the Boost.Python DLL. I started of by trying this, and my extension crashed. So that's why I went about building the example with bjam, to at least see what a successful deployment looked like. But I'll still have to figure out what went wrong with my original attempt; there may be additional bugs to report.
I suppose that a lot of people would have given up building the tutorial example and then given up on Boost.Python altogether.
Michael Rolle (408) 313-8149
_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig
Hi Raghvendra, I'll look at your tutorial shortly. Are you the one responsible for maintaining / bug fixing for Boost.Python? Or would I assume someone else will see my post and act on it? Actually, what I want to know is how I report bugs to whomever will fix them. Thanks. Michael Rolle (408) 313-8149 ----- Original Message ----- From: Raghvendra Jain To: Development of Python/C++ integration Sent: Monday, November 17, 2014 5:09 AM Subject: Re: [C++-sig] Bugs in the Boost.Python tutorial example Hi Michael, I had written a tutorial on Compiling Boost.Python using Visual Studio. Please check. http://www.sigverse.org/wiki/en/index.php?Compiling%20Boost.Python%20for%20P... I know I am not directly answering the questions you raised here, but I guess it might help you since this method is working for me and several others. My aim was to embed Python inside the C++ code, so that I can call Python functions. I guess your aim is to call C++ functions from Python. Nevertheless, the process of building the example, is not so different. Please let me know if it helps. I am running on a deadline, else I would have tried to find a concrete solution to your issues. Sorry again. Cheers, Raghav On Mon, Nov 17, 2014 at 1:10 PM, Michael Rolle <m@rolle.name> wrote: Hi, I had a hard time building the example/tutorial project. Some of this is, I believe, due to bugs in the distribution, and some is probably my lack of understanding. Some of this I posted to the Boost interest group before I could join this (Python C++) SIG, hoping it would get forwarded on here, so there may be a duplication. 1. Calling bjam in the example/tutorial project failed to even start up the build system. This was because the example/bootstrap.jam pointed to the wrong path for the build system root. Instead of ../../../tools/build/v2, it should be ../../../tools/build/src/kernel. When I changed this, bjam now got past the build system startup. 2. Building the project not only compiles hello.cpp, but it also builds a private copy of the Boost/Python library. So I needed to supply the properties needed to correctly build this library (i.e., link=shared, address-mode=64). Of course, I needed to supply those same properties anyway as part of creating the extension. There's probably a way to change something so that the extension uses the library built in the Boost.Python's own project, or if I have obtained the libraries without having to build them, it would use these. I don't know if you intended to have the tutorial example make its own copies, but it seems a waste of resources to do so. 3. The link for debug mode failed, saying that the .pdb file was in the wrong format (LNK1207). This is a bug, due to an option in the link command '/IMPORTLIB:...hello_ext.pdb'. So the linker is creating the .pdb file as an import library, then complaining that it's not a valid pdb file. I changed '.pdb' to '.lib'. I could also have removed this option entirely, since hello_ext.pyd doesn't have anything to export anyway. 4. Before figuring out that the link was the problem, I changed the /Z7 argument to /Zi in the compile command for hello.cpp. I don't know if this was necessary, or if it was necessary to leave it in place. For now, I just wanted to get it to build. Without /Z7, the debug symbols go into example/tutorial/vc120.pdb. I don't know if the linker found these or not. When I try stepping into the extension, I'll know for sure. Microsoft prefers that .pdb files be used for debug symbols rather than embedding them in the .obj files, so this might be the only real reason to make the change. 4. The link for both release mode failed with two undefined symbols, __imp_DecodePointer and __imp_EncodePointer, which are in kernel32.lib. I tried adding kernel32.lib to the link's inputs. But then it warned that there could be static constructors/destructors not getting called. After much research on this topic, I found that the source of the problem was the /NOENTRY argument in the link command, which has the consequence that the automatic CRT initialization of the DLL doesn't occur. So I remove the /NOENTRY and got not warnings, and I didn't need to add kernerl32.lib either. 5. A minor point. The MACHINE:X64 is redundant. The linker knows it's X64 because of the architecture of the input files and libraries. Nothing wrong with it being there, but it's just clutter in the code. 6. Now bjam was successful in building whatever it wanted to build. It said that hello.test passed. Sounds great, I thought. But I then went into Python and tried 'import hello_ext' and that failed. So I have an issue with the test program passing, while the extension didn't actually work. 7. The problem was that bjam didn't put the hello_ext.pyd file in my Python's lib/site-packages folder. It built the .pyd and .pdb files in the example/tutorial/bin/... staging area, and copied (ONLY) the .pyd file to example/tutorial. So not only did the .pyd file get put in the wrong place, but the .pdb was left out. If I am going to debug my extension (such as with Visual Studio's Python Tools), the .pdb file also needs to be in the lib/site-packages folder. Without the .pdb file, the PTVS debugger will not set breakpoints or step into the extension code. To summarize, the fixes I made in order to build the example and use it in Python were: * Change the build system path in examples/bootstrap.jam. * Change the /IMPORTLIB:....pdb to ....lib in the link command. I could have removed it altogether. * Remove /NOENTRY from the link command. * Manually copy the example/tutorial/bin/.../hello_ext.(pyd,pdb) files to Python's lib/site-packages. This cost me about a whole day's time, and I'm pretty resourceful. And I actually want to build my own extensions in VS instead of bjam, linking to the Boost.Python DLL. I started of by trying this, and my extension crashed. So that's why I went about building the example with bjam, to at least see what a successful deployment looked like. But I'll still have to figure out what went wrong with my original attempt; there may be additional bugs to report. I suppose that a lot of people would have given up building the tutorial example and then given up on Boost.Python altogether. Michael Rolle (408) 313-8149 _______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig ------------------------------------------------------------------------------ _______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig
On Mon, Nov 17, 2014 at 4:35 PM, Michael Rolle <m@rolle.name> wrote:
Are you the one responsible for maintaining / bug fixing for Boost.Python? Or would I assume someone else will see my post and act on it? Actually, what I want to know is how I report bugs to whomever will fix them.
To my knowledge, the answer to this question is actually a rather unhappy one - I'm not sure anyone is currently maintaining Boost.Python. I think there may still be general Boost devs who will fix critical bugs, especially if the bug report is accompanied by a patch, but there are a lot of stale tickets in the issue tracker (https://svn.boost.org/trac/boost), and I don't know of anyone who really feels any strong ownership of the library at this stage. I've thought about stepping up to do some of it myself, but given that my day job now involves working with Swig (even though I prefer Boost.Python), it's just not something I can do. I'd love to be corrected by anyone on this list who can say otherwise, of course. Jim
On 17/11/14 04:50 PM, Jim Bosch wrote:
On Mon, Nov 17, 2014 at 4:35 PM, Michael Rolle <m@rolle.name <mailto:m@rolle.name>> wrote:
Are you the one responsible for maintaining / bug fixing for Boost.Python? Or would I assume someone else will see my post and act on it? Actually, what I want to know is how I report bugs to whomever will fix them.
To my knowledge, the answer to this question is actually a rather unhappy one - I'm not sure anyone is currently maintaining Boost.Python. I think there may still be general Boost devs who will fix critical bugs, especially if the bug report is accompanied by a patch, but there are a lot of stale tickets in the issue tracker (https://svn.boost.org/trac/boost), and I don't know of anyone who really feels any strong ownership of the library at this stage. I've thought about stepping up to do some of it myself, but given that my day job now involves working with Swig (even though I prefer Boost.Python), it's just not something I can do.
I'd love to be corrected by anyone on this list who can say otherwise, of course.
Unfortunately, I can't. In addition to that, it seems the problems reported in this thread are rather about the build system, which at least some of those with expertise in Boost.Python aren't very familiar with, in particular if it affects platforms they don't use themselves. (Specifically: I'd be happy to help fix issues with the Boost.Python code itself, but I don't feel competent with bjam / Boost.Build, and my knowledge of MSVC is almost non-existent, not to speak of the fact that I have no computer near me running Windows, so I couldn't even test any Windows-specific patches.) Stefan -- ...ich hab' noch einen Koffer in Berlin...
Thanks, Stefan. I've created a ticket on the tracker that you provided the link to. It's number 10799. In this case, the fixes should be pretty simple and probably involve the rules for linking DLLs generally and for making Python extensions. One of the fixes just involves a change to a boost-build.jam file, so that it will point to the same build system that the tutorial just told me to create with the bootstrap command. Michael Rolle (408) 313-8149 ----- Original Message ----- From: "Stefan Seefeld" <stefan@seefeld.name> To: <cplusplus-sig@python.org> Sent: Monday, November 17, 2014 2:01 PM Subject: Re: [C++-sig] Bugs in the Boost.Python tutorial example
On 17/11/14 04:50 PM, Jim Bosch wrote:
On Mon, Nov 17, 2014 at 4:35 PM, Michael Rolle <m@rolle.name <mailto:m@rolle.name>> wrote:
Are you the one responsible for maintaining / bug fixing for Boost.Python? Or would I assume someone else will see my post and act on it? Actually, what I want to know is how I report bugs to whomever will fix them.
To my knowledge, the answer to this question is actually a rather unhappy one - I'm not sure anyone is currently maintaining Boost.Python. I think there may still be general Boost devs who will fix critical bugs, especially if the bug report is accompanied by a patch, but there are a lot of stale tickets in the issue tracker (https://svn.boost.org/trac/boost), and I don't know of anyone who really feels any strong ownership of the library at this stage. I've thought about stepping up to do some of it myself, but given that my day job now involves working with Swig (even though I prefer Boost.Python), it's just not something I can do.
I'd love to be corrected by anyone on this list who can say otherwise, of course.
Unfortunately, I can't. In addition to that, it seems the problems reported in this thread are rather about the build system, which at least some of those with expertise in Boost.Python aren't very familiar with, in particular if it affects platforms they don't use themselves.
(Specifically: I'd be happy to help fix issues with the Boost.Python code itself, but I don't feel competent with bjam / Boost.Build, and my knowledge of MSVC is almost non-existent, not to speak of the fact that I have no computer near me running Windows, so I couldn't even test any Windows-specific patches.)
Stefan
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig
Thanks, Jim. I've created a ticket on the tracker that you provided the link to. It's number 10799. In this case, the fixes should be pretty simple and probably involve the rules for linking DLLs generally and for making Python extensions. One of the fixes just involves a change to a boost-build.jam file, so that it will point to the same build system that the tutorial just told me to create with the bootstrap command. Michael Rolle (408) 313-8149 ----- Original Message ----- From: Jim Bosch To: Development of Python/C++ integration Sent: Monday, November 17, 2014 1:50 PM Subject: Re: [C++-sig] Bugs in the Boost.Python tutorial example On Mon, Nov 17, 2014 at 4:35 PM, Michael Rolle <m@rolle.name> wrote: Are you the one responsible for maintaining / bug fixing for Boost.Python? Or would I assume someone else will see my post and act on it? Actually, what I want to know is how I report bugs to whomever will fix them. To my knowledge, the answer to this question is actually a rather unhappy one - I'm not sure anyone is currently maintaining Boost.Python. I think there may still be general Boost devs who will fix critical bugs, especially if the bug report is accompanied by a patch, but there are a lot of stale tickets in the issue tracker (https://svn.boost.org/trac/boost), and I don't know of anyone who really feels any strong ownership of the library at this stage. I've thought about stepping up to do some of it myself, but given that my day job now involves working with Swig (even though I prefer Boost.Python), it's just not something I can do. I'd love to be corrected by anyone on this list who can say otherwise, of course. Jim ------------------------------------------------------------------------------ _______________________________________________ Cplusplus-sig mailing list Cplusplus-sig@python.org https://mail.python.org/mailman/listinfo/cplusplus-sig
On 11/17/2014 2:01 PM, Stefan Seefeld wrote:
(Specifically: I'd be happy to help fix issues with the Boost.Python code itself, but I don't feel competent with bjam / Boost.Build, and my knowledge of MSVC is almost non-existent, not to speak of the fact that I have no computer near me running Windows, so I couldn't even test any Windows-specific patches.)
Thanks for offering! Here's a report for a crash bug with a patch that I filed 16 months ago. The formatting is busted in the original report; please see the first comment for a properly formatted test case and patch. https://svn.boost.org/trac/boost/ticket/8978 We've been running this patch successfully since that ticket was filed. Alex
On 17/11/14 06:54 PM, Alex Mohr wrote:
On 11/17/2014 2:01 PM, Stefan Seefeld wrote:
(Specifically: I'd be happy to help fix issues with the Boost.Python code itself, but I don't feel competent with bjam / Boost.Build, and my knowledge of MSVC is almost non-existent, not to speak of the fact that I have no computer near me running Windows, so I couldn't even test any Windows-specific patches.)
Thanks for offering! Here's a report for a crash bug with a patch that I filed 16 months ago. The formatting is busted in the original report; please see the first comment for a properly formatted test case and patch.
https://svn.boost.org/trac/boost/ticket/8978
We've been running this patch successfully since that ticket was filed.
While the issue looks clear (and the patch good), can you attach a test case that would allow me to reproduce the issue (and observe the fix with the patch applied) ? Minimal test cases always accelerate the processing of reported issues. :-) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 11/17/2014 4:17 PM, Stefan Seefeld wrote:
On 17/11/14 06:54 PM, Alex Mohr wrote:
On 11/17/2014 2:01 PM, Stefan Seefeld wrote:
(Specifically: I'd be happy to help fix issues with the Boost.Python code itself, but I don't feel competent with bjam / Boost.Build, and my knowledge of MSVC is almost non-existent, not to speak of the fact that I have no computer near me running Windows, so I couldn't even test any Windows-specific patches.)
Thanks for offering! Here's a report for a crash bug with a patch that I filed 16 months ago. The formatting is busted in the original report; please see the first comment for a properly formatted test case and patch.
https://svn.boost.org/trac/boost/ticket/8978
We've been running this patch successfully since that ticket was filed.
While the issue looks clear (and the patch good), can you attach a test case that would allow me to reproduce the issue (and observe the fix with the patch applied) ? Minimal test cases always accelerate the processing of reported issues. :-)
Sorry, the minimal test case is included in the report. I will inline it here also. Alex // C++ #include <boost/python.hpp> static void f1(int a0, int a1) { } static void f2(int a0, int a1, int a2) { } BOOST_PYTHON_MODULE(kwargCrash) { boost::python::def("f", f1, (arg("a1")=2)); boost::python::def("f", f2, (arg("a2")=2)); } # Python import kwargCrash kwargCrash.f(0, a1=2)
On 11/17/14 4:17 PM, Stefan Seefeld wrote:
While the issue looks clear (and the patch good), can you attach a test case that would allow me to reproduce the issue (and observe the fix with the patch applied) ? Minimal test cases always accelerate the processing of reported issues. :-)
Also please refer to this message detailing my analysis of the problem with the minimal test case: https://mail.python.org/pipermail/cplusplus-sig/2013-August/017012.html Thanks again! Alex
participants (5)
-
Alex Mohr -
Jim Bosch -
Michael Rolle -
Raghvendra Jain -
Stefan Seefeld