cppyy fails to build on gcc 5 and clang
Hi guys, please see (reposting here to grab attention as hinted by Armin) https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5-an... Note: for now I am fine, I managed to build it using gcc 4.9 toolchain. Need to see how far I get with captnproto now .. Cheers, /Tobias
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from. \-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else <http://refi64.com/> On Jan 17 2017, at 11:53 am, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:
Hi guys,
please see (reposting here to grab attention as hinted by Armin)
https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5 -and-clang
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain. Need to see how far I get with captnproto now ..
Cheers, /Tobias
pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi, On Tuesday 2017-01-17 18:03, Ryan Gonzalez wrote:
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
that will likely be a solution, b/c that wrapper file is put in place if the libcxx from clang is not used. Seems a configuration mismatch: to use libcxx, the cmake machinery needs to be told to do so (-Dlibcxx=ON IIRC), as it's off by default. Sorry, is my first ever pip and had to intermarry python and C++ installs w/o wanting to rewrite the infrastructure. :} I'll have to look how to provide these extra arguments to cmake, but should be possible. I don't think libcxx has been tested outside MacOS X, though. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
I think you misunderstood me. Your build log shows: oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ make cpppypy CC=`which clang` \ CXX=`which clang` \ CXXFLAGS="-std=c++11 -stdlib=libc++" \ LDFLAGS="-stdlib=libc++" \ MAKE_NPROCS=4 \ ~/pypy-5.6-linux_x86_64-portable/bin/pip install --verbose PyPy-cppyy-backend I'm saying to _remove_ the `-stdlib=libc++` that's currently there. \-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else <http://refi64.com/> On Jan 17 2017, at 12:20 pm, wlavrijsen@lbl.gov wrote:
Hi,
On Tuesday 2017-01-17 18:03, Ryan Gonzalez wrote: What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
that will likely be a solution, b/c that wrapper file is put in place if the
libcxx from clang is not used. Seems a configuration mismatch: to use libcxx, the cmake machinery needs to be told to do so (-Dlibcxx=ON IIRC), as it's off by default.
Sorry, is my first ever pip and had to intermarry python and C++ installs w/o wanting to rewrite the infrastructure. :} I'll have to look how to provide
these extra arguments to cmake, but should be possible. I don't think libcxx has been tested outside MacOS X, though.
Best regards, Wim \-- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Ryan, On Tuesday 2017-01-17 18:28, Ryan Gonzalez wrote:
I think you misunderstood me. Your build log shows:
no, I didn't. :) It's not my build log, it's Tobias's.
I'm saying to _remove_ the `-stdlib=libc++` that's currently there.
Yes and I expect it to work. But I wrote the pip, and ideally, I get that fixed to allow what Tobias is doing. B/c if the '-stdlib=libc++' is removed, the build system will not pick it up either (it's off by default). But the option exists, so if someone wanted libc++ for whatever purpose, it seems reasonable enough to support it. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim,
I'm saying to _remove_ the `-stdlib=libc++` that's currently there.
Yes and I expect it to work. But I wrote the pip, and ideally, I get that fixed to allow what Tobias is doing. B/c if the '-stdlib=libc++' is removed, the build system will not pick it up either (it's off by default). But the option exists, so if someone wanted libc++ for whatever purpose, it seems reasonable enough to support it.
FreeBSD is fully a clang/libc++ system (both kernel and userland), and it would be very nice to support it=) However, my problem right now is: I cannot get the basic example working on my desktop (Ubuntu 16.04) with any toolchain: 1. gcc 5 / libstdc++ 2. gcc 4.9 / libstdc++ 3. clang / libc++ 4. clang / libstdc++ 1/3/4 fails building cppy. 2 builds cppyy, but the example bails out. So I am stuck =( What toolchain/system are you using? Do you have a Docker image that works? Cheers, /Tobias
Best regards, Wim
Hi Tobias,
1. gcc 5 / libstdc++ 2. gcc 4.9 / libstdc++ 3. clang / libc++ 4. clang / libstdc++
1/3/4 fails building cppy.
how does 3 fail? I presume differently than 1, 4?
2 builds cppyy, but the example bails out.
In which way?
What toolchain/system are you using?
Gentoo, SuSE and Mac. But the build files are taken from a larger project that I've stripped down. The larger project has a plethora of systems that have nightly builds: http://cdash.cern.ch/index.php?project=ROOT So in theory, it should work (famous last words).
Do you have a Docker image that works?
No, haven't. ROOT itself is part of most distro's. I think it's called "root-cern" on Ubuntu? That only gets you most of the way, however: there's also a clingcwrapper.cxx (in the pip and the cxx source) that provides all the C entry points that the PyPy cppyy module needs and it's not currently in the ROOT builds (yet). Sorry for the trouble, but it hasn't even been announced, so you're if not the first, then one of the first users of this pip. :} Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, Am 17.01.2017 um 20:01 schrieb wlavrijsen@lbl.gov:
Hi Tobias,
1. gcc 5 / libstdc++ 2. gcc 4.9 / libstdc++ 3. clang / libc++ 4. clang / libstdc++
1/3/4 fails building cppy.
how does 3 fail? I presume differently than 1, 4?
https://gist.github.com/oberstet/1771d9142cf566c5a877de0df5500899 Note: right now, I try what Ryan suggests in his new reply .. CXX=clang++ (not clang) - I'd expected that clang is clever enough to autoselect/do the right thing, but ..
2 builds cppyy, but the example bails out.
In which way?
https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870
What toolchain/system are you using?
Gentoo, SuSE and Mac. But the build files are taken from a larger project that I've stripped down. The larger project has a plethora of systems that have nightly builds:
Wow.
So in theory, it should work (famous last words).
The build matrix seems to include Ubuntu 16.04 / gcc 5 http://cdash.cern.ch/buildSummary.php?buildid=314808 .. so it "should work";)
Do you have a Docker image that works?
No, haven't. ROOT it
FWIW, I do think that would be very useful to get started. All this C++ machinery, combined with PyPy etc etc is tricky. A "known good" can be very helpful, if only as a starting point .. self is part of most distro's. I think it's called
"root-cern" on Ubuntu? That only gets you most of the way, however: there's also a clingcwrapper.cxx (in the pip and the cxx source) that provides all the C entry points that the PyPy cppyy module needs and it's not currently in the ROOT builds (yet).
There is a lot in your words I don't understand;) I need to dig deeper ..
Sorry for the trouble, but it hasn't even been announced, so you're if not the first, then one of the first users of this pip. :}
Nothing to apologize! I am very well aware of / used to the pain that C++ stuff usually comes with;) Expected. Thanks for a cool tool! Cheers, /Tobias
Best regards, Wim
Hi Tobias,
https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870
those symbols (ZN4ROOT17TGenericClassInfoD1Ev and friends) live in libCore.so which is installed under site packages and you can link with that. However, the backend should add all those symbols and make them available to the linker (it's loaded with RTLD_GLOBAL), so not sure why it doesn't work as-is.
FWIW, I do think that would be very useful to get started.
Yes; but it's another thing for me to learn (albeit a good thing). Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
On Jan 17, 2017 12:37 PM, <wlavrijsen@lbl.gov> wrote: Hi Ryan, On Tuesday 2017-01-17 18:28, Ryan Gonzalez wrote:
I think you misunderstood me. Your build log shows:
no, I didn't. :) It's not my build log, it's Tobias's. *facepalm* I need to double-check who exactly I'm replying to... :/ I'm saying to _remove_ the `-stdlib=libc++` that's currently there.
Yes and I expect it to work. But I wrote the pip, and ideally, I get that fixed to allow what Tobias is doing. B/c if the '-stdlib=libc++' is removed, the build system will not pick it up either (it's off by default). But the option exists, so if someone wanted libc++ for whatever purpose, it seems reasonable enough to support it. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net -- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://kirbyfan64.github.io/
Am 17.01.2017 um 19:03 schrieb Ryan Gonzalez:
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
1) removing explicit stdlib selection leads to the following linker error https://gist.github.com/oberstet/f75b9618b12b1e61289b147fcfd90bc0#file-gistf... makefile target was: https://github.com/oberstet/scratchbox/blob/master/cpp/capnproto/test1/Makef... 2) explicitly selecting GNU c++ stdlib for clang results in https://gist.github.com/oberstet/df1f59b9ce674d8984ee5f9ce045dccb makefile target was: https://github.com/oberstet/scratchbox/blob/master/cpp/capnproto/test1/Makef... 3) when using the gcc-4.9 build of cppyy, and trying the basic example (http://doc.pypy.org/en/latest/cppyy.html#basic-bindings-example) leads to: https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870 --- so right now I cannot get the basic example working =(
\--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
On Jan 17 2017, at 11:53 am, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:
Hi guys,
please see (reposting here to grab attention as hinted by Armin)
https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5 -and-clang
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain. Need to see how far I get with captnproto now ..
Cheers, /Tobias
pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
In your Makefile, change this: CXX=`which clang` \ to: CXX=`which clang++` \ -- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://kirbyfan64.github.io/ On Jan 17, 2017 12:41 PM, "Tobias Oberstein" <tobias.oberstein@gmail.com> wrote:
Am 17.01.2017 um 19:03 schrieb Ryan Gonzalez:
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
1) removing explicit stdlib selection leads to the following linker error
https://gist.github.com/oberstet/f75b9618b12b1e61289b147fcfd 90bc0#file-gistfile1-txt-L885
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L10
2) explicitly selecting GNU c++ stdlib for clang results in
https://gist.github.com/oberstet/df1f59b9ce674d8984ee5f9ce045dccb
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L27
3) when using the gcc-4.9 build of cppyy, and trying the basic example ( http://doc.pypy.org/en/latest/cppyy.html#basic-bindings-example) leads to:
https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870
---
so right now I cannot get the basic example working =(
\--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
On Jan 17 2017, at 11:53 am, Tobias Oberstein <tobias.oberstein@gmail.com
wrote:
Hi guys,
please see (reposting here to grab attention as hinted by Armin)
https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5
-and-clang
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain.
Need to see how far I get with captnproto now ..
Cheers,
/Tobias _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Ryan, with CXX=clang++ the build nearly finishes (I am a moron, sorry, clang isn't that clever), that is CC=`which clang` \ CXX=`which clang++` \ CXXFLAGS="-std=c++11" \ MAKE_NPROCS=4 \ ~/pypy-5.6-linux_x86_64-portable/bin/pip install --no-cache-dir --verbose PyPy-cppyy-backend I get to [ 95%] Linking CXX shared library ../../lib/libCling.so make[3]: Verzeichnis „/tmp/pip-build-EJ5mSe/PyPy-cppyy-backend/builddir“ wird verlassen [ 95%] Built target Cling but then it fails compiling clingcwrapper.cxx.o https://gist.github.com/oberstet/4f60bd02bab8921e09b94c47188c291c#file-gistf... This is oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ clang++ --version clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin Thanks for your help! Tobias Am 17.01.2017 um 20:02 schrieb Ryan Gonzalez:
In your Makefile, change this:
CXX=`which clang` \
to:
CXX=`which clang++` \
-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://kirbyfan64.github.io/
On Jan 17, 2017 12:41 PM, "Tobias Oberstein" <tobias.oberstein@gmail.com> wrote:
Am 17.01.2017 um 19:03 schrieb Ryan Gonzalez:
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
1) removing explicit stdlib selection leads to the following linker error
https://gist.github.com/oberstet/f75b9618b12b1e61289b147fcfd 90bc0#file-gistfile1-txt-L885
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L10
2) explicitly selecting GNU c++ stdlib for clang results in
https://gist.github.com/oberstet/df1f59b9ce674d8984ee5f9ce045dccb
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L27
3) when using the gcc-4.9 build of cppyy, and trying the basic example ( http://doc.pypy.org/en/latest/cppyy.html#basic-bindings-example) leads to:
https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870
---
so right now I cannot get the basic example working =(
\--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
On Jan 17 2017, at 11:53 am, Tobias Oberstein <tobias.oberstein@gmail.com
wrote:
Hi guys,
please see (reposting here to grab attention as hinted by Armin)
https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5
-and-clang
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain.
Need to see how far I get with captnproto now ..
Cheers,
/Tobias _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
This one looks like a bug in cppyy! Try going to clingcwrapper.cxx and changing line 525 from: cppresult->std::string::~string(); to: cppresult->std::string::~basic_string(); (e.g. change `~string` to `~basic_string`). \-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else <http://refi64.com/> On Jan 17 2017, at 1:51 pm, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:
Hi Ryan,
with CXX=clang++ the build nearly finishes (I am a moron, sorry, clang isn't that clever), that is
CC=`which clang` \ CXX=`which clang++` \ CXXFLAGS="-std=c++11" \ MAKE_NPROCS=4 \ ~/pypy-5.6-linux_x86_64-portable/bin/pip install --no-cache-dir \--verbose PyPy-cppyy-backend
I get to
[ 95%] Linking CXX shared library ../../lib/libCling.so make[3]: Verzeichnis „/tmp/pip-build-EJ5mSe/PyPy-cppyy-backend/builddir“ wird verlassen [ 95%] Built target Cling
but then it fails compiling clingcwrapper.cxx.o
https://gist.github.com/oberstet/4f60bd02bab8921e09b94c47188c291c#file- gistfile1-txt-L8
This is
oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ clang++ --version clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final) Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin
Thanks for your help! Tobias
Am 17.01.2017 um 20:02 schrieb Ryan Gonzalez:
In your Makefile, change this:
CXX=`which clang` \
to:
CXX=`which clang++` \
\-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else <http://kirbyfan64.github.io/>
On Jan 17, 2017 12:41 PM, "Tobias Oberstein" <tobias.oberstein@gmail.com> wrote:
Am 17.01.2017 um 19:03 schrieb Ryan Gonzalez:
What happens if you remove the `-stdlib=libc++`? Seems like that's where the Clang build error is coming from.
1) removing explicit stdlib selection leads to the following linker error
https://gist.github.com/oberstet/f75b9618b12b1e61289b147fcfd 90bc0#file-gistfile1-txt-L885
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L10
2) explicitly selecting GNU c++ stdlib for clang results in
https://gist.github.com/oberstet/df1f59b9ce674d8984ee5f9ce045dccb
makefile target was: https://github.com/oberstet/sc ratchbox/blob/master/cpp/capnproto/test1/Makefile#L27
3) when using the gcc-4.9 build of cppyy, and trying the basic example ( <http://doc.pypy.org/en/latest/cppyy.html#basic-bindings-example)> leads to:
https://gist.github.com/oberstet/d260ee15c81954bea8298b7400d04870
\---
so right now I cannot get the basic example working =(
\\--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
On Jan 17 2017, at 11:53 am, Tobias Oberstein <tobias.oberstein@gmail.com
wrote:
Hi guys,
please see (reposting here to grab attention as hinted by Armin)
https://bitbucket.org/pypy/pypy/issues/2467/cppyy-fails-to-build-on-gcc-5
-and-clang
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain.
Need to see how far I get with captnproto now ..
Cheers,
/Tobias _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Ryan, Am 17.01.2017 um 20:56 schrieb Ryan Gonzalez:
This one looks like a bug in cppyy! Try going to clingcwrapper.cxx and changing line 525 from:
Yeah, that helps: the issue at line 525 is gone, but the 2nd issue at line 715 persists: /tmp/pip-BVnCrW-build/src/backend/cppyy/src/clingcwrapper.cxx:715:35: error: invalid operands to binary expression ('basic_ostream<char, std::char_traits<char> >' and 'std::ostringstream' (aka 'basic_ostringstream<char>')) std::cerr << "Warning: " << msg << '\n'; ~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~ https://gist.github.com/oberstet/271719f569087f423197b6d8808ea6f1 /Tobias
Whoops, missed that one! Change line 715 to: std::cerr << "Warning: " << msg.str() << '\n'; \-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else <http://refi64.com/> On Jan 17 2017, at 3:30 pm, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:
Hi Ryan,
Am 17.01.2017 um 20:56 schrieb Ryan Gonzalez: This one looks like a bug in cppyy! Try going to clingcwrapper.cxx and changing line 525 from:
Yeah, that helps: the issue at line 525 is gone, but the 2nd issue at
line 715 persists:
/tmp/pip-BVnCrW-build/src/backend/cppyy/src/clingcwrapper.cxx:715:35: error: invalid operands to binary expression ('basic_ostream<char, std::char_traits<char> >' and 'std::ostringstream' (aka 'basic_ostringstream<char>')) std::cerr << "Warning: " << msg << '\n'; ~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~
https://gist.github.com/oberstet/271719f569087f423197b6d8808ea6f1
/Tobias
Am 17.01.2017 um 22:39 schrieb Ryan Gonzalez:
Whoops, missed that one! Change line 715 to: std::cerr << "Warning: " << msg.str() << '\n';
Success! With these 2 fixes, it now builds using clang (with GNU libstdc++): -- Set runtime path of "/home/oberstet/pypy-5.6-linux_x86_64-portable/site-packages/cppyy_backend/lib/libcppyy_backend.so" to "" install finished done Removing source in /tmp/pip-YrtrlD-build Successfully installed PyPy-cppyy-backend-6.6.9.0 Cleaning up... oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1/PyPy-cppyy-backend-6.6.9.0$ -- However, the basic example spills out weird things: oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ pypy MyClass.py unknown type code: P 1 unknown type code: � -178968160 77 Here: https://gist.github.com/oberstet/a3814c77e5400df3456d7ac74a26bdf7 Code is all here: https://github.com/oberstet/scratchbox/tree/master/cpp/capnproto/test1 Example was built like this: https://github.com/oberstet/scratchbox/tree/master/cpp/capnproto/test1#test --
\--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
On Jan 17 2017, at 3:30 pm, Tobias Oberstein <tobias.oberstein@gmail.com> wrote:
Hi Ryan,
Am 17.01.2017 um 20:56 schrieb Ryan Gonzalez: This one looks like a bug in cppyy! Try going to clingcwrapper.cxx and changing line 525 from:
Yeah, that helps: the issue at line 525 is gone, but the 2nd issue at line 715 persists:
/tmp/pip-BVnCrW-build/src/backend/cppyy/src/clingcwrapper.cxx:715:35: error: invalid operands to binary expression ('basic_ostream<char, std::char_traits<char> >' and 'std::ostringstream' (aka 'basic_ostringstream<char>')) std::cerr << "Warning: " << msg << '\n'; ~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~
https://gist.github.com/oberstet/271719f569087f423197b6d8808ea6f1
/Tobias
Hi Ryan, Tobias, On Tuesday 2017-01-17 21:39, Ryan Gonzalez wrote:
Whoops, missed that one! Change line 715 to: std::cerr << "Warning: " << msg.str() << '\n';
indeed, thanks! Now fixed in the repository. Hopefully up soon in the pip (takes more testing, hence time) ... Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, Am 18.01.2017 um 01:34 schrieb wlavrijsen@lbl.gov:
Hi Ryan, Tobias,
On Tuesday 2017-01-17 21:39, Ryan Gonzalez wrote:
Whoops, missed that one! Change line 715 to: std::cerr << "Warning: " << msg.str() << '\n';
indeed, thanks! Now fixed in the repository. Hopefully up soon in the pip (takes more testing, hence time) ...
Awesome, thanks! However, I still can't get the basic example working =( It spills out weird things: oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ pypy MyClass.py unknown type code: P 1 unknown type code: � -178968160 77 This doesn't seem right. Here: https://gist.github.com/oberstet/a3814c77e5400df3456d7ac74a26bdf7 Code is all here: https://github.com/oberstet/scratchbox/tree/master/cpp/capnproto/test1 Example was built like this: https://github.com/oberstet/scratchbox/tree/master/cpp/capnproto/test1#test Cheers, /Tobias
Best regards, Wim
Hi Tobias,
unknown type code: P 1 unknown type code: ??? -178968160 77
that falls in the category "should never happen." Arguments are passed as a patch of memory, with a type code indicating how the individual arguments should be treated. The only way this can be wrong, is if the offset is wrong. I don't understand how that can happen: the offset is provided at run-time by the backend, where it is compiled in. The pypy-side module then uses that offset, so there should be only one value throughout. Still, is pypy-c build with a different compiler than the pip? Not that I think that's the problem, but maybe it provides some clues. Also, can you try the following (this is not a public interface): % pypy-c
import cppyy cppyy.gbl.gInterpreter.Declare("""class MyClass { .... public: .... MyClass(int i = -99) : m_myint(i) {} .... .... int GetMyInt() { return m_myint; } .... void SetMyInt(int i) { m_myint = i; } .... .... public: .... int m_myint; .... };""") True from cppyy.gbl import MyClass m = MyClass(42) print m.m_myint 42
(The reason this is not a public interface, is b/c I want to hide all this behind an interface that mimics cffi, call it cppffi. Lacking time, though, so if anyone is interested? :) ) Anyway, I expect it to give the same problem, but if it does, it excludes one other place where the bug could hide. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, Am 18.01.2017 um 18:34 schrieb wlavrijsen@lbl.gov:
Hi Tobias,
unknown type code: P 1 unknown type code: ??? -178968160 77
that falls in the category "should never happen." Arguments are passed as a patch of memory, with a type code indicating how the individual arguments should be treated. The only way this can be wrong, is if the offset is wrong. I don't understand how that can happen: the offset is provided at run-time by the backend, where it is compiled in. The pypy-side module then uses that offset, so there should be only one value throughout.
Still, is pypy-c build with a different compiler than the pip? Not that I
Yes. I am using https://github.com/squeaky-pl/portable-pypy and this is built with GCC 6.2.0. As there isn't a standard C++ ABI, this would explain all sorts of weird things happening. Or does clang deliberately try to follow gcc ABI? I am aware that the term "C++ ABI" is an oxymoron in a way .. what's the ABI of a templated template? =)
Also, can you try the following (this is not a public interface):
import cppyy cppyy.gbl.gInterpreter.Declare(""" .... class MyClass { .... public: .... MyClass(int i = -99) : m_myint(i) {} .... int GetMyInt() { return m_myint; } .... void SetMyInt(int i) { m_myint = i; } .... public: .... int m_myint; .... };""") Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: <class '__main__.::'> object has no attribute 'gInterpreter' (details: "function 'cppyy_get_template' not found in
Doesn't work for me: oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ pypy MyClass.py unknown type code: � 1 unknown type code: 1294245728 77 oberstet@thinkpad-t430s:~/scm/oberstet/scratchbox/cpp/capnproto/test1$ pypy Python 2.7.12 (aff251e543859ce4508159dd9f1a82a2f553de00, Nov 12 2016, 08:50:18) [PyPy 5.6.0 with GCC 6.2.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. library 'libcppyy_backend.so'")
-- Given all these problems even running a basic example (I haven't even got to capnproto generated C++ .. and this is heavily templated code, so even more will go wrong), and given that I have a specific use case (captnproto) rather than a general need to wrap C++, I think I will evaluate capnpy .. -- But I think we didn't waste time: - 2 bugs identified and fixed in cppyy - using clang _might_ work, but pypy binaries are gcc most of the time, and that leads to more issues So I think gcc 5 support is more important: "Found GCC compiler with new ABI. Cling does not support the GCC 5 ABI yet" The bundled Cling is too old? I am totally guessing here .. in fact I don't really know what I am doing;) Cheers, /Tobias
Hi Tobias,
Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: <class '__main__.::'> object has no attribute 'gInterpreter' (details: "function 'cppyy_get_template' not found in library 'libcppyy_backend.so'")
okay, problem understood: the version of pypy-c that you have does not support the Cling backend (there is no release of that yet: next pypy). See, 'cppyy_get_template' is from the Reflex era: it no longer exists. So in your case the cppyy module (the RPython portion inside pypy-c) is doing Reflex-thingies while being fed Cling-thingies from the (C++) backend. The real bug here is that I need to have a version check in cppyy. I did plan to, but didn't get around implementing it. Sorry about that! Btw., if it interests you: the reason that 'cppyy_get_template' disappeared is that Cling supports proper overloading of templated and non-templated methods (including instantiation if necessary). So a returned method may well be templated, and thus the relevant question is "is_template". That is, if a method exists, it may be either templated or not, but an additional call to look for a templated method would be pointless. As for ABI-compatibility, it's not that bad: contemporary compilers on the same OS are designed to be compatible. Their standard libraries are a different matter. That said, cppyy per se should not be hampered by that, as all the offsets are provided to the module by the backend at runtime. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim,
okay, problem understood: the version of pypy-c that you have does not support the Cling backend (there is no release of that yet: next pypy).
Ah ..
See, 'cppyy_get_template' is from the Reflex era: it no longer exists. So in your case the cppyy module (the RPython portion inside pypy-c) is doing Reflex-thingies while being fed Cling-thingies from the (C++) backend.
The real bug here is that I need to have a version check in cppyy. I did plan to, but didn't get around implementing it. Sorry about that!
No problem .. great that it will be in the next release! Also, fixing the gcc5 issue I have on Ubuntu 16.04 would be really worth it, not least because this is such a common (developer) platform nowerdays. If that would have worked for me, I wouldn't have run down the clang rabbit hole;) Notwithstanding that clang/libc++ support would _also_ be cool (because of FreeBSD ..)
As for ABI-compatibility, it's not that bad: contemporary compilers on the same OS are designed to be compatible. Their standard libraries are a different matter. That said, cppyy per se should not be hampered by that, as all the offsets are provided to the module by the backend at runtime.
Ok, sounds good .. I believe. I have skimmed into your paper .. heavy stuff .. you obviously know more than me here;) Sadly, I don't have time to really dig into it. Thanks for all your assistence and time! Cheers, /Tobias
Hi, On Tue, Jan 17, 2017 at 5:51 PM, Tobias Oberstein < tobias.oberstein@gmail.com> wrote:
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain. Need to see how far I get with captnproto now ..
if you are interested in capnproto, you might want to try capnpy: it's written in pure python and it has been designed to be super fast on pypy (and it's very fast on CPython as well): https://github.com/antocuni/capnpy http://capnpy.readthedocs.io/en/latest/
Hi Antonio,
Note: for now I am fine, I managed to build it using gcc 4.9 toolchain. Need to see how far I get with captnproto now ..
if you are interested in capnproto, you might want to try capnpy: it's written in pure python and it has been designed to be super fast on pypy (and it's very fast on CPython as well): https://github.com/antocuni/capnpy http://capnpy.readthedocs.io/en/latest/
Fantastic!! Actually, if this is true (performance), this would be wonderful - I can spare my nerves (hello C++, you are a moooonster!), and eat my cake .. The benchmarks http://capnpy.readthedocs.io/en/latest/benchmarks.html for scalar attribute access look awesome (close to mere instance access). Structured value access (lists/dict): bigger difference. I am also concerned about GC pressure - is it still "zero copy"? Note: I am purely interested in performance on PyPy .. In general: I thought it would be a good idea to use capnproto C++ generator, and then cppyy to get the best performance (on pypy). Given there is "antocuni/capnpy", do you think this is a pointless endeavour? -- Anyways, thanks so much for that hint - I will try it. (I will try the other hints given to me by Ryan and Wim too, since I already consumed their time, and maybe there is some gain/insight) Cheers, /Tobias
Hi Tobias, On Tue, Jan 17, 2017 at 7:27 PM, Tobias Oberstein < tobias.oberstein@gmail.com> wrote:
Hi Antonio,
Fantastic!!
Actually, if this is true (performance), this would be wonderful - I can spare my nerves (hello C++, you are a moooonster!), and eat my cake ..
The benchmarks
http://capnpy.readthedocs.io/en/latest/benchmarks.html
for scalar attribute access look awesome (close to mere instance access).
Structured value access (lists/dict): bigger difference. I am also concerned about GC pressure - is it still "zero copy"?
yes: under the hood, capnpy objects are represented as immutable strings; then the generated classes contain accessors which look like this (overly simplified): class Point(_Struct): @property def x(self): offset = statically_known_x_offset + self._offset return struct.unpack_from('q', self._buf.s, offset) where self._buf.s is a string. The nice thing is that the pypy JIT does a very good job at optimizing struct.unpack_from: if you look at the generated code, you see that it loads the 8 bytes directly from the in-memory buffer. So, it's very close to optimal performance. For lists is the same: when you look up a list field, you get a wrapper around the very same buffer, then the custom __getitem__ does the actual lookup in memory. Note that currently lists on pypy are slowish, but I should be able to solve the problem in the near future. The only exception are strings: getting a text attribute means to take a slice of the buffer. However, if the returned value is short-lived, the JIT might be able to optimize the slicing away. (note: there is no automatic conversion to unicode, although I might add an option for that in the future). I don't understand what you mean by dicts are there are no dictionaries in capnproto. Note also that capnpy is serialization only: there is no support for RPC stuff.
Note: I am purely interested in performance on PyPy ..
In general: I thought it would be a good idea to use capnproto C++ generator, and then cppyy to get the best performance (on pypy). Given there is "antocuni/capnpy", do you think this is a pointless endeavour?
The original goal of capnpy was to be as fast as possible on PyPy. However, if you find that C++ + cppyy is faster, I'd be very interested to know :). ciao, Anto
On 18 January 2017 at 21:02, Antonio Cuni <anto.cuni@gmail.com> wrote:
Hi Tobias,
Note also that capnpy is serialization only: there is no support for RPC stuff.
Although if you want to work on it, I've got some preliminary support for it in my fork that I mostly ripped from an internal CapTP implementation I wrote at work. I wasn't going to look at it in the short term, but if there's interest I can get back to it, fix interface generation and rewrite it to be SansIO but otherwise feel free to use whatever code you like from it. https://bitbucket.org/william_ml_leslie/capnpy I'd love too see CapnProto hooked up to Autobahn/WS on Twisted too! I don't think there's a good client-side implementation of CapnProto though, so there's still a missing part of that pipeline. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement.
Hi William, Am 18.01.2017 um 14:44 schrieb William ML Leslie:
On 18 January 2017 at 21:02, Antonio Cuni <anto.cuni@gmail.com> wrote:
Hi Tobias,
Note also that capnpy is serialization only: there is no support for RPC stuff.
Although if you want to work on it, I've got some preliminary support for it in my fork that I mostly ripped from an internal CapTP implementation I wrote at work. I wasn't going to look at it in the short term, but if there's interest I can get back to it, fix interface generation and rewrite it to be SansIO but otherwise feel free to use whatever code you like from it.
https://bitbucket.org/william_ml_leslie/capnpy
I'd love too see CapnProto hooked up to Autobahn/WS on Twisted too! I don't think there's a good client-side implementation of CapnProto though, so there's still a missing part of that pipeline.
Thanks for the pointer! I don't have a use case for the RPC part of Captnproto though. Autobahn/Crossbar.io has WAMP, which already has RPC, more so _routed RPC_. And there are more than a dozen client side WAMP implementations. I just need a ultra-fast way for IPC between Python processes .. doing away with serialization/deserialization overhead and copying. Cheers, /Tobias
Hi Anto, On Wednesday 2017-01-18 10:02, Antonio Cuni wrote:
The original goal of capnpy was to be as fast as possible on PyPy. However, if you find that C++ + cppyy is faster, I'd be very interested to know :).
since you wrote the initial data access part yourself way back when, I'd expect you to fix cppyy if it were slower! :) (Only changes I made is to support multiple virtual inheritance and to specialize to make sure the common case is not slowed down by that.) But the only time that I have found where C++/cppyy is faster on PyPy than pure Python is for data member access precisely b/c the C++ structs have fixed offsets, and you seem to have covered that case by specializing. See this report, figure 4: http://conferences.computer.org/pyhpc/2016/papers/5220a027.pdf But, in cppyy, if there is string handling going on, with C-strings being a stand-in for "byte*" (a type that was again voted down, so is not coming soon), there may be spurious copies if not otherwise specialized. Best regards, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Wim, On Wed, Jan 18, 2017 at 5:09 PM, <wlavrijsen@lbl.gov> wrote:
since you wrote the initial data access part yourself way back when, I'd expect you to fix cppyy if it were slower! :)
that's a fair point, indeed :). But on top of that you need to put a layer which exposes a pythonic interface (for example, offering list-like classes with an __iter__ and a __getitem__). So I have no idea of how the final speed of the thing will be, until someone tries and measure :). ciao, Anto
Anto, On Thursday 2017-01-19 10:52, Antonio Cuni wrote:
But on top of that you need to put a layer which exposes a pythonic interface (for example, offering list-like classes with an __iter__ and a __getitem__). So I have no idea of how the final speed of the thing will be, until someone tries and measure :).
yes, I did something like that for std::vector (C++11 guarantees that the memory it contains is contiguous). Still, I'm going to compare what I did with your code as std::vector in C++ is as efficient as it gets, so there is a lot of ground to cover. Thanks, Wim -- WLavrijsen@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net
Hi Antonio,
The only exception are strings: getting a text attribute means to take a slice of the buffer. However, if the returned value is short-lived, the JIT might be able to optimize the slicing away. (note: there is no automatic conversion to unicode, although I might add an option for that in the future).
Are you aware of https://github.com/alex/zero_buffer ? This emulates Python strings using zero-copy read-only buffer views. I think zero_buffer only supports byte strings not unicode. But for that, maybe it would be cool to have capnpy return zero_buffers upon byte string attribute access, because that should finally provide zero copy .. Caveat: this is a hypothesis I haven't really explored / thought through .. -- Holy grail for me (use case being IPC) would be: Python process 1 mmap's a file shared with Python process 2. Python process 1 puts a string into mmap'ed file, pointer to that is "somehow transferred" to process 2 (eg pushing the index into the mmap'ed file over Unix domain socket .. a single uint64), and Python code in process 2 can do stuff with this string _without_ copying - probably via zero_buffer.
I don't understand what you mean by dicts are there are no dictionaries in capnproto.
Right. Structs.
Note also that capnpy is serialization only: there is no support for RPC stuff.
That's fine .. I don't need RPC (we have WAMP for that in Autobahn/Crossbar.io).
Note: I am purely interested in performance on PyPy ..
In general: I thought it would be a good idea to use capnproto C++ generator, and then cppyy to get the best performance (on pypy). Given there is "antocuni/capnpy", do you think this is a pointless endeavour?
The original goal of capnpy was to be as fast as possible on PyPy. However, if you find that C++ + cppyy is faster, I'd be very interested to know :).
Sure .. right now I cannot progress on cppyy as I can't get the basic example running.. Cheers, /Tobias
ciao, Anto
Hi Tobias, On Wed, Jan 18, 2017 at 5:27 PM, Tobias Oberstein < tobias.oberstein@gmail.com> wrote:
Are you aware of
https://github.com/alex/zero_buffer
?
This emulates Python strings using zero-copy read-only buffer views.
yes, I saw it in the past and I considered to use it for capnpy. IIRC, I measured that at the end of the days, the overhead of using it was larger than simply doing string slicing, especially for short strings. It might be useful for very large strings, however. Depending of what you need to do, for zero copy you could also consider returning a memoryview slice of the original underlying buffer.
Holy grail for me (use case being IPC) would be:
Python process 1 mmap's a file shared with Python process 2.
Python process 1 puts a string into mmap'ed file, pointer to that is "somehow transferred" to process 2 (eg pushing the index into the mmap'ed file over Unix domain socket .. a single uint64), and Python code in process 2 can do stuff with this string _without_ copying - probably via zero_buffer.
Have you actually measured that copying the data between processes is the bottleneck? Using shared memory is something I tried also for a client of mine but at the end we switched back to pass messages using network because the extra complexity was not worth the gain. But again, I suppose it depends on the size of the message. ciao, Anto
Holy grail for me (use case being IPC) would be:
Python process 1 mmap's a file shared with Python process 2.
Python process 1 puts a string into mmap'ed file, pointer to that is "somehow transferred" to process 2 (eg pushing the index into the mmap'ed file over Unix domain socket .. a single uint64), and Python code in process 2 can do stuff with this string _without_ copying - probably via zero_buffer.
Have you actually measured that copying the data between processes is the bottleneck? Using shared memory is something I tried also for a client of mine but at the end we switched back to pass messages using network because the extra complexity was not worth the gain. But again, I suppose it depends on the size of the message.
I have measured (and profiled using vmprof) that we have bottlenecks in serialization/deserialization .. the overhead is lowest (on PyPy) when using a pure CBOR implementation, but still. Hence my interest in captnproto/flatbuffers. I also benchmarked Unix domain socket pure bandwidth (in C) .. which is very high indeed (UDS only involves a single copy when done in C, not 2 copies like loopback TCP). I probably should stop worrying/guessing and try capnpy over UDS and see how far this will get us;) Cheers, /Tobias
ciao, Anto
participants (5)
-
Antonio Cuni -
Ryan Gonzalez -
Tobias Oberstein -
William ML Leslie -
wlavrijsen＠lbl.gov