Extension module difficulty w/pgen.
Hi all, I know this is what I get for trying to integrate pgen into an extension module: I can't get it to link properly. I first saw the following problem on a FreeBSD box. Now, I have the following two external dependencies in my extension module (.../src/Modules/pgenmodule.c): extern grammar * _Py_pgen (node * n); extern grammar * _Py_meta_grammar (void); I added an entry to setup.py for it: exts.append( Extension('pgen', ['pgenmodule.c'])) Now when I run make, the extension module is not built, with the system complaining about being unable to resolve "_Py_meta_grammar", but not "_Py_pgen". When I run nm, I can see both symbols in libpython.2.3.a (these symbols are in pgen.c and metagrammar.c, both of which have been added to the libpython build target): ~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_pgen [26] | 2676| 48|FUNC |GLOB |0 |2 |_Py_pgen ~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_meta [36] | 0| 12|FUNC |GLOB |0 |2 |_Py_meta_grammar On the FreeBSD box, I was able to add "-L. -lpython2.3" to the command line, and this builds. However, when I use this hack on a Solaris platform, it complains about being unable to reserve a text offset for most if not all of the symbols in libpython. It seems to me that I should not have to use this workaround, which only works on one of the systems I use. Does anyone have an idea as to what I should do now? I am a bit confused by this, since Fred Drake's parser extension does not require any of this wackiness. As an aside, the code for the modules I am working on and the diffs are on Sourceforge (PEP 269 implementation), so you can play too, if so inclined. Thanks! -Jon
I know this is what I get for trying to integrate pgen into an extension module: I can't get it to link properly. I first saw the following problem on a FreeBSD box.
Now, I have the following two external dependencies in my extension module (.../src/Modules/pgenmodule.c):
extern grammar * _Py_pgen (node * n); extern grammar * _Py_meta_grammar (void);
I added an entry to setup.py for it:
exts.append( Extension('pgen', ['pgenmodule.c']))
Now when I run make, the extension module is not built, with the system complaining about being unable to resolve "_Py_meta_grammar", but not "_Py_pgen".
Maybe you only get an error for the first unresolved symbol?
When I run nm, I can see both symbols in libpython.2.3.a (these symbols are in pgen.c and metagrammar.c, both of which have been added to the libpython build target):
~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_pgen [26] | 2676| 48|FUNC |GLOB |0 |2 |_Py_pgen ~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_meta [36] | 0| 12|FUNC |GLOB |0 |2 |_Py_meta_grammar
Linux nm output looks very different, so I don't know what this means. Are you *sure* it doesn't mean that there are global references but no definitions for these symbols? And what does the 0 in the second column for _Py_meta_grammar mean?
On the FreeBSD box, I was able to add "-L. -lpython2.3" to the command line, and this builds. However, when I use this hack on a Solaris platform, it complains about being unable to reserve a text offset for most if not all of the symbols in libpython.
It seems to me that I should not have to use this workaround, which only works on one of the systems I use. Does anyone have an idea as to what I should do now? I am a bit confused by this, since Fred Drake's parser extension does not require any of this wackiness.
As an aside, the code for the modules I am working on and the diffs are on Sourceforge (PEP 269 implementation), so you can play too, if so inclined.
Maybe the problem is that nothing else uses these symbols? Try sticking dummy references (e.g. an unreachable call) to them in main.c, to see if that makes a difference. I recall we had to do this for something else that wasn't used by Python itself. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 27 Sep 2002, Guido van Rossum wrote:
Maybe you only get an error for the first unresolved symbol?
Yup. When I comment out the references to _Py_meta_grammar(), it still complains about not being able to see _Py_pgen().
When I run nm, I can see both symbols in libpython.2.3.a (these symbols are in pgen.c and metagrammar.c, both of which have been added to the libpython build target):
~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_pgen [26] | 2676| 48|FUNC |GLOB |0 |2 |_Py_pgen ~/cvs/python/dist/src> nm ./libpython2.3.a | grep _Py_meta [36] | 0| 12|FUNC |GLOB |0 |2 |_Py_meta_grammar
Linux nm output looks very different, so I don't know what this means. Are you *sure* it doesn't mean that there are global references but no definitions for these symbols? And what does the 0 in the second column for _Py_meta_grammar mean?
Come on Guido, I thought you were at one point a Solaris hack ;-). FUNC and GLOB means it is a global function defined in the module. The second column is some sort of memory offset value, and incidently the third column is the byte size reserved for the object (i.e. the objects should be in the library, and are not just place holders). Here is the output from Linux (where I have just duplicated the problem): $ nm libpython2.3.a | egrep "_Py_(meta|pgen)" 00000000 T _Py_meta_grammar 00000e38 T _Py_pgen If I understood the GNU info file for binutils, this means that the symbols are defined in the text segment, and should be available for external linkage.
Maybe the problem is that nothing else uses these symbols? Try sticking dummy references (e.g. an unreachable call) to them in main.c, to see if that makes a difference. I recall we had to do this for something else that wasn't used by Python itself.
I tried this just now, but to no avail. Maybe I am not being thorough enough. If the linker is excluding these symbols because they are not used, why would nm seem to say they are there, and why would statically linking libpython work (on FreeBSD, anyway)? Conversely, I seem to remember this working on an earlier, but abandoned attempt I made on a Linux box. Maybe I just need more vacation. :P Thanks! -Jon
On Fri, 27 Sep 2002, Guido van Rossum wrote: <
Maybe the problem is that nothing else uses these symbols? Try sticking dummy references (e.g. an unreachable call) to them in main.c, to see if that makes a difference. I recall we had to do this for something else that wasn't used by Python itself.
I tried this just now, but to no avail. Maybe I am not being thorough enough. If the linker is excluding these symbols because they are not used, why would nm seem to say they are there, and why would statically linking libpython work (on FreeBSD, anyway)? Conversely, I seem to remember this working on an earlier, but abandoned attempt I made on a Linux box.
It turns out I wasn't being thorough enough. I used nm on my python build, and didn't see the symbols there. I moved the dummy calls to a global function in python.c and only then were the symbols linked into the interpreter. Keeping a dummy function in one of the core modules doesn't seem like a terribly elegant solution, even if it allows me to keep developing the pgen module. What would you suggest be done to ensure that statically linked builds link those symbols? I would assume that since the required symbols *are* in libpython (per my modifications to Makefile.pre.in), building a python using dynamic libraries would allow the extension module to "see" those symbols. You had mentioned doing something like this before. Is there some linkage graveyard where I can bury calls to these symbols in order to ensure they are linked? Or are some of those fancy API macros used to ensure linkage (perhaps by API functions that are utilities for extension writers and not needed by the python core)? Thanks! -Jon
Maybe the problem is that nothing else uses these symbols? Try sticking dummy references (e.g. an unreachable call) to them in main.c, to see if that makes a difference. I recall we had to do this for something else that wasn't used by Python itself.
I tried this just now, but to no avail. Maybe I am not being thorough enough. If the linker is excluding these symbols because they are not used, why would nm seem to say they are there, and why would statically linking libpython work (on FreeBSD, anyway)? Conversely, I seem to remember this working on an earlier, but abandoned attempt I made on a Linux box.
It turns out I wasn't being thorough enough. I used nm on my python build, and didn't see the symbols there. I moved the dummy calls to a global function in python.c and only then were the symbols linked into the interpreter.
Keeping a dummy function in one of the core modules doesn't seem like a terribly elegant solution, even if it allows me to keep developing the pgen module. What would you suggest be done to ensure that statically linked builds link those symbols? I would assume that since the required symbols *are* in libpython (per my modifications to Makefile.pre.in), building a python using dynamic libraries would allow the extension module to "see" those symbols.
Yes, I think building a shared lib would work. But we don't build shared libs for all platforms.
You had mentioned doing something like this before. Is there some linkage graveyard where I can bury calls to these symbols in order to ensure they are linked? Or are some of those fancy API macros used to ensure linkage (perhaps by API functions that are utilities for extension writers and not needed by the python core)?
No, AFAIK you have to create a dummy reference somewhere. I'd suggest adding it to the end of Python/pythonrun.c. --Guido van Rossum (home page: http://www.python.org/~guido/)
Jonathan Riehl <jriehl@spaceship.com> writes:
It seems to me that I should not have to use this workaround, which only works on one of the systems I use. Does anyone have an idea as to what I should do now?
It appears that metagrammar.o is not needed in the Python executable, that's why the linker does not fetch it from the library when building Python. As for pgen, the first question would be why you need metagrammar.o in your extension module. Assuming there is a good reason to expose it, you should arrange to 1. exclude metagrammar.o from libpython.a, 2. include it explicitly into as a source for building the pgen module. HTH, Martin
participants (3)
-
Guido van Rossum -
Jonathan Riehl -
martin＠v.loewis.de