Sjoerd Mullender writes:

David Abrahams wrote:

...

Apparently Python on some linux distros is being linked by g++ rather than gcc, resulting in the C++ runtime library being linked into Python; this has bad consequences for compatibility between C++ extension modules and Pythons that have been built with different versions of GCC. Is this behavior intentional?

Configure with --without-cxx to not use g++. Since there is an option in configure, I assume it is intentional.

O-kay... any idea what the rationale for this decision might be? -- Dave Abrahams Boost Consulting www.boost-consulting.com

Skip Montanaro writes:

>> Configure with --without-cxx to not use g++. Since there is an >> option in configure, I assume it is intentional.

Dave> O-kay... any idea what the rationale for this decision might be?

I believe it's so that people can link in libraries written in C++ and have them initialized properly.

Can you give specifics? What do you mean by "link in?" Do you mean "statically link into the Python interpreter," or something else? Boost.Python is a library written in C++ and I've never had trouble using it with a Python executable... until I ran into a Python that was linked with libstdc++! -- Dave Abrahams Boost Consulting www.boost-consulting.com

Skip Montanaro writes:

>> I believe it's so that people can link in libraries written in C++ >> and have them initialized properly.

Dave> Can you give specifics? What do you mean by "link in?" Do you Dave> mean "statically link into the Python interpreter," or something Dave> else?

Probably not. I'm not a C++ guy. My understanding is that global (maybe static?) C++ objects need the help of C++'s version of crt0 to get properly initialized at program start.

Yes.

If there is some library with such objects that happens to get wrapped and dynamically linked into a Python interpreter

Whoa there. How would that ever happen under ordinary circumstances? Doesn't Python's makefile control what gets linked to Python?

that was linked with a regular C linker (and thus had a C crt0), that initialization wouldn't happen.

Right, and linking would fail, IIRC. It seems to me that if someone decides to build a wacky Python executable that links in C++ code directly, they can afford to use a configure option. There's no need to punish all the other writers of C++ extension modules out there by tying the default build of Python to a particular version of libstdc++.

Dave> Boost.Python is a library written in C++ and I've never had Dave> trouble using it with a Python executable... until I ran into a Dave> Python that was linked with libstdc++!

Sorry, I can't help. I'm just recounting my remembering of the reasons for C++ linkage. Personally, I avoid C++ as much as I can...

If there's someone around here who is responsible for this change and knows its real rationale, can (s)he please tell me what it is? If not, can we please change things back so Python doesn't get linked to the C++ runtime by default? Thanks, -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Martin v. Löwis" writes:

David Abrahams wrote:

...

...

If there is some library with such objects that happens to get wrapped and dynamically linked into a Python interpreter

Whoa there. How would that ever happen under ordinary circumstances? Doesn't Python's makefile control what gets linked to Python?

Not entirely. By extending Modules/Setup

You mean http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/Setup.d... ?

e.g. in the way freeze works), it is well possible to have additional extension modules linked into the Python interpreter, extension modules which are not part of the standard Python distribution.

In fact, before Python supported dynamic loading of extension modules, this was the *only* way to use non-standard extension modules. You always had to build your own version of the Python interpreter. I believe ccpython.cc dates back to these times.

That explains a lot. I contend that either: a. Anyone making that sort of extension with a C++ module should explicitly request --with-cxx, or b. The python build system should automatically detect that --with-cxx is needed based on the presence of C++ extension modules. Frankly I think b. would require an impractical amount of work and, speaking as an author of C++ extension modules, I don't think a. is much of a burden to impose.

...

If there's someone around here who is responsible for this change and knows its real rationale, can (s)he please tell me what it is? If not, can we please change things back so Python doesn't get linked to the C++ runtime by default?

ccpython.cc and --with-cxx was first published in Python 1.6, and hasn't changed much since. So for some of you, it has "always" been there. It was contributed by Geoff Furnish.

What *has* changed now is that the configure test suddenly determines that you need to link with g++ on Linux if main was compiled with g++. This was not the case before, but now is (since g++ 3.2 or something).

I see. Well, everything has become clear, thank you. My proposed remedy hasn't changed, though. -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Martin v. Löwis" writes:

David Abrahams wrote:

...

...

Not entirely. By extending Modules/Setup

You mean http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/Setup.d... ?

I mean Modules/Setup. It is generated from Modules/Setup.dist (plus some additional information) in the build process.

...

I contend that either:

a. Anyone making that sort of extension with a C++ module should explicitly request --with-cxx, or

b. The python build system should automatically detect that --with-cxx is needed based on the presence of C++ extension modules.

Frankly I think b. would require an impractical amount of work and, speaking as an author of C++ extension modules, I don't think a. is much of a burden to impose.

It is the burden of change.

Well, as you say: What *has* changed now is that the configure test suddenly determines that you need to link with g++ on Linux if main was compiled with g++. This was not the case before, but now is (since g++ 3.2 or something).

Contributions are welcome.

Let me first try to discover what contribution would be acceptable. What if: - we add a configure test that runs after the existing test determines that --with-cxx is needed (but not when --with-cxx is explicitly specified on the command line) - This test runs a 'C' executable that tries to load a C++ dynamic library with dlopen. - The test returns an error code if the the dynamic library's static and dynamic initializers have not been run properly - If the test fails we disable --with-cxx ?? I'm trying to intrude on the existing behavior as little as possible, yet get the semantics we want for ELF/Linux in a way that stands a good chance of generalizing to other platforms.

However, you will find that with a), people will still pass --with-cxx, because they tend to "enable" all features they can find.

That's acceptable to me. We could probably circumvent some of those cases by improving the configure --help text.

I personally could accept --with-cxx and ccpython.cc to be removed again, but I'm uncertain whether that may break distutils in some way.

Well, that would certainly be an easy "solution," but it would break HP/UX, and it would break anyone that needs to statically link C++ modules on some platforms. It's a much more drastic change than it would be to only have the system use --with-cxx when the person running configure asks for it explicitly. -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Martin v. Löwis" writes:

David Abrahams wrote:

...

- we add a configure test that runs after the existing test determines that --with-cxx is needed (but not when --with-cxx is explicitly specified on the command line)

- This test runs a 'C' executable that tries to load a C++ dynamic library with dlopen.

- The test returns an error code if the the dynamic library's static and dynamic initializers have not been run properly

- If the test fails we disable --with-cxx

??

What would be the purpose of such a test? The test may fail for many reasons, for example, dlopen might not be available.

I was assuming we only disable --with-cxx if the test builds successfully. I presume dlopen will fail linking if it's unavailable?

OTOH, on current Linux systems the test would succeed, so configure would conclude to link with g++.

Uhh, whoops. Change the sense of my last bullet - If the test passes we disable --with-cxx

...

I'm trying to intrude on the existing behavior as little as possible, yet get the semantics we want for ELF/Linux in a way that stands a good chance of generalizing to other platforms.

I now think that the code should be made more simple, not more complex. Aspects of a solution I could accept:

- ccpython.cc and linking with g++ is removed entirely. or,

That would be bad for C++ users on HP/UX. Is that acceptable?

- the logic is fixed so that linking with g++ is only done if main is in ccpython.o

I don't see how that works. Somehow we need to decide whether to put main in ccpython.o in the first place, don't we?

- the configure test is extended to better match current g++ behaviour.

What do you have in mind?

...

Well, that would certainly be an easy "solution," but it would break HP/UX, and it would break anyone that needs to statically link C++ modules on some platforms. It's a much more drastic change than it would be to only have the system use --with-cxx when the person running configure asks for it explicitly.

I just checked, and it seems that the logic in use is still somewhat different. If the configure test determines that a C++ main() must be linked with CXX, it unconditionally changes the linker to CXX. The test, in turn, is run always if a C++ compiler was found, i.e. independently of whether --with-cxx was provided.

That doesn't match up with reports from my testers who say they can run with C++ extension modules from many different GCC versions if they just configure their Python --without-cxx. If what you were saying were true, wouldn't --without-cxx be ignored on ELF/Linux? -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Martin v. Löwis" writes:

...

...

- the logic is fixed so that linking with g++ is only done if main is in ccpython.o

I don't see how that works. Somehow we need to decide whether to put main in ccpython.o in the first place, don't we?

Yes, that is done through --with-cxx (alone). However, the decision to use CXX for linking is independent on whether --with-cxx was given.

Is the above a description of existing behavior or a behavior you're proposing?

...

...

- the configure test is extended to better match current g++ behaviour.

What do you have in mind?

Somebody reported that the test works better for g++ if the function is marked extern "C".

Which function? What does "works better" mean?

This should be done for 2.4 regardless of any other change.

...

...

I just checked, and it seems that the logic in use is still somewhat different. If the configure test determines that a C++ main() must be linked with CXX, it unconditionally changes the linker to CXX. The test, in turn, is run always if a C++ compiler was found, i.e. independently of whether --with-cxx was provided.

That doesn't match up with reports from my testers who say they can run with C++ extension modules from many different GCC versions if they just configure their Python --without-cxx. If what you were saying were true, wouldn't --without-cxx be ignored on ELF/Linux?

Ok, it's still different. I see three cases now: 1. --without-cxx or --with-cxx=no specified. configure does not look for a C++ compiler, and does not check whether linking needs to be done with a C++ compiler, and decides to use Modules/python.c. 2. --with-cxx not given. configure does look for a C++ compiler, and does check whether linking with the C++ compiler is necessary, and still uses Modules/python.c 3. --with-cxx is given. configure requires it to point to a C++ compiler, performs the linking check, and uses Modules/ccpython.cc.

Is the above a description of existing behavior or is it a behavior you're proposing?

It would help discussion if you would use the actual code, too, instead of just using reports from your testers.

I don't know what you mean by "use the actual code." Do you mean, refer to the actual code in the discussion, or do you mean actually building and running Python --without-cxx myself? If the latter, I don't see a reason to repeat what people I trust have already done. -- Dave Abrahams Boost Consulting www.boost-consulting.com

"Martin v. Löwis" writes:

David Abrahams wrote:

...

...

...

I don't see how that works. Somehow we need to decide whether to put main in ccpython.o in the first place, don't we?

You wouldn't have to ask these questions if you had investigated the answers yourself.

The questions should have been more precisely phrased as, "Do you mean to say <whatever>?" Since your statements about the code have not always been accurate (not blaming you; everyone makes mistakes) I'd still need to know how you intend your statements to be interpreted, not only how they correlate with the code. -- Dave Abrahams Boost Consulting www.boost-consulting.com

[Michael Hudson]

--with-fpectl, for example. Does anyone lurking here actually use that, know what it does and require the functionality? Inquiring minds want to know.

I know what it intends to do: fpectlmodule.c intends to enable the HW FPU divide-by-0, overflow, and invalid operation traps; if any of those traps trigger, raise the C-level SIGFPE signal; and convert SIGFPE to a Python-level FloatingPointError exception. The comments in pyfpe.h explain this best.

Nobody uses it. It should be ripped out. If someone disagrees, let them speak up. On 7/12/05, Michael Hudson wrote:

Tim Peters writes:

...

[Michael Hudson]

...

--with-fpectl, for example. Does anyone lurking here actually use that, know what it does and require the functionality? Inquiring minds want to know.

I know what it intends to do:

Surprise!

...

fpectlmodule.c intends to enable the HW FPU divide-by-0, overflow, and invalid operation traps; if any of those traps trigger, raise the C-level SIGFPE signal; and convert SIGFPE to a Python-level FloatingPointError exception. The comments in pyfpe.h explain this best.

But do you use it? I know what it intends to do too, but I don't use it. The questions I asked were in the order they were for a reason.

Cheers, mwh

-- <cube> If you are anal, and you love to be right all the time, C++ gives you a multitude of mostly untimportant details to fret about so you can feel good about yourself for getting them "right", while missing the big picture entirely -- from Twisted.Quotes _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters writes:

[Michael Hudson]

...

...

...

--with-fpectl, for example. Does anyone lurking here actually use that, know what it does and require the functionality? Inquiring minds want to know.

[Tim, explains what it intends to do]

...

...

...

[Michael]

...

But do you use it? I know what it intends to do too,

Then why ask what it does? You're asking a lot of questions lately that you're irked by when you get an answer <0.9 wink>.

Isn't that the Socratic Method? :) Well, no it's not, but I do ask a lot of questions fishing for perhaps not the most obvious answer.

...

but I don't use it. The questions I asked were in the order they were for a reason.

No, I don't use it -- last I looked seriously (which was years ago), it didn't work on Windows.

Right, that's what I thought.

I don't know of anyone who does use it.

Even better! Cheers, mwh -- I really hope there's a catastrophic bug in some future e-mail program where if you try and send an attachment it cancels your ISP account, deletes your harddrive, and pisses in your coffee -- Adam Rixey

"Martin v. Löwis" writes:

David Abrahams wrote:

...

Apparently Python on some linux distros is being linked by g++ rather than gcc, resulting in the C++ runtime library being linked into Python; this has bad consequences for compatibility between C++ extension modules and Pythons that have been built with different versions of GCC. Is this behavior intentional?

It's as Skip says. According to the C++ standard, a "C++ program" is one where all translation units are written in C++. While most platforms offer interoperability between C and C++ in the sense that C libraries can be linked into C++ programs, interoperability in the other direction is not always supported, i.e. C programs may not be able to use C++ libraries. This is the specific case you are talking about: Python is written in C, yet the extension might be written in C++.

The C++ standard doesn't cover interoperability with 'C', or dynamic linking (we on the C++ committee are working to fix that, but for now that is the case) and it doesn't cover dynamic loading without linking, which is what happens when Python loads an extension written in C++. You can't appeal to the standard for the rules about what needs to be done. All this stuff is entirely implementation-dependent.

Now, on some of these implementations, things can be fixed by writing main() as a C++ translation unit, and compiling it with the C++ compiler. Then, Python itself is a C library to this C++ program, and the extension modules are C++ libraries. Then everything works fine.

C++ extension modules work fine even when main() is a 'C' program on Linux/GCC. The only reason that main would have to be a C++ program on this platform and compiler is if there were C++ translation units _linked_ into it (not loaded as in with dlopen). Since whoever writes the Makefile for Python also controls what's being linked into it, there's no reason to speculate and make Python a C++ program based on what might be linked in.

To support this, main() must be a C++ program. Python compiles main using the C++ compiler if configure thinks this is necessary.

Doing so is necessary for example on systems using the G++ collect2 mechanism for static initializers: compiling main with g++ leads to a call to __main(), which is then created by collect2.

Extension modules that get loaded with dlopen don't get their static initializers run by __main() anyway.

configure thinks that using CXX for linking is necessary if compiling a program using CXX and linking it using CC fails.

That might be the right thing to do for some programs, but AFAICT that's the wrong logic to use for Python. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Jeff Epler writes:

If we change the linker back to gcc, not g++, will it work if extension module 1 gets linked with libstdc++ A and ABI Q, and extension module 2 gets linked with libstdc++ B and ABI Z?

Yes, unless they are using sys.setdlopenflags to force symbols to be shared across these extension modules. That's a very intentional act and should (obviously?) only be undertaken when the extension modules share an ABI.

What if a built-in module is written in C++, as it might be for some people embedding C++?

"Embedding" usually refers to embedding a _Python_ interpreter in a program written in some language other than Python. But I think that's what you meant (just correct me if I'm wrong).

(this will force use of g++ as the linker, right?)

Yes.

Don't these cases matter too?

Yes. But the 2nd case is not one in which the Python executable is being built. The person building a program that embeds Python can control how (s)he does linking.

Assuming they can fail now, how will changing the use of CXX as the linker fix them?

I don't understand the question.

Jeff PS The Python 2.3 and Python 2.4 binaries installed on my Fedora Core machine don't list libstdc++ in `rpm -q --requires python' or `ldd /usr/bin/python'. I don't see a patch that would change Python's behavior in the SRPM, though. I wonder what the difference is between my FC2 and the other systems...

I don't know; the ones we happen to be testing are Debian ("sarge," I think). -- Dave Abrahams Boost Consulting www.boost-consulting.com

David Abrahams wrote:

...

configure thinks that using CXX for linking is necessary if compiling a program using CXX and linking it using CC fails.

That might be the right thing to do for some programs, but AFAICT that's the wrong logic to use for Python.

Why do you say that? Python compiles Modules/ccpython.cc as the main function, using the C++ compiler, and then tries to link it somehow. On some systems (including some Linux installations), linking will *fail* if linking is done using gcc (instead of g++). So we *must* link using g++, or else it won't link at all. Regards, Martin

David Abrahams wrote:

Unless, of course, I'm missing something. So if I am missing something, what is it?

You are missing something, and I can only repeat myself. Some systems require main() to be compiled as C++, or else constructors may not work (and perhaps other things fail as well). The configure option --with-cxx (documented as "enable C++ support") make Python C++ options work on such systems. It is automatically enabled if a C++ compiler is found. There is configure auto-detection for what linker is used when ccpython.o becomes main(). This is the state of the things as it is. In what way would you like to see that state changed? I could personally accept if ccpython and --with-cxx would be dropped entirely (i.e. deliberately breaking systems which require it); I just notice the irony of history: ccpython.cc was originally introduced to better support C++ extension modules - now it might get removed for the very same reason (to better support C++ extension modules). Regards, Martin

David Abrahams schrieb:

"Martin v. Löwis" writes:

...

David Abrahams wrote:

...

Unless, of course, I'm missing something. So if I am missing something, what is it?

You are missing something, and I can only repeat myself. Some systems require main() to be compiled as C++, or else constructors may not work (and perhaps other things fail as well).

Yes, and that becomes important in programs that have constructors. I.e., C++ programs. The Python executable is not such a program, except for one C++ file: ccpython.cc. There is no reason that file couldn't be rewritten as a pure 'C' file and any need for Python to be linked with G++ would disappear.

...

The configure option --with-cxx (documented as "enable C++ support") make Python C++ options

What are "Python C++ options?"

...

work on such systems. It is automatically enabled if a C++ compiler is found.

There is configure auto-detection for what linker is used when ccpython.o becomes main().

This is the state of the things as it is. In what way would you like to see that state changed?

I would like the Python executable never to be linked (or compiled either) by g++ unless that is explicitly requested by the person invoking configure or make.

...

I could personally accept if ccpython and --with-cxx would be dropped entirely (i.e. deliberately breaking systems which require it);

I don't believe any systems require it. I realize you have said otherwise, but after years of working with Boost.Python I'm very familiar with the issues of dynamic linking and C/C++ interoperability on a wide variety of platforms, and I'm not convinced by your assertion. If such a system exists, it should be easy for someone to point me at it, and show that something breaks.

If you build C++ extensions on HP-UX with aCC, Python must be compiled and linked as a C++ program. This is documented. It will not work if Python is compiled and linked as a normal C program (I have tried it). I haven't tried gcc on this platform, but I guess it is the same (compile and link with g++). Ulli

David Abrahams schrieb:

Ulrich Berning writes:

...

If you build C++ extensions on HP-UX with aCC, Python must be compiled and linked as a C++ program. This is documented.

You mean dynamically loaded C++ extensions, or the kind that are linked into the Python executable?

Dynamically loaded extensions, especially SIP/PyQt (http://www.riverbankcomputing.co.uk).

I'm willing to believe almost anything about HP-UX. Until recently, aCC was so broken as a C++ compiler that there was little point in trying to get Boost.Python to work on it, and I don't have much data for that system.

I'm using the HP aC++ Compiler C.03.50 together with the patches PHSS_29483 and PHSS_30967 on HP-UX B.11.00 and had no problems to build Python (2.3.5), Qt, SIP and PyQt and all other extensions with it.

...

It will not work if Python is compiled and linked as a normal C program (I have tried it).

Even if you take out the use of C++ constructs in ccpython.cc? I just need to check all the obvious angles.

What do you mean? The only C++ construct in ccpython.cc is the extern "C" declaration of Py_Main() and this is necessary if a C++ program references symbols from a C library. HP says, that a C++ shared library or a C++ shared object can only be loaded by a C++ main program. I can't remember the error message/symptoms, but I tried to build Python using python.c and couldn't load any C++ extensions. Because I'm going on vacation for the next three weeks, I can't try anything on HP-UX at the moment.

David Abrahams wrote:

Yes, and that becomes important in programs that have constructors. I.e., C++ programs. The Python executable is not such a program, except for one C++ file: ccpython.cc.

That is not true. ccpython was introduced to support builds of the Python interpreter where some modules are C++. As you know, people do write extension modules in C++.

There is no reason that file couldn't be rewritten as a pure 'C' file and any need for Python to be linked with G++ would disappear.

There is already a version of that file in C, Modules/python.c. This is used when --with-cxx is not given.

...

The configure option --with-cxx (documented as "enable C++ support") make Python C++ options work on such systems.

What are "Python C++ options?"

Oops, meant "Python C++ extension modules" (it must have been late when I wrote that).

...

I could personally accept if ccpython and --with-cxx would be dropped entirely (i.e. deliberately breaking systems which require it);

I don't believe any systems require it. I realize you have said otherwise, but after years of working with Boost.Python I'm very familiar with the issues of dynamic linking and C/C++ interoperability on a wide variety of platforms, and I'm not convinced by your assertion. If such a system exists, it should be easy for someone to point me at it, and show that something breaks.

I well remember that gcc 2.5.8 on Linux a.out required this sort of setup. Dynamic linking was not supported at all on that system (atleast not in a way where users could easily write shared libraries themselves). Rebuilding the Python interpreter was the only option to integrate additional modules. Regards, Martin

On 7/10/05, David Abrahams wrote:

"Martin v. Löwis" writes:

...

I well remember that gcc 2.5.8 on Linux a.out required this sort of setup.

Sorry, a.out? Isn't that the default name a C compiler gives to the executable it builds on Unix? Is it also (part of) the name of an OS?

It's the name of an executable format that predates ELF. I don't know if it's in current use anywhere. Paul.