[New-bugs-announce] [issue10243] Packaged Pythons

[Max Skaller]

New submission from Max Skaller <max.skaller at gmail.com>:

Not sure if this is a bug or not. I am unable to find libpython.so for Python3 on either my Mac or Ubuntu. Perhaps this is a packaging fault, however some documentation in the Wiki suggests otherwise. It appears the builders have reverted to an archaic linkage pattern which I helped to get rid of (lets see, perhaps a decade ago?). Python 2.6, for example, does ship a shared library.

Python must be shipped with code linked as follows, I will explain below why, but first the list:

1) The mainline (C main() function) must be a stub which calls the real mainline, which is located in libpython.

2) The mainline MUST be compiled with C++ not C.

3) All extension libraries and add-ons to Python provided as shared libraries must be explicitly linked against libpython.

In particular it is NOT acceptable for any extension or shared library component to expect to find its symbols in the host application executable as the Wiki documentation seems to suggest (in a section which explains a bit of a workaround for OSX frameworks).

Now the reason it MUST be this way. First, any C++ code which is to be linked into an application, either statically, dynamically at load time, or under program control at run time, may require certain stuff to be in place (RTTI, streams, exception handling stuff, or whatever) which can only be put in place in the initialisation of the main application. Although the details are platform specific, it is simply not safe to permit C++ extension modules unless this step is taken.

Legacy or embedded systems may have to make do with a C mainline, and systems which don't support dynamic loading can also do without C++ compilation provided the pre-loaded extensions are all C.

On most major platforms, however, a C++ driver stub is required.

The second issue is also quite simple. It is quite incorrect in a modern computing environment to assume an *application* will be hosting the python interpreter directly. It is not only possible, it is in fact the case for my project, that were the Python interpreter to be called, it would from a shared library loaded under program control at run time.

Such an interpreter cannot be loaded at all if it isn't present in a library: it either has to be statically linked into the shared library making the call, with some ugly linker switches to make sure no symbols are dropped, or it has to be loaded dynamically. The latter case is the only viable option if the run time linker is unable to share symbols to a loaded application, and even if that is possible and can be arranged it is not likely to work so well if multiple shared libraries try to do it.

Similarly, even if you managed to load it somehow, any dynamically loaded extensions may or may not be able to find the symbols.

The ONLY reliable way to ensure extensions can find libpython symbols is to link them against libpython.

In fact, the mainline application should not only contain NO libpython symbols specifically to disable this wrong practice and break any bad extensions that rely on it, it should also, as explained, contain exactly one reference to libpython, which calls Python with argv, argc[] as if it were the mainline.

Just as motivation here: my product is an ultra-high performance programming language with a special construction to allow Python C-extension modules to be built. Its target is usually a shared library and that library is produced by first generating C++ and then compiling it with your native C++ compiler.

For a generated program to call Python interpreter, it HAS to be available in a shared library, and for any extension modules that interpreter loads, they HAVE to get their symbols from that shared library, and, if the generated program is itself a Python module, then if that module is to be loaded from any C extension, including itself or some other extension, it HAS to be linked against libpython which HAS to be loaded dynamically by the loaded.

The unfortunate downside of this is that it is NOT POSSIBLE to have a huge statically linked Python executable which just loads C extensions and nothing else happens. If you're loading any C extensions dynamically libpython must be loaded dynamically too.

Just to be clear: I can easily build it the way I want it but this will not solve my problem, which is to support clients who wish to use my product to generate high performance Python callable modules which will just work "out of the box" with existing Python code. In particular, replacing some slow modules with optimised ones would be more or less entirely transparent .. except that at the moment it could only work with Python 2.x since Python 3.x shipments don't seem to have any shared libpython included (and I just changed my compiler to support Python 3 modules instead of Python 2).

----------
components: Installation
messages: 119963
nosy: Max.Skaller
priority: normal
severity: normal
status: open
title: Packaged Pythons
type: resource usage
versions: Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10243>
_______________________________________