[Python-Dev] Re: shared library

[Francois Pinard]

[Martin v. Löwis]

> Francois Pinard <pinard at iro.umontreal.ca> writes:

> > But I might be missing the meaning of the original question.  If by any
> > chance you feel like elaborating a bit, that could be instructive for me.

> The question is whether the code containing the Python implementation
> should be a shared library, i.e. libpythonxy.dll, which would give a
> very small stub python executable.

Thanks for having replied.

> So it has only disadvantages and no advantages, [...]

I would not go that far.  It might have a few advantages.

A few times in the past, not so long ago, I compiled C main programs
using Python-written modules and functions, and with Pyrex-generated
interfacing code.  I surely did this once or twice for getting a static
debugging environment, but I also got actual, real, production use for such
things.  The usual "rewrite-everything-in-Python" does not always apply for
legacy code.  I could try to defend the "add-new-parts-in-Python" stanza,
but not always the idea to rewrite what already exists -- this would
takes years, as if Python makes faster writing and easier maintenance,
it does not necessarily transform the actual problems into simple things.

However, each such generated executable is significantly bulkier as it
holds a copy of the whole interpreter, and the final linking calls is
neither evident to write, nor to explain to other team members.  So,
a kind of effect is that using Python from C is not used as often as it
might be.  I guess that if it was more "innocuous", using Python modules
together with C programs will happen more often, and might become more
a natural after a while, than still an extraordinary thing.  But it is
easier to hypothesise than ascertain, the truth is that I'm not so sure.

Last week, I had to recompile `vim' on many machines to enable Python
support, and this doubles the size of the installed binary.  Here again,
one might hypothesise that the Linux packagers (SuSE in this case) would
have more naturally provided a Python-enabled `vim' if its size was not
bloating so evidently.

Similar considerations might apply each time Python is considered as an
extension languages for a system written in another language, who knows.
The system has to be pretty big to start with, for Python-as-an-extension
to appear as something else than an enterprise.

> I think this causes more bad than good because:

> 1. If there is a single <prefix>/python executable, there is just
>    about the same amount of sharing.

Of course, if Python is never used as an extension, it is much simpler
and better to merely share the whole executable.  On the other hand,
this is also encouraging people to _not_ use Python as an extension.
The snake is eating its tail, here. :-)

> 2. Having libpythonxy.so causes significant maintenance problems.

At least on Linux, with the number of shared libraries already around,
I would presume that the technology has some maturity.  On this system,
it is likely that it would work painlessly.  However, there are many
platforms where things are not so controlled, and since Python development
aims a lot of platform, the is a significant maintenance problem indeed.

> 3. Having libpythonxy.so increases the start-up time,

Surely.  Yet, whenever Python starts nowadays, there is already oodles
and tons of disk accesses, and I'm often surprised that it starts so
speedily nevertheless.  Machines got fast, and the memory cache of the
system is very effective.  In that sea of disk accesses, I wonder if the
increase of start-up time would be that significant.  Optimisation should
probably address other areas before this one.

> 4. Having libpythonxy.so decreases the run-time efficiency, as now all
>    code needs to be compiled as PIC.

This might account for a few percents of CPU, according to what I read
(I did not benchmark those things myself).  I've seen extra-ordinary
efforts on python-dev for getting Python more speedy, so your argument
may be a strong one for many people.  No doubt that I like speed, but I
would probably not crusade for a few percents.  This was lengthily and
hotly discussed when ELF was first introduced, and it seems that the
overall consensus which emerged is that the community is ready to drop
a few percents for the gains brought by shared libraries.

Of course, now that many softwares accepted to move and resultantly are
a bit slower, Python might gain a few points by standing still. :-)

> 5. Having libpythonxy.so creates a maintenance cost, [...]

This is more or less a repeat of point 2., and there is a maintenance cost,
no doubt.  This might kill the idea more than anything else, in my opinion.

Would it be a reasonable compromise if Python was compiled twice?
Once non-PIC for the usual static installation, and a another one for
preparing a shared library?  Both would be installed.  The `python'
stand-alone interpreter would continue to use static linking, and its
executable would be shared among all currently running Python scripts.
The library would be available for writing extensions of all kinds.

System packagers would be much encouraged to provide both whenever it works
for their system.  Linux packagers among them, of course.  Maybe that for
the usual "configure-without-options" way of installing that most people
use themselves, the status quo would apply for at least a long good while.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard