[Patches] [ python-Patches-460751 ] --with-shared: python with shared lib

noreply@sourceforge.net noreply@sourceforge.net
Tue, 25 Sep 2001 14:53:02 -0700


Patches item #460751, was opened at 2001-09-11 14:29
You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=460751&group_id=5470

Category: Build
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Frederic Giacometti (giacometti)
Assigned to: Martin v. Löwis (loewis)
Summary: --with-shared: python with shared lib

Initial Comment:
As a follow up to patch 400938
(https://sourceforge.net/tracker/?group_id=5470&atid=305470&func=detail&aid=400938),
here is an implementation of the --with-shared
configure option for building python on Posix systems
as a shared library.

Building python as shared library is activated with the
--with-shared option on ./configure

The following points are drawn to particular attention:

- A configbuild.py module is generated and placed in
the machine-dependent directory. It records most 
Python/Makefile variables (CC, CFLAG, LDSHARED...), and
make them available in Python.

- The build has been tested on:
   Linux2: OK
   SunOS5: OK
   IRIX64: Build OK, crash on test (bus error)

- Inference with distutils: I found it very difficult
understanding the distutil architecture and finding the
right points of change; I minimized my interaction
there. Furthermore, the availability of the
'configbuild' module overlaps some of the distutils;
but 'configbuild' is very easy to use ...

- Unresolved symbols: I turned on the symbol checking
flags whenever possible, on behalf the its better
generating the error on build than at runtime.

- configure.in: I only edited the configure Bourne
script, and did not bother with configure.in (actually,
I deleted it) and its autoconfig/m4 macro comparses.

Frederic Giacometti





----------------------------------------------------------------------

>Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-25 14:53

Message:
Logged In: YES 
user_id=93657

Martin, here is a patch update.

- I fixed the missing library problem on bulding statically
the main Python. The static build work now. I had not tested
it yesterday; I was losing myself in patch/autoconf pb
instead.

- The configure script is now generated from configure.in,
with autoconf. Thanks for your help on this.

For the patch division issue, I propose to work in two
steps:
1) First the global patch working
2) When the global patch works, I'll edit manually the diff
file, and create the independent patch files, as well as
will remove what I need to troubleshoot the thing.

This work is no small piece of cake (at least for me).
Please keep in mind that I patch everything from a Python
2.1.1 branch; which is the one in production, and I'd rather
keep the manual editing of the patch to the end, as the last
step; especially since this is no straightforward business,
technically.

Otherwise, yes, I've been maintaining my own branch,
fortunately.

Thanks,

FG


----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-25 14:51

Message:
Logged In: YES 
user_id=93657

Martin, here is a patch update.

- I fixed the missing library problem on bulding statically
the main Python. The static build work now. I had not tested
it yesterday; I was losing myself in patch/autoconf pb
instead.

- The configure script is now generated from configure.in,
with autoconf. Thanks for your help on this.

For the patch division issue, I propose to work in two
steps:
1) First the global patch working
2) When the global patch works, I'll edit manually the diff
file, and create the independent patch files, as well as
will remove what I need to troubleshoot the thing.

This work is no small piece of cake (at least for me).
Please keep in mind that I patch everything from a Python
2.1.1 branch; which is the one in production, and I'd rather
keep the manual editing of the patch to the end, as the last
step; especially since this is no straightforward business,
technically.

Otherwise, yes, I've been maintaining my own branch,
fortunately.

Thanks,

FG


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-25 06:27

Message:
Logged In: YES 
user_id=6380

Martin, feel free to reject and close this.  Frederic's
attitude has been impossible to deal with, here and in other
cases.

Frederic, the real issues here are non-technical -- you
appear unable to understand and adapt to the conventions of
open source development that almost every other contributor
picks up without effort. We don't have the time to educate
you -- we've tried and failed. We'll continue to review your
patches, but we'll be quick to reject them if you continue
to ignour our conventions, which have served Python and the
Python community well over the years. Feel free to maintain
your own fork of the Python sources -- that may be less of a
waste of time for all of us.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-09-25 03:37

Message:
Logged In: YES 
user_id=21627

PLEASE do not put unrelated changes into your patches. What
does "--with-shared: python with shared lib" have to do with
removing @ prefixes in the makefile???? I'm close to giving
up reviewing your patches, and waiting for somebody else to
reject them.

Likewise, why did you remove the makedepend support? It may
be broken, but the objective of this patch has nothing to do
with this. Same for reconfigure; to re-run configure with
the same options, just invoke 'make'. If you think there is
a problem, report a bug.

The problem with the generated configure not recognizing
--with-shared is that you put the AC_ARG_WITH after the
check for build_as_shared into configure.in. Just move the
AC_ARG_WITH somewhat up.

Never use ld to link on Unix, always use $(CC); otherwise,
people will complain that linking in C++ modules will fail.
To pass options to the linker using the SunPRO compiler, use
the -Wl,<option> option; see the description of -W in cc(1).
Also, I propose that you put the -zdefs,-no_unresolved in a
separate patch, to be applied after this one got integrated.
It is not essential for building libpython.so.

It appears that your -lsocket changes broke IPv6 support,
atleast on Solaris 8: Since -lsocket -lnsl is not linked
into programs produced by AC_TRY_{LINK,RUN}, the tests for
creating AF_INET6 sockets and for getaddrinfo fail. Please
reconsider whether putting the libraries together with the
socket module is really essential for this patch. I believe
the module will continue to work even if just the Python
executable is linked with the socket library (of course,
assuming that you get rid of the -z defs requirement).


----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-24 18:06

Message:
Logged In: YES 
user_id=93657

Martin,

I'm attaching a tar containing a diff against the CVS
mainline (sync last Friday), and the comments on the changes
in a different file.

I put the changes in configure.in, from configure, but after
running autoconf, the --with-shared option has no effect.
Hopefully, this is not much.

The small changes for building threaded python on OSF/1 are
in the same zone as those edited for the change, so I could
not separate them out.

FG

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-20 12:16

Message:
Logged In: YES 
user_id=93657

The patch is against Python 2.1.1.
I'll try to integrate this with the current source. I'll
work with a single checkout across the multiple platforms.

Regarding buildno: I was getting recursive dependency links,
after I made the shared library dependent upon its .o files.
I had to break the recursivity. 

What is dguxR4 ? Is it OSF1 V4.0, a.k.a. Digital Unix 4.0,
a.k.a. Tru64 ? I have now access to an OSF1 v4.0 (a.k.a. DU
4.0). I'll do the build there.
uname returns 'OSF1 ...'

The -lsocket is needed for building the socket module, and
had to be moved there (cf. Python/setup.py, I think).
It is not used in the core, so I dropped it there.
Dito for -ldsl.
No symbols are missing when linking libpython2.1.so, anyway.
These libraries just have now to be linked with the modules
which use them.
The rule I applied for deciding on libraries are:
- link with all libraries necessaries so as to suppress all
unresolved symbol output
- link only with libraries whose absence triggers unresolved
symbol outputs

- -lmpc: You might be right, but then -lmpc should be
present only on systems that need it (and, first of all
these systems should be identified/documented).

- fpectl: dito

FG


----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-09-18 10:05

Message:
Logged In: YES 
user_id=21627

I've tried testing your patch, but I get numerous failed
chunks when applying it against the current CVS. What
version does this patch apply to? To simplify the
integration, it would be greatly appreciated if you could
use the current source base.

>From reviewing the patch, I still have a number of concerns:
- it appears that you change the mechanism by which buildno
is incremented. Why?
- There appear to extra white space changes (e.g. in the
distclean chunk); those are naturally unrelated to the patch
- please remove them
- it appears that configuration knowledge for some platforms
is lost when applying this patch, e.g. the dguxR4
special-casing of passing -pic is gone. Please try to
integrate any such knowledge faithfully into your patch,
instead of restricting its applicability to just the
platforms where you've tested it on.
- There is something going on with dropping -lsocket from
the patch. What is that, and why does it happen? Instead, it
appears that you put 'socket' and 'nsl' into setup.py.
According to configure.in, this is wrong on some systems:
they may not have the libraries, or the libraries may be
broken.
- Other libraries appear to suffer as well, such as removing
-lmpc. That might mean that you break thread support on some
systems.
- the chunk to disable fpectl if nothing is known about the
systems appears to be unrelated.


----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-17 10:36

Message:
Logged In: YES 
user_id=93657

I'm attaching an updated patch where, following Martin
suggestions:
  - I removed the configbuild thing, and replaced it with
distutils.sysconfig.get_conf_var calls
  - I manually cleaned up the patch from unrelated changes
(.cvsignore, @ Makefile modifiers...)

Remains the configure.in thing.

If one looks at autoconf:

- it takes care of plateform idiosyncracies, for platform
which do not follow posix API specs; this was of use in
previous generation Unix OS's, and these legacy systems do
not evolve anymore. On present Unixes, and with regard to
the features used for building the core python engine,
nothing moves, and autoconf updates bring nothing.
 Doesn't the Python 1.5.2's configure still works as it did
the first day it was generated ... ?

- autoconf is of use for configuring particular packages and
libraries (e.g.: for configuring and building python
extensions, Tkinter/Tk/TCL...); unfortunately this never
worked with python, and the task is now assumed by the
distutils.

- In the Python core build, autoconf is of no help in
anything sensible: thread, shared library, debug build,
compiler options... 
All these were programmed manually in Bourne; and for doing
the later, autoconf is a complexifying factor on something
already fairly complex - i.e. a handicap -.

- not to say, autoconf is of no use on non-posix plateforms
(windows...)

- last, and the least: autoconf now requires that perl be
installed...

Frederic Giacometti



----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-12 18:36

Message:
Logged In: YES 
user_id=6380

> What is the point of keeping using the autoconf thing when
> it is easier to maintain configure directly, than going
> through configure.in ?

But it's not. One line of configure.in often expands to
dozens (occasionally hundreds) of lines of configure.
autoconf has arcane knowledge that gets updated with new
autoconf releases.

You are entitled to your opinion, but in this case, it's
wrong. Go fork your own version of Python if you don't want
to learn autoconf.

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-12 16:06

Message:
Logged In: YES 
user_id=93657

I am not willing  to make anything difficult for anybody,
but:

What is the point of keeping using the autoconf thing when
it is easier to maintain configure directly, than going
through configure.in ?

After configure has been generated once, it makes more sense
to me to work directly on configure, and forget about
configure.in.
configure (pure Bourne shell) is a easier to understand, to
troubleshoot, and to maintain that configure.in (a mixture
of Bourne shell and autoconf m4 macros).

I'll have a second look at the autoconf thing latter  on -
at present I hardly understand anything to configure.in, and
got 'configure --with-shared' working with satisfaction
across multiple platforms without undue difficulties.






----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-12 11:37

Message:
Logged In: YES 
user_id=6380

> Autoconf is not installed on the commercial Unix
> platforms, it is useless for me to learn it, and I was
> better off directly editing ./configure. However, if
others
> are familiar with the tool, it should be straightforward
for
> them to retrocede the changes from configure to
configure.in

Sure. You'll just have to wait until we have time for that.
You can make it easy for us to integrate your changes, or
you can make it difficult. Your choice.

----------------------------------------------------------------------

Comment By: Frederic Giacometti (giacometti)
Date: 2001-09-12 11:12

Message:
Logged In: YES 
user_id=93657


- I will remove the configbuild; since Martin pointed to me
the existence of  distutils.sysconfig.get_config_vars(). I
did not know about this function... Thanks!
On the fly, you'll also note the following potential problem
with using get_config_vars() instead of configbuild: 
configbuild will report the actual values used for the
build, whereas get_config_vars() will report erroneous
values whenever the Makefile internal variables were
overriden at build time (by either the environment, or by
commandline assignment of the type 'make CC=gxx ...'); not
to mention that configbuild does not need to find and parse
the original Makefile (with all the potential problems
associated).

- The .purify line should be in the CVSROOT/cvsignore CVS
admin file, not in .cvsignore in the root directory of the
source distribution...

- Autoconf is not installed on the commercial Unix
platforms, it is useless for me to learn it, and I was
better off directly editing ./configure. However, if others
are familiar with the tool, it should be straightforward for
them to retrocede the changes from configure to configure.in

- I had to introduce THREADOBJ because you will note that,
in the present implementation, Modules/thread.o appears
multiple times in  LIBOBJS, which causes warning at link
time. Now, Modules/thread.o appears only once




----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-09-12 01:06

Message:
Logged In: YES 
user_id=21627

As for libtool: I strongly advise not to use it. It is
broken in too many ways to enumerate.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2001-09-12 00:18

Message:
Logged In: YES 
user_id=21627

I recommend to rework this patch to get rid of 
configbuild. I cannot see what you achieve with the 
configbuild module that you couldn't do with 
distutils.sysconfig.get_config_vars (which is capable of 
parsing the Makefile directly).

I also recommend to go through the patch chunk by chunk, 
asking yourself whether this chunk *really* is needed to 
achieve the desired functionality. E.g. I cannot see the 
relationship of the .cvsignore chunk; I also believe that 
removal of .purify is incorrect. Likewise, while I 
sympathize with the unresolved symbols change, I believe 
it is unrelated to the main objective of this patch 
(building Python shared).

What is the rationale for not changing configure.in? 
As-is, this patch is useless: configure will be 
overwritten everytime you run autoconf. Please get a copy 
of autoconf on one of your systems, change configure.in, 
run autoconf, and then provide us with a patch that only 
changes configure.in: everybody reviewing this patch 
should be capable of running autoconf. configure is a 
generated file and should not be edited manually.

What is the rationale for the introduction of the 
THREADOBJ variable?



----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-09-11 19:03

Message:
Logged In: YES 
user_id=6380

Haven't seen the patch yet, but I like the idea -- the
command line option suggests that it shouldn't hurt
platforms where it can't be done or has to be done
differently.

Alternative: we could also consider using libtool. I didn't
like that much in the past, but maybe it would work. Dunno.
Anybody know about it?

----------------------------------------------------------------------

You can respond by visiting: 
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=460751&group_id=5470