Mailman 3 December 1998 - Distutils-SIG

Note on building on AIX
by Andrew M. Kuchling 16 Dec '98

16 Dec '98

On the XML-SIG, Konrad Hinsen mentioned the following problem with shared libraries on AIX. I'm forwarding this just to read it into the Distutils record, as something that should also be fixed. -- A.M. Kuchling http://starship.skyport.net/crew/amk/ Quaint though this attitude seems now, it was unquestionably the prevalent one in the nineteenth century, and it would be over-bold to say that it will never return to favour, for the range of human folly is infinite. -- Robertson Davies, _A Voice from the Attic_ ============================= Konrad Hinsen wrote: > > - Some people need patched versions; for example, the standard > > version does not work for AIX. > > What's the patch that's required for AIX? And is there some > reason it can't be rolled into the Makefile.pre.in for 1.5.2? I don't know the system well enough to decide. The problem is that shared library linking is a rather complicated process under AIX, which is handled by two shell scripts. These shell scripts come with the Python distribution (they are "ld_so_aix" and "makexp_aix") and are ultimately installed in the "config" subdirectory of the Python library. But during the compilation of the interpreter and its standard library modules, they reside in the "Modules" subdirectory of the Python distribution. The settings in the configuration reflect this initial situation, not the one after installation. So if you use the standard Makefile.pre.in, the two critical definitions becom LINKCC= $(srcdir)/makexp_aix python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(srcdir)/ld_so_aix $(CC) whereas they should be LINKCC= $(LIBPL)/makexp_aix $(LIBPL)/python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(LIBPL)/ld_so_aix $(CC) -bI:$(LIBPL)/python.exp I suppose this could be arranged during the installation process, but I don't really want to figure out how that works! ...

1 0

Re: [Distutils] Observations
by Greg Ward 16 Dec '98

16 Dec '98

Quoth Robertson, Joseph M, on 15 December 1998: > I like the scriptable builds stuff. Just like Instalit of the old days. If > fact I'd suggest we use a similar method. The build script tags data such > as whether to check file dates, versions, etc when installing; what > compression type to use, etc, etc. What is "Instalit"? -- I've heard the name pop up once or twice in the last week or so, but never heard of it before. Remember, we're not all on the same wavelength here! > Question: How does your system install a working python program if the > target has no python whatsoever on it? That's something I for one haven't considered. It's not currently on the plate of the SIG, but should probably be considered for the future. Here's one possible long-term roadmap: * first, be able to build, distribute, and install Python modules: this will make it easy to extend an existing Python installation * next, be able to install a complete Python application (scripts + modules): this will mainly involve doing the module installation thing plus dependency tracking (ie. what *other* modules are required by this application) * finally, be able to distribute a Python application to systems that don't have Python at all All three are worthy goals indeed, and "distribution utilities" certainly covers all three. But there's a clear order to be followed, and right now we're concentrating on the first job. The second should follow without a great deal of difficulty, and third could either be a snap (just install one of the large number of pre-built Python distributions) or a morass. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 16 Dec '98

16 Dec '98

At 16:33 15/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 16 December 1998: >> I want the build process to put that >> information into a standard module. So I can get it out, >> and call up the compiler to make a C extension FROM PYTHON. > >A-ha! Thank you -- now it comes clear. If I were more weasely, I could >squirm out of it by saying "Well, this just follows from the requirement >to easily build extension modules from source" or, "That's not a >requirement, it's an implementation detail". You could, you'd be right in both cases, and you cannot be expected to do everything: which is why I wrote the article. :-) >But I'm not a weasel, You're not god. You started this list to get help from others. >and I think at some point I meant to mention this >in the requirements document. So I've added it. Great! Thanks! >I'm managing to keep implementation details out of the requirements >list, though. In particular, my proposal is for a new 'sys.config' >module which would contain everything you might need to know about how >Python was configured. YES!!!! I agree! >The most important items are the compiler, >compiler flags, platform, and library installation directories. Can I recommend you look at the interscript compilers module again? Instead of storing compiler flags, etc, you need an actual command that does the compilation. I.e. a python function that builds a command string and os.system's it. I don't think that 'data' can store this information, there are too many options. (Store the data as well!) >For an example of the degree of flaming overkill possible in this sort >of module, try this > perl -MConfig -e 'map { print "$_: $Config{$_}\n" } sort keys %Config;' >You'll note that Perl's Config module stores *every* piece of >information, gleaned, gathered, or queried from the user while running >Perl's 'Configure' script. I don't think Python's sys.config need be >*quite* so exhaustive, I agree. After all, we can upgrade it later. The important thing is to get the module into the distribution. That way people can try it, and we'll find out what is missing. Anyone on this list have a Next machine?? >Here's a very rough start >(loosely based on my long experience with building Perl and my very >brief experience with building Python, and a bit of poking through >Python's makefiles): This is too Unixy. > cc C compiler used to build Python > cppflags C preprocessor flags used to build Python (-I, -D, -U) > optimize optimization/debugging flags used to build Python > ccsharedflags C compiler flags required to build shared objects > extra_ccflags any other C compiler flags used to build Python > ccflags all compiler flags used to build Python: > cppflags + optimize + extra_ccflags > ccextflags all compiler flags used to build Python extensions: > cppflags + optimize + ccsharedflags + extra_ccflags > > ldlibdirs -L flags used to build Python > ldlibs -l flags used to build Python > extra_ldflags any other needed linker flags > ldsharedflags linker flags required to build shared objects > ldflags all linker flags used to build Python: > ldlibdirs + ldlibs + extra_ldflags > ldextflags linker flags used to build Python extensions: > ldlibdirs + ldlibs + ldsharedflags + extra_ldflags I do not think _any_ of these flags, etc, are required individually. Nor do I see how they can possibly be enough. You are making assumptions about how an arbitrary compiler is invoked. What you need is a _function_ which compiles a c file for a shared library, and another which links them. In other words, you just need a string with some 'slots' in it for the filenames, and which has all the flags already set in the right places. The function is a generalisation of that. Have a look at the compilers module I posted here. > platform hardware/os/vendor (or something along those lines) Yes. Something like that. This is necessarily crude information. > install_bindir directory to install binary executables > install_scriptdir directory to install non-binary executables > install_includedir directory to install C header files > install_confincludedir directory to install platform- and > build-specific C header files > install_libdir main Python library directory > install_sitelibdir directory for site-specific modules Each of these things is what i call an 'install point'. It isn't clear that the list is enough: clients may want to organise things differently for different modules. For example, the place PYTHON puts scripts is definitely NOT where I want extension packages: I want them in the site-packages directory. This is tough. I think your list is a good start. Please add: install_htmldir directory to put HTML doco install_mandir directory for man pages (does anyone provide them?) >Obviously this is very sketchy. An excellent sketch though. Why? because it is concrete enough to be argued about! ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 16 Dec '98

16 Dec '98

At 10:36 15/12/98 -0500, Greg Ward wrote: >*This is not a flame* Please do not interpret it as such! > >Quoth John Skaller, on 13 December 1998: >> This proposal is intended as a detail of the general requrements >> document for distutils. >So, first question: are you proposing to throw out my requirements >document and replace it with this? Or are you suggesting >additions/modifications to mine? An addition. I though that was clear by describing it as a detail. I'm proposing that, under one of your headings, to add a subheading. OK? >Next, you speak of the "Python build process" creating "run time >accessible identification data". The first phrase gives me the >impression that you're talking about building Python itself (and its >library), which really isn't the province of the distutils suite. Yes, I AM taking about building Python itself. Sigh. It it that hard to understand: when python is built, it know the name of the compiler. I want a module: import build_compiler cc = build_compiler.cc which is the name of the compiler used to build Python, so I can invoke that same compiler to build an extension. And all the other info needed, such as where various libraries are, what -DMACRO options are needed, etc, (If this SIG produces usable code soon enough, Guido might find it useful to >rework how the standard library is built for Python 1.6. That's the >only connection I see between the two.) Then you are missing the link: the build processes know how to build C. I want the build process to put that information into a standard module. So I can get it out, and call up the compiler to make a C extension FROM PYTHON. >I have no idea what you mean by "run time accessible identification >data". Or rather, I can think of a half-dozen things you might mean. >Please explain, or rephrase in a more conventional way. have I explained now? This is the key problem for distutils BECAUSE it REQUIRES Guido to modify the build process to create the information needed to call up the compiler from python script. At least, this is by far the most reliable way to get the information: all that autoconfig and makefile and setup stuff has the information, and it is needed to be able to build C code later as well. That is the connection. Distutils cannot work if Guido doesn't give the information. SOME of the information is already available in some form: the name of the compiler is already encoded in the version string. But it isn't enough to actually emulate the compilation and linkage processes the same way Setup builds things. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

4 3

Enough on software versions
by Greg Ward 15 Dec '98

15 Dec '98

Hi all: I'd like to declare the "Module version numbers" thread a CLOSED subject. I may spend some time in the near future writing a module to provide classes for version numbers in the anarchist/realist vein proposed by Greg Stein, and in the control-freak/idealist vein proposed by me. If you have any further comments, please keep them in private email; I'm curious to hear any opinions that have *not* already been voiced on the list. (Or you can send me a quick "me too" note if you just want your voice heard.) But let's keep that traffic off the list and move on to important things. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

>[John Skaller] >> So, it would be relatively easily to generate a standard >> python module? > >Without more context I don't know what you mean by "generate a >standard Python module". Sorry: the intent is to generate a python module containing enough information to compile any 'standard' C extension to python without makefiles and on any platform. Note: the assumption is that the extension builder is the same system as that which compiled Python. So a C compiler is available by assumption. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Observations
by Robertson, Joseph M 15 Dec '98

15 Dec '98

Hi all, I like the scriptable builds stuff. Just like Instalit of the old days. If fact I'd suggest we use a similar method. The build script tags data such as whether to check file dates, versions, etc when installing; what compression type to use, etc, etc. Question: How does your system install a working python program if the target has no python whatsoever on it? Thanks Joe Robertson 256-730-8216 jmrober1(a)ingr.com jmrobert(a)ro.com

1 0

Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

This proposal is intended as a detail of the general requrements document for distutils. The proposal is for the Python build process to create run time accessible identification data whose purpose is to facilitate run time building of a) dynamically loadable binary python module objects *.so, .dll etc) b) external (shell) application tools (.exe etc) This proposal does not fix the format of the information. --- I would like to consider a class instance, with one instance for each supported compiler/kind combination. I recommend the Python make process attempt to provide access to C, C++, and Java compilers. A binary build, such as Mark Hammonds Windows build, cannot be pre-configured for the client's system. Note that interscript contains modules which compile C/C++ code. At present, the modules have to rewritten by the client (unless they're running RedHat Linux :-) It is not clear to me if an _executable_ representation of the above requirement is appropriate. If not, at least the information required to create one should be made available. The representation must be editable by the client somehow. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

At 10:27 13/12/98 -0500, Guido van Rossum wrote: >> The proposal is for the Python build process to create run time accessible >> identification data whose purpose is to facilitate run time building of >> >> a) dynamically loadable binary python module objects *.so, .dll etc) >> b) external (shell) application tools (.exe etc) >> >> This proposal does not fix the format of the information. > >Note that (for Unix at least) all this info is already being gathered >by the configure script, and stored in the various Makefile. The info >also ends up being installed (look in >/usr/local/lib/python1.5/config/). So, it would be relatively easily to generate a standard python module? >For Windows, since the only supported compiler is VC++, it could be >hardcoded. However, there's a very serious problem on Windows with >Interscript's approach of invoking the compiler at run-time: most >users don't have this compiler. I wouldn't have call that a serious problem with the approach. It is a difficulty :-) The current implementation doesn't handle failure gracefully. But, after all, it is just python script: more work needs to be done to fetch or use pre-built binaries, and to detect _whether_ a compiler is available. Etc. >I realize that gcc is available for >free, but I think it isn't compatible with VC++. As far as I know, >VC++ is required if you want to use any of Mark Hammond's stuff (COM >and MFC). I don't know if Tcl/Tk can be used (without recompilation) >from a gcc-based app, and I don't know how easy it would be to >recompile. (Does gcc on Windows have support for Win32 APIs at all?) To the best of my knowledge, CygWin will allow a Python build on Windows: it supplies bash, ecgs version of C/C++, and some other utilities. It also supplies Tcl/Tk/Tix. And it allows standard Windows API calls as well. The advantage of the Interscript approach for Windows users is that SOME of them do have a C/C++ compiler. So instead of just ONE person supporting Windows, everyone with a compiler would be able to build and contribute binaries. In other words, I can't see how the interscript approach makes anything _worse_ for Windows users. I do agree the current mechanism is inadequate! Hopefully, people can suggest ways to improve the mechanism. Note also: JPython works on Windows! So Mark Hammonds isn't the only build of Python that works on Windows :-)) ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Attempted summary of version number thread
by John Skaller 15 Dec '98

15 Dec '98

>Perhaps an "API version number" belongs in the package meta-data, along >with the multitude of variables that John Skaller considers part of the >"version" (again, I think John is talking about something much bigger >than "version numbers", and I think what he's talking about can be >considered package meta-data). Yes: but isn't the point here that version numbers are 'just another piece of meta-data'? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0