Mailman 3 March 1999 - Distutils-SIG

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Some code to play with
by Greg Ward 14 Apr '99

14 Apr '99

Hi all -- as promised, I have finally checked in some real Distutils code. There's just enough functionality for the Distutils to build and install itself. To try it out, you'll need to download the code from the anonymous CVS archive at cvs.python.org; see http://www.python.org/sigs/distutils-sig/cvs.html for instructions. If you just want to try it out, then from the top-level Distutils directory (the one that has setup.py), run: ./setup.py build install This will copy all the .py files to a mockup installation tree in 'build', compile them (to .pyc -- don't run setup.py with "python -O", as I don't handle that yet and it will get confused), and then copy the whole mockup tree to the 'site-packages' directory under your Python library directory. Figuring the installation directory relies on Fred Drake's 'sysconfig' module -- I don't recall what the status of sysconfig was on non-Unix systems; hope someone out there can try it and let us know! Come to think of it, I'd like to hear how it works on *any* system apart from my Red Hat Linux 5.2, stock Python 1.5.1, home PC. ;-) If you want to build in a different directory, use the '--basedir' option to the 'build' command; to install in a different place, you can use either the '--prefix' or '--install-lib' options to the 'install' command. Here are a couple of samples to demonstrate: ./setup.py build --basedir=/tmp/build install --prefix=/tmp/usr/local or ./setup.py build --basedir=/tmp/build ./setup.py install --build-base=/tmp/build --prefix=/tmp/usr/local Go crazy. No documentation yet -- please read the code! Start with distutils/core.py which, as its name implies, is the start of everything. I went nuts with docstrings in that module last night, so hopefully there's just enough there that you can figure things out in the absence of the "Distutils Implementation Notes" document that I'd like to write. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 2

Re: [Distutils] Compiler abstractiom model
by Paul F. Dubois 01 Apr '99

01 Apr '99

Just to chime in about the unmentionable, I have a tool for connecting Fortran and Python (as do others; mine isn't ready for the light of day yet) and unfortunately the existing Setup scheme makes me compile my Fortran separately in a library. I don't have any way to piggyback on what Python has learned about the platform. This is too bad. I am sure it isn't the Python community's job to solve the Fortran 90 make problem but there are days when I dream about it. The "right" object-oriented approach should allow some way to extend the system, e.g. adding suffices or file names together with info about how to compile them. -----Original Message----- From: Mark Hammond <MHammond(a)skippinet.com.au> To: distutils-sig(a)python.org <distutils-sig(a)python.org> Date: Tuesday, March 30, 1999 3:50 PM Subject: RE: [Distutils] Compiler abstractiom model >> > Quoth David Ascher, on 29 March 1999: >> > > On windows, it is sometimes needed to specify other files >> which aren't .c >> > > files, but .def files (possibly all can be done with >> command line options, >> > > but might as well build this in). I don't know how these >> should fit in... >> > >> > Are they just listed in the command line like .c files? Or >are they >> > specified by a command-line option? Would you use these in >> code that's >> > meant to be portable to other platforms? >> >> Yes, no, and yes. =) >> >> E.g. Python extensions need to declare the 'exported' entry >> point. This >> can be done either by modifying the source code (bad for >> portable code, >> requires #ifdef's etc.), by specifying a command-line option, >or by >> including a .DEF file. > >Just to follow up on this, many obscure options may need to be >passed to the Windows linker, but they do require their own >option - they are not passed as normal files. Examples are >/DEF: - the .def file David mentions, /NOD:lib - to prevent a >default library from being linked, /implib:lib to name the built >.lib file, etc. > >It wasnt clear from David's post that these need their own >option, and are not passed as a simple filename param to the >linker.... > >Building from what Greg said, I agree that _certain_ command-line >params can be mandated - they are designed not to be >configurable, as messing them will likely break the build. But >there also needs to be a secondary class that are virtually >unconstrained. > >[Sorry - as usual, speaking having kept only a quick eye on the >posts, and not having seen the latest code drop...] > >Mark. > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG(a)python.org >http://www.python.org/mailman/listinfo/distutils-sig > >

3 3

Compiler abstractiom model
by Greg Ward 31 Mar '99

31 Mar '99

Hi all -- I've finally done some thinking and scribbling on how to build extensions -- well, C/C++ extensions for CPython. Java extensions for JPython will have to wait, but they are definitely looming on the horizon as something Distutils will have to handle. Anyways, here are the conclusions I've arrived at. * Stick with C/C++ for now; don't worry about other languages (yet). That way we can be smart about C/C++ things like preprocessor tokens and macros, include directories, shared vs static libraries, source and object files, etc. * At the highest level, we should just be able to say "I know nothing, just give me a compiler object". This implies a factory function returning instances of concrete classes derived from an abstract CCompiler class. These compiler objects must know how to: - compile .c -> .o (or local equivalent) - compile multiple .c's to matching .o's - be able to define/undefine preprocessor macros/tokens - be able to supply preprocessor search directories - link multiple .o's to static library (libfoo.a, or local equiv.) - link multiple .o's to shared library (libfoo.so, or local equiv.) - link multiple .o's to shared object (foo.so, or local equiv.) - for all link steps: + be able to supply explicit libraries (/foo/bar/libbaz.a) + be able to supply implicit libraries (-lbaz) + be able to supply search directories for implicit libraries - do all this with timestamp-based dependency analysis (non-trivial because it requires analyzing header dependencies!) Linking to static/shared libraries and dependency analysis are optional for now; everything else is required to build C/C++ extensions for Python. (At least that's my impression!) "Local equivalent" is meant to encompass different filenames for C++ (eg. .C -> .o) and different operating systems/compilers (eg. .c -> .obj, multiple .obj's to foo.dll or foo.lib) BIG QUESTION: I know this will work on Unix, and from my distant recollections of past work on other systems, it should work on MS-DOS and VMS too. I gather that Windows is pretty derivative of MS-DOS, so will this model work for Windows compilers too? Do we have to worry about Windows compilers other than VC++? But I have *no clue* about Macintosh compilers -- presumably somebody "out there" (not necessarily on this SIG, but I hope so!) knows how to compile Python on the Mac, so hopefully it's possible to compile Python extensions on the Mac. But will this compiler abstraction model work there? Brushing that moment of self-doubt aside, here's a proposed interface for CCompiler and derived classes. define_macro (name [, value]) define a preprocessor macro or token; this will affect all invocations of the 'compile()' method undefine_macro (name) undefine a preprocessor macro or token add_include_dir (dir) add 'dir' to the list of directories that will be searched by the preprocessor for header files set_include_dir ([dirs]) reset the list of preprocessor search directories; 'dirs' should be a list or tuple of directory names; if not supplied, the list is cleared compile (source, define=macro_list, undef=names, include_dirs=dirs) compile source file(s). 'source' may be a sequence of source filenames, all of which will be compiled, or a single filename to compile. The optional 'define', 'undef', and 'include_dirs' named parameters all augment the lists setup by the above four methods. 'macro_list' is a list of either 2-tuples (macro_name, value) or bare macro names. 'names' is a list of macro names, and 'dirs' a list of directories. add_lib (libname) add a library name to the list of implicit libraries ("-lfoo") to link with set_libs ([libnames]) reset the list of implicit libraries (or clear if 'libnames' not supplied) add_lib_dir (dir) add a directory to the list of library search directories ("-L/foo/bar/baz") used when we link set_lib_dirs ([dirs]) reset (or clear) the list of library search directorie link_shared_object (objects, shared_object, libs=libnames, lib_dirs=dirs) link a set of object files together to create a shared object file. The optional 'libs' and 'lib_dirs' parameters only augment the lists setup by the previous four methods. Things to think about: should there be explicit support for "explicit libraries" (eg. where you put "/foo/bar/libbaz.a" on the command line instead of trusting "-lbaz" to figure it out)? I don't think we can expect the caller to put them in the 'objects' list, because the filenames are too system-dependent. My inclination, as you could probably guess, would be to add methods 'add_explicit_lib()' and 'set_explicit_libs()', and a named parameter 'explicit_libs' to 'link_shared_objects()'. Also, there would have to be methods to support creating static and shared libraries: I would call them 'link_static_lib()' and 'link_shared_lib()'. They would have the same interface as 'link_shared_object()', except the output filename would of course have to be handled differently. (To illustrate: on Unix-y systems, passing shared_object='foo' to 'link_shared_object()' would result in an output file 'foo.so'. But passing output_lib='foo' to 'link_shared_lib()' would result in 'libfoo.so', and passing it to 'link_static_lib()' would result in 'libfoo.a'. So, to all the Windows and Mac experts out there: will this cover it? Can the variations in filename conventions and compilation/link schemes all be shoved under this umbrella? Or is it back to the drawing board? Thanks for your comments! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

6 11

Re: [Distutils] I browsed through the distutils source!
by Michel Sanner 31 Mar '99

31 Mar '99

Hi Greg, sorry , didn't have time to look at this yet .. just a question/suggestion What I really like about Makefiles is the -n mode where I can see what it would do before it actually does it. Is duch an option available in setup.py and if not could it be added. Cheers -Michel On Mar 30, 2:01pm, Greg Ward wrote: > > report on what happened. > > There's not much to running it; from the top distutils directory, do: > > ./setup.py build > > should work pretty much anywhere. A slightly more risky proposition is > > ./setup.py install > > which relies on distutils.sysconfig to get the installation directories; > it in turn relies on finding Python's Makefiles in the usual place. I > have no idea if they're even installed under Win 95 -- please let me > know! > > > Greg, anything you'd like me to examine especially? > > The weak spots! Search for "XXX" in the code; I'm liberal with X-rated > comments. Also check my "Current weaknesses" post from last night; see > if you can correlate my opinions of current problems with the code. > > When it occurs to you that commands are a lot like subroutines, and then > when you start to wonder why parameter passing is done backwards, then > you'll be up to speed. (The whole problem of communicating options > between commands was bigger than I expected. The implementation isn't > overly complicated, but I think it'll take some bouncing around across > various classes and thinking about the alternatives before it becomes > apparent why I did it that way.) > > Oh, the big reason I put the code up now is this: I think it's close to > being at a state where development can be in parallel. The basic > framework is in place, all that's missing is a lot of commands to do the > work. The beginnings of building and installation are in place, and > I've started thinking about the 'build_ext' command -- witness the > thread on compiler abstraction models. But the "dist" and "bdist" > commands -- to create source and built distributions -- are important > and could easily be done by someone else. > > Greg > -- > Greg Ward - software developer gward(a)cnri.reston.va.us > Corporation for National Research Initiatives > 1895 Preston White Drive voice: +1-703-620-8990 x287 > Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://www.python.org/mailman/listinfo/distutils-sig > >-- End of excerpt from Greg Ward

2 1

generating pyc and pyo files
by Andrew Dalke 30 Mar '99

30 Mar '99

When should the .pyc and .pyo files be generated during the install process, in the "build" directory or the "install" one? I ask because I've been looking into GNU autoconf and automake. They compile emacs lisp code (.el->.elc) in the build dir before the actual installation, and it might be nice to follow their lead on that, and it would ensure that the downloaded python files are compileable before they are installed. OTOH, I know the normal Python install does a compileall after the .py files have been transfered to the install directory. Also, looking at buildall, it doesn't give any sort of error status on exit if a file could not be compiled. I would prefer the make process stop if that occurs, so compileall needs to exit with a non-zero value. Attached is a context diff patch to "compileall" from 1.5.1 which supports this ability. I can send the full file as well if someone wants it. Andrew dalke(a)bioreason.com *** /usr/local/lib/python1.5/compileall.py Mon Jul 20 12:33:01 1998 --- compileall.py Sun Mar 28 17:50:51 1999 *************** *** 34,39 **** --- 34,40 ---- print "Can't list", dir names = [] names.sort() + success = 1 for name in names: fullname = os.path.join(dir, name) if ddir: *************** *** 54,64 **** --- 55,67 ---- else: exc_type_name = sys.exc_type.__name__ print 'Sorry:', exc_type_name + ':', print sys.exc_value + success = 0 elif maxlevels > 0 and \ name != os.curdir and name != os.pardir and \ os.path.isdir(fullname) and \ not os.path.islink(fullname): compile_dir(fullname, maxlevels - 1, dfile) + return success def compile_path(skip_curdir=1, maxlevels=0): """Byte-compile all module on sys.path. *************** *** 69,79 **** maxlevels: max recursion level (default 0) """ for dir in sys.path: if (not dir or dir == os.curdir) and skip_curdir: print 'Skipping current directory' else: ! compile_dir(dir, maxlevels) def main(): """Script main program.""" --- 72,84 ---- maxlevels: max recursion level (default 0) """ + success = 1 for dir in sys.path: if (not dir or dir == os.curdir) and skip_curdir: print 'Skipping current directory' else: ! success = success and compile_dir(dir, maxlevels) ! return success def main(): """Script main program.""" *************** *** 98,109 **** sys.exit(2) try: if args: for dir in args: ! compile_dir(dir, maxlevels, ddir) else: ! compile_path() except KeyboardInterrupt: print "\n[interrupt]" if __name__ == '__main__': ! main() --- 103,118 ---- sys.exit(2) try: if args: + success = 1 for dir in args: ! success = success and compile_dir(dir, maxlevels, ddir) else: ! success = compile_path() except KeyboardInterrupt: print "\n[interrupt]" + return success + if __name__ == '__main__': ! if not main(): ! sys.exit(1)

5 15

I browsed through the distutils source!
by Martijn Faassen 30 Mar '99

30 Mar '99

Hi there, This is NOT major news, just a note of encouragement to Greg Ward, and making sure he knows that people are indeed looking at things: I just installed CVS on this win95 system at work, downloaded the distutils source, and read through the sources some. It looks pretty neat. I haven't actually tried *running* distutils yet on this windows box, but I'll try to get to that later this week and give you all a report on what happened. Note on installing CVS on a win95 box: be sure to set the environment variable HOME to some directory or it won't work -- this was not mentioned in any CVS docs I found; presumably in win NT, HOME is already set. Greg, anything you'd like me to examine especially? Regards, Martijn

2 1

Oops; here's my "checkpycs" script
by Fred L. Drake 30 Mar '99

30 Mar '99

Sorry, I forgot to attach the promised script. (That seems to be my standard ritual for attachments!) Anyway, the "checkpycs" script is attached, really. -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives #! /usr/bin/env python # -*- Python -*- import errno import os import stat import string import sys VERBOSE = 0 COLUMN_WIDTHS = (40, 10, 10, 10) s = '' for cw in COLUMN_WIDTHS: if s: s = "%s %%%ds" % (s, cw) else: s = "%%%ds" % cw FORMAT = s del s, cw def process_dirs(dirlist, verbose=VERBOSE): fp = os.popen("find %s -name \*.py -print" % string.join(dirlist)) writeline("", " .py Size", ".pyc Size", ".pyo Size") writeline("", "---------", "---------", "---------") while 1: line = fp.readline() if not line: break filename = line[:-1] pysize = os.stat(filename)[stat.ST_SIZE] or "0" pycsize = pyosize = None if os.path.isfile(filename + "c"): pycsize = os.stat(filename + "c")[stat.ST_SIZE] if os.path.isfile(filename + "o"): pyosize = os.stat(filename + "o")[stat.ST_SIZE] if pycsize or pyosize or VERBOSE: writeline(filename, pysize, pycsize, pyosize) def writeline(c1, c2, c3, c4): c1 = c1 or "" c2 = c2 or "" c3 = c3 or "" c4 = c4 or "" if len(c1) > COLUMN_WIDTHS[0]: c1 = "..." + c1[-(COLUMN_WIDTHS[0] - 3):] print FORMAT % (c1, c2, c3, c4) def main(): try: process_dirs(sys.argv[1:] or ["."]) except IOError, e: if e.errno != errno.EPIPE: raise if __name__ == "__main__": main()

1 0

Current weaknesses
by Greg Ward 29 Mar '99

29 Mar '99

Well, it's been about a week since I announced the first bundle of Distutils code. Haven't heard much back yet, so I assume that it has worked for those of you who tried it. Has anyone really dived in and started poking around the code? If so, you must have stumbled across some of the difficulties I had, including: * it doesn't seem like there's a way to control whether 'compile' generates .pyc or .pyo files * worse, it doesn't look like there's even a way to find out what 'compile' will generate! * the command options describing installation directories are haphazard at best. The problem is, I know pretty much what to call platform-specific library directories: "install_platlib", "install_site_platlib", and so forth. That seems in keeping with the Python Makefiles. But I don't really know what to call non-platform-specific library directories. I take solace in knowing that I am not alone; Perl's MakeMaker just calls them "INSTALLLIBDIR", "INSTALLSITELIB", and so forth, which is where my cop-out of "install_lib" and "install_site_lib" came from. But this isn't really satisfactory... anyone got better ideas? * how do we handle copying file metadata under Mac OS? do the copy routines in distutils.util work under Windows as well as they do under Unix? (the code is mostly stolen from the standard shutil module, so if it works then my copying stuff should) * how should we deal with "wildcard" listing of modules in setup.py? Even for a moderate sized distribution like Distutils, it's already obvious that explicitly listing every module in the distribution is a no-go. See the comments in setup.py for some of my thinking on this matter. More importantly, has anyone gotten deeply confused by trying to associate the code with my two-month-old design proposal? I still need to revisit that document to make sure I haven't grossly violated any of its principles (and change the principles to match the violation if so ;-), so if anybody is getting confused it's probably not just you. Hope to hear some comments soon! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

install location(s)
by Andrew Dalke 29 Mar '99

29 Mar '99

This is an easy couple of questions (I hope). 1) What is the correct location for "pure" python module installations? $(prefix)/lib/python$VERSION/site-packages or $(prefix)/lib/site-python I believe most packages tend to install in site-packages but site-python makes more sense since I usually write my modules to deal with differences in python versions. 2) where should python shared libraries be placed if you want to have one install of python per site which includes different architectures? (For example, we may distribute our software on both SGI and Linux boxes.) As I understand it now, the python specific .so files need to be on the PYTHONPATH, which currently contains no information about the specific platform. The only information available is from sys.platform (along with os.uname) and that doesn't appear sufficient to distinguish between different SGI binary interfaces. Specifically, SGIs have 3 interfaces: "old" 32, "new" 32 and 64. These are specified during compilation by the environment variable SGI_ABI or by the command-line options (-32/-o32, -n32, -64). In theory we could compile libraries for each of the different interfaces, though we only support o32 at present. To do it fully we would have to write a wrapper script which runs the specified ABI version of Python ... oh, but then we could modify the PYTHONPATH to reflect the differences. Has there been a proposal for how to distribute/manage/install distributions with multiple architectures? The best I can think of for now is to modify site.py to add os.path.join(prefix, "lib", "python" + sys.version[:3], "site-packages", sys.platform), to the sitedirs list. Does this make sense, and should it be added to the 1.5.2 (or discussed more on c.l.py)? Andrew dalke(a)bioreason.com BTW, here's how SGI's java, which is only for 32 bits, manages things. "java" is actuall a shell script containing the following three snippets: # use -n32 binaries by default if [[ $SGI_ABI = -32 ]] then export JAVA_N32=0 elif [[ $SGI_ABI = -o32 ]] then export JAVA_N32=0 else export JAVA_N32=1 fi case $a in -32) JAVA_N32=0 shift ;; -o32) JAVA_N32=0 shift ;; -n32) JAVA_N32=1 shift ;; if [ $JAVA_N32 = 1 ] then check_path $LD_LIBRARYN32_PATH "LD_LIBRARYN32_PATH" if [ -z "$LD_LIBRARYN32_PATH" ] then if [ -z "$LD_LIBRARY_PATH" ] then LD_LIBRARYN32_PATH=$JAVA_HOME/lib32/sgi/$THREADS_TYPE else check_path $LD_LIBRARY_PATH "LD_LIBRARY_PATH" LD_LIBRARYN32_PATH="$JAVA_HOME/lib32/sgi/$THREADS_TYPE:$LD_LIBRARY_PATH" fi else LD_LIBRARYN32_PATH="$JAVA_HOME/lib32/sgi/$THREADS_TYPE:$LD_LIBRARYN32_PATH" fi export LD_LIBRARYN32_PATH prog=$JAVA_HOME/bin32/sgi/${THREADS_TYPE}/${progname} else check_path $LD_LIBRARY_PATH "LD_LIBRARY_PATH" if [ -z "$LD_LIBRARY_PATH" ] then LD_LIBRARY_PATH=$JAVA_HOME/lib/sgi/$THREADS_TYPE else LD_LIBRARY_PATH="$JAVA_HOME/lib/sgi/$THREADS_TYPE:$LD_LIBRARY_PATH" fi export LD_LIBRARY_PATH prog=$JAVA_HOME/bin/sgi/${THREADS_TYPE}/${progname} fi

3 3