Mailman 3 - Distutils-SIG

extensions in packages
by Greg Ward May 20, 1999

May 20, 1999

[I'm going to try to yank this thread over from PSA-members, which should have been done *long* ago!] [on psa-members, Michel Sanner opined:] > I do not think that trying to bend over bacakwards to circumevent this > "limitation" is the right way to proceed .. unless changing the import > mechanism in Python itself is something that would be extremely difficult to > do. I agree that a change to the import mechanism is in order. My understanding (from another one of those office-hallway conversations with Fred) is that it would be very sensible to add a package's platform-dependent directory to the package's __path__ attribute. No frobbing of sys.path is necessary, and thus no danger of stupid name conflicts in extension modules that are supposed to be buried deep in some package structure. Perhaps imputil.py can help us in the playing around stage; I'm not familiar with it, though, so I'll refrain from further comment. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

More about packages
by Michel Sanner May 3, 1999

May 3, 1999

It seems to me that the platform dependent and independent trees of a Python installation are not symetric in some sense: on one side we have: $prefix/include/python$version/ $prefix/lib/python$version/ $prefix/man on the other side $exec_prefix/bin $exec_prefix/include/python$version $exec_prefix/lib/python$version/config $exec_prefix/lib/python$version/lib-dynload $exec_prefix/lib/python$version/site-packages what bothers me is that we do not have that extra level under lib in the platform independent tree. I'd like to have something like: $prefix/lib/python$version/standard/ (equivalent of lib-dynload) $prefix/lib/python$version/packages/ (for paltform independant packages) and these directories should be part of the Python PATH built by default. I am not sure where pur Python packages are supposed to be installed right now ? -Michel

1 0

Python packages
by Michel Sanner May 3, 1999

May 3, 1999

Hello, I am once again in the Python update phase and (of course) running into the same type of problems regarding the installation of packages. I believe that Python should provide support for installing packages that contain both platform independant files (.py) and platform dependant files (.so, .dso .pyd). The reason is that I do not want to install (and maintain) multiple copies of the .py (one for each paltform I support). It seems to me that this is one ogf the benefits of platform independance. The problem right now is that the only way to do this (I know of) if to hack together an __init__.py file for the package placed in the paltform independant part of the installation tree and that would add the right directory to the PATH for importing .so files. The problem with that is that I need to do that for every single package. Would it be unreasonable to have the Python import mechanism check for packages in the $prefix AND the $exec_prefix directory ? -Michel

1 0

PATCH: sysconfig and NT
by Amos Latteier April 16, 1999

April 16, 1999

Here's a patch that seems to get sysconfig minimally working under NT for me. Has anyone else done better? I noticed that distutils installs itself under 'lib\site-packages' which does not seem to be a convention under NT, at least the directory does not exist by default, and is not in the Python path by default. It seems that Windows packages normally go in the top level Python directory--at least that's where the win32 packages install themselves, and it *is* in the Python path by default... Anyway here's the simple-minded patch. -Amos RCS file: /projects/cvsroot/distutils/distutils/sysconfig.py,v retrieving revision 1.5 diff -r1.5 sysconfig.py 130a131,139 > def _init_nt(): > """Initialize the module as appropriate for NT""" > g=globals() > # load config.h, though I don't know how useful this is > parse_config_h(open( > os.path.join(sys.exec_prefix, "include", "config.h")), g) > # set basic install directories > g['LIBDEST']=os.path.join(sys.exec_prefix, "Lib") > g['BINLIBDEST']=os.path.join(sys.exec_prefix, "Lib") 132c141 < --- > 141a151 > del _init_nt

2 2

(final?) automake related questions
by Andrew Dalke April 16, 1999

April 16, 1999

As I've been going on about, I've got the changes made so that GNU automake supports Python. Because it is the FSF, I had to sign paperwork to disclaim copyright interest in our contributions before they would accept it. I mailed them in today. Thus, I've been touching up my changes before submitting them. I wanted to make sure I was using a sensible (or at least reasonable) set of names before I did that. Here are the names for which I am uncertain: o PYTHON_SITE This is the value sys.prefix + "/python" + sys.version[:3] + "/site-packages" It is the location where platform independent modules should be installed. It can be overridden on the configure line with the parameter --with-python-site=DIR o PYTHON_SITE_PACKAGE automake defines the variable PACKAGE, which I am taking also as the module name (nothing unusual about that, I hope). o PYTHON_SITE_EXEC This is the value sys.exec_prefix + "/python" + sys.version[:3] + "/site-packages" It can be overridded on the configure line with the parameter --with-python-site-exec=DIR. I played around with a few names. These seem to make sense to me (eg, PYTHON_SITE_EXEC is better than PYTHON_SITE_SHARED which I had). I cannot find if the standard Python libraries refer to these directories in some other fashion. The closest I can find is "site.py" which combines the first and last directory names directly into "sitedirs". *** question 1: Do these names make sense to others? o I added a new m4 macro for "aclocal" called "AM_CHECK_PYTHON". This is the command which goes in "configure.in" to tell autoconf that it needs to generate Python information. The macro takes one optional parameter. If the value is "classes" the Python files in the package are installed directly in PYTHON_SITE. If the value is "module" then they are installed into PYTHON_SITE_PACKAGE. The installation directory name is stored in the variable PYTHON_SITE_INSTALL. Eg: If you have the file "Vector.py" in the autoconf package "vecmath" with the PYTHON_SITE set to /usr/local/lib/python1.5/site-packages AM_CHECK_PYTHON(module) puts it in /usr/local/lib/python1.5/site-packages/vecmath/Vector.py AM_CHECK_PYTHON(classes) puts it in /usr/local/lib/python1.5/site-packages/Vector.py *** question 2: Should it be "classes" or something else? Since there's no existing python command-line program to compile a *single* python program (as compared to compileall which compiles, well, all :), I'm using the following in the Makefile (the value of PYTHON is the full location of the python executable and was found during the configure step): PYTHON_COMPILE_CMD = "-c 'import py_compile, sys; \ py_compile.compile(sys.argv[1])' PYTHON_OCOMPILE = $(PYTHON) $(PYTHON_COMPILE_CMD) PYTHON_OCOMPILE = $(PYTHON) -c $(PYTHON_COMPILE_CMD) .py.pyc: @echo "Compiling python file $< " @$(PYTHON_COMPILE) $< .py.pyo: @echo "Compiling python file $< (optimized)" @$(PYTHON_OCOMPILE) $< The idea is that I hope distutils will have a command-line way to do this, so I can replace all that with: .py.pyc: @$(PYTHON) $pythondir/distutils/compile.pr $< .py.pyo: @$(PYTHON) -O $pythondir/distutils/compile.py $< (I hide the actual compile with @ because otherwise you just see the full "import .... argv[1])" which is hard to read. By having a program to byte compile a file directly I can get rid of the noise.) *** question 3: Can distutils provide this sort of service? *** question 4: Anyone interested in getting the latest CVS distribution of automake (after I get the changes in) to play^U test things out? To help out, I'm also working on documentation so people can use it without reading the full automake documentation (which has a strong emphasis on building C programs). Not suprisingly, this email will be a hefty part of said documentation :) Andrew dalke(a)bioreason.com P.S. Here's the list of other definitions created by configure.in. I cannot see how there's any real problem with these names, but who knows .... PYTHON = full name of the python executable PYTHON_PREFIX = sys.prefix PYTHON_EXEC_PREFIX = sys.exec_prefix PYTHON_VERSION = sys.version PYTHON_PLATFORM = sys.platform pythondir = sys.prefix + "/python" + sys.version[:3]

4 10

Some code to play with
by Greg Ward April 14, 1999

April 14, 1999

Hi all -- as promised, I have finally checked in some real Distutils code. There's just enough functionality for the Distutils to build and install itself. To try it out, you'll need to download the code from the anonymous CVS archive at cvs.python.org; see http://www.python.org/sigs/distutils-sig/cvs.html for instructions. If you just want to try it out, then from the top-level Distutils directory (the one that has setup.py), run: ./setup.py build install This will copy all the .py files to a mockup installation tree in 'build', compile them (to .pyc -- don't run setup.py with "python -O", as I don't handle that yet and it will get confused), and then copy the whole mockup tree to the 'site-packages' directory under your Python library directory. Figuring the installation directory relies on Fred Drake's 'sysconfig' module -- I don't recall what the status of sysconfig was on non-Unix systems; hope someone out there can try it and let us know! Come to think of it, I'd like to hear how it works on *any* system apart from my Red Hat Linux 5.2, stock Python 1.5.1, home PC. ;-) If you want to build in a different directory, use the '--basedir' option to the 'build' command; to install in a different place, you can use either the '--prefix' or '--install-lib' options to the 'install' command. Here are a couple of samples to demonstrate: ./setup.py build --basedir=/tmp/build install --prefix=/tmp/usr/local or ./setup.py build --basedir=/tmp/build ./setup.py install --build-base=/tmp/build --prefix=/tmp/usr/local Go crazy. No documentation yet -- please read the code! Start with distutils/core.py which, as its name implies, is the start of everything. I went nuts with docstrings in that module last night, so hopefully there's just enough there that you can figure things out in the absence of the "Distutils Implementation Notes" document that I'd like to write. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 2

Fixed "-n" bug
by Greg Ward April 4, 1999

April 4, 1999

Hi all -- I've just fixed the bug that Michel Sanner inadvertently pointed out -- the "-n" ("--dry-run") flag was universally ignored. This was solved by adding several methods to the Command base class through which all "external actions" (anything that affects the outside world, most particularly the filesystem) should be channelled. Now the verbose and dry-run flags can be respected in a consistently painless way by *all* command classes. In addition to "cvs update", you can now download a snapshot tarball: http://www.python.org/sigs/distutils-sig/distutils-19990403.tar.gz I make no promises about the longevity or reliability of that URL -- it'll probably go away the next time I make a snapshot. Let me know if you like the snapshot, but would prefer a zip file to a tarball; under duress, I could hack the 'mksnap' script to make both. ;-) Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Compiler abstractiom model
by Paul F. Dubois April 1, 1999

April 1, 1999

Just to chime in about the unmentionable, I have a tool for connecting Fortran and Python (as do others; mine isn't ready for the light of day yet) and unfortunately the existing Setup scheme makes me compile my Fortran separately in a library. I don't have any way to piggyback on what Python has learned about the platform. This is too bad. I am sure it isn't the Python community's job to solve the Fortran 90 make problem but there are days when I dream about it. The "right" object-oriented approach should allow some way to extend the system, e.g. adding suffices or file names together with info about how to compile them. -----Original Message----- From: Mark Hammond <MHammond(a)skippinet.com.au> To: distutils-sig(a)python.org <distutils-sig(a)python.org> Date: Tuesday, March 30, 1999 3:50 PM Subject: RE: [Distutils] Compiler abstractiom model >> > Quoth David Ascher, on 29 March 1999: >> > > On windows, it is sometimes needed to specify other files >> which aren't .c >> > > files, but .def files (possibly all can be done with >> command line options, >> > > but might as well build this in). I don't know how these >> should fit in... >> > >> > Are they just listed in the command line like .c files? Or >are they >> > specified by a command-line option? Would you use these in >> code that's >> > meant to be portable to other platforms? >> >> Yes, no, and yes. =) >> >> E.g. Python extensions need to declare the 'exported' entry >> point. This >> can be done either by modifying the source code (bad for >> portable code, >> requires #ifdef's etc.), by specifying a command-line option, >or by >> including a .DEF file. > >Just to follow up on this, many obscure options may need to be >passed to the Windows linker, but they do require their own >option - they are not passed as normal files. Examples are >/DEF: - the .def file David mentions, /NOD:lib - to prevent a >default library from being linked, /implib:lib to name the built >.lib file, etc. > >It wasnt clear from David's post that these need their own >option, and are not passed as a simple filename param to the >linker.... > >Building from what Greg said, I agree that _certain_ command-line >params can be mandated - they are designed not to be >configurable, as messing them will likely break the build. But >there also needs to be a secondary class that are virtually >unconstrained. > >[Sorry - as usual, speaking having kept only a quick eye on the >posts, and not having seen the latest code drop...] > >Mark. > > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG(a)python.org >http://www.python.org/mailman/listinfo/distutils-sig > >

3 3

Compiler abstractiom model
by Greg Ward March 31, 1999

March 31, 1999

Hi all -- I've finally done some thinking and scribbling on how to build extensions -- well, C/C++ extensions for CPython. Java extensions for JPython will have to wait, but they are definitely looming on the horizon as something Distutils will have to handle. Anyways, here are the conclusions I've arrived at. * Stick with C/C++ for now; don't worry about other languages (yet). That way we can be smart about C/C++ things like preprocessor tokens and macros, include directories, shared vs static libraries, source and object files, etc. * At the highest level, we should just be able to say "I know nothing, just give me a compiler object". This implies a factory function returning instances of concrete classes derived from an abstract CCompiler class. These compiler objects must know how to: - compile .c -> .o (or local equivalent) - compile multiple .c's to matching .o's - be able to define/undefine preprocessor macros/tokens - be able to supply preprocessor search directories - link multiple .o's to static library (libfoo.a, or local equiv.) - link multiple .o's to shared library (libfoo.so, or local equiv.) - link multiple .o's to shared object (foo.so, or local equiv.) - for all link steps: + be able to supply explicit libraries (/foo/bar/libbaz.a) + be able to supply implicit libraries (-lbaz) + be able to supply search directories for implicit libraries - do all this with timestamp-based dependency analysis (non-trivial because it requires analyzing header dependencies!) Linking to static/shared libraries and dependency analysis are optional for now; everything else is required to build C/C++ extensions for Python. (At least that's my impression!) "Local equivalent" is meant to encompass different filenames for C++ (eg. .C -> .o) and different operating systems/compilers (eg. .c -> .obj, multiple .obj's to foo.dll or foo.lib) BIG QUESTION: I know this will work on Unix, and from my distant recollections of past work on other systems, it should work on MS-DOS and VMS too. I gather that Windows is pretty derivative of MS-DOS, so will this model work for Windows compilers too? Do we have to worry about Windows compilers other than VC++? But I have *no clue* about Macintosh compilers -- presumably somebody "out there" (not necessarily on this SIG, but I hope so!) knows how to compile Python on the Mac, so hopefully it's possible to compile Python extensions on the Mac. But will this compiler abstraction model work there? Brushing that moment of self-doubt aside, here's a proposed interface for CCompiler and derived classes. define_macro (name [, value]) define a preprocessor macro or token; this will affect all invocations of the 'compile()' method undefine_macro (name) undefine a preprocessor macro or token add_include_dir (dir) add 'dir' to the list of directories that will be searched by the preprocessor for header files set_include_dir ([dirs]) reset the list of preprocessor search directories; 'dirs' should be a list or tuple of directory names; if not supplied, the list is cleared compile (source, define=macro_list, undef=names, include_dirs=dirs) compile source file(s). 'source' may be a sequence of source filenames, all of which will be compiled, or a single filename to compile. The optional 'define', 'undef', and 'include_dirs' named parameters all augment the lists setup by the above four methods. 'macro_list' is a list of either 2-tuples (macro_name, value) or bare macro names. 'names' is a list of macro names, and 'dirs' a list of directories. add_lib (libname) add a library name to the list of implicit libraries ("-lfoo") to link with set_libs ([libnames]) reset the list of implicit libraries (or clear if 'libnames' not supplied) add_lib_dir (dir) add a directory to the list of library search directories ("-L/foo/bar/baz") used when we link set_lib_dirs ([dirs]) reset (or clear) the list of library search directorie link_shared_object (objects, shared_object, libs=libnames, lib_dirs=dirs) link a set of object files together to create a shared object file. The optional 'libs' and 'lib_dirs' parameters only augment the lists setup by the previous four methods. Things to think about: should there be explicit support for "explicit libraries" (eg. where you put "/foo/bar/libbaz.a" on the command line instead of trusting "-lbaz" to figure it out)? I don't think we can expect the caller to put them in the 'objects' list, because the filenames are too system-dependent. My inclination, as you could probably guess, would be to add methods 'add_explicit_lib()' and 'set_explicit_libs()', and a named parameter 'explicit_libs' to 'link_shared_objects()'. Also, there would have to be methods to support creating static and shared libraries: I would call them 'link_static_lib()' and 'link_shared_lib()'. They would have the same interface as 'link_shared_object()', except the output filename would of course have to be handled differently. (To illustrate: on Unix-y systems, passing shared_object='foo' to 'link_shared_object()' would result in an output file 'foo.so'. But passing output_lib='foo' to 'link_shared_lib()' would result in 'libfoo.so', and passing it to 'link_static_lib()' would result in 'libfoo.a'. So, to all the Windows and Mac experts out there: will this cover it? Can the variations in filename conventions and compilation/link schemes all be shoved under this umbrella? Or is it back to the drawing board? Thanks for your comments! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

6 11

Re: [Distutils] I browsed through the distutils source!
by Michel Sanner March 31, 1999

March 31, 1999

Hi Greg, sorry , didn't have time to look at this yet .. just a question/suggestion What I really like about Makefiles is the -n mode where I can see what it would do before it actually does it. Is duch an option available in setup.py and if not could it be added. Cheers -Michel On Mar 30, 2:01pm, Greg Ward wrote: > > report on what happened. > > There's not much to running it; from the top distutils directory, do: > > ./setup.py build > > should work pretty much anywhere. A slightly more risky proposition is > > ./setup.py install > > which relies on distutils.sysconfig to get the installation directories; > it in turn relies on finding Python's Makefiles in the usual place. I > have no idea if they're even installed under Win 95 -- please let me > know! > > > Greg, anything you'd like me to examine especially? > > The weak spots! Search for "XXX" in the code; I'm liberal with X-rated > comments. Also check my "Current weaknesses" post from last night; see > if you can correlate my opinions of current problems with the code. > > When it occurs to you that commands are a lot like subroutines, and then > when you start to wonder why parameter passing is done backwards, then > you'll be up to speed. (The whole problem of communicating options > between commands was bigger than I expected. The implementation isn't > overly complicated, but I think it'll take some bouncing around across > various classes and thinking about the alternatives before it becomes > apparent why I did it that way.) > > Oh, the big reason I put the code up now is this: I think it's close to > being at a state where development can be in parallel. The basic > framework is in place, all that's missing is a lot of commands to do the > work. The beginnings of building and installation are in place, and > I've started thinking about the 'build_ext' command -- witness the > thread on compiler abstraction models. But the "dist" and "bdist" > commands -- to create source and built distributions -- are important > and could easily be done by someone else. > > Greg > -- > Greg Ward - software developer gward(a)cnri.reston.va.us > Corporation for National Research Initiatives > 1895 Preston White Drive voice: +1-703-620-8990 x287 > Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG(a)python.org > http://www.python.org/mailman/listinfo/distutils-sig > >-- End of excerpt from Greg Ward

2 1