Mailman 3 - Distutils-SIG

packages and Maintaining Python in general
by Sanner Michel 19 Jan '99

19 Jan '99

Following Barry's advice I post my message on this list too. Sorry for those of you you get it twice, but I wasn't aware of that SIG ! now I am a subscriber :) Hello, I have spend my week end moving opn from Python1.5 to Python1.5.2b1 and have come to the conclusion that there OUGHT to be better way !!! It is a lot of work because I try to maintain Python for - Sgi under Irix5.3 Irix6 and Irix646 - Sun - Dec alpha under OSF1.2 and OSF 1.4 - HP - Linux - Cray T3e and we use many "packages", NumPy, PIL, PyOpenGL. In addition I want to build only one Python environment for each architecture/OS and place it on some disk mounted by all these computers. Although scripts coming with some of the packages help greatly (thanks David) there are still a number of things that I have problems with. Here are some of the problems I have been strugling with: 1 - I started to use Makefile.pre.in to generate make file for additional packages. Currently when any user types "python" he gets 1.5 (current public version). So I decided to build Python1.5.2b1 and install it somewhere not in the users path. The problem is that when I use Makefile.pre.in it uses "python" and gets version 1.5. The only work around I found was to edit the copy of Makefile.pre.in and make it point to the 1.5.2b1 binary. 2 - When building Python for several architectures, the installation script allows to separate platform-dependant and platform-independant files. I'd like the same for packages, i.e. when a package has both types of files I'd like to be able to have a package with that name in the platform-independant AND in the platform-dependant tree. This DOES NOT work currently because import stops searching after it find a package matching that name. 3 - Numeric Python should really be part of the standard distribution .. (please :). Actually I would even argue that it should be part of the Kernel such that developers can rely on the fact that this data type, CRUCIAL for any scientific application and key to Python's success in that field is available. This would also allow for instance to have this datatype included in the Corba mappings for Python. 4 - How difficult would it be to provide some more flexibility in the directory sctructure created by "make install": I'd like to have something like: PYTHONROOT = /mgl/tools/python/$VERSION $PYTHONROOT/include $PYTHONROOT/lib $PYTHONROOT/man $PYTHONROOT/packages # arch independent packages $PYTHONROOT/$ARCH_OS/bin $PYTHONROOT/$ARCH_OS/include $PYTHONROOT/$ARCH_OS/lib $PYTHONROOT/$ARCH_OS/packages # arch dependent packages I do not need the Python version "python1.5" below that. In that way I can install any new version in parallele with an existing one. Any advice/help/modification is very welcome :) I am also going to post some messages that are more specific to particular packages ... so bear with me ! Thanks a lot -Michel -- ------------------------------------------------------------------ Sanner Michel Ph.D. The Scripps Research Institute 10550 North Torrey Pines Road La Jolla, CA 92037 Tel. (619) 784-2341 Fax. (619) 784-2860 e-mail sanner(a)scripps.edu Home Page http://www.scripps.edu/pub/olson-web/people/sanner

2 1

Using getopt for the proposed option processing
by Fred L. Drake 19 Jan '99

19 Jan '99

The option processing Greg proposed in his design doc matches that of CVS: there are a bunch of global options, and then each command has its own options. The big difference is that cvs commands take arguments as well (the list of files to operate on), whereas Greg's proposal is to allow multiple commands, each of which has private options. I don't see anywhere in Greg's proposal a requirement that commands take arguments as well as options, so there's no reason not to use the getopt module. The only aspect of it is remotely interesting is switching in different sets of options for each different command class, and that's pretty trivial. Here's a little demo code. It accepts anything as a command, and just keeps the same options for each, but the comments should make it more than clear what needs to be done where to get the expected behaviour. ------------------------------------------------------------------------ """Example of the command line syntax used by distutils, using getopt.""" import getopt import sys LONGOPTS = ["help", "verbose"] SHORTOPTS = "hv" def main(): args = ["<global>"] + sys.argv[1:] while args: cmd = args[0] # switch in different SHORTOPTS and LONGOPTS in this call to # support different options for different commands: cmdinfo = sys.modules[__name__] # cmdinfo = load_command(cmd) opts, args = getopt.getopt(args[1:], cmdinfo.SHORTOPTS, cmdinfo.LONGOPTS) # do something with the command here: print cmd, opts # create_command(opts).run() if __name__ == "__main__": main() ------------------------------------------------------------------------ -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191

1 0

Comments on proposed design?
by Greg Ward 14 Jan '99

14 Jan '99

Hey folks -- that Distutils design proposal was meant to provoke some discussion! Don't tell me everybody thinks my proposal was so sublimely perfect that I should just go home and knock off version 0.01 over the weekend. There are known problems, things I need help with, etc. Please take a look at it and post your reactions to the list. However, be warned that I am hoping to revise the proposal and post a revised version tonight/tomorrow. I would really like to have some feedback before I start revising. Thanks -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Design proposal
by Greg Ward 11 Jan '99

11 Jan '99

Well, things have been a little quiet on this list. Time to shake it up a bit, methinks. So I've written a design proposal -- here it is. (NB. #1: to avoid sounding overly wafflish, I've written this in a fairly imperious way -- "X will be done this way, Y is like that". Don't take it *too* seriously; this is open for discussion, up for grabs, welcomes your comments and criticisms, etc. Of course, I wouldn't have designed it this way if I didn't think it was good idea, so expect a reaction if you tell me I'm full of baloney... ;-) (NB. #2: this file is in the Distutils CVS archive as 'text/design-proposal.txt'. Once it is better firmed up through the discussion that I'm sure is forthcoming, I'll HTMLify it, put it in the SIG's web space, and start coding!) $Id: design-proposal.txt,v 1.1 1999/01/11 04:06:01 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd [cmd_options] (XXX no provision for multiple commands here! shouldn't be hard, though, as long as we're firm about what a command is and what an option is) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for his module distribution) may define a set of distribution-specific command-line options, which will be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one function, also named for the command -- e.g. 'distutils.build.build'. This function takes a 'Distribution' instance as its only required argument, and then "does its thing" (eg. build extensions, install everything, etc.). All information needed to "do its thing" is contained in the 'Distribution' instance. (XXX this isn't very OO. As long as we can fully parameterize Distutils commands, that's fine: the client (or user) just provides attributes to go into a 'Distribution' instance, and the command modules use those attributes to know what to do. Inevitably, though, someone somewhere will have to override actual code, and this scheme will break down -- either that or we'll have to put a lot of command-specific stuff into the 'Distribution' class, which is bad [makes it harder to add new commands]. A possible alternate formulation: have each command module define a specially-named class, which is then used to "do its thing". If a client setup.py needs to override, say, installation behaviour, then it can create a subclass of distutils.install.Install. It would then need to tell the 'Distribution' class to use its derived class -- not an impossible task!) Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when the 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour -- not just data -- and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward <gward(a)cnri.reston.va.us>" description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure: eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; saves the command, and saves the command-specific options for passing to the command module * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup calls the command module's interface function (eg. 'distutils.build.build'), passing it the Distribution instance and any command-specific options from the setup.py command-line * [alternate formulation: distutils.core.setup instantiates the command module's interface class (eg. 'distutils.build.Build', or an alternate supplied by the client using the as-yet-unmentioned 'command_class' attribute. The Distribution instance and all command-specific options are supplied to the command class so it can "do its thing"] (XXX again, no provision for multiple commands, although it shouldn't be too hard. And the OO approach to writing command modules needs to be better fleshed out.) Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed. (Undoubtedly.) How does the client specify them, and how does the Distutils run them? -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Anonymous CVS up-and-running
by John Skaller 28 Dec '98

28 Dec '98

At 17:43 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > I have done this, it is about 5 minutes work for a makefile > > expert. It took me half a day to get working properly :-) > > Cool. Send Guido a patch and rationale as to why it should be in >the standard distribution. I can't. It was a patch to version 1.5.0 which i don't have any more. It got clobbered when I installed 1.5.1 of course. >Frankly, the reason to install pythonpp is >lost on me. The you don't need to invoke it. But other people in the C++ SIG want to use C++ to extend Python. This is not the place to argue about it, they want it. :-) ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 28 Dec '98

28 Dec '98

At 11:58 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > This depends on knowing where the makefile and > > the config file are. Which suggest a further factorisation: > > Ok, I've modified distutils.sysconfig to contain: > > get_config_h_filename() > get_makefile_filename() > parse_config_h(fileobject, g=None) > parse_makefile(fileobject, g=None) > > Note that the parse_*() routines take a file object instead of a >filename, Ah, that seems even better, yes! >and the dictionary to update may be passed in. If no >dictionary is provided a new one is used. However the dict is >obtained, it is returned by the function. Right. Seems OK. > Equivalents for non-Unix platforms are still needed; has anyone >thought about Windows or Mac platforms? I don't think we (necessarily) need equivalents for the following reason: the sysconfig data isn't meant for user access: its there to support implementation of functions like 'compile_this_c_as_a_module'. It is these functions which need to be standardised (that is, cross platform). For the VC++ windows build, sysconfig could be omitted entirely, and a hand written 'compile' function provided. For Unix, it is convenient to parameterise such a function with the config data -- for that is the way the autoconf/ make/blah blah process works. none of this implies there shouldn't be an equivalent; but I think the important interfaces is the 'OO' one: <grin> the one characterized by function semantics rather than values of data attributes. > I passed on this for now. I think the cross-platform issues need to >be explored more. I agree. I would like people to explore the interface of my compilers module to see if it is _possible_ for that interface to work on all platforms. It may be that the Unix implementation uses sysconfig data to adapt to various Unix platforms, whereas the Mac or Windows version is hard coded, or uses a different scheme. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 24 Dec '98

24 Dec '98

>Can you send around a URL to the interscript online doc? Yes, but I should say that it has been cited many times, it exists on the www.python.org home page in the 'news' list, and it is in comp.lang.python.announce. http://www.triode.net.au/~skaller/interscript is the home page. >And when you >refer to it, could you please cite a more specific reference? Call it a >character flaw, but I'm not going to just blindly go read a bunch of >doc. A concrete reference helps (me, at least) provide some >focus/structure to why I'm there reading the thing in the first place. I agree, but I want to apologise profusely that I am not able to do this easily at the moment. firstly, my working copy is not the same as the one on the Internet, and secondly the HTML pages are all numbered at the moment, and the numbers are arbitrary and will change completely with every build i do that adds or removes even a single heading. [There are over 500 pages] I am addressing this problem, but at the moment the best I can do is usually to say 'read the section about XXX'. The table of contents is not small, but it should guide finding relevant information quickly. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] High level options interface
by John Skaller 24 Dec '98

24 Dec '98

At 11:26 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > BTW: One weakness of this interfaces is that there > > is no provision for creating an object library > > (archive, ar, etc etc). > > Is this needed? I don't know. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Enough about feature tests!
by Greg Ward 23 Dec '98

23 Dec '98

Hmmm, this seems to be a recurring theme; I guess this is just what a moderator is supposed to do, though. Anyways, it should be obvious but apparently is not: arguing about how best to test for OS features through Python is not really on topic for this SIG. We should be arguing, err discussing, high-level design of the distutils suite; or, failing that, nit-picking implementation details. For example, the sysconfig module, what to call it, when it runs, how it provides its information, etc. are all on-topic. Please stick to relevant things and learn to live with the fact that there's no formal specification for the Python Library, nor is there likely to be one anytime soon, nor will we solve that perceived problem with the distutils. Thanks -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Anonymous CVS up-and-running
by John Skaller 23 Dec '98

23 Dec '98

> Agreed. I seem to recall someone had sent patches to support C++, >possibly to the C++-SIG. Does anyone know exactly what's needed? Yes, I know what is 'needed', but not how to get it. What is required is that the python executable be built using C, and another version of it built using C++. Let me call them python.exe and pythonpp.exe. For gcc/g++, three extra steps are required in the makefile: 1) compile python.c using C++ as well as C, name it pythonpp.o(bj). This is the _only_ file that has to be compiled with C++. 2) Link that main object file with the C built python library. The result, pythonpp.exe, works just like python.exe, _except_ that it will accept full C++ dynamic link libraries as python modules: that is, it will support RTTI, exception handling, and proper constructor initialisation of virtual tables, correctly. 3) Install pythonpp.exe as well as python.exe I have done this, it is about 5 minutes work for a makefile expert. It took me half a day to get working properly :-) > > 2) Some of the data, such as CFLAGS, is useless because it reflects > > the python source build tree. On my system > > Misc/Makefile.pre.in manages to get things right in practice; some >time needs to go into figuring out just how it does things and get >sysconfig to do the right thing as well. In my opinion, part of the solution is to factor the 'variables' more logically, and then modify the makefile to use that factorisation. To start with, may I suggest _private_ variables which are not really required for sysconfig, be renamed in the makefile to some convention, such as a leading underscore: these can be ignored by the sysconfig parser as implementation details. Anything with a pathname related to the build directory is likely to be private: for example the current 'CFLAGS' contains -I../../something which is clearly useless to the installed system. So I'd rename that variable _CFLAGS. Then I'd create a real CFLAGS which _only_ has the option flags, no include directories. This allows the flags to be combined with the include directory options obtained for the _installed_ system. (That information is available now in sysconfig -- and indeed directly from Python) > > 3) The directory for config.h is WRONG! > > config.h lives in > > What definition are you looking at? Huh? The code in the CVS archive is wrong (for my system). It creates a directory name using '/usr/local/lib/python1.5/' which is NOT where config.h is installed. It is installed in /usr/local/include/python1.5 I.e. 'lib' should read 'include'. As a result, the module fails on my system. It is a stock standard Linux installation, IMHO it has put config.h in the right place, therefore sysconfig.py is wrong. :-) (Sorry, it is hard for me to quote stuff because I email on NT, but do real work on Linux :-) > > 4) On my system: > > > > SIZEOF_INT = '4' > > Patch attached which gets this right in a general fashion. Thanks! > > 5) I think the module should NOT just put values > > into itself like > > I don't see any need to complicate the interface. Neither do I. Factorisation doesn't complicate the interface, it simplifies it. There are two categories of information, and they come from separate sources. The data strucure should surely reflect this? > > 6) There is no information from Setup. It would be useful to > > get the Tcl/Tk library and headers into the configuration > > if it was built. Hmmm. :( > > Variables defined in Setup & friends are included. Otherwise, >nothing from Setup is included. This can be changed. It may make >sense to use a separate module for this (modconfig?). I think each source of data should be separate, including the makefile info and the autoconf info. The existing architecture is correct, IMHO: its the simplest possible 'pythonic' representation of the raw data. [This is not enough for the high level goal, but it seems correct to build this lower level first] > > 7) _init_posix should NOT modify a dictionary, but return one: > > CHANGE: > > This item is tightly related to item 5; let's deal with that one as >the real issue. The rest will fall out of that result. OK. > > 8) LINKCC = ' gcc' has a bad leading space. > > Fixed in the patch. Thanks. > > 9) I have no idea how to use the available information. > > There are flags all over the place. > > Yes, that's what configure produces. The next layer needs to put >this information together in a usable form. Reading the Makefile >should help a lot in determining how the various bits should be put >together. Yes, I agree. It is clear that while the current set of attributes is not quite right, it contains a lot of information. Therefore, the first step of just getting it into a pythonic format is a good one. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2