Mailman 3 January 1999 - Distutils-SIG

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

Re: new import mechanism and a "small" distribution
by Greg Stein Feb. 1, 1999

Feb. 1, 1999

Just van Rossum wrote: > > At 8:00 AM -0800 1/31/99, Greg Stein wrote: > >... > >Okay... enough background and rambling. If you're interested, go look at the > >four messages in the thread titled "Freeze and new import architecture" in > >the distutils-sig archive at: > >http://www.python.org/pipermail/distutils-sig/1998-December/thread.html > > I'm not on that sig so I missed your post originally. I agree with most you > say here: > http://www.python.org/pipermail/distutils-sig/1998-December/000077.html > Especially that entries in sys.path should be loader instances (or > directory paths). > > Some questions: > - what is the interface of a loader By "loader", I will presume that you mean an instance of an Importer subclass that is defining get_code(). Here is the method as defined by imputil.Importer: def get_code(self, parent, modname, fqname): """Find and retrieve the code for the given module. parent specifies a parent module to define a context for importing. It may be None, indicating no particular context for the search. modname specifies a single module (not dotted) within the parent. fqname specifies the fully-qualified module name. This is a (potentially) dotted name from the "root" of the module namespace down to the modname. If there is no parent, then modname==fqname. This method should return None, a 2-tuple, or a 3-tuple. * If the module was not found, then None should be returned. * The first item of the 2- or 3-tuple should be the integer 0 or 1, specifying whether the module that was found is a package or not. * The second item is the code object for the module (it will be executed within the new module's namespace). * If present, the third item is a dictionary of name/value pairs that will be inserted into new module before the code object is executed. This provided in case the module's code expects certain values (such as where the module was found). """ raise RuntimeError, "get_code not implemented" That method is the sole interface used by Importer subclasses. To define a custom import mechanism, you would just derive from imputil.Importer and override that one method. I'm not sure if that answers your question, however. Please let me know if something is unclear so that I can correct the docstring. Oh, geez. And I totally spaced on one feature of imputil.py. There is a way to define an import mechanism for very simple uses. I created a subclass named FuncImporter that delegates the get_code() method to a user-supplied function. This allows a user to do something like this: import imputil def get_code(parent, modname, fqname): ...do something... imputil.install_with(get_code) The install_with() utility simply creates a FuncImporter with the specified function and then installs the importer. No need to mess with subclasses. > - how are packages identified If get_code() determines that the requested module is a package, then it should return the integer 1 along with the code object for that package's module. In the standard package system, the code object is loaded from __init__.py. An example: let's say that get_code() is called with (None, "mypkg", "mypkg") for its arguments. get_code() finds "mypkg" in whatever path it is configured for, and then determines that it represents a directory. It looks in the directory for __init__.py or .pyc. If it finds it, then mypkg is a real package. It loads code from __init__.py and returns (1, code). The Importer superclass will create a new module for "mypkg" and execute the code within it, and then label it as a package (for future reference). Internally, packages are labelled with a module-level name: __ispkg__. That is set to 0 or 1 accordingly. The Importer that actually imports a module places itself into the module with "__importer__ = self". This latter variable is to allow Importers to only work on modules that they imported themselves, and helps with identifying the context of an importer (whether an import is being performed by a module within a package, or not). > - can it be in Python 1.6 or sooner? I imagine that it could be in Python 1.6, but I doubt that it would go into Python 1.5.2, as it has not had enough exposure yet. Guido's call on all counts, though :-) > PS: I was just wondering, why doesn't reload() use the standard import hook? The standard import hook returns a *new* module object. reload() must repopulate the module object. I'm not sure that I answered your questions precisely, as they were rather non-specific. If I haven't answered well enough, then please ask again and I'll try again :-). Cheers, -g -- Greg Stein, http://www.lyra.org/

2 5

FW: new import mechanism and a "small" distribution
by Greg Stein Jan. 31, 1999

Jan. 31, 1999

crap... missed the "s" in distutils. what fun now -- divergent threads (here and python-list). sigh. -----Original Message----- From: Greg Stein [mailto:gstein@lyra.org] Sent: Sunday, January 31, 1999 8:00 AM To: distutil-sig(a)python.org Cc: python-list(a)python.org Subject: new import mechanism and a "small" distribution A while back, Mark Hammond brought up a "new import architecture" proposal after some discussion with Jack Jansen and Guido. I responded in my usual diplomatic style and said "feh. the two-step import style is bunk." Then, I expounded on a few of the ideas that I had come up with after working with a lot of custom import stuff (for COM support, for sticking .pyc files into resources or structured-storage files, or for creating distribs with very few files). In essence, I maintained that importers should really be a single step find-and-load, and that Python would be best served by having a list of object instances, each performing this find-and-load step. Imagine, if you will, replacing sys.path with a list of "directory importers" where each importer checks a directory for the requested file. Next step in our hypothetical case is to add a new importer into sys.path that checks the Windows Registry for paths and looks there, or an importer that looks inside the resource fork of a Mac file. Heck, or use the typical example: have an importer that grabs a code object from over the web. Okay... enough background and rambling. If you're interested, go look at the four messages in the thread titled "Freeze and new import architecture" in the distutils-sig archive at: http://www.python.org/pipermail/distutils-sig/1998-December/thread.html What I'm posting here is two things: 1) imputil.py : this is a module that defines an import utility class (imputil.Importer). There are a few subclasses in here for different types of importers. More detail below. 2) a "small distribution" : the demo is for Win32, but it the basic mechanism is quite portable. In this small distro, you can place onto somebody's computer a mere 5 files and NO registry keys. Python starts up just fine and you can import all modules from the standard library. Basically, imputil.Importer's logic was derived from knee.py, which is an emulation of Python's algorithm. Importer defines one method that subclasses should override: get_code(). The doc strings and examples in the module cover that method relatively well. The Importer class will install an import hook and "chain" to the previous hook. This allows the use of multiple Importers, each one attempting to perform the import, and finally falling back to the builtin import mechanism. (note that there is no way to deinstall an importer at the moment, as it can't go "up" the chain to detach itself; the "right" way to do this is to define an Importer that contains a manageable list of Importers, and removing the chaining in favor of the manager iterating over the list) The small distribution replaces site.py and uses an archive of the standard library to fetch code objects from. This library is named "py15.pyl" in the demo, and is generated by the batch file in support/regen.bat (with the help of support/easygen.py). The resulting library is portable, as it is merely a giant glom of .pyc files and a marshalled table of contents (mapping module names to file positions). Look in site.py for the SimpleArchive class to get a feel for how to write a simple subclass of Importer. As another demo, imputil.py contains a class named DirectoryImporter. This subclass attempts to emulate Python's behavior for importing from a directory. It is instantiated with a directory to look in, for modules and packages. If a module is found, it will perform the usual .pyc caching if necessary (or use the cached .pyc). There is a demo function named imputil._test_dir() that will read sys.path and construct a chain of DirectoryImporter instances. You can then import away and Python will actually use the Importers rather than sys.path iteration. Lastly, imputil.py contains two (untested) subclasses for importing from a package contained in archive-type files (such as .gz, .zip, .tar, or a Mac file of resources). For example, you could distribute "package.gz" along with a gzip Importer. When somebody said "import package.module", it would look inside package.gz for the "module" module. I think that's it. This is the first release of this stuff, so there are bound to be improvements or bugfixes or just simple comments :-). Please feel free to speak up. thx, -g -- Greg Stein, http://www.lyra.org/

1 0

Responses to proposal
by Greg Ward Jan. 20, 1999

Jan. 20, 1999

[Fred says the need for application-specific command-line options is not clear] My impetus for this was the need for specific module distributions to be told where to go looking for external C libraries. For instance, the setup.py for zlib would need to know where to find zlib.h and libz.a -- that is, how to generate a -I option for compiling the extension, and a -L option for linking it (modulo the usual Unixisms). Likewise, tkinter needs to be told where to find the Tcl and Tk headers and libraries. Granted, there's already a standard way of doing this: the Setup file. Perhaps we should preserve the Setup file, or perhaps we should mutate it into a Distutils-specific 'options' file as per Martijn's suggestion for replacing my command-line options. However, let's not limit ourselves to "where's this header file, where's that library?" questions: lots of modules could have configuration options that the builder might want to supply at build time. And keep in mind that ultimately we want to be able to write setup.py for a whole Python *applications*, which almost certainly will have configuration options: where to put temporary files? permanent data files? installation directory? etc. [Martijn Faassen berates me for forgetting about abstract compiler classes] > You don't expose Unixish C compilation variables to the user at all, > perhaps? At least in principle. In practice you'd still allow > modification, but it shouldn't be *necessary* to look at these or even > know about them, even to write a C extension. (unless you're on some new > platform) Duh, you're right. I had *Programming Perl* sitting on my lap and open to the section on MakeMaker as I wrote that part of the design proposal. (Mainly I wanted to be sure I didn't forget anything obvious.) Small wonder a few Unixisms creeped in! Anyways, this raises the broader question: how should the class(es) that abstract compilation and linking behave? I'll try to come up with a few ideas and post them separately. (Arg, really must look at Interscript's compilation stuff... presumably John has already thought through this stuff!) Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

3 3

Revised design proposal
by Greg Ward Jan. 20, 1999

Jan. 20, 1999

Hi all -- as threatened, here's a revised design proposal. Main changes: I've commited myself to and firmed up the language regarding 1) multiple commands on the setup.py command line, and 2) OO command modules. There're still some unresolved issues, and I'll be posting on them separately soon. BTW if you can't be bothered to read the whole thing, at least skip down to and read "Part 3", a sort of executive summary (for executives who want to know every step the code will take, but not know the background -- that's what the other 250 lines of text are for!). Enjoy. --snip----------------------------------------------------------------- $Id: design-proposal.txt,v 1.2 1999/01/15 03:52:30 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd1 [cmd_options] [cmd2 [cmd_options]] ... (Distinguishing the end of the option list for cmd1 from cmd2 might be a bit tricky if command options can take arguments, which they certainly should do. If we want getopt-style syntax, it looks like we'll have to reinvent the getopt module. :-( And if we opt for a different syntax, we'll just have to write something to parse that! It's worth the effort in order to be able to say "./setup.py build test install", though.) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for this module distribution) may define a set of distribution-specific command-line options, which may be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business is to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine Please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. 'distutils.core' also defines the 'Command' class which comes in handy for implementin Distutils commands. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one class, also named for the command -- e.g. 'distutils.build.Build'. These command classes will inherit from the 'Command' class, which (at the very least) will provide a means of dealing with command-specific options. (Probably 'Command' will provide a constructor that takes a 'Distribution' class and an optional list of arguments for this command, and parse the argument list by inspecting getopt-style option specifiers in the 'Command'-derived instance.) Each command class must provide a method ('go'? 'run'? 'doit'?) that uses the information in the 'Distribution' instance and the command options to "do its thing". Well-written command classes will parcel this task out into several well-defined (and documented) methods, so that the client setup.py may inherit from and override specific behaviours of a Distutils command class. This also means that the 'Distribution' class must have a way to communicate overridden command classes to the main dispatcher. Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour, not just data, and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward <gward(a)cnri.reston.va.us>" description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure, eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST by default] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; figures out the commands and options for each command, saving them all up for later processing * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution, probably with a bunch of other named arguments specifying various attributes for the Distribution instance) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup determines the command class (usually just named for the command, eg. 'distutils.build.Build, but possibly a class supplied by the client as one of the attributes of the Distribution instance) and instantiates it * the command class constructor takes as arguments the Distribution instance and any command-line arguments specific to this command on the setup.py command line * the command class constructor parses its options to set/override some instance attributes * distutils.core.setup calls the "make it so" method provided by the command class * that method does whatever the command is supposed to do: build modules, process documentation, install files, etc. * distutils.core.setup determines the next command class (if multiple commands were given), and proceeds as before Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed? (Undoubtedly.) How does the client specify them, and how do the Distutils run them? -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

6 7

Re: [Distutils] packages and Maintaining Python in general
by Michel Sanner Jan. 19, 1999

Jan. 19, 1999

On Jan 19, 8:57am, Greg Ward wrote: > Subject: Re: [Distutils] packages and Maintaining Python in general > Quoth Sanner Michel, on 18 January 1999: > > > 1 - I started to use Makefile.pre.in to generate make file for > > additional packages. Currently when any user types "python" he gets > > 1.5 (current public version). So I decided to build Python1.5.2b1 and > > install it somewhere not in the users path. The problem is that when I > > use Makefile.pre.in it uses "python" and gets version 1.5. The only > > work around I found was to edit the copy of Makefile.pre.in and make > > it point to the 1.5.2b1 binary. > > This will definitely be addressed by the Distutils suite: > Makefile.pre.in will be a thing of the past, replaced by a (usually) > tiny Python script. Normally this will just be run as "./setup.py", but > of course you can always say "python1.5.2b1 setup.py" and the build > process will follow the defaults associated with *that* particular > binary (eg. compiler flags, installation directories, etc.). > Cool I see you guys already have a plan :). I guess I will have to browse the archive of the mailing list to come up to speed ! Anyways is the plan to have something available under 1.6 ? or maybe 1.5.2 (non beta ?) > > 2 - When building Python for several architectures, the installation > > script allows to separate platform-dependant and platform-independant > > files. I'd like the same for packages, i.e. when a package has both > > types of files I'd like to be able to have a package with that name in > > the platform-independant AND in the platform-dependant tree. This DOES > > NOT work currently because import stops searching after it find a > > package matching that name. > > I completely agree. I think this will require some cooperation on the > part of Python's configure/build process, and probably the startup code > that initializes sys.path. So it's only likely to happen when/if a) > Distutils comes to pass as planned, b) Python 1.6 goes out. But it's > definitely an important thing to keep in mind. > Yeap, the main problem currently is that import stops searching after it find a package with the right name. So you could NOT have an OpenGL package (for instance) under $prefix and another OpenGL package under $exec_prefix. Is someone talking to Guido about this. It seems to me that this separation between acrh-dependant and arch-independant files should be allowed for packages. > > 3 - Numeric Python should really be part of the standard distribution > > .. (please :). Actually I would even argue that it should be part of > > the Kernel such that developers can rely on the fact that this data > > type, CRUCIAL for any scientific application and key to Python's > > success in that field is available. This would also allow for > > instance to have this datatype included in the Corba mappings for > > Python. > > I disagree. The whole point of the Distutils SIG is to make it trivial > to add third-party modules to an existing Python installation; if NumPy > being crucial for scientists justifies adding it to the core, then why > not add PIL for image-processors, and so forth? Where do you draw the > line? > OK ! I think my level of frustration made me loose my objectivity :) .. although there is something special about Numeric: it implements a basic datatype: ARRAYS that are as basic as lists or tuples. This I think makes a difference with PIL for instance that might use Numeric Arrays to store images for instance. Moreover having this datatype be a part of Python itself would allow to define CORBA mappings (which again I am personally interested in :)... Just pushing my luck here :) -Michel

1 0

Weekly/biweekly development report?
by Andrew M. Kuchling Jan. 19, 1999

Jan. 19, 1999

In the Distutils-SIG, Martijn Faassen wrote: >Perhaps a special 'distutils SIG' status report, posted once every while >and updating everybody on developments would be useful to attract >interest. And what about something in the Python Journal? A recent trend for development projects has been to issue a summary of activity at periodic intervals. It may have started with the development summary at Linux Weekly News, but it's been picked up by other groups such as the Debian GNU/Linux distribution and Mozilla.org. There are also development diaries from various people, most notably Alan Cox: LWN Development page for this week: http://www.lwn.net/1999/0114/devel.phtml LWN Kernel development page for this week: http://www.lwn.net/1999/0114/kernel.phtml Debian Weekly News: http://www.debian.org/~joeyh/debiannews.html Mozilla.org status update: http://www.mozilla.org/status/ Linux kernel update: http://lynx.neu.edu/z/zbrown/kt.html Alan Cox's diary: http://www.linux.org.uk/diary/ Should we attempt to have a monthly summary of Python development activity? This is slightly different from the Python-URL postings, which aren't focused exclusively on development, and don't have much editorial commentary. I like LWN's habit of occasionally commenting on the news, saying "This flamewar was rather pointless" or "This recent argument was upsetting because...". The Python version of this might look like: Distutils SIG: Greg Ward posted a proposed interface (URL to a copy). Once some people have commented on it, implementation work should begin. Zope: DC released new database adapters X,Y,Z. Skip Montanaro announced thing W. Python-list: Endless discussion of assignment in while statements; GvR said he's not interested. ... ...etc... Pulling together the information requires several volunteers, because no one can possibly be reading every single relevant mailing list and newsgroup. How could that be made workable? Have a central editor? A CGI script which lets anyone add suggested items, which an editor draws upon for each issue and then gets cleaned out when the issue is released? Include it as part of the Python Journal? Suggestions? -- A.M. Kuchling http://starship.skyport.net/crew/amk/ He told me he was having the time of his life and then fell to the floor unconscious. I didn't take this too personally, although he was not the first person to collapse while talking to me; to tell the truth it has happened quite often. -- Tom Baker, in his autobiography

1 0

setup.py command-line syntax
by Greg Ward Jan. 19, 1999

Jan. 19, 1999

OK, time for the first of those unresolved issues. This one isn't actually mentioned as such in the "Unresolved issues" section of the design proposal, because if you carefully read and think through what's already there you'll start to see the holes in my proposal. In particular, we need to figure out what the syntax for the 'setup.py' command line is going to be. I have a couple of ideas, none of them wholly satisfactory. In vague terms, I'm currently thinking that the syntax will be like this: ./setup.py [global_options] cmd1 [cmd1_options] [cmd2 [cmd2_options]] ... The specific problem I'm having is: what will options look like, in particular options with arguments, and how do we know when the options for cmd1 are over and we're staring at cmd2? Option 1: getopt-ish Options are either of the '-x' form (single dash, single letter, may be jammed together, arguments may follow with or without intervening space) or of '--long-opt' form (two dashes, dash-or-underscore-separated-words, arguments will follow either '=' or whitespace). Pros: * familar interface: just like the getopt module, except that there are these "command" things intermingled with options, and there are multiple sources for option descriptors (the standard 'Distribution' class provided by distutils.setup.core, the individual command classes provided by the command modules, and possibly setup.py). Cons: * those complications mean that we'd have to reimplement the getopt module for distutils, with hooks to the various sources of option descriptors specific to distutils * due to many ways of specifying option arguments ("-a foo", "-afoo", "--aardvark=foo", "--aardvark foo"), can't tell where option lists ends and next command begins without fully processing the option descriptors and argument lists. This could induce temporal whiplash. Option 2: restricted getopt-ish Options are only of the '--long-opt' form, and option arguments may only be specified as '--option=value' -- no spaces between option and value (ie. they're the same command line word). Pros: * much easier to implement than a getopt knock-off * don't have to fully process options to know where cmd1's options stop and cmd2 starts: just look for first ! /^--/ after cmd1 * easy to make a standard way to negate boolean (no argument) options: just accept "--no-flag" in addition to "--flag" Cons: * perversion of a familiar interface; people will think that because they can say "--option=value" they can equivalently say "--option value" and will get confused when distutils tells them "no such command 'value'" (err, except it will *also* tell them "option '--option' requires an argument"... maybe not so bad then) * needless typing; double-dashes on long options are really only necessary to distinguish them from short options, and if we don't support short options why bother with the double dashes? Option 3: who needs dashes anyways? Options are always of the "option=value" form -- no dashes, must always have a value (even if that value is only 'yes' or 'no'). Pros: * simple to implement, simple to explain * no confusion with existing interfaces (except... see "Cons") * dead easy to spot end of 'option=value' list after each command Cons: * looks like we're running a Makefile! * no nice way to do boolean (argument-less) options Option 4: who needs command options anyways? Change the syntax to ./setup.py [options] cmd1 [cmd2 ...] and make all options the responsibility of distutils.core (or ensure that the client seutp.py and the various command classes get their option descriptors to distutils.core). Pros: * assuming we can get all the option descriptors to distutils.core in time, can just use standard 'getopt' module; the leftover arguments are of course the commands Cons: * unclear from the command-line whose responsibility each option is Anyone have any other schemes? Comments on these? Preferences? Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

packages and Maintaining Python in general
by Sanner Michel Jan. 19, 1999

Jan. 19, 1999

Following Barry's advice I post my message on this list too. Sorry for those of you you get it twice, but I wasn't aware of that SIG ! now I am a subscriber :) Hello, I have spend my week end moving opn from Python1.5 to Python1.5.2b1 and have come to the conclusion that there OUGHT to be better way !!! It is a lot of work because I try to maintain Python for - Sgi under Irix5.3 Irix6 and Irix646 - Sun - Dec alpha under OSF1.2 and OSF 1.4 - HP - Linux - Cray T3e and we use many "packages", NumPy, PIL, PyOpenGL. In addition I want to build only one Python environment for each architecture/OS and place it on some disk mounted by all these computers. Although scripts coming with some of the packages help greatly (thanks David) there are still a number of things that I have problems with. Here are some of the problems I have been strugling with: 1 - I started to use Makefile.pre.in to generate make file for additional packages. Currently when any user types "python" he gets 1.5 (current public version). So I decided to build Python1.5.2b1 and install it somewhere not in the users path. The problem is that when I use Makefile.pre.in it uses "python" and gets version 1.5. The only work around I found was to edit the copy of Makefile.pre.in and make it point to the 1.5.2b1 binary. 2 - When building Python for several architectures, the installation script allows to separate platform-dependant and platform-independant files. I'd like the same for packages, i.e. when a package has both types of files I'd like to be able to have a package with that name in the platform-independant AND in the platform-dependant tree. This DOES NOT work currently because import stops searching after it find a package matching that name. 3 - Numeric Python should really be part of the standard distribution .. (please :). Actually I would even argue that it should be part of the Kernel such that developers can rely on the fact that this data type, CRUCIAL for any scientific application and key to Python's success in that field is available. This would also allow for instance to have this datatype included in the Corba mappings for Python. 4 - How difficult would it be to provide some more flexibility in the directory sctructure created by "make install": I'd like to have something like: PYTHONROOT = /mgl/tools/python/$VERSION $PYTHONROOT/include $PYTHONROOT/lib $PYTHONROOT/man $PYTHONROOT/packages # arch independent packages $PYTHONROOT/$ARCH_OS/bin $PYTHONROOT/$ARCH_OS/include $PYTHONROOT/$ARCH_OS/lib $PYTHONROOT/$ARCH_OS/packages # arch dependent packages I do not need the Python version "python1.5" below that. In that way I can install any new version in parallele with an existing one. Any advice/help/modification is very welcome :) I am also going to post some messages that are more specific to particular packages ... so bear with me ! Thanks a lot -Michel -- ------------------------------------------------------------------ Sanner Michel Ph.D. The Scripps Research Institute 10550 North Torrey Pines Road La Jolla, CA 92037 Tel. (619) 784-2341 Fax. (619) 784-2860 e-mail sanner(a)scripps.edu Home Page http://www.scripps.edu/pub/olson-web/people/sanner

2 1

Using getopt for the proposed option processing
by Fred L. Drake Jan. 18, 1999

Jan. 18, 1999

The option processing Greg proposed in his design doc matches that of CVS: there are a bunch of global options, and then each command has its own options. The big difference is that cvs commands take arguments as well (the list of files to operate on), whereas Greg's proposal is to allow multiple commands, each of which has private options. I don't see anywhere in Greg's proposal a requirement that commands take arguments as well as options, so there's no reason not to use the getopt module. The only aspect of it is remotely interesting is switching in different sets of options for each different command class, and that's pretty trivial. Here's a little demo code. It accepts anything as a command, and just keeps the same options for each, but the comments should make it more than clear what needs to be done where to get the expected behaviour. ------------------------------------------------------------------------ """Example of the command line syntax used by distutils, using getopt.""" import getopt import sys LONGOPTS = ["help", "verbose"] SHORTOPTS = "hv" def main(): args = ["<global>"] + sys.argv[1:] while args: cmd = args[0] # switch in different SHORTOPTS and LONGOPTS in this call to # support different options for different commands: cmdinfo = sys.modules[__name__] # cmdinfo = load_command(cmd) opts, args = getopt.getopt(args[1:], cmdinfo.SHORTOPTS, cmdinfo.LONGOPTS) # do something with the command here: print cmd, opts # create_command(opts).run() if __name__ == "__main__": main() ------------------------------------------------------------------------ -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191

1 0