Mailman 3 - Distutils-SIG

Installer
by Gordon McMillan 22 Feb '99

22 Feb '99

Hi all, I've taken Greg Stein's "small" distribution and (with his advice(*) and help) created a mechanism for building compressed single-file importable python libraries. This is cross-platform, depending only on zlib (and Greg's imputil.py). There are a number of ways you can create these archives - grabbing directories, packages or computing the dependencies of a particular script. Then, for Win32 users, I've taken it a few steps further. I've mixed this with Christian Tismer's sqfreeze (based on /F's squeeze) and freeze and created a compiler-less Python installer. You can also use it much like freeze, except you don't need a compiler. The major difference is that, once installed, it won't be a completely standalone executable. The installer will unpack the dependencies into the exe's directory. There will be no dependencies outside that directory(**). You can pack in extension modules, dlls and anything else you like. The python lib support will be contained in a single archive file. So the equivalent of a frozen pure-python script would be one self-installing exe that expands to 5 files - the "squeezed" main script, python15.dll, zlib.dll, zlib.pyd and your private support lib (something.pyz). Check it out - http://www.mcmillan-inc.com/install.html Oh yes, while the code has my copyright, it's released under a do-what-you-want license. Mostly, I glued together what others had done (freeze, sqfreeze, Greg's stuff...). - Gordon (*) Not always followed. Don't blame Greg. (**) Note that if you want to install something that makes use of Mark's COM extensions (particularly, Python COM servers), you can't get away with this.

3 3

The $0.02 of a packager ;-)
by Oliver Andrich 10 Feb '99

10 Feb '99

Hi, after being invited by Greg Ward to this list, I tried to read all documents available concerning the topic of the list and I also tried to get uptodate with the mailing list by reading the archives. Now I like to give my $0.02 as someone who tries to keep a quite big distribution uptodate which also includes a lot of third party modules. This mail is rather long, cause I like to comment all all things in one mail, cause I think that quite a lot relates to each other. And please keep in mind that I am speaking as an enduser in one or another way. That means I will have to use this as a developer and will have to use it as a packager of third party developer. So I won't comment on any implementation issue where it is not absolutely critical. And please keep also in mind that may be I am talking about things you have already talked about, but I am quite new to this list and I only read the archives of this and the last month. ;-) - Tasks and division of labour I think that the packager and installer role are a little bit mixed up. In my eyes the packagers job is to build and test the package on a specific platform and also build the binary distribution. This is also what you wrote on the webpage. But the installers role should only be to install the prebuilt package, cause his normal job is to provide uptodate software that the users of the system he manages use. And he has enough to do with that. - The proposed Interface Basically the that I can say that I like the idea that I can write in my RPM spec file setup.py build setup.py test setup.py install and afterwards I have installed the package somewhere on my machine. And I am absolutely sure that it works as indented. I think that this is the way it works for most Perl modules. But I have problems with bdist option of setup.py cause I think that this is hard to implement. If I got this right I as a RPM and debian package maintainer should be able to say setup.py bdist rpm setup.py bdist debian And afterwards I have a debian linux and rpm package of the Python package. Nice in theory but this would require that setup.py or the distutils packages how to create these packages, that means we have to implement a meta packaging system on top of existing packaging systems which are powerful themselves. So what would it look like when I call these commands above? Would the distutils stuff create a spec file (input file to create a rpm) and then call rpm -ba <specfile>? And inside the rpm build process setup.py is called again to compile and install the packages content? Finally rpm creates the two normal output files, which are the actual binary package and the other is the source rpm from which you can recompile the binary package on your machine. This is the same for debian linux, slackaware linux, rpm based linux versions, Solaris packages and BeOS software packages. The last is only a vague guess cause I only looked into the Be system very shortly. - What I would suggest what setup.py should do The options that I have no problem with are build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution test - run test suite install - install on local machine What should make_blib do ? But I require is that I can tell the build_ext which compiler switches to use, cause may be I need on my system different switches then the original developer can use. I also like to provide the install option with an argument to tell where the files should be installed, cause I can tell rpm for example that it should compile the extension package as if it would be installed in /usr/lib/python1.5 but could it in the install stage to install it in /tmp/py-root/usr/lib/python1.5. So I can build and install the package without overwriting an existing installation of a older version and I also have a clean way to determine what files actually got installed. install should also be split up into install and install_doc and installdoc should also be able to take an argument where I tell it where to install the files to. I would remove the bdist option cause it would introduce a lot of work, cause you not only have to tackle various systems but also various packaging systems. I would add an option files instead which returns a list of files this packages consists of. And consequently an option doc_files is also required cause I like to stick to the way rpm manages doc files, I simply tell it what files are doc files and it installs them the right way. Another that would be fine if I could extract the package information with setup.py. Something like setup description returns the full description and so on. And I would also add an option system to the command line options, cause I like to tell the setup.py script an option from which it can determine on which system it is running. Why this is required will follow. - ARCH dependent sections should be added What is not clear in my eyes, may be I have missed something, but how do you deal with different architectures? What I would suggest here is we should use a dictionary instead of plain definitions of cc, ccshared, cflags and ldflags. Such a dictionary may look like that compilation_flags = { "Linux" : { "cc": "gcc", "cflags": "-O3", ...}, "Linux2.2": { "cc": "egcs", ....}, "Solaris": { "cc": "cc", ....} } And now I would call setup.py like that setup.py -system Linux build or whatever convention you want to use for command line arguments. - Subpackages are also required Well, this is something that I like very much and what I really got accustomed to. They you build PIL and also a Tkinter version that supports PIL, then like to create both packages and also state that PIL-Tkinter requires PIL. Conclusion (or whatever you want to call it) I as a packager don't require the distutils stuff to be some kind of meta packaging system that generates from some kind of meta information the actual package creation file from which it is called again. And I don't believe that have to develop a complete new packaging system, cause for quite a lot systems such systems exist. And I also think that we introduce such a system the acceptance wouldn't be very high. The people want to maintain the software basis with their natural tools. A RedHat Linux user would like to use rpm, a Solaris user would like to use pkg and a WindowsUser would like to use INstallShield (or whatever is the standard). The target of distutils should be to develop a package which can be configured to compile and install the extension package. The developed software should be usable by the packager to extract all required information to create his native package and the installer, should use the prebuilt packages at best or should be able to install the package by calling setup install. I hope that I described as good as possible what I require as packager and I think that is not a business of distutils but of the native packaging system. Any comments are welcome and I am willing to discuss this, as I am absolutely aware that we need a standard way of installing python extensions. Best regards, Oliver -- Oliver Andrich, RZ-Online, Schlossstrasse Str. 42, D-56068 Koblenz Telefon: 0261-3921027 / Fax: 0261-3921033 / Web: http://rhein-zeitung.de Private Homepage: http://andrich.net/

4 8

Re: new import mechanism and a "small" distribution
by Greg Stein 01 Feb '99

01 Feb '99

Just van Rossum wrote: > > At 8:00 AM -0800 1/31/99, Greg Stein wrote: > >... > >Okay... enough background and rambling. If you're interested, go look at the > >four messages in the thread titled "Freeze and new import architecture" in > >the distutils-sig archive at: > >http://www.python.org/pipermail/distutils-sig/1998-December/thread.html > > I'm not on that sig so I missed your post originally. I agree with most you > say here: > http://www.python.org/pipermail/distutils-sig/1998-December/000077.html > Especially that entries in sys.path should be loader instances (or > directory paths). > > Some questions: > - what is the interface of a loader By "loader", I will presume that you mean an instance of an Importer subclass that is defining get_code(). Here is the method as defined by imputil.Importer: def get_code(self, parent, modname, fqname): """Find and retrieve the code for the given module. parent specifies a parent module to define a context for importing. It may be None, indicating no particular context for the search. modname specifies a single module (not dotted) within the parent. fqname specifies the fully-qualified module name. This is a (potentially) dotted name from the "root" of the module namespace down to the modname. If there is no parent, then modname==fqname. This method should return None, a 2-tuple, or a 3-tuple. * If the module was not found, then None should be returned. * The first item of the 2- or 3-tuple should be the integer 0 or 1, specifying whether the module that was found is a package or not. * The second item is the code object for the module (it will be executed within the new module's namespace). * If present, the third item is a dictionary of name/value pairs that will be inserted into new module before the code object is executed. This provided in case the module's code expects certain values (such as where the module was found). """ raise RuntimeError, "get_code not implemented" That method is the sole interface used by Importer subclasses. To define a custom import mechanism, you would just derive from imputil.Importer and override that one method. I'm not sure if that answers your question, however. Please let me know if something is unclear so that I can correct the docstring. Oh, geez. And I totally spaced on one feature of imputil.py. There is a way to define an import mechanism for very simple uses. I created a subclass named FuncImporter that delegates the get_code() method to a user-supplied function. This allows a user to do something like this: import imputil def get_code(parent, modname, fqname): ...do something... imputil.install_with(get_code) The install_with() utility simply creates a FuncImporter with the specified function and then installs the importer. No need to mess with subclasses. > - how are packages identified If get_code() determines that the requested module is a package, then it should return the integer 1 along with the code object for that package's module. In the standard package system, the code object is loaded from __init__.py. An example: let's say that get_code() is called with (None, "mypkg", "mypkg") for its arguments. get_code() finds "mypkg" in whatever path it is configured for, and then determines that it represents a directory. It looks in the directory for __init__.py or .pyc. If it finds it, then mypkg is a real package. It loads code from __init__.py and returns (1, code). The Importer superclass will create a new module for "mypkg" and execute the code within it, and then label it as a package (for future reference). Internally, packages are labelled with a module-level name: __ispkg__. That is set to 0 or 1 accordingly. The Importer that actually imports a module places itself into the module with "__importer__ = self". This latter variable is to allow Importers to only work on modules that they imported themselves, and helps with identifying the context of an importer (whether an import is being performed by a module within a package, or not). > - can it be in Python 1.6 or sooner? I imagine that it could be in Python 1.6, but I doubt that it would go into Python 1.5.2, as it has not had enough exposure yet. Guido's call on all counts, though :-) > PS: I was just wondering, why doesn't reload() use the standard import hook? The standard import hook returns a *new* module object. reload() must repopulate the module object. I'm not sure that I answered your questions precisely, as they were rather non-specific. If I haven't answered well enough, then please ask again and I'll try again :-). Cheers, -g -- Greg Stein, http://www.lyra.org/

2 5

Re: new import mechanism and a "small" distribution
by Greg Stein 01 Feb '99

01 Feb '99

Gordon McMillan wrote: > > [Greg puts code where his mouth is] > > > A while back, Mark Hammond brought up a "new import architecture" > > proposal after some discussion with Jack Jansen and Guido. I > > responded in my usual diplomatic style and said "feh. the two-step > > import style is bunk." > > [snip...] > > This is outrageously cool! > > It appears that you're aiming for a rewrite of import.c (and > friends) and giving new meaing to "sys.path". (I love it - Java says > "it can be a directory or a zip file" and Greg says "why stop there? > It can be anything at all" - hee hee). Yup :-) Last month, I even argued that the import mechanism could simply be shifted entirely to Python, too, since the overhead of interpreted Python code is minimal next to the parsing, execution, and/or I/O involved with importing. hehe... > What is enabling this magic in this prototype? We pick up your > site.py automatically, but is your python.exe doing something in > between Py_Initialize and Py_Main? Or is this all based on making a > chain out of __builtin__.__import__? If python.exe doesn't find its registry settings, then it assumes a default sys.path (which includes the current directory). Using that, it loads exceptions.py and then site.py. Once site.py loads, then I install the custom import hook so that all future imports will be yanked from py15.pyl. The builtin modules aren't in there, of course, so those fall down the chain to the builtin importer. Cheers, -g -- Greg Stein, http://www.lyra.org/

1 0

FW: new import mechanism and a "small" distribution
by Greg Stein 31 Jan '99

31 Jan '99

crap... missed the "s" in distutils. what fun now -- divergent threads (here and python-list). sigh. -----Original Message----- From: Greg Stein [mailto:gstein@lyra.org] Sent: Sunday, January 31, 1999 8:00 AM To: distutil-sig(a)python.org Cc: python-list(a)python.org Subject: new import mechanism and a "small" distribution A while back, Mark Hammond brought up a "new import architecture" proposal after some discussion with Jack Jansen and Guido. I responded in my usual diplomatic style and said "feh. the two-step import style is bunk." Then, I expounded on a few of the ideas that I had come up with after working with a lot of custom import stuff (for COM support, for sticking .pyc files into resources or structured-storage files, or for creating distribs with very few files). In essence, I maintained that importers should really be a single step find-and-load, and that Python would be best served by having a list of object instances, each performing this find-and-load step. Imagine, if you will, replacing sys.path with a list of "directory importers" where each importer checks a directory for the requested file. Next step in our hypothetical case is to add a new importer into sys.path that checks the Windows Registry for paths and looks there, or an importer that looks inside the resource fork of a Mac file. Heck, or use the typical example: have an importer that grabs a code object from over the web. Okay... enough background and rambling. If you're interested, go look at the four messages in the thread titled "Freeze and new import architecture" in the distutils-sig archive at: http://www.python.org/pipermail/distutils-sig/1998-December/thread.html What I'm posting here is two things: 1) imputil.py : this is a module that defines an import utility class (imputil.Importer). There are a few subclasses in here for different types of importers. More detail below. 2) a "small distribution" : the demo is for Win32, but it the basic mechanism is quite portable. In this small distro, you can place onto somebody's computer a mere 5 files and NO registry keys. Python starts up just fine and you can import all modules from the standard library. Basically, imputil.Importer's logic was derived from knee.py, which is an emulation of Python's algorithm. Importer defines one method that subclasses should override: get_code(). The doc strings and examples in the module cover that method relatively well. The Importer class will install an import hook and "chain" to the previous hook. This allows the use of multiple Importers, each one attempting to perform the import, and finally falling back to the builtin import mechanism. (note that there is no way to deinstall an importer at the moment, as it can't go "up" the chain to detach itself; the "right" way to do this is to define an Importer that contains a manageable list of Importers, and removing the chaining in favor of the manager iterating over the list) The small distribution replaces site.py and uses an archive of the standard library to fetch code objects from. This library is named "py15.pyl" in the demo, and is generated by the batch file in support/regen.bat (with the help of support/easygen.py). The resulting library is portable, as it is merely a giant glom of .pyc files and a marshalled table of contents (mapping module names to file positions). Look in site.py for the SimpleArchive class to get a feel for how to write a simple subclass of Importer. As another demo, imputil.py contains a class named DirectoryImporter. This subclass attempts to emulate Python's behavior for importing from a directory. It is instantiated with a directory to look in, for modules and packages. If a module is found, it will perform the usual .pyc caching if necessary (or use the cached .pyc). There is a demo function named imputil._test_dir() that will read sys.path and construct a chain of DirectoryImporter instances. You can then import away and Python will actually use the Importers rather than sys.path iteration. Lastly, imputil.py contains two (untested) subclasses for importing from a package contained in archive-type files (such as .gz, .zip, .tar, or a Mac file of resources). For example, you could distribute "package.gz" along with a gzip Importer. When somebody said "import package.module", it would look inside package.gz for the "module" module. I think that's it. This is the first release of this stuff, so there are bound to be improvements or bugfixes or just simple comments :-). Please feel free to speak up. thx, -g -- Greg Stein, http://www.lyra.org/

1 0

Responses to proposal
by Greg Ward 20 Jan '99

20 Jan '99

[Fred says the need for application-specific command-line options is not clear] My impetus for this was the need for specific module distributions to be told where to go looking for external C libraries. For instance, the setup.py for zlib would need to know where to find zlib.h and libz.a -- that is, how to generate a -I option for compiling the extension, and a -L option for linking it (modulo the usual Unixisms). Likewise, tkinter needs to be told where to find the Tcl and Tk headers and libraries. Granted, there's already a standard way of doing this: the Setup file. Perhaps we should preserve the Setup file, or perhaps we should mutate it into a Distutils-specific 'options' file as per Martijn's suggestion for replacing my command-line options. However, let's not limit ourselves to "where's this header file, where's that library?" questions: lots of modules could have configuration options that the builder might want to supply at build time. And keep in mind that ultimately we want to be able to write setup.py for a whole Python *applications*, which almost certainly will have configuration options: where to put temporary files? permanent data files? installation directory? etc. [Martijn Faassen berates me for forgetting about abstract compiler classes] > You don't expose Unixish C compilation variables to the user at all, > perhaps? At least in principle. In practice you'd still allow > modification, but it shouldn't be *necessary* to look at these or even > know about them, even to write a C extension. (unless you're on some new > platform) Duh, you're right. I had *Programming Perl* sitting on my lap and open to the section on MakeMaker as I wrote that part of the design proposal. (Mainly I wanted to be sure I didn't forget anything obvious.) Small wonder a few Unixisms creeped in! Anyways, this raises the broader question: how should the class(es) that abstract compilation and linking behave? I'll try to come up with a few ideas and post them separately. (Arg, really must look at Interscript's compilation stuff... presumably John has already thought through this stuff!) Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

3 3

Revised design proposal
by Greg Ward 20 Jan '99

20 Jan '99

Hi all -- as threatened, here's a revised design proposal. Main changes: I've commited myself to and firmed up the language regarding 1) multiple commands on the setup.py command line, and 2) OO command modules. There're still some unresolved issues, and I'll be posting on them separately soon. BTW if you can't be bothered to read the whole thing, at least skip down to and read "Part 3", a sort of executive summary (for executives who want to know every step the code will take, but not know the background -- that's what the other 250 lines of text are for!). Enjoy. --snip----------------------------------------------------------------- $Id: design-proposal.txt,v 1.2 1999/01/15 03:52:30 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd1 [cmd_options] [cmd2 [cmd_options]] ... (Distinguishing the end of the option list for cmd1 from cmd2 might be a bit tricky if command options can take arguments, which they certainly should do. If we want getopt-style syntax, it looks like we'll have to reinvent the getopt module. :-( And if we opt for a different syntax, we'll just have to write something to parse that! It's worth the effort in order to be able to say "./setup.py build test install", though.) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for this module distribution) may define a set of distribution-specific command-line options, which may be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business is to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine Please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. 'distutils.core' also defines the 'Command' class which comes in handy for implementin Distutils commands. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one class, also named for the command -- e.g. 'distutils.build.Build'. These command classes will inherit from the 'Command' class, which (at the very least) will provide a means of dealing with command-specific options. (Probably 'Command' will provide a constructor that takes a 'Distribution' class and an optional list of arguments for this command, and parse the argument list by inspecting getopt-style option specifiers in the 'Command'-derived instance.) Each command class must provide a method ('go'? 'run'? 'doit'?) that uses the information in the 'Distribution' instance and the command options to "do its thing". Well-written command classes will parcel this task out into several well-defined (and documented) methods, so that the client setup.py may inherit from and override specific behaviours of a Distutils command class. This also means that the 'Distribution' class must have a way to communicate overridden command classes to the main dispatcher. Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour, not just data, and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward <gward(a)cnri.reston.va.us>" description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure, eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST by default] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; figures out the commands and options for each command, saving them all up for later processing * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution, probably with a bunch of other named arguments specifying various attributes for the Distribution instance) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup determines the command class (usually just named for the command, eg. 'distutils.build.Build, but possibly a class supplied by the client as one of the attributes of the Distribution instance) and instantiates it * the command class constructor takes as arguments the Distribution instance and any command-line arguments specific to this command on the setup.py command line * the command class constructor parses its options to set/override some instance attributes * distutils.core.setup calls the "make it so" method provided by the command class * that method does whatever the command is supposed to do: build modules, process documentation, install files, etc. * distutils.core.setup determines the next command class (if multiple commands were given), and proceeds as before Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed? (Undoubtedly.) How does the client specify them, and how do the Distutils run them? -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

6 7

Re: [Distutils] packages and Maintaining Python in general
by Michel Sanner 19 Jan '99

19 Jan '99

On Jan 19, 8:57am, Greg Ward wrote: > Subject: Re: [Distutils] packages and Maintaining Python in general > Quoth Sanner Michel, on 18 January 1999: > > > 1 - I started to use Makefile.pre.in to generate make file for > > additional packages. Currently when any user types "python" he gets > > 1.5 (current public version). So I decided to build Python1.5.2b1 and > > install it somewhere not in the users path. The problem is that when I > > use Makefile.pre.in it uses "python" and gets version 1.5. The only > > work around I found was to edit the copy of Makefile.pre.in and make > > it point to the 1.5.2b1 binary. > > This will definitely be addressed by the Distutils suite: > Makefile.pre.in will be a thing of the past, replaced by a (usually) > tiny Python script. Normally this will just be run as "./setup.py", but > of course you can always say "python1.5.2b1 setup.py" and the build > process will follow the defaults associated with *that* particular > binary (eg. compiler flags, installation directories, etc.). > Cool I see you guys already have a plan :). I guess I will have to browse the archive of the mailing list to come up to speed ! Anyways is the plan to have something available under 1.6 ? or maybe 1.5.2 (non beta ?) > > 2 - When building Python for several architectures, the installation > > script allows to separate platform-dependant and platform-independant > > files. I'd like the same for packages, i.e. when a package has both > > types of files I'd like to be able to have a package with that name in > > the platform-independant AND in the platform-dependant tree. This DOES > > NOT work currently because import stops searching after it find a > > package matching that name. > > I completely agree. I think this will require some cooperation on the > part of Python's configure/build process, and probably the startup code > that initializes sys.path. So it's only likely to happen when/if a) > Distutils comes to pass as planned, b) Python 1.6 goes out. But it's > definitely an important thing to keep in mind. > Yeap, the main problem currently is that import stops searching after it find a package with the right name. So you could NOT have an OpenGL package (for instance) under $prefix and another OpenGL package under $exec_prefix. Is someone talking to Guido about this. It seems to me that this separation between acrh-dependant and arch-independant files should be allowed for packages. > > 3 - Numeric Python should really be part of the standard distribution > > .. (please :). Actually I would even argue that it should be part of > > the Kernel such that developers can rely on the fact that this data > > type, CRUCIAL for any scientific application and key to Python's > > success in that field is available. This would also allow for > > instance to have this datatype included in the Corba mappings for > > Python. > > I disagree. The whole point of the Distutils SIG is to make it trivial > to add third-party modules to an existing Python installation; if NumPy > being crucial for scientists justifies adding it to the core, then why > not add PIL for image-processors, and so forth? Where do you draw the > line? > OK ! I think my level of frustration made me loose my objectivity :) .. although there is something special about Numeric: it implements a basic datatype: ARRAYS that are as basic as lists or tuples. This I think makes a difference with PIL for instance that might use Numeric Arrays to store images for instance. Moreover having this datatype be a part of Python itself would allow to define CORBA mappings (which again I am personally interested in :)... Just pushing my luck here :) -Michel

1 0

Weekly/biweekly development report?
by Andrew M. Kuchling 19 Jan '99

19 Jan '99

In the Distutils-SIG, Martijn Faassen wrote: >Perhaps a special 'distutils SIG' status report, posted once every while >and updating everybody on developments would be useful to attract >interest. And what about something in the Python Journal? A recent trend for development projects has been to issue a summary of activity at periodic intervals. It may have started with the development summary at Linux Weekly News, but it's been picked up by other groups such as the Debian GNU/Linux distribution and Mozilla.org. There are also development diaries from various people, most notably Alan Cox: LWN Development page for this week: http://www.lwn.net/1999/0114/devel.phtml LWN Kernel development page for this week: http://www.lwn.net/1999/0114/kernel.phtml Debian Weekly News: http://www.debian.org/~joeyh/debiannews.html Mozilla.org status update: http://www.mozilla.org/status/ Linux kernel update: http://lynx.neu.edu/z/zbrown/kt.html Alan Cox's diary: http://www.linux.org.uk/diary/ Should we attempt to have a monthly summary of Python development activity? This is slightly different from the Python-URL postings, which aren't focused exclusively on development, and don't have much editorial commentary. I like LWN's habit of occasionally commenting on the news, saying "This flamewar was rather pointless" or "This recent argument was upsetting because...". The Python version of this might look like: Distutils SIG: Greg Ward posted a proposed interface (URL to a copy). Once some people have commented on it, implementation work should begin. Zope: DC released new database adapters X,Y,Z. Skip Montanaro announced thing W. Python-list: Endless discussion of assignment in while statements; GvR said he's not interested. ... ...etc... Pulling together the information requires several volunteers, because no one can possibly be reading every single relevant mailing list and newsgroup. How could that be made workable? Have a central editor? A CGI script which lets anyone add suggested items, which an editor draws upon for each issue and then gets cleaned out when the issue is released? Include it as part of the Python Journal? Suggestions? -- A.M. Kuchling http://starship.skyport.net/crew/amk/ He told me he was having the time of his life and then fell to the floor unconscious. I didn't take this too personally, although he was not the first person to collapse while talking to me; to tell the truth it has happened quite often. -- Tom Baker, in his autobiography

1 0

setup.py command-line syntax
by Greg Ward 19 Jan '99

19 Jan '99

OK, time for the first of those unresolved issues. This one isn't actually mentioned as such in the "Unresolved issues" section of the design proposal, because if you carefully read and think through what's already there you'll start to see the holes in my proposal. In particular, we need to figure out what the syntax for the 'setup.py' command line is going to be. I have a couple of ideas, none of them wholly satisfactory. In vague terms, I'm currently thinking that the syntax will be like this: ./setup.py [global_options] cmd1 [cmd1_options] [cmd2 [cmd2_options]] ... The specific problem I'm having is: what will options look like, in particular options with arguments, and how do we know when the options for cmd1 are over and we're staring at cmd2? Option 1: getopt-ish Options are either of the '-x' form (single dash, single letter, may be jammed together, arguments may follow with or without intervening space) or of '--long-opt' form (two dashes, dash-or-underscore-separated-words, arguments will follow either '=' or whitespace). Pros: * familar interface: just like the getopt module, except that there are these "command" things intermingled with options, and there are multiple sources for option descriptors (the standard 'Distribution' class provided by distutils.setup.core, the individual command classes provided by the command modules, and possibly setup.py). Cons: * those complications mean that we'd have to reimplement the getopt module for distutils, with hooks to the various sources of option descriptors specific to distutils * due to many ways of specifying option arguments ("-a foo", "-afoo", "--aardvark=foo", "--aardvark foo"), can't tell where option lists ends and next command begins without fully processing the option descriptors and argument lists. This could induce temporal whiplash. Option 2: restricted getopt-ish Options are only of the '--long-opt' form, and option arguments may only be specified as '--option=value' -- no spaces between option and value (ie. they're the same command line word). Pros: * much easier to implement than a getopt knock-off * don't have to fully process options to know where cmd1's options stop and cmd2 starts: just look for first ! /^--/ after cmd1 * easy to make a standard way to negate boolean (no argument) options: just accept "--no-flag" in addition to "--flag" Cons: * perversion of a familiar interface; people will think that because they can say "--option=value" they can equivalently say "--option value" and will get confused when distutils tells them "no such command 'value'" (err, except it will *also* tell them "option '--option' requires an argument"... maybe not so bad then) * needless typing; double-dashes on long options are really only necessary to distinguish them from short options, and if we don't support short options why bother with the double dashes? Option 3: who needs dashes anyways? Options are always of the "option=value" form -- no dashes, must always have a value (even if that value is only 'yes' or 'no'). Pros: * simple to implement, simple to explain * no confusion with existing interfaces (except... see "Cons") * dead easy to spot end of 'option=value' list after each command Cons: * looks like we're running a Makefile! * no nice way to do boolean (argument-less) options Option 4: who needs command options anyways? Change the syntax to ./setup.py [options] cmd1 [cmd2 ...] and make all options the responsibility of distutils.core (or ensure that the client seutp.py and the various command classes get their option descriptors to distutils.core). Pros: * assuming we can get all the option descriptors to distutils.core in time, can just use standard 'getopt' module; the leftover arguments are of course the commands Cons: * unclear from the command-line whose responsibility each option is Anyone have any other schemes? Comments on these? Preferences? Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

packages and Maintaining Python in general
by Sanner Michel 19 Jan '99

19 Jan '99

Following Barry's advice I post my message on this list too. Sorry for those of you you get it twice, but I wasn't aware of that SIG ! now I am a subscriber :) Hello, I have spend my week end moving opn from Python1.5 to Python1.5.2b1 and have come to the conclusion that there OUGHT to be better way !!! It is a lot of work because I try to maintain Python for - Sgi under Irix5.3 Irix6 and Irix646 - Sun - Dec alpha under OSF1.2 and OSF 1.4 - HP - Linux - Cray T3e and we use many "packages", NumPy, PIL, PyOpenGL. In addition I want to build only one Python environment for each architecture/OS and place it on some disk mounted by all these computers. Although scripts coming with some of the packages help greatly (thanks David) there are still a number of things that I have problems with. Here are some of the problems I have been strugling with: 1 - I started to use Makefile.pre.in to generate make file for additional packages. Currently when any user types "python" he gets 1.5 (current public version). So I decided to build Python1.5.2b1 and install it somewhere not in the users path. The problem is that when I use Makefile.pre.in it uses "python" and gets version 1.5. The only work around I found was to edit the copy of Makefile.pre.in and make it point to the 1.5.2b1 binary. 2 - When building Python for several architectures, the installation script allows to separate platform-dependant and platform-independant files. I'd like the same for packages, i.e. when a package has both types of files I'd like to be able to have a package with that name in the platform-independant AND in the platform-dependant tree. This DOES NOT work currently because import stops searching after it find a package matching that name. 3 - Numeric Python should really be part of the standard distribution .. (please :). Actually I would even argue that it should be part of the Kernel such that developers can rely on the fact that this data type, CRUCIAL for any scientific application and key to Python's success in that field is available. This would also allow for instance to have this datatype included in the Corba mappings for Python. 4 - How difficult would it be to provide some more flexibility in the directory sctructure created by "make install": I'd like to have something like: PYTHONROOT = /mgl/tools/python/$VERSION $PYTHONROOT/include $PYTHONROOT/lib $PYTHONROOT/man $PYTHONROOT/packages # arch independent packages $PYTHONROOT/$ARCH_OS/bin $PYTHONROOT/$ARCH_OS/include $PYTHONROOT/$ARCH_OS/lib $PYTHONROOT/$ARCH_OS/packages # arch dependent packages I do not need the Python version "python1.5" below that. In that way I can install any new version in parallele with an existing one. Any advice/help/modification is very welcome :) I am also going to post some messages that are more specific to particular packages ... so bear with me ! Thanks a lot -Michel -- ------------------------------------------------------------------ Sanner Michel Ph.D. The Scripps Research Institute 10550 North Torrey Pines Road La Jolla, CA 92037 Tel. (619) 784-2341 Fax. (619) 784-2860 e-mail sanner(a)scripps.edu Home Page http://www.scripps.edu/pub/olson-web/people/sanner

2 1

Using getopt for the proposed option processing
by Fred L. Drake 18 Jan '99

18 Jan '99

The option processing Greg proposed in his design doc matches that of CVS: there are a bunch of global options, and then each command has its own options. The big difference is that cvs commands take arguments as well (the list of files to operate on), whereas Greg's proposal is to allow multiple commands, each of which has private options. I don't see anywhere in Greg's proposal a requirement that commands take arguments as well as options, so there's no reason not to use the getopt module. The only aspect of it is remotely interesting is switching in different sets of options for each different command class, and that's pretty trivial. Here's a little demo code. It accepts anything as a command, and just keeps the same options for each, but the comments should make it more than clear what needs to be done where to get the expected behaviour. ------------------------------------------------------------------------ """Example of the command line syntax used by distutils, using getopt.""" import getopt import sys LONGOPTS = ["help", "verbose"] SHORTOPTS = "hv" def main(): args = ["<global>"] + sys.argv[1:] while args: cmd = args[0] # switch in different SHORTOPTS and LONGOPTS in this call to # support different options for different commands: cmdinfo = sys.modules[__name__] # cmdinfo = load_command(cmd) opts, args = getopt.getopt(args[1:], cmdinfo.SHORTOPTS, cmdinfo.LONGOPTS) # do something with the command here: print cmd, opts # create_command(opts).run() if __name__ == "__main__": main() ------------------------------------------------------------------------ -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191

1 0

Comments on proposed design?
by Greg Ward 14 Jan '99

14 Jan '99

Hey folks -- that Distutils design proposal was meant to provoke some discussion! Don't tell me everybody thinks my proposal was so sublimely perfect that I should just go home and knock off version 0.01 over the weekend. There are known problems, things I need help with, etc. Please take a look at it and post your reactions to the list. However, be warned that I am hoping to revise the proposal and post a revised version tonight/tomorrow. I would really like to have some feedback before I start revising. Thanks -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Design proposal
by Greg Ward 11 Jan '99

11 Jan '99

Well, things have been a little quiet on this list. Time to shake it up a bit, methinks. So I've written a design proposal -- here it is. (NB. #1: to avoid sounding overly wafflish, I've written this in a fairly imperious way -- "X will be done this way, Y is like that". Don't take it *too* seriously; this is open for discussion, up for grabs, welcomes your comments and criticisms, etc. Of course, I wouldn't have designed it this way if I didn't think it was good idea, so expect a reaction if you tell me I'm full of baloney... ;-) (NB. #2: this file is in the Distutils CVS archive as 'text/design-proposal.txt'. Once it is better firmed up through the discussion that I'm sure is forthcoming, I'll HTMLify it, put it in the SIG's web space, and start coding!) $Id: design-proposal.txt,v 1.1 1999/01/11 04:06:01 gward Exp $ RECAP: User Interface ===================== Recall from the proposed user interface (posted at http://www.python.org/sigs/distutils-sig/interface.html) that the Distutils will operate via a (usually) trivial Python script, conventionally called setup.py. setup.py has the following syntax: setup.py [global_options] cmd [cmd_options] (XXX no provision for multiple commands here! shouldn't be hard, though, as long as we're firm about what a command is and what an option is) The Distutils will define a standard set of global command-line options; each Distutils command will define a set of command options. The module developer (the person who wrote the setup.py for his module distribution) may define a set of distribution-specific command-line options, which will be mixed in with the global options (and extracted from them before they can cause any harm). The first order of business to decide on the set of standard Distutils commands. Again, from the proposed interface, here is my initial list: make_blib - create mockup installation tree ("build library") build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution bdist - create built distribution for current platform test - run test suite install - install on local machine please see the above URL for details on these. REAL STUFF: Proposed Design =========================== Part 1: from the Distutils' point of view ----------------------------------------- setup.py only has to import one module, 'distutils.core'. This module is responsible for parsing all command-line arguments to setup.py (even though the interpretation of options is distributed across the various Distutils commands, and possibly the client setup.py). It also takes care of receiving control from setup.py, and passing it as appropriate to Distutils commands. Most importantly, 'distutils.core' defines the 'Distribution' class, which is the heart and soul of the Distutils. The client (setup.py) exists mainly to provide attributes for a 'Distribution' instance, and all the Distutils commands operate on that instance. Speaking of Distutils commands: each one is implemented as a Python module, e.g. the 'build' command is implemented by the 'distutils.build' module. Each command module is required to define one function, also named for the command -- e.g. 'distutils.build.build'. This function takes a 'Distribution' instance as its only required argument, and then "does its thing" (eg. build extensions, install everything, etc.). All information needed to "do its thing" is contained in the 'Distribution' instance. (XXX this isn't very OO. As long as we can fully parameterize Distutils commands, that's fine: the client (or user) just provides attributes to go into a 'Distribution' instance, and the command modules use those attributes to know what to do. Inevitably, though, someone somewhere will have to override actual code, and this scheme will break down -- either that or we'll have to put a lot of command-specific stuff into the 'Distribution' class, which is bad [makes it harder to add new commands]. A possible alternate formulation: have each command module define a specially-named class, which is then used to "do its thing". If a client setup.py needs to override, say, installation behaviour, then it can create a subclass of distutils.install.Install. It would then need to tell the 'Distribution' class to use its derived class -- not an impossible task!) Part 2: from the client's point of view --------------------------------------- As I said above, the client (setup.py) only has to import 'distutils.core' -- everything else Distutils-ish is taken care of by this core module. However, the client needs a way to communicate its particular options into the Distutils core (and out to the command modules). I have two possible schemes for this: one short and convenient (but not too extensible), and the other a bit verbose and clunky (but more OO and extensible). There's no reason we can't have our cake and eat it too; the convenient interface could just be a wrapper for the full-blown interface for the many module distributions that don't need a lot of fancy customization. First, here's an example of the simple interface, used for a module distribution with a single "pure Python" module (mymod.py). ------------------------------------------------------------------------ from distutils.core import setup setup (name = "mymod", version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") ------------------------------------------------------------------------ Note that we don't explicitly list "mymod.py" anywhere: Distutils assumes that this is a one-horse distribution named after its sole module ('mymod'). Those who enjoy defining subclasses might prefer to phrase this differently: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mymod" version = "1.2", author = "Greg Ward <gward(a)cnri.reston.va.us>", description = "A very simple, one-module distribution") setup (distclass = MyDistribution) ------------------------------------------------------------------------ This is overkill for a small distribution: we're defining a new class solely to provide attribute values, when the 'distutils.core.setup' exists mainly to let do this anyways. Nevertheless, OO purists will like this -- and undoubtedly there will be times when the client *will* have to override behaviour -- not just data -- and the OO interface will be necessary. And more complex module distributions, with lots of attributes to customize, might be easier to read/maintain with things broken up like this. Consider a distribution with two pure Python modules ('mymod' and 'my_othermod') and a C extension ('myext'); the C extension must be linked with two ancillary C files and a C library. Oh yeah, this distribution requires Python 1.5 and any version of the 're' module: ------------------------------------------------------------------------ from distutils.core import Distribution, setup class MyDistribution (Distribution): name = "mydist", version = "1.3.4", author = "Greg Ward <gward(a)cnri.reston.va.us>" description = """This is an example module distribution. It provides no useful code, but is an interesting example of the Distutils in action.""" # Dependencies requires = { 'python': '1.5', # I like class-based exceptions 're': '', # and I love Perl-style regexps! ;-) } # (and yes, I *know* that "Python 1.5" # implies 're'...) # Actual files that need to be processed and installed in some form py_modules = ['mymod.py', 'my_othermod.py'], ext_modules = {'myext.c': {'other_c': ['extra1.c', 'extra2.c'], 'c_libraries': ['mylib']} } setup (distclass = MyDistribution) ------------------------------------------------------------------------ A couple of things to note: * I'm not afraid to use deeply nested data structures; if you're writing and distributing Python modules, this shouldn't be a problem! * every attribute has a particular type (string, list, dictionary, ...) * the attributes with complex types (especially dictionaries) will have a well-known and well-documented internal structure: eg. """ext_modules is a hash mapping names of C source files (each containing a Python extension module) to a nested hash of information about how to build that module. The allowed keys to this nested hash are: - other_c: other C files that must be compiled and linked with the main C file to create the module - c_libraries: C libraries that must be included in the link ... """ No doubt the 'ext_modules' nested hashes would have more options, and no doubt other Distribution attributes would have complex, documented structure. Finally, the list of all Distribution attributes must be well-known and well-documented! These seem to fall into a couple of broad categories. Here's an initial attempt at a list: Distribution meta-data name version author description Dependencies requires Files to be processed and installed py_modules ext_modules doc_files [eg. SGML source - or whatever std. we get for documentation] Build directories [all under "./blib" by default] build_lib - where to put platform-independent library files build_platlib - where to put platform-dependent library files build_exe - where to put executable programs (ie. scripts) build_html - where to put processed documentation (HTML) (etc... more documentation formats, at least) Installation directories [under sysconfig.LIBDEST] install_lib install_platlib install_exe install_html C compilation cc ccshared cflags ldflags ...well, that's a start. I still don't know how to make all those Unixish C compilation variables more cross-platform. Part 3: revisiting the Distutils' point of view ----------------------------------------------- To sum up, let's go through what happens when the user runs 'setup.py'. Whether setup.py is written in the simple (call-a-function) or general (define-a-subclass) form doesn't matter too much, so I won't split things up into two streams. * setup.py imports distutils.core * distutils.core startup code parses command-line arguments: processes global options that it knows about, and saves the rest for the client (setup.py) to deal with; saves the command, and saves the command-specific options for passing to the command module * setup.py calls distutils.core.setup (possibly with a 'distclass' argument specifying a subclass of Distribution) * distutils.core.setup instantiates Distribution (or the subclass supplied by the client), and uses its arguments (apart from 'distclass') to override attributes of this instance * distutils.core.setup loads the command module (eg. 'distutils.build') * distutils.core.setup calls the command module's interface function (eg. 'distutils.build.build'), passing it the Distribution instance and any command-specific options from the setup.py command-line * [alternate formulation: distutils.core.setup instantiates the command module's interface class (eg. 'distutils.build.Build', or an alternate supplied by the client using the as-yet-unmentioned 'command_class' attribute. The Distribution instance and all command-specific options are supplied to the command class so it can "do its thing"] (XXX again, no provision for multiple commands, although it shouldn't be too hard. And the OO approach to writing command modules needs to be better fleshed out.) Part 4: Unresolved issues ------------------------- * Where do we take care of platform dependencies? Somewhere, sometime, we'll need a class or function or attribute named 'foo_posix', 'foo_win32', 'foo_mac', etc. We might also need 'foo_linux_i86', 'foo_solaris2, 'foo_irix5', 'foo_winnt', 'foo_win98', 'foo_macos8', etc. * ...And that's just within Distutils itself. What about client code -- what if I have a module that sets itself up differently for different distributions; how do I specify that? * Recusive setup: what if my distribution has subdirectories containing other module distributions, with their own setup.py's? Will this ever be needed. (Undoubtedly.) How does the client specify them, and how does the Distutils run them? -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Anonymous CVS up-and-running
by John Skaller 28 Dec '98

28 Dec '98

At 17:43 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > I have done this, it is about 5 minutes work for a makefile > > expert. It took me half a day to get working properly :-) > > Cool. Send Guido a patch and rationale as to why it should be in >the standard distribution. I can't. It was a patch to version 1.5.0 which i don't have any more. It got clobbered when I installed 1.5.1 of course. >Frankly, the reason to install pythonpp is >lost on me. The you don't need to invoke it. But other people in the C++ SIG want to use C++ to extend Python. This is not the place to argue about it, they want it. :-) ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 28 Dec '98

28 Dec '98

At 11:58 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > This depends on knowing where the makefile and > > the config file are. Which suggest a further factorisation: > > Ok, I've modified distutils.sysconfig to contain: > > get_config_h_filename() > get_makefile_filename() > parse_config_h(fileobject, g=None) > parse_makefile(fileobject, g=None) > > Note that the parse_*() routines take a file object instead of a >filename, Ah, that seems even better, yes! >and the dictionary to update may be passed in. If no >dictionary is provided a new one is used. However the dict is >obtained, it is returned by the function. Right. Seems OK. > Equivalents for non-Unix platforms are still needed; has anyone >thought about Windows or Mac platforms? I don't think we (necessarily) need equivalents for the following reason: the sysconfig data isn't meant for user access: its there to support implementation of functions like 'compile_this_c_as_a_module'. It is these functions which need to be standardised (that is, cross platform). For the VC++ windows build, sysconfig could be omitted entirely, and a hand written 'compile' function provided. For Unix, it is convenient to parameterise such a function with the config data -- for that is the way the autoconf/ make/blah blah process works. none of this implies there shouldn't be an equivalent; but I think the important interfaces is the 'OO' one: <grin> the one characterized by function semantics rather than values of data attributes. > I passed on this for now. I think the cross-platform issues need to >be explored more. I agree. I would like people to explore the interface of my compilers module to see if it is _possible_ for that interface to work on all platforms. It may be that the Unix implementation uses sysconfig data to adapt to various Unix platforms, whereas the Mac or Windows version is hard coded, or uses a different scheme. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 24 Dec '98

24 Dec '98

>Can you send around a URL to the interscript online doc? Yes, but I should say that it has been cited many times, it exists on the www.python.org home page in the 'news' list, and it is in comp.lang.python.announce. http://www.triode.net.au/~skaller/interscript is the home page. >And when you >refer to it, could you please cite a more specific reference? Call it a >character flaw, but I'm not going to just blindly go read a bunch of >doc. A concrete reference helps (me, at least) provide some >focus/structure to why I'm there reading the thing in the first place. I agree, but I want to apologise profusely that I am not able to do this easily at the moment. firstly, my working copy is not the same as the one on the Internet, and secondly the HTML pages are all numbered at the moment, and the numbers are arbitrary and will change completely with every build i do that adds or removes even a single heading. [There are over 500 pages] I am addressing this problem, but at the moment the best I can do is usually to say 'read the section about XXX'. The table of contents is not small, but it should guide finding relevant information quickly. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] High level options interface
by John Skaller 24 Dec '98

24 Dec '98

At 11:26 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > BTW: One weakness of this interfaces is that there > > is no provision for creating an object library > > (archive, ar, etc etc). > > Is this needed? I don't know. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Enough about feature tests!
by Greg Ward 23 Dec '98

23 Dec '98

Hmmm, this seems to be a recurring theme; I guess this is just what a moderator is supposed to do, though. Anyways, it should be obvious but apparently is not: arguing about how best to test for OS features through Python is not really on topic for this SIG. We should be arguing, err discussing, high-level design of the distutils suite; or, failing that, nit-picking implementation details. For example, the sysconfig module, what to call it, when it runs, how it provides its information, etc. are all on-topic. Please stick to relevant things and learn to live with the fact that there's no formal specification for the Python Library, nor is there likely to be one anytime soon, nor will we solve that perceived problem with the distutils. Thanks -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Anonymous CVS up-and-running
by John Skaller 22 Dec '98

22 Dec '98

> Agreed. I seem to recall someone had sent patches to support C++, >possibly to the C++-SIG. Does anyone know exactly what's needed? Yes, I know what is 'needed', but not how to get it. What is required is that the python executable be built using C, and another version of it built using C++. Let me call them python.exe and pythonpp.exe. For gcc/g++, three extra steps are required in the makefile: 1) compile python.c using C++ as well as C, name it pythonpp.o(bj). This is the _only_ file that has to be compiled with C++. 2) Link that main object file with the C built python library. The result, pythonpp.exe, works just like python.exe, _except_ that it will accept full C++ dynamic link libraries as python modules: that is, it will support RTTI, exception handling, and proper constructor initialisation of virtual tables, correctly. 3) Install pythonpp.exe as well as python.exe I have done this, it is about 5 minutes work for a makefile expert. It took me half a day to get working properly :-) > > 2) Some of the data, such as CFLAGS, is useless because it reflects > > the python source build tree. On my system > > Misc/Makefile.pre.in manages to get things right in practice; some >time needs to go into figuring out just how it does things and get >sysconfig to do the right thing as well. In my opinion, part of the solution is to factor the 'variables' more logically, and then modify the makefile to use that factorisation. To start with, may I suggest _private_ variables which are not really required for sysconfig, be renamed in the makefile to some convention, such as a leading underscore: these can be ignored by the sysconfig parser as implementation details. Anything with a pathname related to the build directory is likely to be private: for example the current 'CFLAGS' contains -I../../something which is clearly useless to the installed system. So I'd rename that variable _CFLAGS. Then I'd create a real CFLAGS which _only_ has the option flags, no include directories. This allows the flags to be combined with the include directory options obtained for the _installed_ system. (That information is available now in sysconfig -- and indeed directly from Python) > > 3) The directory for config.h is WRONG! > > config.h lives in > > What definition are you looking at? Huh? The code in the CVS archive is wrong (for my system). It creates a directory name using '/usr/local/lib/python1.5/' which is NOT where config.h is installed. It is installed in /usr/local/include/python1.5 I.e. 'lib' should read 'include'. As a result, the module fails on my system. It is a stock standard Linux installation, IMHO it has put config.h in the right place, therefore sysconfig.py is wrong. :-) (Sorry, it is hard for me to quote stuff because I email on NT, but do real work on Linux :-) > > 4) On my system: > > > > SIZEOF_INT = '4' > > Patch attached which gets this right in a general fashion. Thanks! > > 5) I think the module should NOT just put values > > into itself like > > I don't see any need to complicate the interface. Neither do I. Factorisation doesn't complicate the interface, it simplifies it. There are two categories of information, and they come from separate sources. The data strucure should surely reflect this? > > 6) There is no information from Setup. It would be useful to > > get the Tcl/Tk library and headers into the configuration > > if it was built. Hmmm. :( > > Variables defined in Setup & friends are included. Otherwise, >nothing from Setup is included. This can be changed. It may make >sense to use a separate module for this (modconfig?). I think each source of data should be separate, including the makefile info and the autoconf info. The existing architecture is correct, IMHO: its the simplest possible 'pythonic' representation of the raw data. [This is not enough for the high level goal, but it seems correct to build this lower level first] > > 7) _init_posix should NOT modify a dictionary, but return one: > > CHANGE: > > This item is tightly related to item 5; let's deal with that one as >the real issue. The rest will fall out of that result. OK. > > 8) LINKCC = ' gcc' has a bad leading space. > > Fixed in the patch. Thanks. > > 9) I have no idea how to use the available information. > > There are flags all over the place. > > Yes, that's what configure produces. The next layer needs to put >this information together in a usable form. Reading the Makefile >should help a lot in determining how the various bits should be put >together. Yes, I agree. It is clear that while the current set of attributes is not quite right, it contains a lot of information. Therefore, the first step of just getting it into a pythonic format is a good one. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2

Re: [Distutils] Access to Python config info
by John Skaller 22 Dec '98

22 Dec '98

>[example has_symlinks() elided] > > But this can be written on top of feature tests. Why not? because it cannot be done. > def has_symlinks(): > global has_symlinks > import os > if hasattr(os, "symlink"): > has_symlinks = true > return 1 > else: > has_symlinks = false > return 0 Obviously, this does not work. It tests if there is a symlinks attribute in os, not what should be tested: 1) if there is an attribute symlinks, it works according to spec 2) if there is not, then the os doesn't support symlinks To perform (1) would require actually creating some symlinks, and seeing if they 'worked'. This is what autoconf does; it tries to compile various features, and it tests them as well (where they're _apparently_ available, and where it is possible to do a short test). It would be much easier to _document_ that the symlinks attribute is present if and only if creation of symbolic links is supported with semantics XXXX (fill in specification here). Then I could rely on the above code. To perform (2) is hard, but might involve using shell scripts. (Assuming you have a reliable way of launching them :-) > Have you created a list of the system aspects that this is desirable >for? Is the list *finite*? No. I just keep running into problems. And, obviously, it is desirable to have a finite list: tradeoffs are necessary. >It may well be, but there will be a lot >of disagreement over what it should be. yes. But there is nothing new here :-) >For now, let's call this the >"features" module. Are you ready to propose an API? No. But I will work on it with others. I do have some architecture developed in interscript for this: several frames for different categories of features: * platform * user * site with some work done on building these frames, but I'm not actually using any of it. You might start by reading the relevant sections in the online interscript doco. Then there is something concrete to discuss. It's quite clear that the separate 'compilers' stuff ought be part of this(but isn't, in interscript 1.0a7-9). >I'm sure several >of us would be willing to read it and carefully consider it (I will, >though it may not sound like it;). I think the biggest problem is >defining it in a way that provides more information or more convenient >access to the information represented. Yes, I agree. One way to help that process is to try to _use_ an actual implementation. I have an implementation, but I'm not using it. I could use help structuring it; doing so requires the combined knowledge of many people with experience on many different platforms. I, by myself, do not have wide enough experience. I do have one comment to guide your thoughts. Consider JPython everywhere. Don't assume CPython. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 3

Re: [Distutils] Access to Python config info
by John Skaller 22 Dec '98

22 Dec '98

At 14:22 21/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > HOW do you know this works? HOW do you know > > that the method is not, in fact, present, but doesn't work? > > WHERE is the standards document that specifies the > > precise semantics of the 'os' interface?? > > Sounds like I'd better get the documentation for "os" fixed before I >release again; it is on my list. I don't think that is really my point. I'm interested in a STANDARD, not just good documentation. Saying what the library does is a useful hint. Like reading the code. It isn't the same as a specification; which _requires_ certain behaviour. The python _language_ document is written more like a specification. If the language doesn't do what the doco says, its a bug in the implementation (or a fault in the doco). Not an excuse from Guido that it doesn't work on that platform 'because'. I don't want an explanation why something doesn't work, I want an authoritative document that I can cite as evidence of a bug. In other words, relieve myself of the responsibility of fixing it or working around it. I've been shown that my interpretation of what Python does is wrong, by citation of the language specification. I want to be shown wrong about the library (when I am), the same way. [On testing things to see if they work] > > This technique doesn't seem very reliable. :-( > > How so? Because it requires testing every single function in the whole system before using anything. Because there is no definitive specification of what might work or not. Do you see? And just _how_ do I test the functions? Using other functions that are themnselves suspect??? > But this can be written on top of feature tests. Why not? Sigh. Because there is no reliable way to test features. FOR EXAMPLE: consider the module 'commands'. How do I know if it works?? I do not see how to test it .. WITHOUT assuming it works. In fact, you can see by examining the code it will NOT work on NT. So interscript has it's own module called 'commands' that works on both posix and NT. I can't 'test' if it works, because it is one of the modules I need to test if tools such as patch and diff work. I have to rely on it working. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2

Re: [Distutils] Access to Python config info
by John Skaller 22 Dec '98

22 Dec '98

At 01:40 23/12/98 +1000, John Skaller wrote: Duh, sorry for replying to my own post again but: >EXECUTIVE SUMMARY > > 1. module parse_configuration, function get_config > returns a dictionary of config data parsed from > the makefile This depends on knowing where the makefile and the config file are. Which suggest a further factorisation: def parse_config_h(filename) def parse_makefile(filename) which would be generally useful! And truly statless. Now, provide: def get_config_h_filename() def get_makefile_filename() and we can wrap them up to get: def get_config(): dict = parse_config_h(get_config_h_filename()) dict.update(parse_config_h(get_makefile_filename())) return dict Again, this factorisation simplifies the design, and also creates added functionality. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 3

Re: [Distutils] High level options interface
by John Skaller 22 Dec '98

22 Dec '98

At 18:57 21/12/98 +0100, Martijn Faassen wrote: >class extension_compiler: > def compile_C(filename): > pass > def compile_C++(filename): > pass There's no need for a base class, just a specification. Please examine the following interface and see a gcc implementation below that. At least criticize the interface if you don't like it. The interface: # module 'interscript/compilers.c class python_module: def __init__(self,**kwds): # initialise the object with kwds as configuration. def configure(self,**kwds): # extend or change configuration def compile(self,filename, **kwds): # compile the file, with given options # return the string name of the object file def link(self,modname, filenames, **kwds): # link the list of object files, with given options # to generate the named python module. # returns the name of the shared library class application: # exactly the same, except produces an executable # tool that can be invoked from the command line # module 'interscript/compilers.cpp # exactly the same, except for C++ ---------------------------------------------------------------------------- ------ #line 121 "compilers.ipk" import os import sys import string import interscript.compilers.cconfig class python_module: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc = 'gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,modname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) dll = modname +'.so' cc ='gcc -shared -Xlinker -export-dynamic ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + dll print 'system',repr(arg) os.system(arg) return dll class application: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,appname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) cc ='gcc ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + appname print 'system',repr(arg) os.system(arg) return appname ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] High level options interface
by John Skaller 22 Dec '98

22 Dec '98

At 21:59 22/12/98 +1000, John Skaller wrote: >The interface: > ># module 'interscript/compilers.c > >class python_module: > def __init__(self,**kwds): > # initialise the object with kwds as configuration. > > def configure(self,**kwds): > # extend or change configuration > > def compile(self,filename, **kwds): > # compile the file, with given options > # return the string name of the object file > > def link(self,modname, filenames, **kwds): > # link the list of object files, with given options > # to generate the named python module. > # returns the name of the shared library > >class application: > # exactly the same, except produces an executable > # tool that can be invoked from the command line > ># module 'interscript/compilers.cpp ># exactly the same, except for C++ BTW: One weakness of this interfaces is that there is no provision for creating an object library (archive, ar, etc etc). ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 22 Dec '98

22 Dec '98

At 08:07 22/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 20 December 1998: >> I think you are missing the point here: if the configuration >> doesn't work, the client has NO CHOICE but to edit it. >> And if it isn't editable, the client has NO CHOICE, full stop. >> Now, you don't want to leave the client without options, in >> case something as complex as configuring a compiler fails, do you? > >John -- I don't think not being able to edit sysconfig leaves people out >in the cold. Fred's current implementation is, in fact, quite >non-editable, as it scans Makefile and friends as late as possible >(ie. when sysconfig is itself imported). Let me cut the arguments, and simply phrase a request. This gives us BOTH what we want. :-) EXECUTIVE SUMMARY 1. module parse_configuration, function get_config returns a dictionary of config data parsed from the makefile 2. module sysconfig exhibits attributes corresponding to the result of executing get_config when loaded 3. module build_config, by default, exhibits the attributes corresponding to the result of executing get_config AT BUILD TIME This module may be edited by the client. 4. The tool 'refresh_config.py' can be executed to print the (dynamic) configuration to standard output, and thus be used to generate a configuration module, including the standard one (3). DETAILS. Allow me to assume sysconfig.py, as it stands, is copied to parse_configuration, renamed get_config, and edited slightly to conform to the specified interface. Here is the new implementation of module sysconfig, with the same semantics as the existing one: # distutils.sysconfig from distutils.parse_configuration import get_config globals().update(get_config()) Nifty, eh? :-) Here is 'refresh_config.py': # refresh_config.py from distutils.parse_configuration import get_config for key, value in get_config().items(): print key+'='+repr(value) As you can see, both these auxilliary things are a single line of code. This is packaging. It is possible just by isolating the parser, and not doing anything tricky. :-) I think that invoking the refresh utility in the make process is also a single line change to the makefile! WAFFLY IMHO THEORETICAL COMMENTS: It is best to have stateless entities do the work, and then provide state as required by the architecture. Separating out the functionality is crucial to building multiple architectures: it allows the architectural implementation to be largely independent of the implementation of the functionality. This is the secret of Python methods: they're just plain old stateless functions. Bound methods are just architecture, independent of the functionality encoded in the unbound method. We should emulate this good, orthogonal, design! /WAFFLE Comments? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Access to Python config info
by John Skaller 22 Dec '98

22 Dec '98

>It *should* be done by the installation Makefile. I would hope that we >obsolete the third-party use of Setup/config.h/Makefile.pre.in. Those >are even close to the convenience that we can establish. Yes! I think this is a primary goal. I will note: it will _not_ work for all modules because of the problem finding libraries and header files. But it will work for all modules which implement 'pure' data structures, and it will work for all 'opt' modules that are just optimisations of Python script. This latter class is _crucial_; in fact it is vastly more important than distribution of modules: it permits dynamic (run-time) generation of C modules. In particular, one could write a 'compiler' that generated either Python or C from some language (such as Python :-) .. IN Python. (Because apart from generating text, which Python can already do, the only thing else needed is to compile that text into a binary: this is the missing link which will make C extensions work like Python modules). BTW: It is _also_ recommended to have the ability to generate applications, not just Python modules. The 'make' system know how to do that too, and it is useful for extending Python with 'shell' tools. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Anonymous CVS up-and-running
by John Skaller 21 Dec '98

21 Dec '98

At 18:54 18/12/98 -0500, Greg Ward wrote: >Hi all -- > >undoubtedly it's wildly premature to have source available before we >have any sort of high-level design, but what the hell. Absolutely NOT! Gee, haven't you learned some OO? <g> OO is a bottom up. You do detail work BEFORE you have a high level design. What you have done is absolutely right. Thanks!! >Only things there so far are my version number module and Fred's >sysconfig module. --------------------------------------------- Please put my compilers module in it too. It provides the high level abstract interface that is actually needed. At least, it tries to. It needs work! It needs a third component for Java as well. --------------------------------------------- I have incorporated the sysconfig module into interscript. Here are my comments on the sysconfig module. 1) C++ is not supported. If the option is not provided to build py++, a separate distribution may follow. Do we want that? [py++ is the same as python, except that the main translation unit is built with C++: the C built library is still used] We need the Makefile to support C++ so sysconfig can get C++ compilation flags. 2) Some of the data, such as CFLAGS, is useless because it reflects the python source build tree. On my system CFLAGS = '-g -O2 -I./../Include -I.. -DHAVE_CONFIG_H' 3) The directory for config.h is WRONG! config.h lives in /usr/local/include/python1.5 but the sysconfig module looks in /usr/local/lib/python1.5 CHANGE the line which gets this wrong TO: config_dir = os.join( sys.exec_prefix, 'include','python'+version) 4) On my system: SIZEOF_INT = '4' which is wrong, it should be SIZEOF_INT = 4 I got this using 'repr(value)'. Most other numeric values are OK. 5) I think the module should NOT just put values into itself like FLAG = VALUE I suggest: C_CAPABILITIES -- dictionary or module containing stuff from config.h MAKE_FLAGS -- dictionary or module containing Makefile variables be attributes of sysconfig. If you do not like writing sysconfig.C_CAPABILITIES['HAVE_STDDEF_H'] then go for the module, which allows: sysconfig.C_CAPABILITIES.HAVE_STDDEF_H 6) There is no information from Setup. It would be useful to get the Tcl/Tk library and headers into the configuration if it was built. Hmmm. :( 7) _init_posix should NOT modify a dictionary, but return one: CHANGE: g = globals() to g = {} and add return g to the end of the routine. Replace the exec line with g = eval('_init_%s()' % os.name) and add for k in g: globals()[k] = g[k] if you really want to modify the module dictionary. I don't. I use the data to generate the python source of another module. This is important to me: the generated python can be transmitted over the Internet. It can be edited. And I can have several distinct configurations. What I do is that the build code is NOT in sysconfig, but inline interscript (python) code which is executed to BUILD a file 'sysconfig.py'. 8) LINKCC = ' gcc' has a bad leading space. 9) I have no idea how to use the available information. There are flags all over the place. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Access to Python config info
by John Skaller 21 Dec '98

21 Dec '98

At 10:49 16/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > 1) does the OS support symbolic or hard links? > > import os > hasattr(os, "link") > hasattr(os, "symlink") > HOW do you know this works? HOW do you know that the method is not, in fact, present, but doesn't work? WHERE is the standards document that specifies the precise semantics of the 'os' interface?? This technique doesn't seem very reliable. :-( > > It may be the proper place for the answer is not in sys.config, > > but in os. But the os module can compute the answers from > > sys.config. > > For most of this, probably so. Recall also that Guido & others >recommend "feature tests" to determine the availability of >functionality rather than having a separate flag to indicate >availability. There are many ways in which this is more reliable and >maintainable, And some in which it is not reliable :-( >though what we're currently describing as sysconfig >remains pertinant. I would expect sysconfig to be used mostly by >things like distutils and not by typical programs. Hence, it would be >good for os to not import it. That may be so. However, 'feature tests' are useless for meta-programming. What I mean is: suppose one generated some code. If it had to be different depending on 'features', one could do the feature tests and generated it that way; but this would only work on the current platform. If you were generating code for someone else; then the feature tests would be useless ... unless they were done _by_ the generated code. Which is the autoconfig nightmare all over again. What Guido is recommending is exactly like the old fashioned DOS driver problem: test the video, or printer to see what kind it is, and configure a driver: usually by selecting it from a list a mile long _supplied with every single program_. I don't think this is precisely the way to do it. Dynamical feature tests are fine if they are done ONCE on installation; and present the results in a platform independent way. All modern operating systems have an abstract device driver API, and the _operating system_ installs the appropriate driver once when it is configured. With any luck, the configuration record can be ported, so it is possible to generate code for any configuration, (i.e. on a remote host): the generator _might_ optimise the code my providing some dynamic testing (of the configuration record on the client). Of course, the reliability of this mechanism does depend on establishing standards, and then making the implementation(s) comply. There's no reason the configuration module shouldn't do these dynamical tests _as implementation details_ of the implementation of a standardised API. If they're appropriate. _I_ don't want to do this hackery in code, however. It's too much like a bag of trickery. I'd be happy with a module containing: def has_symlinks(): import os return hasattr(os, "link") with a _specification_ of the function has_symlinks because this would allow me to create an alternate implementation (which could, for example, use a data record to represent the configuration of some remote system). Does this make sense? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

High level options interface
by Martijn Faassen 21 Dec '98

21 Dec '98

Hello, I've been following the discussion on how to build C (or C++, or other languages) extensions with interest. I'm thus talking about the 'build_extensions' command of the proposed interface, if I understand it well. Some talk has gone into reading a lot of low level Autoconf results into Python, and exposing them through a sysconfig module. Some objections were raised that this was rather low level. It's also not very portable, right? Options that go into gcc won't make much sense for Visual C++, I imagine. Now, we could try to abstract a compiler framework in Python. I understand this is what John Skaller is working on, but I may be wrong as I haven't looked yet. Abstracting a compiler with its myriads of options could be a pain. We might end up having to abstract makefiles as well in Python. And building C libraries. This could be a rather big project. We'd be duplicating a lot of stuff, as well. I'm not too keen on simply calling 'make' directly from our builder script either, though. That's rather tied to the platform. So let's try abstracting some more. See if it becomes any simpler. I think this is what other people might've proposed as well, and if so, I'm just saying 'me too', here. I'll sketch out a very simplistic and anarchic high level interface here. The base class: class extension_compiler: def compile_C(filename): pass def compile_C++(filename): pass That's about it, unless we want to choose to support more extension languages by default. For each platform we want to support, we supply a subclass of this. For example: class gcc_extension_compiler(extension_compiler): def compile_C(filename): os.system("gcc -c " + filename) def compile_C++(filename): os.system("gcc -c " + filename) That's just about all. We don't have any fancy options (though we might want to enable optimizations by default). But what about special optimizations, debugging, profiling, other fancy stuff? Our abstraction doesn't support this! We need more methods. We need set_optimization_level(10)! Do we? Who does? Generally, only the extension developer knows which special compiler options could work best with the extension code. The extension writer only knows this for his platform(s) of choice, too. Extensions should still build cleanly for anyone on any platform. Thus we do best with as abstract an interface as possible. The extension builder should be given the option to subclass, and provide special compilation options for optimization purposes. Profiling and debugging builds are generally only necessary for the extension developer, and could be handled by special throw-away subclasses. If the developer has figured out how to make extensions build best and fastest on Linux, he can always include a special purpose linux_extension_compiler subclass which calls the compiler with special optimization options. If later it turns out we do need special option settings (for all compilers on all platforms!), we can always add it later. Anyway, I'm leaving lots of stuff out here. We probably need to do the same simplistic abstraction to the linker. The emphasis is to keep the interface as small as possible. A large interface means supporting it on all kinds of different platforms, and that is a lot of work. Once we got a small basic interface going, we can always extend it, platform by platform, with more options, if necessary. A simple 'RISC' interface does limit developers. They can't do all kinds of fancy makefile stuff they usually do. But we don't want to have to support all the fancy makefile stuff anyway. ....remembers ./configure ./make and why the complicated ./configure step is there in the first place; to enhance portability. Hm. *ponder* Okay. Perhaps that invalidates my entire argument above. It might be that I'm limiting developers from using any external libraries in their extensions. Still, philosophically, I think my arguments are sound. In practice, however... Leaving-confused-ly yours, Martijn

1 0

Re: [Distutils] Freeze and new import architecture
by John Skaller 19 Dec '98

19 Dec '98

At 18:15 17/12/98 -0800, Greg Stein wrote: >> ... >> sys.path not be restricted to path names. > sys.path has "strings", and >> an associated map of "module finders". Thus, a sys.path entry could >> have a directory name (like now) or .zip file, URL, etc. > >I would much prefer to see the module finder instances in the sys.path. I agree. But that would be a compatibility problem? >Sometimes, it is *very* difficult to map strings to module finders. Yes. I think this is a serious weakness of the proposal. >> 1. Finding the module in a specific namespace. >> 2. Importing a module of a specific type, once it has been found. > >I think the separation is bogus. I don't: I'd _like_ to add two things, as a client of the system: 1) Add a new type. For example, allow .c files to be loaded, by compiling them first. 2) Add a new kind of namespace. For example, an FTP server. Or a hook to Trove. Or a private data structure I designed myself. (1) has to do with what kinds of things are loaded, whereas (2) has to do with where they are. Note that the finder must be able to _fetch_ the data to a place that the loader can load it from. If a single place is enough, then the two features are orthogonal, and thus each is separately amenable to Object Oriented development. So .. it is desirable to build an abstraction in which the functionality is separate. >> Regarding 2: the finder currently returns a structure that enables the >> correct >> importer to be called later on. Importers that we have are for >> builtin, >> frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind >> the >> same interface, PYC-resource importers (mac-only) and PYD-resource >> importers >> (mac-only). > >Punt this. Just import the dumb thing in one shot. But you can't: a .dll file is imported by saying dlopen(), whereas a .py file is imported by compiling it to a .pyc file which is then imported. Etc. 'One shot' implies a single function which is not extensible. >Take the example of an HTTP-based import. Separating that into *two* >transactions would be painful. It should be imported in one fell swoop. >And no, you can't just keep the socket open and pass that to the loader >-- that implies that you can defer the passing for a while, but the web >server will time out your connection and close it. Conversely, if the >intent is *not* to hold the "structure" for a while, then why the heck >have two pieces? The way I see it, the _finder_ is responsible for downloading the file to the local file system, where the loader requires it to be. The loader turns these raw bits into a module. >Both of our proposals guarantee that stuff in sys.path are not >pathnames. If I insert a "foo.zip" or a >"http://host.domain.name/pymodules/", then you certainly dont have >pathnames. > >I believe the biggest issue with my proposal is the fact that the values >are no longer strings. That's easy to fix: have a default 'finder' that is used if the sys.path entry is a string. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Freeze and new import architecture
by John Skaller 19 Dec '98

19 Dec '98

At 11:45 18/12/98 +1100, Mark Hammond wrote: > WHAT WE WANT > ============ > >What we'd like I'll try to describe top-down (hopefully better to >understand >than bottom-up). > >importing a module becomes something like > > for pathentry in sys.path: > finder = getfinder(pathentry) > loader = finder.find(module) > module = loader() >getfinder() is something like > if not path_to_finder.has_key(pathentry): > for f in all_finder_types: > finder = f.create_finder(pathentry) > if finder: > path_to_finder[pathentry] = finder > break > return path_to_finder[pathentry] I think this is excellent! But I'm not clear on how every kind of finder can be created for every path. [Also, it should be possible to flush the cache] ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Access to Python config info
by John Skaller 19 Dec '98

19 Dec '98

At 15:46 17/12/98 -0800, Greg Stein wrote: >hehe... boy, you do like things in a particular way, don't you? Now I >know where the version number stuff came from :-) i do think it is fair for the coordinator to ask for help organising things. Although I'll have a problem with CVS/RCS myself. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Access to Python config info
by John Skaller 19 Dec '98

19 Dec '98

At 18:36 17/12/98 -0500, Greg Ward wrote: >Well, Fred, you've set the ball rolling... I guess coding and design >will, as usual, proceed hand-in-hand... ;-) I'll modify my compilers modules to use the information as presented in this module. That should upgrade it from Redhat Linux to 'arbitrary Unix system'. And might even work on a Cygwin system. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Access to Python config info
by John Skaller 19 Dec '98

19 Dec '98

At 17:20 16/12/98 +0100, M.-A. Lemburg wrote: >Greg Ward wrote: >> >> On the superficial front, Marc-Andre hasn't convinced me that sysconfig >> is a better name than sys.config. >So the reasons I renamed it from sys.config to sysconfig were simply: I think I favour this, but not strongly. >> No! At least not in the long term: as Greg Stein pointed out, the >> config module (whatever it gets called) should just be generated when >> Python is built. Or were you talking about the near term, where we will >> need a distutils release that works with Python 1.5.2 and is allowed all >> manner of grotesque hackery to work around the lack of a config module, >> such as digging up and parsing Python's config.h, Makefile, and Setup? > >I wouldn't consider this grotesque hackery. I would. But it doesn't matter. What matters is not how it is implemented, but the interface it is required to have. (Where 'interface' includes semantics) The principal task of this effort is _not_ implementation but standardisation. >The parsed information could then be stored in a _sysconfig module >which subsequent invocations then use which is close to what Greg >proposed except that the first run is not necessarily done by the >installation Makefile. Yes. And there are number of approaches, some to be used before the module is distributed as standard, and some after. We need to know we _can_ implement it, and we need some actual implementations to examine, but this is not as important as standardising the API. What we need to produce is a specification, not (just) an implementation. And I come back again to the point that what the client needs is a module that actually compiles C code to a dynamically loadable shared library, so that the supplier can provide the C code _without_ any configuration stuff. Getting the compiler flags, etc, is all very well, but it isn't what is required by the end user. The person who needs all these flags is probably the supplier .. when the client complains the code doesn't compile. So it is important that the detail data be portable (i.e. can be physically sent across the internet). In my opinion, what is important is the API: the actual function signatures used to compile C code. Not how they're implemented. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Access to Python config info
by John Skaller 19 Dec '98

19 Dec '98

At 11:11 16/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 17 December 1998 [on whether the config module >should be builtin or a separate .py file]: >> I don't see any real choice: it has to be separate AND editable. >> Because if I download a binary Windows version, the information will be wrong >> for my system. The Unix version might be wrong too: perhaps I built >> Python with gcc, but I want to build extensions with ecgs, or with edg. > >There are a number of reasons why you don't want to do this, and it all >boils down to the fact that in the C world you cannot reliably link two >object files if one was compiled with compiler X and the other with >compiler Y. I agree: this is a valid point. >Anyways, these all constitute reasons for *strong recommendations* >against fiddling with the compiler or flags. I'll concede that someone >who *really* knows what they're doing should be allowed to do this (but >they should also have to sign a disclaimer first ;-). I think you are missing the point here: if the configuration doesn't work, the client has NO CHOICE but to edit it. And if it isn't editable, the client has NO CHOICE, full stop. Now, you don't want to leave the client without options, in case something as complex as configuring a compiler fails, do you? At least, they should be able to ask for help on the newsgroup, and edit the file with help from experts, shouldn't they? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Anonymous CVS up-and-running
by Greg Ward 18 Dec '98

18 Dec '98

Hi all -- undoubtedly it's wildly premature to have source available before we have any sort of high-level design, but what the hell. Go visit http://www.python.org/sigs/distutils-sig/cvs.html to find out how to access it. The quick version: use CVS (1.9 or greater) to login to cvs.python.org as 'distutilscvs'; the password is 'painless'. Checkout the 'distutils' tree. Poke around. The interesting source is in 'distutils' (ie. 'distutils/distutils' from the directory where you run 'cvs checkout). Only things there so far are my version number module and Fred's sysconfig module. (Which is currently actually 'distutils.sysconfig', maybe not a bad thing.) version.py is copiously commented and guaranteed to start another flamewar. So if you do look at it, *please* don't post your immediate thoughts to the list. Send 'em my way if you must vent your spleen. Version numbers are still a banned topic until we have some idea of what they'll be used for (ie. high-level design). Anyone who wants to take a stab at a high-level design should once again pore over the requirements and proposed user interface on the SIG web site. We haven't had any discussion about the proposed interface, and I very much doubt that it's perfect in every way. I think talking that over is a sensible thing to do before starting in on the design. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Configuration etc.
by Konrad Hinsen 18 Dec '98

18 Dec '98

I don't claim that I read everything that all of you posted on the topic of retrieving and storing the configuration information from the Python build process - I also have to work in between ;-) But it seems to me that this discussion misses the point, or at least deals with a minor implementation detail. The build phase of the installation process (needed only for source installation of extension modules, of course) should use a much higher-level interface to the DistUtil core than just the information from the Makefile. As a package provider, I want to write something like from DistUtils import compileExtensionModule compileExtensionModule("foo", ["foomodule.c", "bar.c"]) Everything lower-level is an implementation detail and not worth 10 KB of discussion ;-) A DistUtils implementation for Python 1.5.x on Unix will have to parse the configuration file or Makefile from the Python build, using some module much like the one posted by Fred. For other platforms, this kind of information may not be available, or not necessary, or useless. For later Python versions, everything might be different. So I see no reason at all to store such specific data in a documented standard library module, whatever it is called. It may even be better not to store it at all and parse the original input every time the information is needed. And now for something completely different: implementation strategy. At some point we will have to move from discussing to programming, after all. It seems to me that the whole DistUtils project divides nicely into two almost independent subunits: - building binaries from source code - installing Python modules and binary extension modules So maybe we should create two "task forces" that work on these subtopics, of course communicating as necessary. That said, I'll disappear for a two-week Christmas vacation, hoping that the discussion during these two weeks won't fill up all my disk space... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

4 5

Access to Python config info
by Greg Ward 18 Dec '98

18 Dec '98

OK, we seem to have a better degree of consensus on this than on that version number thing. Good -- hopefully we won't scare anyone off with an extended argument. On the superficial front, Marc-Andre hasn't convinced me that sysconfig is a better name than sys.config. (On the other hand, I never said whether I thought it should be a sub-module of sys, or a [potentially ambiguous] module under a package called 'sys'.) My rationale for calling it sys.config is that sys is "Python system stuff"; sys.config would basically be more of the same, but probably a *lot* more -- so I don't think it should clutter up the well-known, well-understood 'sys' module. One other possible name is, of course, 'config', but then the Perl guys might sue us for copyright infringement. >joke!< Marc-Andre also said: > Rather than discuss which things to include or not, just > put all the information in there you can find by simply parsing > Makefile. Note that the parsing mechanism should also allow > for fairly decent $(VARIABLE) expansion. No! At least not in the long term: as Greg Stein pointed out, the config module (whatever it gets called) should just be generated when Python is built. Or were you talking about the near term, where we will need a distutils release that works with Python 1.5.2 and is allowed all manner of grotesque hackery to work around the lack of a config module, such as digging up and parsing Python's config.h, Makefile, and Setup? [Marc-Andre again] > But the install dirs are changeable during configure... we might > need some help from Guido here, e.g. the sys module could > include the information: sys.configdir, sys.incldir and sys.libdir. Now you're cluttering 'sys' up with stuff in the yet-to-be-named config module. These directories should just be accessible variables in sys.config, eg. sys.config.install_bindir sys.config.install_scriptdir sys.config.install_libdir etc. Greg Stein pointed out: > Actually, at the discussion, we planned on compiling them right into the > interpreter. That would prevent version skews between a compiled Python > and the libraries. > > However, now that I'm thinking about it... screw that. If "import > sysconfig" picks up the wrong version, then you have plenty of other > issues to deal with. I'd much rather see a .py file that a bunch of > stuff compiled into the interpreter. I'm torn on this one. Building into Python prevents version skew, keeping it separate means it's easily readable without writing code. Both are compelling arguments. Greg has given one voice in favour of a separate .py file... anyone else? [Greg S. again] > I'd also recommend ignoring all the dictionary vs instance stuff. Just > have a bunch of assignments dropped into a sysconfig.py module. IMO, it > would look a lot like FCNTL.py (h2py output files). No need to get fancy > with defaults and stuff. Just write the dumb file out. I (almost) completely agree. If we put *everything* from config.h into the config module, it might be nice to store it separately (eg. in a hash). But the stuff from Makefile and Setup should just be dumped in flat as variables. Are there other sources of information we should be worrying about? (Eg. does config.cache have anything valuable that doesn't wind up in config.h or Makefile?) Finally, John Skaller respond to my sketch of what should be in the config module: > This is too Unixy. Absolutely true. I never claimed to be anything but a Unix bigot. Two rebuttals though: * if platform X supports running the compiler from a command line or a script, surely that compiler can take some arguments... and surely some of those arguments will deal with optimization/debugging, some will deal with preprocessor settings, and some are just plain miscellaneous * splitting the flags up into all those categories can be useful, but it could be that I'm confusing the Python build process with the extension module build process. Perhaps the way to build an extension should be carved in stone when Python itself is built, and extension builders shouldn't be allowed to muck with it. If so, then John is absolutely right about just calling a function to invoke the compiler for such-and-such a situation. Gee, I guess that second one wasn't much of a rebuttal. So much for my combative spirit. ;-) > An excellent sketch though. Why? because it is concrete > enough to be argued about! Thank you! that was the whole idea; I hate abstract arguments (the only courses I came close to flunking in university were philosophy and psychology...). And I have a nasty tendency to attack anyone who brings up airy-fairy concepts with *something* concrete to explain his ideas. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

6 15

Freeze and new import architecture
by Mark Hammond 18 Dec '98

18 Dec '98

[Apols in advance for the length of this - but the background is probably necessary] I will make no attempt to outline what the freeze tool should ultimately be! Im not even going to outline what freeze is now! Im just going to focus on one single requirement. We want freeze to be capable of working with any number of "code containers" - ie, to be able to locate code in "frozen C modules" (as it works now), or potentially in a .zip file etc. The justification is fairly easy to explain: Consider that you may want to use "freeze" to distribute an application that consists of a number of Python programs. Using freeze as it stands now, the entire Python library is frozen into each application. Thus the result is each program is frozen to an executable that may be megabytes in size, even though the bulk is likely to be the identical, standard Python library. Attempting to cut a long story short, Guido, Jack Jansen, myself and Just came up with another idea for an extensible import mechanism: sys.path not be restricted to path names. sys.path has "strings", and an associated map of "module finders". Thus, a sys.path entry could have a directory name (like now) or .zip file, URL, etc. I have included below a mail from Jack Jansen on this topic. To paraphrase Guido's response, it was: "looks good - but lets define a Python interface so it may work with JPython" Accordingly, I have mocked up a few .py files which start to implement the ideas in Jack's post. However, Im now floundering a little as to the best way to take this any further. Im really looking for further support or critisism of the idea, and some interest from people in helping take this further... If there is general interest, I will post my mock-up, which allows directories and URLs to exist on sys.path... Thanks, Mark. -----Original Message----- From: Jack Jansen [mailto:Jack.Jansen@cwi.nl] Sent: Wednesday, 12 August 1998 1:11 AM To: guido(a)python.org; just(a)letterror.com; mhammond(a)skippinet.com.au Subject: An import architecture I thought I'd just mail you the results of a discussion Mark and myself had about the import architecture. I'm just sending it out to us four, but I guess we should get more people into the discussion if Guido thinks that this is something worth "solving". The problem we're looking at is that import.c and importdl.c have lots of special case code, and everytime someone comes up with a new neat way to import modules more special case code has to be added. This happened with Mac PYC resource modules and PYD modules, and there's also importdl.c which nows about DLL modules for umpteen different systems. It would be nice if there was an architecture in import.c that would allow the machine specific code to hook into (and, ultimately, Python code as well). WHAT WE HAVE ============ We noticed that there are really two issues involved in importing modules: 1. Finding the module in a specific namespace. 2. Importing a module of a specific type, once it has been found. For 1. we currently have 5 namespaces: builtin modules, frozen modules, the filesystem namespace, the PYC resource namespace in a file (mac-only) and the PYD resource namespace in a file (again mac-only). Other namespaces can be envisioned, for instance the namespace inside a squeeze .pyz archive, a web-based namespace, whatever. The builtin and frozen namespace are currently special, in that they don't occur in sys.path and are always virtually at the very front of sys.path. On the mac sys.path entries can be either filenames (in which case the latter two modulefinders are invoked) or directories (in which case the filesystem finder is invoked), on other platforms there are only directories in sys.path right now. Regarding 2: the finder currently returns a structure that enables the correct importer to be called later on. Importers that we have are for builtin, frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind the same interface, PYC-resource importers (mac-only) and PYD-resource importers (mac-only). WHAT WE WANT ============ What we'd like I'll try to describe top-down (hopefully better to understand than bottom-up). importing a module becomes something like for pathentry in sys.path: finder = getfinder(pathentry) loader = finder.find(module) module = loader() getfinder() is something like if not path_to_finder.has_key(pathentry): for f in all_finder_types: finder = f.create_finder(pathentry) if finder: path_to_finder[pathentry] = finder break return path_to_finder[pathentry] And there would be a call whereby a finder type registers itself (adds itself to all_finder_types). finder.create_finder() would examine the current path component, see if it could handle it, and, if so, return a finder object that will search this path component everytime it is invoked. The usual case is that getfinder doesn't do much: only the first time you see a new entry in sys.path you'll have to ask all the finders in turn whether they support it, ad you just remember this. The loaders register themselves with the finders, passing finder-specific arguments. For instance, the .py loader registers itself with the filesystem finder, telling it that the ".py" extension should result in a .py loader being created. The unix-specific dll-loader does the same for .so extensions, the windows-dll-loader for .dll, the mac-dll-loader for .slb etc. The PYC-resource loader tells the mac-resource-finder to look for 'PYC ' resources, the PYD-resource loader tells it to look for 'PYD ' resources, etc. The information that is passed from the finder to loader when it has found a module is again finder-specific: the filesystem loader will probably pass an open file and a filename, etc. A loader can register itself with multiple finders, assuming their interfaces are similar. So, the .py loader could register itself not only with the filesystem finder but also with a url-based finder or something, as long as that url-based finder uses the same calling convention for creating the loader. WHAT DOES IT BUY US =================== A greatly simplified import.c, importdl.c split out over the various platforms (and with the possibility to pass machine-specific info from the find phase to the load phase, something that cant be done now and leads to double work on various platforms) and easy extensibility. There is the issue of performance. The description above is all Pythonish, but going through the Python calling sequence for all these things is probably not a good idea from a performance standpoint. This is however fixable. The objects involved are - The finder "class", the thing you use to check whether a certain sys.path component can be handled by this code - The finder instance returned by this class - The importer class (called by the finder objects) - The importer instance (returned by the finder instance through invoking the importer class). These 4 things could well be 4 specific PyObject types, with the needed C-routines in the object struct. Calling these for the normal case (i.e. when they're implemented in C) would be at most a single (C-) indirection more expensive than the current scheme. Moreover, it would be easy to create a module that would implement these 4 types as wrappers around Python code. The C-routines would then do the usual PyEval_CallObject stuff to call the Python implementation. So, you'd have the generality but you'd only pay for it when you put Python-handled entries in sys.path and actually hit those entries. ODDS AND ENDS ============= A side issue: this stuff would also allow us to put the builtin and frozen namespace into sys.path explicitly, for instance as "__builtins__" and "__frozen__", something I would like. The disadvantage would be that you can't be sure that everything on sys.path is a pathname, but the advantage would be that you could, for instance create a frozen program that you could patch: set sys.path to ["/usr/local/FooPatches", "__frozen__", "__builtins__"], and whenever you have a patch to a single module in a frozen executable you just send your clients the single .pyc file and tell them to put it in the FooPatches directory. There'd probably have to be a bit of code that explicitly prepends "__frozen__" and "__builtins__" to sys.path if they aren't there already or something. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)cwi.nl | ++++ if you agree copy these lines to your sig ++++ http://www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

Re: Proposal: C/C++ compiler identification
by David Ascher 16 Dec '98

16 Dec '98

[sorry, I was disabled due to bounces, I'm just reading up now]. GvR: > For Windows, since the only supported compiler is VC++, it could be > hardcoded. However, there's a very serious problem on Windows with > Interscript's approach of invoking the compiler at run-time: most > users don't have this compiler. I realize that gcc is available for > free, but I think it isn't compatible with VC++. As far as I know, > VC++ is required if you want to use any of Mark Hammond's stuff (COM > and MFC). I don't know if Tcl/Tk can be used (without recompilation) > from a gcc-based app, and I don't know how easy it would be to > recompile. (Does gcc on Windows have support for Win32 APIs at all?) FYI: The status of gcc on Windows as far as I understand it: -- it's called the Cygwin project -- it's supported by Cygnus (see www.cygnus.com) -- it uses egcs, not gcc -- current build is b20 (note: doesn't comply with version number scheme discussed earlier =) -- it provides access to the win32 api if you want -- it needs access to cygwin.dll on the machine (not a standard MS library =) -- in the past, it was very hard or impossible to have gcc-compiled and VC-compiled objects linked together, even statically. -- I believe things are better now, but far from trivial (I'm still at b19 because of final comment below) -- I could imagine doing things like compiling Python + NumPy together, but I'd never try Python+PIL+NumPy+Tk+readline+threads+... Final comment on Cygwin: While Cygwin is free, it is *hard* to install and setup in a proper way. My advice to all Windows users who want to compile things is to buy VC -- the Cygwin stuff is just not there yet when it comes to novices or folks who are short on time. Final comment on GvR's point above: I don't think VC is needed to use MHammond's COM stuff. Just the DLLs, which are installed on reasonable machines. --david "my Mom still can't forgive me for using Windows" ascher

2 1

Re: [Distutils] Access to Python config info
by John Skaller 16 Dec '98

16 Dec '98

>I'm torn on this one. Building into Python prevents version skew, >keeping it separate means it's easily readable without writing code. >Both are compelling arguments. Greg has given one voice in favour of a >separate .py file... anyone else? I don't see any real choice: it has to be separate AND editable. Because if I download a binary Windows version, the information will be wrong for my system. The Unix version might be wrong too: perhaps I built Python with gcc, but I want to build extensions with ecgs, or with edg. BTW: I see no harm in putting all the configuration options into a dictionary as Marc Andre suggests. But I don't think that is _enough_. >Finally, John Skaller respond to my sketch of what should be in the >config module: >> This is too Unixy. > >Absolutely true. I never claimed to be anything but a Unix bigot. I'm a 'platform independent' bigot ;-) >Two rebuttals though: > > * if platform X supports running the compiler from a command line or > a script, surely that compiler can take some arguments... and surely > some of those arguments will deal with optimization/debugging, some > will deal with preprocessor settings, and some are just plain > miscellaneous Yes, but just having a bunch of flags doesn't tell me how to invoke the compiler. For example, Unix compilers use a -l flag, followed by the name of a library _without_ the first three letters (lib) and _without_ an extension, and _without_ a pathname. Windows compilers don't use this convention. You have to supply a full pathname, and some of them do not support any kind of library search. For example, some compilers will compile a .c file as c. And a .obj will get linked but not compiled. I'd need to know. Is there a space allowed after the -o option?? Do I put the name of the executable there, or the name _without_ .exe, which is added automatically? Now, you could add meta-flags to answer these questions. And it would become very complex to calculate how to invoke the compiler. > * splitting the flags up into all those categories can be useful, but > it could be that I'm confusing the Python build process with the > extension module build process. Perhaps the way to build an > extension should be carved in stone when Python itself is built, and > extension builders shouldn't be allowed to muck with it. If so, > then John is absolutely right about just calling a function to > invoke the compiler for such-and-such a situation. > >Gee, I guess that second one wasn't much of a rebuttal. So much for my >combative spirit. ;-) Do both! I don't see the 'function' as 'carved in stone', i see it as the default easy way to do it. It will _not_ always be enough! But it should always work for ISO compliant 'pure C', i.e. stuff that doesn't call non-standard libraries (other than Python of course :-) For example, it is enough for mxTools. Not enough for _tkinter. ----------------------- Here are a questions that sys.config could answer: 1) does the OS support symbolic or hard links? 2) what information is in fstat? 3) does the OS supply big/little endian Unicode files? 4) What operating system is it? [This is a hard question] 5) is os.system available? 6) What is the size of a Python 'integer' 'float'? 7) What is the eol character sequence? 8) is there a 'drive' prefix? 9) Can you delete an open file (Yes under Unix, No under NT) 10) Is there a stderr and/or stdlog file 11) Is there a console at all? 12) Is there a compiler available? 13) Is this J or C Python? It may be the proper place for the answer is not in sys.config, but in os. But the os module can compute the answers from sys.config. Hmm. Something I'm trying to say is that the point of the module has to be to provide a _standard_ interface to various features such as compiler invocation and installation. That is, a platform independent one (where possible of course). Note that as soon as you consider JPython, the system compiler is a java compiler, not a C compiler. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2

RE: [Distutils] Observations
by Robertson, Joseph M 16 Dec '98

16 Dec '98

Hi all, This is only day 2 for me so I apologize if I haven't absorbed all your info yet. Instalit was a generalized installation toolkit for DOS/Windows similar to (but more powerful than) InstallShield. I used to work at the company. Alas it is no more, but its salient features were scriptable 'build' process and 'scriptable' install process. The scritpable build process was extremely valuable, for remote or timed builds across the enterprise (IBM used it for this). I'm just gonna hang here for a while and try to grasp the 'big picture'... Joe Robertson 256-730-8216 jmrober1(a)ingr.com jmrobert(a)ro.com > -----Original Message----- > From: Greg Ward [SMTP:gward@cnri.reston.va.us] > Sent: Wednesday, December 16, 1998 7:33 AM > To: distutils-sig(a)python.org > Subject: Re: [Distutils] Observations > > Quoth Robertson, Joseph M, on 15 December 1998: > > I like the scriptable builds stuff. Just like Instalit of the old days. > If > > fact I'd suggest we use a similar method. The build script tags data > such > > as whether to check file dates, versions, etc when installing; what > > compression type to use, etc, etc. > > What is "Instalit"? -- I've heard the name pop up once or twice in the > last week or so, but never heard of it before. Remember, we're not all > on the same wavelength here!

1 0

Re: [Distutils] Access to Python config info
by Robin Friedrich 16 Dec '98

16 Dec '98

> From: Andrew M. Kuchling <akuchlin(a)cnri.reston.va.us> > I'd sum it up as: if it goes in the Python binary, sys.config > is OK. If it goes in an external .py file, sys.config is not OK, and > I'd vote for sysconfig or some new package. > > >I'm torn on this one. Building into Python prevents version skew, > >keeping it separate means it's easily readable without writing code. > >Both are compelling arguments. Greg has given one voice in favour of a > >separate .py file... anyone else? > > Go for the .py file, IMHO. Another vote for impeac^H^H^H^H^H^Ha separate .py file to store this. Mostly it's better since it allows tweaks to be made easily, (say the installpath changes, or a new compiler), a hundred different little things can happen at a local site. It'll also help during the development phases of this tool to have visibility into what's actually happening. -- Robin K. Friedrich Houston, Texas

1 0

Note on building on AIX
by Andrew M. Kuchling 16 Dec '98

16 Dec '98

On the XML-SIG, Konrad Hinsen mentioned the following problem with shared libraries on AIX. I'm forwarding this just to read it into the Distutils record, as something that should also be fixed. -- A.M. Kuchling http://starship.skyport.net/crew/amk/ Quaint though this attitude seems now, it was unquestionably the prevalent one in the nineteenth century, and it would be over-bold to say that it will never return to favour, for the range of human folly is infinite. -- Robertson Davies, _A Voice from the Attic_ ============================= Konrad Hinsen wrote: > > - Some people need patched versions; for example, the standard > > version does not work for AIX. > > What's the patch that's required for AIX? And is there some > reason it can't be rolled into the Makefile.pre.in for 1.5.2? I don't know the system well enough to decide. The problem is that shared library linking is a rather complicated process under AIX, which is handled by two shell scripts. These shell scripts come with the Python distribution (they are "ld_so_aix" and "makexp_aix") and are ultimately installed in the "config" subdirectory of the Python library. But during the compilation of the interpreter and its standard library modules, they reside in the "Modules" subdirectory of the Python distribution. The settings in the configuration reflect this initial situation, not the one after installation. So if you use the standard Makefile.pre.in, the two critical definitions becom LINKCC= $(srcdir)/makexp_aix python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(srcdir)/ld_so_aix $(CC) whereas they should be LINKCC= $(LIBPL)/makexp_aix $(LIBPL)/python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(LIBPL)/ld_so_aix $(CC) -bI:$(LIBPL)/python.exp I suppose this could be arranged during the installation process, but I don't really want to figure out how that works! ...

1 0

Re: [Distutils] Observations
by Greg Ward 16 Dec '98

16 Dec '98

Quoth Robertson, Joseph M, on 15 December 1998: > I like the scriptable builds stuff. Just like Instalit of the old days. If > fact I'd suggest we use a similar method. The build script tags data such > as whether to check file dates, versions, etc when installing; what > compression type to use, etc, etc. What is "Instalit"? -- I've heard the name pop up once or twice in the last week or so, but never heard of it before. Remember, we're not all on the same wavelength here! > Question: How does your system install a working python program if the > target has no python whatsoever on it? That's something I for one haven't considered. It's not currently on the plate of the SIG, but should probably be considered for the future. Here's one possible long-term roadmap: * first, be able to build, distribute, and install Python modules: this will make it easy to extend an existing Python installation * next, be able to install a complete Python application (scripts + modules): this will mainly involve doing the module installation thing plus dependency tracking (ie. what *other* modules are required by this application) * finally, be able to distribute a Python application to systems that don't have Python at all All three are worthy goals indeed, and "distribution utilities" certainly covers all three. But there's a clear order to be followed, and right now we're concentrating on the first job. The second should follow without a great deal of difficulty, and third could either be a snap (just install one of the large number of pre-built Python distributions) or a morass. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 16 Dec '98

16 Dec '98

At 16:33 15/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 16 December 1998: >> I want the build process to put that >> information into a standard module. So I can get it out, >> and call up the compiler to make a C extension FROM PYTHON. > >A-ha! Thank you -- now it comes clear. If I were more weasely, I could >squirm out of it by saying "Well, this just follows from the requirement >to easily build extension modules from source" or, "That's not a >requirement, it's an implementation detail". You could, you'd be right in both cases, and you cannot be expected to do everything: which is why I wrote the article. :-) >But I'm not a weasel, You're not god. You started this list to get help from others. >and I think at some point I meant to mention this >in the requirements document. So I've added it. Great! Thanks! >I'm managing to keep implementation details out of the requirements >list, though. In particular, my proposal is for a new 'sys.config' >module which would contain everything you might need to know about how >Python was configured. YES!!!! I agree! >The most important items are the compiler, >compiler flags, platform, and library installation directories. Can I recommend you look at the interscript compilers module again? Instead of storing compiler flags, etc, you need an actual command that does the compilation. I.e. a python function that builds a command string and os.system's it. I don't think that 'data' can store this information, there are too many options. (Store the data as well!) >For an example of the degree of flaming overkill possible in this sort >of module, try this > perl -MConfig -e 'map { print "$_: $Config{$_}\n" } sort keys %Config;' >You'll note that Perl's Config module stores *every* piece of >information, gleaned, gathered, or queried from the user while running >Perl's 'Configure' script. I don't think Python's sys.config need be >*quite* so exhaustive, I agree. After all, we can upgrade it later. The important thing is to get the module into the distribution. That way people can try it, and we'll find out what is missing. Anyone on this list have a Next machine?? >Here's a very rough start >(loosely based on my long experience with building Perl and my very >brief experience with building Python, and a bit of poking through >Python's makefiles): This is too Unixy. > cc C compiler used to build Python > cppflags C preprocessor flags used to build Python (-I, -D, -U) > optimize optimization/debugging flags used to build Python > ccsharedflags C compiler flags required to build shared objects > extra_ccflags any other C compiler flags used to build Python > ccflags all compiler flags used to build Python: > cppflags + optimize + extra_ccflags > ccextflags all compiler flags used to build Python extensions: > cppflags + optimize + ccsharedflags + extra_ccflags > > ldlibdirs -L flags used to build Python > ldlibs -l flags used to build Python > extra_ldflags any other needed linker flags > ldsharedflags linker flags required to build shared objects > ldflags all linker flags used to build Python: > ldlibdirs + ldlibs + extra_ldflags > ldextflags linker flags used to build Python extensions: > ldlibdirs + ldlibs + ldsharedflags + extra_ldflags I do not think _any_ of these flags, etc, are required individually. Nor do I see how they can possibly be enough. You are making assumptions about how an arbitrary compiler is invoked. What you need is a _function_ which compiles a c file for a shared library, and another which links them. In other words, you just need a string with some 'slots' in it for the filenames, and which has all the flags already set in the right places. The function is a generalisation of that. Have a look at the compilers module I posted here. > platform hardware/os/vendor (or something along those lines) Yes. Something like that. This is necessarily crude information. > install_bindir directory to install binary executables > install_scriptdir directory to install non-binary executables > install_includedir directory to install C header files > install_confincludedir directory to install platform- and > build-specific C header files > install_libdir main Python library directory > install_sitelibdir directory for site-specific modules Each of these things is what i call an 'install point'. It isn't clear that the list is enough: clients may want to organise things differently for different modules. For example, the place PYTHON puts scripts is definitely NOT where I want extension packages: I want them in the site-packages directory. This is tough. I think your list is a good start. Please add: install_htmldir directory to put HTML doco install_mandir directory for man pages (does anyone provide them?) >Obviously this is very sketchy. An excellent sketch though. Why? because it is concrete enough to be argued about! ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 16 Dec '98

16 Dec '98

At 10:36 15/12/98 -0500, Greg Ward wrote: >*This is not a flame* Please do not interpret it as such! > >Quoth John Skaller, on 13 December 1998: >> This proposal is intended as a detail of the general requrements >> document for distutils. >So, first question: are you proposing to throw out my requirements >document and replace it with this? Or are you suggesting >additions/modifications to mine? An addition. I though that was clear by describing it as a detail. I'm proposing that, under one of your headings, to add a subheading. OK? >Next, you speak of the "Python build process" creating "run time >accessible identification data". The first phrase gives me the >impression that you're talking about building Python itself (and its >library), which really isn't the province of the distutils suite. Yes, I AM taking about building Python itself. Sigh. It it that hard to understand: when python is built, it know the name of the compiler. I want a module: import build_compiler cc = build_compiler.cc which is the name of the compiler used to build Python, so I can invoke that same compiler to build an extension. And all the other info needed, such as where various libraries are, what -DMACRO options are needed, etc, (If this SIG produces usable code soon enough, Guido might find it useful to >rework how the standard library is built for Python 1.6. That's the >only connection I see between the two.) Then you are missing the link: the build processes know how to build C. I want the build process to put that information into a standard module. So I can get it out, and call up the compiler to make a C extension FROM PYTHON. >I have no idea what you mean by "run time accessible identification >data". Or rather, I can think of a half-dozen things you might mean. >Please explain, or rephrase in a more conventional way. have I explained now? This is the key problem for distutils BECAUSE it REQUIRES Guido to modify the build process to create the information needed to call up the compiler from python script. At least, this is by far the most reliable way to get the information: all that autoconfig and makefile and setup stuff has the information, and it is needed to be able to build C code later as well. That is the connection. Distutils cannot work if Guido doesn't give the information. SOME of the information is already available in some form: the name of the compiler is already encoded in the version string. But it isn't enough to actually emulate the compilation and linkage processes the same way Setup builds things. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

4 3

Enough on software versions
by Greg Ward 15 Dec '98

15 Dec '98

Hi all: I'd like to declare the "Module version numbers" thread a CLOSED subject. I may spend some time in the near future writing a module to provide classes for version numbers in the anarchist/realist vein proposed by Greg Stein, and in the control-freak/idealist vein proposed by me. If you have any further comments, please keep them in private email; I'm curious to hear any opinions that have *not* already been voiced on the list. (Or you can send me a quick "me too" note if you just want your voice heard.) But let's keep that traffic off the list and move on to important things. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

>[John Skaller] >> So, it would be relatively easily to generate a standard >> python module? > >Without more context I don't know what you mean by "generate a >standard Python module". Sorry: the intent is to generate a python module containing enough information to compile any 'standard' C extension to python without makefiles and on any platform. Note: the assumption is that the extension builder is the same system as that which compiled Python. So a C compiler is available by assumption. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Observations
by Robertson, Joseph M 15 Dec '98

15 Dec '98

Hi all, I like the scriptable builds stuff. Just like Instalit of the old days. If fact I'd suggest we use a similar method. The build script tags data such as whether to check file dates, versions, etc when installing; what compression type to use, etc, etc. Question: How does your system install a working python program if the target has no python whatsoever on it? Thanks Joe Robertson 256-730-8216 jmrober1(a)ingr.com jmrobert(a)ro.com

1 0

Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

This proposal is intended as a detail of the general requrements document for distutils. The proposal is for the Python build process to create run time accessible identification data whose purpose is to facilitate run time building of a) dynamically loadable binary python module objects *.so, .dll etc) b) external (shell) application tools (.exe etc) This proposal does not fix the format of the information. --- I would like to consider a class instance, with one instance for each supported compiler/kind combination. I recommend the Python make process attempt to provide access to C, C++, and Java compilers. A binary build, such as Mark Hammonds Windows build, cannot be pre-configured for the client's system. Note that interscript contains modules which compile C/C++ code. At present, the modules have to rewritten by the client (unless they're running RedHat Linux :-) It is not clear to me if an _executable_ representation of the above requirement is appropriate. If not, at least the information required to create one should be made available. The representation must be editable by the client somehow. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

3 2

Re: [Distutils] Proposal: C/C++ compiler identification
by John Skaller 15 Dec '98

15 Dec '98

At 10:27 13/12/98 -0500, Guido van Rossum wrote: >> The proposal is for the Python build process to create run time accessible >> identification data whose purpose is to facilitate run time building of >> >> a) dynamically loadable binary python module objects *.so, .dll etc) >> b) external (shell) application tools (.exe etc) >> >> This proposal does not fix the format of the information. > >Note that (for Unix at least) all this info is already being gathered >by the configure script, and stored in the various Makefile. The info >also ends up being installed (look in >/usr/local/lib/python1.5/config/). So, it would be relatively easily to generate a standard python module? >For Windows, since the only supported compiler is VC++, it could be >hardcoded. However, there's a very serious problem on Windows with >Interscript's approach of invoking the compiler at run-time: most >users don't have this compiler. I wouldn't have call that a serious problem with the approach. It is a difficulty :-) The current implementation doesn't handle failure gracefully. But, after all, it is just python script: more work needs to be done to fetch or use pre-built binaries, and to detect _whether_ a compiler is available. Etc. >I realize that gcc is available for >free, but I think it isn't compatible with VC++. As far as I know, >VC++ is required if you want to use any of Mark Hammond's stuff (COM >and MFC). I don't know if Tcl/Tk can be used (without recompilation) >from a gcc-based app, and I don't know how easy it would be to >recompile. (Does gcc on Windows have support for Win32 APIs at all?) To the best of my knowledge, CygWin will allow a Python build on Windows: it supplies bash, ecgs version of C/C++, and some other utilities. It also supplies Tcl/Tk/Tix. And it allows standard Windows API calls as well. The advantage of the Interscript approach for Windows users is that SOME of them do have a C/C++ compiler. So instead of just ONE person supporting Windows, everyone with a compiler would be able to build and contribute binaries. In other words, I can't see how the interscript approach makes anything _worse_ for Windows users. I do agree the current mechanism is inadequate! Hopefully, people can suggest ways to improve the mechanism. Note also: JPython works on Windows! So Mark Hammonds isn't the only build of Python that works on Windows :-)) ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

2 1

Re: [Distutils] Attempted summary of version number thread
by John Skaller 15 Dec '98

15 Dec '98

>Perhaps an "API version number" belongs in the package meta-data, along >with the multitude of variables that John Skaller considers part of the >"version" (again, I think John is talking about something much bigger >than "version numbers", and I think what he's talking about can be >considered package meta-data). Yes: but isn't the point here that version numbers are 'just another piece of meta-data'? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Attempted summary of version number thread
by John Skaller 15 Dec '98

15 Dec '98

>So perhaps the following scheme might be acceptable: Version numbers >start with an API version number, then any combination of letters and >punctuation that is likely to make sense to a human reader, followed >by an optional "alpha" or "beta" indicator. The first number would >change only if a package cannot be used as a drop-in replacement for >its predecessors. Is that something the "anarchists" could agree on? Here is an idea of an alternative: every package supplies a package identification script, which is a Python script that builds some data structures and a function: isupgradeof(pkg) to discover if a package is an upgrade of another, the client executes BOTH package identification scripts, and then calls BOTH isupgradeof functions (each on the other package). A package is an upgrade of another if one of the functions says so, otherwise not. It may be worth considering adding this kind of functionality to the __init__ module of a package. Just an idea. Remember, we all know some OO, and we know it is better to represent abstractions with methods than data <grin> ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Attempted summary of version number thread
by John Skaller 15 Dec '98

15 Dec '98

At 17:46 14/12/98 +0100, Konrad Hinsen wrote: >> the anarchists, the middle-of-the-roaders, and John Skaller. The >> control freaks (me, Fred, and Konrad -- at least you two said you agree >> with me!) are in favour of prescribing *syntax* for version numbers and >> *suggesting* semantics. The *proposed* syntax and semantics that I > >Right, and it seems worth pointing out again why, or rather to what >extent, the interpretation of version numbers should be unified. > >The criterion is that it must be clear both for human users and for >installation programs >1) which of two versions is newer You are making a hidden assumption. That the development process in linear. A comparator for packages is at best a partial order: some packages may be linearly ordered, but it is far too much of a constraint on developers to require this. Therefore, 'newer' is not an appropriate requirement. A more appropriate comparator is a partial order 'is compatible with'; meaning that you can safely upgrade. It also follows that 'numbers' are an inadequate representation of upgrade compatibility. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Freeze on topic for this SIG?
by John Skaller 15 Dec '98

15 Dec '98

At 12:42 14/12/98 +1100, Mark Hammond wrote: >Im really just looking for a place to discuss enhancements to Freeze >itself. Specifically, Guido, Jack Jansen, Just and myself have had >very very preliminary discussions on a new "import architecture" for >Python itself. Why not, by private email, create a 'requirements and analysis' document: what is wrong with freeze? [I've never used it] What is desirable? How can it be done, roughly? [BTW: I know nothing of freeze but I'm very interested in a new import architecture!!!!] The post that as a discussion starter? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Attempted summary of version number thread
by Greg Ward 14 Dec '98

14 Dec '98

I'm going to see if I can't make some sense out of the version-numbering thread of the last couple of days. First: if you've already said your two cents' worth in this thread, please don't chime in again. I think everyone has made their point quite clear. (However, if I misunderstood you and misrepresent you in the summaries below, a brief correction to the list would be appreciated.) It seems to me that there are four camps out there: the control freaks, the anarchists, the middle-of-the-roaders, and John Skaller. The control freaks (me, Fred, and Konrad -- at least you two said you agree with me!) are in favour of prescribing *syntax* for version numbers and *suggesting* semantics. The *proposed* syntax and semantics that I posted are up for debate, but at least three of us like the idea of imposing a little order on how version numbers look, and giving developers some general guidelines -- which they are free to ignore! -- about what they should mean. The anarchists are Greg Stein and Marc-Andre Lemburg; these two take a "live and let live" approach to version numbers. They seem to concede that some very light syntactic requirements are necessary, but want developers to be free to define their semantics however they like. (Or, Greg and Marc-Andre just want to write code to describe the existing multitude of version number systems.) I think Andrew Kuchling is the only middle-of-the-roader, and I might be saying that just because I know Andrew's a pretty relaxed guy and not prone to radical positions like anarchy. Anyways, he posted an anti-prescriptionist statement to the list right after my syntax and semantics proposal. Maybe Andrew is really our third anarchist. Finally, John Skaller is out on his own. As I read it, John has a much wider definition of "version" than I (or the others who've posted on the topic) do. Or maybe it's more accurate to say: I started this thread to talk about *version numbers*, and John is more concerned with software versions *in general* -- all the things that can't be expressed in a simple linear progression. So: do any of you object to the above characterizations? I'd be interested to hear the opinions of anyone outside the above six, especially if you feel there're more sides to this debate than "control freaks", "anarchists", and "John Skaller". (Gee, I hope John is back on the list -- mailman disabled him because of excessive bounces, and I'd hate for him to miss this. ;-) One possible way to resolve this is to provide a couple of standard classes to implement version numbering systems. I won't say any more in detail right now, because I don't want to start another heated debate. But think about it... Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

5 5

Freeze on topic for this SIG?
by Mark Hammond 14 Dec '98

14 Dec '98

Hi all, The first paragraph of this SIG's charter states: """ The distutils sig exists to discuss the design and implementation of a suite of module distribution utilities for Python. These utilities will take the form of some standard modules, probably grouped in the 'distutils' package, for building, packaging, and installing third-party modules. This includes both modules written purely in Python and extension modules written in C/C++. """ This implies that the freeze tool would indeed be on topic (even though I fairly certain that this was not the intent!). Im really just looking for a place to discuss enhancements to Freeze itself. Specifically, Guido, Jack Jansen, Just and myself have had very very preliminary discussions on a new "import architecture" for Python itself. It is driven by freeze, but would probably become a standard part of Python if proven worthy. Im concerned it is too esoteric for the newgroup itself, but also that it really would be stretching the charter of this SIG. Please let me know if this is acceptable here (would make an interesting diversion from version numbers, tho :-), or if I should attempt a newsgroup assault! Mark.

2 1

Version numbers for module distributions
by Greg Ward 11 Dec '98

11 Dec '98

There have been a few rumbles about module version numbering on and off the list, and I thought I should bring it all out into the open. First, I think I tossed off a random opinion along these lines: Module version numbers should follow the GNU standard: X.Y.Z, where X is the major version, Y is the minor version, and Z is the patch level. Alpha and beta versions can be noted by appending "aN" or "bN", where N is any integer. This is still my opinion, although John Skaller raised a bunch of issues on the list, and Greg Stein noted a minor wrinlke in private email. I'll address those momentarily. First, let me eludicate the position above. This was (roughly) my opinion going into the Developer's Day session back at the Python Conference, and when Eric Raymond said roughly the same thing (minus the bit about alpha/beta version) based on his experience running the giant sunsite archive, I was glad to hear I wasn't the only one who likes this scheme. (And after all, the GNU folks have a hell of a lot of experience in these matters, and they have been right before!) Finally, this seems to be the version numbering system that Guido uses for Python itself, which is another good precedent. To be formal for a moment: a version number must match the regular expression (ignore whitespace in the regex): (\d+) \. (\d+) \. (\d+) ([ab](\d+))? Two possible changes I would consider open to debate: * replace [ab] with [a-z], to allow test versions beyond "beta"... although any programmer who goes up to "zeta" (or should that be omega?) really needs to look at his quality control... ;-) * make the patch number optional (and treat 1.0 and 1.0.0 as the same version number) Yes, this is very rigid. But it also makes it pretty easy to parse, split up, and compare version numbers. Making alpha/beta numbers a formal part is, IMHO, necessary, because intuitively 1.0.0a1 should be less than 1.0.0 -- but no lexical or numeric comparison will draw this conclusion! Now, there should also be semi-formal guidelines on what the different components of a version number *for modules* might mean: * a major number of 0 means the module is still being actively developed and, while it may be ready for evaluation, nobody should depend on stability of interface or implementation * a change in the major number beyond 1 means a major change in the interface, possibly not backwards compatible with previous major releases * a change in the minor number means some new functionality has been added in a way that doesn't break the interface * a change in the patch level means bugs have been fixed without adding new functionality * an alpha version (X.Y.Z.aN) is an advance preview of a future release where the interface has not yet stabilized * a beta version (X.Y.Z.aN) is an advance preview of a future release where the interface has stabilized, but known bugs exist and are still being fixed These rules have a number of implications that hadn't occurred to me until just now, when I wrote them down. In particular, whole classes of version numbers make no sense at all. Having an alpha version of something with major number 0 (0.1a3) is silly, because that leading zero already implies an unstable interface. Likewise, having an alpha version of a release with a non-zero patchlevel (1.3.4a2) is silly for the opposite reason; a change in the patchlevel means only bugs are being fixed, so there's no new interface to be unstable. This of course directly contradicts the semantics of Python version numbers; witness 1.5.2a2. But Guido's allowed to make up his own semantics; I'm just suggesting rules for Python module distributions. Oh yeah, let me stress that: *module distributions*. If you distribute a bunch of modules together, they might each have a version number, *but* there should also be a version number for the whole distribution. (And the individual modules in the distribution could just go in lockstep with the distribution version, if you want.) Now I'll try to address John Skaller's points about the multidimensionality of software versions... Quoth John Skaller (2 December 1998): > No, that isn't enough. Consider: > > MacVersion > UnixVersion > NTVersion I think that platform differences can be handled by the naming scheme for built distributions. For instance, version 1.3.2 of module 'foo' might be distributed under the following names: foo-1.3.2.tar.gz # source distribution foo-1.3.2-sparc-sun-solaris2.tar.gz # 'dumb' built distribution # for a certain Unix platform foo-1.3.2-i386-linux.rpm # 'smart' built distribution # for one popular PC platform foo-1.3.2-win32.??? # 'smart' built distribution for # some obscure PC platform This of course assumes that the same source distribution applies to all platforms. This is a lot easier with modern programming languages like Python, Java, or Perl than with crufty old beasts like C or C++. (In the immortal words of Larry Wall, "C isn't so much portable as it is ubiquitous".) But lots of C and C++ programmers go to *great* effort to ensure that their source distribution is the same for all platforms, even if they need to resort to conditional compilation, complicated configuration scripts, etc. to make it so. I think for the restricted domain of Python modules (even those written in C or C++), we can assume that people will write cross-platform source distributions (or at least have them as a goal). Next issue: > Now consider: > > FreeVersion > Commercial Version > > and then: > > BareBones version > Delux Version > Everything including the Kitchen Sink Version I see these are basically the same: different releases of essentially the same product but with varying degrees of functionality. (Where "functionality" might be defined as things like a license server for the commercial version. It could be argued that these *detract* from functionality, but let's not go there...) I'm not sure how to accomodate this one. A hack would be to give different names to the distributions: foo-1.3.2 for the standard free distribution, foo-commercial-1.3.2, foo-barebones-1.3.2, foo-delux-1.3.2 etc. for varying levels of functionality (and price!). However, again we come back to the fact that the product being distributed is *not* arbitrary software, but collections of Python modules together with their documentation and test suites. If you want different levels of functionality, you should split up your modules accordingly and make several distributions. Give away foo-base-1.3.2 to anyone who wants it, and charge extra for foo-bronze-1.3.2, foo-silver-1.3.2, and foo-gold-1.3.2, each of which contain a couple of modules that add still more functionality to the whole system. I think a lot of "version" problems can be solved by carefully organizing your modules so that blocks of functionality (barebones vs deluxe, informix vs sybase, etc.) correspond to modules or blocks of modules. Hell, this is just good solid software engineering practice, so you should be doing it that way anyways. That answer should also suffice where you have non-cross-platform source code (which is about the only reason I can think of for having platform-specific source releases). Finally, Greg Stein raised the following wrinkle in private email: > I wanted to briefly let you know that Apache modules are typically > versioned as: 0.9.3-1.3.3. The first set refers to the module itself. > The second set refers to Apache (which is currently at 1.3.3). Good point! I certainly hope we don't have to deal with this, i.e. have module developers worry whether their module will work with Python 1.5.2 or 1.6 or 1.6.1 (etc.). There is, however, the distinct possibility of API breakage with Python 2.0, which could affect potentially all extension modules. Umm, I think I'll let Guido handle this one, if he's still reading this SIG...

6 16

Package descriptions
by M.-A. Lemburg 11 Dec '98

11 Dec '98

Since there seems to be some interest in capturing meta information about packages, I suggest having a look at the (currently sleeping) Trove project initiated by E. Raymond: http://tuxedo.org/~esr/trove/ It defines quite a few different ways of identifying relationships between packages, subpackages and other resources. No need to discuss all of this again... <sigh>. -- Marc-Andre Lemburg Y2000: 385 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : ---------------------------------------------------------

1 0

My view of installation control
by John Skaller 11 Dec '98

11 Dec '98

Here is roughly what I am planning for interscript installation control. Executive summary ----------------- 1) In the source, the author provides category tags for files. 2) The client is required to bind the tags to installation points. Explanation ----------- Clients may wish to install parts of packages in different ways. For example they may wish to install doco in the same directory as each module: package/something.py package/something.html or in separate isomorphic directories: code/package/something.py doco/package/something.html There are a lot of variations on this scheme, and we must cope with ALL of them. To do this, we rely on the AUTHOR to identify the category of a file by tagging it: the file extension can be used for defaulting, as well as a few other things (such as the kind of generating process). This means that absolute filenames are not allowed in the source package. The CLIENT must be able find out what tags the author used, and provide a binding to an installation point for every file based exclusively on the tags of that file. For example: for a PY file, the installation point might be deemed /usr/local/lib/python1.5/site-packages and the [DOCO, HTML] installation point might be /home/httpd/html/python/installed-packages This mechanism is NOT enough, but it should cover a very wide variety of cases. [See below] The issues involved are: 1) how does the author decide what tags to use? 2) how does the client find out what the tags the author used are, and what they are intended to mean? 3) how does the client specify the binding of tags to installation points? --------------------------- [Below] There is a more general solution than mere tags, which I'm working on. The basic idea is that the author provides a _category_ which defines the abstract structure of the package, and the client provides a _functor_ which binds the category to the installation space. This mechanism is fully general, it can necessarily do everything. Of course, the source category has to be specified, which limits the domain to those categories for which there are effective implementations; the same comment applies to the installation functor. This mechanism has many advantages, apart from complete generality. The main one is that the installation function can be _verified_ with many representations, that is, it can be checked mnechanically that the installation mapping preserves the abstract structure of the package. With some modifications to the aboave abstraction, installation can be extended to upgrading. This mechanism automatically subsumes all notions of version control, dependency checking, etc. It allows, for example, any kind of version numbering system _provided that system is coherent_, (where coherent has a very precise meaning). ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

interscript
by John Skaller 11 Dec '98

11 Dec '98

I'm going to put the case, semi-seriously, that the 'solution' to the problem of distributing packages is to use interscript. I'll try to distinguish what works now, what is planned, and what I have no real idea about yet. For those not familiar with it: it is a source format in which 1) code 2) documentation 3) installation and building instructions can all be encoded. First, interscript allows all packages to be archived in a single text file. [Works now]. It is possible to store encoded binary if it is _really_ necesssary [planned]. The client can extract stuff from the archive with a command line tool [works now], or by embedding the package in another application [works now]. The client decides if documentation is to be generated [works now] and if so, what format it should be in [works now: plain text, flat HTML, advanced HTML, Latex2e, Lambda work now, XML planned] The client decides what language the documentation is to be produced in [planned] The client determines where the documentation is installed [works now] and where the code is installed [works now]. The package automatically builds any components written in C or C++ [works now]. It can configure itself for any platform/user/site/ preferences [planned, what works now depends on the ability to 'introspect' the information from Python itself, which is not enough] The package automatically tests everything. [Works now, but more work is required to create a proper test harness] Interscript can read source directly off the Internet [FTP and HTTP works now]. Interscript can read compressed archives [planned]. Interscript can generate interscript archives from any existing packages [sort of works :-] ----------------------------------------------------------- There is a full categorical installation/version control/dependency mechanism. [Some plans: much help needed here!!!] ----------------------------------------------------------- So there you have it: interscript already does Almost Everything (TM) a disttribution mechanism requires, and it will support almost all programming languages, not just C/C++/Java/Python, although support for these four languages is the highest priority (because they're needed for interscript itself). Interscript is written entirely in Python, although there are some C modules which do optimisations, and there is some dependency on external tools (C/C++/Java compilers, patch, diff) for full functionality. ----------------------------------------------------------- Objection: Interscript is too big. Answer: At present, everything is bundled togther in an experimental package. Because there is no standard or simple way to distribute the separate components separately. The intent is to use this experimental version to create the tools needed to unbundle itself. This is called 'bootstrapping' <grin> Objection: I'm not interesting in Literate Programming. Answer: Irrelevant. Interscript is a build/test/install environment. The mechanism includes a lot of support for literate programming, but this doesn't _have_ to be used by anyone. Objection: My packages aren't built using interscript, I don't want to convert them, it's too much work. Answer: an automated tool can package up all existing distributions. All that is required is listing the files to be packaged, with an indication if they're text or binary. Objection: not enough is working properly yet. Answer. This is a valid objection. More people working on the components would help remove this objection. See next post. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Version numbers for module distributions
by John Skaller 11 Dec '98

11 Dec '98

>My rationale for having some *suggested guidelines* is that most >developers spend several years groping in the dark trying to figure out >how to assign version numbers to their software. Experienced hackers >eventually figure it out, and really smart, far-thinking hackers just >"get it" immediately and come up with a system that will work for them >henceforth and forevermore. I think most people, though, would benefit >from a little friendly advice informed by the many decades of collective >experience that will inform the "How to Develop and Distribute Python >Modules" document that will come of all this. The problem I have with this is that the many years of collective experience only shows the common mechanisms are grossly inadequate. After all, many years of collective experience shows the world is flat. Truth is not an issue of consensus. Neither is doing version control anything to do with collective experience with old, broken, paradigms. >As long as we can a) explain the syntax, and b) write code to parse and >compare version numbers, then any proposal is valid. This one sounds >valid. I don't agree. This sounds like "I'm Ok, Your're Ok". The only 'valid' scheme is one which works. There are no such schemes, none of them come even close to working. By FAR the best scheme is the Microsoft Active Update mechanism, because it figures it all out, and installs updates on demand. In my opinion. :-)) ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Version numbers for module distributions
by John Skaller 11 Dec '98

11 Dec '98

>And I strongly agree with Marc-Andre about staying out of the semantics >of the version numbers. I'll use a format for the automated tool, but >stay away from *my* semantics. <grin> Add requirement to distutils requirements document: "Must accomodate existing schemes" >I would also argue very strongly more more flexibility in the format >schemes. For example, looking at the RedHat 5.1 distribution, I see a >few formats: > >xzip-161-2.i386.rpm >yp-tools-1.4.1-2.i386.rpm >xv-3.10a-10.i386.rpm >xrn-8.02-7.i386.rpm >xpm-3.4j-2.i386.rpm >xmorph-1996.07.12-4.i386.rpm >xboard-3.2.pl0-9.i386.rpm >x3270-3.1.1.6-2.i386.rpm >spice-2g6-7.i386.rpm >sox-11g-7.i386.rpm >rdate-0.960923-4.i386.rpm >nfs-server-2.2beta29-5.i386.rpm >nenscript-1.13++-11.i386.rpm >mailx-5.5.kw-9.i386.rpm >dhcp-2.0b1pl0-2.i386.rpm And .. they're all RPMs. :-) >You have to be MUCH more flexible in what you can take for the version >number. You cannot legislate a numbering scheme to the masses. It simply >won't work because too many people have an idea of the "right" way to >number. Agree. >As long as you can specify rules for HOW numbers will be compared, then >you're okay. For example: > >1) a version number has 1 or more numbers separate by a period or by >sequences of letters. If only periods, then these are compared >left-to-right to determine an ordering. >2) sequences of letters are part of the tuple for comparison and are >compared lexicographically >3) recognize the numeric components may have leading zeroes > >That should be about it. > >Welcome to Other Peoples' Versioning. Have a nice stay. :-) I disagree with the above rule :-) Solution: each package must provide a function to compare its own version numbers. Comments? ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] Version numbers for module distributions
by John Skaller 11 Dec '98

11 Dec '98

At 11:14 10/12/98 -0500, Greg Ward wrote: >Yes, this is very rigid. But it also makes it pretty easy to parse, >split up, and compare version numbers. Making alpha/beta numbers a >formal part is, IMHO, necessary, because intuitively 1.0.0a1 should be >less than 1.0.0 -- but no lexical or numeric comparison will draw this >conclusion! It depends on your personal 'intuition'. I could say that an alpha version of a product is a _different_ product, in other words, the lexical ordering is: alpha/beta/production tag product name major version minor version subminor version In other words, if I am running 1.2.3 production, the latest release will NEVER be 1.2.4 alpha. That is a completely different product family! I don't run any alphas! But you are entitled to a different policy! >Now I'll try to address John Skaller's points about the >multidimensionality of software versions... > >Quoth John Skaller (2 December 1998): >> No, that isn't enough. Consider: >> >> MacVersion >> UnixVersion >> NTVersion > >I think that platform differences can be handled by the naming scheme >for built distributions. >I think for the restricted >domain of Python modules (even those written in C or C++), we can assume >that people will write cross-platform source distributions (or at least >have them as a goal). They can't. Just compare a JPython vs CPython version of something that also contains C/Java code. Smly: compare a MAC version of something whose Unix version assumes a command line. >Next issue: > >> Now consider: >> FreeVersion >> Commercial Version >> and then: >> >> BareBones version >> Delux Version >> Everything including the Kitchen Sink Version > >I see these are basically the same: different releases of essentially >the same product but with varying degrees of functionality. >I'm not sure how to accomodate this one. I have the idea for a general mechanism that will accomodate ALL the various 'dimensions'. I haven't implemented it yet, however. But I have made a crude start. I think one needs to assume arbitrary tags on components: some of which are standardised. For example: name: interscript interface: 1 implementation: 3.2 platform supported: Mac, NT, Unix python required: C1.5.1 included components: basic stability: beta languages: English, Spanish requires: name: mxTools interface: 1 implementation: >2 included components: delux >I think a lot of "version" problems can be solved by carefully >organizing your modules so that blocks of functionality (barebones vs >deluxe, informix vs sybase, etc.) correspond to modules or blocks of >modules. Hell, this is just good solid software engineering practice, >so you should be doing it that way anyways. You are right. But most filing systems don't have the require functionality: they're hieratchical, which is simply NOT good enough. This is also why version numbers in sequence like 1.2.3 are NOT enough. Heirarchies are not enough. More general categories are required. Interscript solves this problem by _generating_ code. This is the only way to cope with multi-factorial installation issues. This is also what 'autoconf' et-al do. BTW: a decision on _anything_, even with limitations, is better than no standard! Because there will always be limitations. But I think it is good practice to see how far we can push things, before backtracking to something we can agree on and implement, and which both clients and suppliers will find useful. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils] DistUtils requirements
by John Skaller 11 Dec '98

11 Dec '98

At 10:22 10/12/98 -0500, Fred L. Drake wrote: > >Konrad Hinsen writes: > > - Package installation via FTP/HTTP from a server. That may seem like > > a minor convenience, but once you realize how many system administrators > > don't know how to use FTP, it gains importance... > > This, in particular, seems to indicate that an additional script is >installed along with Python. > What's been discussed so far is how to build/install an unpacked >distribution. I think this indicates a 'design bug'. It shouldn't be necessary to unpack anything. Python can read data straight out of ZIP/Tar files. I mean: it can't at the moment, but it is on the TODO list. This requires some support from Python itself, but I feel Guido will be wll disposed to being able to access a package which is packed in an archive. [I.e. 'import' directly from the archive, _without_ unpacking it] This is also on the Interscript TODO list: interscript can already read files by FTP, HTTP, as well as from the local file system, but it cannot read files from archives. But the modules required to do this seem to be part of the standard distribution already. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

sample version number code
by Greg Stein 10 Dec '98

10 Dec '98

Here is some sample code for version numbering, a-la-GJS. # # version numbering test code # samples = [ '1.5.1', '1.5.2b2', '161', '3.10a', '8.02', '3.4j', '1996.07.12', '3.2.pl0', '3.1.1.6', '2g6', '11g', '0.960923', '2.2beta29', '1.13++', '5.5.kw', '2.0b1pl0', ] import re r = re.compile('([0-9]+|[a-z]+|\.)') def removeall(l, v): while 1: try: l.remove(v) except ValueError: return def vsn(s): comps = r.split(s) removeall(comps, '.') removeall(comps, '') for i in range(len(comps)): try: comps[i] = int(comps[i]) except ValueError: pass return tuple(comps) for s in samples: print vsn(s) -- Greg Stein, http://www.lyra.org/

1 0

Re: DistUtils requirements
by Konrad Hinsen 10 Dec '98

10 Dec '98

> covered in the requirements list, under "meta-data". The only thing I > had in mind there, though, is that the distutils would have some way of > representing and comparing version numbers -- largely a passive role. > RPM-style upgrade/downgrade support was not in mind, but it's a > possibility. Judging from how I use RPM, updates could be more frequent than installation! > > - Package installation via FTP/HTTP from a server. That may seem like > > a minor convenience, but once you realize how many system administrators > > don't know how to use FTP, it gains importance... > > I'm not entirely sure what you mean here. One possible interpretation > is the following scenario: At the simplest level it would be like RPM, i.e. you could specify an URL instead of a package filename. Then of course more intelligence could be added to help the user with finding a server that has a specific package or finding a package that solves a certain problem. But that sounds like something for DistUtils 2.0! First there must be enough DistUtil-enabled packages around before automatic searching becomes useful. > The plan is to stay away from Makefiles except for people who want and > can benefit from them (mainly developers writing collections of C > extensions under Unix). As long as you can run the compiler and linker Fine. Anyway, Makefiles and make are not essential for compiling code. But calling a compiler from a Python script may be hard enough if all you have is an IDE! > (However, I am a raving Unix bigot and delight in my ignorance of > bitty-box platforms like Windows and Macintosh. Anyone who actually Me too, and I hope I can remain ignorant about them for a long time ;-) But even for Unix, we should plan for maximum flexibility. The current Makefile.pre.in system (based on the information gathered by autoconf) is not really sufficient; for example, it doesn't work for AIX unless Makefile.pre.in is patched manually (and even then it's not flexible enough to deal with all situations!). At worst, we'd have one platform-specific module in DistUtils. > sort of configuration code. However, I think it would be useful to > provide some of the functionality that Autoconf provides, e.g. Python > classes and/or functions to check for a particular C library, or C > include file, or see if a certain token is defined, or if a certain Right. Otherwise everyone will have to start from scratch. > It certainly doesn't deal with cross-platform issues. One of the things > I love about Perl is the towering arrogance of the whole Perl community > regarding the utter and complete superiority of Unix over those "other" > so-called operating systems. Last time I checked, the semi-official Hey, maybe I should take a closer look at Perl after all... ;-) > Thanks for catching my slips in the requirements list. Anyone else > notice anything? Or have any wish-list items? Now's the time to get > 'em in... Nothing fundamental, but I just remembered one detail that many extension module writers overlook: if an extension module provides header files for use by other extension modules, they must be installed in a public place as well and the compilation system should take care of including this public place in the include directory list. There is no provision for this at all in the current Makefile.pre.in procedure. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

1 0

DistUtils requirements
by Konrad Hinsen 10 Dec '98

10 Dec '98

Hi everybody, it's nice to see this SIG starting work; about a year ago, I proposed something similar in c.l.p and there was little interest and even less willingness to invest time into such a project. And then I didn't have enough time, meaning I never did anything that could be called an implementation. However, I did have a rather clear idea of what I wanted, with a few question marks. Many of the features I had in mind are already listed on Greg's requirements list, so I'll restrict myself to what's missing. The model I wanted to imitate was RPM, which offers three features that are not on the current requirements list: - Version control, i.e. the possibility to update installed packages easily with some sanity checks (no "downgrade" unless explicitly requested) - Dependence checking. This is particularly important for fine-grained packages (i.e. many small interdependent packages that are maintained independently), as it makes sure that everything works well together. - Package installation via FTP/HTTP from a server. That may seem like a minor convenience, but once you realize how many system administrators don't know how to use FTP, it gains importance... And now for the question marks: - Compilation on non-Unix platforms. I couldn't find out whether this is doable at all in an automatic way on all platforms; Windows compilers in particular seem to be oriented towards interactive code development rather than batch-mode compilation. - Dependence checking and configuration for non-Python libraries. For example, finding out whether Tk is installed and how to link with it. Does Perl packaging deal with any of these points, and if yes, how? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

2 1

[Distutils]New stuff on SIG web site
by Greg Ward 07 Dec '98

07 Dec '98

Hi all -- I've started writing up some detailed requirements/design documentation for the distutils suite. Most of this builds on what was covered at the Developer's Day session back in November, so those of you who were there should read it to make sure I'm not interpolating *too* much extra nonsense. I am definitely interpolating some things, though: there's a lot more detail in these documents than in the Developer's Day summary I wrote up. Anyways, just visit the SIG's web home at http://www.python.org/sigs/distutils-sig/distutils.html and read the "Requirements", "Common tasks", and "Proposed interface" documents. Please post substantive comments to the list; typos and such can just be mailed straight to me. Incidentally, posting these documents could be interpreted as a snub to those of you who have been suggesting your own extension building systems. Au contraire! Rather, this is meant to be a prelude to diving into those other systems; after all, how can we evaluate their suitability unless we have an explicit list of requirements? And how can we evaluate their usability unless we can compare them to an interface that, while theoretical, was agreed upon as a "good thing" by (apparently) all the people in the Developer's Day session? If an existing system can be found that meets (or can be made to meet) all the suitability and usability requirements, then our job is made a lot easier. But we must have objective grounds for evaluating those requirements, and what I've done is a stab in that direction. FWIW, I've also tweaked and renamed the "Introduction" page, and converted the SIG charter to HTML. Woo-hoo. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

[Distutils-sig] Compilers module
by John Skaller 04 Dec '98

04 Dec '98

Well, for those who haven't looked at interscript, here is the compilers module for comment. These modules work with gcc on my box. The idea is to _require_ the client implement these modules (providing some common implementation such as this one). That way, there is a standard _interface_ for compiling external tools and python modules, which can be used by author build software in a platform independent way. ------------ compilers.py --------------------------------------- #line 13 "compilers.ipk" import os import sys import string import interscript.compilers.cconfig class python_module: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,modname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) dll = modname +'.so' cc ='gcc -shared -Xlinker -export-dynamic ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + dll print 'system',repr(arg) os.system(arg) return dll class application: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,appname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) cc ='gcc ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + appname print 'system',repr(arg) os.system(arg) return appname ------------ cconfig.py --------- #line 188 "compilers.ipk" import os import sys import string class config: def __init__(self,**kwds): self.libdirs = [] self.incdirs = [] self.libs = [] self.macros = {} self.switches = {} self.extra = '' self.append_dict(kwds) def copy(self): c = config() c.libdirs = self.libdirs[:] c.incdirs = self.incdirs[:] c.libs = self.libs[:] c.macros = self.macros.copy() c.switches = self.switches.copy() c.extra = self.extra return c def append_kwds(self,**kwds): self.append_dict(kwds) def append_dict(self,kwds): if kwds.has_key('libdirs'): self.libdirs[-1:-1]=kwds['libdirs'] if kwds.has_key('incdirs'): self.incdirs[-1:-1]=kwds['incdirs'] if kwds.has_key('libs'): self.libs[-1:-1]=kwds['libs'] if kwds.has_key('extra'): self.extra = self.extra + ' ' + kwds['extra'] if kwds.has_key('macros'): macros = kwds['macros'] for macro in macros: self.macros[macro] = macros[macro] if kwds.has_key('switches'): switches = kwds['switches'] for switch in switches: self.switches[switch] = switches[switch] def __str__(self): s = self.extra for x in self.libdirs: s = s + ' -L' + x for x in self.incdirs : s = s + ' -I' + x for x in self.libs: s = s + ' -l' + x for x in self.macros.keys(): s = s + ' -D' + x if self.macros[x]: s = s + '=' + self.macros[x] for x in self.switches.keys(): s = s + ' -' + x + self.switches[x] return s ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils-sig] distutils charter and interscript
by John Skaller 02 Dec '98

02 Dec '98

At 11:43 1/12/98 -0500, Greg Ward wrote: >On Tue, Dec 01, 1998 at 09:22:15AM +1000, John Skaller wrote: >> Interscript is designed to do all this. Except, it has a much wider >> scope: it isn't limited to building Python, and it includes testing >> and documentation. (And a few other things -- some of which are implemented >> already, and some of which are not [such as version control]) >> >> There's a generic description of requirements in the documentation at >> >> http://www.triode.net.au/~skaller/interscript > >Cool! I'm looking through the Interscript docs now. I'm going to grab Macr-Andre's mxDistUtils just as soon as I can; to see what can be integrated. {I can't seem to get the URL at the moment} >I have long >thought that Python and Perl would both be eminently suitable for >literate programming because of their nifty embedded documentation >features. I've never really figured out how you would resolve the >conflict between the various target audiences implicit in the >conventional ways those embedded documentation standards are used. I haven't really figured it out either. I don't think what interscript does now is a solution, so much as a toolkit, with a couple of tools that seem useful. I guess the idea is to add more, and _then_ see how to integrate them. One of the key integrating tools, a parser, is sitting there, unused so far. >For >instance, in the Perl world, pod documentation is generally targeted at >the user of the module, and the better pods provide examples and plenty >of explanatory text in addition to the nuts and bolts of "here are the >subroutines/methods provided, and here are the parameters they take". Yeah. There's a little program I have somewhere that typesets the whole perl module library. I used that to test the POD features of the Perl tangler. Hmmm. It isn't clear I put it in the distribution, since it has the location of the library (on my machine) hard coded into it. >My impression of the use of docstrings in the Python world is that >because they wind up in the runtime VM code, people tend to make them a >lot terser, and only give nuts 'n bolts descriptions of modules, >classes, and subroutines. Thus building a useful document for most >modules simply by gluing docstrings together would be a dubious >prospect. But still, Python docstrings are targeted at the module's >users. Some doc strings are available at run time. This is important. They're a kind of introspection utility. I'm considering using them as links to their interscript web documentation. >The third target audience, and a much smaller one, is the people who >really want to understand the implementation. It has always been my >impression that this was the goal of literate programming: to provide >explanations of the data structures and algorithms embodied in the code >as a high-tech replacement for poorly-maintained or non-existent >comments. The "complete documentation" approach of POD, or the >"barebones nuts 'n bolts documentation" of Python docstrings both seem >at odds with this. My idea is that documentation at all the 'levels' has to be linked. For example, I like the idea that regressions tests belong in the tutorial. That way the tutorial has lots of examples, and the documentation describing the client interface can be 'verified' reasonably easily, since the description is lexically close to examples that test it. Similarly, there must be 'links' between requirements and design documents, and actual implementation etc etc. >Anyways, this is way off topic. How can your comments about literate programming be considered off topic in a discussion of literate programming? <grin> >I've always been intrigued by the idea >of literate programming, but never really got much past poking around >the TeX source and looking (briefly) at CWeb once. I had heard of >Interscript from several of the real Python gurus (who I happen to work >with), but nobody mentioned that it includes an extension building >system! It doesn't. Yet. It includes a _plan_ for one. At least, a notice of intent :-) >> Almost done: interscript creates a source tree in the doco. >> It doesn't yet provide the ability to 'tar' up those files, >> but that is fairly trivial. > >I assume this is part of tangling: extract source code to make source >files, and then you know what goes into a source distribution. From the interscript point of view, generated source code is a transient output, not source. The sources are the _interscript_ sources (called 'original sources'). But it depends on what you are building for who, when, etc etc. That is, it seems simple to package up files for distribution, but more complex to decide which ones, and _tell_ interscript. I plan to use categories for this. That's one reason the 'felix' stuff is in the workspace. ['Felix' is named after a certain cat who had a black bag of tricks.] >Of >course, documentation and test suites also belong in the source >distributions, so I guess weaving comes into it as well. Hmmm... In the end, the distinction between weavers and tanglers isn't justified. That is, the distinction between documentation and code isn't justified. One is just a more 'formal' version of the other .. or something :-) >> > install - install a built library on the local machine >> >> This is MUCH harder. >I don't think installation has to be that hard. If you go back and look >at the summary of the "Extension Building" Developer's Day session, >you'll find a bit about the "blib" directory approach, which I want to >steal from Perl's MakeMaker. .. OK. Remember, though, that I have a slightly different problem: I have to cater for 'installation' of systems written in Java, C++, and MyFavouriteLanguage <TM>, on machines with one user, networks of workstations in a research lab, commercial production houses, complience testing laboritories .... I firmly believe that a specialised Python installation mechanism is best designed as a 'subclass' <there, I can say it too :-> of a more general one. Else you get too many non-orthogonal quirks. >My idea for the distutils is to blatantly rip off as many of these good >ideas as possible, while making them more cross-platform (ie. no >requirement for Makefiles) and a bit more object-oriented. However, >most of the ideas carry over quite cleanly from Perl to Python. I guess what I want is a toolkit for you to build your installation model _with_. That is, I want to support multiple models .. in a consistent way. >> > gen_make - generate a Makefile to do some of the above tasks >> > (mainly 'build', for developer convenience and efficiency) >> >> Please don't! Please generate PYTHON script if it is necessary. >> Python runs on all platforms, and it has -- or should have -- >> access to all the configuration information. > >Yech! I can't imagine why you'd want to generate a Python script -- >Python is such a dynamic, module-oriented, introspective language that >it seems crazy to have to generate Python code. Just write enough >modules, and make them flexible and well-documented, so that every >needed build/package/install task is trivial. Should be doable. Hang on! It is clear, that the best way to generate python script, is to have a library available so the generated script is small. But you still need to generate the script that ties it all togther. In other words, I agree with you. But it still entails generating script: a library cannot 'do' anything unless something call it :-) >Please note: ***NO ONE WILL HAVE TO USE MAKE FOR ANY OF THIS*** The >reason I propose a 'gen_make' option is largely for the convenience of >developers writing collections of C extensions under Unix. The point of the interscript 'compilers' module is that often you don't need 'make' for compiling C extensions. Interscript contains a couple of small optimisations written in C. If you build interscript .. they get compiled automatically. As long as you have gcc running on a Linux box configured like mine. If not, you can edit the compilers module to suit. All it does is call os.system('gcc '+flags+' '+modulename) [more or less]. Why do we need make? >However, the people who write collections of C extensions under Unix are >a small but crucial segment of the Python world. If we can convince >these people to use distutils, we win. Some way of specifying external libraries is required. Make isn't needed at all. It can all be done with os.system() calls to external tools, plus other Python features. Python is quite powerful enough for this!!!!! Why use make? >It would be of great convenience >for these folks if the distutils can generate a Makefile that does >everything (or almost everything) they need. MakeMaker can do it -- so >why can't we? We can. But why not generate python script instead of makescript! >There appears to be growing support for writing a next-generation 'make' >in Python. Great idea, but I don't think this is the place for that; if >such a beast does come into existence, I'll try again. Interscript IS that tool. Interscript already builds both Python and C. Make is not required. But more 'tools' certainly are. Such as a way to read and wrire tar and zip files. [Planned] >> For example, the compilers module I provide happens to >> use gnu gcc/g++. So it won't work on NT. Or a Solaris system >> where the client wants to use the native cc. Etc etc. > >OK, that's a problem. But as you said in your first message, we should >look more at your interface than your implementation. Do you think your >implementation could be easily extended to work with more compilers. It was design to. But it was designed by a person without the experience a whole group of users have. So it will probably need work and a lot of input and discussion to turn the prototype into a platform independent standard. >(Most importantly, it should work with the compiler and compiler flags >used to build Python. I agree. That information should be available. A distutils package to provide it would be superb. Alternatively, or as well as this, one could think about the platform, site, user, and global frames in interscript, which are designed to carry that information -- but don't. >If you don't have access to that information and >don't use it, then you can't build properly build extensions to be >dynamically loaded by Python. Yes. At the moment, the compilers module just has to be rewritten by the client, for example. It would be better if all the required information was available to make a single platform independent compilers module. There must be a way to _extend_ the build information for extensions, i.e. to add new libraries and header files to some kind of data base, with a platform independent key that allows a platform independent build script to be written.(in python) >That, incidentally, is why I think a >patch might be needed for 1.5.2 -- it would probably be a patch to the >configure/build stuff, and the addition of whatever code is needed to >make a builtin 'sys.config' module which provides access to everything >Python's configure/build process knows. The intention is that this >stuff would be standard in 1.6.) Ah, OK! I agree!! I think there is not enough information. Shall we make a list of all the information we think could be needed? >Cool! That's a neat idea -- had never occured to me. Not sure if it >belongs in the "standard Python way of doing things", though. I'm more interested in a 'standard' way of doing things that covers a wider scope than just Python. After all, Python has to interface to other things, so you need them too, whatever they are. >Do you >find that having test code intermingled with production code greatly >increases the size of things? Not so much as having the documentation in there as well. But then, human written source code is limited in size by the number of contributors and they're typing speed. :-) The point is to avoid too much repetitive code, by writing code to generate it. In Python, I think you would use this ability less than say C++, where there's a fair bit of ugly repetition (declarations in two places, for example). >I've released a couple of fair-sized >collections of Perl modules complete with test suites, and I wrote >roughly as much test code as production code. Then you probably didn't write enough. :-) I think it should be more like 2/1. For doco, more like 5/1. In other words, I expect that the total amount of production code in publication quality software should be about 10%. PS: I'm talking about 'original sources'. >I'm not sure if I'd want >to wade through the test code at the same time as the production code, This is a very important point. I do not agree. I do not disagree. That is the point! It is necessary to be able to have multiple views of systems. Ideally, they're dynamically configurable. In the current batch oriented version of interscript, this is not the case. So eventually, it will need a GUI. The 'web' output is a rough test bed/prototype for the kind of thing the client will need to configure as a view. >but I can certainly see the ease-of-access benefits: add a feature to >the production code, and immediately add a test for it (in addition to >documenting it). or .. at least, have a reminder that you have not yet added a test. I think the key has to be a flexible system: different programming languages, systems, and people ... >> > * a standard for representing and comparing version numbers >> >> I think this is very hard. It isn't enough to just use numbers. >> RCS/CVS provides a tree. With labelled cut points. AFAIK, it cannot >> handle the important thing: dependencies. > >RCS/CVS (and SCCS, for that matter) address the developer; version >numbers should address the user. I think one could make the following argument: 1. There are version controls for the developer 2. There are version controls for the client 3. They not the same. 4. They are related. and that the problem is how to relate them. Since I do not know how, I plan to use categories to express the relation .. because that allows any model to be built. If you cannot fix a specific soluton, fix an abstract one: or, 'the solution to all problems in computing is one more level of indirection' :-) >And they should be in an obvious >linear sequence, or your users will get terribly confused. No, that isn't enough. Consider: MacVersion UnixVersion NTVersion Now consider: FreeVersion Commercial Version and then: BareBones version Delux Version Everything including the Kitchen Sink Version and then there is: Informix version Sybase Version etc etc etc.... There cannot be a 'linear' solution to the actual problem. >There are a >whole bunch of possible ways to do version numbering; I'm inclined >towards the GNU style (1.2.3) with optional alpha/beta tags >(eg. "1.5.2a2"). Coincentally, this seems to be Guido's version >numbering system for Python... Yes, I plan the same kind of numbering for interscript; at least until I have something significantly better. >Thanks for picking apart my proposal -- I was getting worried that >everyone would agree with everything I proposed, and I'd be stuck with >writing all the code. ;-) No, I'll do everything I can to help. The first thing to do is look at Marc-Andre's stuff. No point wasting good code! ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

[Distutils-sig] Administrivia
by Greg Ward 01 Dec '98

01 Dec '98

Hi all -- I'm pleased to mention that after being in existence for less than a week, this SIG already has 25 subscribers! Thanks all for joining up and further convincing me that extension building really is considered painful. ;-) Next, I finally got around to making a tiny little web presence for the SIG. You'll find it in the usual place: http://www.python.org/sigs/distutils-sig/ Currently there's not much there -- if it weren't for the fact that I'm a bit of a blabbermouth in front of a keyboard (blabbertype?), it would just be links to the mailing list info page, the SIG charter, and the summary I wrote of the Developer's Day session. Still, it's a start. Your friendly neighbourhood SIG administrator... Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils-sig] Some tools...
by M.-A. Lemburg 01 Dec '98

01 Dec '98

Mark Hammond wrote: > > It hates either you or me :-) > > The requested URL /~lemburg/mxDistTools.py was not found on this > server. Sorry, the correct URL is: http://starship.skyport.net/~lemburg/mxDistTools.zip -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : ---------------------------------------------------------

1 0

[Distutils-sig] Some tools...
by M.-A. Lemburg 01 Dec '98

01 Dec '98

As promised, here are some of my tools I use for packaging my extension modules (and also my commercial stuff). http://starship.skyport.net/~lemburg/mxDistTools.py -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : ---------------------------------------------------------

1 0

[Distutils-sig] Compiler module available for comment
by John Skaller 30 Nov '98

30 Nov '98

This is to advise that I have already implemented Python modules with the ability to compile C or C++ Python dynamically loadable modules C or C++ external applications and these are incorporated in interscript in the 'compilers' subdirectory. The implementation is not important. What is important is the _interface_ provided. Please examine. I would appreciate any comments on the generality of the interface. In my opinion, these modules should be in the standard distribution, and be hooked by 'import' so that C/C++ extension developers don't have to provide _any_ makefiles or other installation crud. Just copy the C file into the right place and it gets compiled automatically, just like a .py file gets compiled to a .pyc file automatically. Obviously, this technology cannot handle linking to arbitrary libraries in a platform independent way -- but it will handle a large class of C modules of the kind 'Could be done in Python, but C is faster'. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

[Distutils-sig] (fwd) PROPOSAL: XML based Make for C++, Perl, Python, Java
by Markus Fleck 29 Nov '98

29 Nov '98

1 0

[Distutils-sig] Charter for this SIG
by Greg Ward 25 Nov '98

25 Nov '98

distutils-sig: forum to discuss module distribution utilities for Python The distutils sig exists to discuss the design and implementation of a suite of module distribution utilities for Python. These utilities will take the form of some standard modules, probably grouped in the 'distutils' package, for building, packaging, and installing third-party modules. This includes both modules written purely in Python and extension modules written in C/C++. The main deliverable will be the following core modules (in the distutils package): build - provides code for building a module, which might include copying .py files into a staging area, compiling and linking .c files, processing documentation to an installable form, etc. dist - create a source distribution bdist - create a 'built distribution' (the equivalent of a 'binary distribution', except that binaries won't necessarily be present install - install a built library on the local machine gen_make - generate a Makefile to do some of the above tasks (mainly 'build', for developer convenience and efficiency) I will tentatively put forward March 1999 as a target for a working release to run on top of Python 1.5.2, which *might* require a patch-and-rebuild-Python step (or might find ways to work with the information available under 1.5). For a complete, tested, documented suite ready to bundle with Python 1.6, let's shoot for June 1999. Other topics of interest: * encouraging module developers to write test suites by having a standard place for them in module distributions * ditto for documentation -- although this is probably the job of the doc-sig, it would be nice to tie the two together at some point * a standard for representing and comparing version numbers * social engineering in general, ie. convincing module developers to start using the system * the possible need for tweaks to the configure/build process for Python itself, and a place to hold configuration information (possibly a new built-in module, 'sys.config') * possibly rewriting the configure/build/install process for Python 1.6 -- especially useful if the distutils are bundled with 1.6! * ties to Trove -- any module distribution scheme must include a way to describe the package metadata, and there will need to be hooks between any future Trove archive for Python and this metadata * recognizing SWIG-assisted extensions in addition to "ordinary" C extensions The starting point for discussion on the sig will be the summary of the IPC7 Developer's Day session which got this whole thing rolling: http://www.foretec.com/python/workshops/1998-11/dd-ward-sum.html

1 0

[Distutils-sig] Test message
by Greg Ward 25 Nov '98

25 Nov '98

Hello to everyone out there in distutils-sig land (just me, so far). Let the fun begin! Greg

1 0