Mailman 3 - Distutils-SIG

[Distutils]New stuff on SIG web site
by Greg Ward 07 Dec '98

07 Dec '98

Hi all -- I've started writing up some detailed requirements/design documentation for the distutils suite. Most of this builds on what was covered at the Developer's Day session back in November, so those of you who were there should read it to make sure I'm not interpolating *too* much extra nonsense. I am definitely interpolating some things, though: there's a lot more detail in these documents than in the Developer's Day summary I wrote up. Anyways, just visit the SIG's web home at http://www.python.org/sigs/distutils-sig/distutils.html and read the "Requirements", "Common tasks", and "Proposed interface" documents. Please post substantive comments to the list; typos and such can just be mailed straight to me. Incidentally, posting these documents could be interpreted as a snub to those of you who have been suggesting your own extension building systems. Au contraire! Rather, this is meant to be a prelude to diving into those other systems; after all, how can we evaluate their suitability unless we have an explicit list of requirements? And how can we evaluate their usability unless we can compare them to an interface that, while theoretical, was agreed upon as a "good thing" by (apparently) all the people in the Developer's Day session? If an existing system can be found that meets (or can be made to meet) all the suitability and usability requirements, then our job is made a lot easier. But we must have objective grounds for evaluating those requirements, and what I've done is a stab in that direction. FWIW, I've also tweaked and renamed the "Introduction" page, and converted the SIG charter to HTML. Woo-hoo. Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

[Distutils-sig] Compilers module
by John Skaller 04 Dec '98

04 Dec '98

Well, for those who haven't looked at interscript, here is the compilers module for comment. These modules work with gcc on my box. The idea is to _require_ the client implement these modules (providing some common implementation such as this one). That way, there is a standard _interface_ for compiling external tools and python modules, which can be used by author build software in a platform independent way. ------------ compilers.py --------------------------------------- #line 13 "compilers.ipk" import os import sys import string import interscript.compilers.cconfig class python_module: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,modname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) dll = modname +'.so' cc ='gcc -shared -Xlinker -export-dynamic ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + dll print 'system',repr(arg) os.system(arg) return dll class application: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,appname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) cc ='gcc ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + appname print 'system',repr(arg) os.system(arg) return appname ------------ cconfig.py --------- #line 188 "compilers.ipk" import os import sys import string class config: def __init__(self,**kwds): self.libdirs = [] self.incdirs = [] self.libs = [] self.macros = {} self.switches = {} self.extra = '' self.append_dict(kwds) def copy(self): c = config() c.libdirs = self.libdirs[:] c.incdirs = self.incdirs[:] c.libs = self.libs[:] c.macros = self.macros.copy() c.switches = self.switches.copy() c.extra = self.extra return c def append_kwds(self,**kwds): self.append_dict(kwds) def append_dict(self,kwds): if kwds.has_key('libdirs'): self.libdirs[-1:-1]=kwds['libdirs'] if kwds.has_key('incdirs'): self.incdirs[-1:-1]=kwds['incdirs'] if kwds.has_key('libs'): self.libs[-1:-1]=kwds['libs'] if kwds.has_key('extra'): self.extra = self.extra + ' ' + kwds['extra'] if kwds.has_key('macros'): macros = kwds['macros'] for macro in macros: self.macros[macro] = macros[macro] if kwds.has_key('switches'): switches = kwds['switches'] for switch in switches: self.switches[switch] = switches[switch] def __str__(self): s = self.extra for x in self.libdirs: s = s + ' -L' + x for x in self.incdirs : s = s + ' -I' + x for x in self.libs: s = s + ' -l' + x for x in self.macros.keys(): s = s + ' -D' + x if self.macros[x]: s = s + '=' + self.macros[x] for x in self.switches.keys(): s = s + ' -' + x + self.switches[x] return s ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

Re: [Distutils-sig] distutils charter and interscript
by John Skaller 02 Dec '98

02 Dec '98

At 11:43 1/12/98 -0500, Greg Ward wrote: >On Tue, Dec 01, 1998 at 09:22:15AM +1000, John Skaller wrote: >> Interscript is designed to do all this. Except, it has a much wider >> scope: it isn't limited to building Python, and it includes testing >> and documentation. (And a few other things -- some of which are implemented >> already, and some of which are not [such as version control]) >> >> There's a generic description of requirements in the documentation at >> >> http://www.triode.net.au/~skaller/interscript > >Cool! I'm looking through the Interscript docs now. I'm going to grab Macr-Andre's mxDistUtils just as soon as I can; to see what can be integrated. {I can't seem to get the URL at the moment} >I have long >thought that Python and Perl would both be eminently suitable for >literate programming because of their nifty embedded documentation >features. I've never really figured out how you would resolve the >conflict between the various target audiences implicit in the >conventional ways those embedded documentation standards are used. I haven't really figured it out either. I don't think what interscript does now is a solution, so much as a toolkit, with a couple of tools that seem useful. I guess the idea is to add more, and _then_ see how to integrate them. One of the key integrating tools, a parser, is sitting there, unused so far. >For >instance, in the Perl world, pod documentation is generally targeted at >the user of the module, and the better pods provide examples and plenty >of explanatory text in addition to the nuts and bolts of "here are the >subroutines/methods provided, and here are the parameters they take". Yeah. There's a little program I have somewhere that typesets the whole perl module library. I used that to test the POD features of the Perl tangler. Hmmm. It isn't clear I put it in the distribution, since it has the location of the library (on my machine) hard coded into it. >My impression of the use of docstrings in the Python world is that >because they wind up in the runtime VM code, people tend to make them a >lot terser, and only give nuts 'n bolts descriptions of modules, >classes, and subroutines. Thus building a useful document for most >modules simply by gluing docstrings together would be a dubious >prospect. But still, Python docstrings are targeted at the module's >users. Some doc strings are available at run time. This is important. They're a kind of introspection utility. I'm considering using them as links to their interscript web documentation. >The third target audience, and a much smaller one, is the people who >really want to understand the implementation. It has always been my >impression that this was the goal of literate programming: to provide >explanations of the data structures and algorithms embodied in the code >as a high-tech replacement for poorly-maintained or non-existent >comments. The "complete documentation" approach of POD, or the >"barebones nuts 'n bolts documentation" of Python docstrings both seem >at odds with this. My idea is that documentation at all the 'levels' has to be linked. For example, I like the idea that regressions tests belong in the tutorial. That way the tutorial has lots of examples, and the documentation describing the client interface can be 'verified' reasonably easily, since the description is lexically close to examples that test it. Similarly, there must be 'links' between requirements and design documents, and actual implementation etc etc. >Anyways, this is way off topic. How can your comments about literate programming be considered off topic in a discussion of literate programming? <grin> >I've always been intrigued by the idea >of literate programming, but never really got much past poking around >the TeX source and looking (briefly) at CWeb once. I had heard of >Interscript from several of the real Python gurus (who I happen to work >with), but nobody mentioned that it includes an extension building >system! It doesn't. Yet. It includes a _plan_ for one. At least, a notice of intent :-) >> Almost done: interscript creates a source tree in the doco. >> It doesn't yet provide the ability to 'tar' up those files, >> but that is fairly trivial. > >I assume this is part of tangling: extract source code to make source >files, and then you know what goes into a source distribution. From the interscript point of view, generated source code is a transient output, not source. The sources are the _interscript_ sources (called 'original sources'). But it depends on what you are building for who, when, etc etc. That is, it seems simple to package up files for distribution, but more complex to decide which ones, and _tell_ interscript. I plan to use categories for this. That's one reason the 'felix' stuff is in the workspace. ['Felix' is named after a certain cat who had a black bag of tricks.] >Of >course, documentation and test suites also belong in the source >distributions, so I guess weaving comes into it as well. Hmmm... In the end, the distinction between weavers and tanglers isn't justified. That is, the distinction between documentation and code isn't justified. One is just a more 'formal' version of the other .. or something :-) >> > install - install a built library on the local machine >> >> This is MUCH harder. >I don't think installation has to be that hard. If you go back and look >at the summary of the "Extension Building" Developer's Day session, >you'll find a bit about the "blib" directory approach, which I want to >steal from Perl's MakeMaker. .. OK. Remember, though, that I have a slightly different problem: I have to cater for 'installation' of systems written in Java, C++, and MyFavouriteLanguage <TM>, on machines with one user, networks of workstations in a research lab, commercial production houses, complience testing laboritories .... I firmly believe that a specialised Python installation mechanism is best designed as a 'subclass' <there, I can say it too :-> of a more general one. Else you get too many non-orthogonal quirks. >My idea for the distutils is to blatantly rip off as many of these good >ideas as possible, while making them more cross-platform (ie. no >requirement for Makefiles) and a bit more object-oriented. However, >most of the ideas carry over quite cleanly from Perl to Python. I guess what I want is a toolkit for you to build your installation model _with_. That is, I want to support multiple models .. in a consistent way. >> > gen_make - generate a Makefile to do some of the above tasks >> > (mainly 'build', for developer convenience and efficiency) >> >> Please don't! Please generate PYTHON script if it is necessary. >> Python runs on all platforms, and it has -- or should have -- >> access to all the configuration information. > >Yech! I can't imagine why you'd want to generate a Python script -- >Python is such a dynamic, module-oriented, introspective language that >it seems crazy to have to generate Python code. Just write enough >modules, and make them flexible and well-documented, so that every >needed build/package/install task is trivial. Should be doable. Hang on! It is clear, that the best way to generate python script, is to have a library available so the generated script is small. But you still need to generate the script that ties it all togther. In other words, I agree with you. But it still entails generating script: a library cannot 'do' anything unless something call it :-) >Please note: ***NO ONE WILL HAVE TO USE MAKE FOR ANY OF THIS*** The >reason I propose a 'gen_make' option is largely for the convenience of >developers writing collections of C extensions under Unix. The point of the interscript 'compilers' module is that often you don't need 'make' for compiling C extensions. Interscript contains a couple of small optimisations written in C. If you build interscript .. they get compiled automatically. As long as you have gcc running on a Linux box configured like mine. If not, you can edit the compilers module to suit. All it does is call os.system('gcc '+flags+' '+modulename) [more or less]. Why do we need make? >However, the people who write collections of C extensions under Unix are >a small but crucial segment of the Python world. If we can convince >these people to use distutils, we win. Some way of specifying external libraries is required. Make isn't needed at all. It can all be done with os.system() calls to external tools, plus other Python features. Python is quite powerful enough for this!!!!! Why use make? >It would be of great convenience >for these folks if the distutils can generate a Makefile that does >everything (or almost everything) they need. MakeMaker can do it -- so >why can't we? We can. But why not generate python script instead of makescript! >There appears to be growing support for writing a next-generation 'make' >in Python. Great idea, but I don't think this is the place for that; if >such a beast does come into existence, I'll try again. Interscript IS that tool. Interscript already builds both Python and C. Make is not required. But more 'tools' certainly are. Such as a way to read and wrire tar and zip files. [Planned] >> For example, the compilers module I provide happens to >> use gnu gcc/g++. So it won't work on NT. Or a Solaris system >> where the client wants to use the native cc. Etc etc. > >OK, that's a problem. But as you said in your first message, we should >look more at your interface than your implementation. Do you think your >implementation could be easily extended to work with more compilers. It was design to. But it was designed by a person without the experience a whole group of users have. So it will probably need work and a lot of input and discussion to turn the prototype into a platform independent standard. >(Most importantly, it should work with the compiler and compiler flags >used to build Python. I agree. That information should be available. A distutils package to provide it would be superb. Alternatively, or as well as this, one could think about the platform, site, user, and global frames in interscript, which are designed to carry that information -- but don't. >If you don't have access to that information and >don't use it, then you can't build properly build extensions to be >dynamically loaded by Python. Yes. At the moment, the compilers module just has to be rewritten by the client, for example. It would be better if all the required information was available to make a single platform independent compilers module. There must be a way to _extend_ the build information for extensions, i.e. to add new libraries and header files to some kind of data base, with a platform independent key that allows a platform independent build script to be written.(in python) >That, incidentally, is why I think a >patch might be needed for 1.5.2 -- it would probably be a patch to the >configure/build stuff, and the addition of whatever code is needed to >make a builtin 'sys.config' module which provides access to everything >Python's configure/build process knows. The intention is that this >stuff would be standard in 1.6.) Ah, OK! I agree!! I think there is not enough information. Shall we make a list of all the information we think could be needed? >Cool! That's a neat idea -- had never occured to me. Not sure if it >belongs in the "standard Python way of doing things", though. I'm more interested in a 'standard' way of doing things that covers a wider scope than just Python. After all, Python has to interface to other things, so you need them too, whatever they are. >Do you >find that having test code intermingled with production code greatly >increases the size of things? Not so much as having the documentation in there as well. But then, human written source code is limited in size by the number of contributors and they're typing speed. :-) The point is to avoid too much repetitive code, by writing code to generate it. In Python, I think you would use this ability less than say C++, where there's a fair bit of ugly repetition (declarations in two places, for example). >I've released a couple of fair-sized >collections of Perl modules complete with test suites, and I wrote >roughly as much test code as production code. Then you probably didn't write enough. :-) I think it should be more like 2/1. For doco, more like 5/1. In other words, I expect that the total amount of production code in publication quality software should be about 10%. PS: I'm talking about 'original sources'. >I'm not sure if I'd want >to wade through the test code at the same time as the production code, This is a very important point. I do not agree. I do not disagree. That is the point! It is necessary to be able to have multiple views of systems. Ideally, they're dynamically configurable. In the current batch oriented version of interscript, this is not the case. So eventually, it will need a GUI. The 'web' output is a rough test bed/prototype for the kind of thing the client will need to configure as a view. >but I can certainly see the ease-of-access benefits: add a feature to >the production code, and immediately add a test for it (in addition to >documenting it). or .. at least, have a reminder that you have not yet added a test. I think the key has to be a flexible system: different programming languages, systems, and people ... >> > * a standard for representing and comparing version numbers >> >> I think this is very hard. It isn't enough to just use numbers. >> RCS/CVS provides a tree. With labelled cut points. AFAIK, it cannot >> handle the important thing: dependencies. > >RCS/CVS (and SCCS, for that matter) address the developer; version >numbers should address the user. I think one could make the following argument: 1. There are version controls for the developer 2. There are version controls for the client 3. They not the same. 4. They are related. and that the problem is how to relate them. Since I do not know how, I plan to use categories to express the relation .. because that allows any model to be built. If you cannot fix a specific soluton, fix an abstract one: or, 'the solution to all problems in computing is one more level of indirection' :-) >And they should be in an obvious >linear sequence, or your users will get terribly confused. No, that isn't enough. Consider: MacVersion UnixVersion NTVersion Now consider: FreeVersion Commercial Version and then: BareBones version Delux Version Everything including the Kitchen Sink Version and then there is: Informix version Sybase Version etc etc etc.... There cannot be a 'linear' solution to the actual problem. >There are a >whole bunch of possible ways to do version numbering; I'm inclined >towards the GNU style (1.2.3) with optional alpha/beta tags >(eg. "1.5.2a2"). Coincentally, this seems to be Guido's version >numbering system for Python... Yes, I plan the same kind of numbering for interscript; at least until I have something significantly better. >Thanks for picking apart my proposal -- I was getting worried that >everyone would agree with everything I proposed, and I'd be stuck with >writing all the code. ;-) No, I'll do everything I can to help. The first thing to do is look at Marc-Andre's stuff. No point wasting good code! ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

[Distutils-sig] Administrivia
by Greg Ward 01 Dec '98

01 Dec '98

Hi all -- I'm pleased to mention that after being in existence for less than a week, this SIG already has 25 subscribers! Thanks all for joining up and further convincing me that extension building really is considered painful. ;-) Next, I finally got around to making a tiny little web presence for the SIG. You'll find it in the usual place: http://www.python.org/sigs/distutils-sig/ Currently there's not much there -- if it weren't for the fact that I'm a bit of a blabbermouth in front of a keyboard (blabbertype?), it would just be links to the mailing list info page, the SIG charter, and the summary I wrote of the Developer's Day session. Still, it's a start. Your friendly neighbourhood SIG administrator... Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

1 0

Re: [Distutils-sig] Some tools...
by M.-A. Lemburg 01 Dec '98

01 Dec '98

Mark Hammond wrote: > > It hates either you or me :-) > > The requested URL /~lemburg/mxDistTools.py was not found on this > server. Sorry, the correct URL is: http://starship.skyport.net/~lemburg/mxDistTools.zip -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : ---------------------------------------------------------

1 0

[Distutils-sig] Some tools...
by M.-A. Lemburg 01 Dec '98

01 Dec '98

As promised, here are some of my tools I use for packaging my extension modules (and also my commercial stuff). http://starship.skyport.net/~lemburg/mxDistTools.py -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : ---------------------------------------------------------

1 0

[Distutils-sig] Compiler module available for comment
by John Skaller 30 Nov '98

30 Nov '98

This is to advise that I have already implemented Python modules with the ability to compile C or C++ Python dynamically loadable modules C or C++ external applications and these are incorporated in interscript in the 'compilers' subdirectory. The implementation is not important. What is important is the _interface_ provided. Please examine. I would appreciate any comments on the generality of the interface. In my opinion, these modules should be in the standard distribution, and be hooked by 'import' so that C/C++ extension developers don't have to provide _any_ makefiles or other installation crud. Just copy the C file into the right place and it gets compiled automatically, just like a .py file gets compiled to a .pyc file automatically. Obviously, this technology cannot handle linking to arbitrary libraries in a platform independent way -- but it will handle a large class of C modules of the kind 'Could be done in Python, but C is faster'. ------------------------------------------------------- John Skaller email: skaller(a)maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia

1 0

[Distutils-sig] (fwd) PROPOSAL: XML based Make for C++, Perl, Python, Java
by Markus Fleck 29 Nov '98

29 Nov '98

1 0

[Distutils-sig] Charter for this SIG
by Greg Ward 25 Nov '98

25 Nov '98

distutils-sig: forum to discuss module distribution utilities for Python The distutils sig exists to discuss the design and implementation of a suite of module distribution utilities for Python. These utilities will take the form of some standard modules, probably grouped in the 'distutils' package, for building, packaging, and installing third-party modules. This includes both modules written purely in Python and extension modules written in C/C++. The main deliverable will be the following core modules (in the distutils package): build - provides code for building a module, which might include copying .py files into a staging area, compiling and linking .c files, processing documentation to an installable form, etc. dist - create a source distribution bdist - create a 'built distribution' (the equivalent of a 'binary distribution', except that binaries won't necessarily be present install - install a built library on the local machine gen_make - generate a Makefile to do some of the above tasks (mainly 'build', for developer convenience and efficiency) I will tentatively put forward March 1999 as a target for a working release to run on top of Python 1.5.2, which *might* require a patch-and-rebuild-Python step (or might find ways to work with the information available under 1.5). For a complete, tested, documented suite ready to bundle with Python 1.6, let's shoot for June 1999. Other topics of interest: * encouraging module developers to write test suites by having a standard place for them in module distributions * ditto for documentation -- although this is probably the job of the doc-sig, it would be nice to tie the two together at some point * a standard for representing and comparing version numbers * social engineering in general, ie. convincing module developers to start using the system * the possible need for tweaks to the configure/build process for Python itself, and a place to hold configuration information (possibly a new built-in module, 'sys.config') * possibly rewriting the configure/build/install process for Python 1.6 -- especially useful if the distutils are bundled with 1.6! * ties to Trove -- any module distribution scheme must include a way to describe the package metadata, and there will need to be hooks between any future Trove archive for Python and this metadata * recognizing SWIG-assisted extensions in addition to "ordinary" C extensions The starting point for discussion on the sig will be the summary of the IPC7 Developer's Day session which got this whole thing rolling: http://www.foretec.com/python/workshops/1998-11/dd-ward-sum.html

1 0

[Distutils-sig] Test message
by Greg Ward 25 Nov '98

25 Nov '98

Hello to everyone out there in distutils-sig land (just me, so far). Let the fun begin! Greg

1 0