From mal@lemburg.com Tue Dec 1 09:36:56 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 01 Dec 1998 10:36:56 +0100 Subject: [Distutils-sig] Some tools... Message-ID: <3663B8B8.20C0CC74@lemburg.com> As promised, here are some of my tools I use for packaging my extension modules (and also my commercial stuff). http://starship.skyport.net/~lemburg/mxDistTools.py -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gward@cnri.reston.va.us Tue Dec 1 16:43:27 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Dec 1998 11:43:27 -0500 Subject: [Distutils-sig] distutils charter and interscript In-Reply-To: <1.5.4.32.19981130232215.00f4a578@triode.net.au>; from John Skaller on Tue, Dec 01, 1998 at 09:22:15AM +1000 References: <1.5.4.32.19981130232215.00f4a578@triode.net.au> Message-ID: <19981201114327.A16182@thrak.cnri.reston.va.us> On Tue, Dec 01, 1998 at 09:22:15AM +1000, John Skaller wrote: > Interscript is designed to do all this. Except, it has a much wider > scope: it isn't limited to building Python, and it includes testing > and documentation. (And a few other things -- some of which are implemented > already, and some of which are not [such as version control]) > > There's a generic description of requirements in the documentation at > > http://www.triode.net.au/~skaller/interscript Cool! I'm looking through the Interscript docs now. I have long thought that Python and Perl would both be eminently suitable for literate programming because of their nifty embedded documentation features. I've never really figured out how you would resolve the conflict between the various target audiences implicit in the conventional ways those embedded documentation standards are used. For instance, in the Perl world, pod documentation is generally targeted at the user of the module, and the better pods provide examples and plenty of explanatory text in addition to the nuts and bolts of "here are the subroutines/methods provided, and here are the parameters they take". My impression of the use of docstrings in the Python world is that because they wind up in the runtime VM code, people tend to make them a lot terser, and only give nuts 'n bolts descriptions of modules, classes, and subroutines. Thus building a useful document for most modules simply by gluing docstrings together would be a dubious prospect. But still, Python docstrings are targeted at the module's users. The third target audience, and a much smaller one, is the people who really want to understand the implementation. It has always been my impression that this was the goal of literate programming: to provide explanations of the data structures and algorithms embodied in the code as a high-tech replacement for poorly-maintained or non-existent comments. The "complete documentation" approach of POD, or the "barebones nuts 'n bolts documentation" of Python docstrings both seem at odds with this. Anyways, this is way off topic. I've always been intrigued by the idea of literate programming, but never really got much past poking around the TeX source and looking (briefly) at CWeb once. I had heard of Interscript from several of the real Python gurus (who I happen to work with), but nobody mentioned that it includes an extension building system! > Almost done: interscript creates a source tree in the doco. > It doesn't yet provide the ability to 'tar' up those files, > but that is fairly trivial. I assume this is part of tangling: extract source code to make source files, and then you know what goes into a source distribution. Of course, documentation and test suites also belong in the source distributions, so I guess weaving comes into it as well. Hmmm... > > install - install a built library on the local machine > > This is MUCH harder. If you read my documentation, and examine the > sections on installation control (site-frame, platform-frame, user-frame) > you will see I have made fairly advanced provision for installation control. > I'm not using any of this yet. > > My system discriminates the status of the package, and may install > it in different places depending on the status. For example, > when I download a new version of a package, I might put it into > a test directory and test it, before I install it into a more > widely accessible place. Furthermore, the install point is conditioned > by the authors evaluation: is it alpha, beta, or production software? I don't think installation has to be that hard. If you go back and look at the summary of the "Extension Building" Developer's Day session, you'll find a bit about the "blib" directory approach, which I want to steal from Perl's MakeMaker. Basically, ./blib ('build library') is a mock installation tree that looks quite a lot like a subset of /usr/local/lib/perl5/site_perl (or, in the distutils case, will look quite a lot like a subset of /usr/local/lib/python1.x). (Note that this is for varying values of "/usr/local/lib/perl5" -- Perl lets you install its library anywhere, and this information is available via the standard Config module -- "/usr/local/lib/python1.x" -- Python lets you install its library anywhere, and this information *should* be available through some standard module (which I refer to as 'sys.config').) When you build a Perl module distribution, C extensions are compiled and the .so files wind up in ./blib/arch; pure-Perl modules (.pm files) are simply copied into ./blib/lib; and documentation (POD) is converted to *roff format in ./blib/man. The advantage of this is (at least) two-fold: first, running the test suites in a "realistic" environment is trivial: just prepend ./blib/lib and ./blib/arch to Perl's library search path, and run the test programs. Second, installation is trivial: just do recursive copies of ./blib{lib,arch,man} to the appropriate places under /usr/local/lib/perl5/site_perl. (Actually, it's a bit smarter than that: it only copies files that are actually different from corresponding files in the "official" library directory.) MakeMaker also allows users to specify a different installation base than /usr/local/lib/perl5/site_perl, so non-superusers (or superusers who are just messing around) can install things to their home directory, to a temp directory, etc. My idea for the distutils is to blatantly rip off as many of these good ideas as possible, while making them more cross-platform (ie. no requirement for Makefiles) and a bit more object-oriented. However, most of the ideas carry over quite cleanly from Perl to Python. > > gen_make - generate a Makefile to do some of the above tasks > > (mainly 'build', for developer convenience and efficiency) > > Please don't! Please generate PYTHON script if it is necessary. > Python runs on all platforms, and it has -- or should have -- > access to all the configuration information. Yech! I can't imagine why you'd want to generate a Python script -- Python is such a dynamic, module-oriented, introspective language that it seems crazy to have to generate Python code. Just write enough modules, and make them flexible and well-documented, so that every needed build/package/install task is trivial. Should be doable. Please note: ***NO ONE WILL HAVE TO USE MAKE FOR ANY OF THIS*** The reason I propose a 'gen_make' option is largely for the convenience of developers writing collections of C extensions under Unix. It won't be needed for people writing single-.py-file-distributions, it won't be needed (but it might be nice) for people writing single-.c-file- distributions, and it most certainly will be not be needed for people just installing the stuff. However, the people who write collections of C extensions under Unix are a small but crucial segment of the Python world. If we can convince these people to use distutils, we win. It would be of great convenience for these folks if the distutils can generate a Makefile that does everything (or almost everything) they need. MakeMaker can do it -- so why can't we? Anyways, I completely agree with your statements about Make being unreliable, unportable, flaky, and a bear to debug. I also agree that we have something better available; that's why the whole proposal is built around something called 'setup.py'. The generation of makefiles is just an optional feature for a small but important segment of the population. There appears to be growing support for writing a next-generation 'make' in Python. Great idea, but I don't think this is the place for that; if such a beast does come into existence, then we should add a 'gen_ngmake' command to distutils, but examining time-based dependences amongst files is not really germane to most of this discussion. It's certainly something that people writing collections of C extensions have to worry about, and those of them using Unix have a solution -- just not a very satisfactory (or portable) one. > For example, the compilers module I provide happens to > use gnu gcc/g++. So it won't work on NT. Or a Solaris system > where the client wants to use the native cc. Etc etc. OK, that's a problem. But as you said in your first message, we should look more at your interface than your implementation. Do you think your implementation could be easily extended to work with more compilers. (Most importantly, it should work with the compiler and compiler flags used to build Python. If you don't have access to that information and don't use it, then you can't build properly build extensions to be dynamically loaded by Python. That, incidentally, is why I think a patch might be needed for 1.5.2 -- it would probably be a patch to the configure/build stuff, and the addition of whatever code is needed to make a builtin 'sys.config' module which provides access to everything Python's configure/build process knows. The intention is that this stuff would be standard in 1.6.) > Yes. Although interscript allows a slightly different approach: > test code is embedded directly in the source and is executed during > the build process. The test code isn't installed anywhere; it's > written to a temporary file for testing and then deleted afterwards. Cool! That's a neat idea -- had never occured to me. Not sure if it belongs in the "standard Python way of doing things", though. Do you find that having test code intermingled with production code greatly increases the size of things? I've released a couple of fair-sized collections of Perl modules complete with test suites, and I wrote roughly as much test code as production code. I'm not sure if I'd want to wade through the test code at the same time as the production code, but I can certainly see the ease-of-access benefits: add a feature to the production code, and immediately add a test for it (in addition to documenting it). > > * a standard for representing and comparing version numbers > > I think this is very hard. It isn't enough to just use numbers. > RCS/CVS provides a tree. With labelled cut points. AFAIK, it cannot > handle the important thing: dependencies. RCS/CVS (and SCCS, for that matter) address the developer; version numbers should address the user. And they should be in an obvious linear sequence, or your users will get terribly confused. There are a whole bunch of possible ways to do version numbering; I'm inclined towards the GNU style (1.2.3) with optional alpha/beta tags (eg. "1.5.2a2"). Coincentally, this seems to be Guido's version numbering system for Python... The only two other widely-known models I can think of offhand are Linux (basically GNU, but with the stable/development version wrinkle) and Perl (where version numbers can be compared as floating point numbers -- which leads to the madness of the current Perl version being 5.00502!). I don't think either of these are appropriate. OK, enough for now. I have to spend some more time digesting the Interscript docs, and then I'll try to read the rest of your email in detail. Thanks for picking apart my proposal -- I was getting worried that everyone would agree with everything I proposed, and I'd be stuck with writing all the code. ;-) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From da@skivs.ski.org Tue Dec 1 19:40:36 1998 From: da@skivs.ski.org (David Ascher) Date: Tue, 1 Dec 1998 11:40:36 -0800 (Pacific Standard Time) Subject: [Distutils-sig] distutils charter and interscript In-Reply-To: <19981201114327.A16182@thrak.cnri.reston.va.us> Message-ID: On Tue, 1 Dec 1998, Greg Ward wrote: > My impression of the use of docstrings in the Python world is that > because they wind up in the runtime VM code, people tend to make them a > lot terser, Side note -- only docstrings which are the first statements in a class or function or module are kept -- everything else is thrown out currently. --david From mal@lemburg.com Tue Dec 1 23:04:25 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 02 Dec 1998 00:04:25 +0100 Subject: [Distutils-sig] Some tools... References: <003001be1d7b$4169d3b0$0801a8c0@bobcat.skippinet.com.au> Message-ID: <366475F9.4FF0468A@lemburg.com> Mark Hammond wrote: > > It hates either you or me :-) > > The requested URL /~lemburg/mxDistTools.py was not found on this > server. Sorry, the correct URL is: http://starship.skyport.net/~lemburg/mxDistTools.zip -- Marc-Andre Lemburg Y2000: 395 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gward@cnri.reston.va.us Tue Dec 1 23:30:00 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 1 Dec 1998 18:30:00 -0500 Subject: [Distutils-sig] Administrivia Message-ID: <19981201183000.H16249@thrak.cnri.reston.va.us> Hi all -- I'm pleased to mention that after being in existence for less than a week, this SIG already has 25 subscribers! Thanks all for joining up and further convincing me that extension building really is considered painful. ;-) Next, I finally got around to making a tiny little web presence for the SIG. You'll find it in the usual place: http://www.python.org/sigs/distutils-sig/ Currently there's not much there -- if it weren't for the fact that I'm a bit of a blabbermouth in front of a keyboard (blabbertype?), it would just be links to the mailing list info page, the SIG charter, and the summary I wrote of the Developer's Day session. Still, it's a start. Your friendly neighbourhood SIG administrator... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skaller@maxtal.com.au Wed Dec 2 08:18:10 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 02 Dec 1998 18:18:10 +1000 Subject: [Distutils-sig] distutils charter and interscript Message-ID: <1.5.4.32.19981202081810.00f71814@triode.net.au> At 11:43 1/12/98 -0500, Greg Ward wrote: >On Tue, Dec 01, 1998 at 09:22:15AM +1000, John Skaller wrote: >> Interscript is designed to do all this. Except, it has a much wider >> scope: it isn't limited to building Python, and it includes testing >> and documentation. (And a few other things -- some of which are implemented >> already, and some of which are not [such as version control]) >> >> There's a generic description of requirements in the documentation at >> >> http://www.triode.net.au/~skaller/interscript > >Cool! I'm looking through the Interscript docs now. I'm going to grab Macr-Andre's mxDistUtils just as soon as I can; to see what can be integrated. {I can't seem to get the URL at the moment} >I have long >thought that Python and Perl would both be eminently suitable for >literate programming because of their nifty embedded documentation >features. I've never really figured out how you would resolve the >conflict between the various target audiences implicit in the >conventional ways those embedded documentation standards are used. I haven't really figured it out either. I don't think what interscript does now is a solution, so much as a toolkit, with a couple of tools that seem useful. I guess the idea is to add more, and _then_ see how to integrate them. One of the key integrating tools, a parser, is sitting there, unused so far. >For >instance, in the Perl world, pod documentation is generally targeted at >the user of the module, and the better pods provide examples and plenty >of explanatory text in addition to the nuts and bolts of "here are the >subroutines/methods provided, and here are the parameters they take". Yeah. There's a little program I have somewhere that typesets the whole perl module library. I used that to test the POD features of the Perl tangler. Hmmm. It isn't clear I put it in the distribution, since it has the location of the library (on my machine) hard coded into it. >My impression of the use of docstrings in the Python world is that >because they wind up in the runtime VM code, people tend to make them a >lot terser, and only give nuts 'n bolts descriptions of modules, >classes, and subroutines. Thus building a useful document for most >modules simply by gluing docstrings together would be a dubious >prospect. But still, Python docstrings are targeted at the module's >users. Some doc strings are available at run time. This is important. They're a kind of introspection utility. I'm considering using them as links to their interscript web documentation. >The third target audience, and a much smaller one, is the people who >really want to understand the implementation. It has always been my >impression that this was the goal of literate programming: to provide >explanations of the data structures and algorithms embodied in the code >as a high-tech replacement for poorly-maintained or non-existent >comments. The "complete documentation" approach of POD, or the >"barebones nuts 'n bolts documentation" of Python docstrings both seem >at odds with this. My idea is that documentation at all the 'levels' has to be linked. For example, I like the idea that regressions tests belong in the tutorial. That way the tutorial has lots of examples, and the documentation describing the client interface can be 'verified' reasonably easily, since the description is lexically close to examples that test it. Similarly, there must be 'links' between requirements and design documents, and actual implementation etc etc. >Anyways, this is way off topic. How can your comments about literate programming be considered off topic in a discussion of literate programming? >I've always been intrigued by the idea >of literate programming, but never really got much past poking around >the TeX source and looking (briefly) at CWeb once. I had heard of >Interscript from several of the real Python gurus (who I happen to work >with), but nobody mentioned that it includes an extension building >system! It doesn't. Yet. It includes a _plan_ for one. At least, a notice of intent :-) >> Almost done: interscript creates a source tree in the doco. >> It doesn't yet provide the ability to 'tar' up those files, >> but that is fairly trivial. > >I assume this is part of tangling: extract source code to make source >files, and then you know what goes into a source distribution. From the interscript point of view, generated source code is a transient output, not source. The sources are the _interscript_ sources (called 'original sources'). But it depends on what you are building for who, when, etc etc. That is, it seems simple to package up files for distribution, but more complex to decide which ones, and _tell_ interscript. I plan to use categories for this. That's one reason the 'felix' stuff is in the workspace. ['Felix' is named after a certain cat who had a black bag of tricks.] >Of >course, documentation and test suites also belong in the source >distributions, so I guess weaving comes into it as well. Hmmm... In the end, the distinction between weavers and tanglers isn't justified. That is, the distinction between documentation and code isn't justified. One is just a more 'formal' version of the other .. or something :-) >> > install - install a built library on the local machine >> >> This is MUCH harder. >I don't think installation has to be that hard. If you go back and look >at the summary of the "Extension Building" Developer's Day session, >you'll find a bit about the "blib" directory approach, which I want to >steal from Perl's MakeMaker. .. OK. Remember, though, that I have a slightly different problem: I have to cater for 'installation' of systems written in Java, C++, and MyFavouriteLanguage , on machines with one user, networks of workstations in a research lab, commercial production houses, complience testing laboritories .... I firmly believe that a specialised Python installation mechanism is best designed as a 'subclass' of a more general one. Else you get too many non-orthogonal quirks. >My idea for the distutils is to blatantly rip off as many of these good >ideas as possible, while making them more cross-platform (ie. no >requirement for Makefiles) and a bit more object-oriented. However, >most of the ideas carry over quite cleanly from Perl to Python. I guess what I want is a toolkit for you to build your installation model _with_. That is, I want to support multiple models .. in a consistent way. >> > gen_make - generate a Makefile to do some of the above tasks >> > (mainly 'build', for developer convenience and efficiency) >> >> Please don't! Please generate PYTHON script if it is necessary. >> Python runs on all platforms, and it has -- or should have -- >> access to all the configuration information. > >Yech! I can't imagine why you'd want to generate a Python script -- >Python is such a dynamic, module-oriented, introspective language that >it seems crazy to have to generate Python code. Just write enough >modules, and make them flexible and well-documented, so that every >needed build/package/install task is trivial. Should be doable. Hang on! It is clear, that the best way to generate python script, is to have a library available so the generated script is small. But you still need to generate the script that ties it all togther. In other words, I agree with you. But it still entails generating script: a library cannot 'do' anything unless something call it :-) >Please note: ***NO ONE WILL HAVE TO USE MAKE FOR ANY OF THIS*** The >reason I propose a 'gen_make' option is largely for the convenience of >developers writing collections of C extensions under Unix. The point of the interscript 'compilers' module is that often you don't need 'make' for compiling C extensions. Interscript contains a couple of small optimisations written in C. If you build interscript .. they get compiled automatically. As long as you have gcc running on a Linux box configured like mine. If not, you can edit the compilers module to suit. All it does is call os.system('gcc '+flags+' '+modulename) [more or less]. Why do we need make? >However, the people who write collections of C extensions under Unix are >a small but crucial segment of the Python world. If we can convince >these people to use distutils, we win. Some way of specifying external libraries is required. Make isn't needed at all. It can all be done with os.system() calls to external tools, plus other Python features. Python is quite powerful enough for this!!!!! Why use make? >It would be of great convenience >for these folks if the distutils can generate a Makefile that does >everything (or almost everything) they need. MakeMaker can do it -- so >why can't we? We can. But why not generate python script instead of makescript! >There appears to be growing support for writing a next-generation 'make' >in Python. Great idea, but I don't think this is the place for that; if >such a beast does come into existence, I'll try again. Interscript IS that tool. Interscript already builds both Python and C. Make is not required. But more 'tools' certainly are. Such as a way to read and wrire tar and zip files. [Planned] >> For example, the compilers module I provide happens to >> use gnu gcc/g++. So it won't work on NT. Or a Solaris system >> where the client wants to use the native cc. Etc etc. > >OK, that's a problem. But as you said in your first message, we should >look more at your interface than your implementation. Do you think your >implementation could be easily extended to work with more compilers. It was design to. But it was designed by a person without the experience a whole group of users have. So it will probably need work and a lot of input and discussion to turn the prototype into a platform independent standard. >(Most importantly, it should work with the compiler and compiler flags >used to build Python. I agree. That information should be available. A distutils package to provide it would be superb. Alternatively, or as well as this, one could think about the platform, site, user, and global frames in interscript, which are designed to carry that information -- but don't. >If you don't have access to that information and >don't use it, then you can't build properly build extensions to be >dynamically loaded by Python. Yes. At the moment, the compilers module just has to be rewritten by the client, for example. It would be better if all the required information was available to make a single platform independent compilers module. There must be a way to _extend_ the build information for extensions, i.e. to add new libraries and header files to some kind of data base, with a platform independent key that allows a platform independent build script to be written.(in python) >That, incidentally, is why I think a >patch might be needed for 1.5.2 -- it would probably be a patch to the >configure/build stuff, and the addition of whatever code is needed to >make a builtin 'sys.config' module which provides access to everything >Python's configure/build process knows. The intention is that this >stuff would be standard in 1.6.) Ah, OK! I agree!! I think there is not enough information. Shall we make a list of all the information we think could be needed? >Cool! That's a neat idea -- had never occured to me. Not sure if it >belongs in the "standard Python way of doing things", though. I'm more interested in a 'standard' way of doing things that covers a wider scope than just Python. After all, Python has to interface to other things, so you need them too, whatever they are. >Do you >find that having test code intermingled with production code greatly >increases the size of things? Not so much as having the documentation in there as well. But then, human written source code is limited in size by the number of contributors and they're typing speed. :-) The point is to avoid too much repetitive code, by writing code to generate it. In Python, I think you would use this ability less than say C++, where there's a fair bit of ugly repetition (declarations in two places, for example). >I've released a couple of fair-sized >collections of Perl modules complete with test suites, and I wrote >roughly as much test code as production code. Then you probably didn't write enough. :-) I think it should be more like 2/1. For doco, more like 5/1. In other words, I expect that the total amount of production code in publication quality software should be about 10%. PS: I'm talking about 'original sources'. >I'm not sure if I'd want >to wade through the test code at the same time as the production code, This is a very important point. I do not agree. I do not disagree. That is the point! It is necessary to be able to have multiple views of systems. Ideally, they're dynamically configurable. In the current batch oriented version of interscript, this is not the case. So eventually, it will need a GUI. The 'web' output is a rough test bed/prototype for the kind of thing the client will need to configure as a view. >but I can certainly see the ease-of-access benefits: add a feature to >the production code, and immediately add a test for it (in addition to >documenting it). or .. at least, have a reminder that you have not yet added a test. I think the key has to be a flexible system: different programming languages, systems, and people ... >> > * a standard for representing and comparing version numbers >> >> I think this is very hard. It isn't enough to just use numbers. >> RCS/CVS provides a tree. With labelled cut points. AFAIK, it cannot >> handle the important thing: dependencies. > >RCS/CVS (and SCCS, for that matter) address the developer; version >numbers should address the user. I think one could make the following argument: 1. There are version controls for the developer 2. There are version controls for the client 3. They not the same. 4. They are related. and that the problem is how to relate them. Since I do not know how, I plan to use categories to express the relation .. because that allows any model to be built. If you cannot fix a specific soluton, fix an abstract one: or, 'the solution to all problems in computing is one more level of indirection' :-) >And they should be in an obvious >linear sequence, or your users will get terribly confused. No, that isn't enough. Consider: MacVersion UnixVersion NTVersion Now consider: FreeVersion Commercial Version and then: BareBones version Delux Version Everything including the Kitchen Sink Version and then there is: Informix version Sybase Version etc etc etc.... There cannot be a 'linear' solution to the actual problem. >There are a >whole bunch of possible ways to do version numbering; I'm inclined >towards the GNU style (1.2.3) with optional alpha/beta tags >(eg. "1.5.2a2"). Coincentally, this seems to be Guido's version >numbering system for Python... Yes, I plan the same kind of numbering for interscript; at least until I have something significantly better. >Thanks for picking apart my proposal -- I was getting worried that >everyone would agree with everything I proposed, and I'd be stuck with >writing all the code. ;-) No, I'll do everything I can to help. The first thing to do is look at Marc-Andre's stuff. No point wasting good code! ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 4 14:45:45 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sat, 05 Dec 1998 00:45:45 +1000 Subject: [Distutils-sig] Compilers module Message-ID: <1.5.4.32.19981204144545.00f68fa4@triode.net.au> Well, for those who haven't looked at interscript, here is the compilers module for comment. These modules work with gcc on my box. The idea is to _require_ the client implement these modules (providing some common implementation such as this one). That way, there is a standard _interface_ for compiling external tools and python modules, which can be used by author build software in a platform independent way. ------------ compilers.py --------------------------------------- #line 13 "compilers.ipk" import os import sys import string import interscript.compilers.cconfig class python_module: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,modname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) dll = modname +'.so' cc ='gcc -shared -Xlinker -export-dynamic ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + dll print 'system',repr(arg) os.system(arg) return dll class application: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,appname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) cc ='gcc ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + appname print 'system',repr(arg) os.system(arg) return appname ------------ cconfig.py --------- #line 188 "compilers.ipk" import os import sys import string class config: def __init__(self,**kwds): self.libdirs = [] self.incdirs = [] self.libs = [] self.macros = {} self.switches = {} self.extra = '' self.append_dict(kwds) def copy(self): c = config() c.libdirs = self.libdirs[:] c.incdirs = self.incdirs[:] c.libs = self.libs[:] c.macros = self.macros.copy() c.switches = self.switches.copy() c.extra = self.extra return c def append_kwds(self,**kwds): self.append_dict(kwds) def append_dict(self,kwds): if kwds.has_key('libdirs'): self.libdirs[-1:-1]=kwds['libdirs'] if kwds.has_key('incdirs'): self.incdirs[-1:-1]=kwds['incdirs'] if kwds.has_key('libs'): self.libs[-1:-1]=kwds['libs'] if kwds.has_key('extra'): self.extra = self.extra + ' ' + kwds['extra'] if kwds.has_key('macros'): macros = kwds['macros'] for macro in macros: self.macros[macro] = macros[macro] if kwds.has_key('switches'): switches = kwds['switches'] for switch in switches: self.switches[switch] = switches[switch] def __str__(self): s = self.extra for x in self.libdirs: s = s + ' -L' + x for x in self.incdirs : s = s + ' -I' + x for x in self.libs: s = s + ' -l' + x for x in self.macros.keys(): s = s + ' -D' + x if self.macros[x]: s = s + '=' + self.macros[x] for x in self.switches.keys(): s = s + ' -' + x + self.switches[x] return s ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gward@cnri.reston.va.us Mon Dec 7 03:28:06 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 6 Dec 1998 22:28:06 -0500 Subject: [Distutils]New stuff on SIG web site Message-ID: <19981206222806.A21373@cnri.reston.va.us> Hi all -- I've started writing up some detailed requirements/design documentation for the distutils suite. Most of this builds on what was covered at the Developer's Day session back in November, so those of you who were there should read it to make sure I'm not interpolating *too* much extra nonsense. I am definitely interpolating some things, though: there's a lot more detail in these documents than in the Developer's Day summary I wrote up. Anyways, just visit the SIG's web home at http://www.python.org/sigs/distutils-sig/distutils.html and read the "Requirements", "Common tasks", and "Proposed interface" documents. Please post substantive comments to the list; typos and such can just be mailed straight to me. Incidentally, posting these documents could be interpreted as a snub to those of you who have been suggesting your own extension building systems. Au contraire! Rather, this is meant to be a prelude to diving into those other systems; after all, how can we evaluate their suitability unless we have an explicit list of requirements? And how can we evaluate their usability unless we can compare them to an interface that, while theoretical, was agreed upon as a "good thing" by (apparently) all the people in the Developer's Day session? If an existing system can be found that meets (or can be made to meet) all the suitability and usability requirements, then our job is made a lot easier. But we must have objective grounds for evaluating those requirements, and what I've done is a stab in that direction. FWIW, I've also tweaked and renamed the "Introduction" page, and converted the SIG charter to HTML. Woo-hoo. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From hinsen@cnrs-orleans.fr Thu Dec 10 14:53:03 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 10 Dec 1998 15:53:03 +0100 Subject: [Distutils] DistUtils requirements Message-ID: <199812101453.PAA25622@dirac.cnrs-orleans.fr> Hi everybody, it's nice to see this SIG starting work; about a year ago, I proposed something similar in c.l.p and there was little interest and even less willingness to invest time into such a project. And then I didn't have enough time, meaning I never did anything that could be called an implementation. However, I did have a rather clear idea of what I wanted, with a few question marks. Many of the features I had in mind are already listed on Greg's requirements list, so I'll restrict myself to what's missing. The model I wanted to imitate was RPM, which offers three features that are not on the current requirements list: - Version control, i.e. the possibility to update installed packages easily with some sanity checks (no "downgrade" unless explicitly requested) - Dependence checking. This is particularly important for fine-grained packages (i.e. many small interdependent packages that are maintained independently), as it makes sure that everything works well together. - Package installation via FTP/HTTP from a server. That may seem like a minor convenience, but once you realize how many system administrators don't know how to use FTP, it gains importance... And now for the question marks: - Compilation on non-Unix platforms. I couldn't find out whether this is doable at all in an automatic way on all platforms; Windows compilers in particular seem to be oriented towards interactive code development rather than batch-mode compilation. - Dependence checking and configuration for non-Python libraries. For example, finding out whether Tk is installed and how to link with it. Does Perl packaging deal with any of these points, and if yes, how? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Fred L. Drake, Jr." References: <199812101453.PAA25622@dirac.cnrs-orleans.fr> Message-ID: <13935.59195.297200.97122@weyr.cnri.reston.va.us> Konrad Hinsen writes: > - Package installation via FTP/HTTP from a server. That may seem like > a minor convenience, but once you realize how many system administrators > don't know how to use FTP, it gains importance... This, in particular, seems to indicate that an additional script is installed along with Python. What's been discussed so far is how to build/install an unpacked distribution. I think a wrapper around this that can retrieve and unpack the distribution and then start the build/install operation on it is useful as well. Such a script might even know about a registry for packages, so you could ask it to load package spamandeggs, and it could go find the latest version compatible with your version of Python, download it, and install it. Sounds cool! Now we just need the registry and a real archive of packages. And a dependency system. And a host with disk space for the archive. ... ;-) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From hinsen@cnrs-orleans.fr Thu Dec 10 16:04:32 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 10 Dec 1998 17:04:32 +0100 Subject: [Distutils] Re: DistUtils requirements In-Reply-To: <19981210101503.D25928@cnri.reston.va.us> (message from Greg Ward on Thu, 10 Dec 1998 10:15:03 -0500) References: <199812101453.PAA25622@dirac.cnrs-orleans.fr> <19981210101503.D25928@cnri.reston.va.us> Message-ID: <199812101604.RAA26234@dirac.cnrs-orleans.fr> > covered in the requirements list, under "meta-data". The only thing I > had in mind there, though, is that the distutils would have some way of > representing and comparing version numbers -- largely a passive role. > RPM-style upgrade/downgrade support was not in mind, but it's a > possibility. Judging from how I use RPM, updates could be more frequent than installation! > > - Package installation via FTP/HTTP from a server. That may seem like > > a minor convenience, but once you realize how many system administrators > > don't know how to use FTP, it gains importance... > > I'm not entirely sure what you mean here. One possible interpretation > is the following scenario: At the simplest level it would be like RPM, i.e. you could specify an URL instead of a package filename. Then of course more intelligence could be added to help the user with finding a server that has a specific package or finding a package that solves a certain problem. But that sounds like something for DistUtils 2.0! First there must be enough DistUtil-enabled packages around before automatic searching becomes useful. > The plan is to stay away from Makefiles except for people who want and > can benefit from them (mainly developers writing collections of C > extensions under Unix). As long as you can run the compiler and linker Fine. Anyway, Makefiles and make are not essential for compiling code. But calling a compiler from a Python script may be hard enough if all you have is an IDE! > (However, I am a raving Unix bigot and delight in my ignorance of > bitty-box platforms like Windows and Macintosh. Anyone who actually Me too, and I hope I can remain ignorant about them for a long time ;-) But even for Unix, we should plan for maximum flexibility. The current Makefile.pre.in system (based on the information gathered by autoconf) is not really sufficient; for example, it doesn't work for AIX unless Makefile.pre.in is patched manually (and even then it's not flexible enough to deal with all situations!). At worst, we'd have one platform-specific module in DistUtils. > sort of configuration code. However, I think it would be useful to > provide some of the functionality that Autoconf provides, e.g. Python > classes and/or functions to check for a particular C library, or C > include file, or see if a certain token is defined, or if a certain Right. Otherwise everyone will have to start from scratch. > It certainly doesn't deal with cross-platform issues. One of the things > I love about Perl is the towering arrogance of the whole Perl community > regarding the utter and complete superiority of Unix over those "other" > so-called operating systems. Last time I checked, the semi-official Hey, maybe I should take a closer look at Perl after all... ;-) > Thanks for catching my slips in the requirements list. Anyone else > notice anything? Or have any wish-list items? Now's the time to get > 'em in... Nothing fundamental, but I just remembered one detail that many extension module writers overlook: if an extension module provides header files for use by other extension modules, they must be installed in a public place as well and the compilation system should take care of including this public place in the include directory list. There is no provision for this at all in the current Makefile.pre.in procedure. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gward@cnri.reston.va.us Thu Dec 10 16:14:36 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 10 Dec 1998 11:14:36 -0500 Subject: [Distutils] Version numbers for module distributions Message-ID: <19981210111436.A26034@cnri.reston.va.us> There have been a few rumbles about module version numbering on and off the list, and I thought I should bring it all out into the open. First, I think I tossed off a random opinion along these lines: Module version numbers should follow the GNU standard: X.Y.Z, where X is the major version, Y is the minor version, and Z is the patch level. Alpha and beta versions can be noted by appending "aN" or "bN", where N is any integer. This is still my opinion, although John Skaller raised a bunch of issues on the list, and Greg Stein noted a minor wrinlke in private email. I'll address those momentarily. First, let me eludicate the position above. This was (roughly) my opinion going into the Developer's Day session back at the Python Conference, and when Eric Raymond said roughly the same thing (minus the bit about alpha/beta version) based on his experience running the giant sunsite archive, I was glad to hear I wasn't the only one who likes this scheme. (And after all, the GNU folks have a hell of a lot of experience in these matters, and they have been right before!) Finally, this seems to be the version numbering system that Guido uses for Python itself, which is another good precedent. To be formal for a moment: a version number must match the regular expression (ignore whitespace in the regex): (\d+) \. (\d+) \. (\d+) ([ab](\d+))? Two possible changes I would consider open to debate: * replace [ab] with [a-z], to allow test versions beyond "beta"... although any programmer who goes up to "zeta" (or should that be omega?) really needs to look at his quality control... ;-) * make the patch number optional (and treat 1.0 and 1.0.0 as the same version number) Yes, this is very rigid. But it also makes it pretty easy to parse, split up, and compare version numbers. Making alpha/beta numbers a formal part is, IMHO, necessary, because intuitively 1.0.0a1 should be less than 1.0.0 -- but no lexical or numeric comparison will draw this conclusion! Now, there should also be semi-formal guidelines on what the different components of a version number *for modules* might mean: * a major number of 0 means the module is still being actively developed and, while it may be ready for evaluation, nobody should depend on stability of interface or implementation * a change in the major number beyond 1 means a major change in the interface, possibly not backwards compatible with previous major releases * a change in the minor number means some new functionality has been added in a way that doesn't break the interface * a change in the patch level means bugs have been fixed without adding new functionality * an alpha version (X.Y.Z.aN) is an advance preview of a future release where the interface has not yet stabilized * a beta version (X.Y.Z.aN) is an advance preview of a future release where the interface has stabilized, but known bugs exist and are still being fixed These rules have a number of implications that hadn't occurred to me until just now, when I wrote them down. In particular, whole classes of version numbers make no sense at all. Having an alpha version of something with major number 0 (0.1a3) is silly, because that leading zero already implies an unstable interface. Likewise, having an alpha version of a release with a non-zero patchlevel (1.3.4a2) is silly for the opposite reason; a change in the patchlevel means only bugs are being fixed, so there's no new interface to be unstable. This of course directly contradicts the semantics of Python version numbers; witness 1.5.2a2. But Guido's allowed to make up his own semantics; I'm just suggesting rules for Python module distributions. Oh yeah, let me stress that: *module distributions*. If you distribute a bunch of modules together, they might each have a version number, *but* there should also be a version number for the whole distribution. (And the individual modules in the distribution could just go in lockstep with the distribution version, if you want.) Now I'll try to address John Skaller's points about the multidimensionality of software versions... Quoth John Skaller (2 December 1998): > No, that isn't enough. Consider: > > MacVersion > UnixVersion > NTVersion I think that platform differences can be handled by the naming scheme for built distributions. For instance, version 1.3.2 of module 'foo' might be distributed under the following names: foo-1.3.2.tar.gz # source distribution foo-1.3.2-sparc-sun-solaris2.tar.gz # 'dumb' built distribution # for a certain Unix platform foo-1.3.2-i386-linux.rpm # 'smart' built distribution # for one popular PC platform foo-1.3.2-win32.??? # 'smart' built distribution for # some obscure PC platform This of course assumes that the same source distribution applies to all platforms. This is a lot easier with modern programming languages like Python, Java, or Perl than with crufty old beasts like C or C++. (In the immortal words of Larry Wall, "C isn't so much portable as it is ubiquitous".) But lots of C and C++ programmers go to *great* effort to ensure that their source distribution is the same for all platforms, even if they need to resort to conditional compilation, complicated configuration scripts, etc. to make it so. I think for the restricted domain of Python modules (even those written in C or C++), we can assume that people will write cross-platform source distributions (or at least have them as a goal). Next issue: > Now consider: > > FreeVersion > Commercial Version > > and then: > > BareBones version > Delux Version > Everything including the Kitchen Sink Version I see these are basically the same: different releases of essentially the same product but with varying degrees of functionality. (Where "functionality" might be defined as things like a license server for the commercial version. It could be argued that these *detract* from functionality, but let's not go there...) I'm not sure how to accomodate this one. A hack would be to give different names to the distributions: foo-1.3.2 for the standard free distribution, foo-commercial-1.3.2, foo-barebones-1.3.2, foo-delux-1.3.2 etc. for varying levels of functionality (and price!). However, again we come back to the fact that the product being distributed is *not* arbitrary software, but collections of Python modules together with their documentation and test suites. If you want different levels of functionality, you should split up your modules accordingly and make several distributions. Give away foo-base-1.3.2 to anyone who wants it, and charge extra for foo-bronze-1.3.2, foo-silver-1.3.2, and foo-gold-1.3.2, each of which contain a couple of modules that add still more functionality to the whole system. I think a lot of "version" problems can be solved by carefully organizing your modules so that blocks of functionality (barebones vs deluxe, informix vs sybase, etc.) correspond to modules or blocks of modules. Hell, this is just good solid software engineering practice, so you should be doing it that way anyways. That answer should also suffice where you have non-cross-platform source code (which is about the only reason I can think of for having platform-specific source releases). Finally, Greg Stein raised the following wrinkle in private email: > I wanted to briefly let you know that Apache modules are typically > versioned as: 0.9.3-1.3.3. The first set refers to the module itself. > The second set refers to Apache (which is currently at 1.3.3). Good point! I certainly hope we don't have to deal with this, i.e. have module developers worry whether their module will work with Python 1.5.2 or 1.6 or 1.6.1 (etc.). There is, however, the distinct possibility of API breakage with Python 2.0, which could affect potentially all extension modules. Umm, I think I'll let Guido handle this one, if he's still reading this SIG... From akuchlin@cnri.reston.va.us Thu Dec 10 16:29:43 1998 From: akuchlin@cnri.reston.va.us (Andrew M. Kuchling) Date: Thu, 10 Dec 1998 11:29:43 -0500 (EST) Subject: [Distutils] Version numbers for module distributions In-Reply-To: <19981210111436.A26034@cnri.reston.va.us> References: <19981210111436.A26034@cnri.reston.va.us> Message-ID: <13935.62516.587442.819078@amarok.cnri.reston.va.us> Greg Ward writes: >Now, there should also be semi-formal guidelines on what the different >components of a version number *for modules* might mean: Disagreement; I'd vote for mechanism, not policy. For example, recently I went through 3 pre-releases of xml-0.5; they could have been called 0.5a{1,2,3}. ILU's been at 2.0a{1-12}, and I have no idea if the developers ever plan to go to beta. Different developers use version numbers differently, in ways that are consistent in the large but inconsistent in small details. A side note: for the XML package, I think things are getting to the point that I want to create a fake build tree (the blib/ idea) in order to allow running the test suite without having to actually perform the installation. So, is there any implementation work that can be done? -- A.M. Kuchling http://starship.skyport.net/crew/amk/ Once you accept that the world is a giant computer run by white mice, all other movies fade into insignificance. -- Mutsumi Takahashi, on _The Hitch-hiker's Guide to the Galaxy_ From Fred L. Drake, Jr." References: <19981210111436.A26034@cnri.reston.va.us> <13935.62516.587442.819078@amarok.cnri.reston.va.us> Message-ID: <13935.64545.246413.945053@weyr.cnri.reston.va.us> Andrew M. Kuchling writes: > Disagreement; I'd vote for mechanism, not policy. For > example, recently I went through 3 pre-releases of xml-0.5; they could > have been called 0.5a{1,2,3}. ILU's been at 2.0a{1-12}, and I have no > idea if the developers ever plan to go to beta. Different developers > use version numbers differently, in ways that are consistent in the > large but inconsistent in small details. I don't see any problem with offering mechanism in the software and a short set of guidelines in documentation. Perhaps the docs should indicate that if the guidelines are not followed, perpetrators will be tossed over... er, they should describe how they are using the parts of the number in their documentation. > A side note: for the XML package, I think things are getting > to the point that I want to create a fake build tree (the blib/ idea) > in order to allow running the test suite without having to actually On this topic, it may make sense for the xml package to be a directory within the package, rather than the package toplevel. So xml-VERSION.tar.gz would expand to: xml-VERSION/ xml/ dom/ sax/ parsers/ utils/ ... doc/ README ... > perform the installation. So, is there any implementation work that > can be done? I think a version number "object" that can handle parsing and comparison is a clear need and fairly straightforward to implement. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <19981210111436.A26034@cnri.reston.va.us> Message-ID: <13935.65110.309959.994019@weyr.cnri.reston.va.us> Greg Ward writes: > * replace [ab] with [a-z], to allow test versions beyond > "beta"... although any programmer who goes up to "zeta" (or should > that be omega?) really needs to look at his quality control... ;-) Make it [ab] and don't change it unless there are substantial complaints (meaning someone actually wants to use something else and can present a good argument for doing so, *not* that someone might theoretically want to). > * make the patch number optional (and treat 1.0 and 1.0.0 as the > same version number) Definately. In fact, I'd want to hide the patch number if it's 0. If the version number class is distutils.version.Number, this should happen: >>> from distutils.version import Number >>> v = Number("1.0.0a1") >>> str(v) '1.0a1' > Quoth John Skaller (2 December 1998): > > No, that isn't enough. Consider: > > > > MacVersion > > UnixVersion > > NTVersion > > I think that platform differences can be handled by the naming scheme > for built distributions. For instance, version 1.3.2 of module 'foo' > might be distributed under the following names: I agree. The version number should not be platform dependent, as that can only create confusion. > > Now consider: > > > > FreeVersion > > Commercial Version > > > > and then: > > > > BareBones version > > Delux Version > > Everything including the Kitchen Sink Version > > I see these are basically the same: different releases of essentially > the same product but with varying degrees of functionality. (Where These are different products and should have different base names. Your examples describe this effectively. > Finally, Greg Stein raised the following wrinkle in private email: > > > I wanted to briefly let you know that Apache modules are typically > > versioned as: 0.9.3-1.3.3. The first set refers to the module itself. > > The second set refers to Apache (which is currently at 1.3.3). This is a dependency issue, and the dependency checking component should be able to deal with version numbers as appropriate. This is how other packaging systems deal with it (to the best of my knowledge, which is not comprehensive ;). -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From hinsen@cnrs-orleans.fr Thu Dec 10 17:06:31 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 10 Dec 1998 18:06:31 +0100 Subject: [Distutils] Version numbers for module distributions In-Reply-To: <19981210111436.A26034@cnri.reston.va.us> (message from Greg Ward on Thu, 10 Dec 1998 11:14:36 -0500) References: <19981210111436.A26034@cnri.reston.va.us> Message-ID: <199812101706.SAA16466@dirac.cnrs-orleans.fr> > There have been a few rumbles about module version numbering on and off > the list, and I thought I should bring it all out into the open. First, > I think I tossed off a random opinion along these lines: [lots of arguments deleted because I agree completely] > Now, there should also be semi-formal guidelines on what the different > components of a version number *for modules* might mean: Better formal ones. If you feel you need more precision to indicate subtle variations in your code, put them into a README file! > their documentation and test suites. If you want different levels of > functionality, you should split up your modules accordingly and make > several distributions. Give away foo-base-1.3.2 to anyone who wants it, > and charge extra for foo-bronze-1.3.2, foo-silver-1.3.2, and > foo-gold-1.3.2, each of which contain a couple of modules that add still > more functionality to the whole system. That's the only reasonable approach if we want to establish a dependency verification system. > I think a lot of "version" problems can be solved by carefully > organizing your modules so that blocks of functionality (barebones vs > deluxe, informix vs sybase, etc.) correspond to modules or blocks of > modules. Hell, this is just good solid software engineering practice, Right. But this should be explained carefully in the manual for package developers! > Good point! I certainly hope we don't have to deal with this, i.e. have > module developers worry whether their module will work with Python 1.5.2 > or 1.6 or 1.6.1 (etc.). We will have to consider this for binary distributions, because the binary APIs do change from version to version (not always, but sometimes). But since Python keeps a C API version number, all it takes is to note this version number in the package meta-data. At least for verification; distinguishing binary distributions for different Python versions by their file name is another question. But not really a difficult one, unless we want to support DOS file systems! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Fred L. Drake, Jr." References: <19981210111436.A26034@cnri.reston.va.us> <199812101706.SAA16466@dirac.cnrs-orleans.fr> Message-ID: <13936.487.51992.408655@weyr.cnri.reston.va.us> Konrad Hinsen writes: > At least for verification; distinguishing binary distributions > for different Python versions by their file name is another > question. But not really a difficult one, unless we want to support > DOS file systems! The package name should be stored in the package somehow, easily retrievable. There are probably a lot of NT boxes that use FAT file systems because they weren't born NT, and some companies deliver new NT boxes using FAT filesystems, and require that you explicitly convert the filesystem if you want NTFS. (DELL does this, and the conversion seems to have worked fine for me.) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From mal@lemburg.com Thu Dec 10 17:18:09 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 10 Dec 1998 18:18:09 +0100 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> Message-ID: <36700251.754E3533@lemburg.com> Greg Ward wrote: > > There have been a few rumbles about module version numbering on and off > the list, and I thought I should bring it all out into the open. First, > I think I tossed off a random opinion along these lines: > > Module version numbers should follow the GNU standard: X.Y.Z, where > X is the major version, Y is the minor version, and Z is the > patch level. Alpha and beta versions can be noted by appending > "aN" or "bN", where N is any integer. Sounds ok. There should also be some way to name current snapshots that are not meant as official distribution, e.g. when I get bug reports I usually fix them and then put a pre-version on the server for the envolved people to download and test. > Now, there should also be semi-formal guidelines on what the different > components of a version number *for modules* might mean: > > * a major number of 0 means the module is still being actively > developed and, while it may be ready for evaluation, nobody > should depend on stability of interface or implementation > * a change in the major number beyond 1 means a major change in the > interface, possibly not backwards compatible with previous > major releases > * a change in the minor number means some new functionality has been > added in a way that doesn't break the interface > * a change in the patch level means bugs have been fixed without > adding new functionality > * an alpha version (X.Y.Z.aN) is an advance preview of a future > release where the interface has not yet stabilized > * a beta version (X.Y.Z.aN) is an advance preview of a future > release where the interface has stabilized, but known bugs > exist and are still being fixed I don't think that a major version 0 does necessarily imply alpha status. Many Python extensions start their lifetime with 0.1 even though they could have started with 1.0 (I guess Pythoneers are attracted to understatement somehow). Likewise a change in the major number doesn't necessarily imply a change in the interface, e.g. a move from 0.9.9 to 1.0.0 may not include any changes at all. As a general rule I would not be too specific about the semantics of the version numbers: modules interdependence can safely be handled by comparing version numbers (a standard version number class would be a good idea, IMHO). The module authors will have to check compatibility anyway, so it's not the user's problem. Well, my 0.02e anyway... BTW: What's the general way of distributing subpackages ? My mxODBC stuff is getting a little "fat" and I'm thinking of splitting it up in smaller bits&pieces with a base package and n optional subpackages. -- Marc-Andre Lemburg Y2000: 386 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Thu Dec 10 18:46:16 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 10:46:16 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> <13935.65110.309959.994019@weyr.cnri.reston.va.us> Message-ID: <367016F8.75BAAF47@lyra.org> Fred L. Drake wrote: > ... > > Finally, Greg Stein raised the following wrinkle in private email: > > > > > I wanted to briefly let you know that Apache modules are typically > > > versioned as: 0.9.3-1.3.3. The first set refers to the module itself. > > > The second set refers to Apache (which is currently at 1.3.3). > > This is a dependency issue, and the dependency checking component > should be able to deal with version numbers as appropriate. This is > how other packaging systems deal with it (to the best of my > knowledge, which is not comprehensive ;). This is bunk. I'm not putting it there for an automated tool! It is there for the *USER* to figure out which one he wants. What about when I publish 1.1.3-2.0.1 and maintain *both*. And I strongly agree with Marc-Andre about staying out of the semantics of the version numbers. I'll use a format for the automated tool, but stay away from *my* semantics. I would also argue very strongly more more flexibility in the format schemes. For example, looking at the RedHat 5.1 distribution, I see a few formats: xzip-161-2.i386.rpm yp-tools-1.4.1-2.i386.rpm xv-3.10a-10.i386.rpm xrn-8.02-7.i386.rpm xpm-3.4j-2.i386.rpm xmorph-1996.07.12-4.i386.rpm xboard-3.2.pl0-9.i386.rpm x3270-3.1.1.6-2.i386.rpm spice-2g6-7.i386.rpm sox-11g-7.i386.rpm rdate-0.960923-4.i386.rpm nfs-server-2.2beta29-5.i386.rpm nenscript-1.13++-11.i386.rpm mailx-5.5.kw-9.i386.rpm dhcp-2.0b1pl0-2.i386.rpm I can draw a number of conclusions from this about types of formats that need to be allowed. But that's a straight-forward exercise. The point is: You have to be MUCH more flexible in what you can take for the version number. You cannot legislate a numbering scheme to the masses. It simply won't work because too many people have an idea of the "right" way to number. As long as you can specify rules for HOW numbers will be compared, then you're okay. For example: 1) a version number has 1 or more numbers separate by a period or by sequences of letters. If only periods, then these are compared left-to-right to determine an ordering. 2) sequences of letters are part of the tuple for comparison and are compared lexicographically 3) recognize the numeric components may have leading zeroes That should be about it. Welcome to Other Peoples' Versioning. Have a nice stay. :-) -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Thu Dec 10 19:17:31 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 10 Dec 1998 14:17:31 -0500 Subject: [Distutils] Version numbers for module distributions In-Reply-To: <13935.62516.587442.819078@amarok.cnri.reston.va.us>; from Andrew M. Kuchling on Thu, Dec 10, 1998 at 11:29:43AM -0500 Message-ID: <19981210141731.B26125@cnri.reston.va.us> Quoth Andrew M. Kuchling, on 10 December 1998: > Disagreement; I'd vote for mechanism, not policy. For > example, recently I went through 3 pre-releases of xml-0.5; they could > have been called 0.5a{1,2,3}. ILU's been at 2.0a{1-12}, and I have no > idea if the developers ever plan to go to beta. Different developers > use version numbers differently, in ways that are consistent in the > large but inconsistent in small details. Module developers are perfectly free to use illogical version numbers if they wish. I certainly don't intend to enforce through code the version number *guidelines* that I proposed, nor do I intend to check for the version numbers occupying the "illogical" subsets of version-number-space. I just wanted to point out the possible consequences of *that particular* set of guidelines. If anyone has suggestions for version number guidelines that don't have unexpected consequences, I'd like to hear 'em. (Of course, such guidelines should be short, simple, and obvious. That's what I like about my proposal.) > A side note: for the XML package, I think things are getting > to the point that I want to create a fake build tree (the blib/ idea) > in order to allow running the test suite without having to actually > perform the installation. So, is there any implementation work that > can be done? Yeah, the distutils.build module. It would have to be told various lists of source files (for a start: .py, ancillary .c files, and extension .c files) and then would Do The Right Thing: copy .py into blib, compile to .py and .po, compile all .c files, and link ancillary .o with extension .o to make .so. (Platform independence can come later.) Of course, we haven't really looked at the existing extension-building systems, e.g. the interface put on the table by John Skaller. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein@lyra.org Thu Dec 10 19:22:32 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 11:22:32 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210141731.B26125@cnri.reston.va.us> Message-ID: <36701F78.1B07493C@lyra.org> Greg Ward wrote: > ... > > A side note: for the XML package, I think things are getting > > to the point that I want to create a fake build tree (the blib/ idea) > > in order to allow running the test suite without having to actually > > perform the installation. So, is there any implementation work that > > can be done? > > Yeah, the distutils.build module. It would have to be told various > lists of source files (for a start: .py, ancillary .c files, and > extension .c files) and then would Do The Right Thing: copy .py into > blib, compile to .py and .po, compile all .c files, and link ancillary > .o with extension .o to make .so. (Platform independence can come > later.) IMO, a distutils test harness only needs to compile a .pyo. There isn't a need to do both because we can presume the Python interpreter has the appropriate testing. -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Dec 10 19:30:27 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 11:30:27 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> <36700251.754E3533@lemburg.com> Message-ID: <36702153.3CE88D7A@lyra.org> M.-A. Lemburg wrote: > ... > I don't think that a major version 0 does necessarily imply > alpha status. Many Python extensions start their lifetime > with 0.1 even though they could have started with 1.0 (I guess > Pythoneers are attracted to understatement somehow). > > Likewise a change in the major number doesn't necessarily imply a > change in the interface, e.g. a move from 0.9.9 to 1.0.0 may not include > any changes at all. > ... Absolutely. When my mod_dav is ready, I'm going to change it from 0.9.x to 1.0.0. No API changes, no user functionality changes. It is bogus to have distutils try to proscribe rules of semantics to the version numbering. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Dec 10 19:47:00 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 11:47:00 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> Message-ID: <36702534.A03FB89@lyra.org> Greg Ward wrote: > ... > I think a lot of "version" problems can be solved by carefully > organizing your modules so that blocks of functionality (barebones vs > deluxe, informix vs sybase, etc.) correspond to modules or blocks of > modules. Hell, this is just good solid software engineering practice, > so you should be doing it that way anyways. That answer should also > suffice where you have non-cross-platform source code (which is about > the only reason I can think of for having platform-specific source > releases). People will use whatever practices they want. Many people do not have software engineering degrees (heck, many don't even have degrees!). If you organize distutils such that you require a change on their part, they'll simply ignore distutils. Back to .tar.gz files. I believe that you cannot attempt to enforce much policy on the coders. Some, sure. But pick your fights. -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Thu Dec 10 19:47:53 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 10 Dec 1998 14:47:53 -0500 Subject: [Distutils] Version numbers for module distributions In-Reply-To: <36702153.3CE88D7A@lyra.org>; from Greg Stein on Thu, Dec 10, 1998 at 11:30:27AM -0800 References: <19981210111436.A26034@cnri.reston.va.us> <36700251.754E3533@lemburg.com> <36702153.3CE88D7A@lyra.org> Message-ID: <19981210144753.C26125@cnri.reston.va.us> Wow, looks like I found the hot-button issue that this SIG was looking for. Fun! Quoth Greg Stein, on 10 December 1998: > It is bogus to have distutils try to proscribe rules of semantics to the > version numbering. ^^^^^^^^^ I think you meant "prescribe" not "proscribe". Whatever! Anyways, everyone who's howling in protest against forcing certain semantics down module developers' throats missed an important point: those were only *guidelines*, and what's more they were only *proposed* guidelines! (OK, OK, only Greg Stein howled. Andrew and Marc-Andre stated their case nicely.) Heck, you could even interpret the fact that "0.5a1" or "1.5.2a1" are "illogical" version numbers as flaws in my proposal, and come up with different guidelines that don't have that problem. Feel free! One possible modification to the guidelines based on the feedback received so far: "changes in major version number imply a change in the interface, *except* for the 0.9.x to 1.0 change." It would be good to stress that you really shouldn't break the interface in a "mature" (>= 1.0) product without bumping the major version number. The fact remains, though, that under my proposed version numbering system (which mandates syntax and suggests semantics), developers would be free to use version numbers however they like. The proposed semantics are *suggested guidelines*. How on earth could such things be enforced, anyways? (OK, here's an idea: if you upload version 1.3.2 to an archive, the archive software unpacks 1.3.2 next to 1.3.1, and makes sure that you haven't added any new methods or functions, or any parameters to existing methods or functions. JUST KIDDING!) My rationale for having some *suggested guidelines* is that most developers spend several years groping in the dark trying to figure out how to assign version numbers to their software. Experienced hackers eventually figure it out, and really smart, far-thinking hackers just "get it" immediately and come up with a system that will work for them henceforth and forevermore. I think most people, though, would benefit from a little friendly advice informed by the many decades of collective experience that will inform the "How to Develop and Distribute Python Modules" document that will come of all this. Greg also had a suggestion about loosening up the syntax, essentially by reducing it to a set of comparison rules: > 1) a version number has 1 or more numbers separate by a period or by > sequences of letters. If only periods, then these are compared > left-to-right to determine an ordering. > 2) sequences of letters are part of the tuple for comparison and are > compared lexicographically > 3) recognize the numeric components may have leading zeroes As long as we can a) explain the syntax, and b) write code to parse and compare version numbers, then any proposal is valid. This one sounds valid. However, keep in mind that we're dealing with a much smaller and more cohesive population of existing packages than the guys at Red Hat deal with. It might turn out that a heck of a lot of existing Python module developers think there should be a standard (and fairly rigid) *syntax* for version numbers, and are willing to change their habits to go along with this. If not, we can adopt Greg's much looser syntax proposal, and if there's anybody left over who can't or won't fit their version numbers into that syntax... well, too bad. Greg (the other one) -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19981210111436.A26034@cnri.reston.va.us> <36702534.A03FB89@lyra.org> Message-ID: <13936.9913.548949.621070@weyr.cnri.reston.va.us> Greg Stein writes: > People will use whatever practices they want. Many people do not have > software engineering degrees (heck, many don't even have degrees!). If > you organize distutils such that you require a change on their part, > they'll simply ignore distutils. Back to .tar.gz files. I don't think .tar.gz went anywhere. Are we supposed to be doing a new archive format as well? I don't see any point. > I believe that you cannot attempt to enforce much policy on the coders. > Some, sure. But pick your fights. Gosh, that's what we say about dealing with kids, too! ;-) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From gstein@lyra.org Thu Dec 10 20:00:55 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 12:00:55 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> <36702534.A03FB89@lyra.org> <13936.9913.548949.621070@weyr.cnri.reston.va.us> Message-ID: <36702877.4EBF5026@lyra.org> Fred L. Drake wrote: > > Greg Stein writes: > > People will use whatever practices they want. Many people do not have > > software engineering degrees (heck, many don't even have degrees!). If > > you organize distutils such that you require a change on their part, > > they'll simply ignore distutils. Back to .tar.gz files. > > I don't think .tar.gz went anywhere. Are we supposed to be doing a > new archive format as well? I don't see any point. nah, I was trying to imply a simple archive of files, rather than a cool package. -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Thu Dec 10 20:31:29 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 10 Dec 1998 12:31:29 -0800 Subject: [Distutils] sample version number code Message-ID: <36702FA1.7639923F@lyra.org> Here is some sample code for version numbering, a-la-GJS. # # version numbering test code # samples = [ '1.5.1', '1.5.2b2', '161', '3.10a', '8.02', '3.4j', '1996.07.12', '3.2.pl0', '3.1.1.6', '2g6', '11g', '0.960923', '2.2beta29', '1.13++', '5.5.kw', '2.0b1pl0', ] import re r = re.compile('([0-9]+|[a-z]+|\.)') def removeall(l, v): while 1: try: l.remove(v) except ValueError: return def vsn(s): comps = r.split(s) removeall(comps, '.') removeall(comps, '') for i in range(len(comps)): try: comps[i] = int(comps[i]) except ValueError: pass return tuple(comps) for s in samples: print vsn(s) -- Greg Stein, http://www.lyra.org/ From skaller@maxtal.com.au Fri Dec 11 03:33:11 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:11 +1000 Subject: [Distutils] Version numbers for module distributions Message-ID: <1.5.4.32.19981211033311.00f816ec@triode.net.au> At 11:14 10/12/98 -0500, Greg Ward wrote: >Yes, this is very rigid. But it also makes it pretty easy to parse, >split up, and compare version numbers. Making alpha/beta numbers a >formal part is, IMHO, necessary, because intuitively 1.0.0a1 should be >less than 1.0.0 -- but no lexical or numeric comparison will draw this >conclusion! It depends on your personal 'intuition'. I could say that an alpha version of a product is a _different_ product, in other words, the lexical ordering is: alpha/beta/production tag product name major version minor version subminor version In other words, if I am running 1.2.3 production, the latest release will NEVER be 1.2.4 alpha. That is a completely different product family! I don't run any alphas! But you are entitled to a different policy! >Now I'll try to address John Skaller's points about the >multidimensionality of software versions... > >Quoth John Skaller (2 December 1998): >> No, that isn't enough. Consider: >> >> MacVersion >> UnixVersion >> NTVersion > >I think that platform differences can be handled by the naming scheme >for built distributions. >I think for the restricted >domain of Python modules (even those written in C or C++), we can assume >that people will write cross-platform source distributions (or at least >have them as a goal). They can't. Just compare a JPython vs CPython version of something that also contains C/Java code. Smly: compare a MAC version of something whose Unix version assumes a command line. >Next issue: > >> Now consider: >> FreeVersion >> Commercial Version >> and then: >> >> BareBones version >> Delux Version >> Everything including the Kitchen Sink Version > >I see these are basically the same: different releases of essentially >the same product but with varying degrees of functionality. >I'm not sure how to accomodate this one. I have the idea for a general mechanism that will accomodate ALL the various 'dimensions'. I haven't implemented it yet, however. But I have made a crude start. I think one needs to assume arbitrary tags on components: some of which are standardised. For example: name: interscript interface: 1 implementation: 3.2 platform supported: Mac, NT, Unix python required: C1.5.1 included components: basic stability: beta languages: English, Spanish requires: name: mxTools interface: 1 implementation: >2 included components: delux >I think a lot of "version" problems can be solved by carefully >organizing your modules so that blocks of functionality (barebones vs >deluxe, informix vs sybase, etc.) correspond to modules or blocks of >modules. Hell, this is just good solid software engineering practice, >so you should be doing it that way anyways. You are right. But most filing systems don't have the require functionality: they're hieratchical, which is simply NOT good enough. This is also why version numbers in sequence like 1.2.3 are NOT enough. Heirarchies are not enough. More general categories are required. Interscript solves this problem by _generating_ code. This is the only way to cope with multi-factorial installation issues. This is also what 'autoconf' et-al do. BTW: a decision on _anything_, even with limitations, is better than no standard! Because there will always be limitations. But I think it is good practice to see how far we can push things, before backtracking to something we can agree on and implement, and which both clients and suppliers will find useful. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 11 03:33:18 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:18 +1000 Subject: [Distutils] interscript Message-ID: <1.5.4.32.19981211033318.00f89310@triode.net.au> I'm going to put the case, semi-seriously, that the 'solution' to the problem of distributing packages is to use interscript. I'll try to distinguish what works now, what is planned, and what I have no real idea about yet. For those not familiar with it: it is a source format in which 1) code 2) documentation 3) installation and building instructions can all be encoded. First, interscript allows all packages to be archived in a single text file. [Works now]. It is possible to store encoded binary if it is _really_ necesssary [planned]. The client can extract stuff from the archive with a command line tool [works now], or by embedding the package in another application [works now]. The client decides if documentation is to be generated [works now] and if so, what format it should be in [works now: plain text, flat HTML, advanced HTML, Latex2e, Lambda work now, XML planned] The client decides what language the documentation is to be produced in [planned] The client determines where the documentation is installed [works now] and where the code is installed [works now]. The package automatically builds any components written in C or C++ [works now]. It can configure itself for any platform/user/site/ preferences [planned, what works now depends on the ability to 'introspect' the information from Python itself, which is not enough] The package automatically tests everything. [Works now, but more work is required to create a proper test harness] Interscript can read source directly off the Internet [FTP and HTTP works now]. Interscript can read compressed archives [planned]. Interscript can generate interscript archives from any existing packages [sort of works :-] ----------------------------------------------------------- There is a full categorical installation/version control/dependency mechanism. [Some plans: much help needed here!!!] ----------------------------------------------------------- So there you have it: interscript already does Almost Everything (TM) a disttribution mechanism requires, and it will support almost all programming languages, not just C/C++/Java/Python, although support for these four languages is the highest priority (because they're needed for interscript itself). Interscript is written entirely in Python, although there are some C modules which do optimisations, and there is some dependency on external tools (C/C++/Java compilers, patch, diff) for full functionality. ----------------------------------------------------------- Objection: Interscript is too big. Answer: At present, everything is bundled togther in an experimental package. Because there is no standard or simple way to distribute the separate components separately. The intent is to use this experimental version to create the tools needed to unbundle itself. This is called 'bootstrapping' Objection: I'm not interesting in Literate Programming. Answer: Irrelevant. Interscript is a build/test/install environment. The mechanism includes a lot of support for literate programming, but this doesn't _have_ to be used by anyone. Objection: My packages aren't built using interscript, I don't want to convert them, it's too much work. Answer: an automated tool can package up all existing distributions. All that is required is listing the files to be packaged, with an indication if they're text or binary. Objection: not enough is working properly yet. Answer. This is a valid objection. More people working on the components would help remove this objection. See next post. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 11 03:33:20 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:20 +1000 Subject: [Distutils] My view of installation control Message-ID: <1.5.4.32.19981211033320.00f795bc@triode.net.au> Here is roughly what I am planning for interscript installation control. Executive summary ----------------- 1) In the source, the author provides category tags for files. 2) The client is required to bind the tags to installation points. Explanation ----------- Clients may wish to install parts of packages in different ways. For example they may wish to install doco in the same directory as each module: package/something.py package/something.html or in separate isomorphic directories: code/package/something.py doco/package/something.html There are a lot of variations on this scheme, and we must cope with ALL of them. To do this, we rely on the AUTHOR to identify the category of a file by tagging it: the file extension can be used for defaulting, as well as a few other things (such as the kind of generating process). This means that absolute filenames are not allowed in the source package. The CLIENT must be able find out what tags the author used, and provide a binding to an installation point for every file based exclusively on the tags of that file. For example: for a PY file, the installation point might be deemed /usr/local/lib/python1.5/site-packages and the [DOCO, HTML] installation point might be /home/httpd/html/python/installed-packages This mechanism is NOT enough, but it should cover a very wide variety of cases. [See below] The issues involved are: 1) how does the author decide what tags to use? 2) how does the client find out what the tags the author used are, and what they are intended to mean? 3) how does the client specify the binding of tags to installation points? --------------------------- [Below] There is a more general solution than mere tags, which I'm working on. The basic idea is that the author provides a _category_ which defines the abstract structure of the package, and the client provides a _functor_ which binds the category to the installation space. This mechanism is fully general, it can necessarily do everything. Of course, the source category has to be specified, which limits the domain to those categories for which there are effective implementations; the same comment applies to the installation functor. This mechanism has many advantages, apart from complete generality. The main one is that the installation function can be _verified_ with many representations, that is, it can be checked mnechanically that the installation mapping preserves the abstract structure of the package. With some modifications to the aboave abstraction, installation can be extended to upgrading. This mechanism automatically subsumes all notions of version control, dependency checking, etc. It allows, for example, any kind of version numbering system _provided that system is coherent_, (where coherent has a very precise meaning). ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 11 03:33:16 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:16 +1000 Subject: [Distutils] Version numbers for module distributions Message-ID: <1.5.4.32.19981211033316.00f8f970@triode.net.au> >My rationale for having some *suggested guidelines* is that most >developers spend several years groping in the dark trying to figure out >how to assign version numbers to their software. Experienced hackers >eventually figure it out, and really smart, far-thinking hackers just >"get it" immediately and come up with a system that will work for them >henceforth and forevermore. I think most people, though, would benefit >from a little friendly advice informed by the many decades of collective >experience that will inform the "How to Develop and Distribute Python >Modules" document that will come of all this. The problem I have with this is that the many years of collective experience only shows the common mechanisms are grossly inadequate. After all, many years of collective experience shows the world is flat. Truth is not an issue of consensus. Neither is doing version control anything to do with collective experience with old, broken, paradigms. >As long as we can a) explain the syntax, and b) write code to parse and >compare version numbers, then any proposal is valid. This one sounds >valid. I don't agree. This sounds like "I'm Ok, Your're Ok". The only 'valid' scheme is one which works. There are no such schemes, none of them come even close to working. By FAR the best scheme is the Microsoft Active Update mechanism, because it figures it all out, and installs updates on demand. In my opinion. :-)) ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 11 03:33:10 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:10 +1000 Subject: [Distutils] DistUtils requirements Message-ID: <1.5.4.32.19981211033310.00f7c0bc@triode.net.au> At 10:22 10/12/98 -0500, Fred L. Drake wrote: > >Konrad Hinsen writes: > > - Package installation via FTP/HTTP from a server. That may seem like > > a minor convenience, but once you realize how many system administrators > > don't know how to use FTP, it gains importance... > > This, in particular, seems to indicate that an additional script is >installed along with Python. > What's been discussed so far is how to build/install an unpacked >distribution. I think this indicates a 'design bug'. It shouldn't be necessary to unpack anything. Python can read data straight out of ZIP/Tar files. I mean: it can't at the moment, but it is on the TODO list. This requires some support from Python itself, but I feel Guido will be wll disposed to being able to access a package which is packed in an archive. [I.e. 'import' directly from the archive, _without_ unpacking it] This is also on the Interscript TODO list: interscript can already read files by FTP, HTTP, as well as from the local file system, but it cannot read files from archives. But the modules required to do this seem to be part of the standard distribution already. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Fri Dec 11 03:33:14 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 11 Dec 1998 13:33:14 +1000 Subject: [Distutils] Version numbers for module distributions Message-ID: <1.5.4.32.19981211033314.00f7b684@triode.net.au> >And I strongly agree with Marc-Andre about staying out of the semantics >of the version numbers. I'll use a format for the automated tool, but >stay away from *my* semantics. Add requirement to distutils requirements document: "Must accomodate existing schemes" >I would also argue very strongly more more flexibility in the format >schemes. For example, looking at the RedHat 5.1 distribution, I see a >few formats: > >xzip-161-2.i386.rpm >yp-tools-1.4.1-2.i386.rpm >xv-3.10a-10.i386.rpm >xrn-8.02-7.i386.rpm >xpm-3.4j-2.i386.rpm >xmorph-1996.07.12-4.i386.rpm >xboard-3.2.pl0-9.i386.rpm >x3270-3.1.1.6-2.i386.rpm >spice-2g6-7.i386.rpm >sox-11g-7.i386.rpm >rdate-0.960923-4.i386.rpm >nfs-server-2.2beta29-5.i386.rpm >nenscript-1.13++-11.i386.rpm >mailx-5.5.kw-9.i386.rpm >dhcp-2.0b1pl0-2.i386.rpm And .. they're all RPMs. :-) >You have to be MUCH more flexible in what you can take for the version >number. You cannot legislate a numbering scheme to the masses. It simply >won't work because too many people have an idea of the "right" way to >number. Agree. >As long as you can specify rules for HOW numbers will be compared, then >you're okay. For example: > >1) a version number has 1 or more numbers separate by a period or by >sequences of letters. If only periods, then these are compared >left-to-right to determine an ordering. >2) sequences of letters are part of the tuple for comparison and are >compared lexicographically >3) recognize the numeric components may have leading zeroes > >That should be about it. > >Welcome to Other Peoples' Versioning. Have a nice stay. :-) I disagree with the above rule :-) Solution: each package must provide a function to compare its own version numbers. Comments? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From mal@lemburg.com Fri Dec 11 00:02:49 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 11 Dec 1998 01:02:49 +0100 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> <36702534.A03FB89@lyra.org> <13936.9913.548949.621070@weyr.cnri.reston.va.us> <36702877.4EBF5026@lyra.org> Message-ID: <36706129.7C55E1C6@lemburg.com> I think the discussion is moving in a wrong direction here. We should maybe start to rethink what the intent of version numbering is to start with: · A newer version should be recognizable by comparing it to an older one and this must be doable by both computers and humans · Newer versions might introduce interface changes, but these are described in the package history -- the version number may not reflect this change (e.g. Python's API level will change even though it's only a new patch level that's being released) So basically all that's needed is to be able to compare the version numbers of packages within a release sequence. It doesn't matter whether they use numbers and dots, words or dates. Trying to automate compatibility checks by only using version numbers isn't really a feasable approach, IMHO. This will always remain a manual and errorprone process... -- Marc-Andre Lemburg Y2000: 386 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From mal@lemburg.com Fri Dec 11 10:05:17 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 11 Dec 1998 11:05:17 +0100 Subject: [Distutils] Package descriptions Message-ID: <3670EE5D.15CEEFFD@lemburg.com> Since there seems to be some interest in capturing meta information about packages, I suggest having a look at the (currently sleeping) Trove project initiated by E. Raymond: http://tuxedo.org/~esr/trove/ It defines quite a few different ways of identifying relationships between packages, subpackages and other resources. No need to discuss all of this again... . -- Marc-Andre Lemburg Y2000: 385 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Fri Dec 11 12:31:04 1998 From: gstein@lyra.org (Greg Stein) Date: Fri, 11 Dec 1998 04:31:04 -0800 Subject: [Distutils] Version numbers for module distributions References: <19981210111436.A26034@cnri.reston.va.us> <36702534.A03FB89@lyra.org> <13936.9913.548949.621070@weyr.cnri.reston.va.us> <36702877.4EBF5026@lyra.org> <36706129.7C55E1C6@lemburg.com> Message-ID: <36711088.61AB5D30@lyra.org> M.-A. Lemburg wrote: >... > So basically all that's needed is to be able to compare the version > numbers of packages within a release sequence. It doesn't > matter whether they use numbers and dots, words or dates. > > Trying to automate compatibility checks by only using version > numbers isn't really a feasable approach, IMHO. This will always > remain a manual and errorprone process... I very much agree, which is why I posted my view on the rules we can use for comparing them. The code I posted returns tuples which should be directly comparable. I believe they define a good ordering, but it would also take a bit of review to see what happens when elements are inserted/removed (e.g. the a2 at the end of 1.5.2a2) -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Fri Dec 11 23:49:04 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 11 Dec 1998 18:49:04 -0500 Subject: [Distutils] Attempted summary of version number thread Message-ID: <19981211184904.A27438@cnri.reston.va.us> I'm going to see if I can't make some sense out of the version-numbering thread of the last couple of days. First: if you've already said your two cents' worth in this thread, please don't chime in again. I think everyone has made their point quite clear. (However, if I misunderstood you and misrepresent you in the summaries below, a brief correction to the list would be appreciated.) It seems to me that there are four camps out there: the control freaks, the anarchists, the middle-of-the-roaders, and John Skaller. The control freaks (me, Fred, and Konrad -- at least you two said you agree with me!) are in favour of prescribing *syntax* for version numbers and *suggesting* semantics. The *proposed* syntax and semantics that I posted are up for debate, but at least three of us like the idea of imposing a little order on how version numbers look, and giving developers some general guidelines -- which they are free to ignore! -- about what they should mean. The anarchists are Greg Stein and Marc-Andre Lemburg; these two take a "live and let live" approach to version numbers. They seem to concede that some very light syntactic requirements are necessary, but want developers to be free to define their semantics however they like. (Or, Greg and Marc-Andre just want to write code to describe the existing multitude of version number systems.) I think Andrew Kuchling is the only middle-of-the-roader, and I might be saying that just because I know Andrew's a pretty relaxed guy and not prone to radical positions like anarchy. Anyways, he posted an anti-prescriptionist statement to the list right after my syntax and semantics proposal. Maybe Andrew is really our third anarchist. Finally, John Skaller is out on his own. As I read it, John has a much wider definition of "version" than I (or the others who've posted on the topic) do. Or maybe it's more accurate to say: I started this thread to talk about *version numbers*, and John is more concerned with software versions *in general* -- all the things that can't be expressed in a simple linear progression. So: do any of you object to the above characterizations? I'd be interested to hear the opinions of anyone outside the above six, especially if you feel there're more sides to this debate than "control freaks", "anarchists", and "John Skaller". (Gee, I hope John is back on the list -- mailman disabled him because of excessive bounces, and I'd hate for him to miss this. ;-) One possible way to resolve this is to provide a couple of standard classes to implement version numbering systems. I won't say any more in detail right now, because I don't want to start another heated debate. But think about it... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal@lemburg.com Sat Dec 12 00:52:17 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 12 Dec 1998 01:52:17 +0100 Subject: [Distutils] Attempted summary of version number thread References: <19981211184904.A27438@cnri.reston.va.us> Message-ID: <3671BE41.6B8D8B1B@lemburg.com> Greg Ward wrote: > > I'm going to see if I can't make some sense out of the version-numbering > thread of the last couple of days. > > First: if you've already said your two cents' worth in this thread, > please don't chime in again. I think everyone has made their point > quite clear. (However, if I misunderstood you and misrepresent you in > the summaries below, a brief correction to the list would be > appreciated.) > > ... > > The anarchists are Greg Stein and Marc-Andre Lemburg; these two take a > "live and let live" approach to version numbers. They seem to concede > that some very light syntactic requirements are necessary, but want > developers to be free to define their semantics however they like. (Or, > Greg and Marc-Andre just want to write code to describe the existing > multitude of version number systems.) I wouldn't describe myself as anarchist... realist is more like it ;-) We should get on with more important things than version numbers, IMHO, really. What's inside the package is important, not how it is named. And that information is quite well captured by the Trove meta records. ------------------------------------------------------------------- What I'd like to see is a way that automates the build process. Automating software updates is by far more complicated and should be deferred for now. BTW, Fredrik mentioned that building with VC++ on WinXX is doable with command line invocation only. Wouldn't that be a good start into unifying the make process on Unix and WinXX [what about Macs ?]. A WISE kind of installer for Unices wouldn't look bad either. Could be done using Tkinter or ncurses or simple text based dialogs: /home/sweet/home> ./setup.py PACKAGE FooBar Show version information ? [No] Show install guide ? [Yes] Which Browser should I use ? [netscape] ...netscape fires up and displays the install guide... Where do you want to install today ? [/usr/holidays/beach] >>> building... ok Should I run the test suite now ? [Yes] >>> running tests... ok (output stored in FooBar.tests) Display install summary ? [No] Should I save your answers for future installations ? [Yes] >>> saved to FooBar.installation FooBar installed successfully. All this is doable and could even be fun... I suggest just hacking away at something like it and then abstracting all meaningful bits & pieces into nice distutils modules. PS: For the typical Unix freak we should also include a README pointing her to ./setup.py and maybe even a Makefile with all targets running ./setup.py using additional parameters, e.g. setup.py --make install, setup.py --make clean, etc. PPS: For the typical WinXX freak there'll have to be an install.bat of course ;-) PPPS: Don't know how the Macs do it... they probably just click on the icon :-) -- Marc-Andre Lemburg Y2000: 385 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From MHammond@skippinet.com.au Sat Dec 12 01:01:24 1998 From: MHammond@skippinet.com.au (Mark Hammond) Date: Sat, 12 Dec 1998 12:01:24 +1100 Subject: [Distutils] Attempted summary of version number thread In-Reply-To: <19981211184904.A27438@cnri.reston.va.us> Message-ID: <001101be256a$f16cc0f0$0801a8c0@bobcat> > So: do any of you object to the above characterizations? I'd be > interested to hear the opinions of anyone outside the above six, > especially if you feel there're more sides to this debate Well, you asked :-) Im definately on the side of the Anarchists. IMO, the "deliverables" from this SIG should be unobtrusive as possible, and should conform to _my_ wishes, rather than the other way around. For example, my Windows extensions use a simple "build number", and this is what I want to continue to use. My reasoning is simply that I make many incremental releases - its not a product per se. There is never a clear distinction between "versions" - what would "version 1" of my extensions be, or "version 2"? Further, it is never clear to me when my releases are "beta" - new features may not work completely as advertised, but core features are typically more stable than previous versions. This is almost certainly an artifcat of the volume of things in my single package - but that is the point - the way _I_ do things dont necessarily fit with the other version branding models being imposed. Of course, other things I have in mind _do_ fit better with some of the proposed schemes. So one size doesnt fit all, even just for me! If the deliverables require me to rework too many things, or shoehorn into a scheme that doesnt fit for me, I may simply not bother. Without presuming to speak for Marc and Greg, IMO, the message is clear - if some people on this SIG dont see it working for them, then the chances of it becoming truly general purpose to the wider community are slight. Just MHO (Mark Hammond's Opinion :-), of course! Mark. From skaller@maxtal.com.au Sun Dec 13 12:31:40 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 13 Dec 1998 22:31:40 +1000 Subject: [Distutils] Proposal: C/C++ compiler identification Message-ID: <1.5.4.32.19981213123140.00f6e05c@triode.net.au> This proposal is intended as a detail of the general requrements document for distutils. The proposal is for the Python build process to create run time accessible identification data whose purpose is to facilitate run time building of a) dynamically loadable binary python module objects *.so, .dll etc) b) external (shell) application tools (.exe etc) This proposal does not fix the format of the information. --- I would like to consider a class instance, with one instance for each supported compiler/kind combination. I recommend the Python make process attempt to provide access to C, C++, and Java compilers. A binary build, such as Mark Hammonds Windows build, cannot be pre-configured for the client's system. Note that interscript contains modules which compile C/C++ code. At present, the modules have to rewritten by the client (unless they're running RedHat Linux :-) It is not clear to me if an _executable_ representation of the above requirement is appropriate. If not, at least the information required to create one should be made available. The representation must be editable by the client somehow. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From guido@CNRI.Reston.VA.US Sun Dec 13 15:27:15 1998 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Sun, 13 Dec 1998 10:27:15 -0500 Subject: [Distutils] Proposal: C/C++ compiler identification In-Reply-To: Your message of "Sun, 13 Dec 1998 22:31:40 +1000." <1.5.4.32.19981213123140.00f6e05c@triode.net.au> References: <1.5.4.32.19981213123140.00f6e05c@triode.net.au> Message-ID: <199812131527.KAA18636@eric.cnri.reston.va.us> > The proposal is for the Python build process to create run time accessible > identification data whose purpose is to facilitate run time building of > > a) dynamically loadable binary python module objects *.so, .dll etc) > b) external (shell) application tools (.exe etc) > > This proposal does not fix the format of the information. Note that (for Unix at least) all this info is already being gathered by the configure script, and stored in the various Makefile. The info also ends up being installed (look in /usr/local/lib/python1.5/config/). For Windows, since the only supported compiler is VC++, it could be hardcoded. However, there's a very serious problem on Windows with Interscript's approach of invoking the compiler at run-time: most users don't have this compiler. I realize that gcc is available for free, but I think it isn't compatible with VC++. As far as I know, VC++ is required if you want to use any of Mark Hammond's stuff (COM and MFC). I don't know if Tcl/Tk can be used (without recompilation) from a gcc-based app, and I don't know how easy it would be to recompile. (Does gcc on Windows have support for Win32 APIs at all?) --Guido van Rossum (home page: http://www.python.org/~guido/) From MHammond@skippinet.com.au Mon Dec 14 01:42:59 1998 From: MHammond@skippinet.com.au (Mark Hammond) Date: Mon, 14 Dec 1998 12:42:59 +1100 Subject: [Distutils] Freeze on topic for this SIG? Message-ID: <000601be2703$150951b0$0801a8c0@bobcat> Hi all, The first paragraph of this SIG's charter states: """ The distutils sig exists to discuss the design and implementation of a suite of module distribution utilities for Python. These utilities will take the form of some standard modules, probably grouped in the 'distutils' package, for building, packaging, and installing third-party modules. This includes both modules written purely in Python and extension modules written in C/C++. """ This implies that the freeze tool would indeed be on topic (even though I fairly certain that this was not the intent!). Im really just looking for a place to discuss enhancements to Freeze itself. Specifically, Guido, Jack Jansen, Just and myself have had very very preliminary discussions on a new "import architecture" for Python itself. It is driven by freeze, but would probably become a standard part of Python if proven worthy. Im concerned it is too esoteric for the newgroup itself, but also that it really would be stretching the charter of this SIG. Please let me know if this is acceptable here (would make an interesting diversion from version numbers, tho :-), or if I should attempt a newsgroup assault! Mark. From gstein@lyra.org Mon Dec 14 01:52:22 1998 From: gstein@lyra.org (Greg Stein) Date: Sun, 13 Dec 1998 17:52:22 -0800 Subject: [Distutils] Freeze on topic for this SIG? References: <000601be2703$150951b0$0801a8c0@bobcat> Message-ID: <36746F56.55B17209@lyra.org> Mark Hammond wrote: > ... > Please let me know if this is acceptable here (would make an > interesting diversion from version numbers, tho :-), or if I should > attempt a newsgroup assault! IMO, Freeze is a perfectly valid distribution mechanism. Including it here makes sense to me. No reason there can't be multiple discussion threads within the SIG. -g -- Greg Stein, http://www.lyra.org/ From hinsen@cnrs-orleans.fr Mon Dec 14 16:46:45 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 14 Dec 1998 17:46:45 +0100 Subject: [Distutils] Attempted summary of version number thread In-Reply-To: <19981211184904.A27438@cnri.reston.va.us> (message from Greg Ward on Fri, 11 Dec 1998 18:49:04 -0500) References: <19981211184904.A27438@cnri.reston.va.us> Message-ID: <199812141646.RAA21126@dirac.cnrs-orleans.fr> > the anarchists, the middle-of-the-roaders, and John Skaller. The > control freaks (me, Fred, and Konrad -- at least you two said you agree > with me!) are in favour of prescribing *syntax* for version numbers and > *suggesting* semantics. The *proposed* syntax and semantics that I Right, and it seems worth pointing out again why, or rather to what extent, the interpretation of version numbers should be unified. The criterion is that it must be clear both for human users and for installation programs 1) which of two versions is newer 2) if two versions are compatible, i.e. use the same APIs The first criterion could be met by any "reasonable" version numbering scheme, plus a package-specific routine for comparison, as has been pointed out by the "anarchists". But the second one is less obvious to realize. For the installation program, another package-supplied routine would be sufficient, but it should also be clear for a normal user whether version 1.2.3 is merely a bug-fix update to 1.1.0 or uses a non-compatible API. Another distinction that matters for the user is whether code is production level or experimental. Within these limits I am willing to leave all the freedom people want, such as whether to start at 0 or at 1. So perhaps the following scheme might be acceptable: Version numbers start with an API version number, then any combination of letters and punctuation that is likely to make sense to a human reader, followed by an optional "alpha" or "beta" indicator. The first number would change only if a package cannot be used as a drop-in replacement for its predecessors. Is that something the "anarchists" could agree on? Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From gward@cnri.reston.va.us Mon Dec 14 17:38:53 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 14 Dec 1998 12:38:53 -0500 Subject: [Distutils] Attempted summary of version number thread In-Reply-To: <199812141646.RAA21126@dirac.cnrs-orleans.fr>; from Konrad Hinsen on Mon, Dec 14, 1998 at 05:46:45PM +0100 References: <19981211184904.A27438@cnri.reston.va.us> <199812141646.RAA21126@dirac.cnrs-orleans.fr> Message-ID: <19981214123853.B27675@cnri.reston.va.us> Quoth Konrad Hinsen, on 14 December 1998: > The criterion is that it must be clear both for human users and for > installation programs > 1) which of two versions is newer > 2) if two versions are compatible, i.e. use the same APIs OK, just to keep everyone on their toes, I'm going to take the anarchists' side here. I have said several times that I am against *imposing* semantics of version numbering, and if I read Konrad right here that's what he's suggesting: a certain part of the version number implies "API version". I don't think this'll work, and not just because Greg and Mark and Marc-Andre (and probably lots of other people) don't like it. I do kind of like it, being a control-freak myself, but I don't think it'll work! Most developers just don't think *that* carefully about what their version number means. Or they might think carefully about it, but come to different conclusions: if you completely rewrite your software, you might feel that justifies bumping from 1.x to 2.0 -- even with full API compatability. We simply cannot mandate these sorts of details on something like the world of all Python modules. Perhaps an "API version number" belongs in the package meta-data, along with the multitude of variables that John Skaller considers part of the "version" (again, I think John is talking about something much bigger than "version numbers", and I think what he's talking about can be considered package meta-data). While I think it's a worthy goal to try to automatically handle all forms of module inter-dependency, I also think it's a vast swamp with no discernible exit. The goals for the distutils should be modest: be able to compare version numbers and express module dependencies in terms of version numbers. Expressing compatability on the many axes John proposes, plus the "API version", would be very difficult. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19981211184904.A27438@cnri.reston.va.us> Message-ID: <13941.19733.211781.795180@weyr.cnri.reston.va.us> Greg Ward writes: > control freaks (me, Fred, and Konrad -- at least you two said you agree ... > So: do any of you object to the above characterizations? I'd be Greg, I can live with being a control freak; I hear this at home, too. ;) As far as the relationship with the rest of the distutils deliverables is concerned, there's a lot which is not related to the versioning issue, and there's no reason to let versioning delay other aspects of the project. All of the build/install/package code is independent of versioning. Versioning is important with regard to dependencies, both for dependency declarations for a package and availability analysis for locating dependency implementations. For this, some sort of reqiures/ provides model with version comparisons may be appropriate. As far as this goes, comparability of version numbers is more important than a specific scheme, so I'm inclined to accept that multiple versioning "schemes" be possible and have each be identifiable by name. A default that is somewhere between the "control freak" model and Greg Stein's variant should be fairly usable. Mark Hammond's "build number" approach seems to fit the syntactic model just fine; he's only using the first number, and it can get large. But that's ok for our immediate purposes, and compatibility analysis will probably require explicit statements of compatibility of various aspects (API, back-end data, whatever makes sense for each) on a per-release basis anyway. Using a requires/provides model allows each package to "provide" a number of "aspects" which can be independently versioned, as well. So each data format or API can be versioned independently. My frobnitz package can then do the following: requires xml-omnibus-package 0.6 provides frobnitz-data 1.0 provides frobnitz-API 1.0 provides frobnitz-API 1.1 -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From skaller@maxtal.com.au Tue Dec 15 00:55:05 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 15 Dec 1998 10:55:05 +1000 Subject: [Distutils] Proposal: C/C++ compiler identification Message-ID: <1.5.4.32.19981215005505.00f7b3d4@triode.net.au> At 10:27 13/12/98 -0500, Guido van Rossum wrote: >> The proposal is for the Python build process to create run time accessible >> identification data whose purpose is to facilitate run time building of >> >> a) dynamically loadable binary python module objects *.so, .dll etc) >> b) external (shell) application tools (.exe etc) >> >> This proposal does not fix the format of the information. > >Note that (for Unix at least) all this info is already being gathered >by the configure script, and stored in the various Makefile. The info >also ends up being installed (look in >/usr/local/lib/python1.5/config/). So, it would be relatively easily to generate a standard python module? >For Windows, since the only supported compiler is VC++, it could be >hardcoded. However, there's a very serious problem on Windows with >Interscript's approach of invoking the compiler at run-time: most >users don't have this compiler. I wouldn't have call that a serious problem with the approach. It is a difficulty :-) The current implementation doesn't handle failure gracefully. But, after all, it is just python script: more work needs to be done to fetch or use pre-built binaries, and to detect _whether_ a compiler is available. Etc. >I realize that gcc is available for >free, but I think it isn't compatible with VC++. As far as I know, >VC++ is required if you want to use any of Mark Hammond's stuff (COM >and MFC). I don't know if Tcl/Tk can be used (without recompilation) >from a gcc-based app, and I don't know how easy it would be to >recompile. (Does gcc on Windows have support for Win32 APIs at all?) To the best of my knowledge, CygWin will allow a Python build on Windows: it supplies bash, ecgs version of C/C++, and some other utilities. It also supplies Tcl/Tk/Tix. And it allows standard Windows API calls as well. The advantage of the Interscript approach for Windows users is that SOME of them do have a C/C++ compiler. So instead of just ONE person supporting Windows, everyone with a compiler would be able to build and contribute binaries. In other words, I can't see how the interscript approach makes anything _worse_ for Windows users. I do agree the current mechanism is inadequate! Hopefully, people can suggest ways to improve the mechanism. Note also: JPython works on Windows! So Mark Hammonds isn't the only build of Python that works on Windows :-)) ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 15 00:55:13 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 15 Dec 1998 10:55:13 +1000 Subject: [Distutils] Freeze on topic for this SIG? Message-ID: <1.5.4.32.19981215005513.00f730d4@triode.net.au> At 12:42 14/12/98 +1100, Mark Hammond wrote: >Im really just looking for a place to discuss enhancements to Freeze >itself. Specifically, Guido, Jack Jansen, Just and myself have had >very very preliminary discussions on a new "import architecture" for >Python itself. Why not, by private email, create a 'requirements and analysis' document: what is wrong with freeze? [I've never used it] What is desirable? How can it be done, roughly? [BTW: I know nothing of freeze but I'm very interested in a new import architecture!!!!] The post that as a discussion starter? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 15 00:55:18 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 15 Dec 1998 10:55:18 +1000 Subject: [Distutils] Attempted summary of version number thread Message-ID: <1.5.4.32.19981215005518.00f65530@triode.net.au> >Perhaps an "API version number" belongs in the package meta-data, along >with the multitude of variables that John Skaller considers part of the >"version" (again, I think John is talking about something much bigger >than "version numbers", and I think what he's talking about can be >considered package meta-data). Yes: but isn't the point here that version numbers are 'just another piece of meta-data'? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 15 00:55:16 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 15 Dec 1998 10:55:16 +1000 Subject: [Distutils] Attempted summary of version number thread Message-ID: <1.5.4.32.19981215005516.00f7dd60@triode.net.au> >So perhaps the following scheme might be acceptable: Version numbers >start with an API version number, then any combination of letters and >punctuation that is likely to make sense to a human reader, followed >by an optional "alpha" or "beta" indicator. The first number would >change only if a package cannot be used as a drop-in replacement for >its predecessors. Is that something the "anarchists" could agree on? Here is an idea of an alternative: every package supplies a package identification script, which is a Python script that builds some data structures and a function: isupgradeof(pkg) to discover if a package is an upgrade of another, the client executes BOTH package identification scripts, and then calls BOTH isupgradeof functions (each on the other package). A package is an upgrade of another if one of the functions says so, otherwise not. It may be worth considering adding this kind of functionality to the __init__ module of a package. Just an idea. Remember, we all know some OO, and we know it is better to represent abstractions with methods than data ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 15 00:55:14 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 15 Dec 1998 10:55:14 +1000 Subject: [Distutils] Attempted summary of version number thread Message-ID: <1.5.4.32.19981215005514.00f772e8@triode.net.au> At 17:46 14/12/98 +0100, Konrad Hinsen wrote: >> the anarchists, the middle-of-the-roaders, and John Skaller. The >> control freaks (me, Fred, and Konrad -- at least you two said you agree >> with me!) are in favour of prescribing *syntax* for version numbers and >> *suggesting* semantics. The *proposed* syntax and semantics that I > >Right, and it seems worth pointing out again why, or rather to what >extent, the interpretation of version numbers should be unified. > >The criterion is that it must be clear both for human users and for >installation programs >1) which of two versions is newer You are making a hidden assumption. That the development process in linear. A comparator for packages is at best a partial order: some packages may be linearly ordered, but it is far too much of a constraint on developers to require this. Therefore, 'newer' is not an appropriate requirement. A more appropriate comparator is a partial order 'is compatible with'; meaning that you can safely upgrade. It also follows that 'numbers' are an inadequate representation of upgrade compatibility. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From guido@CNRI.Reston.VA.US Tue Dec 15 15:15:23 1998 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Tue, 15 Dec 1998 10:15:23 -0500 Subject: [Distutils] Proposal: C/C++ compiler identification In-Reply-To: Your message of "Tue, 15 Dec 1998 10:55:05 +1000." <1.5.4.32.19981215005505.00f7b3d4@triode.net.au> References: <1.5.4.32.19981215005505.00f7b3d4@triode.net.au> Message-ID: <199812151515.KAA00855@eric.cnri.reston.va.us> [someone] > >> The proposal is for the Python build process to create run time accessible > >> identification data whose purpose is to facilitate run time building of > >> > >> a) dynamically loadable binary python module objects *.so, .dll etc) > >> b) external (shell) application tools (.exe etc) > >> > >> This proposal does not fix the format of the information. [me] > >Note that (for Unix at least) all this info is already being gathered > >by the configure script, and stored in the various Makefile. The info > >also ends up being installed (look in > >/usr/local/lib/python1.5/config/). [John Skaller] > So, it would be relatively easily to generate a standard > python module? Without more context I don't know what you mean by "generate a standard Python module". If you are asking whether it is easy to compile a Python extension given just its source, yes: just copy /usr/local/lib/python1.5/config/Makefile.pre.in in the directory where the extension lives, write a one-line Setup file containing the module name and the source file name, and execute these two shell commands: make -f Makefile.pre.in boot make This assumes that "python" is on your $PATH. > >For Windows, since the only supported compiler is VC++, it could be > >hardcoded. However, there's a very serious problem on Windows with > >Interscript's approach of invoking the compiler at run-time: most > >users don't have this compiler. > > I wouldn't have call that a serious problem with the > approach. It is a difficulty :-) > > The current implementation doesn't handle failure > gracefully. But, after all, it is just python script: more work > needs to be done to fetch or use pre-built binaries, > and to detect _whether_ a compiler is available. Etc. > > >I realize that gcc is available for > >free, but I think it isn't compatible with VC++. As far as I know, > >VC++ is required if you want to use any of Mark Hammond's stuff (COM > >and MFC). I don't know if Tcl/Tk can be used (without recompilation) > >from a gcc-based app, and I don't know how easy it would be to > >recompile. (Does gcc on Windows have support for Win32 APIs at all?) > > To the best of my knowledge, CygWin will allow a Python > build on Windows: it supplies bash, ecgs version of C/C++, > and some other utilities. It also supplies Tcl/Tk/Tix. And it > allows standard Windows API calls as well. I know all that, but it requires building Python with CygWin. Do you realize that most Python users on Windows download the installer from python.org? That doesn't contain the C sources (they don't have a C compiler so what good would it do them) and has a PYTHON15.DLL created with VC++. Oh well, I guess that if your software requires them to have a compiler, you might as well require them to grab the sources and rebuild Python -- but you must realize that this will put off most potential users... > The advantage of the Interscript approach for Windows > users is that SOME of them do have a C/C++ compiler. So instead > of just ONE person supporting Windows, everyone with a compiler > would be able to build and contribute binaries. *IF* they can produce compatible binaries. > In other words, I can't see how the interscript approach > makes anything _worse_ for Windows users. I do agree the current > mechanism is inadequate! Hopefully, people can suggest ways > to improve the mechanism. > > Note also: JPython works on Windows! So Mark Hammonds > isn't the only build of Python that works on Windows :-)) I never said that. --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@cnri.reston.va.us Tue Dec 15 15:36:20 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 15 Dec 1998 10:36:20 -0500 Subject: [Distutils] Proposal: C/C++ compiler identification In-Reply-To: <1.5.4.32.19981213123140.00f6e05c@triode.net.au>; from John Skaller on Sun, Dec 13, 1998 at 10:31:40PM +1000 References: <1.5.4.32.19981213123140.00f6e05c@triode.net.au> Message-ID: <19981215103620.A28098@cnri.reston.va.us> *This is not a flame* Please do not interpret it as such! Quoth John Skaller, on 13 December 1998: > This proposal is intended as a detail of the general requrements > document for distutils. > > The proposal is for the Python build process to create run time accessible > identification data whose purpose is to facilitate run time building of > > a) dynamically loadable binary python module objects *.so, .dll etc) > b) external (shell) application tools (.exe etc) > > This proposal does not fix the format of the information. I have a couple of comments/questions on this -- mostly relating to the fact that I don't understand what you're talking about. First, when you talk about a "requirements document" for distutils, I can't help but be reminded that I've already written such a thing, posted it to the web, and asked for comments on this list. The only comments I got were a reminder to mention cross-platform friendliness and to add a blurb on dependency checking. That is, everyone on the list seemed pretty happy with the my proposed requirements document and just corrected a few lapses on my part. So, first question: are you proposing to throw out my requirements document and replace it with this? Or are you suggesting additions/modifications to mine? Next, you speak of the "Python build process" creating "run time accessible identification data". The first phrase gives me the impression that you're talking about building Python itself (and its library), which really isn't the province of the distutils suite. (If this SIG produces usable code soon enough, Guido might find it useful to rework how the standard library is built for Python 1.6. That's the only connection I see between the two.) I have no idea what you mean by "run time accessible identification data". Or rather, I can think of a half-dozen things you might mean. Please explain, or rephrase in a more conventional way. > --- > I would like to consider a class instance, with one instance > for each supported compiler/kind combination. > > I recommend the Python make process attempt to provide > access to C, C++, and Java compilers. > > A binary build, such as Mark Hammonds Windows build, > cannot be pre-configured for the client's system. > > Note that interscript contains modules which compile > C/C++ code. At present, the modules have to rewritten > by the client (unless they're running RedHat Linux :-) > > It is not clear to me if an _executable_ representation > of the above requirement is appropriate. If not, > at least the information required to create one > should be made available. > > The representation must be editable by the client > somehow. Now I'm really confused. The above statements are a mix of requirements/suggestions (providing access to C, C++, and Java compilers), design proposals (one class instance per supported compiler/kind combination), and random comments on existing systems (Mark Hammond's Windows build, your Interscript). John, *please* try to have your say in an organized fashion, using terms familiar to the whole audience. Remember, we're *not* all on the same wavelength -- I can't shake the impression that you're talking about something entirely different from the proposal we fleshed out at the Developer's Day session. Not having been there, obviously, places you at a bit of a disadvantage, but the summary I posted, along with the requirements and proposed interface, should give anyone a good flavour of what was decided in that session. If you have constructive comments or suggestions, we're open to them. If you want to tear the whole edifice down and replace it with a gleaming tower of ivory perfection, then you'll have to be a lot more coherent. (Please, *not* more voluminous: more *coherent*.) Again, *this is not a flame*, it is a plea for cogent and lucid exposition. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From jmrober1@ingr.com Tue Dec 15 20:02:17 1998 From: jmrober1@ingr.com (Robertson, Joseph M) Date: Tue, 15 Dec 1998 14:02:17 -0600 Subject: [Distutils] Observations Message-ID: Hi all, I like the scriptable builds stuff. Just like Instalit of the old days. If fact I'd suggest we use a similar method. The build script tags data such as whether to check file dates, versions, etc when installing; what compression type to use, etc, etc. Question: How does your system install a working python program if the target has no python whatsoever on it? Thanks Joe Robertson 256-730-8216 jmrober1@ingr.com jmrobert@ro.com From skaller@maxtal.com.au Tue Dec 15 20:38:32 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 16 Dec 1998 06:38:32 +1000 Subject: [Distutils] Proposal: C/C++ compiler identification Message-ID: <1.5.4.32.19981215203832.00f79ba8@triode.net.au> >[John Skaller] >> So, it would be relatively easily to generate a standard >> python module? > >Without more context I don't know what you mean by "generate a >standard Python module". Sorry: the intent is to generate a python module containing enough information to compile any 'standard' C extension to python without makefiles and on any platform. Note: the assumption is that the extension builder is the same system as that which compiled Python. So a C compiler is available by assumption. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 15 20:38:34 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 16 Dec 1998 06:38:34 +1000 Subject: [Distutils] Proposal: C/C++ compiler identification Message-ID: <1.5.4.32.19981215203834.00f709f4@triode.net.au> At 10:36 15/12/98 -0500, Greg Ward wrote: >*This is not a flame* Please do not interpret it as such! > >Quoth John Skaller, on 13 December 1998: >> This proposal is intended as a detail of the general requrements >> document for distutils. >So, first question: are you proposing to throw out my requirements >document and replace it with this? Or are you suggesting >additions/modifications to mine? An addition. I though that was clear by describing it as a detail. I'm proposing that, under one of your headings, to add a subheading. OK? >Next, you speak of the "Python build process" creating "run time >accessible identification data". The first phrase gives me the >impression that you're talking about building Python itself (and its >library), which really isn't the province of the distutils suite. Yes, I AM taking about building Python itself. Sigh. It it that hard to understand: when python is built, it know the name of the compiler. I want a module: import build_compiler cc = build_compiler.cc which is the name of the compiler used to build Python, so I can invoke that same compiler to build an extension. And all the other info needed, such as where various libraries are, what -DMACRO options are needed, etc, (If this SIG produces usable code soon enough, Guido might find it useful to >rework how the standard library is built for Python 1.6. That's the >only connection I see between the two.) Then you are missing the link: the build processes know how to build C. I want the build process to put that information into a standard module. So I can get it out, and call up the compiler to make a C extension FROM PYTHON. >I have no idea what you mean by "run time accessible identification >data". Or rather, I can think of a half-dozen things you might mean. >Please explain, or rephrase in a more conventional way. have I explained now? This is the key problem for distutils BECAUSE it REQUIRES Guido to modify the build process to create the information needed to call up the compiler from python script. At least, this is by far the most reliable way to get the information: all that autoconfig and makefile and setup stuff has the information, and it is needed to be able to build C code later as well. That is the connection. Distutils cannot work if Guido doesn't give the information. SOME of the information is already available in some form: the name of the compiler is already encoded in the version string. But it isn't enough to actually emulate the compilation and linkage processes the same way Setup builds things. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gward@cnri.reston.va.us Tue Dec 15 21:17:56 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 15 Dec 1998 16:17:56 -0500 Subject: [Distutils] Enough on software versions Message-ID: <19981215161756.A28684@cnri.reston.va.us> Hi all: I'd like to declare the "Module version numbers" thread a CLOSED subject. I may spend some time in the near future writing a module to provide classes for version numbers in the anarchist/realist vein proposed by Greg Stein, and in the control-freak/idealist vein proposed by me. If you have any further comments, please keep them in private email; I'm curious to hear any opinions that have *not* already been voiced on the list. (Or you can send me a quick "me too" note if you just want your voice heard.) But let's keep that traffic off the list and move on to important things. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Tue Dec 15 21:33:04 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 15 Dec 1998 16:33:04 -0500 Subject: [Distutils] Proposal: C/C++ compiler identification In-Reply-To: <1.5.4.32.19981215203834.00f709f4@triode.net.au>; from John Skaller on Wed, Dec 16, 1998 at 06:38:34AM +1000 References: <1.5.4.32.19981215203834.00f709f4@triode.net.au> Message-ID: <19981215163304.D28107@cnri.reston.va.us> Quoth John Skaller, on 16 December 1998: > > Yes, I AM taking about building Python itself. > Sigh. It it that hard to understand: when python is built, > it know the name of the compiler. [...deletia...] > I want the build process to put that > information into a standard module. So I can get it out, > and call up the compiler to make a C extension FROM PYTHON. A-ha! Thank you -- now it comes clear. As it happens, this was one of the things discussed at the Developer's Day session which I failed to mention in the online summary. Mea culpa. I also forgot to mention it on the requirements list. If I were more weasely, I could squirm out of it by saying "Well, this just follows from the requirement to easily build extension modules from source" or, "That's not a requirement, it's an implementation detail". But I'm not a weasel, and I think at some point I meant to mention this in the requirements document. So I've added it. I also changed the wording of the point that immediately leads into it; here're the changed and new bullet:
  • It must be possible (and preferably easy) to instead download and build from a source distribution. (Necessary for people who don't happen to be using one of the major platforms, or who don't trust built distributions.)
  • To aid in installing all modules, and in building extension modules from C/C++ source, there must be a standardized way to get Python's configuration data (such as compiler, compiler flags, platform, default library directory, etc.). I'm managing to keep implementation details out of the requirements list, though. In particular, my proposal is for a new 'sys.config' module which would contain everything you might need to know about how Python was configured. The most important items are the compiler, compiler flags, platform, and library installation directories. For an example of the degree of flaming overkill possible in this sort of module, try this (assuming you have a fairly recent [>= 5.003 I think] version of Perl installed): perl -MConfig -e 'map { print "$_: $Config{$_}\n" } sort keys %Config;' You'll note that Perl's Config module stores *every* piece of information, gleaned, gathered, or queried from the user while running Perl's 'Configure' script. I don't think Python's sys.config need be *quite* so exhaustive, but I can personally vouch that the Config module has more utility than simply knowing where to install modules. (F'r instance, if you're wondering "Gee, is Perl's 'syscall' available on this platform?", or, "Was this Perl built with stdio or sfio?", then you can answer your questions with Config. That's pretty cool.) So, perhaps this is a good place to start a new flamewar^H^H^H^H^H^H^H^H discussion. Does anyone violently disagree with the concept of adding a sys.config module? And what should go in it? Here's a very rough start (loosely based on my long experience with building Perl and my very brief experience with building Python, and a bit of poking through Python's makefiles): cc C compiler used to build Python cppflags C preprocessor flags used to build Python (-I, -D, -U) optimize optimization/debugging flags used to build Python ccsharedflags C compiler flags required to build shared objects extra_ccflags any other C compiler flags used to build Python ccflags all compiler flags used to build Python: cppflags + optimize + extra_ccflags ccextflags all compiler flags used to build Python extensions: cppflags + optimize + ccsharedflags + extra_ccflags ldlibdirs -L flags used to build Python ldlibs -l flags used to build Python extra_ldflags any other needed linker flags ldsharedflags linker flags required to build shared objects ldflags all linker flags used to build Python: ldlibdirs + ldlibs + extra_ldflags ldextflags linker flags used to build Python extensions: ldlibdirs + ldlibs + ldsharedflags + extra_ldflags platform hardware/os/vendor (or something along those lines) install_bindir directory to install binary executables install_scriptdir directory to install non-binary executables install_includedir directory to install C header files install_confincludedir directory to install platform- and build-specific C header files install_libdir main Python library directory install_sitelibdir directory for site-specific modules Obviously this is very sketchy. A proper proposal would have examples and/or defaults (eg. "What do I get if I just do "configure ; make ; make install" on a standard POSIX-compliant system?). Any replies should be implicitly prefixed, "Greg, you are a newbie and don't know what you're talking about" because it's true. *But* I have configured and built Perl more times than I care to admit, and I know what works and doesn't work there. All you long-time Python hackers surely know what works and doesn't work *here*, so together we should be able to cobble something together. Go to it... Greg PS. oh, I guess I should briefly address John's remaining points: > This is the key problem for distutils BECAUSE it > REQUIRES Guido to modify the build process to create > the information needed to call up the compiler > from python script. At least, this is by far the most > reliable way to get the information: all that autoconfig > and makefile and setup stuff has the information, > and it is needed to be able to build C code later as well. > > That is the connection. Distutils cannot work > if Guido doesn't give the information. I agree, but I hadn't considered it the *key* problem. After all, the first version of distutils will have to work with 1.5.2; one possibility is (hack, blech, gaack) to chew up config.status and/or Makefile from the Python source directory. Obviously, having sys.config available in the standard library is a far nicer solution in every respect. -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal@lemburg.com Wed Dec 16 10:41:21 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Dec 1998 11:41:21 +0100 Subject: [Distutils] Proposal: standard sysconfig module References: <1.5.4.32.19981215203834.00f709f4@triode.net.au> <19981215163304.D28107@cnri.reston.va.us> Message-ID: <36778E51.2A029EFF@lemburg.com> Greg Ward wrote: > > In particular, my proposal is for a new 'sys.config' > module which would contain everything you might need to know about how > Python was configured. The most important items are the compiler, > compiler flags, platform, and library installation directories. Good idea, though the module should probably be named sysconfig (without the dot). I'd suggest sticking in the config.h file as dictionary, e.g. {'HAVE_TZNAME':1, 'HAVE_TM_ZONE':0,...} and the information from Makefile (the one in /usr/local/lib/python1.5/config) parsed and stored in a class instance (using the same names as in the Makefile for clarity), e.g. sysconfig.make.CC C compiler .OPT compiler flags etc. Rather than discuss which things to include or not, just put all the information in there you can find by simply parsing Makefile. Note that the parsing mechanism should also allow for fairly decent $(VARIABLE) expansion. .../Setup should also go into a dictionary: {'cPickle.c':'cPickle.c',...} Why dictionaries for config.h and Setup and an instance for make ? Because the Makefile entries will not change to often and the class can provide reasonable defaults. config.h and Setup change a lot between versions and maintaining them as instances would probably cause lot's of compatibility headaches. The remaining problem (?) is where to find the three files. Looking at my Makefile they seem to be at: sys.prefix + '/lib/python' + sys.version[:3] + '/config/' and sys.prefix + '/include/python' + sys.version[:3] + '/' But the install dirs are changeable during configure... we might need some help from Guido here, e.g. the sys module could include the information: sys.configdir, sys.incldir and sys.libdir. Note that the configuration for WinXX is hard-coded, so the sysconfig module could also provide the data on those platforms. Together with the other information in sys and os.environ this should provide a pretty complete picture of how the machine is setup. -- Marc-Andre Lemburg Y2000: 380 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From gstein@lyra.org Wed Dec 16 11:17:58 1998 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Dec 1998 03:17:58 -0800 Subject: [Distutils] Proposal: standard sysconfig module References: <1.5.4.32.19981215203834.00f709f4@triode.net.au> <19981215163304.D28107@cnri.reston.va.us> <36778E51.2A029EFF@lemburg.com> Message-ID: <367796E6.1AA05494@lyra.org> M.-A. Lemburg wrote: > > Greg Ward wrote: > > > > In particular, my proposal is for a new 'sys.config' > > module which would contain everything you might need to know about how > > Python was configured. The most important items are the compiler, > > compiler flags, platform, and library installation directories. > > Good idea, though the module should probably be named sysconfig > (without the dot). > > I'd suggest sticking in the config.h file as > dictionary, e.g. {'HAVE_TZNAME':1, 'HAVE_TM_ZONE':0,...} and > the information from Makefile (the one in /usr/local/lib/python1.5/config) > parsed and stored in a class instance (using the same names as > in the Makefile for clarity), e.g. > > sysconfig.make.CC C compiler > .OPT compiler flags > etc. > > Rather than discuss which things to include or not, just > put all the information in there you can find by simply parsing > Makefile. Note that the parsing mechanism should also allow > for fairly decent $(VARIABLE) expansion. > > ../Setup should also go into a dictionary: {'cPickle.c':'cPickle.c',...} > > Why dictionaries for config.h and Setup and an instance for > make ? Because the Makefile entries will not change to often > and the class can provide reasonable defaults. config.h and > Setup change a lot between versions and maintaining them as > instances would probably cause lot's of compatibility > headaches. > > The remaining problem (?) is where to find the three files. Looking at > my Makefile they seem to be at: > ... Actually, at the discussion, we planned on compiling them right into the interpreter. That would prevent version skews between a compiled Python and the libraries. However, now that I'm thinking about it... screw that. If "import sysconfig" picks up the wrong version, then you have plenty of other issues to deal with. I'd much rather see a .py file that a bunch of stuff compiled into the interpreter. I'd also recommend ignoring all the dictionary vs instance stuff. Just have a bunch of assignments dropped into a sysconfig.py module. IMO, it would look a lot like FCNTL.py (h2py output files). No need to get fancy with defaults and stuff. Just write the dumb file out. I agree with Marc on "just put it all in there". No reason to sort it out. Cheers, -g -- Greg Stein, http://www.lyra.org/ From skaller@maxtal.com.au Wed Dec 16 12:29:22 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 16 Dec 1998 22:29:22 +1000 Subject: [Distutils] Proposal: C/C++ compiler identification Message-ID: <1.5.4.32.19981216122922.00f7c0dc@triode.net.au> At 16:33 15/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 16 December 1998: >> I want the build process to put that >> information into a standard module. So I can get it out, >> and call up the compiler to make a C extension FROM PYTHON. > >A-ha! Thank you -- now it comes clear. If I were more weasely, I could >squirm out of it by saying "Well, this just follows from the requirement >to easily build extension modules from source" or, "That's not a >requirement, it's an implementation detail". You could, you'd be right in both cases, and you cannot be expected to do everything: which is why I wrote the article. :-) >But I'm not a weasel, You're not god. You started this list to get help from others. >and I think at some point I meant to mention this >in the requirements document. So I've added it. Great! Thanks! >I'm managing to keep implementation details out of the requirements >list, though. In particular, my proposal is for a new 'sys.config' >module which would contain everything you might need to know about how >Python was configured. YES!!!! I agree! >The most important items are the compiler, >compiler flags, platform, and library installation directories. Can I recommend you look at the interscript compilers module again? Instead of storing compiler flags, etc, you need an actual command that does the compilation. I.e. a python function that builds a command string and os.system's it. I don't think that 'data' can store this information, there are too many options. (Store the data as well!) >For an example of the degree of flaming overkill possible in this sort >of module, try this > perl -MConfig -e 'map { print "$_: $Config{$_}\n" } sort keys %Config;' >You'll note that Perl's Config module stores *every* piece of >information, gleaned, gathered, or queried from the user while running >Perl's 'Configure' script. I don't think Python's sys.config need be >*quite* so exhaustive, I agree. After all, we can upgrade it later. The important thing is to get the module into the distribution. That way people can try it, and we'll find out what is missing. Anyone on this list have a Next machine?? >Here's a very rough start >(loosely based on my long experience with building Perl and my very >brief experience with building Python, and a bit of poking through >Python's makefiles): This is too Unixy. > cc C compiler used to build Python > cppflags C preprocessor flags used to build Python (-I, -D, -U) > optimize optimization/debugging flags used to build Python > ccsharedflags C compiler flags required to build shared objects > extra_ccflags any other C compiler flags used to build Python > ccflags all compiler flags used to build Python: > cppflags + optimize + extra_ccflags > ccextflags all compiler flags used to build Python extensions: > cppflags + optimize + ccsharedflags + extra_ccflags > > ldlibdirs -L flags used to build Python > ldlibs -l flags used to build Python > extra_ldflags any other needed linker flags > ldsharedflags linker flags required to build shared objects > ldflags all linker flags used to build Python: > ldlibdirs + ldlibs + extra_ldflags > ldextflags linker flags used to build Python extensions: > ldlibdirs + ldlibs + ldsharedflags + extra_ldflags I do not think _any_ of these flags, etc, are required individually. Nor do I see how they can possibly be enough. You are making assumptions about how an arbitrary compiler is invoked. What you need is a _function_ which compiles a c file for a shared library, and another which links them. In other words, you just need a string with some 'slots' in it for the filenames, and which has all the flags already set in the right places. The function is a generalisation of that. Have a look at the compilers module I posted here. > platform hardware/os/vendor (or something along those lines) Yes. Something like that. This is necessarily crude information. > install_bindir directory to install binary executables > install_scriptdir directory to install non-binary executables > install_includedir directory to install C header files > install_confincludedir directory to install platform- and > build-specific C header files > install_libdir main Python library directory > install_sitelibdir directory for site-specific modules Each of these things is what i call an 'install point'. It isn't clear that the list is enough: clients may want to organise things differently for different modules. For example, the place PYTHON puts scripts is definitely NOT where I want extension packages: I want them in the site-packages directory. This is tough. I think your list is a good start. Please add: install_htmldir directory to put HTML doco install_mandir directory for man pages (does anyone provide them?) >Obviously this is very sketchy. An excellent sketch though. Why? because it is concrete enough to be argued about! ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gward@cnri.reston.va.us Wed Dec 16 13:33:28 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 16 Dec 1998 08:33:28 -0500 Subject: [Distutils] Observations Message-ID: <19981216083328.A28888@cnri.reston.va.us> Quoth Robertson, Joseph M, on 15 December 1998: > I like the scriptable builds stuff. Just like Instalit of the old days. If > fact I'd suggest we use a similar method. The build script tags data such > as whether to check file dates, versions, etc when installing; what > compression type to use, etc, etc. What is "Instalit"? -- I've heard the name pop up once or twice in the last week or so, but never heard of it before. Remember, we're not all on the same wavelength here! > Question: How does your system install a working python program if the > target has no python whatsoever on it? That's something I for one haven't considered. It's not currently on the plate of the SIG, but should probably be considered for the future. Here's one possible long-term roadmap: * first, be able to build, distribute, and install Python modules: this will make it easy to extend an existing Python installation * next, be able to install a complete Python application (scripts + modules): this will mainly involve doing the module installation thing plus dependency tracking (ie. what *other* modules are required by this application) * finally, be able to distribute a Python application to systems that don't have Python at all All three are worthy goals indeed, and "distribution utilities" certainly covers all three. But there's a clear order to be followed, and right now we're concentrating on the first job. The second should follow without a great deal of difficulty, and third could either be a snap (just install one of the large number of pre-built Python distributions) or a morass. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Wed Dec 16 14:03:59 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 16 Dec 1998 09:03:59 -0500 Subject: [Distutils] Access to Python config info Message-ID: <19981216090359.A28921@cnri.reston.va.us> OK, we seem to have a better degree of consensus on this than on that version number thing. Good -- hopefully we won't scare anyone off with an extended argument. On the superficial front, Marc-Andre hasn't convinced me that sysconfig is a better name than sys.config. (On the other hand, I never said whether I thought it should be a sub-module of sys, or a [potentially ambiguous] module under a package called 'sys'.) My rationale for calling it sys.config is that sys is "Python system stuff"; sys.config would basically be more of the same, but probably a *lot* more -- so I don't think it should clutter up the well-known, well-understood 'sys' module. One other possible name is, of course, 'config', but then the Perl guys might sue us for copyright infringement. >joke!< Marc-Andre also said: > Rather than discuss which things to include or not, just > put all the information in there you can find by simply parsing > Makefile. Note that the parsing mechanism should also allow > for fairly decent $(VARIABLE) expansion. No! At least not in the long term: as Greg Stein pointed out, the config module (whatever it gets called) should just be generated when Python is built. Or were you talking about the near term, where we will need a distutils release that works with Python 1.5.2 and is allowed all manner of grotesque hackery to work around the lack of a config module, such as digging up and parsing Python's config.h, Makefile, and Setup? [Marc-Andre again] > But the install dirs are changeable during configure... we might > need some help from Guido here, e.g. the sys module could > include the information: sys.configdir, sys.incldir and sys.libdir. Now you're cluttering 'sys' up with stuff in the yet-to-be-named config module. These directories should just be accessible variables in sys.config, eg. sys.config.install_bindir sys.config.install_scriptdir sys.config.install_libdir etc. Greg Stein pointed out: > Actually, at the discussion, we planned on compiling them right into the > interpreter. That would prevent version skews between a compiled Python > and the libraries. > > However, now that I'm thinking about it... screw that. If "import > sysconfig" picks up the wrong version, then you have plenty of other > issues to deal with. I'd much rather see a .py file that a bunch of > stuff compiled into the interpreter. I'm torn on this one. Building into Python prevents version skew, keeping it separate means it's easily readable without writing code. Both are compelling arguments. Greg has given one voice in favour of a separate .py file... anyone else? [Greg S. again] > I'd also recommend ignoring all the dictionary vs instance stuff. Just > have a bunch of assignments dropped into a sysconfig.py module. IMO, it > would look a lot like FCNTL.py (h2py output files). No need to get fancy > with defaults and stuff. Just write the dumb file out. I (almost) completely agree. If we put *everything* from config.h into the config module, it might be nice to store it separately (eg. in a hash). But the stuff from Makefile and Setup should just be dumped in flat as variables. Are there other sources of information we should be worrying about? (Eg. does config.cache have anything valuable that doesn't wind up in config.h or Makefile?) Finally, John Skaller respond to my sketch of what should be in the config module: > This is too Unixy. Absolutely true. I never claimed to be anything but a Unix bigot. Two rebuttals though: * if platform X supports running the compiler from a command line or a script, surely that compiler can take some arguments... and surely some of those arguments will deal with optimization/debugging, some will deal with preprocessor settings, and some are just plain miscellaneous * splitting the flags up into all those categories can be useful, but it could be that I'm confusing the Python build process with the extension module build process. Perhaps the way to build an extension should be carved in stone when Python itself is built, and extension builders shouldn't be allowed to muck with it. If so, then John is absolutely right about just calling a function to invoke the compiler for such-and-such a situation. Gee, I guess that second one wasn't much of a rebuttal. So much for my combative spirit. ;-) > An excellent sketch though. Why? because it is concrete > enough to be argued about! Thank you! that was the whole idea; I hate abstract arguments (the only courses I came close to flunking in university were philosophy and psychology...). And I have a nasty tendency to attack anyone who brings up airy-fairy concepts with *something* concrete to explain his ideas. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From akuchlin@cnri.reston.va.us Wed Dec 16 14:13:16 1998 From: akuchlin@cnri.reston.va.us (Andrew M. Kuchling) Date: Wed, 16 Dec 1998 09:13:16 -0500 (EST) Subject: [Distutils] Access to Python config info In-Reply-To: <19981216090359.A28921@cnri.reston.va.us> References: <19981216090359.A28921@cnri.reston.va.us> Message-ID: <13943.48793.185434.288473@amarok.cnri.reston.va.us> Greg Ward writes: >On the superficial front, Marc-Andre hasn't convinced me that sysconfig >is a better name than sys.config. (On the other hand, I never said >whether I thought it should be a sub-module of sys, or a [potentially >ambiguous] module under a package called 'sys'.) My rationale for A lot of Python programs import sys, more than will need sys.config, so there seems no reason to force all those programs to import another .py file that they're aren't going to actually use. On the other hand, if the config information winds up stored in the Python binary, then sys.config will just add a few dictionary entries, so it might as well be there. Alternatively, sys.config could be some sort of magic object that only imports the .py if it's actually accessed, but that's an ugly hack. I'd sum it up as: if it goes in the Python binary, sys.config is OK. If it goes in an external .py file, sys.config is not OK, and I'd vote for sysconfig or some new package. >I'm torn on this one. Building into Python prevents version skew, >keeping it separate means it's easily readable without writing code. >Both are compelling arguments. Greg has given one voice in favour of a >separate .py file... anyone else? Go for the .py file, IMHO. -- A.M. Kuchling http://starship.skyport.net/crew/amk/ The age of chivalry is gone. That of sophisters, economists and calculators has succeeded: and the glory of Europe is extinguished for ever. -- Edmund Burke, _Reflections on The Revolution in France_ From akuchlin@cnri.reston.va.us Wed Dec 16 14:15:59 1998 From: akuchlin@cnri.reston.va.us (Andrew M. Kuchling) Date: Wed, 16 Dec 1998 09:15:59 -0500 (EST) Subject: [Distutils] Note on building on AIX Message-ID: <13943.49161.594450.313098@amarok.cnri.reston.va.us> On the XML-SIG, Konrad Hinsen mentioned the following problem with shared libraries on AIX. I'm forwarding this just to read it into the Distutils record, as something that should also be fixed. -- A.M. Kuchling http://starship.skyport.net/crew/amk/ Quaint though this attitude seems now, it was unquestionably the prevalent one in the nineteenth century, and it would be over-bold to say that it will never return to favour, for the range of human folly is infinite. -- Robertson Davies, _A Voice from the Attic_ ============================= Konrad Hinsen wrote: > > - Some people need patched versions; for example, the standard > > version does not work for AIX. > > What's the patch that's required for AIX? And is there some > reason it can't be rolled into the Makefile.pre.in for 1.5.2? I don't know the system well enough to decide. The problem is that shared library linking is a rather complicated process under AIX, which is handled by two shell scripts. These shell scripts come with the Python distribution (they are "ld_so_aix" and "makexp_aix") and are ultimately installed in the "config" subdirectory of the Python library. But during the compilation of the interpreter and its standard library modules, they reside in the "Modules" subdirectory of the Python distribution. The settings in the configuration reflect this initial situation, not the one after installation. So if you use the standard Makefile.pre.in, the two critical definitions becom LINKCC= $(srcdir)/makexp_aix python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(srcdir)/ld_so_aix $(CC) whereas they should be LINKCC= $(LIBPL)/makexp_aix $(LIBPL)/python.exp "" $(LIBRARY); $(PURIFY) $(CC) LDSHARED= $(LIBPL)/ld_so_aix $(CC) -bI:$(LIBPL)/python.exp I suppose this could be arranged during the installation process, but I don't really want to figure out how that works! ... From friedrich@pythonpros.com Wed Dec 16 14:34:29 1998 From: friedrich@pythonpros.com (Robin Friedrich) Date: Wed, 16 Dec 1998 08:34:29 -0600 Subject: [Distutils] Access to Python config info Message-ID: <199812161437.GAA31736@ns1.lyra.org> > From: Andrew M. Kuchling > I'd sum it up as: if it goes in the Python binary, sys.config > is OK. If it goes in an external .py file, sys.config is not OK, and > I'd vote for sysconfig or some new package. > > >I'm torn on this one. Building into Python prevents version skew, > >keeping it separate means it's easily readable without writing code. > >Both are compelling arguments. Greg has given one voice in favour of a > >separate .py file... anyone else? > > Go for the .py file, IMHO. Another vote for impeac^H^H^H^H^H^Ha separate .py file to store this. Mostly it's better since it allows tweaks to be made easily, (say the installpath changes, or a new compiler), a hundred different little things can happen at a local site. It'll also help during the development phases of this tool to have visibility into what's actually happening. -- Robin K. Friedrich Houston, Texas From jmrober1@ingr.com Wed Dec 16 14:40:29 1998 From: jmrober1@ingr.com (Robertson, Joseph M) Date: Wed, 16 Dec 1998 08:40:29 -0600 Subject: [Distutils] Observations Message-ID: Hi all, This is only day 2 for me so I apologize if I haven't absorbed all your info yet. Instalit was a generalized installation toolkit for DOS/Windows similar to (but more powerful than) InstallShield. I used to work at the company. Alas it is no more, but its salient features were scriptable 'build' process and 'scriptable' install process. The scritpable build process was extremely valuable, for remote or timed builds across the enterprise (IBM used it for this). I'm just gonna hang here for a while and try to grasp the 'big picture'... Joe Robertson 256-730-8216 jmrober1@ingr.com jmrobert@ro.com > -----Original Message----- > From: Greg Ward [SMTP:gward@cnri.reston.va.us] > Sent: Wednesday, December 16, 1998 7:33 AM > To: distutils-sig@python.org > Subject: Re: [Distutils] Observations > > Quoth Robertson, Joseph M, on 15 December 1998: > > I like the scriptable builds stuff. Just like Instalit of the old days. > If > > fact I'd suggest we use a similar method. The build script tags data > such > > as whether to check file dates, versions, etc when installing; what > > compression type to use, etc, etc. > > What is "Instalit"? -- I've heard the name pop up once or twice in the > last week or so, but never heard of it before. Remember, we're not all > on the same wavelength here! From skaller@maxtal.com.au Wed Dec 16 15:36:22 1998 From: skaller@maxtal.com.au (John Skaller) Date: Thu, 17 Dec 1998 01:36:22 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981216153622.00f7680c@triode.net.au> >I'm torn on this one. Building into Python prevents version skew, >keeping it separate means it's easily readable without writing code. >Both are compelling arguments. Greg has given one voice in favour of a >separate .py file... anyone else? I don't see any real choice: it has to be separate AND editable. Because if I download a binary Windows version, the information will be wrong for my system. The Unix version might be wrong too: perhaps I built Python with gcc, but I want to build extensions with ecgs, or with edg. BTW: I see no harm in putting all the configuration options into a dictionary as Marc Andre suggests. But I don't think that is _enough_. >Finally, John Skaller respond to my sketch of what should be in the >config module: >> This is too Unixy. > >Absolutely true. I never claimed to be anything but a Unix bigot. I'm a 'platform independent' bigot ;-) >Two rebuttals though: > > * if platform X supports running the compiler from a command line or > a script, surely that compiler can take some arguments... and surely > some of those arguments will deal with optimization/debugging, some > will deal with preprocessor settings, and some are just plain > miscellaneous Yes, but just having a bunch of flags doesn't tell me how to invoke the compiler. For example, Unix compilers use a -l flag, followed by the name of a library _without_ the first three letters (lib) and _without_ an extension, and _without_ a pathname. Windows compilers don't use this convention. You have to supply a full pathname, and some of them do not support any kind of library search. For example, some compilers will compile a .c file as c. And a .obj will get linked but not compiled. I'd need to know. Is there a space allowed after the -o option?? Do I put the name of the executable there, or the name _without_ .exe, which is added automatically? Now, you could add meta-flags to answer these questions. And it would become very complex to calculate how to invoke the compiler. > * splitting the flags up into all those categories can be useful, but > it could be that I'm confusing the Python build process with the > extension module build process. Perhaps the way to build an > extension should be carved in stone when Python itself is built, and > extension builders shouldn't be allowed to muck with it. If so, > then John is absolutely right about just calling a function to > invoke the compiler for such-and-such a situation. > >Gee, I guess that second one wasn't much of a rebuttal. So much for my >combative spirit. ;-) Do both! I don't see the 'function' as 'carved in stone', i see it as the default easy way to do it. It will _not_ always be enough! But it should always work for ISO compliant 'pure C', i.e. stuff that doesn't call non-standard libraries (other than Python of course :-) For example, it is enough for mxTools. Not enough for _tkinter. ----------------------- Here are a questions that sys.config could answer: 1) does the OS support symbolic or hard links? 2) what information is in fstat? 3) does the OS supply big/little endian Unicode files? 4) What operating system is it? [This is a hard question] 5) is os.system available? 6) What is the size of a Python 'integer' 'float'? 7) What is the eol character sequence? 8) is there a 'drive' prefix? 9) Can you delete an open file (Yes under Unix, No under NT) 10) Is there a stderr and/or stdlog file 11) Is there a console at all? 12) Is there a compiler available? 13) Is this J or C Python? It may be the proper place for the answer is not in sys.config, but in os. But the os module can compute the answers from sys.config. Hmm. Something I'm trying to say is that the point of the module has to be to provide a _standard_ interface to various features such as compiler invocation and installation. That is, a platform independent one (where possible of course). Note that as soon as you consider JPython, the system compiler is a java compiler, not a C compiler. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From Fred L. Drake, Jr." References: <1.5.4.32.19981216153622.00f7680c@triode.net.au> Message-ID: <13943.54939.159850.120664@weyr.cnri.reston.va.us> John Skaller writes: > 1) does the OS support symbolic or hard links? import os hasattr(os, "link") hasattr(os, "symlink") > 2) what information is in fstat? > 3) does the OS supply big/little endian Unicode files? > 4) What operating system is it? [This is a hard question] > 5) is os.system available? hasattr(os, "system") > 6) What is the size of a Python 'integer' 'float'? > 7) What is the eol character sequence? I thought this was in there, but can't recall where. > 8) is there a 'drive' prefix? > 9) Can you delete an open file (Yes under Unix, No under NT) > 10) Is there a stderr and/or stdlog file > 11) Is there a console at all? > 12) Is there a compiler available? > 13) Is this J or C Python? # true for JPython sys.platform[:4] == "java" > It may be the proper place for the answer is not in sys.config, > but in os. But the os module can compute the answers from > sys.config. For most of this, probably so. Recall also that Guido & others recommend "feature tests" to determine the availability of functionality rather than having a separate flag to indicate availability. There are many ways in which this is more reliable and maintainable, though what we're currently describing as sysconfig remains pertinant. I would expect sysconfig to be used mostly by things like distutils and not by typical programs. Hence, it would be good for os to not import it. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From gward@cnri.reston.va.us Wed Dec 16 16:11:50 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 16 Dec 1998 11:11:50 -0500 Subject: [Distutils] Access to Python config info In-Reply-To: <1.5.4.32.19981216153622.00f7680c@triode.net.au>; from John Skaller on Thu, Dec 17, 1998 at 01:36:22AM +1000 References: <1.5.4.32.19981216153622.00f7680c@triode.net.au> Message-ID: <19981216111150.C28854@cnri.reston.va.us> Quoth John Skaller, on 17 December 1998 [on whether the config module should be builtin or a separate .py file]: > I don't see any real choice: it has to be separate AND editable. > Because if I download a binary Windows version, the information will be wrong > for my system. The Unix version might be wrong too: perhaps I built > Python with gcc, but I want to build extensions with ecgs, or with edg. There are a number of reasons why you don't want to do this, and it all boils down to the fact that in the C world you cannot reliably link two object files if one was compiled with compiler X and the other with compiler Y. A concrete example that has burned me in the past: the 'assert' macro supplied with certain versions of GCC uses a function that is only available when you use GCC to drive the link, ie. when you the GCC runtime library is linked in. I have been bitten by this in two widely separated areas: distributing a C library that was used by ImageMagick and GIMP; people had problems if they compiled ImageMagick or GIMP with the system compiler and my library with gcc. Likewise, I have seen problems with a C library/Perl extension that I wrote; if someone manages to build the library and extension with gcc and link against a Perl built with cc, they stumble across this problem. (Less likely, because the SOP in Perl-world is to build extensions the same way Perl itself was built -- for this very reason, no doubt!) Similarly, different C++ compilers might use different name-mangling algorithms, again causing symbol mismatches. Anyways, these all constitute reasons for *strong recommendations* against fiddling with the compiler or flags. I'll concede that someone who *really* knows what they're doing should be allowed to do this (but they should also have to sign a disclaimer first ;-). And if they want to edit a machine-generated Python library to make this change permanent, well, again the responsibility rests on their shoulders. Extension builders should certainly have the option to override defaults from the config module, eg. ./setup.py build --cc=gcc --optimize=-g or ./setup.py install --prefix=/home/greg or something like that. (Fodder for another thread, perhaps.) However, I can definitely buy Andrew's and Robin's arguments in favour of [config-module-name].py rather than building it into Python. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From mal@lemburg.com Wed Dec 16 16:20:30 1998 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 16 Dec 1998 17:20:30 +0100 Subject: [Distutils] Access to Python config info References: <19981216090359.A28921@cnri.reston.va.us> Message-ID: <3677DDCE.6FA9DC89@lemburg.com> Greg Ward wrote: > > On the superficial front, Marc-Andre hasn't convinced me that sysconfig > is a better name than sys.config. (On the other hand, I never said > whether I thought it should be a sub-module of sys, or a [potentially > ambiguous] module under a package called 'sys'.) Gee, naming issues again... anything with a dot in it reminds me of packages and thus should act like one (submodules are really not a good idea: they contain code that is executed but not necessarily needed by the one who imports the main module). Yet, the sys module is written in C and all the distutils stuff should, IMHO, be done in Python. So the reasons I renamed it from sys.config to sysconfig were simply: · don't make it look like a package when it is not · don't have 'import sys' fill unneeded dictionaries · just naming it 'config' would probably cause name clashes Alternative name: 'pyconfig' > > Rather than discuss which things to include or not, just > > put all the information in there you can find by simply parsing > > Makefile. Note that the parsing mechanism should also allow > > for fairly decent $(VARIABLE) expansion. > > No! At least not in the long term: as Greg Stein pointed out, the > config module (whatever it gets called) should just be generated when > Python is built. Or were you talking about the near term, where we will > need a distutils release that works with Python 1.5.2 and is allowed all > manner of grotesque hackery to work around the lack of a config module, > such as digging up and parsing Python's config.h, Makefile, and Setup? I wouldn't consider this grotesque hackery. The idea I had was to have the sysconfig module initialize itself at import time (rather than install time). Since it is not often needed this should be a feasable approach. Most of all it works without any changes to the distribution as it is, except maybe... > > But the install dirs are changeable during configure... we might > > need some help from Guido here, e.g. the sys module could > > include the information: sys.configdir, sys.incldir and sys.libdir. > > Now you're cluttering 'sys' up with stuff in the yet-to-be-named config > module. > > These directories should just be accessible variables in > sys.config, eg. > > sys.config.install_bindir > sys.config.install_scriptdir > sys.config.install_libdir They are... once the module is initialized. But for it to find the necessary templates (Setup, config.h and Makefile) it needs some path information first. Looking at the current Makefile.pre.in though, it seems that this doesn't directly support non-standard install directories either. The parsed information could then be stored in a _sysconfig module which subsequent invocations then use which is close to what Greg proposed except that the first run is not necessarily done by the installation Makefile. > Greg Stein pointed out: > > Actually, at the discussion, we planned on compiling them right into the > > interpreter. That would prevent version skews between a compiled Python > > and the libraries. > > > > However, now that I'm thinking about it... screw that. If "import > > sysconfig" picks up the wrong version, then you have plenty of other > > issues to deal with. I'd much rather see a .py file that a bunch of > > stuff compiled into the interpreter. > > I'm torn on this one. Building into Python prevents version skew, > keeping it separate means it's easily readable without writing code. > Both are compelling arguments. Greg has given one voice in favour of a > separate .py file... anyone else? The current system has the same problems: Makefile.pre.in looks in the install dirs at boot time. If it finds wrong information then the resulting Makefile will use bogus data. Some people have reported effects of this: strange core dumps and unresolved symbols, even bus errors. Most of them had the install dirs mounted via NFS containing data for other machines than the one they built the extensions on. > Are there other sources of information we should be > worrying about? (Eg. does config.cache have anything valuable that > doesn't wind up in config.h or Makefile?) I think the three files pretty much cover all information that is available to Python at compile time (except maybe the os.environ at compile time). -- Marc-Andre Lemburg Y2000: 380 days left --------------------------------------------------------------------- : Python Pages >>> http://starship.skyport.net/~lemburg/ : --------------------------------------------------------- From da@skivs.ski.org Wed Dec 16 19:06:03 1998 From: da@skivs.ski.org (David Ascher) Date: Wed, 16 Dec 1998 11:06:03 -0800 (Pacific Standard Time) Subject: [Distutils] Re: Proposal: C/C++ compiler identification Message-ID: [sorry, I was disabled due to bounces, I'm just reading up now]. GvR: > For Windows, since the only supported compiler is VC++, it could be > hardcoded. However, there's a very serious problem on Windows with > Interscript's approach of invoking the compiler at run-time: most > users don't have this compiler. I realize that gcc is available for > free, but I think it isn't compatible with VC++. As far as I know, > VC++ is required if you want to use any of Mark Hammond's stuff (COM > and MFC). I don't know if Tcl/Tk can be used (without recompilation) > from a gcc-based app, and I don't know how easy it would be to > recompile. (Does gcc on Windows have support for Win32 APIs at all?) FYI: The status of gcc on Windows as far as I understand it: -- it's called the Cygwin project -- it's supported by Cygnus (see www.cygnus.com) -- it uses egcs, not gcc -- current build is b20 (note: doesn't comply with version number scheme discussed earlier =) -- it provides access to the win32 api if you want -- it needs access to cygwin.dll on the machine (not a standard MS library =) -- in the past, it was very hard or impossible to have gcc-compiled and VC-compiled objects linked together, even statically. -- I believe things are better now, but far from trivial (I'm still at b19 because of final comment below) -- I could imagine doing things like compiling Python + NumPy together, but I'd never try Python+PIL+NumPy+Tk+readline+threads+... Final comment on Cygwin: While Cygwin is free, it is *hard* to install and setup in a proper way. My advice to all Windows users who want to compile things is to buy VC -- the Cygwin stuff is just not there yet when it comes to novices or folks who are short on time. Final comment on GvR's point above: I don't think VC is needed to use MHammond's COM stuff. Just the DLLs, which are installed on reasonable machines. --david "my Mom still can't forgive me for using Windows" ascher From da@skivs.ski.org Wed Dec 16 19:16:41 1998 From: da@skivs.ski.org (David Ascher) Date: Wed, 16 Dec 1998 11:16:41 -0800 (Pacific Standard Time) Subject: [Distutils] Re: Access to Python config info Message-ID: Robin: > Another vote for impeac^H^H^H^H^H^Ha separate .py file to store this. > Mostly it's better since it allows tweaks to be made easily, (say the > installpath changes, or a new compiler), a hundred different little > things can happen at a local site. It'll also help during the > development phases of this tool to have visibility into what's actually > happening. Same here. Furthermore, one can imagine writing a program which generates sysconfig.py for distributed heteregeneous systems, or even writing a sysconfig.py module which *does stuff* at import time. Why limit ourselves... --david From gstein@lyra.org Wed Dec 16 19:40:01 1998 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Dec 1998 11:40:01 -0800 Subject: [Distutils] Access to Python config info References: <19981216090359.A28921@cnri.reston.va.us> Message-ID: <36780C91.59EF05A4@lyra.org> Greg Ward wrote: > ... > Marc-Andre also said: > > Rather than discuss which things to include or not, just > > put all the information in there you can find by simply parsing > > Makefile. Note that the parsing mechanism should also allow > > for fairly decent $(VARIABLE) expansion. > > No! At least not in the long term: as Greg Stein pointed out, the > config module (whatever it gets called) should just be generated when > Python is built. Or were you talking about the near term, where we will > need a distutils release that works with Python 1.5.2 and is allowed all > manner of grotesque hackery to work around the lack of a config module, > such as digging up and parsing Python's config.h, Makefile, and Setup? I believe the public interface will/should always be a set of names and values. Even if we do a hack in lieu of native Python support, we should just set globals for use by clients. There may be functions in there, too, for compiler invocation and stuff (as John recommends). I'm not clear on whether that needs to be in sysconfig.py or part of the distutils package (forgot the pkg name). > ... > [Greg S. again] > > I'd also recommend ignoring all the dictionary vs instance stuff. Just > > have a bunch of assignments dropped into a sysconfig.py module. IMO, it > > would look a lot like FCNTL.py (h2py output files). No need to get fancy > > with defaults and stuff. Just write the dumb file out. > > I (almost) completely agree. If we put *everything* from config.h into > the config module, it might be nice to store it separately (eg. in a > hash). But the stuff from Makefile and Setup should just be dumped in > flat as variables. Are there other sources of information we should be > worrying about? (Eg. does config.cache have anything valuable that > doesn't wind up in config.h or Makefile?) All right, you Perl hacker. We'll have none of this here! Python doesn't have "hashes" ... it has dictionaries! Also, it is always possible to view module globals as a dictionary (with vars(sysconfig)), so there isn't a need for another level containing a dictionary. Go home Perl hacker! :-) > ... Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Wed Dec 16 21:09:01 1998 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Dec 1998 13:09:01 -0800 Subject: [Distutils] Re: Proposal: C/C++ compiler identification References: Message-ID: <3678216D.7021C225@lyra.org> David Ascher wrote: > ... > -- it's called the Cygwin project > -- it's supported by Cygnus (see www.cygnus.com) > -- it uses egcs, not gcc > -- current build is b20 (note: doesn't comply with version number scheme > discussed earlier =) Works for the code that I posted :-) >... > Final comment on GvR's point above: I don't think VC is needed to use > MHammond's COM stuff. Just the DLLs, which are installed on reasonable > machines. VC is needed to compile our stuff. It certainly isn't needed to run it. -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Wed Dec 16 21:37:33 1998 From: gstein@lyra.org (Greg Stein) Date: Wed, 16 Dec 1998 13:37:33 -0800 Subject: [Distutils] Access to Python config info References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> Message-ID: <3678281D.4FC0C4F6@lyra.org> M.-A. Lemburg wrote: > ... > Greg Ward wrote: > > No! At least not in the long term: as Greg Stein pointed out, the > > config module (whatever it gets called) should just be generated when > > Python is built. Or were you talking about the near term, where we will > > need a distutils release that works with Python 1.5.2 and is allowed all > > manner of grotesque hackery to work around the lack of a config module, > > such as digging up and parsing Python's config.h, Makefile, and Setup? > > I wouldn't consider this grotesque hackery. The idea I had > was to have the sysconfig module initialize itself at import > time (rather than install time). Since it is not often needed > this should be a feasable approach. Most of all it works > without any changes to the distribution as it is, except maybe... I don't see much of a rationale for *not* doing this at install time. The file is a bunch of constants that are easily determined when you build/install Python. There is a caveat for trying to get it to function with 1.5.2, but I don't believe that changes the long-term interface, which I maintain are simple names/values. > Greg Ward wrote: > > These directories should just be accessible variables in > > sys.config, eg. > > > > sys.config.install_bindir > > sys.config.install_scriptdir > > sys.config.install_libdir > > They are... once the module is initialized. But for it to find the > necessary templates (Setup, config.h and Makefile) it needs some > path information first. Looking at the current Makefile.pre.in > though, it seems that this doesn't directly support non-standard install > directories either. > > The parsed information could then be stored in a _sysconfig module > which subsequent invocations then use which is close to what Greg > proposed except that the first run is not necessarily done by the > installation Makefile. It *should* be done by the installation Makefile. I would hope that we obsolete the third-party use of Setup/config.h/Makefile.pre.in. Those are even close to the convenience that we can establish. So, I'd maintain that we don't need to find those three files, so there isn't a bootstrap issue here. Note that these files are always installed at $(exec_prefix)/lib/python$(version)/config/ (as far as I can tell, that's fixed). Cheers, -g -- Greg Stein, http://www.lyra.org/ From Fred L. Drake, Jr." References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> Message-ID: <13945.16759.231398.717797@weyr.cnri.reston.va.us> --udwOQwHuZC Content-Type: text/plain; charset=us-ascii Content-Description: message body text Content-Transfer-Encoding: 7bit Greg Stein writes: > I don't see much of a rationale for *not* doing this at install time. > The file is a bunch of constants that are easily determined when you > build/install Python. Yes. It can include code to regenerate the "right stuff" when run as a script as well. Running it as a script can be done at the end of installation. > There is a caveat for trying to get it to function with 1.5.2, but I > don't believe that changes the long-term interface, which I maintain are > simple names/values. What's the caveat? > Note that these files are always installed at > $(exec_prefix)/lib/python$(version)/config/ (as far as I can tell, > that's fixed). I think you're right. I've attached a sysconfig.py that works on Unix. Someone familiar with other platforms will need to make it work there as well by adding additional _init_() functions; should be os.name. When run as a script, it rewrites itself with the discovered information. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 --udwOQwHuZC Content-Type: text/plain Content-Description: preliminary sysconfig.py Content-Disposition: inline; filename="sysconfig.py" Content-Transfer-Encoding: 7bit """Prototype sysconfig module that loads information when run as a script, but only defines constants when imported. This should be run as a script as one of the last steps of the Python installation process. """ # START DATA # END DATA def _init_posix(): import os import re import sys g = globals() version = sys.version[:3] config_dir = os.path.join( sys.exec_prefix, "lib", "python" + version, "config") # load the installed config.h: define_rx = re.compile("#define ([A-Z][A-Z0-9_]+) (.*)\n") undef_rx = re.compile("/[*] #undef ([A-Z][A-Z0-9_]+) [*]/\n") fp = open(os.path.join(config_dir, "config.h")) while 1: line = fp.readline() if not line: break m = define_rx.match(line) if m: n, v = m.group(1, 2) if v == "1": g[n] = 1 else: g[n] = v else: m = undef_rx.match(line) if m: g[m.group(1)] = 0 # load the installed Makefile.pre.in: variable_rx = re.compile("([a-zA-Z][a-zA-Z0-9_]+)\s*=\s*(.*)\n") done = {} notdone = {} fp = open(os.path.join(config_dir, "Makefile")) while 1: line = fp.readline() if not line: break m = variable_rx.match(line) if m: n, v = m.group(1, 2) if "$" in v: notdone[n] = v else: done[n] = v # do variable interpolation here findvar1_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*).") findvar2_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*)}") while notdone: for name in notdone.keys(): value = notdone[name] m = findvar1_rx.search(value) if not m: m = findvar2_rx.search(value) if m: n = m.group(1) if done.has_key(n): after = value[m.end():] value = value[:m.start()] + done[n] + after if "$" in after: notdone[name] = value else: done[name] = value del notdone[name] elif notdone.has_key(n): # get it on a subsequent round pass else: done[n] = "" after = value[m.end():] value = value[:m.start()] + after if "$" in after: notdone[name] = value else: done[name] = value del notdone[name] else: del notdone[name] # save the results in the global dictionary g.update(done) # save the results in the source file filename = os.path.splitext(sys.argv[0])[0] + ".py" lines = open(filename).readlines() start = lines.index("# START DATA\n") end = lines.index("# END DATA\n") newdata = [] items = g.items() items.sort() for name, value in items: if name[0] != "_": newdata.append("%s = %s\n" % (name, `value`)) lines[start+1:end] = newdata open(filename, "w").writelines(lines) if __name__ == "__main__": import os exec "_init_%s()" % os.name del _init_posix --udwOQwHuZC-- From gward@cnri.reston.va.us Thu Dec 17 23:36:16 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 17 Dec 1998 18:36:16 -0500 Subject: [Distutils] Access to Python config info In-Reply-To: <13945.16759.231398.717797@weyr.cnri.reston.va.us>; from Fred L. Drake on Thu, Dec 17, 1998 at 12:37:59PM -0500 References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> Message-ID: <19981217183616.A29344@cnri.reston.va.us> Quoth Fred L. Drake, on 17 December 1998: > I've attached a sysconfig.py that works on Unix. Someone familiar > with other platforms will need to make it work there as well by adding > additional _init_() functions; should be os.name. When > run as a script, it rewrites itself with the discovered information. Cool! I have two itty-bitty teeny-tiny nits to pick: > """Prototype sysconfig module that loads information when run as a script, > but only defines constants when imported. > > This should be run as a script as one of the last steps of the Python > installation process. > """ *This* version should run as one of the last steps of the distutils installation process, ie. this is the hack that'll work with Python 1.5.2. The nice version that will be bundled with Python 1.6 (I'm hoping!) might just be this again, or it might have more direct knowledge of Python's configure-time information. But it should run as part of Python's build process, so it won't go looking in sys.exec_prefix + lib + python + ... -- it'll just find stuff in the current (Python build) directory. But for the first version of distutils, I think this is what we need. > # do variable interpolation here > findvar1_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*).") > findvar2_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*)}") Ummm... is this right? Shouldn't the first regex be for $(...) and the second for ${...}? If that's what you were thinking, this'll still work (because of the .'s where there should be \( and \)), but it's a bit inexact. (Feel free to slap me with a wet noodle if I'm missing something obvious here.) Well, Fred, you've set the ball rolling... I guess coding and design will, as usual, proceed hand-in-hand... ;-) Greg P.S. I would like to suggest one coding standard for the project: every source file should have the name and email address of the person who wrote it, the date it was started, and and RCS/CVS identifier of some kind (I'm partial to $Id$ myself, but $Revision$ is also fine). Any objections? P.P.S. I'll see about getting an anonymous CVS archive set up tonight or tomorrow. -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein@lyra.org Thu Dec 17 23:46:08 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 17 Dec 1998 15:46:08 -0800 Subject: [Distutils] Access to Python config info References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> Message-ID: <367997C0.68C6903@lyra.org> Greg Ward wrote: > ... > *This* version should run as one of the last steps of the distutils > installation process, ie. this is the hack that'll work with Python > 1.5.2. The nice version that will be bundled with Python 1.6 (I'm > hoping!) might just be this again, or it might have more direct > knowledge of Python's configure-time information. But it should run as > part of Python's build process, so it won't go looking in > sys.exec_prefix + lib + python + ... -- it'll just find stuff in the > current (Python build) directory. > > But for the first version of distutils, I think this is what we need. Re: Fred Drake asking about the caveat I mentioned. This is it: in the future, the build process just spits out a bunch of assignments. For 1.5.2, we don't have that luxury, so it gets a bit more complicated. Cheers, -g > ... > P.S. I would like to suggest one coding standard for the project: every > source file should have the name and email address of the person who > wrote it, the date it was started, and and RCS/CVS identifier of some > kind (I'm partial to $Id$ myself, but $Revision$ is also fine). Any > objections? hehe... boy, you do like things in a particular way, don't you? Now I know where the version number stuff came from :-) For better or worse, I don't think you're going to get the level of compliance that you want here. You'll see :-) hehe.... -- Greg Stein, http://www.lyra.org/ From MHammond@skippinet.com.au Fri Dec 18 00:45:53 1998 From: MHammond@skippinet.com.au (Mark Hammond) Date: Fri, 18 Dec 1998 11:45:53 +1100 Subject: [Distutils] Freeze and new import architecture Message-ID: <001c01be2a1f$c4f0f0f0$0801a8c0@bobcat> [Apols in advance for the length of this - but the background is probably necessary] I will make no attempt to outline what the freeze tool should ultimately be! Im not even going to outline what freeze is now! Im just going to focus on one single requirement. We want freeze to be capable of working with any number of "code containers" - ie, to be able to locate code in "frozen C modules" (as it works now), or potentially in a .zip file etc. The justification is fairly easy to explain: Consider that you may want to use "freeze" to distribute an application that consists of a number of Python programs. Using freeze as it stands now, the entire Python library is frozen into each application. Thus the result is each program is frozen to an executable that may be megabytes in size, even though the bulk is likely to be the identical, standard Python library. Attempting to cut a long story short, Guido, Jack Jansen, myself and Just came up with another idea for an extensible import mechanism: sys.path not be restricted to path names. sys.path has "strings", and an associated map of "module finders". Thus, a sys.path entry could have a directory name (like now) or .zip file, URL, etc. I have included below a mail from Jack Jansen on this topic. To paraphrase Guido's response, it was: "looks good - but lets define a Python interface so it may work with JPython" Accordingly, I have mocked up a few .py files which start to implement the ideas in Jack's post. However, Im now floundering a little as to the best way to take this any further. Im really looking for further support or critisism of the idea, and some interest from people in helping take this further... If there is general interest, I will post my mock-up, which allows directories and URLs to exist on sys.path... Thanks, Mark. -----Original Message----- From: Jack Jansen [mailto:Jack.Jansen@cwi.nl] Sent: Wednesday, 12 August 1998 1:11 AM To: guido@python.org; just@letterror.com; mhammond@skippinet.com.au Subject: An import architecture I thought I'd just mail you the results of a discussion Mark and myself had about the import architecture. I'm just sending it out to us four, but I guess we should get more people into the discussion if Guido thinks that this is something worth "solving". The problem we're looking at is that import.c and importdl.c have lots of special case code, and everytime someone comes up with a new neat way to import modules more special case code has to be added. This happened with Mac PYC resource modules and PYD modules, and there's also importdl.c which nows about DLL modules for umpteen different systems. It would be nice if there was an architecture in import.c that would allow the machine specific code to hook into (and, ultimately, Python code as well). WHAT WE HAVE ============ We noticed that there are really two issues involved in importing modules: 1. Finding the module in a specific namespace. 2. Importing a module of a specific type, once it has been found. For 1. we currently have 5 namespaces: builtin modules, frozen modules, the filesystem namespace, the PYC resource namespace in a file (mac-only) and the PYD resource namespace in a file (again mac-only). Other namespaces can be envisioned, for instance the namespace inside a squeeze .pyz archive, a web-based namespace, whatever. The builtin and frozen namespace are currently special, in that they don't occur in sys.path and are always virtually at the very front of sys.path. On the mac sys.path entries can be either filenames (in which case the latter two modulefinders are invoked) or directories (in which case the filesystem finder is invoked), on other platforms there are only directories in sys.path right now. Regarding 2: the finder currently returns a structure that enables the correct importer to be called later on. Importers that we have are for builtin, frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind the same interface, PYC-resource importers (mac-only) and PYD-resource importers (mac-only). WHAT WE WANT ============ What we'd like I'll try to describe top-down (hopefully better to understand than bottom-up). importing a module becomes something like for pathentry in sys.path: finder = getfinder(pathentry) loader = finder.find(module) module = loader() getfinder() is something like if not path_to_finder.has_key(pathentry): for f in all_finder_types: finder = f.create_finder(pathentry) if finder: path_to_finder[pathentry] = finder break return path_to_finder[pathentry] And there would be a call whereby a finder type registers itself (adds itself to all_finder_types). finder.create_finder() would examine the current path component, see if it could handle it, and, if so, return a finder object that will search this path component everytime it is invoked. The usual case is that getfinder doesn't do much: only the first time you see a new entry in sys.path you'll have to ask all the finders in turn whether they support it, ad you just remember this. The loaders register themselves with the finders, passing finder-specific arguments. For instance, the .py loader registers itself with the filesystem finder, telling it that the ".py" extension should result in a .py loader being created. The unix-specific dll-loader does the same for .so extensions, the windows-dll-loader for .dll, the mac-dll-loader for .slb etc. The PYC-resource loader tells the mac-resource-finder to look for 'PYC ' resources, the PYD-resource loader tells it to look for 'PYD ' resources, etc. The information that is passed from the finder to loader when it has found a module is again finder-specific: the filesystem loader will probably pass an open file and a filename, etc. A loader can register itself with multiple finders, assuming their interfaces are similar. So, the .py loader could register itself not only with the filesystem finder but also with a url-based finder or something, as long as that url-based finder uses the same calling convention for creating the loader. WHAT DOES IT BUY US =================== A greatly simplified import.c, importdl.c split out over the various platforms (and with the possibility to pass machine-specific info from the find phase to the load phase, something that cant be done now and leads to double work on various platforms) and easy extensibility. There is the issue of performance. The description above is all Pythonish, but going through the Python calling sequence for all these things is probably not a good idea from a performance standpoint. This is however fixable. The objects involved are - The finder "class", the thing you use to check whether a certain sys.path component can be handled by this code - The finder instance returned by this class - The importer class (called by the finder objects) - The importer instance (returned by the finder instance through invoking the importer class). These 4 things could well be 4 specific PyObject types, with the needed C-routines in the object struct. Calling these for the normal case (i.e. when they're implemented in C) would be at most a single (C-) indirection more expensive than the current scheme. Moreover, it would be easy to create a module that would implement these 4 types as wrappers around Python code. The C-routines would then do the usual PyEval_CallObject stuff to call the Python implementation. So, you'd have the generality but you'd only pay for it when you put Python-handled entries in sys.path and actually hit those entries. ODDS AND ENDS ============= A side issue: this stuff would also allow us to put the builtin and frozen namespace into sys.path explicitly, for instance as "__builtins__" and "__frozen__", something I would like. The disadvantage would be that you can't be sure that everything on sys.path is a pathname, but the advantage would be that you could, for instance create a frozen program that you could patch: set sys.path to ["/usr/local/FooPatches", "__frozen__", "__builtins__"], and whenever you have a patch to a single module in a frozen executable you just send your clients the single .pyc file and tell them to put it in the FooPatches directory. There'd probably have to be a bit of code that explicitly prepends "__frozen__" and "__builtins__" to sys.path if they aren't there already or something. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@cwi.nl | ++++ if you agree copy these lines to your sig ++++ http://www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From gstein@lyra.org Fri Dec 18 02:15:37 1998 From: gstein@lyra.org (Greg Stein) Date: Thu, 17 Dec 1998 18:15:37 -0800 Subject: [Distutils] Freeze and new import architecture References: <001c01be2a1f$c4f0f0f0$0801a8c0@bobcat> Message-ID: <3679BAC9.3D252028@lyra.org> I've messed around quite a bit with funky import mechanisms also and have found the current setup a bit tough to work within. I'll insert some comments below, but never fear... I've got more, too :-) Mark Hammond wrote: > ... > sys.path not be restricted to path names. sys.path has "strings", and > an associated map of "module finders". Thus, a sys.path entry could > have a directory name (like now) or .zip file, URL, etc. I would much prefer to see the module finder instances in the sys.path. Sometimes, it is *very* difficult to map strings to module finders. For example, if you have a .dll with frozen code in it, and the code has been frozen in one of N formats, then how can you determine which module finder to use for the format? IMO, it is better to insert the finder itself: sys.path.append(GZippedDLLResource("modulename", "mycode.dll")) sys.path.append(DLLResourceGroup("mymodules.dll")) > ... Jack Jansen wrote: > ... > WHAT WE HAVE > ============ > > We noticed that there are really two issues involved in importing > modules: > 1. Finding the module in a specific namespace. > 2. Importing a module of a specific type, once it has been found. I think the separation is bogus. Trying to fit into the Finder/Loader paradigm of the ihooks has always been a total, non-intuitive pain-in-the-ass for me (to be blunt :-). Instead, I just go straight for the import hook and ignore the whole ihooks thing. I would put forward that we ignore the find/load paradigm and simply go to: 1. Import the given module if you can If an element on sys.path can't do it (returning None), then you move to the next one. > ... > The builtin and frozen namespace are currently special, in that they > don't > occur in sys.path and are always virtually at the very front of > sys.path. On > the mac sys.path entries can be either filenames (in which case the > latter two > modulefinders are invoked) or directories (in which case the > filesystem finder > is invoked), on other platforms there are only directories in sys.path > right > now. Simple to do: sys.path.insert(0, BuiltinImporter()) The BuiltinImporter handles compiled-in and frozen modules. > Regarding 2: the finder currently returns a structure that enables the > correct > importer to be called later on. Importers that we have are for > builtin, > frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind > the > same interface, PYC-resource importers (mac-only) and PYD-resource > importers > (mac-only). Punt this. Just import the dumb thing in one shot. Take the example of an HTTP-based import. Separating that into *two* transactions would be painful. It should be imported in one fell swoop. And no, you can't just keep the socket open and pass that to the loader -- that implies that you can defer the passing for a while, but the web server will time out your connection and close it. Conversely, if the intent is *not* to hold the "structure" for a while, then why the heck have two pieces? > > WHAT WE WANT > ============ > > What we'd like I'll try to describe top-down (hopefully better to > understand > than bottom-up). > > importing a module becomes something like > > for pathentry in sys.path: > finder = getfinder(pathentry) > loader = finder.find(module) > module = loader() I'll amend this to: for pathentry in sys.path: if type(pathentry) == StringType: module = old_import(pathentry, modname) else: module = pathentry.do_import(modname) if module: return module else: raise ImportError, modname + " not found." > getfinder() is something like > if not path_to_finder.has_key(pathentry): > for f in all_finder_types: > finder = f.create_finder(pathentry) > if finder: > path_to_finder[pathentry] = finder > break > return path_to_finder[pathentry] The above code is basically keeping a mirror of the sys.path list, but with importer instances in it. Just put those into sys.path itself. > And there would be a call whereby a finder type registers itself (adds > itself > to all_finder_types). In my proposal, this wouldn't be necessary. You insert finders right into sys.path. > ... > A loader can register itself with multiple finders, assuming their > interfaces > are similar. So, the .py loader could register itself not only with > the > filesystem finder but also with a url-based finder or something, as > long as > that url-based finder uses the same calling convention for creating > the loader. I'd rephrase this as you have multiple importer instances, each configured for a different "path" to its module namespace. > WHAT DOES IT BUY US > =================== > > A greatly simplified import.c, importdl.c split out over the various > platforms > (and with the possibility to pass machine-specific info from the find > phase to > the load phase, something that cant be done now and leads to double > work on > various platforms) and easy extensibility. Agreed. Quite necessary. I would think that we could have different little code bits for each platform, much like we have thread_*.h in the Python/ subdirectory. > There is the issue of performance. The description above is all > Pythonish, but > going through the Python calling sequence for all these things is > probably not > a good idea from a performance standpoint. This is however fixable. I disagree. In almost all cases, you are talking about bringing a module into the interpreter. Your performance is going to be characterized by I/O, memory allocations for all the structures that get built when the module executes, and the actual execution time of that module. The time spent using Python to perform the import sequence is a non-issue. > ... > > ODDS AND ENDS > ============= > > A side issue: this stuff would also allow us to put the builtin and > frozen > namespace into sys.path explicitly, for instance as "__builtins__" and > "__frozen__", something I would like. The disadvantage would be that > you can't > be sure that everything on sys.path is a pathname, but the advantage > would be > that you could, for instance create a frozen program that you could > patch: Both of our proposals guarantee that stuff in sys.path are not pathnames. If I insert a "foo.zip" or a "http://host.domain.name/pymodules/", then you certainly dont have pathnames. I believe the biggest issue with my proposal is the fact that the values are no longer strings. However, the out-of-the-box version of Python can easily contain *just* strings. If people bring in custom importers into their application, AND they have code that depends on the "string-ness" of sys.path, then it is their problem. By definition, they've altered the behavior of their app and they need to compensate; my proposal is backwards compatible for existing apps. [ the tweak would be to avoid inserting BuiltinImporter() -- the instance/type could still exist, but merely be *implied* rather than explicitly within sys.path ] > set sys.path to ["/usr/local/FooPatches", "__frozen__", > "__builtins__"], and > whenever you have a patch to a single module in a frozen executable > you just > send your clients the single .pyc file and tell them to put it in the > FooPatches directory. There'd probably have to be a bit of code that > explicitly prepends "__frozen__" and "__builtins__" to sys.path if > they aren't > there already or something. This is quite humorous... Small world: I've used this approach before. In the Microsoft Merchant Server 1.0, we had a bunch of frozen Python code. However, we also looked in a specific directly for patches. We never patched it :-), but it was possible. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Fri Dec 18 14:52:32 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 18 Dec 1998 09:52:32 -0500 Subject: [Distutils] Access to Python config info In-Reply-To: <367997C0.68C6903@lyra.org>; from Greg Stein on Thu, Dec 17, 1998 at 03:46:08PM -0800 References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> <367997C0.68C6903@lyra.org> Message-ID: <19981218095232.A1992@cnri.reston.va.us> Quoth Greg Stein, on 17 December 1998: > > P.S. I would like to suggest one coding standard for the project: every > > source file should have the name and email address of the person who > > wrote it, the date it was started, and and RCS/CVS identifier of some > > kind (I'm partial to $Id$ myself, but $Revision$ is also fine). Any > > objections? > > hehe... boy, you do like things in a particular way, don't you? Now I > know where the version number stuff came from :-) I am anal boy; hear me squeak! Actually, I'm doing all this to convince the Python world that not *all* people named Greg are left-coast long-haired anarchists. ;-)^2 But seriously: is it *really* such a burden to put something like this: # # distutils/version.py # # Implements multiple version numbering conventions for the # Python Module Distribution Utilities. # # written by Greg Ward , 1998/12/17 # # $Id$ # in the top of a source file? I don't think so, and it's especially trivial compared to the time spent to write documentation and a test suite. (Which I'm also a fan of. But I figured insisting on them would just get me laughed off the sig, so I kept the request [name, date, RCS id] pretty small. I honestly didn't think even *you* would howl at that... silly me!) Which reminds me, by writing sysconfig.py Fred has shamed me into writing version.py. Should be available in that yet-to-be-created anonymous CVS archive Real Soon Now. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gstein@lyra.org Fri Dec 18 15:05:23 1998 From: gstein@lyra.org (Greg Stein) Date: Fri, 18 Dec 1998 07:05:23 -0800 Subject: [Distutils] Access to Python config info References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> <367997C0.68C6903@lyra.org> <19981218095232.A1992@cnri.reston.va.us> Message-ID: <367A6F33.7CA8070E@lyra.org> Greg Ward wrote: > ... > Actually, I'm doing all this to convince the Python world that not *all* > people named Greg are left-coast long-haired anarchists. ;-)^2 Feh. > But seriously: is it *really* such a burden to put something like this: > > # > # distutils/version.py > # > # Implements multiple version numbering conventions for the > # Python Module Distribution Utilities. > # > # written by Greg Ward , 1998/12/17 > # > # $Id$ > # > > in the top of a source file? I don't think so, and it's especially > trivial compared to the time spent to write documentation and a test > suite. (Which I'm also a fan of. But I figured insisting on them would > just get me laughed off the sig, so I kept the request [name, date, RCS > id] pretty small. I honestly didn't think even *you* would howl at > that... silly me!) I'm not howling, and I never said it was a burden (so there! :-). In fact, I don't really have much of a problem with it. If I *remembered* to do it, then I'll add them for any code I drop off. However, I'm a realist... I'm laying odds that you won't get everybody to add those headers. Call me a cynic, but you're working against the habits of a bunch of programmers. There is very little on the planet that has the stubborness of a programmer :-) > Which reminds me, by writing sysconfig.py Fred has shamed me into > writing version.py. Should be available in that yet-to-be-created > anonymous CVS archive Real Soon Now. Hey, I gave you some code for versions. It worked great for me :-) -g -- Greg Stein, http://www.lyra.org/ From hinsen@cnrs-orleans.fr Fri Dec 18 15:28:21 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 18 Dec 1998 16:28:21 +0100 Subject: [Distutils] Configuration etc. Message-ID: <199812181528.QAA17720@dirac.cnrs-orleans.fr> I don't claim that I read everything that all of you posted on the topic of retrieving and storing the configuration information from the Python build process - I also have to work in between ;-) But it seems to me that this discussion misses the point, or at least deals with a minor implementation detail. The build phase of the installation process (needed only for source installation of extension modules, of course) should use a much higher-level interface to the DistUtil core than just the information from the Makefile. As a package provider, I want to write something like from DistUtils import compileExtensionModule compileExtensionModule("foo", ["foomodule.c", "bar.c"]) Everything lower-level is an implementation detail and not worth 10 KB of discussion ;-) A DistUtils implementation for Python 1.5.x on Unix will have to parse the configuration file or Makefile from the Python build, using some module much like the one posted by Fred. For other platforms, this kind of information may not be available, or not necessary, or useless. For later Python versions, everything might be different. So I see no reason at all to store such specific data in a documented standard library module, whatever it is called. It may even be better not to store it at all and parse the original input every time the information is needed. And now for something completely different: implementation strategy. At some point we will have to move from discussing to programming, after all. It seems to me that the whole DistUtils project divides nicely into two almost independent subunits: - building binaries from source code - installing Python modules and binary extension modules So maybe we should create two "task forces" that work on these subtopics, of course communicating as necessary. That said, I'll disappear for a two-week Christmas vacation, hoping that the discussion during these two weeks won't fill up all my disk space... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Fred L. Drake, Jr." References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> <367997C0.68C6903@lyra.org> Message-ID: <13946.29916.320317.654162@weyr.cnri.reston.va.us> Greg Stein writes: > Re: Fred Drake asking about the caveat I mentioned. > > This is it: in the future, the build process just spits out a bunch of > assignments. For 1.5.2, we don't have that luxury, so it gets a bit more Greg, 1.5.2 isn't out yet! -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <199812181528.QAA17720@dirac.cnrs-orleans.fr> Message-ID: <13946.33189.821489.368289@weyr.cnri.reston.va.us> Konrad Hinsen writes: > The build phase of the installation process (needed only for source > installation of extension modules, of course) should use a much > higher-level interface to the DistUtil core than just the information I agree; I think the expectation is a slightly higher level interface than that: you provide something declarative, telling the distutils package what your inputs are and what sort of module you want to build. And it does the rest. > Everything lower-level is an implementation detail and not worth > 10 KB of discussion ;-) Probably not, if someone gets to go off and do the whole thing. I think we're trying to share the load, though. And we just like email. ;-) > I see no reason at all to store such specific data in a documented > standard library module, whatever it is called. It may even be better > not to store it at all and parse the original input every time the > information is needed. I'm fine with this. It can just be distutils.sysconfig, have a comment saying it's just an implementation detail, and ... > after all. It seems to me that the whole DistUtils project divides > nicely into two almost independent subunits: > > - building binaries from source code > > - installing Python modules and binary extension modules Good breakdown. Each of these needs to consider at least three platforms for the initial release: Unix, Windows, and Mac. > That said, I'll disappear for a two-week Christmas vacation, hoping > that the discussion during these two weeks won't fill up all my disk Hah! Never leave your email unattended.... ;-) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> Message-ID: <13946.35058.910774.305483@weyr.cnri.reston.va.us> Greg Ward writes: > *This* version should run as one of the last steps of the distutils > installation process, ie. this is the hack that'll work with Python If we can agree that it includes sufficient information, and we can get the other platforms supported, why can't we ask Guido to include it in 1.5.2? That would also allow other projects that have differing views to use the same fundamental information. That way running it is only needed as part of the Python build/install (should be install, esp. for people using things like RPM, where the locations may change after the build). > 1.5.2. The nice version that will be bundled with Python 1.6 (I'm > hoping!) might just be this again, or it might have more direct > knowledge of Python's configure-time information. But it should run as > part of Python's build process, so it won't go looking in > sys.exec_prefix + lib + python + ... -- it'll just find stuff in the > current (Python build) directory. Stuffing in the build directory isn't useful. Now that you have me thinking about this, it perhaps *is* useful to at least build the paths at import time. The Python distribution is supposed to be relocatable *after installation* for Unix (at least). The interpreter already does a search for the libraries, and sets sys.prefix and sys.exec_prefix accordingly (I think those get set properly... yes, they are). So: the right way to do this is to set most of the paths at import time. This should not be hard. > > # do variable interpolation here > > findvar1_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*).") > > findvar2_rx = re.compile(r"\$.([A-Za-z][A-Za-z0-9_]*)}") > > Ummm... is this right? Shouldn't the first regex be for $(...) and the > second for ${...}? If that's what you were thinking, this'll still work Ouch! Yes. That was a change I forgot I made while debugging something that was entirely a different problem with the code (I was using match() instead of search(), probably because it's usually what I need). > P.S. I would like to suggest one coding standard for the project: every > source file should have the name and email address of the person who > wrote it, the date it was started, and and RCS/CVS identifier of some Cool. I typically use __version__ = "$Revision$" for storing revision control information. This corresponds to a recommendation from long ago; I don't recall the source, but it was probably c.l.py. $Id$ could be added in a comment if desired, but I don't know how useful it is since I don't use it. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From da@skivs.ski.org Fri Dec 18 17:29:19 1998 From: da@skivs.ski.org (David Ascher) Date: Fri, 18 Dec 1998 09:29:19 -0800 (Pacific Standard Time) Subject: [Distutils] Configuration etc. In-Reply-To: <199812181528.QAA17720@dirac.cnrs-orleans.fr> Message-ID: On Fri, 18 Dec 1998, Konrad Hinsen wrote: > It seems to me that the whole DistUtils project divides > nicely into two almost independent subunits: > > - building binaries from source code > > - installing Python modules and binary extension modules > I would like to add another, somewhat win/mac-centric twist: - making installers of Python & add-ons. More specifically: i'm working on a program which is going to use Python, Tk, PIL, OpenGL, NumPy, Pmw, and some custom code. I'd like to be able to make a single installer which installed all of those parts, not necessarily frozen. For all of the extensions above, I think that I can generate .WSE files (the input to the WISE installer maker, free for Python extensions) which do the job, starting with Python's, and assuming I or someone else can figure out how to do the Tk bit. I'm sure this kind of thing would be useful in the mac world as well, where even fewer users have compilers. Wouldn't this sort of thing be useful in Unix as well, in some distribution models? I think so, judging from the efforts of the Digital Creations folks and their binary porting kit for Zope (from http://www.zope.org/Download/Contrib). Note that this is a different kind of distribution -- it's a distribution of applications for users, not of extensions for Python programmers. --david From hinsen@cnrs-orleans.fr Fri Dec 18 17:57:16 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 18 Dec 1998 18:57:16 +0100 Subject: [Distutils] Configuration etc. In-Reply-To: (message from David Ascher on Fri, 18 Dec 1998 09:29:19 -0800 (Pacific Standard Time)) References: Message-ID: <199812181757.SAA14516@dirac.cnrs-orleans.fr> > I would like to add another, somewhat win/mac-centric twist: > > - making installers of Python & add-ons. > > More specifically: i'm working on a program which is going to use Python, > Tk, PIL, OpenGL, NumPy, Pmw, and some custom code. I'd like to be able to > make a single installer which installed all of those parts, not > necessarily frozen. The package installer we are currently discussing could handle that as well - almost. It should in any case be possible to install several packages at the same time; otherwise packages with circular dependencies could not be installed at all. So if the installer could also treat an archive of packages (in whatever convenient format), it would be easy to set up distributions of multiple packages. The reason I said "almost" is that the Python interpreter must always be there first. At least for Unix, I see no way around this, and for other platforms I don't know. And (again for Unix) I don't think anyone would object to install two things for some application. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Fri Dec 18 18:02:29 1998 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 18 Dec 1998 19:02:29 +0100 Subject: [Distutils] Access to Python config info In-Reply-To: <13946.35058.910774.305483@weyr.cnri.reston.va.us> (fdrake@cnri.reston.va.us) References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> <13946.35058.910774.305483@weyr.cnri.reston.va.us> Message-ID: <199812181802.TAA03844@dirac.cnrs-orleans.fr> > Greg Ward writes: > > *This* version should run as one of the last steps of the distutils > > installation process, ie. this is the hack that'll work with Python > > If we can agree that it includes sufficient information, and we can > get the other platforms supported, why can't we ask Guido to include > it in 1.5.2? That would also allow other projects that have differing Because if we later find out that we need a somewhat different interface or functionality, we have a compatibility problem. I see no need to rush - all information is already available in Python 1.5.1, it just takes some parsing. Konrad [now definitely leaving...] -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Fred L. Drake, Jr." References: <19981216090359.A28921@cnri.reston.va.us> <3677DDCE.6FA9DC89@lemburg.com> <3678281D.4FC0C4F6@lyra.org> <13945.16759.231398.717797@weyr.cnri.reston.va.us> <19981217183616.A29344@cnri.reston.va.us> <13946.35058.910774.305483@weyr.cnri.reston.va.us> <199812181802.TAA03844@dirac.cnrs-orleans.fr> Message-ID: <13946.39259.976078.780082@weyr.cnri.reston.va.us> Konrad Hinsen writes: > Because if we later find out that we need a somewhat different > interface or functionality, we have a compatibility problem. > I see no need to rush - all information is already available in I don't think this is a problem, really. The question about whether it really belongs in the standard distribution at all is more important. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From da@skivs.ski.org Fri Dec 18 18:02:38 1998 From: da@skivs.ski.org (David Ascher) Date: Fri, 18 Dec 1998 10:02:38 -0800 (Pacific Standard Time) Subject: [Distutils] Configuration etc. In-Reply-To: <199812181757.SAA14516@dirac.cnrs-orleans.fr> Message-ID: On Fri, 18 Dec 1998, Konrad Hinsen wrote: > The package installer we are currently discussing could handle that as > well - almost. It should in any case be possible to install several > packages at the same time; otherwise packages with circular > dependencies could not be installed at all. So if the installer could > also treat an archive of packages (in whatever convenient format), > it would be easy to set up distributions of multiple packages. > > The reason I said "almost" is that the Python interpreter must always > be there first. At least for Unix, I see no way around this, and for > other platforms I don't know. And (again for Unix) I don't think > anyone would object to install two things for some application. I haven't had a chance to look at all the proposals in detail, but I just wanted to remind folks that not all "extensions" are really Python extensions, but they'd be nice things to have in a distribution anyway -- anything from non-Python specific libraries (netcdf, gd, t1lib, whatever) to large things like Tcl/Tk. Tcl/Tk is especially important for the usual reasons. In a word, some thought should be put into thinking about other non-controllable systems (e.g., no matter what Greg W says, Ousterhout isn't going to follow his recommendations =). I don't think we can or should be all-encompassing, but *thinking* a bit about these problems could avoid some pitfalls. --david From gward@cnri.reston.va.us Fri Dec 18 22:10:21 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 18 Dec 1998 17:10:21 -0500 Subject: [Distutils] Access to Python config info In-Reply-To: <36780C91.59EF05A4@lyra.org>; from Greg Stein on Wed, Dec 16, 1998 at 11:40:01AM -0800 References: <19981216090359.A28921@cnri.reston.va.us> <36780C91.59EF05A4@lyra.org> Message-ID: <19981218171021.B1992@cnri.reston.va.us> Quoth Greg Stein, on 16 December 1998: > I believe the public interface will/should always be a set of names and > values. Even if we do a hack in lieu of native Python support, we should > just set globals for use by clients. > > There may be functions in there, too, for compiler invocation and stuff > (as John recommends). I'm not clear on whether that needs to be in > sysconfig.py or part of the distutils package (forgot the pkg name). I agree: sysconfig.py should be a flat list of globals. A brain-dump of Python's configuration info, as it were. This implies that any functions/methods (eg. for invoking the compiler) belong in some distutils module. After all, this is the 'distutils' project; as much code as possible should go in the 'distutils' package. sysconfig should just be a flat braindump. > All right, you Perl hacker. We'll have none of this here! Python doesn't > have "hashes" ... it has dictionaries! Also, it is always possible to > view module globals as a dictionary (with vars(sysconfig)), so there > isn't a need for another level containing a dictionary. > > Go home Perl hacker! My list of gripes with Python is pretty small, but I think the main one is that the word "Dictionary" is just too long to type. And when you write as many comments as I do, that matters... ;-) Just another /^P\w+/ hacker... Greg PS. Oops, I just realized that regex also lets Pascal in, and PL1 (of course it should be written "PL/1" [or is it "PL/I"?]), and Prolog... hmmm... back to the drawing board... -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From Fred L. Drake, Jr." References: <19981216090359.A28921@cnri.reston.va.us> <36780C91.59EF05A4@lyra.org> <19981218171021.B1992@cnri.reston.va.us> Message-ID: <13946.54495.173514.954136@weyr.cnri.reston.va.us> Greg Ward writes: > I agree: sysconfig.py should be a flat list of globals. A brain-dump of > Python's configuration info, as it were. I don't see any big reason not to keep it in distutils, either, as long as it can be done the way I did the preliminary thing. I think it's even reasonable to just let it do the information discovery each time, which might make it easier to update Python without having to do anything with an existing distutils installation (in case it's not in there). > My list of gripes with Python is pretty small, but I think the main one > is that the word "Dictionary" is just too long to type. And when you That's why many of us spell it "dict". ;-) > PS. Oops, I just realized that regex also lets Pascal in, and PL1 (of > course it should be written "PL/1" [or is it "PL/I"?]), and I think either, depending on which you used. Wasn't one a subset of the other? -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From gward@cnri.reston.va.us Fri Dec 18 22:26:13 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 18 Dec 1998 17:26:13 -0500 Subject: [Distutils] Configuration etc. In-Reply-To: <199812181528.QAA17720@dirac.cnrs-orleans.fr>; from Konrad Hinsen on Fri, Dec 18, 1998 at 04:28:21PM +0100 References: <199812181528.QAA17720@dirac.cnrs-orleans.fr> Message-ID: <19981218172613.C1992@cnri.reston.va.us> Quoth Konrad Hinsen, on 18 December 1998: > Everything lower-level is an implementation detail and not worth > 10 KB of discussion ;-) True to a certain extent -- but hey, we're programmers; we can't help indulge in talking about low-level implementation details even when we don't really have a high-level design. But you're right that this *is* a detail and should take a backseat to design matters for the time being, although I think the discussion on sysconfig has been very useful and productive. > And now for something completely different: implementation strategy. > At some point we will have to move from discussing to programming, > after all. It seems to me that the whole DistUtils project divides > nicely into two almost independent subunits: > > - building binaries from source code > > - installing Python modules and binary extension modules I know you (and most of the people who've been active on the SIG, for that matter) weren't there, but we covered this pretty well at the Developer's Day session. The split is subtly different, into three roles: * developer - builds from source tree, creates source distribution, possibly creates built distribution(s) * packager - builds from source distribution, creates built distribution(s) * installer - one of: - builds from source distribution, installs - installs from built distribution See the session summary at http://www.foretec.com/python/workshops/1998-11/dd-ward-sum.html for details. If you don't want to read the whole bloody thing, scroll down to the big "workflow" diagram. This really illustrates a lot of what you're talking about, and what needs to be automated by the distutils. > So maybe we should create two "task forces" that work on these > subtopics, of course communicating as necessary. Not a bad idea, but it would be *three* task forces: for building, for creating distributions (source or built; built distributions can further be smart or dumb), and for installing. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Fri Dec 18 23:54:36 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 18 Dec 1998 18:54:36 -0500 Subject: [Distutils] Anonymous CVS up-and-running Message-ID: <19981218185436.A2845@cnri.reston.va.us> Hi all -- undoubtedly it's wildly premature to have source available before we have any sort of high-level design, but what the hell. Go visit http://www.python.org/sigs/distutils-sig/cvs.html to find out how to access it. The quick version: use CVS (1.9 or greater) to login to cvs.python.org as 'distutilscvs'; the password is 'painless'. Checkout the 'distutils' tree. Poke around. The interesting source is in 'distutils' (ie. 'distutils/distutils' from the directory where you run 'cvs checkout). Only things there so far are my version number module and Fred's sysconfig module. (Which is currently actually 'distutils.sysconfig', maybe not a bad thing.) version.py is copiously commented and guaranteed to start another flamewar. So if you do look at it, *please* don't post your immediate thoughts to the list. Send 'em my way if you must vent your spleen. Version numbers are still a banned topic until we have some idea of what they'll be used for (ie. high-level design). Anyone who wants to take a stab at a high-level design should once again pore over the requirements and proposed user interface on the SIG web site. We haven't had any discussion about the proposed interface, and I very much doubt that it's perfect in every way. I think talking that over is a sensible thing to do before starting in on the design. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skaller@maxtal.com.au Sat Dec 19 18:16:56 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:16:56 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181656.00f7f3f0@triode.net.au> At 10:49 16/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > 1) does the OS support symbolic or hard links? > > import os > hasattr(os, "link") > hasattr(os, "symlink") > HOW do you know this works? HOW do you know that the method is not, in fact, present, but doesn't work? WHERE is the standards document that specifies the precise semantics of the 'os' interface?? This technique doesn't seem very reliable. :-( > > It may be the proper place for the answer is not in sys.config, > > but in os. But the os module can compute the answers from > > sys.config. > > For most of this, probably so. Recall also that Guido & others >recommend "feature tests" to determine the availability of >functionality rather than having a separate flag to indicate >availability. There are many ways in which this is more reliable and >maintainable, And some in which it is not reliable :-( >though what we're currently describing as sysconfig >remains pertinant. I would expect sysconfig to be used mostly by >things like distutils and not by typical programs. Hence, it would be >good for os to not import it. That may be so. However, 'feature tests' are useless for meta-programming. What I mean is: suppose one generated some code. If it had to be different depending on 'features', one could do the feature tests and generated it that way; but this would only work on the current platform. If you were generating code for someone else; then the feature tests would be useless ... unless they were done _by_ the generated code. Which is the autoconfig nightmare all over again. What Guido is recommending is exactly like the old fashioned DOS driver problem: test the video, or printer to see what kind it is, and configure a driver: usually by selecting it from a list a mile long _supplied with every single program_. I don't think this is precisely the way to do it. Dynamical feature tests are fine if they are done ONCE on installation; and present the results in a platform independent way. All modern operating systems have an abstract device driver API, and the _operating system_ installs the appropriate driver once when it is configured. With any luck, the configuration record can be ported, so it is possible to generate code for any configuration, (i.e. on a remote host): the generator _might_ optimise the code my providing some dynamic testing (of the configuration record on the client). Of course, the reliability of this mechanism does depend on establishing standards, and then making the implementation(s) comply. There's no reason the configuration module shouldn't do these dynamical tests _as implementation details_ of the implementation of a standardised API. If they're appropriate. _I_ don't want to do this hackery in code, however. It's too much like a bag of trickery. I'd be happy with a module containing: def has_symlinks(): import os return hasattr(os, "link") with a _specification_ of the function has_symlinks because this would allow me to create an alternate implementation (which could, for example, use a data record to represent the configuration of some remote system). Does this make sense? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:16:58 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:16:58 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181658.00f75d68@triode.net.au> At 11:11 16/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 17 December 1998 [on whether the config module >should be builtin or a separate .py file]: >> I don't see any real choice: it has to be separate AND editable. >> Because if I download a binary Windows version, the information will be wrong >> for my system. The Unix version might be wrong too: perhaps I built >> Python with gcc, but I want to build extensions with ecgs, or with edg. > >There are a number of reasons why you don't want to do this, and it all >boils down to the fact that in the C world you cannot reliably link two >object files if one was compiled with compiler X and the other with >compiler Y. I agree: this is a valid point. >Anyways, these all constitute reasons for *strong recommendations* >against fiddling with the compiler or flags. I'll concede that someone >who *really* knows what they're doing should be allowed to do this (but >they should also have to sign a disclaimer first ;-). I think you are missing the point here: if the configuration doesn't work, the client has NO CHOICE but to edit it. And if it isn't editable, the client has NO CHOICE, full stop. Now, you don't want to leave the client without options, in case something as complex as configuring a compiler fails, do you? At least, they should be able to ask for help on the newsgroup, and edit the file with help from experts, shouldn't they? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:05 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:05 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181705.00f8cb14@triode.net.au> >It *should* be done by the installation Makefile. I would hope that we >obsolete the third-party use of Setup/config.h/Makefile.pre.in. Those >are even close to the convenience that we can establish. Yes! I think this is a primary goal. I will note: it will _not_ work for all modules because of the problem finding libraries and header files. But it will work for all modules which implement 'pure' data structures, and it will work for all 'opt' modules that are just optimisations of Python script. This latter class is _crucial_; in fact it is vastly more important than distribution of modules: it permits dynamic (run-time) generation of C modules. In particular, one could write a 'compiler' that generated either Python or C from some language (such as Python :-) .. IN Python. (Because apart from generating text, which Python can already do, the only thing else needed is to compile that text into a binary: this is the missing link which will make C extensions work like Python modules). BTW: It is _also_ recommended to have the ability to generate applications, not just Python modules. The 'make' system know how to do that too, and it is useful for extending Python with 'shell' tools. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:07 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:07 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181707.00f85860@triode.net.au> At 18:36 17/12/98 -0500, Greg Ward wrote: >Well, Fred, you've set the ball rolling... I guess coding and design >will, as usual, proceed hand-in-hand... ;-) I'll modify my compilers modules to use the information as presented in this module. That should upgrade it from Redhat Linux to 'arbitrary Unix system'. And might even work on a Cygwin system. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:00 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:00 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181700.00f7a1e4@triode.net.au> At 17:20 16/12/98 +0100, M.-A. Lemburg wrote: >Greg Ward wrote: >> >> On the superficial front, Marc-Andre hasn't convinced me that sysconfig >> is a better name than sys.config. >So the reasons I renamed it from sys.config to sysconfig were simply: I think I favour this, but not strongly. >> No! At least not in the long term: as Greg Stein pointed out, the >> config module (whatever it gets called) should just be generated when >> Python is built. Or were you talking about the near term, where we will >> need a distutils release that works with Python 1.5.2 and is allowed all >> manner of grotesque hackery to work around the lack of a config module, >> such as digging up and parsing Python's config.h, Makefile, and Setup? > >I wouldn't consider this grotesque hackery. I would. But it doesn't matter. What matters is not how it is implemented, but the interface it is required to have. (Where 'interface' includes semantics) The principal task of this effort is _not_ implementation but standardisation. >The parsed information could then be stored in a _sysconfig module >which subsequent invocations then use which is close to what Greg >proposed except that the first run is not necessarily done by the >installation Makefile. Yes. And there are number of approaches, some to be used before the module is distributed as standard, and some after. We need to know we _can_ implement it, and we need some actual implementations to examine, but this is not as important as standardising the API. What we need to produce is a specification, not (just) an implementation. And I come back again to the point that what the client needs is a module that actually compiles C code to a dynamically loadable shared library, so that the supplier can provide the C code _without_ any configuration stuff. Getting the compiler flags, etc, is all very well, but it isn't what is required by the end user. The person who needs all these flags is probably the supplier .. when the client complains the code doesn't compile. So it is important that the detail data be portable (i.e. can be physically sent across the internet). In my opinion, what is important is the API: the actual function signatures used to compile C code. Not how they're implemented. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:08 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:08 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981219181708.00f74c68@triode.net.au> At 15:46 17/12/98 -0800, Greg Stein wrote: >hehe... boy, you do like things in a particular way, don't you? Now I >know where the version number stuff came from :-) i do think it is fair for the coordinator to ask for help organising things. Although I'll have a problem with CVS/RCS myself. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:11 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:11 +1000 Subject: [Distutils] Freeze and new import architecture Message-ID: <1.5.4.32.19981219181711.00f9bba0@triode.net.au> At 18:15 17/12/98 -0800, Greg Stein wrote: >> ... >> sys.path not be restricted to path names. > sys.path has "strings", and >> an associated map of "module finders". Thus, a sys.path entry could >> have a directory name (like now) or .zip file, URL, etc. > >I would much prefer to see the module finder instances in the sys.path. I agree. But that would be a compatibility problem? >Sometimes, it is *very* difficult to map strings to module finders. Yes. I think this is a serious weakness of the proposal. >> 1. Finding the module in a specific namespace. >> 2. Importing a module of a specific type, once it has been found. > >I think the separation is bogus. I don't: I'd _like_ to add two things, as a client of the system: 1) Add a new type. For example, allow .c files to be loaded, by compiling them first. 2) Add a new kind of namespace. For example, an FTP server. Or a hook to Trove. Or a private data structure I designed myself. (1) has to do with what kinds of things are loaded, whereas (2) has to do with where they are. Note that the finder must be able to _fetch_ the data to a place that the loader can load it from. If a single place is enough, then the two features are orthogonal, and thus each is separately amenable to Object Oriented development. So .. it is desirable to build an abstraction in which the functionality is separate. >> Regarding 2: the finder currently returns a structure that enables the >> correct >> importer to be called later on. Importers that we have are for >> builtin, >> frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind >> the >> same interface, PYC-resource importers (mac-only) and PYD-resource >> importers >> (mac-only). > >Punt this. Just import the dumb thing in one shot. But you can't: a .dll file is imported by saying dlopen(), whereas a .py file is imported by compiling it to a .pyc file which is then imported. Etc. 'One shot' implies a single function which is not extensible. >Take the example of an HTTP-based import. Separating that into *two* >transactions would be painful. It should be imported in one fell swoop. >And no, you can't just keep the socket open and pass that to the loader >-- that implies that you can defer the passing for a while, but the web >server will time out your connection and close it. Conversely, if the >intent is *not* to hold the "structure" for a while, then why the heck >have two pieces? The way I see it, the _finder_ is responsible for downloading the file to the local file system, where the loader requires it to be. The loader turns these raw bits into a module. >Both of our proposals guarantee that stuff in sys.path are not >pathnames. If I insert a "foo.zip" or a >"http://host.domain.name/pymodules/", then you certainly dont have >pathnames. > >I believe the biggest issue with my proposal is the fact that the values >are no longer strings. That's easy to fix: have a default 'finder' that is used if the sys.path entry is a string. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Sat Dec 19 18:17:10 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 04:17:10 +1000 Subject: [Distutils] Freeze and new import architecture Message-ID: <1.5.4.32.19981219181710.00f9f778@triode.net.au> At 11:45 18/12/98 +1100, Mark Hammond wrote: > WHAT WE WANT > ============ > >What we'd like I'll try to describe top-down (hopefully better to >understand >than bottom-up). > >importing a module becomes something like > > for pathentry in sys.path: > finder = getfinder(pathentry) > loader = finder.find(module) > module = loader() >getfinder() is something like > if not path_to_finder.has_key(pathentry): > for f in all_finder_types: > finder = f.create_finder(pathentry) > if finder: > path_to_finder[pathentry] = finder > break > return path_to_finder[pathentry] I think this is excellent! But I'm not clear on how every kind of finder can be created for every path. [Also, it should be possible to flush the cache] ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gstein@lyra.org Sat Dec 19 22:20:51 1998 From: gstein@lyra.org (Greg Stein) Date: Sat, 19 Dec 1998 14:20:51 -0800 Subject: [Distutils] Freeze and new import architecture References: <1.5.4.32.19981219181711.00f9bba0@triode.net.au> Message-ID: <367C26C3.6F74412@lyra.org> I think you may have misunderstood some of my points, so I'll try to clarify below. John Skaller wrote: > > At 18:15 17/12/98 -0800, Greg Stein wrote: > >> ... > >> sys.path not be restricted to path names. > > sys.path has "strings", and > >> an associated map of "module finders". Thus, a sys.path entry could > >> have a directory name (like now) or .zip file, URL, etc. > > > >I would much prefer to see the module finder instances in the sys.path. > > I agree. But that would be a compatibility problem? Nope. I pointed out that a default installation would only have strings in there. A person would not have a compatibility problem until they chose to start using a special importer. In that case, I don't define it as a problem because they chose to do so. Of course, we would need to update various standard modules to improve their handle of sys.path. But again that is not a compatibility issue. >... > >> 1. Finding the module in a specific namespace. > >> 2. Importing a module of a specific type, once it has been found. > > > >I think the separation is bogus. > > I don't: I'd _like_ to add two things, as a client > of the system: > > 1) Add a new type. For example, allow .c files to be loaded, > by compiling them first. > > 2) Add a new kind of namespace. For example, an FTP server. > Or a hook to Trove. Or a private data structure I designed > myself. > > (1) has to do with what kinds of things are loaded, > whereas (2) has to do with where they are. Yes, it would be nice to do those things. However, I don't see that you need the separate find/import paradigm to do it. If your particular importer that you've placed into sys.path wants to use two steps, then fine. If your importers share functionality using those steps, then kudos to them. Note that Python's __import__ hook is a single step(!). The two part find/load scheme is an artifact of ihooks, not Python itself. And I disagree that we need to formalize it within Python itself. I simply maintain that we should have a very simple interface from Python to any import system. In summary, that is placing importers into sys.path and invoking a "do_import" method on them. Simple and clean. > Note that the finder must be able to _fetch_ the data to a place > that the loader can load it from. If a single place is enough, > then the two features are orthogonal, and thus each is separately > amenable to Object Oriented development. > > So .. it is desirable to build an abstraction in which > the functionality is separate. You can build your importers this way, but I don't think we need to place that mechanism in Python. Personally, I have issues with the style of "put it into a temporary location, then import it". It seems subject to race conditions and/or /tmp hacks. > > >> Regarding 2: the finder currently returns a structure that enables the > >> correct > >> importer to be called later on. Importers that we have are for > >> builtin, > >> frozen, .py/.pyc/.pyo modules, various dll-importers all hiding behind > >> the > >> same interface, PYC-resource importers (mac-only) and PYD-resource > >> importers > >> (mac-only). > > > >Punt this. Just import the dumb thing in one shot. > > But you can't: a .dll file is imported by saying dlopen(), > whereas a .py file is imported by compiling it to a .pyc file which > is then imported. Etc. Sorry, I meant "punt the whole structure thing". In my little corner of the universe, I don't believe we have two steps, so we don't need to formalize any mechanism for passing state between them. Basically, I see the state thing as a compensation for introducing the two-step find/load into Python's single-step import mechnaism. > 'One shot' implies a single function which is not extensible. This is just argumentative. My proposal is just as extensible, and I would maintain that it is simpler for the interpreter, and simpler for many importers (rather than import-writers needing to deal with the funky two-step). > > >Take the example of an HTTP-based import. Separating that into *two* > ... > The way I see it, the _finder_ is responsible for > downloading the file to the local file system, where the > loader requires it to be. The loader turns these raw bits > into a module. As I mentioned before, I (personally) don't like this style. I'd rather write an importer that loads it straight in from the wire. If it hits the disk, then it would be *very* transitory. The two-step thing that returns state structures implies an indeterminite time between those steps, which I think is wrong. > > >Both of our proposals guarantee that stuff in sys.path are not > >pathnames. If I insert a "foo.zip" or a > >"http://host.domain.name/pymodules/", then you certainly dont have > >pathnames. > > > >I believe the biggest issue with my proposal is the fact that the values > >are no longer strings. > > That's easy to fix: have a default 'finder' that is used > if the sys.path entry is a string. I meant "values are no long [only] strings". Please review my proposal again; you'll note in the path processing that I tested for a string and call "old_import" to import things using Python's current string-based mechanism. Cheers, -g -- Greg Stein, http://www.lyra.org/ From skaller@maxtal.com.au Sun Dec 20 13:14:09 1998 From: skaller@maxtal.com.au (John Skaller) Date: Sun, 20 Dec 1998 23:14:09 +1000 Subject: [Distutils] Anonymous CVS up-and-running Message-ID: <1.5.4.32.19981220131409.0094c5a0@triode.net.au> At 18:54 18/12/98 -0500, Greg Ward wrote: >Hi all -- > >undoubtedly it's wildly premature to have source available before we >have any sort of high-level design, but what the hell. Absolutely NOT! Gee, haven't you learned some OO? OO is a bottom up. You do detail work BEFORE you have a high level design. What you have done is absolutely right. Thanks!! >Only things there so far are my version number module and Fred's >sysconfig module. --------------------------------------------- Please put my compilers module in it too. It provides the high level abstract interface that is actually needed. At least, it tries to. It needs work! It needs a third component for Java as well. --------------------------------------------- I have incorporated the sysconfig module into interscript. Here are my comments on the sysconfig module. 1) C++ is not supported. If the option is not provided to build py++, a separate distribution may follow. Do we want that? [py++ is the same as python, except that the main translation unit is built with C++: the C built library is still used] We need the Makefile to support C++ so sysconfig can get C++ compilation flags. 2) Some of the data, such as CFLAGS, is useless because it reflects the python source build tree. On my system CFLAGS = '-g -O2 -I./../Include -I.. -DHAVE_CONFIG_H' 3) The directory for config.h is WRONG! config.h lives in /usr/local/include/python1.5 but the sysconfig module looks in /usr/local/lib/python1.5 CHANGE the line which gets this wrong TO: config_dir = os.join( sys.exec_prefix, 'include','python'+version) 4) On my system: SIZEOF_INT = '4' which is wrong, it should be SIZEOF_INT = 4 I got this using 'repr(value)'. Most other numeric values are OK. 5) I think the module should NOT just put values into itself like FLAG = VALUE I suggest: C_CAPABILITIES -- dictionary or module containing stuff from config.h MAKE_FLAGS -- dictionary or module containing Makefile variables be attributes of sysconfig. If you do not like writing sysconfig.C_CAPABILITIES['HAVE_STDDEF_H'] then go for the module, which allows: sysconfig.C_CAPABILITIES.HAVE_STDDEF_H 6) There is no information from Setup. It would be useful to get the Tcl/Tk library and headers into the configuration if it was built. Hmmm. :( 7) _init_posix should NOT modify a dictionary, but return one: CHANGE: g = globals() to g = {} and add return g to the end of the routine. Replace the exec line with g = eval('_init_%s()' % os.name) and add for k in g: globals()[k] = g[k] if you really want to modify the module dictionary. I don't. I use the data to generate the python source of another module. This is important to me: the generated python can be transmitted over the Internet. It can be edited. And I can have several distinct configurations. What I do is that the build code is NOT in sysconfig, but inline interscript (python) code which is executed to BUILD a file 'sysconfig.py'. 8) LINKCC = ' gcc' has a bad leading space. 9) I have no idea how to use the available information. There are flags all over the place. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From M.Faassen@vet.uu.nl Mon Dec 21 17:57:55 1998 From: M.Faassen@vet.uu.nl (Martijn Faassen) Date: Mon, 21 Dec 1998 18:57:55 +0100 Subject: [Distutils] High level options interface Message-ID: <367E8C23.EE9D8545@pop.vet.uu.nl> Hello, I've been following the discussion on how to build C (or C++, or other languages) extensions with interest. I'm thus talking about the 'build_extensions' command of the proposed interface, if I understand it well. Some talk has gone into reading a lot of low level Autoconf results into Python, and exposing them through a sysconfig module. Some objections were raised that this was rather low level. It's also not very portable, right? Options that go into gcc won't make much sense for Visual C++, I imagine. Now, we could try to abstract a compiler framework in Python. I understand this is what John Skaller is working on, but I may be wrong as I haven't looked yet. Abstracting a compiler with its myriads of options could be a pain. We might end up having to abstract makefiles as well in Python. And building C libraries. This could be a rather big project. We'd be duplicating a lot of stuff, as well. I'm not too keen on simply calling 'make' directly from our builder script either, though. That's rather tied to the platform. So let's try abstracting some more. See if it becomes any simpler. I think this is what other people might've proposed as well, and if so, I'm just saying 'me too', here. I'll sketch out a very simplistic and anarchic high level interface here. The base class: class extension_compiler: def compile_C(filename): pass def compile_C++(filename): pass That's about it, unless we want to choose to support more extension languages by default. For each platform we want to support, we supply a subclass of this. For example: class gcc_extension_compiler(extension_compiler): def compile_C(filename): os.system("gcc -c " + filename) def compile_C++(filename): os.system("gcc -c " + filename) That's just about all. We don't have any fancy options (though we might want to enable optimizations by default). But what about special optimizations, debugging, profiling, other fancy stuff? Our abstraction doesn't support this! We need more methods. We need set_optimization_level(10)! Do we? Who does? Generally, only the extension developer knows which special compiler options could work best with the extension code. The extension writer only knows this for his platform(s) of choice, too. Extensions should still build cleanly for anyone on any platform. Thus we do best with as abstract an interface as possible. The extension builder should be given the option to subclass, and provide special compilation options for optimization purposes. Profiling and debugging builds are generally only necessary for the extension developer, and could be handled by special throw-away subclasses. If the developer has figured out how to make extensions build best and fastest on Linux, he can always include a special purpose linux_extension_compiler subclass which calls the compiler with special optimization options. If later it turns out we do need special option settings (for all compilers on all platforms!), we can always add it later. Anyway, I'm leaving lots of stuff out here. We probably need to do the same simplistic abstraction to the linker. The emphasis is to keep the interface as small as possible. A large interface means supporting it on all kinds of different platforms, and that is a lot of work. Once we got a small basic interface going, we can always extend it, platform by platform, with more options, if necessary. A simple 'RISC' interface does limit developers. They can't do all kinds of fancy makefile stuff they usually do. But we don't want to have to support all the fancy makefile stuff anyway. ....remembers ./configure ./make and why the complicated ./configure step is there in the first place; to enhance portability. Hm. *ponder* Okay. Perhaps that invalidates my entire argument above. It might be that I'm limiting developers from using any external libraries in their extensions. Still, philosophically, I think my arguments are sound. In practice, however... Leaving-confused-ly yours, Martijn From Fred L. Drake, Jr." References: <1.5.4.32.19981219181656.00f7f3f0@triode.net.au> Message-ID: <13950.40964.713688.311939@weyr.cnri.reston.va.us> John Skaller writes: > HOW do you know this works? HOW do you know > that the method is not, in fact, present, but doesn't work? > WHERE is the standards document that specifies the > precise semantics of the 'os' interface?? Sounds like I'd better get the documentation for "os" fixed before I release again; it is on my list. > This technique doesn't seem very reliable. :-( How so? (Given that functions will be provided only when the underlying system defines the corresponding capability.) Your issue with this sounds like fixing the documentation would be sufficient. > That may be so. However, 'feature tests' are useless > for meta-programming. What I mean is: suppose one generated > some code. If it had to be different depending on 'features', ... > If you were generating code for someone else; > then the feature tests would be useless ... unless they > were done _by_ the generated code. These are runtime tests. Generated code that relies on the result should make them as appropriate. Note that the only time they are appropriate *ever* is if you provide alternate functionality (however simple) for platforms that fail the test, or if you want to remove options from view in a user interface for platforms which don't offer the functionality. Any other time the AttributeError exception is appropriate, and can be caught if needed. > Which is the autoconfig nightmare all over again. > > What Guido is recommending is exactly like the > old fashioned DOS driver problem: test the video, or printer > to see what kind it is, and configure a driver: usually > by selecting it from a list a mile long _supplied with > every single program_. > > I don't think this is precisely the way to do it. > Dynamical feature tests are fine if they are done ONCE > on installation; and present the results in a platform So once installed, the result ends up being platform specific. In many environments, this is not acceptable. If it is, then your installation-time tests and configuration can be built on top. > _I_ don't want to do this hackery in code, > however. It's too much like a bag of trickery. > I'd be happy with a module containing: [example has_symlinks() elided] But this can be written on top of feature tests. Why not? If features are expected to be tested may times, repeatedly, the implementation could be somewhat auto-optomizing by having the functions replace themselves with "fast" versions that just return the result for subsequent queries: def has_symlinks(): global has_symlinks import os if hasattr(os, "symlink"): has_symlinks = true return 1 else: has_symlinks = false return 0 def true(): return 1 def false(): return 0 > with a _specification_ of the function has_symlinks Have you created a list of the system aspects that this is desirable for? Is the list *finite*? It may well be, but there will be a lot of disagreement over what it should be. For now, let's call this the "features" module. Are you ready to propose an API? I'm sure several of us would be willing to read it and carefully consider it (I will, though it may not sound like it;). I think the biggest problem is defining it in a way that provides more information or more convenient access to the information represented. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981220131409.0094c5a0@triode.net.au> Message-ID: <13950.42354.720812.271393@weyr.cnri.reston.va.us> John Skaller writes: > --------------------------------------------- > Please put my compilers module in it too. > It provides the high level abstract interface that is actually needed. Good idea. For those of use that haven't had time to play with interscript, having it available in the repository would make it easier to look at. ;-) It'll also make it easier to modify it as we go along without John having to maintain compatibility with working versions of interscript; there's no reason to tie John's release schedule with our bickering. > I have incorporated the sysconfig module into interscript. > Here are my comments on the sysconfig module. Cool! > 1) C++ is not supported. If the option is not provided to build > py++, a separate distribution may follow. Do we want that? Agreed. I seem to recall someone had sent patches to support C++, possibly to the C++-SIG. Does anyone know exactly what's needed? > 2) Some of the data, such as CFLAGS, is useless because it reflects > the python source build tree. On my system Misc/Makefile.pre.in manages to get things right in practice; some time needs to go into figuring out just how it does things and get sysconfig to do the right thing as well. I won't have time for it today; may in January. ;-( > 3) The directory for config.h is WRONG! > config.h lives in What definition are you looking at? > 4) On my system: > > SIZEOF_INT = '4' Patch attached which gets this right in a general fashion. > 5) I think the module should NOT just put values > into itself like I don't see any need to complicate the interface. > 6) There is no information from Setup. It would be useful to > get the Tcl/Tk library and headers into the configuration > if it was built. Hmmm. :( Variables defined in Setup & friends are included. Otherwise, nothing from Setup is included. This can be changed. It may make sense to use a separate module for this (modconfig?). > 7) _init_posix should NOT modify a dictionary, but return one: > CHANGE: This item is tightly related to item 5; let's deal with that one as the real issue. The rest will fall out of that result. > 8) LINKCC = ' gcc' has a bad leading space. Fixed in the patch. > 9) I have no idea how to use the available information. > There are flags all over the place. Yes, that's what configure produces. The next layer needs to put this information together in a usable form. Reading the Makefile should help a lot in determining how the various bits should be put together. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 Index: sysconfig.py =================================================================== RCS file: /projects/cvsroot/distutils/distutils/sysconfig.py,v retrieving revision 1.1 diff -c -c -r1.1 sysconfig.py *** sysconfig.py 1998/12/18 23:46:33 1.1 --- sysconfig.py 1998/12/21 19:43:10 *************** *** 15,20 **** --- 15,21 ---- def _init_posix(): import os import re + import string import sys g = globals() *************** *** 35,44 **** m = define_rx.match(line) if m: n, v = m.group(1, 2) ! if v == "1": ! g[n] = 1 ! else: ! g[n] = v else: m = undef_rx.match(line) if m: --- 36,44 ---- m = define_rx.match(line) if m: n, v = m.group(1, 2) ! try: v = string.atoi(v) ! except ValueError: pass ! g[n] = v else: m = undef_rx.match(line) if m: *************** *** 57,65 **** --- 57,68 ---- m = variable_rx.match(line) if m: n, v = m.group(1, 2) + v = string.strip(v) if "$" in v: notdone[n] = v else: + try: v = string.atoi(v) + except ValueError: pass done[n] = v # do variable interpolation here *************** *** 79,85 **** if "$" in after: notdone[name] = value else: ! done[name] = value del notdone[name] elif notdone.has_key(n): # get it on a subsequent round --- 82,90 ---- if "$" in after: notdone[name] = value else: ! try: value = string.atoi(value) ! except ValueError: pass ! done[name] = string.strip(value) del notdone[name] elif notdone.has_key(n): # get it on a subsequent round *************** *** 91,99 **** if "$" in after: notdone[name] = value else: ! done[name] = value del notdone[name] else: del notdone[name] # save the results in the global dictionary --- 96,107 ---- if "$" in after: notdone[name] = value else: ! try: value = string.atoi(value) ! except ValueError: pass ! done[name] = string.strip(value) del notdone[name] else: + # bogus variable reference; just drop it since we can't deal del notdone[name] # save the results in the global dictionary From skaller@maxtal.com.au Tue Dec 22 11:59:30 1998 From: skaller@maxtal.com.au (John Skaller) Date: Tue, 22 Dec 1998 21:59:30 +1000 Subject: [Distutils] High level options interface Message-ID: <1.5.4.32.19981222115930.00f7b9b0@triode.net.au> At 18:57 21/12/98 +0100, Martijn Faassen wrote: >class extension_compiler: > def compile_C(filename): > pass > def compile_C++(filename): > pass There's no need for a base class, just a specification. Please examine the following interface and see a gcc implementation below that. At least criticize the interface if you don't like it. The interface: # module 'interscript/compilers.c class python_module: def __init__(self,**kwds): # initialise the object with kwds as configuration. def configure(self,**kwds): # extend or change configuration def compile(self,filename, **kwds): # compile the file, with given options # return the string name of the object file def link(self,modname, filenames, **kwds): # link the list of object files, with given options # to generate the named python module. # returns the name of the shared library class application: # exactly the same, except produces an executable # tool that can be invoked from the command line # module 'interscript/compilers.cpp # exactly the same, except for C++ ---------------------------------------------------------------------------- ------ #line 121 "compilers.ipk" import os import sys import string import interscript.compilers.cconfig class python_module: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc = 'gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,modname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) dll = modname +'.so' cc ='gcc -shared -Xlinker -export-dynamic ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + dll print 'system',repr(arg) os.system(arg) return dll class application: def __init__(self,**kwds): self.config = interscript.compilers.cconfig.config() self.config.append_dict(kwds) def configure(self,**kwds): self.config.append_dict(kwds) def compile(self,filename, **kwds): config = self.config.copy() config.append_dict(kwds) base = string.join(string.split(filename,'.')[:-1],'.') obj = base+'.o' cc ='gcc -g -O2 -fpic -fPIC -pedantic ' inc = '-I' + sys.prefix + '/include/python1.5 ' if sys.prefix != sys.exec_prefix: inc = inc + '-I' + sys.exec_prefix + '/include/python1.5 ' cstr = str(config)+' ' arg = cc + cstr + inc + '-c '+filename + ' -o '+ obj print 'system',repr(arg) os.system(arg) return obj def link(self,appname, filenames, **kwds): config = self.config.copy() config.append_dict(kwds) cc ='gcc ' cstr = str(config) + ' ' lib = '-L' + sys.exec_prefix + '/lib/python1.5 ' files = string.join(filenames) + ' ' arg = cc + cstr + lib + files + '-o ' + appname print 'system',repr(arg) os.system(arg) return appname ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From gward@cnri.reston.va.us Tue Dec 22 13:07:47 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 22 Dec 1998 08:07:47 -0500 Subject: [Distutils] Access to Python config info In-Reply-To: <1.5.4.32.19981219181705.00f8cb14@triode.net.au>; from John Skaller on Sun, Dec 20, 1998 at 04:17:05AM +1000 References: <1.5.4.32.19981219181705.00f8cb14@triode.net.au> Message-ID: <19981222080747.A1197@cnri.reston.va.us> Quoth John Skaller, on 20 December 1998: > I think you are missing the point here: if the configuration > doesn't work, the client has NO CHOICE but to edit it. > And if it isn't editable, the client has NO CHOICE, full stop. > Now, you don't want to leave the client without options, in > case something as complex as configuring a compiler fails, do you? John -- I don't think not being able to edit sysconfig leaves people out in the cold. Fred's current implementation is, in fact, quite non-editable, as it scans Makefile and friends as late as possible (ie. when sysconfig is itself imported). However, that doesn't preclude the ability to override Makefile/config.h settings at module build or install time. I believe my proposed user interface for the distutils touches on this: we could just have command line options intercepted by the appropriate distutils module to override its default behaviour (where the default is usually dictated by the Python config info harvested by sysconfig). Eg. if you know bloody well what you're doing and are willing to take the responsibility for compiling an extension module differently from how Python was compiled: ./setup.py build -cc gcc -cflags -O3 -cppflags '-DSOME_MACRO' ...or something along those lines. It's possible to consider a way to make those override preferences permanent for a site and/or user, but that's a detail. There's certainly no need to go editing a generated sysconfig (if indeed we end up moving away from Fred's current implementation and towards that model), or the Makefile/config.h/Setup in $(execprefix)/lib/python$(version)/config. And again, quoth John Skaller, on 20 December 1998: > BTW: It is _also_ recommended to have the ability > to generate applications, not just Python modules. The 'make' > system know how to do that too, and it is useful for extending > Python with 'shell' tools. I alluded to that in a post a couple of days ago; my opinion is *still* that we should concentrate on module distribution, and hopefully application distribution will emerge as a natural application/extension of module distribution. Then of course, there's also Mark Hammond's thread on freeze-as- distribution-mechanism. Strikes me as a neat idea, especially for distributing full-blown Python applications. I really must play around with freeze before I can have any sort of informed opinion there, though. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skaller@maxtal.com.au Tue Dec 22 14:10:13 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 00:10:13 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981222141013.00f88808@triode.net.au> At 14:22 21/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > HOW do you know this works? HOW do you know > > that the method is not, in fact, present, but doesn't work? > > WHERE is the standards document that specifies the > > precise semantics of the 'os' interface?? > > Sounds like I'd better get the documentation for "os" fixed before I >release again; it is on my list. I don't think that is really my point. I'm interested in a STANDARD, not just good documentation. Saying what the library does is a useful hint. Like reading the code. It isn't the same as a specification; which _requires_ certain behaviour. The python _language_ document is written more like a specification. If the language doesn't do what the doco says, its a bug in the implementation (or a fault in the doco). Not an excuse from Guido that it doesn't work on that platform 'because'. I don't want an explanation why something doesn't work, I want an authoritative document that I can cite as evidence of a bug. In other words, relieve myself of the responsibility of fixing it or working around it. I've been shown that my interpretation of what Python does is wrong, by citation of the language specification. I want to be shown wrong about the library (when I am), the same way. [On testing things to see if they work] > > This technique doesn't seem very reliable. :-( > > How so? Because it requires testing every single function in the whole system before using anything. Because there is no definitive specification of what might work or not. Do you see? And just _how_ do I test the functions? Using other functions that are themnselves suspect??? > But this can be written on top of feature tests. Why not? Sigh. Because there is no reliable way to test features. FOR EXAMPLE: consider the module 'commands'. How do I know if it works?? I do not see how to test it .. WITHOUT assuming it works. In fact, you can see by examining the code it will NOT work on NT. So interscript has it's own module called 'commands' that works on both posix and NT. I can't 'test' if it works, because it is one of the modules I need to test if tools such as patch and diff work. I have to rely on it working. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 22 14:10:15 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 00:10:15 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981222141015.00f87124@triode.net.au> >[example has_symlinks() elided] > > But this can be written on top of feature tests. Why not? because it cannot be done. > def has_symlinks(): > global has_symlinks > import os > if hasattr(os, "symlink"): > has_symlinks = true > return 1 > else: > has_symlinks = false > return 0 Obviously, this does not work. It tests if there is a symlinks attribute in os, not what should be tested: 1) if there is an attribute symlinks, it works according to spec 2) if there is not, then the os doesn't support symlinks To perform (1) would require actually creating some symlinks, and seeing if they 'worked'. This is what autoconf does; it tries to compile various features, and it tests them as well (where they're _apparently_ available, and where it is possible to do a short test). It would be much easier to _document_ that the symlinks attribute is present if and only if creation of symbolic links is supported with semantics XXXX (fill in specification here). Then I could rely on the above code. To perform (2) is hard, but might involve using shell scripts. (Assuming you have a reliable way of launching them :-) > Have you created a list of the system aspects that this is desirable >for? Is the list *finite*? No. I just keep running into problems. And, obviously, it is desirable to have a finite list: tradeoffs are necessary. >It may well be, but there will be a lot >of disagreement over what it should be. yes. But there is nothing new here :-) >For now, let's call this the >"features" module. Are you ready to propose an API? No. But I will work on it with others. I do have some architecture developed in interscript for this: several frames for different categories of features: * platform * user * site with some work done on building these frames, but I'm not actually using any of it. You might start by reading the relevant sections in the online interscript doco. Then there is something concrete to discuss. It's quite clear that the separate 'compilers' stuff ought be part of this(but isn't, in interscript 1.0a7-9). >I'm sure several >of us would be willing to read it and carefully consider it (I will, >though it may not sound like it;). I think the biggest problem is >defining it in a way that provides more information or more convenient >access to the information represented. Yes, I agree. One way to help that process is to try to _use_ an actual implementation. I have an implementation, but I'm not using it. I could use help structuring it; doing so requires the combined knowledge of many people with experience on many different platforms. I, by myself, do not have wide enough experience. I do have one comment to guide your thoughts. Consider JPython everywhere. Don't assume CPython. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 22 14:10:10 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 00:10:10 +1000 Subject: [Distutils] High level options interface Message-ID: <1.5.4.32.19981222141010.00f88b20@triode.net.au> At 21:59 22/12/98 +1000, John Skaller wrote: >The interface: > ># module 'interscript/compilers.c > >class python_module: > def __init__(self,**kwds): > # initialise the object with kwds as configuration. > > def configure(self,**kwds): > # extend or change configuration > > def compile(self,filename, **kwds): > # compile the file, with given options > # return the string name of the object file > > def link(self,modname, filenames, **kwds): > # link the list of object files, with given options > # to generate the named python module. > # returns the name of the shared library > >class application: > # exactly the same, except produces an executable > # tool that can be invoked from the command line > ># module 'interscript/compilers.cpp ># exactly the same, except for C++ BTW: One weakness of this interfaces is that there is no provision for creating an object library (archive, ar, etc etc). ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 22 14:10:17 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 00:10:17 +1000 Subject: [Distutils] Anonymous CVS up-and-running Message-ID: <1.5.4.32.19981222141017.00f9be98@triode.net.au> > Agreed. I seem to recall someone had sent patches to support C++, >possibly to the C++-SIG. Does anyone know exactly what's needed? Yes, I know what is 'needed', but not how to get it. What is required is that the python executable be built using C, and another version of it built using C++. Let me call them python.exe and pythonpp.exe. For gcc/g++, three extra steps are required in the makefile: 1) compile python.c using C++ as well as C, name it pythonpp.o(bj). This is the _only_ file that has to be compiled with C++. 2) Link that main object file with the C built python library. The result, pythonpp.exe, works just like python.exe, _except_ that it will accept full C++ dynamic link libraries as python modules: that is, it will support RTTI, exception handling, and proper constructor initialisation of virtual tables, correctly. 3) Install pythonpp.exe as well as python.exe I have done this, it is about 5 minutes work for a makefile expert. It took me half a day to get working properly :-) > > 2) Some of the data, such as CFLAGS, is useless because it reflects > > the python source build tree. On my system > > Misc/Makefile.pre.in manages to get things right in practice; some >time needs to go into figuring out just how it does things and get >sysconfig to do the right thing as well. In my opinion, part of the solution is to factor the 'variables' more logically, and then modify the makefile to use that factorisation. To start with, may I suggest _private_ variables which are not really required for sysconfig, be renamed in the makefile to some convention, such as a leading underscore: these can be ignored by the sysconfig parser as implementation details. Anything with a pathname related to the build directory is likely to be private: for example the current 'CFLAGS' contains -I../../something which is clearly useless to the installed system. So I'd rename that variable _CFLAGS. Then I'd create a real CFLAGS which _only_ has the option flags, no include directories. This allows the flags to be combined with the include directory options obtained for the _installed_ system. (That information is available now in sysconfig -- and indeed directly from Python) > > 3) The directory for config.h is WRONG! > > config.h lives in > > What definition are you looking at? Huh? The code in the CVS archive is wrong (for my system). It creates a directory name using '/usr/local/lib/python1.5/' which is NOT where config.h is installed. It is installed in /usr/local/include/python1.5 I.e. 'lib' should read 'include'. As a result, the module fails on my system. It is a stock standard Linux installation, IMHO it has put config.h in the right place, therefore sysconfig.py is wrong. :-) (Sorry, it is hard for me to quote stuff because I email on NT, but do real work on Linux :-) > > 4) On my system: > > > > SIZEOF_INT = '4' > > Patch attached which gets this right in a general fashion. Thanks! > > 5) I think the module should NOT just put values > > into itself like > > I don't see any need to complicate the interface. Neither do I. Factorisation doesn't complicate the interface, it simplifies it. There are two categories of information, and they come from separate sources. The data strucure should surely reflect this? > > 6) There is no information from Setup. It would be useful to > > get the Tcl/Tk library and headers into the configuration > > if it was built. Hmmm. :( > > Variables defined in Setup & friends are included. Otherwise, >nothing from Setup is included. This can be changed. It may make >sense to use a separate module for this (modconfig?). I think each source of data should be separate, including the makefile info and the autoconf info. The existing architecture is correct, IMHO: its the simplest possible 'pythonic' representation of the raw data. [This is not enough for the high level goal, but it seems correct to build this lower level first] > > 7) _init_posix should NOT modify a dictionary, but return one: > > CHANGE: > > This item is tightly related to item 5; let's deal with that one as >the real issue. The rest will fall out of that result. OK. > > 8) LINKCC = ' gcc' has a bad leading space. > > Fixed in the patch. Thanks. > > 9) I have no idea how to use the available information. > > There are flags all over the place. > > Yes, that's what configure produces. The next layer needs to put >this information together in a usable form. Reading the Makefile >should help a lot in determining how the various bits should be put >together. Yes, I agree. It is clear that while the current set of attributes is not quite right, it contains a lot of information. Therefore, the first step of just getting it into a pythonic format is a good one. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 22 15:40:15 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 01:40:15 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981222154015.00f98628@triode.net.au> At 08:07 22/12/98 -0500, Greg Ward wrote: >Quoth John Skaller, on 20 December 1998: >> I think you are missing the point here: if the configuration >> doesn't work, the client has NO CHOICE but to edit it. >> And if it isn't editable, the client has NO CHOICE, full stop. >> Now, you don't want to leave the client without options, in >> case something as complex as configuring a compiler fails, do you? > >John -- I don't think not being able to edit sysconfig leaves people out >in the cold. Fred's current implementation is, in fact, quite >non-editable, as it scans Makefile and friends as late as possible >(ie. when sysconfig is itself imported). Let me cut the arguments, and simply phrase a request. This gives us BOTH what we want. :-) EXECUTIVE SUMMARY 1. module parse_configuration, function get_config returns a dictionary of config data parsed from the makefile 2. module sysconfig exhibits attributes corresponding to the result of executing get_config when loaded 3. module build_config, by default, exhibits the attributes corresponding to the result of executing get_config AT BUILD TIME This module may be edited by the client. 4. The tool 'refresh_config.py' can be executed to print the (dynamic) configuration to standard output, and thus be used to generate a configuration module, including the standard one (3). DETAILS. Allow me to assume sysconfig.py, as it stands, is copied to parse_configuration, renamed get_config, and edited slightly to conform to the specified interface. Here is the new implementation of module sysconfig, with the same semantics as the existing one: # distutils.sysconfig from distutils.parse_configuration import get_config globals().update(get_config()) Nifty, eh? :-) Here is 'refresh_config.py': # refresh_config.py from distutils.parse_configuration import get_config for key, value in get_config().items(): print key+'='+repr(value) As you can see, both these auxilliary things are a single line of code. This is packaging. It is possible just by isolating the parser, and not doing anything tricky. :-) I think that invoking the refresh utility in the make process is also a single line change to the makefile! WAFFLY IMHO THEORETICAL COMMENTS: It is best to have stateless entities do the work, and then provide state as required by the architecture. Separating out the functionality is crucial to building multiple architectures: it allows the architectural implementation to be largely independent of the implementation of the functionality. This is the secret of Python methods: they're just plain old stateless functions. Bound methods are just architecture, independent of the functionality encoded in the unbound method. We should emulate this good, orthogonal, design! /WAFFLE Comments? ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Tue Dec 22 15:57:33 1998 From: skaller@maxtal.com.au (John Skaller) Date: Wed, 23 Dec 1998 01:57:33 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981222155733.00f9604c@triode.net.au> At 01:40 23/12/98 +1000, John Skaller wrote: Duh, sorry for replying to my own post again but: >EXECUTIVE SUMMARY > > 1. module parse_configuration, function get_config > returns a dictionary of config data parsed from > the makefile This depends on knowing where the makefile and the config file are. Which suggest a further factorisation: def parse_config_h(filename) def parse_makefile(filename) which would be generally useful! And truly statless. Now, provide: def get_config_h_filename() def get_makefile_filename() and we can wrap them up to get: def get_config(): dict = parse_config_h(get_config_h_filename()) dict.update(parse_config_h(get_makefile_filename())) return dict Again, this factorisation simplifies the design, and also creates added functionality. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From Fred L. Drake, Jr." References: <1.5.4.32.19981222141010.00f88b20@triode.net.au> Message-ID: <13951.51235.981654.909725@weyr.cnri.reston.va.us> John Skaller writes: > BTW: One weakness of this interfaces is that there > is no provision for creating an object library > (archive, ar, etc etc). Is this needed? I don't see where, unless we really want to abstract everything in the build process. Essentially, we end building our own make utility, which I think we should attempt to avoid -- enough people have done that already. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981222155733.00f9604c@triode.net.au> Message-ID: <13951.53201.309992.249208@weyr.cnri.reston.va.us> John Skaller writes: > This depends on knowing where the makefile and > the config file are. Which suggest a further factorisation: Ok, I've modified distutils.sysconfig to contain: get_config_h_filename() get_makefile_filename() parse_config_h(fileobject, g=None) parse_makefile(fileobject, g=None) Note that the parse_*() routines take a file object instead of a filename, and the dictionary to update may be passed in. If no dictionary is provided a new one is used. However the dict is obtained, it is returned by the function. Equivalents for non-Unix platforms are still needed; has anyone thought about Windows or Mac platforms? I think the current implementation is fine for BeOS; can someone who uses the BeOS port confirm? NeXT users might want to pipe up as well! > def get_config(): > dict = parse_config_h(get_config_h_filename()) > dict.update(parse_config_h(get_makefile_filename())) > return dict I passed on this for now. I think the cross-platform issues need to be explored more. > Again, this factorisation simplifies the design, and also > creates added functionality. This I agree with. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981222155733.00f9604c@triode.net.au> Message-ID: <13951.53460.185686.575946@weyr.cnri.reston.va.us> Oops, I forgot the patch. It should be below. If I don't forget again. ;-) Greg, please integrate with the CVS repo. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 Index: sysconfig.py =================================================================== RCS file: /projects/cvsroot/distutils/distutils/sysconfig.py,v retrieving revision 1.1 diff -c -c -r1.1 sysconfig.py *** sysconfig.py 1998/12/18 23:46:33 1.1 --- sysconfig.py 1998/12/22 17:00:01 *************** *** 11,33 **** __version__ = "$Revision: 1.1 $" ! ! def _init_posix(): ! import os ! import re ! import sys ! ! g = globals() ! ! version = sys.version[:3] ! config_dir = os.path.join( ! sys.exec_prefix, "lib", "python" + version, "config") ! ! # load the installed config.h: define_rx = re.compile("#define ([A-Z][A-Z0-9_]+) (.*)\n") undef_rx = re.compile("/[*] #undef ([A-Z][A-Z0-9_]+) [*]/\n") ! fp = open(os.path.join(config_dir, "config.h")) ! while 1: line = fp.readline() if not line: --- 11,36 ---- __version__ = "$Revision: 1.1 $" ! import os ! import re ! import string ! import sys ! ! ! def get_config_h_filename(): ! return os.path.join(sys.exec_prefix, "lib", "python" + sys.version[:3], ! "config", "config.h") ! ! def get_makefile_filename(): ! return os.path.join(sys.exec_prefix, "lib", "python" + sys.version[:3], ! "config", "Makefile") ! ! def parse_config_h(fp, g=None): ! if g is None: ! g = {} define_rx = re.compile("#define ([A-Z][A-Z0-9_]+) (.*)\n") undef_rx = re.compile("/[*] #undef ([A-Z][A-Z0-9_]+) [*]/\n") ! # while 1: line = fp.readline() if not line: *************** *** 35,54 **** m = define_rx.match(line) if m: n, v = m.group(1, 2) ! if v == "1": ! g[n] = 1 ! else: ! g[n] = v else: m = undef_rx.match(line) if m: g[m.group(1)] = 0 ! # load the installed Makefile.pre.in: variable_rx = re.compile("([a-zA-Z][a-zA-Z0-9_]+)\s*=\s*(.*)\n") done = {} notdone = {} - fp = open(os.path.join(config_dir, "Makefile")) while 1: line = fp.readline() --- 38,58 ---- m = define_rx.match(line) if m: n, v = m.group(1, 2) ! try: v = string.atoi(v) ! except ValueError: pass ! g[n] = v else: m = undef_rx.match(line) if m: g[m.group(1)] = 0 + return g ! def parse_makefile(fp, g=None): ! if g is None: ! g = {} variable_rx = re.compile("([a-zA-Z][a-zA-Z0-9_]+)\s*=\s*(.*)\n") done = {} notdone = {} while 1: line = fp.readline() *************** *** 57,65 **** --- 61,72 ---- m = variable_rx.match(line) if m: n, v = m.group(1, 2) + v = string.strip(v) if "$" in v: notdone[n] = v else: + try: v = string.atoi(v) + except ValueError: pass done[n] = v # do variable interpolation here *************** *** 79,85 **** if "$" in after: notdone[name] = value else: ! done[name] = value del notdone[name] elif notdone.has_key(n): # get it on a subsequent round --- 86,94 ---- if "$" in after: notdone[name] = value else: ! try: value = string.atoi(value) ! except ValueError: pass ! done[name] = string.strip(value) del notdone[name] elif notdone.has_key(n): # get it on a subsequent round *************** *** 91,106 **** if "$" in after: notdone[name] = value else: ! done[name] = value del notdone[name] else: del notdone[name] # save the results in the global dictionary g.update(done) - import os exec "_init_%s()" % os.name - del os del _init_posix --- 100,124 ---- if "$" in after: notdone[name] = value else: ! try: value = string.atoi(value) ! except ValueError: pass ! done[name] = string.strip(value) del notdone[name] else: + # bogus variable reference; just drop it since we can't deal del notdone[name] # save the results in the global dictionary g.update(done) + return g + + def _init_posix(): + g = globals() + # load the installed config.h: + parse_config_h(open(get_config_h_filename()), g) + # load the installed Makefile.pre.in: + parse_makefile(open(get_makefile_filename()), g) exec "_init_%s()" % os.name del _init_posix From M.Faassen@vet.uu.nl Tue Dec 22 18:00:00 1998 From: M.Faassen@vet.uu.nl (Martijn Faassen) Date: Tue, 22 Dec 1998 19:00:00 +0100 Subject: [Distutils] High level options interface References: <1.5.4.32.19981222115930.00f7b9b0@triode.net.au> Message-ID: <367FDE20.543AF50C@pop.vet.uu.nl> John Skaller wrote: > > At 18:57 21/12/98 +0100, Martijn Faassen wrote: > >class extension_compiler: > > def compile_C(filename): > > pass > > def compile_C++(filename): > > pass > > There's no need for a base class, just a specification. Not much of a difference here. An abstract base class has the advantage that you can do: def compile_ObscureLanguage(filename): print "Sorry, this language is not supported on this platform." or something like that. But I realize below you modelled the entire thing differently anyhow. > Please examine the following interface and see > a gcc implementation below that. At least criticize > the interface if you don't like it. [actual code snipped] I've looked through the interface, thought about it (later on in my post) and I have some questions (also later in my post). I notice you took the module/application to be the central object that we're doing things to (compiling, linking, etc). My previous interface was to the compiler. I'm pondering the consequences of this difference right now. Hmm. I'm just trying to write up some code that uses the two abstractions. Basic scenario: We have code written. Some of it is in Python, some in C, and some in C++. It all needs to be compiled and linked together, on various platforms. The compiler centric abstraction would be used somewhat like this: # platform dependent initialisation code import platforms # perhaps 'platform' is instead a module like os, already preconfigured # for this platform (what about cross-compilation, though?) platform = platforms.NormalLinux2Platform() ... # platform independent code c_compiler = platform.get_c_compiler() cpp_compiler = platform.get_cpp_compiler() ... for filename in c_filenames: c_compiler.compile(filename) for filename in cpp_filenames: cpp_compiler.compile(filename) # note that the compiler objects know that we compile to '.o' files # on this platform (not for instance to '.obj') c_compiler.link(c_filenames) cpp_compiler.link(cpp_filenames) # ? what if we need to link c and cpp objects together? ... # something needs to be done to get an actual module name The module/application centric abstraction: # platform dependent initialisation import compilers.c # already configured for gcc on this platform import compilers.cpp # already configured for g++ on this platform some_config = "--overdrive" # extra flags (in this case nonsense) ... # platform independent code module = compilers.c.python_module(some_config) # ignoring extra compilation flags for now objectnames = [] for filename in filenames: objectnames.append(module.compile(filename)) # alternatively something like: # objectnames = map(module.compile, filenames) # link it together producing module filename module_filename = module.link(module_name, objectnames) Hmm. Am I using this right? * What if you have to link C, C++, other languages together? (this is an issue with both approaches) * Why is the module name only provided at linking in the module centric interface? Why not initialize the python_module object with the module name? * There's some passing of filenames around, more in the module centric interface. Is this necessary? Are we losing abstraction here or is it okay? Passing around base names seems to be enough (or just the .c names, and stripping away .c each time you need to construct the .o name, or whatever you need). * In the module centric abstraction, the 'compile' method is rather disconnected from the rest of the module. I can see that using the 'link' functionality we produce a nice shared library in the end, which is what we want. But 'compile' doesn't 'feel' like it belongs to module. * I'm not sure if the extra passing of compiler options around (especially to 'compile' and to 'link' makes the thing more flexible, or alternatively more difficult to maintain platform independently. Perhaps a combination of the two approaches would be best. Modules should ideally know how to compile themselves, using simple compiler abstractions. As long as we can keep this simple enough. So far my musings. What do you all think? Regards, Martijn From gstein@lyra.org Tue Dec 22 20:31:06 1998 From: gstein@lyra.org (Greg Stein) Date: Tue, 22 Dec 1998 12:31:06 -0800 Subject: [Distutils] Access to Python config info References: <1.5.4.32.19981222141013.00f88808@triode.net.au> Message-ID: <3680018A.4672D2CB@lyra.org> John Skaller wrote: > ... > I don't think that is really my point. > I'm interested in a STANDARD, not just good documentation. > > Saying what the library does is a useful hint. > Like reading the code. It isn't the same as a specification; > which _requires_ certain behaviour. The python _language_ > document is written more like a specification. > If the language doesn't do what the doco says, its a bug > in the implementation (or a fault in the doco). > > Not an excuse from Guido that it doesn't work > on that platform 'because'. I don't want an explanation > why something doesn't work, I want an authoritative > document that I can cite as evidence of a bug. > In other words, relieve myself of the responsibility of > fixing it or working around it. > > I've been shown that my interpretation of > what Python does is wrong, by citation of the > language specification. I want to be shown wrong > about the library (when I am), the same way. That specification does not exist, nor does anybody plan to write it. "The code is the spec" applies 100% here. What ships in the standard distribution defines the module. Simple. Done. If you really want it, then I would suggest that you take some time to write it. > [On testing things to see if they work] > > > This technique doesn't seem very reliable. :-( > > > > How so? > > Because it requires testing every single > function in the whole system before using > anything. Because there is no definitive specification > of what might work or not. Do you see? > > And just _how_ do I test the functions? > Using other functions that are themnselves suspect??? If a function is in the standard distribution, then it should be assumed that it works. That is what Fred is getting at. A "hasattr" says whether it exists or not. If it does, then it should work. And yes, there may be certain caveats to its behavior. > [more about testing and features and commands.py...] Where is all this going? I have totally lost the point of this discussion. Let's just get back to discussing distutils, can we? Shortcomings of certain modules, how to test the standard distribution, etc, should probably be raised on comp.lang.python. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Tue Dec 22 20:42:46 1998 From: gstein@lyra.org (Greg Stein) Date: Tue, 22 Dec 1998 12:42:46 -0800 Subject: [Distutils] Access to Python config info References: <1.5.4.32.19981222141015.00f87124@triode.net.au> Message-ID: <36800445.6020688B@lyra.org> John Skaller wrote: > ... > > def has_symlinks(): > > global has_symlinks > > import os > > if hasattr(os, "symlink"): > > has_symlinks = true > > return 1 > > else: > > has_symlinks = false > > return 0 > > Obviously, this does not work. It tests if there is > a symlinks attribute in os, not what should be tested: > > 1) if there is an attribute symlinks, it works according to spec > 2) if there is not, then the os doesn't support symlinks > > To perform (1) would require actually creating some symlinks, > and seeing if they 'worked'. This is what autoconf does; No. If it is there, then it should work. If it doesn't, then ask Guido what he'd like to do. His answer will be one of: 1) it should be removed for that platform 2) it will stay; test sys.platform to determine whether it will work. There is no reason for anything to code defensively against the standard distribution library. There is no need to second-guess all of its modules and functions. [trimming the rest of this email note] Can you send around a URL to the interscript online doc? And when you refer to it, could you please cite a more specific reference? Call it a character flaw, but I'm not going to just blindly go read a bunch of doc. A concrete reference helps (me, at least) provide some focus/structure to why I'm there reading the thing in the first place. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Tue Dec 22 20:58:26 1998 From: gstein@lyra.org (Greg Stein) Date: Tue, 22 Dec 1998 12:58:26 -0800 Subject: [Distutils] Anonymous CVS up-and-running References: <1.5.4.32.19981222141017.00f9be98@triode.net.au> Message-ID: <368007F2.3DFE9845@lyra.org> John Skaller wrote: > > > Agreed. I seem to recall someone had sent patches to support C++, > >possibly to the C++-SIG. Does anyone know exactly what's needed? > ... > I have done this, it is about 5 minutes work for a makefile > expert. It took me half a day to get working properly :-) Until these patches are applied to the main distribution, then I think we can safely ignore the issue. > > > 2) Some of the data, such as CFLAGS, is useless because it reflects > > > the python source build tree. On my system > > > > Misc/Makefile.pre.in manages to get things right in practice; some > >time needs to go into figuring out just how it does things and get > >sysconfig to do the right thing as well. > > In my opinion, part of the solution is to factor the > 'variables' more logically, and then modify the makefile to use > that factorisation. > > To start with, may I suggest _private_ variables which > are not really required for sysconfig, be renamed in the makefile > to some convention, such as a leading underscore: these can be > ignored by the sysconfig parser as implementation details. > > Anything with a pathname related to the build directory > is likely to be private: for example the current 'CFLAGS' > contains -I../../something which is clearly useless to the > installed system. So I'd rename that variable _CFLAGS. > Then I'd create a real CFLAGS which _only_ has the option > flags, no include directories. This allows the flags to be > combined with the include directory options obtained > for the _installed_ system. (That information is available > now in sysconfig -- and indeed directly from Python) I have a better idea: rather than all this work on the distribution itself, let's do a Version 1 that places *everything* into the sysconfig as it does now. USERS of the sysconfig can simply ignore most of it. That works, and that gets us to a solution faster. We need code to work with, more than we need the right code. > > > 3) The directory for config.h is WRONG! > > > config.h lives in > > > > What definition are you looking at? > > Huh? The code in the CVS archive is wrong (for my system). > It creates a directory name using '/usr/local/lib/python1.5/' > which is NOT where config.h is installed. It is installed in > > /usr/local/include/python1.5 > > I.e. 'lib' should read 'include'. > As a result, the module fails on my system. > It is a stock standard Linux installation, IMHO it has > put config.h in the right place, therefore sysconfig.py is wrong. :-) Quite true. On my RedHat 5.1 system, I have config.h in /usr/include/python1.5. Note that /usr/lib/python1.5/config *does* exist, but it just doesn't have the headers. Also, with a manual install of Python, it places it into /usr/local/include/python1.5. > > > 5) I think the module should NOT just put values > > > into itself like > > > > I don't see any need to complicate the interface. > > Neither do I. Factorisation doesn't complicate > the interface, it simplifies it. There are two categories > of information, and they come from separate sources. > The data strucure should surely reflect this? Not really. Clients should just use what they want. We should get the information into Python and let the higher levels deal with it. sysconfig should be simple and flat. The less we structure it, the smaller chance of structuring it wrong (in terms of future needs). -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Tue Dec 22 21:08:27 1998 From: gstein@lyra.org (Greg Stein) Date: Tue, 22 Dec 1998 13:08:27 -0800 Subject: [Distutils] Access to Python config info References: <1.5.4.32.19981222155733.00f9604c@triode.net.au> <13951.53201.309992.249208@weyr.cnri.reston.va.us> Message-ID: <36800A4B.53028AE0@lyra.org> Fred L. Drake wrote: > ... > Equivalents for non-Unix platforms are still needed; has anyone > thought about Windows or Mac platforms? I think the current > implementation is fine for BeOS; can someone who uses the BeOS port > confirm? NeXT users might want to pipe up as well! The Windows platform has a hardcoded "makefile" and a hardcoded config.h. If the sysconfig interface is simply a flat of list of names, then the Windows platform can ship a simple sysconfig module defining those names as appropriate (rather than trying to query them). This is partly why I advocate making sysocnfig as simple as possible. Preferably, it would be generate by the build process and shipped that way. NO functions. If you want functions/classes/whatever, then they go in the distutils package, consuming information from sysconfig.py as needed. Cheers, -g -- Greg Stein, http://www.lyra.org/ From Fred L. Drake, Jr." References: <1.5.4.32.19981222141015.00f87124@triode.net.au> <36800445.6020688B@lyra.org> Message-ID: <13952.6455.448768.506564@weyr.cnri.reston.va.us> Greg Stein writes: > No. If it is there, then it should work. If it doesn't, then ask Guido > what he'd like to do. His answer will be one of: > > 1) it should be removed for that platform This one. > 2) it will stay; test sys.platform to determine whether it will work. Never this. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981222141013.00f88808@triode.net.au> Message-ID: <13952.7359.395282.696952@weyr.cnri.reston.va.us> John Skaller writes: > I don't think that is really my point. > I'm interested in a STANDARD, not just good documentation. Until Guido says otherwise, the standard documentation *is* the standard. If it isn't clear, well, I'm always open to specific suggestions and patches . I'm not being defensive here, I don't think, but the documentation has always been the standard for Python. Where there are holes, the implementation is the standard. > language specification. I want to be shown wrong > about the library (when I am), the same way. Was the above paragraph sufficient? ;-) > Because it requires testing every single > function in the whole system before using > anything. Because there is no definitive specification > of what might work or not. Do you see? No. The presence of the function is sufficient. If the underlying system calls don't work as described (where that's an issue), there is either a bug in the documentation or a bug in the syscall implementation. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981222141015.00f87124@triode.net.au> Message-ID: <13952.7859.964895.846491@weyr.cnri.reston.va.us> John Skaller writes: > Obviously, this does not work. It tests if there is > a symlinks attribute in os, not what should be tested: > > 1) if there is an attribute symlinks, it works according to spec > 2) if there is not, then the os doesn't support symlinks > > To perform (1) would require actually creating some symlinks, > and seeing if they 'worked'. This is what autoconf does; > it tries to compile various features, and it tests them > as well (where they're _apparently_ available, and where it > is possible to do a short test). > > It would be much easier to _document_ that > the symlinks attribute is present if and only if > creation of symbolic links is supported with > semantics XXXX (fill in specification here). Oh, you want this to be an *operating system* standard. I don't think PyOS has been released yet. Seriously, a language should expose interfaces to system services, not make guarantees as to the conformance of the O/S to some standard. I think the Open Group is in charge of X/Open conformance these days. If you want a higher level interface, you need to build it. Once you've demonstrated generality, you can describe it as a standard, document it, and try and get others to adopt it. *That's* standardization. There are good reasons the IETF requires at least two independent implementations of a specification to make it an internet standard: that shows that the interface is sufficiently general and useful that it was worth two organizations supporting the development work. Until, it's just a specification. Which is the most we need here. > with some work done on building these frames, > but I'm not actually using any of it. Sounds like it's not really needed. > You might start by reading the relevant sections > in the online interscript doco. Then there > is something concrete to discuss. As Greg Stein asked: please provide *specific* pointers: which sections are relevant? > Consider JPython everywhere. Don't assume CPython. Good point, especially for those of us that don't normally use it. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981222141017.00f9be98@triode.net.au> Message-ID: <13952.8367.554649.741643@weyr.cnri.reston.va.us> John Skaller writes: > I have done this, it is about 5 minutes work for a makefile > expert. It took me half a day to get working properly :-) Cool. Send Guido a patch and rationale as to why it should be in the standard distribution. Frankly, the reason to install pythonpp is lost on me. > To start with, may I suggest _private_ variables which > are not really required for sysconfig, be renamed in the makefile > to some convention, such as a leading underscore: these can be > ignored by the sysconfig parser as implementation details. No. What's in the Makefile is there for the Makefile's use. If we don't need part of it, it can be ignored. Many make variables have a long history, and should not be renamed. CFLAGS definately qualifies! > Huh? The code in the CVS archive is wrong (for my system). > It creates a directory name using '/usr/local/lib/python1.5/' > which is NOT where config.h is installed. It is installed in What's the variable/constant that has this wrong? I don't see "config.h" in the values. > the interface, it simplifies it. There are two categories > of information, and they come from separate sources. > The data strucure should surely reflect this? The specific sources are, in this case, not indicative of useful categories. > I think each source of data should be separate, > including the makefile info and the autoconf info. The Makefile includes information generated by the configure script; that's a significant part of what configure does. *Parsing* the different source types can be intelligently separated; the usefulness of the separation is still open to debate. It doesn't hurt, and is done in the patch I sent earlier. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From gward@cnri.reston.va.us Wed Dec 23 13:31:38 1998 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 23 Dec 1998 08:31:38 -0500 Subject: [Distutils] Enough about feature tests! Message-ID: <19981223083138.A1936@cnri.reston.va.us> Hmmm, this seems to be a recurring theme; I guess this is just what a moderator is supposed to do, though. Anyways, it should be obvious but apparently is not: arguing about how best to test for OS features through Python is not really on topic for this SIG. We should be arguing, err discussing, high-level design of the distutils suite; or, failing that, nit-picking implementation details. For example, the sysconfig module, what to call it, when it runs, how it provides its information, etc. are all on-topic. Please stick to relevant things and learn to live with the fact that there's no formal specification for the Python Library, nor is there likely to be one anytime soon, nor will we solve that perceived problem with the distutils. Thanks -- Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 x287 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From skaller@maxtal.com.au Thu Dec 24 15:54:57 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 25 Dec 1998 01:54:57 +1000 Subject: [Distutils] High level options interface Message-ID: <1.5.4.32.19981224155457.00f7e99c@triode.net.au> At 11:26 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > BTW: One weakness of this interfaces is that there > > is no provision for creating an object library > > (archive, ar, etc etc). > > Is this needed? I don't know. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Thu Dec 24 15:55:04 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 25 Dec 1998 01:55:04 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981224155504.00f74c90@triode.net.au> At 11:58 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > This depends on knowing where the makefile and > > the config file are. Which suggest a further factorisation: > > Ok, I've modified distutils.sysconfig to contain: > > get_config_h_filename() > get_makefile_filename() > parse_config_h(fileobject, g=None) > parse_makefile(fileobject, g=None) > > Note that the parse_*() routines take a file object instead of a >filename, Ah, that seems even better, yes! >and the dictionary to update may be passed in. If no >dictionary is provided a new one is used. However the dict is >obtained, it is returned by the function. Right. Seems OK. > Equivalents for non-Unix platforms are still needed; has anyone >thought about Windows or Mac platforms? I don't think we (necessarily) need equivalents for the following reason: the sysconfig data isn't meant for user access: its there to support implementation of functions like 'compile_this_c_as_a_module'. It is these functions which need to be standardised (that is, cross platform). For the VC++ windows build, sysconfig could be omitted entirely, and a hand written 'compile' function provided. For Unix, it is convenient to parameterise such a function with the config data -- for that is the way the autoconf/ make/blah blah process works. none of this implies there shouldn't be an equivalent; but I think the important interfaces is the 'OO' one: the one characterized by function semantics rather than values of data attributes. > I passed on this for now. I think the cross-platform issues need to >be explored more. I agree. I would like people to explore the interface of my compilers module to see if it is _possible_ for that interface to work on all platforms. It may be that the Unix implementation uses sysconfig data to adapt to various Unix platforms, whereas the Mac or Windows version is hard coded, or uses a different scheme. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Thu Dec 24 15:55:58 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 25 Dec 1998 01:55:58 +1000 Subject: [Distutils] Access to Python config info Message-ID: <1.5.4.32.19981224155558.00f9e820@triode.net.au> >Can you send around a URL to the interscript online doc? Yes, but I should say that it has been cited many times, it exists on the www.python.org home page in the 'news' list, and it is in comp.lang.python.announce. http://www.triode.net.au/~skaller/interscript is the home page. >And when you >refer to it, could you please cite a more specific reference? Call it a >character flaw, but I'm not going to just blindly go read a bunch of >doc. A concrete reference helps (me, at least) provide some >focus/structure to why I'm there reading the thing in the first place. I agree, but I want to apologise profusely that I am not able to do this easily at the moment. firstly, my working copy is not the same as the one on the Internet, and secondly the HTML pages are all numbered at the moment, and the numbers are arbitrary and will change completely with every build i do that adds or removes even a single heading. [There are over 500 pages] I am addressing this problem, but at the moment the best I can do is usually to say 'read the section about XXX'. The table of contents is not small, but it should guide finding relevant information quickly. ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From skaller@maxtal.com.au Thu Dec 24 15:56:11 1998 From: skaller@maxtal.com.au (John Skaller) Date: Fri, 25 Dec 1998 01:56:11 +1000 Subject: [Distutils] Anonymous CVS up-and-running Message-ID: <1.5.4.32.19981224155611.00f8f5dc@triode.net.au> At 17:43 22/12/98 -0500, Fred L. Drake wrote: > >John Skaller writes: > > I have done this, it is about 5 minutes work for a makefile > > expert. It took me half a day to get working properly :-) > > Cool. Send Guido a patch and rationale as to why it should be in >the standard distribution. I can't. It was a patch to version 1.5.0 which i don't have any more. It got clobbered when I installed 1.5.1 of course. >Frankly, the reason to install pythonpp is >lost on me. The you don't need to invoke it. But other people in the C++ SIG want to use C++ to extend Python. This is not the place to argue about it, they want it. :-) ------------------------------------------------------- John Skaller email: skaller@maxtal.com.au http://www.maxtal.com.au/~skaller phone: 61-2-96600850 snail: 10/1 Toxteth Rd, Glebe NSW 2037, Australia From Fred L. Drake, Jr." References: <1.5.4.32.19981224155504.00f74c90@triode.net.au> Message-ID: <13959.45169.106633.164189@weyr.cnri.reston.va.us> John Skaller writes: > I don't think we (necessarily) need equivalents > for the following reason: > the sysconfig data isn't meant for user access: its there to support > implementation of functions like 'compile_this_c_as_a_module'. I was thinking that it needed to be there for use by the "compile..." stuff, but if it's not needed, then importing sysconfig shouldn't break things. (With the last patch, it would raise a NameError when os.name wasn't "posix".) People that know the other platforms better will have to determine if sysconfig-style data is useful. If it is, please provide an _init_() function that does the right thing. I do expect the specific names that are defined to be specific to the platform; there's no need to provide the same names. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191 From Fred L. Drake, Jr." References: <1.5.4.32.19981224155611.00f8f5dc@triode.net.au> Message-ID: <13959.46423.370977.848319@weyr.cnri.reston.va.us> I said: >Frankly, the reason to install pythonpp is >lost on me. John Skaller writes: > The you don't need to invoke it. > But other people in the C++ SIG want to use C++ to extend > Python. This is not the place to argue about it, they want it. :-) *This* I understand. I understand why main() should be built with C++ to extend Python with C++ reliably. Is that the only reason for pythonpp? In that case, just build the standard interpreter with a C++-compiled main() and be done with it. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives 1895 Preston White Dr. Reston, VA 20191