Some things I came across while adding distutils support to some code: * From a docstring in distutils/versions.py: 0.4 0.4.0 (these two are equivalent) 0.4.1 Does this mean that 0.4.1 is equivalent to 0.41 in the strict versioning class? I'd suggest adding that, just to make it crystal clear. * The name option in setup.py: It's not clear if it can/should contain spaces; given that you mention underscores, I assume they're as space substitutes. * Building a distribution: it creates hard links to files. This means that if you delete a file and rebuild the dist without erasing the hard-link-filled subdir, the deleted file is still present in the distribution. It's probably easiest to blow away the whole <product>-<version> directory, rather than attempting to scan its contents and update it. -- A.M. Kuchling http://starship.python.net/crew/amk/ "I don't think we should interfere." "Interfere? Of course we should interfere! Always do what you're best at, that's what I say." -- Romana and the Doctor, in "Nightmare of Eden"
Some other thoughts... On Sat, 27 Nov 1999, A.M. Kuchling wrote: [schnipp]
* Building a distribution: it creates hard links to files. This means that if you delete a file and rebuild the dist without erasing the hard-link-filled subdir, the deleted file is still present in the distribution. It's probably easiest to blow away the whole <product>-<version> directory, rather than attempting to scan its contents and update it.
Yes! I spent a frustrating few minutes trying to work out why files I was trying to exclude from the distribution were turning up in the tarball... Also, excluding files from the dist in general is a pain. I have bytecodehacks in CVS; I don't want to distribute the CVS folders. I found no way to do this except building the MANIFEST file using a shell script (which kind of defeats the point of the manifest, I'd have thought). Also excluding a single file doesn't work. If I want to distribute all files in a directory `bar' except one called, say, `foo', I'd expect to be able to write in the MANIFEST bar !foo but this doesn't work. It doesn't complain, foo is just included in the archive. I'd be happy to beat on these if someone could give me advice where to start. Would a .cvsignore style approach be suitable? Other than these wrinkles distutils is a pleasure to use. Good work! Cheers, Michael
On Sat, 27 Nov 1999, A.M. Kuchling wrote: [schnipp]
* Building a distribution: it creates hard links to files. This means that if you delete a file and rebuild the dist without erasing the hard-link-filled subdir, the deleted file is still present in the distribution. It's probably easiest to blow away the whole <product>-<version> directory, rather than attempting to scan its contents and update it.
Yes! I spent a frustrating few minutes trying to work out why files I was trying to exclude from the distribution were turning up in the tarball...
Also, excluding files from the dist in general is a pain.
I have bytecodehacks in CVS; I don't want to distribute the CVS folders. I found no way to do this except building the MANIFEST file using a shell script (which kind of defeats the point of the manifest, I'd have thought).
Try using cvs export (requires a tag, but you should be tagging your code for release anyway). Only the files with the tag will be exported; it is far easier to "remove" files from the release. You can still use the manifest to restrict the files as needed. Also, from a release management point-of-view, you shouldn't be releasing code from your work area. -Arcege -- ------------------------------------------------------------------------ | Michael P. Reilly, Release Engineer | Email: arcege@shore.net | | Salem, Mass. USA 01970 | | ------------------------------------------------------------------------
Michael P. Reilly writes:
Try using cvs export (requires a tag, but you should be tagging your code for release anyway). Only the files with the tag will be exported; it is far easier to "remove" files from the release. You can still use the manifest to restrict the files as needed.
Also, from a release management point-of-view, you shouldn't be releasing code from your work area.
Yes, yes, yes! Truer words have never been spoken! -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
On Mon, 29 Nov 1999, Fred L. Drake, Jr. wrote:
Michael P. Reilly writes:
Try using cvs export (requires a tag, but you should be tagging your code for release anyway). Only the files with the tag will be exported; it is far easier to "remove" files from the release. You can still use the manifest to restrict the files as needed.
Also, from a release management point-of-view, you shouldn't be releasing code from your work area.
Yes, yes, yes! Truer words have never been spoken!
Does anyone feel like doing a CVS tutorial BOF at IPC8? I think a lot of us could use some of this wisdom... --david
David Ascher writes:
Does anyone feel like doing a CVS tutorial BOF at IPC8? I think a lot of us could use some of this wisdom...
Er, Barry Warsaw is the CVS guru around here. Barry? ;-) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
Does anyone feel like doing a CVS tutorial BOF at IPC8? I think a lot of us could use some of this wisdom...
--david
A number of us were talking with publishers in Monterrey about them putting out a CVS book. I've had a number of ppl asking me if there was anything more than the texinfo files. Beyond that, a CVS BOF would probably be useful. I'm still trying to make it to IPC8. -Arcege PS: This discussion _might_ be a little off-topic. ;) -- ------------------------------------------------------------------------ | Michael P. Reilly, Release Engineer | Email: arcege@shore.net | | Salem, Mass. USA 01970 | | ------------------------------------------------------------------------
Michael P. Reilly writes:
A number of us were talking with publishers in Monterrey about them putting out a CVS book. I've had a number of ppl asking me if there was anything more than the texinfo files.
Actually, there is a CVS book now. Search for "CVS" at amazon.com in the computer section, or at your favorite bookseller. -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
On 28 November 1999, Michael Hudson said:
Yes! I spent a frustrating few minutes trying to work out why files I was trying to exclude from the distribution were turning up in the tarball...
Oops, sorry.
Also, excluding files from the dist in general is a pain.
I have bytecodehacks in CVS; I don't want to distribute the CVS folders. I found no way to do this except building the MANIFEST file using a shell script (which kind of defeats the point of the manifest, I'd have thought).
Also excluding a single file doesn't work. [...more ranting about what doesn't work in the Distutils MANIFEST file...]
Yeah, I've been bitten by a couple of these things myself; others, I haven't seen. As I said in my last post, this was a cobbled-together hack and not-at-all well thought-out. Guess it's time to fix that. What I'm moving towards now is some of the ideas that MAL, Gordan MacMillan, and Fred Drake (anyone else?) were tossing around back in September when I first brought up the MANIFEST thing. In particular: * two-phase manifest; developer writes a MANIFEST.in that is short and simple, something like this (for Distutils itself): USAGE TODO MANIFEST CHANGES examples README *.py (although possibly with a more verbose syntax). Then, at some point the Distutils will process this and output MANIFEST, which is used a) to put together the distribution, b) to allow the developer to peek into the explicit list of files generated, and c) to allow the developer to meddle with the explicit list of files generated. However, this gets complicated when you start thinking about the interactions between MANIFEST.in and MANIFEST. I think there are cases where the module developer will just want to use his own MANIFEST and ignore my MANIFEST.in thing, and there are cases where he'll just want to clobber MANIFEST from MANIFEST.in every time. I started scribbling down some thoughts on this a few weekends ago, but got distracted. * more explicit, clearer syntax Anyways, I'm open to ideas on how best to handle the MANIFEST.in/MANIFEST thing (and syntax for the former). Tell me what features you want, whether you even like the idea of going from a short, simple list of exclude/include patterns to an explicit list of every file, how you would like to use such a system, etc. (My main concern is that we *not* just have a simple MANIFEST file that the developer has to create and maintain; that's how Perl's MakeMaker expects you to work, and it's a PITA. It provides a module to help you generate the MANIFEST the first time, but after that you're pretty much on your own.) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
On Mon, 29 Nov 1999 at 1:18pm, Andrew M. Kuchling wrote:
One field that might be interesting is, for lack of a better name, a 'status' field that said whether a project is maintained or unmaintained.
In fact (i'm happy to say) i have exactly such a field. And for lack of a better name, it is called exactly that. (-: It is non-obvious. My approach (though perhaps wrong-headed in some places) has been to omit any unnecessary information from display (a bit of an interface pet peave of mine is "too much cluttering useless information"---so i may err too much in the other direction). Since the status is set by default to "Alive and Well" on nearly all the items in the database i consider the info redundant and it is not printed. However, as soon as the status changes from Alive and Well, it shows up. I should explicitly document this behavior somewhere (at least), and make a display showing all possible fields. You can see this in the "Lost/Broken" category where i've dumped items with broken links... they have a status flag to that respect... on the details for each of them you'll see the "status" item appear... in bright red letters even... (hopefully to attract someone's attention who can fix the link --- have fixed many already --- many that remain broken sadly on python.org --- i should be keeping track and inform them... but it's hard to keep track... i think now though of Lyntin, and X Extensions --- two recent ones that come to mind --- it is heartening at least to see people actively making the effort to use the "report dead link" function on my detail pages.) In fact the flag is an integer and there's about 6 statuses possible at the moment (though i'm wanting to refine this and have modified it a few times already).... (excuse my even worse dictionary name than field name <G>)... generally they go from best to worst condition... statuses = { 0: 'Alive and Well', 1: 'In transition/Will be back', 2: 'No Longer Supported', 3: 'Unknown/Author Disappeared', 5: 'Now in Standard Distribution', 8: 'Unknown/Dead Link', 9: 'Dead and Gone', } The only flag used so far is the "8" ... oh, and the "5" in the single case of the old cPickle module. (And i think i may have accidentally given a 9 flag where i meant 8 in a few cases... as i say, it's in flux and a bit confusing. The above statuses (as now programmed) only show up on the details page when it is non-zero. Not sure how and where to use the rest... but i have created them in the event that one day i get a flash of insight in this direction. The *real* problem is figuring out when the heck something becomes "unmaintained". Not a lot of authors announce that sort of thing. <g> Perhaps there should be a time limit. It could even happen automatically. If there is no update in x months the flag automatically gets bumped to "unmaintained". But then this is probably not accurate either, as sometimes people "maintain" old work for quite a while... just don't actively develop it anymore, so perhaps there should be a distinction there. Of course there is always then just the "last updated" date in my database. I had intended to impleemnt some filters at some point. For example, so epople can, if they want, filter out everything more than a year old, etc. Or more than a week old, if they insist. Heh. I have been thinking about these issues especially lately as I've been working through the /pub/python/contrib directories. I've added (maintaining the file dates as 'last modified') already everything from Database/ and DataStructures/ (which i later noticed was also symlinked to Math/). There is a lot of really, really old junk in there. Though there's also some really, really old nifty neato stuff too. (-: (Some little obscure gems i had not noticed before---just because it's old doesn't necessarily mean it's bad!) But what to do about the old stuff which is obsolete. And i'm especially bothered by the thought of old stuff which is so old that it will not even work with python 1.5x ... i wish i had some way of flagging that... to warn unsuspecting people. On the other hand, there's not *that* much of the old stuff, so perhaps it's not worth worrying about that much at this time. If you know of any Abandoned/Unsupported code, let me know. I'll update the flags ASAP!! (-:
containing the actual code; Parnassus points to human-readable Web pages, not directly to the .tgz file, and it probably shouldn't have a
This has changed. I have added a second field. Now each item has "URL" (intended to point always and only to human readable page---or ftp directory), and also now a "download" field, which is intended to point whenever possible to binary (or raw script) (though only a few dozen items have this so far). So already i have moved, at least a little, in this direction. Again, this is a bit non-obvious to viewers... in that i only print on the details page what is available. If there is no "download" link defined then it simply does not show up, rather than saying "Not Available", or whatever. But if there is a direct download link, it shows up. If you do an URL search for "gz" for example, you'll notice most if not all of them items will have "download" links now. Also the submission form has the added field.
pointer to it. How does CPAN handle this? Does it list foo-*.tgz and take the one where * is the highest version number?
CPAN lists everything last i checked (which was a long time ago!). It's up to the user to decide what version they want (as if anyone doesn't want the latest <G>)... but i'm far from expert (or even slightly knowledgable) on the workings of CPAN. ...
On Mon, 29 Nov 1999, Greg Ward wrote:
On 28 November 1999, Michael Hudson said:
Yes! I spent a frustrating few minutes trying to work out why files I was trying to exclude from the distribution were turning up in the tarball...
Oops, sorry.
That's OK :-)
Also, excluding files from the dist in general is a pain.
I have bytecodehacks in CVS; I don't want to distribute the CVS folders. I found no way to do this except building the MANIFEST file using a shell script (which kind of defeats the point of the manifest, I'd have thought).
Also excluding a single file doesn't work. [...more ranting about what doesn't work in the Distutils MANIFEST file...]
Yeah, I've been bitten by a couple of these things myself; others, I haven't seen. As I said in my last post, this was a cobbled-together hack and not-at-all well thought-out. Guess it's time to fix that.
It would be good if the distutils could be used to distribute stuff... [snip two phase plan]
Anyways, I'm open to ideas on how best to handle the MANIFEST.in/MANIFEST thing (and syntax for the former). Tell me what features you want, whether you even like the idea of going from a short, simple list of exclude/include patterns to an explicit list of every file, how you would like to use such a system, etc.
Right, here's an idea. I'm not sure it's the best way to go, or even a reasonble one, but I'm just going to braindump for a while. At the base, I think you want to be able to supply a Python function that gets passed a path to a file and returns 1 if it should be included or 0 if not. But better would be a class (more Pythonic) with an "accept" method, e.g. class PermissiveAcceptor(Acceptor): # I'll explain Acceptor later def accept(self,file): return 1 then you need a way of combining Acceptors so that you can combine: class BackupRejector(Acceptor): def accept(self,file): return file[-1] <> '~' class PycRejector(Acceptor): def accept(self,file): if len(file) > 4: return file[-4:] <> ".pyc" Now my idea of how to do this is to define methods in the Acceptor class class Acceptor: def __and__(self,rhs): return AndingAcceptor(self,rhs) def __or__(self,rhs): return OringAcceptor(self,rhs) def __neg__(self): return NotAcceptor(self) class AndingAcceptor(Acceptor): def __init__(self,one,two): self.one = one self.two = two def accept(self,file): return self.one.accept(file) and self.two.accept(file) class OringAcceptor(Acceptor): def __init__(self,one,two): self.one = one self.two = two def accept(self,file): return self.one.accept(file) or self.two.accept(file) then: myacceptor = PycRejector() & BackupRejector() [ Random aside #1: For stateless acceptors like BackupRejector it might be better to do class _BackupRejector(Acceptor): def accept(self,file): return file[-1] <> '~' BackupRejector = _BackupRejector() ] [ Random aside #2: guess who's being doing some functional programming of late? ] Then setup would take another argument like "file_chooser" that would be called with every file in the heirachy. [ Random aside #3: hmm... maybe should have accept_file, accept_dir ] This would not be a particularly efficient solution, but that shouldn't really be a problem, should it? Then one has to provide a set of classes built in to distutils so that in the common cases, one doesn't have to write lots of code. Obvious candiate is MANIFESTAcceptor... A somewhat heavier weight idea of mine is too have files in each directory that are like MANIFESTs, but just for that directory. A bit like .cvsignore ... so you would put in a file called .distutils-manifest lines like: *.py ! hacked_up.py extra_support_file I would personally find that preferable to one top level file. What would be even nicer in some ways would be the ability to have rules that affected deeper level directories sort-of Acquisition style (if that makes sense to anyone - I know what I mean, btu I can't seem to express it properly and I've rabbited on for long enough now...)
(My main concern is that we *not* just have a simple MANIFEST file that the developer has to create and maintain; that's how Perl's MakeMaker expects you to work, and it's a PITA. It provides a module to help you generate the MANIFEST the first time, but after that you're pretty much on your own.)
This is a fate to be avoided. Thank you for your patience. Comments appreciated (I can think of a few problems already...) Michael
On Thu, 2 Dec 1999, Michael Hudson wrote:
... class Acceptor: def __and__(self,rhs): return AndingAcceptor(self,rhs) def __or__(self,rhs): return OringAcceptor(self,rhs) def __neg__(self): return NotAcceptor(self)
Sorry... but this is just silly. This is taking a simple problem of file selection and throwing a mess of procedural code at the thing. How is this *any* better than a simple list of files or file patterns? The distutils package must be *very* simple to gain acceptance. Requiring people to write code, understand how to subclass CCompiler, or provide a bazillion params to a setup() function is just not going to help. The developer ought to be able to say "ship *.py and foomodule.c". distutils would package that into a target directory and see that foomodule.c is compiled to a module on the target system. Done. Grain of salt time: until I step up and provide code to make distutils as simple as I hope it would be, my comments should be viewed as "peanut gallery" comments. Cheers, -g -- Greg Stein, http://www.lyra.org/
On Thu, 2 Dec 1999, Greg Stein wrote:
On Thu, 2 Dec 1999, Michael Hudson wrote:
... class Acceptor: def __and__(self,rhs): return AndingAcceptor(self,rhs) def __or__(self,rhs): return OringAcceptor(self,rhs) def __neg__(self): return NotAcceptor(self)
Sorry... but this is just silly. This is taking a simple problem of file selection and throwing a mess of procedural code at the thing. How is this *any* better than a simple list of files or file patterns?
The distutils package must be *very* simple to gain acceptance. Requiring people to write code, understand how to subclass CCompiler, or provide a bazillion params to a setup() function is just not going to help.
Perhaps I should have been more clear: I am in absolute agreement that it must be possible to be able to use distutils very simply. However it is my contention that it should also be extremely flexible, becuase if you want to do something a little bit unusual you should be able to do it without abandoning the advantages of distutils. I do not believe these aims are incompatible. What should be supplied is an easy to use system with sensible defaults such that the complexity of specificying which files to ship is commensurate with the complexity of your requirements. "ship *.py and foomodule.c" is a simple specification and there should be a simple way of saying it. If your specification is just a little more complex, then there should be a clear and coherent way of extending your setup.py file to accomodate it; if your specification is so fiendishly complex then you are going to need to write code to express it, then you should be able to that too. This applies to far more than just distribution systems of course, and is my main greivance with just about every bit of software that I have ever used... this is part of what the guile project is about. From http://www.gnu.org/software/guile: When you decide you need a scripting language or a configuration file, the first impulse is, ``I'll just do something clean and simple.'' This is the right impulse: a full programming language with conditionals,loops, local scopes, and procedures is just a distraction from your project. But simple languages never stay simple. For example, early releases of PHP, a language for generating web pages on the fly, touted its minute memory footprint and simplicitly. Well, the latest release of PHP has its own object system. Compare Tcl circa 1988 with the modern beast. Same story with Perl. The point here is not to criticize these (very successful) tools, but rather to show that simplicity in a scripting language doesn't last long. The real challenge is to age well. Whatever you may think of guile, they have a point. Oops, got a bit carried away there.
The developer ought to be able to say "ship *.py and foomodule.c". distutils would package that into a target directory and see that foomodule.c is compiled to a module on the target system. Done.
Yes. However that should not (in my opinion) be *all* that can be said. A case could be made that there should be a limit on how complex your specification should be, but I don't buy it. Turing complete is enough for me ;-).
Grain of salt time: until I step up and provide code to make distutils as simple as I hope it would be, my comments should be viewed as "peanut gallery" comments.
Same for me too, of course. But this is Python, so we'll be able to implement our ideas in about fifteen seconds, right <wink>? Michael
On Thu, 2 Dec 1999, Michael Hudson wrote:
... Perhaps I should have been more clear: I am in absolute agreement that it must be possible to be able to use distutils very simply. However it is my contention that it should also be extremely flexible, becuase if you want to do something a little bit unusual you should be able to do it without abandoning the advantages of distutils. I do not believe these aims are incompatible.
Here is where I'll state "we'll agree to disagree." I have yet to find something that is "easy to use, yet flexible enough for whatever you may need to do." Nobody ever does this well, and I further believe that Python's rapid development capability obsoletes the notion of trying to do everything in one package (since it is so easy to write/extend with new code). In fact, your Guile quote kind of supports what I'm trying to say: systems that have tried to encompass every need end up big-and-ugly; therefore, I would maintain that you should *not* attempt to be all-encompassing. I'd rather see a bare-ass simple mechanism that solves the 80% case and tell the other 20% to go write their own (I might actually use stronger terms for those 20% people :-). Anybody can figure out bare-ass simple and use it. They can also figure out how to extend it for their particularly whacky situation. But... IMO... and I don't expect others to share that view or to code that view :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein wrote:
I'd rather see a bare-ass simple mechanism that solves the 80% case and tell the other 20% to go write their own (I might actually use stronger terms for those 20% people :-). Anybody can figure out bare-ass simple and use it. They can also figure out how to extend it for their particularly whacky situation.
But... IMO... and I don't expect others to share that view or to code that view :-)
I agree completely. And in addition it is a Python promotion issue to have the 80% simple solution too. JimA
On 02 December 1999, Greg Stein said:
On Thu, 2 Dec 1999, Michael Hudson wrote:
... class Acceptor: def __and__(self,rhs): return AndingAcceptor(self,rhs) def __or__(self,rhs): return OringAcceptor(self,rhs) def __neg__(self): return NotAcceptor(self)
Sorry... but this is just silly. This is taking a simple problem of file selection and throwing a mess of procedural code at the thing. How is this *any* better than a simple list of files or file patterns?
Once again, I completely agree with Greg here. Michael's scheme is, ummm, interesting, but there is such a thing as being *too* object-oriented. I have not paid a lot of attention to performance in the Distutils, but I draw the line at severl method invocations to test every file in the source tree. The MANIFEST.in file should provide the level of flexibility needed by that all-important 80% of distributions. I think it's close now, but the syntax is inscrutable and the implementation flaky. (And it doesn't use MANIFEST.in, just MANIFEST.) The point of this thread is to fix those problems, not come up with an entirely different scheme. The distutils have been designed so that if a module developer *does* need some totally custom way to specify what files belong in his source distribution, it's not a problem. In theory, it goes something like this: * write a class that subclasses distutils.command.dist.Dist * override the method that finds the files to distribute * tell 'setup()' to attach your new class to the "dist" command rather than the default, distutils.command.dist.Dist and Bob's your uncle. (Disclaimer: this has never actually been tried in practice. It's a lovely design, though, if I may so so myself.) Another wee nit is that the Dist class as currently implemented has *two* methods for finding all files, which makes overriding slightly awkward. So sue me.
The distutils package must be *very* simple to gain acceptance. Requiring people to write code, understand how to subclass CCompiler, or provide a bazillion params to a setup() function is just not going to help.
I think it's getting there for module developers. The setup.py for Distutils itself is as simple as I can imagine it being. The three example setup.py's included in the distribution (for PIL, mxDateTime, and Numeric) are a tad more verbose than they really need to be -- there's a bit too much bureaucracy involved in specifying how to build simple extension modules. That's definitely fixable, though, and once it's fixed I'll be very happy with the developer interface. The end-user interface is still a bit clunky because I picked some bad names for installation directory options, and some of the semantics behind certain options are mildly bogus. (Needs a bit more do-what- I-mean-not-what-I-say logic, which must be doled out with care but can go a long way in this sort of thing.) The bogosity was only revealed as I wrote the documentation (and the IPC8 paper on the Distutils), and I haven't touched the code since then. But these problems are on my mental to-do list.
Grain of salt time: until I step up and provide code to make distutils as simple as I hope it would be, my comments should be viewed as "peanut gallery" comments.
Hey, toss all the peanuts you like. My hide's not quite as thick as an elephant's, but so far nothing on this sig has offended me too deeply. Greg
Greg Ward writes:
I think it's getting there for module developers. The setup.py for Distutils itself is as simple as I can imagine it being. The three
This reminds me of something; I've looked at using Distutils for the PyXML package, and ran aground on compiling Expat, a library required for the other stuff. What I'd want to do is jump out of the Distutils framework and run 'cd expat ; make'. How should that be done? Writing my own subclass for the build command? Is there a 'run this set of shell commands' hook? Or should I just list every single .c file in Expat as a dependency for the relevant module? (After all, running shell commands is a Unix-ism.) -- A.M. Kuchling http://starship.python.net/crew/amk/ The past is always knocking at the door, trying to break through into today. -- Dr Occult, in BOOKS OF MAGIC #1
On 02 December 1999, Andrew M. Kuchling said:
This reminds me of something; I've looked at using Distutils for the PyXML package, and ran aground on compiling Expat, a library required for the other stuff. What I'd want to do is jump out of the Distutils framework and run 'cd expat ; make'. How should that be done? Writing my own subclass for the build command? Is there a 'run this set of shell commands' hook? Or should I just list every single .c file in Expat as a dependency for the relevant module? (After all, running shell commands is a Unix-ism.)
Not handled yet. I had the same problem writing the example setup.py for PIL; it simply doesn't know how to build /F's libimaging.a, so requires the user to do it. Blecchhh. The good news is that the Distutils compiler framework was designed to be able to handle this sort of thing. The bad news is I'm not sure how the hint "please build library X first" should go into the setup file. Perhaps a new command, "build_lib" or "build_clib" that -- like "build_ext" -- uses the compiler framework to build C code, but with a libX.a or X.lib file as the end result, rather than a Python extension module. My main concern is not limiting myself to C libraries; I *assume* C++ libraries are much the same, but I know for a fact that Java is very different (and yes, I still want to support Java extensions for JPython someday). For now, I would put something like this: if not os.path.exists libfile ("expat"): if os.plat == 'posix': system "cd expat ; make" else: raise SystemExit, "you have to build the expat library first" where the function 'libfile' might look like this: def libfile (libname): from distutils.ccompiler import new_compiler compiler = new_compiler () return compiler.library_filename (libname) Paying attention to the directory where libexpat.a (or whatever it's called) should live is left as an exercise for the reader. Hint: the 'library_filename()' method doesn't know about directories (hmmm). Greg
On Thu, 2 Dec 1999, Greg Ward wrote:
... if not os.path.exists libfile ("expat"): if os.plat == 'posix': system "cd expat ; make" else: raise SystemExit, "you have to build the expat library first"
Just to be annoying, I'll point out that the "system" approach won't work ... Expat's makefile doesn't create a libexpat.a :-) Of course, reasonable people add in that line to the Makefile before bundling Expat in their distro. :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
On Thu, 2 Dec 1999, Greg Ward wrote:
On 02 December 1999, Greg Stein said:
On Thu, 2 Dec 1999, Michael Hudson wrote:
... class Acceptor: def __and__(self,rhs): return AndingAcceptor(self,rhs) def __or__(self,rhs): return OringAcceptor(self,rhs) def __neg__(self): return NotAcceptor(self)
Sorry... but this is just silly. This is taking a simple problem of file selection and throwing a mess of procedural code at the thing. How is this *any* better than a simple list of files or file patterns?
Once again, I completely agree with Greg here. Michael's scheme is, ummm, interesting, but there is such a thing as being *too* object-oriented. I have not paid a lot of attention to performance in the Distutils, but I draw the line at severl method invocations to test every file in the source tree.
OK ... <sound of cute idea being sent to bin>.
The MANIFEST.in file should provide the level of flexibility needed by that all-important 80% of distributions. I think it's close now, but the syntax is inscrutable and the implementation flaky. (And it doesn't use MANIFEST.in, just MANIFEST.) The point of this thread is to fix those problems, not come up with an entirely different scheme.
Fine. What are the problems with the current system? I know exclusion of single files doesn't work, and you can provoke some dodgy looking tracebacks with malformed MANIFESTs.
The distutils have been designed so that if a module developer *does* need some totally custom way to specify what files belong in his source distribution, it's not a problem. In theory, it goes something like this:
* write a class that subclasses distutils.command.dist.Dist * override the method that finds the files to distribute * tell 'setup()' to attach your new class to the "dist" command rather than the default, distutils.command.dist.Dist
and Bob's your uncle. (Disclaimer: this has never actually been tried in practice. It's a lovely design, though, if I may so so myself.)
Another wee nit is that the Dist class as currently implemented has *two* methods for finding all files, which makes overriding slightly awkward. So sue me.
My problem with this is that is a discontinous jump from the simple case to the complex one - although as the other Greg pointed out, that's a hard problem noone's got right, so just getting something that works for most cases is a more important goal.
The distutils package must be *very* simple to gain acceptance. Requiring people to write code, understand how to subclass CCompiler, or provide a bazillion params to a setup() function is just not going to help.
I think it's getting there for module developers.
I certainly found it pleasant enough, MANIFEST nits aside.
The setup.py for Distutils itself is as simple as I can imagine it being. The three example setup.py's included in the distribution (for PIL, mxDateTime, and Numeric) are a tad more verbose than they really need to be -- there's a bit too much bureaucracy involved in specifying how to build simple extension modules. That's definitely fixable, though, and once it's fixed I'll be very happy with the developer interface.
The end-user interface is still a bit clunky because I picked some bad names for installation directory options, and some of the semantics behind certain options are mildly bogus. (Needs a bit more do-what- I-mean-not-what-I-say logic, which must be doled out with care but can go a long way in this sort of thing.) The bogosity was only revealed as I wrote the documentation (and the IPC8 paper on the Distutils), and I haven't touched the code since then. But these problems are on my mental to-do list.
I would say it would be essential for python setup.py --help to produce something more helpful than option --help not recognized I found the simple act of installing distutils somewhere other than than the default to involve considerable head scratching and staring at the USAGE file. I'd have thought this would be the commonest option passed to install. Regards, Michael
[I explain how a programmer could theoretically dodge the MANIFEST mechanism and do-it-himself] [Michael Hudson replies]
My problem with this is that is a discontinous jump from the simple case to the complex one - although as the other Greg pointed out, that's a hard problem noone's got right, so just getting something that works for most cases is a more important goal.
The discontinuous jump -- from the simple job of writing MANIFEST.in to the complex job of subclassing distutils.command.dist.Dist -- doesn't bother me. First of all, I don't think the complex case should be all that hard (but then, I seem to have an object-oriented brain). More importantly, you *can* do the complex case -- there's no brick wall barring your way from further progress, just a small hurdle to jump. If you want to do weird things with the Distutils, you have to jump through hoops -- hopefully small hoops, but the key point is that you shouldn't wind up bashing your head against the wall. That's why those last two letters of "setup.py" are so important... no brick walls. ;-)
I certainly found it pleasant enough, MANIFEST nits aside.
Thanks!
I would say it would be essential for
python setup.py --help
to produce something more helpful than
option --help not recognized
D'oh! Somehow that one slipped through my clue-filter. Consider it noted...
I found the simple act of installing distutils somewhere other than than the default to involve considerable head scratching and staring at the USAGE file. I'd have thought this would be the commonest option passed to install.
Yes, there definitely needs to be an easier grade from the "trivial" case to slightly customized to heavily customized cases. This is mostly a documentation problem, I think. (Although I have to actually *try* some of those slightly- and heavily-customized cases before I can be certain...) Greg
On Thu, 2 Dec 1999, Greg Ward wrote:
I found the simple act of installing distutils somewhere other than than the default to involve considerable head scratching and staring at the USAGE file. I'd have thought this would be the commonest option passed to install.
Yes, there definitely needs to be an easier grade from the "trivial" case to slightly customized to heavily customized cases. This is mostly a documentation problem, I think. (Although I have to actually *try* some of those slightly- and heavily-customized cases before I can be certain...)
Further mucking about shows that python setup.py -v install_py --install-dir=/home/mwh21/src/python/ does what I want ... but python setup.py -nv install --install-dir=/home/mwh21/src/python/ produces the dreaded option --install-dir not recognized While python setup.py -nv install --install-site-lib=/home/mwh21/src/python/ works, and python setup.py -nv install_py --install-site-lib=/home/mwh21/src/python/ doesn't ... I don't think that's helpful. OTOH, I don't see how to make it better. I guess one would need to know what people actually would be wanting this command to do on a regular basis and make that easy. As a data point, I like to chuck everything experimental in ~/src/python; I think currently for a mixed extension/Python package I'd have to type something along the lines of python setup.py install --install-site-platlib=~/src/python \ --install-site-lib=~/src/python which is a bit unwieldy. Maybe I should make an alias for that... Would it be reasonable to have an "interactive" mode, or am I just being silly again? Cheers, Michael
Michael Hudson writes:
Further mucking about shows that [Lots of examples of busted behavior elided...]
I tihnk this really shows that many of these things really need to be global options and that the local options thing makes very little sense. Time to simplify!
Would it be reasonable to have an "interactive" mode, or am I just being silly again?
Yes. ;) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
On 02 December 1999, Fred L. Drake, Jr. said:
Michael Hudson writes:
Further mucking about shows that [Lots of examples of busted behavior elided...]
I tihnk this really shows that many of these things really need to be global options and that the local options thing makes very little sense.
I beg to differ -- I'll have to take a closer look at what's going on with Michael's examples of brokenness, but I like to think they can be fixed without completely tossing the existing design. See "perldoc ExtUtils::MakeMaker" for my rationale behind having most behaviour controlled by command options. 22 pages of documentation describing all of MakeMaker's options (which are all global) is a bit overwhelming; I wanted to be able to categorize things a bit. [Michael again]
Would it be reasonable to have an "interactive" mode, or am I just being silly again?
Interesting notion. Would this just be a shell for the Distutils commands, eg. ./setup.py --interactive
build [...output of 'build' command...] install [...output of 'install' command...] quit
or did you have something else in mind? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
I said:
I tihnk this really shows that many of these things really need to be global options and that the local options thing makes very little sense.
Greg Ward writes:
I beg to differ -- I'll have to take a closer look at what's going on with Michael's examples of brokenness, but I like to think they can be fixed without completely tossing the existing design.
I'm not suggesting that these things shouldn't be options, only that they should be global options to the setup.py script, not local to each sub-command. For options that apply to a specific sub-command, go ahead and make them command-local. For options that describe the Python installation that the package is being added to, use global options. These are used to describe the target environment, and there's no good reason not to group them in the global options space. -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
On 07 December 1999, Fred L. Drake, Jr. said:
I'm not suggesting that these things shouldn't be options, only that they should be global options to the setup.py script, not local to each sub-command. For options that apply to a specific sub-command, go ahead and make them command-local. For options that describe the Python installation that the package is being added to, use global options. These are used to describe the target environment, and there's no good reason not to group them in the global options space.
I think that goes hand-in-hand with figuring out what's really needed, what's most commonly needed, and fixing up the installation options to reflect those (perceived) needs. That and rethinking the "dist" command (in particular, the MANIFEST/MANIFEST.in syntax and semantics) are on my list for Distutils 0.2. Greg
On 27 November 1999, A.M. Kuchling said:
Some things I came across while adding distutils support to some code:
* From a docstring in distutils/versions.py: 0.4 0.4.0 (these two are equivalent) 0.4.1
Does this mean that 0.4.1 is equivalent to 0.41 in the strict versioning class? I'd suggest adding that, just to make it crystal clear.
That is not currently the case, and I think the "strict" version regex -- as well as the semantics attached to its bits and pieces -- would need some work for that to happen. It seems attractive at first blush, but I'm not entirely sure that it's a good idea. "0.4.1 == 0.41" sort-of implies "1.11 == 1.1.1", which is definitely *not* attractive. ESR's "Software Release Practices HOWTO" argues in favour of major.minor.patch version numbering, and I agree with that. See http://www.linuxdoc.org/HOWTO/Software-Release-Practice-HOWTO-2.html#ss2.1
* The name option in setup.py: It's not clear if it can/should contain spaces; given that you mention underscores, I assume they're as space substitutes.
Greg Ward writes:
On 27 November 1999, A.M. Kuchling said:
Does this mean that 0.4.1 is equivalent to 0.41 in the strict versioning class? I'd suggest adding that, just to make it crystal clear.
Sorry, I miswrote; I meant, "if that is the case, the example should be added to the docstring to make it clear." If that's not the case, though, I wasn't suggesting adding it, and agree that the 1.11 == 1.1.1 consequence is unpleasant.
...which probably just means I need to work on the documentation.
It would be a good idea to have a brief page or two up top which covers the basics extremely quickly, letting you get started quickly. The Python distrib's README sets a good precedent: If you don't read instructions ------------------------------ Congratulations on getting this far. :-) To start building right away (on UNIX): type "./configure" in the current directory and when it finishes, type "make". The section Build Instructions below is still recommended reading. :-) ... <and then continues for about 900 more lines>
could say "setup.py dist" to make the Distutils 0.1 release. It works for putting together the Distutils release, but isn't much good for anything else. ;-( See my next post...
Actually, I thought it worked fine, once I figured out the hard-link thing, and I really liked the ability to write patterns in the MANIFEST. Another thing I thought of: Tools/versioncheck from the Python distrib should be supported in some way. While versioncheck is for checking the versions of installed packages (so it's not obsoleted by the Distutils), it's still worth adding a 'checkversion' command that would simply print out whether a more recent version of the code is available. -- A.M. Kuchling http://starship.python.net/crew/amk/ The age of chivalry is gone. That of sophisters, economists and calculators has succeeded: and the glory of Europe is extinguished for ever. -- Edmund Burke, _Reflections on The Revolution in France_
participants (10)
-
A.M. Kuchling
-
Andrew M. Kuchling
-
David Ascher
-
Fred L. Drake, Jr.
-
Greg Stein
-
Greg Ward
-
James C. Ahlstrom
-
Michael Hudson
-
Michael P. Reilly
-
T-Mathy Meddleton