Re: [Distutils] The "dist" command
Greg wrote: <braindump> Possible modes of operation: - require an explicit manifest that lists every single file (presumably along with a way to auto-generate the manifest) - require an explicit manifest, but allow it to have globs or filename patterns of some kind (and also have auto-generation) - allow an explict manifest, but automatically augment it at runtime with the source files mentioned in 'packages', 'py_modules', and 'ext_modules' (and any other such things that might come along) Since you can't possibly anticipate the preferences of every sicko out there, I'd suggest a 2 phase approach. Use specifiers / filters to create an explicit manifest. Let the programmer edit the manifest, then go on to phase 2. - Gordon
On 24 September 1999, Gordon McMillan said:
Greg wrote: <braindump> Possible modes of operation: - require an explicit manifest that lists every single file (presumably along with a way to auto-generate the manifest) - require an explicit manifest, but allow it to have globs or filename patterns of some kind (and also have auto-generation) - allow an explict manifest, but automatically augment it at runtime with the source files mentioned in 'packages', 'py_modules', and 'ext_modules' (and any other such things that might come along)
Since you can't possibly anticipate the preferences of every sicko out there, I'd suggest a 2 phase approach. Use specifiers / filters to create an explicit manifest. Let the programmer edit the manifest, then go on to phase 2.
Gordon, Fred, Marc-André: thanks for your ideas. They're all, ummm, interesting. However, I'm trying to balance simplicity and utility here, and I think my original manifest syntax does an adequate job of that. At least it works fine for generating a source distribution for Distutils 0.1. The MANIFEST file is simplicity itself: # # List of files to ship in the Distutils source distribution (apart from # the standard stuff -- Python source files, README and setup script, # test scripts). # # GPW 1999/09/23 # # $Id$ # examples * !*~ !*.py[co] Gee, I haven't even checked it in yet. Oh well, I've got *tons* of changes from the weekend still to checkin. And then I finish writing the USAGE file, which should probably be added to the MANIFEST. (Hmmm: perhaps the "list of default files" should include /^[A-Z_]+(\.txt)?$/, to handle README and USAGE and LICENCE and TODO and INSTALL and all those conventional all-uppercase filenames...) So yes, I'm ignoring Fred's idea and using the "default" files (all source files implied by 'py_modules', 'ext_modules', and 'packages', plus README and setup.py) even when there's a MANIFEST. And I'm ignoring Marc-André's idea: there's only one MANIFEST. I don't need anything fancy for *my* little module distribution, so it's hard for me to predict what more complex distributions might throw in. Oh yeah, I'm ignoring Gordon's suggestion too: if you want to list every file explictly, you're welcome to do so! If you want automated help in doing so, may I suggest the 'find(1)' command? >smirk< Oh, one comment on the idea of separate MANIFESTs for source and binary distributions: no. The binary distribution should be built from the 'build' directory; the whole point of that directory is to make installation and the generation of binary (err, built) distributions trivial. The information about what goes in the built distribution should come from an extra 'build' phase, eg. 'build_doc'. Wow, I never thought there'd come a day when I would refuse complexity and featuritis. This Python thing must be getting to me. Better go spend some time hacking Perl to remind myself of the joys of unbridled complexity. ;-) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
Greg Ward writes:
explictly, you're welcome to do so! If you want automated help in doing so, may I suggest the 'find(1)' command? >smirk<
I don't think there's any equivalent for Windows without buying 3rd party addons (not an acceptable alternative). It would be easy to implement using os.path.walk(); I'm sure an interested Windows-bound reader of this group could contribute such a beast.
Oh, one comment on the idea of separate MANIFESTs for source and binary distributions: no. The binary distribution should be built from the 'build' directory; the whole point of that directory is to make installation and the generation of binary (err, built) distributions
It's too early to say, probably: those of us dragging our feet need more experience with what you already have before we can say that it's absolutely required or you can say that it absolutely isn't. -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
On 27 September 1999, Fred L. Drake, Jr. said:
Greg Ward writes:
explictly, you're welcome to do so! If you want automated help in doing so, may I suggest the 'find(1)' command? >smirk<
I don't think there's any equivalent for Windows without buying 3rd party addons (not an acceptable alternative). It would be easy to implement using os.path.walk(); I'm sure an interested Windows-bound reader of this group could contribute such a beast.
Hey, I was smirking when I said that. The whole point of having some wildcard support in the Distutils MANIFEST format is that people don't need to use 'find', or a separate "make_manifest" script to generate it: they just type a few lines and it works. You and Marc-André have come up with some clever ideas that will allow people to type even fewer lines in some circumstances, at the expense of more complexity in the code.
It's too early to say, probably: those of us dragging our feet need more experience with what you already have before we can say that it's absolutely required or you can say that it absolutely isn't.
Exactly -- I claim that the current MANIFEST syntax is excellent for generating a source distribution of the Distutils. It would also work quite well for the other software I've released in the past, whether it's a C library, a MATLAB toolbox, or Perl module distributions. It's certainly better than the MANIFEST format used by Perl's MakeMaker (all files explicitly listed, with the ExtUtils::MakeManifest module to help you generate it the first time -- good luck maintaining it). I hope it fits the style of other developers, but we'll have to wait and see on that. Hyper-defensively yours, Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
Hi Greg, [Sorry for jumping in rather abruptly; I've been busy moving to and from the UK recently...]
Hey, I was smirking when I said that. The whole point of having some wildcard support in the Distutils MANIFEST format is that people don't need to use 'find', or a separate "make_manifest" script to generate it: they just type a few lines and it works. You and Marc-André have come up with some clever ideas that will allow people to type even fewer lines in some circumstances, at the expense of more complexity in the code.
My two pennies (with a bit of luck I'll comment on something you recognize): Manifest files should be as explicit as possible. I.e., wildcards should only be allowed in a preprocessor tool for generating the initial manifest file. The reason for this is simply that you don't want to accidentally ship your boss' latest hate mail to you, because you accidentally saved it in the source directory :) (so, I prefer the 'make_manifest' approach) As for 'find', I am still, extremely slowly, working on Python replacements for the most common Unix/GNU utilities. It would be kind of nifty to have a 'find' module written so generically, using keyword parameters, that you could use if from your own Python scripts: User = os.environ["USER"] Edits = posixcmd.find("/src", perm="-u+w", name="*.c", type="f", user=User) if Edits == []: print "You can go home now, %s. See you tomorrow." % User else: print "You need to check in the following files %s:" % User for Edit in Edits: print "\t%s" % Edit -- Mikael
On Tue, 28 Sep 1999, Mikael Lyngvig wrote:
As for 'find', I am still, extremely slowly, working on Python replacements for the most common Unix/GNU utilities.
Interesting project. I don't know if you know that TomC has a similar project for Perl: http://language.perl.com/ppt/ I like the notion of a module w/ functions as opposed to just getopt parsing. --david
Interesting project.
And shouldn't be that hard given Python already has built-in regular expressions, etc.
I don't know if you know that TomC has a similar project for Perl: http://language.perl.com/ppt/
I like the notion of a module w/ functions as opposed to just getopt
I almost see it as a question of personal integrity to NOT know what the Perl dudes are doing :) Shame on me! Perl is probably the greatest competitor to Python, so one should now - and try to give Python something better. parsing. Yes, but Greg pointed out a severe short-coming (and I found one more: keyword args don't preserve the order of the options), so it would have to be a list of options, or even a simple command-line string, so there's not much difference. -- Mikael
Mikael Lyngvig writes:
Yes, but Greg pointed out a severe short-coming (and I found one more: keyword args don't preserve the order of the options), so it would have to be a list of options, or even a simple command-line string, so there's not much difference.
Mikael, Still, a lot of the common cases can be handled by convenience functions that either do the right thing simply or build the more complicated paramters to the general implementation. find, in particular, is subject to the ordering problem, but most commands are not. Sounds like a cool thing to have handy! -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
Greg Ward wrote:
Gordon, Fred, Marc-André:
thanks for your ideas. They're all, ummm, interesting. However, I'm trying to balance simplicity and utility here, and I think my original manifest syntax does an adequate job of that. At least it works fine for generating a source distribution for Distutils 0.1. The MANIFEST file is simplicity itself:
# # List of files to ship in the Distutils source distribution (apart from # the standard stuff -- Python source files, README and setup script, # test scripts). # # GPW 1999/09/23 # # $Id$ #
examples * !*~ !*.py[co]
Gee, I haven't even checked it in yet. Oh well, I've got *tons* of changes from the weekend still to checkin. And then I finish writing the USAGE file, which should probably be added to the MANIFEST. (Hmmm: perhaps the "list of default files" should include /^[A-Z_]+(\.txt)?$/, to handle README and USAGE and LICENCE and TODO and INSTALL and all those conventional all-uppercase filenames...)
So yes, I'm ignoring Fred's idea and using the "default" files (all source files implied by 'py_modules', 'ext_modules', and 'packages', plus README and setup.py) even when there's a MANIFEST. And I'm ignoring Marc-André's idea: there's only one MANIFEST. I don't need anything fancy for *my* little module distribution, so it's hard for me to predict what more complex distributions might throw in. Oh yeah, I'm ignoring Gordon's suggestion too: if you want to list every file explictly, you're welcome to do so! If you want automated help in doing so, may I suggest the 'find(1)' command? >smirk<
Oh, one comment on the idea of separate MANIFESTs for source and binary distributions: no. The binary distribution should be built from the 'build' directory; the whole point of that directory is to make installation and the generation of binary (err, built) distributions trivial. The information about what goes in the built distribution should come from an extra 'build' phase, eg. 'build_doc'.
How does that work ? I usually let my pkgcreate run over the same package tree twice: once for source distribution and then again for a binary one -- pretty easy I'd say and without *any* overhead on my part since the default file inclusion/exclusion sets are chosen by sinlge option on the command line.
Wow, I never thought there'd come a day when I would refuse complexity and featuritis. This Python thing must be getting to me. Better go spend some time hacking Perl to remind myself of the joys of unbridled complexity. ;-)
Greg, what we proposed here is not featuritis, it's plain experience. All of the mentioned things are needed one way or another. Anyway, I guess that a simple tool to create the MANIFEST file will probably already do the trick for all of us. Perhaps you could add an option to choose the filename for it... e.g. -M <MANIFEST-filename>. Or have the dist tool look for a MANIFEST.py file and let it return the list of files to process via a get_files() API. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 98 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
On 28 September 1999, M.-A. Lemburg said:
Greg, what we proposed here is not featuritis, it's plain experience. All of the mentioned things are needed one way or another.
Anyway, I guess that a simple tool to create the MANIFEST file will probably already do the trick for all of us. Perhaps you could add an option to choose the filename for it... e.g. -M <MANIFEST-filename>. Or have the dist tool look for a MANIFEST.py file and let it return the list of files to process via a get_files() API.
Already there; from distutils/distutils/command/dist.py: options = [('formats=', 'f', "formats for source distribution (tar, ztar, gztar, or zip)"), ('manifest=', 'm', "name of manifest file"), ] thus: python setup.py -v dist --manifest MyManifestFile or python setup.py -v dist -m MyManifestFile will do what you want. (Well, it *should*, I haven't tested it...) Please note, I have never said all of these ideas that have circulated are baseless or silly or unneeded. I just want some real developers to get real experience using the Distutils on real module distributions before we make a final decision about which all-singing, all-dancing solution is needed for manifest files. There's more than one way to do it, and I don't want to implement (or support or maintain) all of them. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
participants (6)
-
David Ascher -
Fred L. Drake, Jr. -
Gordon McMillan -
Greg Ward -
M.-A. Lemburg -
Mikael Lyngvig