Re: [Distutils] thoughts on distutils 1 & 2 (addendum)
Hey, Couple more thoughts to add to yesterday's: - Decoupling DU build procedures from DU installation procedures also implies delegating control of building and installing to separate scripts (e.g. 'build.py', 'install.py'), rather than having a single 'setup.py' script doing double duty. Superficially this may sound like it's creating more(!) work for module developers, but bear in mind my goal of making these scripts more generic so that in most cases the module developer can simply [re]use existing ones rather than have to code new versions each time. - Regarding the "human-readable flag" aspect of setup.py, as this does cause some concern... In an _ideal_ world, the absence of a setup.py script would simply indicate that a module could be installed via a generic installation process. This _not_ being an ideal world, however (i.e. non-DU-compatible modules also lack a setup.py script, making it hard to tell the two forms apart), there's no reason a standard 'install.py' script couldn't always be included. Also, removing metadata from setup scripts means that most of the time 'install.py' will be a completely generic script that can be automatically added to the .zip at build-time; one less thing for the module developer to have to do themselves. Cheers, has -- http://freespace.virgin.net/hamish.sanderson/
On Wed, May 19, 2004 at 05:14:57PM +0100, has wrote:
Also, removing metadata from setup scripts means that most of the time 'install.py' will be a completely generic script that can be automatically added to the .zip at build-time; one less thing for the module developer to have to do themselves.
You keep saying "the .zip" as if that's the only, or possibly prefered, distribution mechanism. It sounds as if your exposure to heterogenous systems is pretty limited and I think you are imposing the limitations of your view onto Distutils. Your recurring recommendations for monolithic scripts, such as build.py, install.py, whatever.py also seems to be a step backward from Distutils current capabilities. The functionality I think you intend by these scripts is already integrated in Distutils. It is accomplished by subclassing the build and install command classes or their helper commands. In an object oriented language, this is the appropriate way to re-use code; Not to have to include duplicate scripts in each package's source tree. Please review some more setup.py scripts. I agree that metadata should be provided in a consistent place (e.g. setup.py or setup.cfg but not both) but if you review most setup.py scripts, you'll find what is simply a function call that passes metadata to classes that do everything you're describing and more. In more complex cases, you'll see the same function call, referencing user-supplied subclasses of internal commands that don't quite do what the author needs. I'll also agree that how this is done is not inuitive, nor well documented, but it works none-the-less. I appreciate that your suggestions are based on your experience with Distutils. It seems to me however, that your experience is limited and some of the suggestions you are proposing will impose severe limitations on those of us that deal with more diverse environments. mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/
has wrote:
Hey,
Couple more thoughts to add to yesterday's:
- Decoupling DU build procedures from DU installation procedures also implies delegating control of building and installing to separate scripts (e.g. 'build.py', 'install.py'), rather than having a single 'setup.py' script doing double duty.
As others already pointed out, what you suggest doesn't really change anything: 'setup.py' is just a facade to access distutils' functionality. building and installing *is* already controlled by two distinct objects ('commands').
Superficially this may sound like it's creating more(!) work for module developers, but bear in mind my goal of making these scripts more generic so that in most cases the module developer can simply [re]use existing ones rather than have to code new versions each time.
what do you want to reuse that you can't right now ? As we all agree the 'build' class should be more polymorphic, i.e. it should be able to wrap your own build system for a component of the package (and that's one thing where DU1 lacks most IMO). However, in order for 'build' and 'install' to work smoothly together, they both have to respect some conventions, for example use some common metadata format to share information about the things to be installed. Regards, Stefan
Stefan Seefeld wrote:
- Decoupling DU build procedures from DU installation procedures also implies delegating control of building and installing to separate scripts (e.g. 'build.py', 'install.py'), rather than having a single 'setup.py' script doing double duty.
As others already pointed out, what you suggest doesn't really change anything: 'setup.py' is just a facade to access distutils' functionality. building and installing *is* already controlled by two distinct objects ('commands').
Superficially this may sound like it's creating more(!) work for module developers, but bear in mind my goal of making these scripts more generic so that in most cases the module developer can simply [re]use existing ones rather than have to code new versions each time.
what do you want to reuse that you can't right now ?
The 'setup.py' [or whatever you want to call it] script. Reduce the amount of code the module developer needs to write. For simple modules and packages, this could (and should) be zero.
However, in order for 'build' and 'install' to work smoothly together, they both have to respect some conventions, for example use some common metadata format to share information about the things to be installed.
Of course. My point is that some of this data shouldn't be in the setup scripts, and at least some of the rest should be automatically determined by the system rather than specified by the user (unless they need to override the automatics): - Put metadata (by which I mean information describing the module/package itself; its name, version, author, dependencies, etc) into a standard metadata file, e.g. meta.txt, within the package. - Put data/code used to build the distro into a build.py script. Put data/code used to install the distro into a install.py script. i.e. There should be a clear distinction made between information describing the module and information used merely in building/installing it (custom paths, framework bindings, etc). These are two very different things and should be handled accordingly. [Note: any time I say 'metadata' I'm referring to the former, not the latter.] With DU1, all this data is squidged into setup.py. This is suboptimal: it's not very convenient to read/edit, and there's some unnecessary duplication of metadata occurring over the build/install process as some of this data gets duplicated into PKG-INFO (which'd be unnecessary if it was put into a meta.txt file in the first place). Often the only unique information stored in setup.py is said 'metadata', which is really a suboptimal arrangement. (Plus stuff like sub-package names, which can and should be gotten rid of in most, if not all, cases.) Maybe a practical example'd help: Current: from distutils.core import setup setup(name = 'HTMLTemplate', version = '0.4.3', description = 'HTML templating engine.', author = 'HAS', author_email = '', # see Manual.txt url='http://freespace.virgin.net/hamish.sanderson/htmltemplate.html', license = 'LGPL', platforms = ['any'], long_description = 'HTMLTemplate converts XML/HTML/XHTML templates ...', py_modules = ['HTMLTemplate'], ) Suggested: - meta.txt version 0.4.3 description HTML templating engine author HAS author email see Manual.txt url http://freespace.virgin.net/hamish.sanderson/htmltemplate.html license LGPL platforms any long description HTMLTemplate converts XML/HTML/XHTML templates ... - install.py #!/usr/bin/env python from DU2 import install install() - build.py #!/usr/bin/env python from sys import argv from DU2 import build build(argv[1], omit='\.pyc$') Eliminating unnecessary duplication means the name shouldn't need to be declared more than once (i.e. the package folder name). The py_modules value is, by default, unnecessary. PKG-INFO is obsolete. README can be auto-generated (though I kinda wonder just how useful this really is and if it could be eliminated altogether). Both install.py and build.py are generic scripts: the former can be automatically inserted into the distro, the latter run from the shell as a standard build script. It'll still scale up just as well as DU1, so that's not a concern. The aim here is to handle the simplest and [presumably] most common cases more cleanly. HTH has -- http://freespace.virgin.net/hamish.sanderson/
On Wed, May 19, 2004 at 11:07:02PM +0100, has wrote:
Stefan Seefeld wrote:
what do you want to reuse that you can't right now ?
The 'setup.py' [or whatever you want to call it] script. Reduce the amount of code the module developer needs to write. For simple modules and packages, this could (and should) be zero. [snip] My point is that some of this data shouldn't be in the setup scripts, and at least some of the rest should be automatically determined by the system rather than specified by the user (unless they need to override the automatics):
I think, with proper metadata definitions standardized in setup.cfg, DU could be modified such that: from distutils import setup setup() would simply pull all it's information from the setup.cfg and go. That's _almost_ zero. I don't think you can get to absolute zero because somewhere the author has to specify the metadata.
- Put metadata (by which I mean information describing the module/package itself; its name, version, author, dependencies, etc) into a standard metadata file, e.g. meta.txt, within the package.
That's what setup.cfg is.
- Put data/code used to build the distro into a build.py script. Put data/code used to install the distro into a install.py script.
Not necessary at all.
i.e. There should be a clear distinction made between information describing the module and information used merely in building/installing it (custom paths, framework bindings, etc). These are two very different things and should be handled accordingly. [Note: any time I say 'metadata' I'm referring to the former, not the latter.]
That would be nice, but if the package author requires "custom paths, framework bindings, etc." there's simply no way DU could guess where those come from so the author has to put them somewhere. The setup.cfg file (http://python.org/doc/current/dist/setup-config.html) already has different sections for the build and install metadata _if_ they are required.
With DU1, all this data is squidged into setup.py. This is suboptimal: it's not very convenient to read/edit, and there's some unnecessary duplication of metadata occurring over the build/install process as some of this data gets duplicated into PKG-INFO (which'd be unnecessary if it was put into a meta.txt file in the first place).
setup.cfg == meta.txt The PKG-INFO file (iirc) is a(n extractable) subset of the setup.cfg metadata brought up in PEP 301 for the purpose of publishing package information to an "index server". It's not something that should have to be dealt with manually at all (again assuming we nail the metadata correctly).
Often the only unique information stored in setup.py is said 'metadata', which is really a suboptimal arrangement. (Plus stuff like sub-package names, which can and should be gotten rid of in most, if not all, cases.)
setup.py's dist process produces the PKG-INFO file, by combining the setup.py and setup.cfg metadata, although I'd still prefer to see setup.cfg be the "authoritative" metadata source so the "almost zero" setup.py concept would work with the author only having to maintain metadata in one place. (I think that's the force behind your argument, correct?)
Maybe a practical example'd help:
[snip] I believe almost everything in your example is available through setup.cfg _today_. Your argument underscores the point that it is clearly not documented as such. By modifying setup() to search for any unspecified parameters in setup.cfg (which it almost does now; I think the only required arg is "name") doesn't that meet your goals? mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/
I wrote:
Eliminating unnecessary duplication means the name shouldn't need to be declared more than once (i.e. the package folder name). The py_modules value is, by default, unnecessary. PKG-INFO is obsolete. README can be auto-generated (though I kinda wonder just how useful this really is and if it could be eliminated altogether). Both install.py and build.py are generic scripts: the former can be automatically inserted into the distro, the latter run from the shell as a standard build script.
Oops, forgot to mention an obvious benefit for DU developers: eliminating a not insignificant quantity of code from DU. In particular, all module metadata handling (parsing/generating/verifying) duties can be handled by a much more useful general-purpose modulemetadata module in the standard library, to which DU would become just another client. HTH has -- http://freespace.virgin.net/hamish.sanderson/
has wrote:
Oops, forgot to mention an obvious benefit for DU developers: eliminating a not insignificant quantity of code from DU.
What code would that be ? In the simplest case setup.py is quite trivial. It's only when things get more complex and require 'manual' fine-tuning that this script (or other user-specific modules it loads) becomes large. I honestly find this approach quite scalable.
In particular, all module metadata handling (parsing/generating/verifying) duties can be handled by a much more useful general-purpose modulemetadata module in the standard library, to which DU would become just another client.
what is the scope of such a 'module metadata' module ? And who would use it beside distutils ? As I said earlier, I don't believe anything beside the already existing metadata (available through pythonic introspection) is needed or even useful for modules in general. What you seem to have in mind is important for *packages*, and that's exactly distutils' scope. Whenever I hear 'general purpose' a little warning light flashes in my mind... :-) Regards, Stefan
participants (3)
-
has -
Mark W. Alexander -
Stefan Seefeld