Multi-distribution distributions
I'm somewhat surprised by the fact that some distributions (e.g. Numpy, Zodb) have multiple setup.py programs. As far as I know, these setup.py's do not share information so there is no way to do a bdist_wininst or bdist_rpm that builds a single distribution for these multiple sub-packages. I see this as a fairly large problem! The bdist_ functions are an important part of Distutils functionality. Paul Prescod
Explanation: we made these packages optional partly because they ought to be optional but partly because one of them depends on LAPACK and many people wish to configure the packages to use a different LAPACK than the limited "lite" one supplied. It would be correct in the spirit of SourceForge and Python packages to make every single one of these optional packages a separate "project" or at least a separate download. That would raise the overhead. Technically the manual should be split up into pieces too. I don't think all that trouble is worth undergoing in order to "solve" this problem. So, you could just build separate rpms for each of the optional packages. They are NOT subpackages in the true sense of the word, except that a couple of them install into Numeric's directory for backward compatibility. Numeric is not a true package either. Again, people have argued (and I agree) that purity of thought is trumped by backward compatibility and that we should leave well enough alone. You are welcome to add a script which runs the bdist functions on all the optional packages, in much the same way setup_all.py works. You do need to face the issue if making a bdist for the public of which LAPACK you use on which platform. I believe I was one of the earliest and hardest pushers for Distutils, so I have no trouble agreeing with your goals. -----Original Message----- From: numpy-discussion-admin@lists.sourceforge.net [mailto:numpy-discussion-admin@lists.sourceforge.net]On Behalf Of Paul Prescod Sent: Thursday, January 04, 2001 5:22 PM To: distutils-sig@python.org Cc: akuchlin@mems-exchange.org; numpy-discussion@lists.sourceforge.net Subject: [Numpy-discussion] Multi-distribution distributions I'm somewhat surprised by the fact that some distributions (e.g. Numpy, Zodb) have multiple setup.py programs. As far as I know, these setup.py's do not share information so there is no way to do a bdist_wininst or bdist_rpm that builds a single distribution for these multiple sub-packages. I see this as a fairly large problem! The bdist_ functions are an important part of Distutils functionality. Paul Prescod _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net http://lists.sourceforge.net/mailman/listinfo/numpy-discussion
"Paul F. Dubois" wrote:
Explanation: we made these packages optional partly because they ought to be optional but partly because one of them depends on LAPACK and many people wish to configure the packages to use a different LAPACK than the limited "lite" one supplied.
Wow, your "lite" module is half a meg. I'd hate to see the heavy version. :) I may not understand the extension so please tell me if I'm off-base: Would it be simpler to have a single setup.py and a set of flags to turn on and off each of the "secondary" extensions?
You are welcome to add a script which runs the bdist functions on all the optional packages, in much the same way setup_all.py works.
If I did this, I would consider a different strategy. I would suggest that each of the setup.py's could move their "setup" function into an if __name__=="__main__" block. Then setup_all.py could import each one (a little trickery needed there) and then combine the symbols like sourcelist, headers, ext_modules and so forth.
You do need to face the issue if making a bdist for the public of which LAPACK you use on which platform.
I would expect to use lapack_lite.pyd. It's easy enough to override it by copying a custom lapack on top. Paul Prescod
Paul Prescod wrote:
I'm somewhat surprised by the fact that some distributions (e.g. Numpy, Zodb) have multiple setup.py programs. As far as I know, these setup.py's do not share information so there is no way to do a bdist_wininst or bdist_rpm that builds a single distribution for these multiple sub-packages.
I see this as a fairly large problem! The bdist_ functions are an important part of Distutils functionality.
I don't think I understand your question: do you want a single setup.py to call all the other setup.py files similar to a recursive Makefile ? I'd be interested to hear how the other packagers deal with multiple subpackages, since I am planning to start in the same direction with my mx utils. There will be a base installation and multiple optional subpackages, but I'm currently unsure how to split up the task -- it could either be done using a setup.py which takes extra options (for choosing the subpackages in question) or by having multiple setup.py files (one for the base set and then one for each optional add-on). Thoughts ? Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
...
I don't think I understand your question: do you want a single setup.py to call all the other setup.py files similar to a recursive Makefile ?
That's exactly what I do *not want*. Recursive makefiles are easy and I'm not thrilled with recursive setup.py's either. As soon as you do that you have the usual "make" problem that the dependency checking is not unified PLUS the distutils-specific problem that any distributions you build will not be unified. You'll get six RPMs, six windows executables, etc. One of ActiveState's goals is to make extension installation as easy for people without compilers as Python installation is. Binary builds are an important part of that. If I have to tell a NumPy user to install five RPMs/Windows executables/MSIs/PPMs for one "logical" modules (NumPy), they will complain. I would like each logical "Python extension" to have one setup.py. If it is to have optional parts then there should be distutils options that turn those parts on and off. Context: I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to provide compiled versions of these to my customers, I need to be able to automate that process. I need to set up a script that downloads a random module, runs "python setup.py bdist_winexe" and posts it on a web page with little or no human intervention. I've found less than 10 distutils extensions in the world and most of them are idiosyncratic in one way or another. I think that's just growing pains but I'm going to try to iron out the idiosyncracies one by one so that the first extensions can serve as models for the hundreds that are yet to come. Paul Prescod
On Thu, Jan 04, 2001 at 11:49:06PM -0800, Paul Prescod wrote:
I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to
The unified ZODB distribution is really several extensions in one: ExtensionClasses, for example, are potentially useful in their own right without the rest of ZODB. But I don't want to make people download seven different packages, so they're wrapped up in a single distribution. The true solution would be a tool like Debian's apt-get to automatically download the dependencies for a package, but that's a long way away. If you're working on a Python package manager for ActiveState and want to package the ZODB, then all the components can just be packaged separately. I'm not sure if a similar approach would help for the mx* packages, though, and agree that some convention should be selected. Anyone got any ideas? --amk
Andrew Kuchling wrote:
On Thu, Jan 04, 2001 at 11:49:06PM -0800, Paul Prescod wrote:
I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to
The unified ZODB distribution is really several extensions in one: ExtensionClasses, for example, are potentially useful in their own right without the rest of ZODB.
I've been thinking about this today. I'd like feedback on a proposal. Let's start with terminology: there is a "missing term" in our vocabulary. People sometimes use the words "module", "extension" and "package" for a "distutils unit" but those words already have precise meanings in the Python world. I propose the term "distutils component". A distutils component consists of a directory which contains a setup.py and some other code. The component is also the unit of versioning. Some components "embed" other components. The parent component is dependent on the embedded component and for convenience, the embedded component is distributed with the parent. Embedding may go away when distutils has proper dependency tracking -- but it may not. It should be possible to create a "parent component" or "meta component" by specifying the paths to sub-components. Just as you specify "py_modules", "extensions" and so forth, you should be able to specify "sub_components". Then distutils itself would do all of the "for subcomponent in subcomponents: run_setup()" hackery that NumPy, ZODB, etc. do "by hand" today. This could map into installers too. There is probably a way to create RPMs that contain RPMs, windows EXEs that contain windows EXEs and so forth. Paul Prescod
Paul Prescod wrote:
Andrew Kuchling wrote:
On Thu, Jan 04, 2001 at 11:49:06PM -0800, Paul Prescod wrote:
I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to
The unified ZODB distribution is really several extensions in one: ExtensionClasses, for example, are potentially useful in their own right without the rest of ZODB.
I've been thinking about this today. I'd like feedback on a proposal.
Let's start with terminology: there is a "missing term" in our vocabulary. People sometimes use the words "module", "extension" and "package" for a "distutils unit" but those words already have precise meanings in the Python world. I propose the term "distutils component". A distutils component consists of a directory which contains a setup.py and some other code. The component is also the unit of versioning.
Some components "embed" other components. The parent component is dependent on the embedded component and for convenience, the embedded component is distributed with the parent. Embedding may go away when distutils has proper dependency tracking -- but it may not.
It should be possible to create a "parent component" or "meta component" by specifying the paths to sub-components. Just as you specify "py_modules", "extensions" and so forth, you should be able to specify "sub_components". Then distutils itself would do all of the "for subcomponent in subcomponents: run_setup()" hackery that NumPy, ZODB, etc. do "by hand" today.
This could map into installers too. There is probably a way to create RPMs that contain RPMs, windows EXEs that contain windows EXEs and so forth.
Nice idea. Could you put the definitions on the distutils web page somwhere ? -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
"M.-A. Lemburg" wrote:
...
Nice idea.
Could you put the definitions on the distutils web page somwhere ?
I don't think I have write access. I could write it up as an HTML document if someone else wants to post it. Paul
Andrew Kuchling wrote:
On Thu, Jan 04, 2001 at 11:49:06PM -0800, Paul Prescod wrote:
I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to
The unified ZODB distribution is really several extensions in one: ExtensionClasses, for example, are potentially useful in their own right without the rest of ZODB. But I don't want to make people download seven different packages, so they're wrapped up in a single distribution. The true solution would be a tool like Debian's apt-get to automatically download the dependencies for a package, but that's a long way away. If you're working on a Python package manager for ActiveState and want to package the ZODB, then all the components can just be packaged separately.
I'm not sure if a similar approach would help for the mx* packages, though, and agree that some convention should be selected. Anyone got any ideas?
I am currently planning to release the mx Extensions as three packages: BASE, CRYPTO and COMMERCIAL. Each of these packages will contain a set of mx subpackages with CRYPTO and COMMERCIAL depending on BASE. The reason for me to bundle the subpackages is that of simplified maintenance -- as the number of subpackages increases, I can no longer distribute them separately. Since diskspace is cheap and bandwidth is too, I don't see much of a problem with this step. Still, I agree that it would be nice to at tell distutils to create RPM, DEB, etc. dependencies automatically. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
On Mon, Jan 08, 2001 at 12:54:39PM +0100, M.-A. Lemburg wrote:
I am currently planning to release the mx Extensions as three packages: BASE, CRYPTO and COMMERCIAL. Each of these packages will contain a set of mx subpackages with CRYPTO and COMMERCIAL depending on BASE.
But if BASE includes, say, mxDateTime, mxODBC, and mxTextTools, how would a user specify that only mxDateTime and mxODBC should be compiled? I think that's what Paul is wondering about, and I think this should be standardized to keep the process of building packages consistent. --amk
Andrew Kuchling wrote:
On Mon, Jan 08, 2001 at 12:54:39PM +0100, M.-A. Lemburg wrote:
I am currently planning to release the mx Extensions as three packages: BASE, CRYPTO and COMMERCIAL. Each of these packages will contain a set of mx subpackages with CRYPTO and COMMERCIAL depending on BASE.
But if BASE includes, say, mxDateTime, mxODBC, and mxTextTools, how would a user specify that only mxDateTime and mxODBC should be compiled?
He wouldn't... it's either all packages or none. I really don't think that standard Joe User cares about too many packages being installed (at least at the size of a few 100kB each).
I think that's what Paul is wondering about, and I think this should be standardized to keep the process of building packages consistent.
True, but I also fear that many packages will have specialized setup.py files to accomodate for their resp. needs, e.g. the autoconf machinery in distutils is still in its infancy and a more powerful library detection would also be needed to satisfy extensions built around third-party libs. Still distutils is a big step forward, since there's a single interface for the installation process: setup.py in the packages base distribution directory. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Andrew Kuchling wrote:
On Thu, Jan 04, 2001 at 11:49:06PM -0800, Paul Prescod wrote:
I could have a special case for NumPy and a special case for ZODB and a special case for ... But there are around 600 extensions on the Vaults of Parnassus and about 1800 Perl extensions on CPAN. If I'm going to
The unified ZODB distribution is really several extensions in one: ExtensionClasses, for example, are potentially useful in their own right without the rest of ZODB. But I don't want to make people download seven different packages, so they're wrapped up in a single distribution. The true solution would be a tool like Debian's apt-get to automatically download the dependencies for a package, but that's a long way away. If you're working on a Python package manager for ActiveState and want to package the ZODB, then all the components can just be packaged separately.
I have looked at the ZODB toplevel setup script, and it seems wrong to me. The behaviour you get is very confusing: Try running 'python setup.py --help', the --help option will be 'sent' to every single (sub)setup script, and you will get several screens of help. 'python setup.py bdist_dumb' however, does at least something sensible: it builds xxx.zip archives for every subpackage included in ZODB. Wouldn't it be better to take an approach like this: # create a magic dsitribution instance toplevel_dist = MagicDistribution() # collect all subpackages for package_dir in packages: os.chdir(package_dir) dist = run_setup('setup.py', stop_after='init') toplevel_dist.extend(<some magic code which combines all the properties from 'dist') os.chdir(top_dir) # run the top-level distro toplevel_dist.run() Now, 'python setup.py --help' would print the help only once, and 'python setup.py bdist_dumb' would build a single archive containing _all_ subpackages. There are (currently) some problems with this approach, but maybe they can be sorted out... (Note that I used the term 'package' above while 'distutils unit' would be the correct term nowadays) Regards, Thomas
participants (5)
-
Andrew Kuchling -
M.-A. Lemburg -
Paul F. Dubois -
Paul Prescod -
Thomas Heller