Supporting extensibility in disutils
I'd like it to be possible to add additional commands to distutils with modifying distutils itself or the setup.py file included with packages. I don't think this needs to be a particularly risky change. The use case is to allow additional commands to be added for all distutils setup.py scripts, or to replace commands with site-specific. This could then be used to support alternate bdist_* flavors (either for additional types of packages or alternate implementations for currently supported package types). It could also be used to provide alternate implementations of existing commands to support alternate policies. Specifically, I propose that distutils should be extended to search for an implementation class across a list of packages instead of only distutils.command, keeping distutils.command as the default entry in the list. Additional packages could be identified using a command line option or configuration data. I'm willing to work up a patch for this; are there any objections to adding this functionality in Python 2.4? (I'd like to have this in the coming alpha if at all possible.) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
Fred L. Drake, Jr. wrote:
I'd like it to be possible to add additional commands to distutils with modifying distutils itself or the setup.py file included with packages. I don't think this needs to be a particularly risky change.
The use case is to allow additional commands to be added for all distutils setup.py scripts, or to replace commands with site-specific. This could then be used to support alternate bdist_* flavors (either for additional types of packages or alternate implementations for currently supported package types). It could also be used to provide alternate implementations of existing commands to support alternate policies.
Specifically, I propose that distutils should be extended to search for an implementation class across a list of packages instead of only distutils.command, keeping distutils.command as the default entry in the list. Additional packages could be identified using a command line option or configuration data.
I'm willing to work up a patch for this; are there any objections to adding this functionality in Python 2.4? (I'd like to have this in the coming alpha if at all possible.)
I'm +1 on being able to *add* new commands this way, but a strong -1 on overriding distutils own commands by doing import hackery. The reason for the latter is that tracking down errors in such setups is a complete nightmare - both for the developer and the user. How about something like a PYTHONDISTUTILSPATH + distutils.path (mimicing PYTHONPATH and sys.path for command lookups) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 02 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
At 12:39 PM 8/2/04 -0400, Fred L. Drake, Jr. wrote:
I'd like it to be possible to add additional commands to distutils with modifying distutils itself or the setup.py file included with packages. I don't think this needs to be a particularly risky change.
The use case is to allow additional commands to be added for all distutils setup.py scripts, or to replace commands with site-specific. This could then be used to support alternate bdist_* flavors (either for additional types of packages or alternate implementations for currently supported package types). It could also be used to provide alternate implementations of existing commands to support alternate policies.
Specifically, I propose that distutils should be extended to search for an implementation class across a list of packages instead of only distutils.command, keeping distutils.command as the default entry in the list. Additional packages could be identified using a command line option or configuration data.
FYI, this can be done right now with something like: import distutils.command import mycommandpackage distutils.command.__path__.extend(mycommandpackage.__path__) although that's admittedly kludgy. But it's kludgy in a backwards-compatible way. :)
On Monday 02 August 2004 01:04 pm, Phillip J. Eby wrote:
FYI, this can be done right now with something like: ... although that's admittedly kludgy. But it's kludgy in a backwards-compatible way. :)
This requires introducing more Python code, which suggests to be that the setup.py file would need to be modified. This is something I very much want to avoid; if I'm willing to change the setup.py, then there isn't much of a problem to begin with. But you're right; that's *really* kludgy. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
On Monday 02 August 2004 12:49 pm, M.-A. Lemburg wrote:
I'm +1 on being able to *add* new commands this way, but a strong -1 on overriding distutils own commands by doing import hackery. The reason for the latter is that tracking down errors in such setups is a complete nightmare - both for the developer and the user.
I'm not actually suggesting any import hackery that isn't being done already. All that's being done is loading of a class from a computed name; I'm suggesting that distutils try more than one name before failing. I don't think the package developer should need to worry about such setups at all. If they provide a setup.py that works with a vanilla Python, they've done their part. It's the responsibilty of the distutils extension developer to work with what they get.
How about something like a PYTHONDISTUTILSPATH + distutils.path (mimicing PYTHONPATH and sys.path for command lookups) ?
That would certainly work for adding commands. I'm not convinced we don't want to allow overrides, though. (No, I don't like that directories from PYTHONPATH come after the default path, or that site-packages comes after the standard library.) We could certainly point out the potential dangers of overriding stock commands in this way, but it's not clear that we need to disallow overriding either. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
Fred L. Drake, Jr. wrote:
On Monday 02 August 2004 12:49 pm, M.-A. Lemburg wrote:
I'm +1 on being able to *add* new commands this way, but a strong -1 on overriding distutils own commands by doing import hackery. The reason for the latter is that tracking down errors in such setups is a complete nightmare - both for the developer and the user.
I'm not actually suggesting any import hackery that isn't being done already. All that's being done is loading of a class from a computed name; I'm suggesting that distutils try more than one name before failing.
I don't think the package developer should need to worry about such setups at all. If they provide a setup.py that works with a vanilla Python, they've done their part. It's the responsibilty of the distutils extension developer to work with what they get.
How about something like a PYTHONDISTUTILSPATH + distutils.path (mimicing PYTHONPATH and sys.path for command lookups) ?
That would certainly work for adding commands. I'm not convinced we don't want to allow overrides, though. (No, I don't like that directories from PYTHONPATH come after the default path, or that site-packages comes after the standard library.) We could certainly point out the potential dangers of overriding stock commands in this way, but it's not clear that we need to disallow overriding either.
The PYTHONDISTUTILSPATH would allow this without having to change any code, even though it's considered dangerous. Providing distutils.path would also allow munging the path in e.g. sitecustomize.py which is probably what you have in mind. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 02 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
On Monday 02 August 2004 01:38 pm, M.-A. Lemburg wrote:
The PYTHONDISTUTILSPATH would allow this without having to change any code, even though it's considered dangerous.
Agreed; I wasn't objecting to using an environment variable for this, or to exposing some sort of distutils.path. I think what goes into the path is module names, not directory names, though, so it's a tiny bit different from sys.path in construction.
Providing distutils.path would also allow munging the path in e.g. sitecustomize.py which is probably what you have in mind.
Actually, no, I was thinking of loading a setting from the configuration file, which allows per-installation and per-user configuration. That also avoids importing distutils in sitecustomize, which I consider a good thing (the fewer things that affect Python's startup, the better!). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
I've posted a patch at: http://www.python.org/sf/1002241 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
Fred L. Drake, Jr. wrote:
Specifically, I propose that distutils should be extended to search for an implementation class across a list of packages instead of only distutils.command, keeping distutils.command as the default entry in the list. Additional packages could be identified using a command line option or configuration data.
I'm willing to work up a patch for this; are there any objections to adding this functionality in Python 2.4? (I'd like to have this in the coming alpha if at all possible.)
Fine by me. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
Fred L. Drake, Jr. wrote:
I've posted a patch at:
Nice, but does this actually implement what you had in mind ? I don't see how the sysadmin could configure the system to preset the commands package path. Hmm, he could probably do so by tweaking the global distutils.cfg. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 03 2004)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
On Tuesday 03 August 2004 04:49 am, M.-A. Lemburg wrote:
Nice, but does this actually implement what you had in mind ?
I think it does.
I don't see how the sysadmin could configure the system to preset the commands package path. Hmm, he could probably do so by tweaking the global distutils.cfg.
The sysadmin can tweak the global config file to affect all users. Individual users can tweak their personal config file or a local config file (in the directory containing setup.py), or they can give the option from the command line only when they want it to take effect. The tests make sure the option from the configuration file is handled correctly, though they only cover the local configuration file. Mucking aruond with the per-installation and per-user files seemed dangerous, and changing the per-installation file could easily fail if Python is actually installed (due to lack of permissions). In fact, the tests are somewhat fragile since they expect the per-installation and per-user files not to specify a value for this. I'll fix this fragility in the patch today. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
On Tuesday 03 August 2004 09:29 am, Fred L. Drake, Jr. wrote:
installed (due to lack of permissions). In fact, the tests are somewhat fragile since they expect the per-installation and per-user files not to specify a value for this. I'll fix this fragility in the patch today.
I've fixed up the tests and committed the patch to CVS. This feature will be included in Python 2.4a2. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
participants (4)
-
Anthony Baxter -
Fred L. Drake, Jr. -
M.-A. Lemburg -
Phillip J. Eby