Making commands extensible by default
Hello, This is a side discussion but quiet important ihmo. == Problem == Some people complained about the fact that is was hard to extend Distutils commands. You end up rewriting the whole command most of the time. So what's a command ? It's a class that is used by the distribution instance when you run Distutils. roughly: cmd = Command(distribution) cmd.initialize_options() cmd.finalize_options() <--- allows to check the options if subcommands where run cmd.run() <--- runs the code each command can define sub commands, but most of the time it's a harcoded list, so you need to inherit the command if you want to add a new behavior. == work in progress, == What we want to do here is being able to define subsets in run(), sharing the same options environment. so basically, a rough, generic run() method could be: def run(): for func in some_funcs: func(self, options) If "some_funcs" could be defined by a registery with simple names, anyone could provide new functions and configure the registery to run a sequence of function. Given a command name, Distutils can get this list of function, through a registery. Each function could register itself into Distutils, like in what I have started to work here for the manifest file: see http://wiki.python.org/moin/Distutils/ManifestPluginSystem The ordering would be configurable through the setup.cfg file. Any opinion, idea for this part ? Cheers Tarek
Tarek Ziadé wrote:
Hello,
This is a side discussion but quiet important ihmo.
== Problem ==
Some people complained about the fact that is was hard to extend Distutils commands. You end up rewriting the whole command most of the time.
So what's a command ? It's a class that is used by the distribution instance when you run Distutils.
roughly:
cmd = Command(distribution) cmd.initialize_options() cmd.finalize_options() <--- allows to check the options if subcommands where run cmd.run() <--- runs the code
each command can define sub commands, but most of the time it's a harcoded list, so you need to inherit the command if you want to add a new behavior.
== work in progress, ==
What we want to do here is being able to define subsets in run(), sharing the same options environment.
so basically, a rough, generic run() method could be:
def run(): for func in some_funcs: func(self, options)
If "some_funcs" could be defined by a registery with simple names, anyone could provide new functions and configure the registery to run a sequence of function.
Given a command name, Distutils can get this list of function, through a registery. Each function could register itself into Distutils, like in what I have started to work here for the manifest file: see http://wiki.python.org/moin/Distutils/ManifestPluginSystem
The ordering would be configurable through the setup.cfg file.
Any opinion, idea for this part ?
Have you looked at paver? It's syntax makes extension easy. @task def run(): function1() function2() function3() or @task @needs([function1, function2, function3]) def run(): pass So if I want to do everything that setuptools.command.install does and also install locale files using my own function I do: @task def install_locales(): # My code to install locales goes here pass @task def install(): # A new install task that overrides the system install task # Note that I control ordering just by changing the order # subtasks get called call_task('setuptools.command.install') call_task('install_locales') -Toshio
Toshio Kuratomi wrote:
Tarek Ziadé wrote:
Hello,
This is a side discussion but quiet important ihmo.
== Problem ==
Some people complained about the fact that is was hard to extend Distutils commands. You end up rewriting the whole command most of the time.
So what's a command ? It's a class that is used by the distribution instance when you run Distutils.
roughly:
cmd = Command(distribution) cmd.initialize_options() cmd.finalize_options() <--- allows to check the options if subcommands where run cmd.run() <--- runs the code
each command can define sub commands, but most of the time it's a harcoded list, so you need to inherit the command if you want to add a new behavior.
== work in progress, ==
What we want to do here is being able to define subsets in run(), sharing the same options environment.
so basically, a rough, generic run() method could be:
def run(): for func in some_funcs: func(self, options)
If "some_funcs" could be defined by a registery with simple names, anyone could provide new functions and configure the registery to run a sequence of function.
Given a command name, Distutils can get this list of function, through a registery. Each function could register itself into Distutils, like in what I have started to work here for the manifest file: see http://wiki.python.org/moin/Distutils/ManifestPluginSystem
The ordering would be configurable through the setup.cfg file.
Any opinion, idea for this part ?
Have you looked at paver? It's syntax makes extension easy.
Yes, paver is nice, but I don't think it solves the problem that Tarek aimed at solving here. Let me introduce some usercases. For example, if you want to extend say the distutils sdist command, you could do: @task @needs(['distutils.command.sdist']) def mysdist(): # Add some more stuff here to the tarball ... But what if you want to alter the original sdist behavior, e.g. place some files elsewhere ? Paver works nice if you want to extend the existing behavior in a simple way (after doing this, do that), but unfortunately, some things do not fit well if at all in this scheme. I would cite two examples, related to sdist and paver: - http://groups.google.com/group/paver/browse_thread/thread/581534ca19cc541e - http://groups.google.com/group/paver/browse_thread/thread/e153ba3da8cc3334 There are also many things which are annoying once you start using paver for cross-platform deployment, because of distutils commands. For example, you may need to move some distributions files (tarball, binaries installers, doc) somewhere else. Even though the code used to generate the files are in python, there is currently no way to use that code outside distutils - things as simple as platform specific names of the tarballs, etc... are not factored out, and you have to recreate it in your own scripts. So the current problem with distutils commands is not just how to easily extend them - sometimes, you need to alter them as well, hence Tarek's suggestion for plugins/registry. cheers, David
David Cournapeau wrote:
Toshio Kuratomi wrote:
Tarek Ziadé wrote:
Hello,
This is a side discussion but quiet important ihmo.
== Problem ==
Some people complained about the fact that is was hard to extend Distutils commands. You end up rewriting the whole command most of the time.
So what's a command ? It's a class that is used by the distribution instance when you run Distutils.
roughly:
cmd = Command(distribution) cmd.initialize_options() cmd.finalize_options() <--- allows to check the options if subcommands where run cmd.run() <--- runs the code
each command can define sub commands, but most of the time it's a harcoded list, so you need to inherit the command if you want to add a new behavior.
== work in progress, ==
What we want to do here is being able to define subsets in run(), sharing the same options environment.
so basically, a rough, generic run() method could be:
def run(): for func in some_funcs: func(self, options)
If "some_funcs" could be defined by a registery with simple names, anyone could provide new functions and configure the registery to run a sequence of function.
Given a command name, Distutils can get this list of function, through a registery. Each function could register itself into Distutils, like in what I have started to work here for the manifest file: see http://wiki.python.org/moin/Distutils/ManifestPluginSystem
The ordering would be configurable through the setup.cfg file.
Any opinion, idea for this part ?
Have you looked at paver? It's syntax makes extension easy.
Yes, paver is nice, but I don't think it solves the problem that Tarek aimed at solving here. Let me introduce some usercases. For example, if you want to extend say the distutils sdist command, you could do:
@task @needs(['distutils.command.sdist']) def mysdist(): # Add some more stuff here to the tarball ...
But what if you want to alter the original sdist behavior, e.g. place some files elsewhere ? Paver works nice if you want to extend the existing behavior in a simple way (after doing this, do that), but unfortunately, some things do not fit well if at all in this scheme.
@task @needs(['distutils.command.sdist']) def sdist(): # Place some files in an alternate location
I would cite two examples, related to sdist and paver: - http://groups.google.com/group/paver/browse_thread/thread/581534ca19cc541e - http://groups.google.com/group/paver/browse_thread/thread/e153ba3da8cc3334
those are both bugs in paver. (Whether they're resolvable or not within paver, I don't know.) They don't have bearing on talking about redesigning how to design a new architecture that's easy to extend. -Toshio
Toshio Kuratomi wrote:
They don't have bearing on talking about redesigning how to design a new architecture that's easy to extend.
Those examples show why extending distutils commands with subclassing + post processing is not always enough. I don't understand why they would not be relevant for the design to improve distutils extensibility. They are quite typical of the usual problems I have when I need to extend distutils myself. David
David Cournapeau wrote:
Toshio Kuratomi wrote:
They don't have bearing on talking about redesigning how to design a new architecture that's easy to extend.
Those examples show why extending distutils commands with subclassing + post processing is not always enough. I don't understand why they would not be relevant for the design to improve distutils extensibility. They are quite typical of the usual problems I have when I need to extend distutils myself.
+1 to this argument :-) subclassing is a bad way to implement extensibility for essentially imperative tasks. I was just saying that the fact that distutils commands being used from paver having bugs does not invalidate paver's design of having functions be the task unit to build upon. -Toshio
Hi Tarek, Tarek Ziadé wrote:
== work in progress, ==
What we want to do here is being able to define subsets in run(), sharing the same options environment.
so basically, a rough, generic run() method could be:
def run(): for func in some_funcs: func(self, options)
What exactly is options here ? The class instance member user_options or something else (a dictionary of options common to each function, a bit like environment variables in scons or waf). A related problem, but maybe outside the scope of your proposal is dealing with communication between commands. I think the only way to do it at the moment is to attach those data to the Distribution instance - that's very fragile (if only because every new distutils-related tool has its own distribution class, so you have to special case for every one of them). For me, that's one of the main issue in distutils: extending distutils without breaking other tools (paver/setuptools). I think we should also working with precise examples/usecases right away - in my own experience at least, the difficulties with distutils are mostly implementation details. I will work on a few examples which I found quite painful to implement while working on the numpy build system and add them to the wiki, to help the discussion, cheers, David -- View this message in context: http://www.nabble.com/Making-commands-extensible-by-default-tp22978698p23096... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
On Fri, Apr 17, 2009 at 1:54 PM, cdavid <david@ar.media.kyoto-u.ac.jp> wrote:
A related problem, but maybe outside the scope of your proposal is dealing with communication between commands. I think the only way to do it at the moment is to attach those data to the Distribution instance - that's very fragile (if only because every new distutils-related tool has its own distribution class, so you have to special case for every one of them). For me, that's one of the main issue in distutils: extending distutils without breaking other tools (paver/setuptools).
I think we should also working with precise examples/usecases right away - in my own experience at least, the difficulties with distutils are mostly implementation details. I will work on a few examples which I found quite painful to implement while working on the numpy build system and add them to the wiki, to help the discussion,
ok great, thx. I have myself several use cases and I think I might have a working solution on the paper for the extension of existing commands. a new kind of option you can use in any command, but wich is not a simple type like a boolean or a string. It's a class that would works with plugins. class MyCmd(Command): foo = ExtensibleOption('foo') from there, the command would be able to ask in its code the option some values, by calling some plugins with a generic signature : for plugin in self.foo._get_plugins(): values = plugin(self.distribution, self) It's up to the command to define its extension points, and document them. I am doing this code in a code base for another project, when I have something running I'll get back here with it as a support for my example. ++ Tarek -- Tarek Ziadé | http://ziade.org
Here's a draft for an extensible command based on plugin; http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft Let me know how it fits your use cases. Regards Tarek -- Tarek Ziadé | http://ziade.org
participants (4)
-
cdavid -
David Cournapeau
-
Tarek Ziadé -
Toshio Kuratomi