Adding entry points into Distutils ?
Hello I am making a separate email for this topic to make sure no one misses it. There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils. So any strong opinion against them ? If not, I'll add a section in PEP 376 - and the first use case I can see is having a pluggable way to extend the list of files contained in .egg-info. The work to be done would be to extract entry points from pkg_resources (the discovery work that consists of looking for entry_points.txt files will be covered by the APIs already described in that PEP) This is not hard. Cheers Tarek -- Tarek Ziadé | http://ziade.org
On Tue, 5 May 2009 01:46:21 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hello
I am making a separate email for this topic to make sure no one misses it.
There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils.
So any strong opinion against them ?
but what does it provide that we don't already get with the site-packages directory? and .PTH? isn't there already a plug-in system for packages in python? David
On Mon, May 4, 2009 at 6:46 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hello
I am making a separate email for this topic to make sure no one misses it.
There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils.
So any strong opinion against them ?
Not strong, but I have a few issues with how they are currently defined: * There's the issue of activated and unactivated eggs, of course, but I guess that will be moot since there's no activation with just distutils? * I'm uncomfortable with the way entry points are scanned. I haven't looked close enough to back it up with numbers, but I think there's a noticeable performance degradation when the number of installed packages becomes large. (Given the algorithm this would be expected.) * Scanning for entry points by name makes me uncomfortable, but I feel like the API kind of encourages it. * There's no idea of explicitly enabling an entry point, simply installing a package makes the entry point show up. Implicit plugins make me uncomfortable. * There's not much in the way of entry point documentation built into the system. The __doc__ of the entry point objects is one way, but there's no way to document entry point groups. * Entry points that provide functionality to the installation system itself make me very uncomfortable, except insofar as they describe the package. [console_scripts] is okay, but some of the other Setuptools extension points bother me because of the self-referential nature, e.g., egg_file_writers. -- Ian Bicking | http://blog.ianbicking.org
On Tue, May 5, 2009 at 1:57 AM, Ian Bicking <ianb@colorstudy.com> wrote:
Not strong, but I have a few issues with how they are currently defined:
* There's the issue of activated and unactivated eggs, of course, but I guess that will be moot since there's no activation with just distutils?
Yes
* There's no idea of explicitly enabling an entry point, simply installing a package makes the entry point show up. Implicit plugins make me uncomfortable.
I don't see entry points as plugins, but rather the registering of a given piece of code, under a unique name. If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ? The way I see entry points is "potential" plugins, an application can decide to consume, and turn into a real plugin when it uses it. And an entry point that would be "disabled" is an entry point that is not used from the application A point of view, but might be used in the application B. (unless I misunderstood the concept of "group") So enabling/disabling an entry point and keeping track of the activation state should be done by the host application ihmo.
* There's not much in the way of entry point documentation built into the system. The __doc__ of the entry point objects is one way, but there's no way to document entry point groups.
very good point, -- Tarek Ziadé | http://ziade.org
On May 5, 2009, at 4:49 AM, Tarek Ziadé wrote:
On Tue, May 5, 2009 at 1:57 AM, Ian Bicking <ianb@colorstudy.com> wrote:
Not strong, but I have a few issues with how they are currently defined:
* There's the issue of activated and unactivated eggs, of course, but I guess that will be moot since there's no activation with just distutils?
Yes
* There's no idea of explicitly enabling an entry point, simply installing a package makes the entry point show up. Implicit plugins make me uncomfortable.
I don't see entry points as plugins, but rather the registering of a given piece of code, under a unique name.
I don't understand that. I thought the purpose of entry points was to register code such as plugins so that applications didn't have to be manually configured. I think I'm with Ian on that one: Explicit is better than implicit. If I have to "turn on" the plugin, then what benefit does an entry point registry give me? I could just as easily provide that information to the application directly.
If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ?
The user who wants the application to consume the plugin.
The way I see entry points is "potential" plugins, an application can decide to consume, and turn into a real plugin when it uses it.
And an entry point that would be "disabled" is an entry point that is not used from the application A point of view, but might be used in the application B.
But if it's not being used by A, why should A see it at all? Doug
On Tue, May 5, 2009 at 1:57 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote:
If I have to "turn on" the plugin, then what benefit does an entry point registry give me?
I don't understand this sentence, since you say later that you want the user to manually turn a plugin "on" for an application to soncume it.
If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ?
The user who wants the application to consume the plugin.
I am confused with the role of this "man in the middle". In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side. Do you have a use case we can share for mutual comprehension ?
And an entry point that would be "disabled" is an entry point that is not used from the application A point of view, but might be used in the application B.
But if it's not being used by A, why should A see it at all?
A, B or any app can browse all entry points. Entry points are defined by the (group, name) couple. Basically if I want to create a pluggable feature called "myfeature", the group will be "myfeature" and plugins that implement that feature will register themselves under that group. Then my application will browse and consume entry points for the group "myfeature" and do whatever they want with them. In pseudo code:
plugins = iter_entry_points(group="myfeature")
So if A doesn't need the plugins that are under the group "myfeature", it will just ignore the entry points that are in this group. e.g. ignore the group. Maybe A will consume entry_points that are under another group. But I have never browsed *all* entry points from an application. I think the best practice for entry points is to use the most explicit group names possible, but having plugins that can be consumed by several applications is a win ihmo. For instance, if I need to write a specific extensible installation script, I'll probably see if I can consume zc.buildout recipes through the "zc.buildout.recipe" group.
Doug
-- Tarek Ziadé | http://ziade.org
On May 5, 2009, at 8:15 AM, Tarek Ziadé wrote:
On Tue, May 5, 2009 at 1:57 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote:
If I have to "turn on" the plugin, then what benefit does an entry point registry give me?
I don't understand this sentence, since you say later that you want the user to manually turn a plugin "on" for an application to soncume it.
I want each application to be configured separately, rather than having a global registry of plugins. So having a "plugin registry" *library* in stdlib may make sense (so that apps can build their registry databases in a consistent way), but automatically registering entry points just because a package is installed does not.
If you add explicit enabling, who will do it ? the package that has the entry point ? The applications that consumes them ?
The user who wants the application to consume the plugin.
I am confused with the role of this "man in the middle". In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side.
Do you have a use case we can share for mutual comprehension ?
I think we have a different view about how the plugins should work. It sounds like you're advocating a model where all plugins are registered globally, an application can search the global registry for plugins based on categories, and then some administrator enables/ disables them locally for each app. I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort.
So if A doesn't need the plugins that are under the group "myfeature", it will just ignore the entry points that are in this group. e.g. ignore the group.
Maybe A will consume entry_points that are under another group. But I have never browsed *all* entry points from an application.
That depends on how the database of entry points is maintained, but I'll grant that point.
I think the best practice for entry points is to use the most explicit group names possible, but having plugins that can be consumed by several applications is a win ihmo.
For instance, if I need to write a specific extensible installation script, I'll probably see if I can consume zc.buildout recipes through the "zc.buildout.recipe" group.
Doug
-- Tarek Ziadé | http://ziade.org
On Tue, May 5, 2009 at 2:41 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote:
I am confused with the role of this "man in the middle". In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side.
Do you have a use case we can share for mutual comprehension ?
I think we have a different view about how the plugins should work. It sounds like you're advocating a model where all plugins are registered globally, an application can search the global registry for plugins based on categories, and then some administrator enables/disables them locally for each app.
I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort.
That is basically how host applications are dealing with entry points: an explicit configuration. The implicit part is just happening when you look for the plugins. Let's take a real example : I am working on a program called Atomisator which can be extended through plugins. For example people can provide a plugin to read data that are located somewhere (like a rss feed), and return a sequence of entries. So in atomisator, I have a configuration file where I decide which plugin to use to read my data: """ [atomisator] reader = rss """ When reading it, Atomisator knows that it needs the "rss" plugin, and will browse in entry points with a specific group called "atomisator.rss". If it finds it, it uses it, otherwise it throws an exception. ("Install it!") That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the "rss" plugins, so my Atomisatior application can discover then load it at run time. So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. "put your package in this /plugins directory" for example. I am advocating that the entry point mechanism is handy because it relies on an existing mechanism : "install your package" and it allows plugin sharing amongst applications. But I see the caveats you are explaining, and I understand them now; So, what if we didn't have these entry points installed globally when the package is installed, but just a configuration file in the host application level to point them ? a configuration file that reunites all entry points an application uses. For the Atomisator example: [atomisator.reader] rss = somepackage.somemodule:MyPluginClass This would let the application consume the plugins pointed by this configuration file and this would remove the implicit part you don't like. I'd be very happy with such a plugin system on my side. And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg Tarek -- Tarek Ziadé | http://ziade.org
hi,
That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the "rss" plugins, so my Atomisatior application can discover then load it at run time.
So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. "put your package in this /plugins directory" for example.
wouldn't this be tackled by providing a plugin-discoverer plugin ? kind of like the new import hooks of pep-302 [1] where you have module finders separated from module loaders, or like this library I have been using in my field [2] cheers, sebastien. [1] http://www.python.org/dev/peps/pep-0302/ [2] http://cdsweb.cern.ch/record/865639?ln=pl http://wlav.web.cern.ch/wlav/pybus/index.html http://seal.cvs.cern.ch:80/cgi-bin/seal.cgi/seal/Scripting/PyBus -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay #########################################
2009/5/5 Sebastien Binet <seb.binet@gmail.com>:
hi,
That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the "rss" plugins, so my Atomisatior application can discover then load it at run time.
So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. "put your package in this /plugins directory" for example.
wouldn't this be tackled by providing a plugin-discoverer plugin ? kind of like the new import hooks of pep-302 [1] where you have module finders separated from module loaders, or like this library I have been using in my field [2]
That would be better to use these hooks rather than __import__ I believe, but I don't think it changes the problem (eg locate the plugin)
On May 5, 2009, at 9:33 AM, Tarek Ziadé wrote:
On Tue, May 5, 2009 at 2:41 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote:
I am confused with the role of this "man in the middle". In my mind there are plugins on one side, and host applications that consumes them if they wish on they other side.
Do you have a use case we can share for mutual comprehension ?
I think we have a different view about how the plugins should work. It sounds like you're advocating a model where all plugins are registered globally, an application can search the global registry for plugins based on categories, and then some administrator enables/disables them locally for each app.
I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort.
That is basically how host applications are dealing with entry points: an explicit configuration. The implicit part is just happening when you look for the plugins.
Let's take a real example : I am working on a program called Atomisator which can be extended through plugins.
For example people can provide a plugin to read data that are located somewhere (like a rss feed), and return a sequence of entries.
So in atomisator, I have a configuration file where I decide which plugin to use to read my data:
""" [atomisator]
reader = rss """
When reading it, Atomisator knows that it needs the "rss" plugin, and will browse in entry points with a specific group called "atomisator.rss". If it finds it, it uses it, otherwise it throws an exception. ("Install it!")
That allows people to create their own plugins in separate packages, and use them by tweaking the configuration file. The only think that entry point provided here is an automatic registration at installation time of the "rss" plugins, so my Atomisatior application can discover then load it at run time.
I don't think I understand the difference between the step you're calling "discover", scanning the registry, and actually loading the plugin. Does "discovering" the plugin involve importing any of its code?
So in your way of seeing thing, you'd rather see this registration mechanism at the application level, but the you need to provide specific installation instructions for people that want to add plugins. e.g. "put your package in this /plugins directory" for example.
Sort of. I see the installation and configuration of entry points as 2 steps: 1. some variation of "python setup.py install" 2. update the configuration of app to tell it where to find the plugin Some applications may provide automated tools for step 2, and I could see great benefit to making the loading and registration of entry points part of the standard library, as long as the registration is maintained per-application instead of site-wide.
I am advocating that the entry point mechanism is handy because it relies on an existing mechanism : "install your package" and it allows plugin sharing amongst applications.
How then do I install a package but *not* have it available for an application? Suppose, for example, that I have several copies of a web app installed for different customers, but I want to prevent some of them from using an advanced plugin of some sort.
But I see the caveats you are explaining, and I understand them now;
So, what if we didn't have these entry points installed globally when the package is installed, but just a configuration file in the host application level to point them ?
a configuration file that reunites all entry points an application uses. For the Atomisator example:
[atomisator.reader] rss = somepackage.somemodule:MyPluginClass
Yes! We can figure out the exact spelling, but we're talking about the same thing now. If we use dotted notation all the way ("somepackage.somemodule.MyPluginClass") then it's simple to just import the thing directly.
This would let the application consume the plugins pointed by this configuration file and this would remove the implicit part you don't like. I'd be very happy with such a plugin system on my side.
If you follow the model of the logging module and provide an easy "load plugins from an ini file", then all python apps have the feature of making it easy to load plugins, developers have a standard API to code to, and administrators don't have to worry about plugins "sneaking" into an app.
And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg
That sounds good. Doug
On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote
I don't think I understand the difference between the step you're calling "discover", scanning the registry, and actually loading the plugin. Does "discovering" the plugin involve importing any of its code?
No, like Phillip said somewhere in the thread. The discovery pseudo-code is: entry_points = [] for path in paths: if path is egg-info: entry_points.append(load('entry_points.txt')) Then each entry point is just a string that "locates" the plugins ("package.modul.class" for example) The real import occurs only explicitely when you do "entry_point.load()"
a configuration file that reunites all entry points an application uses. For the Atomisator example:
[atomisator.reader] rss = somepackage.somemodule:MyPluginClass
Yes! We can figure out the exact spelling, but we're talking about the same thing now. If we use dotted notation all the way ("somepackage.somemodule.MyPluginClass") then it's simple to just import the thing directly.
I think is simpler with the "somepackage.somemodule:MyPluginClass" notation This is how setuptools does roughly:
parts = "somepackage.somemodule:MyPluginClass".split(':') module = __import__(parts[0]) plugin = getattr(module, parts[1])
And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg
That sounds good.
The only caveat I see though, is that the host app has to know the exact location of each plugin in the code of the third party app, whereas entry points provide this information through the discovery API. I could leave with both notation I believe: location = "somepackage.somemodule:MyPluginClass" *or* location = iter_entry_points('atomisator.reader', 'rss') -- Tarek Ziadé | http://ziade.org
On May 5, 2009, at 11:05 AM, Tarek Ziadé wrote:
On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote
a configuration file that reunites all entry points an application uses. For the Atomisator example:
[atomisator.reader] rss = somepackage.somemodule:MyPluginClass
Yes! We can figure out the exact spelling, but we're talking about the same thing now. If we use dotted notation all the way ("somepackage.somemodule.MyPluginClass") then it's simple to just import the thing directly.
I think is simpler with the "somepackage.somemodule:MyPluginClass" notation
Good point, I was remembering the import syntax incorrectly.
And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg
That sounds good.
The only caveat I see though, is that the host app has to know the exact location of each plugin in the code of the third party app, whereas entry points provide this information through the discovery API.
True. On the other hand, that encourages standard locations like "mylib.yourapp.entry_point" and "mylib.anotherapp.entry_point". I think most of my concerns about the global registry are taken care of by the fact that "discovering" a plugin doesn't involve any imports of unknown code. I still prefer per-app, explicit configuration of entry points, and think we could build a system to support that. I would like to see *some* variation of this in the standard library, though, because I have several uses for it. Doug
On Tue, May 05, 2009 at 05:05:28PM +0200, Tarek Ziadé wrote:
On Tue, May 5, 2009 at 4:29 PM, Doug Hellmann <doug.hellmann@gmail.com> wrote
a configuration file that reunites all entry points an application uses. For the Atomisator example:
[atomisator.reader] rss = somepackage.somemodule:MyPluginClass [...] And this would fit I think in Distutils needs since we can configure it through three levels of configuration files distutils.cfg, pydistutils.cfg and setup.cfg
That sounds good.
So there would be a configuration file for each application that needs it? I like this a lot more then the global entry-point registry too (it avoids name collisions for entry points too). But how can a "python setup.py install" know where to find this configuration file to add it's plugin? Or should this be an explicit manual step? It might be nice to have a --register-plugins option to the install command though if possible. Something else to keep into account is the FHS, I can imagine GNU/Linux distributions would want to place a configuration file somewhere else, like in /etc/PROJECT.conf instead of /usr/lib/pythonX.Y/site-packages/PROJECT.egg-info/plugin_registry (pathnames are fairly random examples). The only thing I can think of is somehow having a file in .egg-info telling you where the plugin configuration file is, just like was proposed for all types of data files earlier on this list. But I think by now I'm taking this too far and the install command of distutils should not be able to register plugins for random projects automatically. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 09:27 PM 5/5/2009 +0100, Floris Bruynooghe wrote:
But how can a "python setup.py install" know where to find this configuration file to add it's plugin?
It doesn't. The whole point of having two stages -- discovery and activation -- is that discovery is an automatic side-effect of installation, whereas activation can be controlled by an application-specific policy or configuration file. Entry points (as currently implemented) are discovery; the application then determines which entry points to actually load or invoke, using its own configuration (or command-line options, or menu selections, or automatic policy, or whatever else).
Tarek Ziadé <ziade.tarek@gmail.com> writes:
I think is simpler with the "somepackage.somemodule:MyPluginClass" notation
This is how setuptools does roughly:
parts = "somepackage.somemodule:MyPluginClass".split(':') […]
Using the standard import notation is no more difficult: >>> parts = "somepackage.somemodule.MyPluginClass".rsplit('.', 1) >>> parts ['somepackage.somemodule', 'MyPluginClass'] versus: >>> parts = "somepackage.somemodule:MyPluginClass".split(':', 1) >>> parts ['somepackage.somemodule', 'MyPluginClass'] I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class. -- \ “The lift is being fixed for the day. During that time we | `\ regret that you will be unbearable.” —hotel, Bucharest | _o__) | Ben Finney
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects? Doug
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects?
I use it; I'm not sure who else does. The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience.
On May 6, 2009, at 1:46 PM, P.J. Eby wrote:
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects?
I use it; I'm not sure who else does.
The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience.
That's pretty much what I expected. It feels a little messy to have plugins exposing "internals" like that but not so much so that I propose we don't allow it. The ":" syntax seems like the right way to go.
Doug Hellmann wrote:
On May 6, 2009, at 1:46 PM, P.J. Eby wrote:
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects?
I use it; I'm not sure who else does.
The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience.
That's pretty much what I expected. It feels a little messy to have plugins exposing "internals" like that but not so much so that I propose we don't allow it. The ":" syntax seems like the right way to go.
I'd be tempted to call this an edge-case. You should be able to expose the internal detail you'd need via a module scope alias for the particular case. That seems easier than providing a whole new notion. Hanno
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hanno Schlichting wrote:
Doug Hellmann wrote:
On May 6, 2009, at 1:46 PM, P.J. Eby wrote:
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class. Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote: parts are attributes within that module. Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects? I use it; I'm not sure who else does.
The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience. That's pretty much what I expected. It feels a little messy to have plugins exposing "internals" like that but not so much so that I propose we don't allow it. The ":" syntax seems like the right way to go.
I'd be tempted to call this an edge-case. You should be able to expose the internal detail you'd need via a module scope alias for the particular case. That seems easier than providing a whole new notion.
I'm actually a big fan of the ':', because it makes explicit the difference between the "import" and the "named thing", even for module-scoped names. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAde9+gerLs4ltQ4RAgzJAJ0RgbDdXFFW/O/mcK3u6BCKOiBW3QCfUXs3 S0lgBewN4w5PqIHBilft29Y= =G7sv -----END PGP SIGNATURE-----
On Wed, May 6, 2009 at 1:32 PM, Tres Seaver <tseaver@palladion.com> wrote:
I'd be tempted to call this an edge-case. You should be able to expose the internal detail you'd need via a module scope alias for the particular case. That seems easier than providing a whole new notion.
I'm actually a big fan of the ':', because it makes explicit the difference between the "import" and the "named thing", even for module-scoped names.
Yeah, I also like it primarily for clarity. Also you can provide better error messages when it fails, matching up the error against the intention. In some contexts I also extend this syntax to do something like: module, path = name.split(':', 1) obj = eval(path, load_module(module).__dict__) I don't propose that we do eval for plugins, but it's a nice side-effect of the syntax that it's possible to add in other contexts. Also I don't think there's any strong precedence for purely dot notation, loading objects from strings is something that's always done ad hoc, and the only widely used library I know of that people use for this is Setuptools (indirectly through entry points). -- Ian Bicking | http://blog.ianbicking.org
Ian Bicking wrote:
On Wed, May 6, 2009 at 1:32 PM, Tres Seaver <tseaver@palladion.com <mailto:tseaver@palladion.com>> wrote:
I'm actually a big fan of the ':', because it makes explicit the difference between the "import" and the "named thing", even for module-scoped names.
Yeah, I also like it primarily for clarity. Also you can provide better error messages when it fails, matching up the error against the intention.
[...]
Also I don't think there's any strong precedence for purely dot notation, loading objects from strings is something that's always done ad hoc, and the only widely used library I know of that people use for this is Setuptools (indirectly through entry points).
The one I and others use in quite many places is zope.dottedname [1] (the actual function is at [2]). This one doesn't make any difference between modules and attributes. It has no zope dependency but happens to be used throughout the entire Zope / Plone stack. Hanno [1] http://pypi.python.org/pypi/zope.dottedname [2] http://svn.zope.org/zope.dottedname/trunk/src/zope/dottedname/resolve.py?vie...
At 08:28 PM 5/6/2009 +0200, Hanno Schlichting wrote:
Doug Hellmann wrote:
On May 6, 2009, at 1:46 PM, P.J. Eby wrote:
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote:
I don't see any advantage, in the context of this discussion, to having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects?
I use it; I'm not sure who else does.
The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience.
That's pretty much what I expected. It feels a little messy to have plugins exposing "internals" like that but not so much so that I propose we don't allow it. The ":" syntax seems like the right way to go.
I'd be tempted to call this an edge-case. You should be able to expose the internal detail you'd need via a module scope alias for the particular case. That seems easier than providing a whole new notion.
Ah, now you reminded me of a forgotten detail: the specific use case was that I had a library with a Command class that was widely used as a base class; simply adding a classmethod to that base class allowed all the existing classes to be used as plugins without any change to their modules' source code: the entry points just always reference the classmethod.
On Thu, May 7, 2009 at 7:18 AM, P.J. Eby <pje@telecommunity.com> wrote:
At 08:28 PM 5/6/2009 +0200, Hanno Schlichting wrote:
Doug Hellmann wrote:
On May 6, 2009, at 1:46 PM, P.J. Eby wrote:
At 10:59 AM 5/6/2009 -0400, Doug Hellmann wrote:
On May 5, 2009, at 10:50 PM, P.J. Eby wrote:
At 12:03 PM 5/6/2009 +1000, Ben Finney wrote: > I don't see any advantage, in the context of this discussion, to > having an additional, incompatible naming for full-path-to-a-class.
Setuptools doesn't limit an entry point to being a class, function, or other top-level name within a module. It can be a method of a class, or an attribute of an attribute. The ':' removes any ambiguity as to which part of the name is the module, and which parts are attributes within that module.
Is that level of complexity useful in practice? I can understand how it came to be implemented, but is it actually used by any projects?
I use it; I'm not sure who else does.
The particular use case I have (and that's most likely to be shared) is that the calling app or framework wants a callable or function, but the providing app or library implements that callable as a classmethod for convenience.
That's pretty much what I expected. It feels a little messy to have plugins exposing "internals" like that but not so much so that I propose we don't allow it. The ":" syntax seems like the right way to go.
I'd be tempted to call this an edge-case. You should be able to expose the internal detail you'd need via a module scope alias for the particular case. That seems easier than providing a whole new notion.
Ah, now you reminded me of a forgotten detail: the specific use case was that I had a library with a Command class that was widely used as a base class; simply adding a classmethod to that base class allowed all the existing classes to be used as plugins without any change to their modules' source code: the entry points just always reference the classmethod.
[slightly off topic] It seems like registration is the canonical use case for classmethods in Python no? Tarek has a good example in Expert Python Programming of this in his observer example. [/end]
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Cheers, Noah
At 08:41 AM 5/5/2009 -0400, Doug Hellmann wrote:
I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort.
Note that this is not incompatible with entry points; an application can simply treat entry points as a list of *available* plugins, rather than as a list of *active* plugins. Chandler, for example, does this for user-level plugins.
On May 5, 2009, at 10:38 AM, P.J. Eby wrote:
At 08:41 AM 5/5/2009 -0400, Doug Hellmann wrote:
I don't want new functionality available to an application just because someone has permission to install a package somewhere in the PYTHONPATH. I would rather have plugins added to an app through an explicit configuration step of some sort.
Note that this is not incompatible with entry points; an application can simply treat entry points as a list of *available* plugins, rather than as a list of *active* plugins. Chandler, for example, does this for user-level plugins.
That's good. I wasn't sure if registering an entry point caused it to be automatically loaded in an app that expressed interest in it, or if there was a step the app could take to verify that it was desirable code before doing any imports. Earlier messages in this thread made me think there was no "protection" from a registered plugin.
Tarek Ziadé <ziade.tarek@gmail.com> writes:
There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils.
I don't see what those advantages are. From what I can see, commands still need to be implemented in separate programs in order to be used as commands.
So any strong opinion against them ?
Until I understand what they do, I don't know what objections I might have :-) -- \ “Only the educated are free.” —Epictetus | `\ | _o__) | Ben Finney
There are *many* benefits of adding entry points into Distutils. In fact, adding a plugin system in there, will help make the commands more extendable and therefore will help us in the long term to remove things out of Distutils.
So any strong opinion against them ?
If not, I'll add a section in PEP 376 - and the first use case I can see is having a pluggable way to extend the list of files contained in .egg-info.
The work to be done would be to extract entry points from pkg_resources (the discovery work that consists of looking for entry_points.txt files will be covered by the APIs already described in that PEP) This is not hard.
Yes, I'd like to see entry_points added to Distutils! I'll also mention my most common use-case for using entry_points is installing console_scripts using zc.recipe.egg. This use-case only needs discovery of entry_points. Once a distribution's entry_points are made available, it's entirely up to the application installer (eg. zc.recipe.egg) as to what is done with the entry_point information. For example, let's say that I have a Python project which I want to use zest.releaser for. The entry_points for this package is: entry_points={ 'console_scripts': [ 'release = zest.releaser.release:main', 'prerelease = zest.releaser.prerelease:main', 'postrelease = zest.releaser.postrelease:main', 'fullrelease = zest.releaser.fullrelease:main', 'longtest = zest.releaser.longtest:main', 'lasttagdiff = zest.releaser.lasttagdiff:main'], } Then I can use buildout to install this package into a project's deployment with: [buildout] parts = releaser [releaser] recipe = zc.recipe.egg eggs = zest.releaser Where the zc.recipe.egg recipe will turn all 'console_scripts' entry_points into scripts in the ./bin directory. Which I find pretty nice :) Anyways, I'm mentioning this use-case just to re-iterate that there is a difference between asking a distribution what entry_points it provides (discovery) and deciding what configuration or actions to take based on that information (activation or script generation). Also, as for discovery, this use-case only requires discovery on a per-distribution basis, it doesn't need a 'iterate over all distribution's entry_points installed in some location' feature. Also, using console_scripts for entry_points means that there is a second way of specifying scripts, since Distutils already has the 'scripts' metadata field. I would say that the 'scripts' field should then be deprecated, but perhaps there are reason's beyond breaking backwards compatability for keeping that field around?
On May 6, 2009, at 9:38 PM, Wheat wrote:
I'll also mention my most common use-case for using entry_points is installing console_scripts using zc.recipe.egg.
I'm curious about that because I've never understood the benefit of using entry points for console scripts. Why not just list the script as a part of the package and have it installed "normally"?
Also, using console_scripts for entry_points means that there is a second way of specifying scripts, since Distutils already has the 'scripts' metadata field. I would say that the 'scripts' field should then be deprecated, but perhaps there are reason's beyond breaking backwards compatability for keeping that field around?
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is? Doug
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is?
Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for scripts is that it generates appropriate wrappers to allow use on all platforms. Having said that, I find setuptools entry points to be over-engineered, and the Windows wrappers (in particular, the fact that they are not version-independent) to be somewhat clumsy. But as a concept, I like the idea of having a way of specifying that a script is intended as an "executable", and having distutils do the job of generating whatever platform cruft is required [1] to make that work. Of course, for even remotely modern Python versions, I'd argue strongly that packages shouldn't be including console scripts, but should rather be supplying modules that can be run as scripts, via the -m argument to python. Users can then build aliases, shell scripts, or whatever is appropriate based on that. Paul. [1] And note especially that .bat files are *not* suitable wrappers on Windows, in spite of the fact that they are commonly used. Their biggest disadvantage is that they don't nest.
On May 7, 2009, at 8:54 AM, Paul Moore wrote:
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is?
Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for scripts is that it generates appropriate wrappers to allow use on all platforms.
I write a python script call hello.py like this: #!/usr/bin/env python def main(): print 'hello!' if __name__ == '__main__': main() Why make me define an entry point for that? I can just put it in /usr/ bin or somewhere in the path on Windows and call it as "hello.py". Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
Having said that, I find setuptools entry points to be over-engineered, and the Windows wrappers (in particular, the fact that they are not version-independent) to be somewhat clumsy. But as a concept, I like the idea of having a way of specifying that a script is intended as an "executable", and having distutils do the job of generating whatever platform cruft is required [1] to make that work.
That's the part I'm not clear about. What "cruft" is needed anywhere?
Of course, for even remotely modern Python versions, I'd argue strongly that packages shouldn't be including console scripts, but should rather be supplying modules that can be run as scripts, via the -m argument to python. Users can then build aliases, shell scripts, or whatever is appropriate based on that.
Paul.
[1] And note especially that .bat files are *not* suitable wrappers on Windows, in spite of the fact that they are commonly used. Their biggest disadvantage is that they don't nest.
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
That works but a lot of Unix users have in the past objected to having '.py' in the name. People then start trying to cater for them by leaving off the '.py', which means it doesn't work on Windows, etc etc. As a Windows user, I'm all in favour of this. But can you persuade the Unix users to agree with me, please? :-)
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
I don't think so, as such. It gives Unix and Windows users who care (== not me, and clearly not you, either) the ability to call the command "hello" rather than "hello.py". Paul.
On Thu, May 07, 2009 at 03:03:29PM +0100, Paul Moore wrote:
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
That works but a lot of Unix users have in the past objected to having '.py' in the name. People then start trying to cater for them by leaving off the '.py', which means it doesn't work on Windows, etc etc.
Why can't the install_scripts command from distutils install with a .py extension on windonws and without on UNIX? (Or rather: what would be wrong with such an approach?) Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Thu, 7 May 2009 15:12:15 +0100, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
Why can't the install_scripts command from distutils install with a .py extension on windonws and without on UNIX? (Or rather: what would be wrong with such an approach?)
Perphaps it is too much of a good idea.... or perphaps it would make things too easy for windows and unix users... David
On May 7, 2009, at 10:03 AM, Paul Moore wrote:
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
I don't think so, as such. It gives Unix and Windows users who care (== not me, and clearly not you, either) the ability to call the command "hello" rather than "hello.py".
Hmm, yeah, I see that as useful. I don't work with Windows a lot myself, but I don't want to explicitly drop support there, either. Doug
Doug Hellmann wrote:
On May 7, 2009, at 8:54 AM, Paul Moore wrote:
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is?
Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for scripts is that it generates appropriate wrappers to allow use on all platforms.
I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python.
[1] And note especially that .bat files are *not* suitable wrappers on Windows, in spite of the fact that they are commonly used. Their biggest disadvantage is that they don't nest.
Which is why it creates .exe wrappers, not batch files. [1] http://peak.telecommunity.com/DevCenter/EasyInstall
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Eric Smith wrote:
Doug Hellmann wrote:
On May 7, 2009, at 8:54 AM, Paul Moore wrote:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is? Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for scripts is that it generates appropriate wrappers to allow use on all
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>: platforms. I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python.
Exactly: using entry points for console scripts guarantees that the python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAu4++gerLs4ltQ4RAjW4AJ0cM5oqDL35qoBq1nnl4YD1RmX1NACghbqM Zom9yb5WCzaXFrMz7V75zQU= =CZLG -----END PGP SIGNATURE-----
2009/5/7 Tres Seaver <tseaver@palladion.com>:
Eric Smith wrote:
Doug Hellmann wrote:
On May 7, 2009, at 8:54 AM, Paul Moore wrote:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is? Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for scripts is that it generates appropriate wrappers to allow use on all
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>: platforms. I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python.
Exactly: using entry points for console scripts guarantees that the python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess.
... and somewhere around here we end up with what I described as an over-engineered solution. By trying to satisfy everyone's requirements, you ultimately satisfy no-one's. Sigh. I keep meaning to avoid getting sucked back into this tar-pit, and I keep failing :-( Just put me down as a hearty +1 for Doug's "just deploy a script called whatever.py" approach for standalone stuff, and using python -m for scripts distributed as part of larger distributions. Paul.
Paul Moore wrote:
2009/5/7 Tres Seaver <tseaver@palladion.com>:
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python. Exactly: using entry points for console scripts guarantees that the
Eric Smith wrote: python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess.
... and somewhere around here we end up with what I described as an over-engineered solution.
Or, what I call a minimal set of required functionality.
By trying to satisfy everyone's requirements, you ultimately satisfy no-one's.
Not sure I agree, but in any event that's certainly not a reason to try and make a generally useful solution. I think entry points and scripts are two issues that shouldn't be conflated. Entry points are generally useful (to me and others), and having a way of the installer knowing about scripts and generating correct wrappers for them (in a cross-platform way) is also generally useful (to me and others). That scripts currently use entry points is a design detail that perhaps doesn't need to be exposed. Eric.
On Fri, May 8, 2009 at 10:04 AM, Eric Smith <eric@trueblade.com> wrote:
Paul Moore wrote:
2009/5/7 Tres Seaver <tseaver@palladion.com>:
Eric Smith wrote:
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python.
Exactly: using entry points for console scripts guarantees that the python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess.
... and somewhere around here we end up with what I described as an over-engineered solution.
Or, what I call a minimal set of required functionality.
By trying to satisfy everyone's requirements, you ultimately satisfy no-one's.
Not sure I agree, but in any event that's certainly not a reason to try and make a generally useful solution.
I think entry points and scripts are two issues that shouldn't be conflated. Entry points are generally useful (to me and others), and having a way of the installer knowing about scripts and generating correct wrappers for them (in a cross-platform way) is also generally useful (to me and others).
Two current problems I have with console scripts with setuptools are: 1. Different versions of Python conflict with previous versions of console scripts. Take paste for example. 2. The entry point mechanism IIRC recursively scans the site-packages directory and loads up the system path with eggs. This is too expensive of an operation for the current environment I work in. 3. There doesn't seem to be a clean way to inject user specific environment details to the console script. I often need the ability to alter the sys.path in a user specific way for the entry point without needing to mess up the global sys.path permanently.
scripts currently use entry points is a design detail that perhaps doesn't need to be exposed.
Eric.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Cheers, Noah
Noah Gift wrote:
3. There doesn't seem to be a clean way to inject user specific environment details to the console script.
os.environ? Either than, or I'm not following you... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Eric Smith wrote:
Paul Moore wrote:
2009/5/7 Tres Seaver <tseaver@palladion.com>:
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python. Exactly: using entry points for console scripts guarantees that the
Eric Smith wrote: python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess.
... and somewhere around here we end up with what I described as an over-engineered solution.
Or, what I call a minimal set of required functionality.
+1. <flame>Anyone who's arguing against this is either not deploying stuff in a repeatable fashion, and so isn't serious in my books, or is so serious that they're cutting vm images to roll out and so dump everything for the app in site packages</flame> Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
2009/5/9 Chris Withers <chris@simplistix.co.uk>:
+1.
<flame>Anyone who's arguing against this is either not deploying stuff in a repeatable fashion, and so isn't serious in my books, or is so serious that they're cutting vm images to roll out and so dump everything for the app in site packages</flame>
Hmm. I'll accept the flame to the extent that I don't deploy stuff to a wide enough audience to be qualified to comment on that side of things. But: 1. Can you clarify what "this" is? Part of the issue I see is that there's never a clear enough statement of a proposal for a non-expert in the field to follow. 2. Distutils is for distributing python modules/packages, so *application* deployment is out of scope. Script support is for small-scale stuff (in my view). The fact that it gets used for more doesn't mean it's appropriate... 3. Accepting that I don't know what you mean by "this", can I point out that as a user, I personally have problems with a significant proportion of scripts distributed with packages - so are you saying that those packages "aren't serious", or that there is no way of doing what they (and I) want at present? Concrete examples of specific packages would probably help. Right this moment, I can't personally provide any because I recently had to rebuild my PC and haven't yet reinstalled the vast number of tried-once-but-never-used-again packages I used to have available... I'll see if I can find some in due course. (Basically, the types of things I see are scripts distributed on Windows with no filetype extension, .bat wrappers for command line scripts which don't work right when called from another bat file, setuptools-built exe wrappers which result in version-specific binaries for pure-python code, to give some examples). Paul.
Paul Moore wrote:
2009/5/9 Chris Withers <chris@simplistix.co.uk>:
+1.
<flame>Anyone who's arguing against this is either not deploying stuff in a repeatable fashion, and so isn't serious in my books, or is so serious that they're cutting vm images to roll out and so dump everything for the app in site packages</flame>
Hmm. I'll accept the flame to the extent that I don't deploy stuff to a wide enough audience to be qualified to comment on that side of things. But:
1. Can you clarify what "this" is?
Given that I'm writing this offline on a train, the answer is "no", I'm afraid ;-)
Part of the issue I see is that there's never a clear enough statement of a proposal for a non-expert in the field to follow.
Which proposal do you think isn't clear enough?
2. Distutils is for distributing python modules/packages, so *application* deployment is out of scope.
Python applications rely on packages and modules. Application in the scope I'm using it might mean something like "YouTube".
Script support is for small-scale stuff (in my view).
Wrong. I use it for creating start/stop scripts for services along with scripts to stick in crontabs. I don't need to bother with that for small-scales stuff.
3. Accepting that I don't know what you mean by "this", can I point out that as a user, I personally have problems with a significant proportion of scripts distributed with packages - so are you saying that those packages "aren't serious",
Probably, or certainly that their authors don't really care about the scripts they generate...
or that there is no way of doing what they (and I) want at present?
I'm afraid I don't remember/understand what you want...
I'll see if I can find some in due course. (Basically, the types of things I see are scripts distributed on Windows with no filetype extension, .bat wrappers for command line scripts which don't work right when called from another bat file, setuptools-built exe wrappers which result in version-specific binaries for pure-python code, to give some examples).
The last is the "right" thing. These scripts work cross-platform and are tied to the correct version of python for the setup as "originally intended", ie: the version the package was installed with. cheers, Chris
On May 7, 2009, at 10:20 AM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Eric Smith wrote:
Doug Hellmann wrote:
On May 7, 2009, at 8:54 AM, Paul Moore wrote:
I would argue the other way. Why force authors of console scripts to deal with entry points instead of just installing the script as-is? Please explain "as-is" with reference to ensuring that the script works cross-platform. I think the benefit of entry points for
2009/5/7 Doug Hellmann <doug.hellmann@gmail.com>: scripts is that it generates appropriate wrappers to allow use on all platforms. I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/bin or somewhere in the path on Windows and call it as "hello.py".
Does setuptools give me something extra for Windows? I'm not a regular Windows user, so it's likely that there are features I don't know about.
Yes. It creates a .exe wrapper [1]. By using entry points, I don't need to care what the target system is. Also, /usr/bin/env might invoke the wrong python.
Exactly: using entry points for console scripts guarantees that the python into which the corresponding distribution is installed is the one used to run the script, which is *highly* desirable. Otherwise, you end up with the "just install everything in the system Python's site-packages" mess.
pip installs my scripts into a virtualenv without any issue and without using entry points, AFAICT. I guess if we move to requiring entry points and disallowing simple script distribution I'll need to find another way to package virtualenvwrapper. Since it's a bash script, it doesn't have entry points. I've been using setuptools to package it so it can be installed via easy_install, since it is a Python development tool. Doug
At 10:38 AM 5/7/2009 -0400, Doug Hellmann wrote:
pip installs my scripts into a virtualenv without any issue and without using entry points, AFAICT.
I guess if we move to requiring entry points and disallowing simple script distribution I'll need to find another way to package virtualenvwrapper. Since it's a bash script, it doesn't have entry points. I've been using setuptools to package it so it can be installed via easy_install, since it is a Python development tool.
Setuptools still supports "classic" scripts, and I don't see any reason to remove that support. People do package non-Python scripts with their projects, after all. easy_install basically examines such scripts to see if they end with .py, have a #! line with 'python' in it, or are valid Python source code. If any of the above are true, it makes a Python script wrapper, otherwise it assumes the script is some other language and leaves it alone.
On May 7, 2009, at 7:28 PM, P.J. Eby wrote:
At 10:38 AM 5/7/2009 -0400, Doug Hellmann wrote:
pip installs my scripts into a virtualenv without any issue and without using entry points, AFAICT.
I guess if we move to requiring entry points and disallowing simple script distribution I'll need to find another way to package virtualenvwrapper. Since it's a bash script, it doesn't have entry points. I've been using setuptools to package it so it can be installed via easy_install, since it is a Python development tool.
Setuptools still supports "classic" scripts, and I don't see any reason to remove that support. People do package non-Python scripts with their projects, after all.
easy_install basically examines such scripts to see if they end with .py, have a #! line with 'python' in it, or are valid Python source code. If any of the above are true, it makes a Python script wrapper, otherwise it assumes the script is some other language and leaves it alone.
That would do what I want. I only brought it up because another poster in this thread mentioned eliminating "scripts" an requiring console apps to use entry points.
I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/ bin or somewhere in the path on Windows and call it as "hello.py".
With an entry point, hello.py could simply be: def main(): print 'hello!' But I agree, the Distutils 'scripts' is conceptually simpler. Especially since it let's you run a script 'as-is' without first running an install tool to generate a script wrapper. It's only once a script has dependencies on other Python libraries where relying on those libraries being available for import may not work. In this case, the 'scripts' field forces the user to install those libraries in a global location (site-packages), this isn't always desirable though! As for including BASH scripts in the 'scripts' field ... I dunno. Clearly entry_points are meant to only annotate Python code! Is including BASH scripts in the 'scripts' field an abuse of Distutils or is it acceptable usage? The Distutils docs aren't explicit on this, so I'm inclined to say it's acceptable, so this is at least one good reason for keeping the 'scripts' field. But then having both 'scripts' and 'entry_points/console_scripts' is less than perfect since it introduces (mostly) unneccessary TIMTOWTDI.
Doug Hellmann <doug.hellmann@gmail.com> writes:
I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/ bin or somewhere in the path on Windows and call it as "hello.py".
To address the issue raised elsewhere in this thread: Why should the name of a *command* include the suffix ‘.py’? That just begs for the situation down the line where one of these commands is being used widely, by the name ‘hello.py’, and then someone wants to provide an alternative implementation in another language. A command implemented in Ruby named ‘hello.py’ (for backward compatibility) is even more annoying that one implemented in Python. This situation is entirely predictable, so naming it *without* a suffix is the right way to go. It also makes the command less annoying to type. The simple rule of thumb: If it's primarily a module to be imported, name it ‘foo.py’, don't mark it executable, and don't use the shebang line. If it's primarily a command to be run as the main program, name it ‘foo’ with no ‘.py’ suffix, mark it executable, and use the shebang line. -- \ “I have had a perfectly wonderful evening, but this wasn't it.” | `\ —Groucho Marx | _o__) | Ben Finney
On May 7, 2009, at 5:53 PM, Ben Finney wrote:
Doug Hellmann <doug.hellmann@gmail.com> writes:
I write a python script call hello.py like this:
#!/usr/bin/env python
def main(): print 'hello!'
if __name__ == '__main__': main()
Why make me define an entry point for that? I can just put it in /usr/ bin or somewhere in the path on Windows and call it as "hello.py".
To address the issue raised elsewhere in this thread:
Why should the name of a *command* include the suffix ‘.py’? That just begs for the situation down the line where one of these commands is being used widely, by the name ‘hello.py’, and then someone wants to provide an alternative implementation in another language.
I personally don't care if it includes the .py suffix or not, most of the time. I tend not to include it in my own code, and don't get bent out of shape if someone else's tool does use it. Of course, as I've been reminded in this thread not including the .py (and not using entry points) means my "scripts" don't work as-is when installed on Windows. Doug
participants (16)
-
Ben Finney -
Chris Withers -
David Lyon -
Doug Hellmann -
Eric Smith -
Floris Bruynooghe -
Greg Ewing -
Hanno Schlichting -
Ian Bicking -
Noah Gift -
P.J. Eby -
Paul Moore -
Sebastien Binet -
Tarek Ziadé -
Tres Seaver -
Wheat