Abstract This PEP proposes a way to represent the setuptools “entry points” feature in standard Python metadata. Entry points are a useful mechanism for advertising or discovering plugins or other exported functionality without having to depend on the module namespace. Since the feature is used by many existing Python distributions and not everyone wants to use setuptools, it is useful to have a way to represent the functionality that is not tied to setuptools itself. The proposed feature defines an extension field for the standard Python distribution metadata and some basic semantics for its use. Overview Entry points are represented as an extension in the metadata: { … “extensions”: { “entry_points” : { … } } } The extension contains the data in a dictionary format similar to that accepted by setuptools’ setup(entry_points={}) keyword argument. It is a dictionary of “group” : [ “key=module.name:attrs.attrs [extra, extra2, ...]”, ]. “group” is the name of this class of entry points. Values in common use include “console_scripts” and “sqlalchemy.dialects”. During discovery, clients usually iterate over all the entry points in a particular group. “key” is the name of a particular entry point in the group. It must be locally unique within this distribution’s group. “module.name” is the Python module that defines the entry point. Client code would import the module to use the entry point. “:attrs.attrs” are the optional attributes of the module that define the entry point (the module itself can be the entry point). Client code uses normal attribute access on the imported module to use the entry point. “[extra, extra2, ...]” are any optional features of the distribution (declared with “extras”) that must be installed to use the declared entry point. This is not common. Complete example { … “extensions”: { “entry_points” : { { “sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] } } } } Use Client code reads every distribution’s metadata file on sys.path, filtering for the entry point group name desired, and, if applicable, the individual entry point name. Once the desired entry point has been found, a utility function imports the necessary module and attribute to return an object which can be inspected or called. distlib and setuptools’ pkg_resources provide APIs for this functionality.
On 18 July 2013 14:03, Daniel Holth <dholth@gmail.com> wrote:
Abstract
This PEP proposes a way to represent the setuptools “entry points” feature in standard Python metadata. Entry points are a useful mechanism for advertising or discovering plugins or other exported functionality without having to depend on the module namespace. Since the feature is used by many existing Python distributions and not everyone wants to use setuptools, it is useful to have a way to represent the functionality that is not tied to setuptools itself.
1. I think that console (and GUI) scripts should be top-level metadata, not an extension. Installers need to be able to create wrappers based on these, and it is useful data for introspection. 2. distlib calls these "exports" and I think that's a better name. But if names are all that we argue over, I'm happy :-) 3. Someone (I think it was PJE) pointed out that having entry points in a separate metadata file was good because it allowed a fast check of "does this distribution expose entry points?" Note that this isn't a useful thing to check for script wrappers, which again argues that those should be handled separately. 4. You seem to have an extra set of curly braces in a few places. You say the value of "entry_points" is a dictionary, but you show it as a set containing one dictionary in the set (and of course sets aren't valid JSON). I'll assume this is just a typo, and you meant { … “extensions”: { “entry_points” : { “sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] , ... } } } 5. What's the logic for having the values (I don't see a good term for these - the "mysql = ..." bit above) be a structured string that the user has to parse? Either it should be completely free format (which I suspect makes it of limited use for introspection, if nothing else) or it should be broken up into JSON - no point in having a blob of data that needs parsing in the middle of an already structured format! Paul
On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 18 July 2013 14:03, Daniel Holth <dholth@gmail.com> wrote:
Abstract
This PEP proposes a way to represent the setuptools “entry points” feature in standard Python metadata. Entry points are a useful mechanism for advertising or discovering plugins or other exported functionality without having to depend on the module namespace. Since the feature is used by many existing Python distributions and not everyone wants to use setuptools, it is useful to have a way to represent the functionality that is not tied to setuptools itself.
1. I think that console (and GUI) scripts should be top-level metadata, not an extension. Installers need to be able to create wrappers based on these, and it is useful data for introspection.
It is an extension so it can be a separate PEP, since there's enough to talk about in the main PEP. The document tries to write down what setuptools does in a straightforward json way.
2. distlib calls these "exports" and I think that's a better name. But if names are all that we argue over, I'm happy :-)
3. Someone (I think it was PJE) pointed out that having entry points in a separate metadata file was good because it allowed a fast check of "does this distribution expose entry points?" Note that this isn't a useful thing to check for script wrappers, which again argues that those should be handled separately.
I am more interested in seeing the installer update some kind of index like a sqlite database. I don't know if it would be faster since I haven't tried it.
4. You seem to have an extra set of curly braces in a few places. You say the value of "entry_points" is a dictionary, but you show it as a set containing one dictionary in the set (and of course sets aren't valid JSON). I'll assume this is just a typo, and you meant
{ … “extensions”: { “entry_points” : { “sqlalchemy.dialects” : [ “mysql = sqlalchemy_mysql:base.dialect”] , ... } } }
5. What's the logic for having the values (I don't see a good term for these - the "mysql = ..." bit above) be a structured string that the user has to parse? Either it should be completely free format (which I suspect makes it of limited use for introspection, if nothing else) or it should be broken up into JSON - no point in having a blob of data that needs parsing in the middle of an already structured format!
For one thing you can have more than one mysql = in the same sqlalchemy.dialects. I think in this instance the string parsing is probably simpler than defining a more JSONy version. FWIW the PEP 426 metadata is also full of structured strings.
Daniel Holth <dholth <at> gmail.com> writes:
On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore <p.f.moore <at> gmail.com> wrote:
It is an extension so it can be a separate PEP, since there's enough to talk about in the main PEP. The document tries to write down what setuptools does in a straightforward json way.
If the JSON we're talking about goes in the main metadata dictionary, perhaps it should go into PEP 426, so that everything is in one place. As we're talking about special handling of script generation by installers, it may make sense to not consider them as extensions but as core metadata.
2. distlib calls these "exports" and I think that's a better name. But if names are all that we argue over, I'm happy
3. Someone (I think it was PJE) pointed out that having entry points in a separate metadata file was good because it allowed a fast check of "does this distribution expose entry points?" Note that this isn't a useful
The reason for picking "exports" is that you can export data, not just code, and "entry point" has a connotation of being code. PJE suggested "exports" and I think it fits the bill better than "entry_points". thing
to check for script wrappers, which again argues that those should be handled separately.
That seems generally true, except that there might be applications out there that want to walk over the scripts associated with different installed distributions. That seems a legitimate, though uncommon, use case. In any case, I think the script metadata should be structured as "scripts": { "console": [spec1, spec2] "gui": [spec1, spec2] } Because that allows a sensible place for e.g. script generation options, as in "scripts": { "console": [spec1, spec2] "gui": [spec1, spec2] "options": { ... } }
I am more interested in seeing the installer update some kind of index like a sqlite database. I don't know if it would be faster since I haven't tried it.
That (a SQLite installation database) is probably some way off, and would require more significant changes elsewhere. The big advantage of the current setup with text files is that every thing is human readable - very handy when things go wrong. I don't know whether this area is a performance bottleneck, but we will be able to deal with it using a separate exports.json file in the short term.
For one thing you can have more than one mysql = in the same sqlalchemy.dialects. I think in this instance the string parsing is
Don't you say in the PEP about the key that "It must be locally unique within this distribution’s group."?
probably simpler than defining a more JSONy version.
I think Paul's point is that if it was JSON, you wouldn't need to parse anything. The current format of entries is name = module:attrs [flag1,flag2] which could be { "name": name, "module": module, "attrs": attrs, "flags": ["flag1", "flag2"] } which is obviously more verbose. Note that I don't see necessarily a connection between extras and flags, though you've mentioned that they're extras. Does setuptools store, against an installed distribution, the extras it was installed with? AFAIK it doesn't. (Even if it did, it would need to keep that updated if one of the extras' dependencies were later uninstalled.) And if not, how would having extras in the specification help, since you can't test the "must be installed" part? On the other hand, you might want to use generalised flags that provide control over how the exported entry is processed. One reason for keeping the format as-is might be in case any migration issues come up (i.e. people depending on this specific format in some way), but I'm not sure whether there are any such issues or what they might be.
FWIW the PEP 426 metadata is also full of structured strings.
True - the dependency specifier, if nothing else. One other thing is that the PEP needs to state that the exports metadata must be written to exports.json in the .dist-info folder by an installer - that isn't mentioned anywhere. Also, whether it should contain the scripts part, or just the other exports (but see my comment above as to why scripts might be considered exports worth iterating over, like any other). Regards, Vinay Sajip
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Daniel Holth <dholth <at> gmail.com> writes:
On Thu, Jul 18, 2013 at 9:27 AM, Paul Moore <p.f.moore <at> gmail.com> wrote:
It is an extension so it can be a separate PEP, since there's enough to talk about in the main PEP. The document tries to write down what setuptools does in a straightforward json way.
If the JSON we're talking about goes in the main metadata dictionary, perhaps it should go into PEP 426, so that everything is in one place. As we're talking about special handling of script generation by installers, it may make sense to not consider them as extensions but as core metadata.
2. distlib calls these "exports" and I think that's a better name. But if names are all that we argue over, I'm happy
The reason for picking "exports" is that you can export data, not just code, and "entry point" has a connotation of being code. PJE suggested "exports" and I think it fits the bill better than "entry_points".
3. Someone (I think it was PJE) pointed out that having entry points in a separate metadata file was good because it allowed a fast check of "does this distribution expose entry points?" Note that this isn't a useful thing to check for script wrappers, which again argues that those should be handled separately.
That seems generally true, except that there might be applications out there that want to walk over the scripts associated with different installed distributions. That seems a legitimate, though uncommon, use case.
In any case, I think the script metadata should be structured as
"scripts": { "console": [spec1, spec2] "gui": [spec1, spec2] }
Because that allows a sensible place for e.g. script generation options, as in
"scripts": { "console": [spec1, spec2] "gui": [spec1, spec2] "options": { ... } }
I am more interested in seeing the installer update some kind of index like a sqlite database. I don't know if it would be faster since I haven't tried it.
That (a SQLite installation database) is probably some way off, and would require more significant changes elsewhere. The big advantage of the current setup with text files is that every thing is human readable - very handy when things go wrong. I don't know whether this area is a performance bottleneck, but we will be able to deal with it using a separate exports.json file in the short term.
Who knows. On some filesystems stat() is painfully slow and you could be better off just parsing the single metadata file.
For one thing you can have more than one mysql = in the same sqlalchemy.dialects. I think in this instance the string parsing is
Don't you say in the PEP about the key that "It must be locally unique within this distribution’s group."?
The kind of entry point has to be unique, but the name inside the spec does not: dialects : [ "mysql = first mysql driver...", "mysql = second mysql driver..." ]
probably simpler than defining a more JSONy version.
I think Paul's point is that if it was JSON, you wouldn't need to parse anything. The current format of entries is
name = module:attrs [flag1,flag2]
which could be { "name": name, "module": module, "attrs": attrs, "flags": ["flag1", "flag2"] } which is obviously more verbose.
Note that I don't see necessarily a connection between extras and flags, though you've mentioned that they're extras. Does setuptools store, against an installed distribution, the extras it was installed with? AFAIK it doesn't. (Even if it did, it would need to keep that updated if one of the extras' dependencies were later uninstalled.) And if not, how would having extras in the specification help, since you can't test the "must be installed" part? On the other hand, you might want to use generalised flags that provide control over how the exported entry is processed.
You might mean the document's mention that in setuptools loading an entry point can require a particular extra. In setuptools this would mean additional eggs could be added to sys.path as a result of loading the entry point.
One reason for keeping the format as-is might be in case any migration issues come up (i.e. people depending on this specific format in some way), but I'm not sure whether there are any such issues or what they might be.
FWIW the PEP 426 metadata is also full of structured strings.
True - the dependency specifier, if nothing else.
One other thing is that the PEP needs to state that the exports metadata must be written to exports.json in the .dist-info folder by an installer - that isn't mentioned anywhere. Also, whether it should contain the scripts part, or just the other exports (but see my comment above as to why scripts might be considered exports worth iterating over, like any other).
I would like to see it timed with and without exports.json Why not keep a single API for iterating over console scripts and other entry points and exports. Seems harmless.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Daniel Holth <dholth <at> gmail.com> writes:
For one thing you can have more than one mysql = in the same sqlalchemy.dialects. I think in this instance the string parsing is
Don't you say in the PEP about the key that "It must be locally unique within this distribution’s group."?
Setuptools requires this per-distribution uniqueness, but note that uniqueness is not required across distributions. So more than one distribution can export a 'mysql' in the 'sqlalchemy.dialects' group. It's up to the application to decide how to handle multiple definitions; typically one either uses all of them or the first one found on sys.path, or some other tie-breaking mechanism. The pkg_resources entry point APIs just provide operations for iterating over entry points defined on either a single distribution, or across all distributions on a specified set of directories. (Via the WorkingSet API.)
Note that I don't see necessarily a connection between extras and flags, though you've mentioned that they're extras. Does setuptools store, against an installed distribution, the extras it was installed with? AFAIK it doesn't. (Even if it did, it would need to keep that updated if one of the extras' dependencies were later uninstalled.) And if not, how would having extras in the specification help, since you can't test the "must be installed" part?
The pkg_resources implementation does a require() for the extras at the time the entry point is loaded, i.e., just before importing. This allows it to dynamically add requirements to sys.path, or alternatively raise an error to indicate the extras aren't available. In addition, various entry point API functions take an 'installer' keyword argument, specifying a callback to handle installation of missing extras. Setuptools uses this feature internally, so that if you use a setup.py command whose entry point needs additional dependencies, those will be fetched on-the-fly.
I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json). They're important enough to include directly. Cheers, Nick.
On 19 Jul 2013 08:42, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
I actually now plan to make scripts and exports first class citizens in
PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json).
They're important enough to include directly.
The PEP that will define the updated dist-info contents will be the sdist 2.0 PEP. Things are probably stable enough for me to write that, now. Cheers, Nick.
Cheers, Nick.
On Thu, Jul 18, 2013 at 6:42 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json).
They're important enough to include directly.
Cheers, Nick.
Must they be two separate features? One of the reasons I use entry_points scripts is that I forget that the scripts= command to setup() exists at all.
OH -scripts would be the distutils-style scrips. On Thu, Jul 18, 2013 at 6:51 PM, Daniel Holth <dholth@gmail.com> wrote:
On Thu, Jul 18, 2013 at 6:42 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted summary files (like the existing pydist-dependencies.json).
They're important enough to include directly.
Cheers, Nick.
Must they be two separate features? One of the reasons I use entry_points scripts is that I forget that the scripts= command to setup() exists at all.
On 18 July 2013 23:51, Daniel Holth <dholth@gmail.com> wrote:
I actually now plan to make scripts and exports first class citizens in PEP 426, with pydist-scripts.json and pydist-exports.json as extracted
On Thu, Jul 18, 2013 at 6:42 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: summary
files (like the existing pydist-dependencies.json).
They're important enough to include directly.
Cheers, Nick.
Must they be two separate features? One of the reasons I use entry_points scripts is that I forget that the scripts= command to setup() exists at all.
This is just the metadata. I would assume that the console-scripts/gui-scripts entry pointdefinitions in setuptools would be extracted and put into the scripts metadata, and any non-script entry points would go into exports. The legacy scripts= arguments would also go into scripts. Nick - How would the 2 methods of specifying scripts (legacy name of a (potentially platform-specific) file, and setuptools module:function style spec requiring wrapper generation) be recorded, under your approach? Paul
On 19 July 2013 18:28, Paul Moore <p.f.moore@gmail.com> wrote:
This is just the metadata. I would assume that the console-scripts/gui-scripts entry pointdefinitions in setuptools would be extracted and put into the scripts metadata, and any non-script entry points would go into exports. The legacy scripts= arguments would also go into scripts.
Nick - How would the 2 methods of specifying scripts (legacy name of a (potentially platform-specific) file, and setuptools module:function style spec requiring wrapper generation) be recorded, under your approach?
As in "we install this script" vs "we need a script wrapper generated at install time for this"? Not sure, I hadn't even the idea of letting people register arbitrary "we install this script". Heck, I haven't even worked out what I want the format to look like :) I'll take that into account now, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 July 2013 09:35, Nick Coghlan <ncoghlan@gmail.com> wrote:
Not sure, I hadn't even the idea of letting people register arbitrary "we install this script". Heck, I haven't even worked out what I want the format to look like :)
That's the big legacy issue. The old distutils script= argument just dumps arbitrary files into the scripts location. I thought people only used that for the same thing that setuptools entry points can do (and so could safely be treated as "legacy, phase it out") but Daniel has scanned PyPI and tells me otherwise :-( There's also setuptools itself, which generates exes and py files (on Windows, on Unix presumably just a #! script) which are then effectively "legacy-style" scripts It could certainly be changed to just write metadata, but as per the other thread this would be a functionality change that people seem to feel would be unacceptable.
I'll take that into account now, though.
Best of luck :-) Paul
On 19 July 2013 01:03, Daniel Holth <dholth@gmail.com> wrote:
Abstract
This PEP proposes a way to represent the setuptools “entry points” feature in standard Python metadata. Entry points are a useful mechanism for advertising or discovering plugins or other exported functionality without having to depend on the module namespace. Since the feature is used by many existing Python distributions and not everyone wants to use setuptools, it is useful to have a way to represent the functionality that is not tied to setuptools itself.
The proposed feature defines an extension field for the standard Python distribution metadata and some basic semantics for its use.
So my question here would be - can we make it faster? We have just been diagnosing a performance problem in nova due to rootwrap being a pkg_resources scripts entry point : just getting to the first line of main() takes 200ms, and we make dozens of subprocess calls (has to be, we're escalating privileges) to the script in question : that time is nearly entirely doing introspection of metadata from disk. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Cloud Services
Robert Collins <robertc <at> robertcollins.net> writes:
So my question here would be - can we make it faster? We have just been diagnosing a performance problem in nova due to rootwrap being a pkg_resources scripts entry point : just getting to the first line of main() takes 200ms, and we make dozens of subprocess calls (has to be, we're escalating privileges) to the script in question : that time is nearly entirely doing introspection of metadata from disk.
Is there more detailed information about where the time is being spent? e.g. os.stat(), file I/O, parsing of the actual metadata files, load_entry_point() etc. Regards, Vinay Sajip
On 19 July 2013 21:24, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Robert Collins <robertc <at> robertcollins.net> writes:
So my question here would be - can we make it faster? We have just been diagnosing a performance problem in nova due to rootwrap being a pkg_resources scripts entry point : just getting to the first line of main() takes 200ms, and we make dozens of subprocess calls (has to be, we're escalating privileges) to the script in question : that time is nearly entirely doing introspection of metadata from disk.
Is there more detailed information about where the time is being spent? e.g. os.stat(), file I/O, parsing of the actual metadata files, load_entry_point() etc.
Not sure. Joe? -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Cloud Services
On Fri, Jul 19, 2013 at 5:32 AM, Robert Collins <robertc@robertcollins.net> wrote:
On 19 July 2013 21:24, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Robert Collins <robertc <at> robertcollins.net> writes:
So my question here would be - can we make it faster? We have just been diagnosing a performance problem in nova due to rootwrap being a pkg_resources scripts entry point : just getting to the first line of main() takes 200ms, and we make dozens of subprocess calls (has to be, we're escalating privileges) to the script in question : that time is nearly entirely doing introspection of metadata from disk.
Is there more detailed information about where the time is being spent? e.g. os.stat(), file I/O, parsing of the actual metadata files, load_entry_point() etc.
Not sure. Joe?
-Rob
You should at least time it against the simpler "import sys, x.main; sys.exit(main())" style wrapper. As a pkg_resources optimization it might be worthwhile to try using https://github.com/benhoyt/scandir/ or in Python 3, the undocumented cache used by the importer system, to try to speed things up. It is a bit tricky to profile pkg_resources since it does a lot of work at import.
I have gone ahead and gathered some information using our standard development environment, devstack, I ran cProfile on our application, with the contents of it mocked out, http://paste.openstack.org/show/40948/ When I try importing pkg_resources in our development environment it is very slow: vagrant@precise64:/opt/stack/nova$ time python -c "from pkg_resources import load_entry_point" real 0m0.185s user 0m0.136s sys 0m0.044s vagrant@precise64:/opt/stack/nova$ time python -c "print 'hi'" hi real 0m0.047s user 0m0.036s sys 0m0.008s I also ran cProfile on just the import line: http://paste.openstack.org/show/40949/ and $ python -vvvvvv -c "import pkg_resources" http://paste.openstack.org/show/40952/ As for python 3, we have to maintain python 2.6 and 2.7 compatibility so a Python 3 only fix isn't acceptable On Fri, Jul 19, 2013 at 5:58 AM, Daniel Holth <dholth@gmail.com> wrote:
On Fri, Jul 19, 2013 at 5:32 AM, Robert Collins <robertc@robertcollins.net> wrote:
On 19 July 2013 21:24, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Robert Collins <robertc <at> robertcollins.net> writes:
So my question here would be - can we make it faster? We have just been diagnosing a performance problem in nova due to rootwrap being a pkg_resources scripts entry point : just getting to the first line of main() takes 200ms, and we make dozens of subprocess calls (has to be, we're escalating privileges) to the script in question : that time is nearly entirely doing introspection of metadata from disk.
Is there more detailed information about where the time is being spent? e.g. os.stat(), file I/O, parsing of the actual metadata files, load_entry_point() etc.
Not sure. Joe?
-Rob
You should at least time it against the simpler "import sys, x.main; sys.exit(main())" style wrapper.
As a pkg_resources optimization it might be worthwhile to try using https://github.com/benhoyt/scandir/ or in Python 3, the undocumented cache used by the importer system, to try to speed things up.
It is a bit tricky to profile pkg_resources since it does a lot of work at import.
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
When I try importing pkg_resources in our development environment it is very slow:
Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.)
On Fri, Jul 19, 2013 at 1:42 PM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
When I try importing pkg_resources in our development environment it is very slow:
Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.)
It looks like zc.buildout is not an option as we are already heavily invested in using pip and virtualenv.
Yeah. Not moving to zc.buildout for anything. I believe it will be a better option to just write by-hand scripts that get installed that just do: from nova.rootwrap import cmd return cmd.main(sys.argv) or something. Basically, a tiny boiler-plate script that does the same thing as a console_scripts entry point thing without loading the module in question via pkg_resources. On Fri, Jul 19, 2013 at 3:10 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
On Fri, Jul 19, 2013 at 1:42 PM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
When I try importing pkg_resources in our development environment it is very slow:
Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.)
It looks like zc.buildout is not an option as we are already heavily invested in using pip and virtualenv.
On 19 July 2013 23:22, Monty Taylor <monty.taylor@gmail.com> wrote:
Yeah. Not moving to zc.buildout for anything. I believe it will be a better option to just write by-hand scripts that get installed that just do:
from nova.rootwrap import cmd
return cmd.main(sys.argv)
or something. Basically, a tiny boiler-plate script that does the same thing as a console_scripts entry point thing without loading the module in question via pkg_resources.
Do you care about Windows compatibility for your app? If not, this is probably your best option. If you do, you'll need to add exe wrappers for the scripts on Windows - if you need that, then using distlib's script generator might be worth investigating. Paul
On Jul 19, 2013, at 6:22 PM, Monty Taylor <monty.taylor@gmail.com> wrote:
Yeah. Not moving to zc.buildout for anything. I believe it will be a better option to just write by-hand scripts that get installed that just do:
from nova.rootwrap import cmd
return cmd.main(sys.argv)
or something. Basically, a tiny boiler-plate script that does the same thing as a console_scripts entry point thing without loading the module in question via pkg_resources.
We could also have pbr do this when it builds the sdist. Doug
On Fri, Jul 19, 2013 at 3:10 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
On Fri, Jul 19, 2013 at 1:42 PM, PJ Eby <pje@telecommunity.com> wrote: On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
When I try importing pkg_resources in our development environment it is very slow:
Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.)
It looks like zc.buildout is not an option as we are already heavily invested in using pip and virtualenv.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
https://review.openstack.org/#/c/38000/ On not-Windows, install a non-pkg_resources based script content. On windows, defer to underlying setuptools functionality. (the test failures showing on the patch were build farm issues which we just sorted, I'll run-check the patch once the farm is good)
On Sat, Jul 20, 2013 at 1:57 PM, Monty Taylor <monty.taylor@gmail.com> wrote:
https://review.openstack.org/#/c/38000/
On not-Windows, install a non-pkg_resources based script content. On windows, defer to underlying setuptools functionality. (the test failures showing on the patch were build farm issues which we just sorted, I'll run-check the patch once the farm is good)
Another way to make your scripts run faster is to use on-demand imports. Mercurial does "from mercurial import demandimport; demandimport.enable()". apipkg is another implementation. The drawback is that some Python programs run incorrectly if the import order changes.
On Fri, Jul 19, 2013 at 6:10 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
On Fri, Jul 19, 2013 at 1:42 PM, PJ Eby <pje@telecommunity.com> wrote:
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon <joe.gordon0@gmail.com> wrote:
When I try importing pkg_resources in our development environment it is very slow:
Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.)
It looks like zc.buildout is not an option as we are already heavily invested in using pip and virtualenv.
Here's where the magic happens: https://bitbucket.org/pypa/setuptools/src/9dc434ac0308749d564d721a19ee412c2e... And: https://bitbucket.org/pypa/setuptools/src/9dc434ac0308749d564d721a19ee412c2e...
participants (9)
-
Daniel Holth -
Doug Hellmann -
Joe Gordon -
Monty Taylor -
Nick Coghlan -
Paul Moore -
PJ Eby -
Robert Collins -
Vinay Sajip