[Distutils] entry points PEP
Joe Gordon
joe.gordon0 at gmail.com
Fri Jul 19 20:09:24 CEST 2013
I have gone ahead and gathered some information using our standard
development environment, devstack, I ran cProfile on our application, with
the contents of it mocked out,
http://paste.openstack.org/show/40948/
When I try importing pkg_resources in our development environment it is
very slow:
vagrant at precise64:/opt/stack/nova$ time python -c "from pkg_resources
import load_entry_point"
real 0m0.185s
user 0m0.136s
sys 0m0.044s
vagrant at precise64:/opt/stack/nova$ time python -c "print 'hi'"
hi
real 0m0.047s
user 0m0.036s
sys 0m0.008s
I also ran cProfile on just the import line:
http://paste.openstack.org/show/40949/
and
$ python -vvvvvv -c "import pkg_resources"
http://paste.openstack.org/show/40952/
As for python 3, we have to maintain python 2.6 and 2.7 compatibility
so a Python 3 only fix isn't acceptable
On Fri, Jul 19, 2013 at 5:58 AM, Daniel Holth <dholth at gmail.com> wrote:
> On Fri, Jul 19, 2013 at 5:32 AM, Robert Collins
> <robertc at robertcollins.net> wrote:
> > On 19 July 2013 21:24, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >> Robert Collins <robertc <at> robertcollins.net> writes:
> >>
> >>> So my question here would be - can we make it faster? We have just
> >>> been diagnosing a performance problem in nova due to rootwrap being a
> >>> pkg_resources scripts entry point : just getting to the first line of
> >>> main() takes 200ms, and we make dozens of subprocess calls (has to be,
> >>> we're escalating privileges) to the script in question : that time is
> >>> nearly entirely doing introspection of metadata from disk.
> >>
> >> Is there more detailed information about where the time is being spent?
> e.g.
> >> os.stat(), file I/O, parsing of the actual metadata files,
> load_entry_point()
> >> etc.
> >
> > Not sure. Joe?
> >
> > -Rob
>
> You should at least time it against the simpler "import sys, x.main;
> sys.exit(main())" style wrapper.
>
> As a pkg_resources optimization it might be worthwhile to try using
> https://github.com/benhoyt/scandir/ or in Python 3, the undocumented
> cache used by the importer system, to try to speed things up.
>
> It is a bit tricky to profile pkg_resources since it does a lot of
> work at import.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130719/a830ed8a/attachment.html>
More information about the Distutils-SIG
mailing list