[Python-Dev] requirements for moving __import__ over to importlib?

[Brett Cannon]

On Tue, Feb 7, 2012 at 15:49, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Tue, 7 Feb 2012 15:07:24 -0500
> Brett Cannon <brett at python.org> wrote:
> >
> > Now I'm going to be upfront and say I really did not want to have this
> > performance conversation now as I have done *NO* profiling or analysis of
> > the algorithms used in importlib in order to tune performance (e.g. the
> > function that handles case-sensitivity, which is on the critical path for
> > importing source code, has a platform check which could go away if I
> > instead had platform-specific versions of the function that were assigned
> > to a global variable at startup).
>
> >From a cursory look, I think you're gonna have to break (special-case)
> some abstractions and have some inner loop coded in C for the common
> cases.
>

Wouldn't shock me if it came to that, but obviously I would like to try to
avoid it.


>
> That said, I think profiling and solving performance issues is critical
> *before* integrating this work. It doesn't need to be done by you, but
> the python-dev community shouldn't feel strong-armed to solve the issue.
>
>
That part of the discussion I'm staying out of since I want to see this in
so I'm biased.


>  > IOW I really do not look forward to someone saying "importlib is so much
> > slower at importing a module containing ``pass``" when (a) that never
> > happens, and (b) most programs do not spend their time importing but
> > instead doing interesting work.
>
> Well, import time is so important that the Mercurial developers have
> written an on-demand import mechanism, to reduce the latency of
> command-line operations.
>

Sure, but they are a somewhat extreme case.


>
> But it's not only important for Mercurial and the like. Even if you're
> developing a Web app, making imports slower will make restarts slower,
> and development more tedious in the first place.
>
>
Fine, startup cost from a hard crash I can buy when you are getting 1000
QPS, but development more tedious?


>  > So, if there is going to be some baseline performance target I need to
> hit
> > to make people happy I would prefer to know what that (real-world)
> > benchmark is and what the performance target is going to be on a
> non-debug
> > build.
>
> - No significant slowdown in startup time.
>

What's significant and measuring what exactly? I mean startup already has a
ton of imports as it is, so this would wash out the point of measuring
practically anything else for anything small. This is why I said I want a
benchmark to target which does actual work since flat-out startup time
measures nothing meaningful but busy work. I would get more out of code
that just stat'ed every file in Lib since at least that did some work.


>
> - Within 25% of current performance when importing, say, the "struct"
>  module (Lib/struct.py) from bytecode.
>

Why struct? It's such a small module that it isn't really a typical module.
The median file size of Lib is 11K (e.g. tabnanny.py), not 238 bytes (which
is barely past Hello World). And is this just importing struct or is this
from startup, e.g. ``python -c "import struct"``?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120207/51b5231e/attachment.html>