[Python-Dev] To reduce Python "application" startup time

[Brett Cannon]

I should mention that I have a prototype design for improving importlib's
lazy loading to be easier to turn on and use. See
https://notebooks.azure.com/Brett/libraries/di2Btqj7zSI/html/Lazy%20importing.ipynb
for my current notes. Part of it includes an explicit lazy_import()
function which would negate needing to hide imports in functions to delay
their importation.

On Wed, 6 Sep 2017 at 20:50 INADA Naoki <songofacandy at gmail.com> wrote:

> > I’m not sure however whether burying imports inside functions (as a kind
> of poor man’s lazy import) is ultimately going to be satisfying.  First,
> it’s not natural, it generally violates coding standards (e.g. PEP 8), and
> can make linters complain.
>
> Of course,  I tried to move imports only when (1) it's only used one
> or two of many functions in the module,
> (2) it's relatively heavy, (3) it's rerely imported from other modules.
>
>
> >   Second, I think you’ll end up chasing imports all over the stdlib and
> third party modules in any sufficiently complicated application.
>
> Agree.  I won't use much time to optimization by moving import from
> top to inner function in stdlib.
>
> I think my import-profiler patch can be polished and committed in Python
> to help
> library maintainers to know import time easily.  (Maybe, `python -X
> import-profile`)
>
>
> > Third, I’m not sure that the gains you’ll get won’t just be overwhelmed
> by lots of other things going on, such as pkg_resources entry point
> processing, pth file processing, site.py effects, command line processing
> libraries such as click, and implicitly added distribution exception hooks
> (e.g. Ubuntu’s apport).
>
> Yes.  I noticed some of them while profiling imports.
> For example, old-style namespace package imports types module for
> types.Module.
> Types module imports functools, and functools imports collections.
> So some efforts in CPython (Avoid importing collections and functools
> from site) is not
> worth enough when there are at least one old-style namespace package
> is installed.
>
>
> >
> > Many of these can’t be blamed on Python itself, but all can contribute
> significantly to Python’s apparent start up time.  It’s definitely worth
> investigating the details of Python import, and a few of us at the core
> sprint have looked at those numbers and thrown around ideas for
> improvement, but we’ll need to look at the effects up and down the stack to
> improve the start up performance for the average Python application.
> >
>
> Yes. I totally agree with you.  That's why I use import-profile.patch
> for some 3rd party libraries.
>
> Currently, I have these ideas to optimize application startup time.
>
> * Faster, or lazily compiling regular expression. (pkg_resources
> imports pyparsing, which has lot regex)
> * More usable lazy import. (which can be solved "PEP 549: Instance
> Properties (aka: module properties)")
> * Optimize enum creation.
> * Faster namedtuple (There is pull request already)
> * Faster ABC
> * Breaking large import tree in stdlib.  (PEP 549 may help this too)
>
> Regards,
>
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170920/b94b9594/attachment.html>