On Fri, 23 Jun 2017 23:03:57 +0000
Brett Cannon <brett(a)python.org> wrote:
> On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh(a)gmail.com> wrote:
>
> > As suggested, I'd like to discuss if lazy-loading is an option for
> > improving python-startup time.And if could be done inside the scope of a
> > GSoc project.
> >
>
> It's a possibility and it could be done in the scope of a GSoC project
> easily. Basically what would be needed is an importlib.util.lazy_import()
> function which does mostly what is outlined in
> https://docs.python.org/3/library/importlib.html#approximating-importlib-im…
> but
> where the proper lazy loader is set on the spec object as an extra step.
> Then every module that is used during startup would use
> importlib.util.lazy_import() for importing instead of the normal import
> statement.
My experience is that:
- you want lazy imports to be implicit, i.e. they should work using the
"import" statement and not any special syntax or function invocation
- you want a blacklist and/or whitelist mechanism to restrict lazy
imports to a particular set of modules and packages, because some
modules may not play well with lazy importing (e.g. anything that
registers plugins, specializations -- think singledispatch --, etc.)
For example, I may want to register the "tornado", "asyncio" and "numpy"
namespaces / packages for lazy importing, but not the "numba" namespace
as it uses registration mechanisms quite heavily.
(and the stdlib could be part of the default lazy import whitelist)
Regards
Antoine.
On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh(a)gmail.com> wrote:
> As suggested, I'd like to discuss if lazy-loading is an option for
> improving python-startup time.And if could be done inside the scope of a
> GSoc project.
>
It's a possibility and it could be done in the scope of a GSoC project
easily. Basically what would be needed is an importlib.util.lazy_import()
function which does mostly what is outlined in
https://docs.python.org/3/library/importlib.html#approximating-importlib-im…
but
where the proper lazy loader is set on the spec object as an extra step.
Then every module that is used during startup would use
importlib.util.lazy_import() for importing instead of the normal import
statement. What this would do is help guarantee that all modules that are
identified as part of startup never import a module needlessly as the lazy
loader would simply postpone the load until necessary. This would also
allow for pulling out local imports that are currently done in modules that
are a part of startup and make them global so they are more visible.
But I have no idea if this will actually speed things up. :) At worst it
would slow things down ever so slightly due to the extra overhead of lazy
loading for things that are known to be needed already during startup. At
best, though, is we accidentally discover modules that are being imported
needlessly at startup as well as not having to hide imports in functions
for performance. This fact that it may not be worth it is why I haven't
bothered to try it out yet.
As suggested, I'd like to discuss if lazy-loading is an option for
improving python-startup time.And if could be done inside the scope of a
GSoc project.
Thank You
Regards,
Bhavishya
