Mailman 3 June 2017 - Speed

Re: [Speed] Lazy-loading to decrease python_startup time
by Antoine Pitrou June 24, 2017

June 24, 2017

On Fri, 23 Jun 2017 23:03:57 +0000 Brett Cannon <brett(a)python.org> wrote: > On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh(a)gmail.com> wrote: > > > As suggested, I'd like to discuss if lazy-loading is an option for > > improving python-startup time.And if could be done inside the scope of a > > GSoc project. > > > > It's a possibility and it could be done in the scope of a GSoC project > easily. Basically what would be needed is an importlib.util.lazy_import() > function which does mostly what is outlined in > https://docs.python.org/3/library/importlib.html#approximating-importlib-im… > but > where the proper lazy loader is set on the spec object as an extra step. > Then every module that is used during startup would use > importlib.util.lazy_import() for importing instead of the normal import > statement. My experience is that: - you want lazy imports to be implicit, i.e. they should work using the "import" statement and not any special syntax or function invocation - you want a blacklist and/or whitelist mechanism to restrict lazy imports to a particular set of modules and packages, because some modules may not play well with lazy importing (e.g. anything that registers plugins, specializations -- think singledispatch --, etc.) For example, I may want to register the "tornado", "asyncio" and "numpy" namespaces / packages for lazy importing, but not the "numba" namespace as it uses registration mechanisms quite heavily. (and the stdlib could be part of the default lazy import whitelist) Regards Antoine.

2 1

Re: [Speed] Lazy-loading to decrease python_startup time
by Brett Cannon June 23, 2017

June 23, 2017

On Fri, 23 Jun 2017 at 12:17 Bhavishya <bhavishyagopesh(a)gmail.com> wrote: > As suggested, I'd like to discuss if lazy-loading is an option for > improving python-startup time.And if could be done inside the scope of a > GSoc project. > It's a possibility and it could be done in the scope of a GSoC project easily. Basically what would be needed is an importlib.util.lazy_import() function which does mostly what is outlined in https://docs.python.org/3/library/importlib.html#approximating-importlib-im… but where the proper lazy loader is set on the spec object as an extra step. Then every module that is used during startup would use importlib.util.lazy_import() for importing instead of the normal import statement. What this would do is help guarantee that all modules that are identified as part of startup never import a module needlessly as the lazy loader would simply postpone the load until necessary. This would also allow for pulling out local imports that are currently done in modules that are a part of startup and make them global so they are more visible. But I have no idea if this will actually speed things up. :) At worst it would slow things down ever so slightly due to the extra overhead of lazy loading for things that are known to be needed already during startup. At best, though, is we accidentally discover modules that are being imported needlessly at startup as well as not having to hide imports in functions for performance. This fact that it may not be worth it is why I haven't bothered to try it out yet.

1 0

Lazy-loading to decrease python_startup time
by Bhavishya June 23, 2017

June 23, 2017

As suggested, I'd like to discuss if lazy-loading is an option for improving python-startup time.And if could be done inside the scope of a GSoc project. Thank You Regards, Bhavishya

1 0