Re: [Speed] Lazy-loading to decrease python_startup time
On Sat, 24 Jun 2017 16:28:29 +0000 Brett Cannon <brett@python.org> wrote:
My experience is that:
- you want lazy imports to be implicit, i.e. they should work using the "import" statement and not any special syntax or function invocation
- you want a blacklist and/or whitelist mechanism to restrict lazy imports to a particular set of modules and packages, because some modules may not play well with lazy importing (e.g. anything that registers plugins, specializations -- think singledispatch --, etc.)
For example, I may want to register the "tornado", "asyncio" and "numpy" namespaces / packages for lazy importing, but not the "numba" namespace as it uses registration mechanisms quite heavily.
(and the stdlib could be part of the default lazy import whitelist)
That's all true for an end user's application, but for the stdlib where there is no knock-on effects from dependencies not being loaded lazily I don't think it's quite as critical. Plus lazy loading does make debugging harder by making loads that trigger an exception happen at the line of first use instead of at the import line, so I don't know if it's desirable to automatically make the whole stdlib be lazily loaded from the outset (which is what would be required since doing it in e.g. sitecustomize.py wouldn't happen until after startup anyway).
Yes, you are right. I was assuming that if we take the time to include a lazy import system, we'd make it available for third-party applications, though ;-)
Regards
Antoine.
On Sat, 24 Jun 2017 at 09:50 Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 24 Jun 2017 16:28:29 +0000 Brett Cannon <brett@python.org> wrote:
My experience is that:
- you want lazy imports to be implicit, i.e. they should work using the "import" statement and not any special syntax or function invocation
- you want a blacklist and/or whitelist mechanism to restrict lazy imports to a particular set of modules and packages, because some modules may not play well with lazy importing (e.g. anything that registers plugins, specializations -- think singledispatch --, etc.)
For example, I may want to register the "tornado", "asyncio" and
"numpy"
namespaces / packages for lazy importing, but not the "numba" namespace as it uses registration mechanisms quite heavily.
(and the stdlib could be part of the default lazy import whitelist)
That's all true for an end user's application, but for the stdlib where there is no knock-on effects from dependencies not being loaded lazily I don't think it's quite as critical. Plus lazy loading does make debugging harder by making loads that trigger an exception happen at the line of first use instead of at the import line, so I don't know if it's desirable to automatically make the whole stdlib be lazily loaded from the outset (which is what would be required since doing it in e.g. sitecustomize.py wouldn't happen until after startup anyway).
Yes, you are right. I was assuming that if we take the time to include a lazy import system, we'd make it available for third-party applications, though ;-)
That's step 2. :) While EIBTI for the lazy_import(), "practicality beats purity" in making it easy to just flip a switch to make all imports lazy. Plus I have not thought through the design of the "switch" solution yet while the function solution is already done thanks to importlib.import_module() (which I honestly thought of giving a 'lazy' keyword-only argument to, but since the algorithm would change I figured it was probably best not to go down that route).
participants (2)
-
Antoine Pitrou -
Brett Cannon