[Python-Dev] Python startup optimization: script vs. service

Christian Heimes christian at python.org
Mon Oct 2 05:47:05 EDT 2017


Hello python-dev,

it's great to see that so many developers are working on speeding up
Python's startup. The improvements are going to make Python more
suitable for command line scripts. However I'm worried that some
approaches are going to make other use cases slower and less efficient.
I'm talking about downsides of lazy initialization and deferred imports.


For short running command line scripts, lazy initialization of regular
expressions and deferred import of rarely used modules can greatly
reduce startup time and reduce memory usage.


For long running processes, deferring imports and initialization can be
a huge performance problem. A typical server application should
initialize as much as possible at startup and then signal its partners
that it is ready to serve requests. A deferred import of a module is
going to slow down the first request that happens to require the module.
This is unacceptable for some applications, e.g. Raymond's example of
speed trading.

It's even worse for forking servers. A forking HTTP server handles each
request in a forked child. Each child process has to compile a lazy
regular expression or important a deferred module over and over.
uWSGI's emperor / vassal mode us a pre-fork model with multiple server
processes to efficiently share memory with copy-on-write semantics. Lazy
imports will make the approach less efficient and slow down forking of
new vassals.


TL;DR please refrain from moving imports into functions or implementing
lazy modes, until we have figured out how to satisfy requirements of
both scripts and long running services. We probably need a PEP...

Christian


More information about the Python-Dev mailing list