[Python-Dev] this is what happens if you freeze all the modules required for startup

[Stefan Behnel]

Brett Cannon, 14.04.2014 23:51:
> It was realized during PyCon that since we are freezing importlib we could
> now consider freezing all the modules to cut out having to stat or read
> them from disk. So for day 1 of the sprints I I decided to hack up a
> proof-of-concept to see what kind of performance gain it would get.
> 
> Freezing everything except encodings.__init__, os, and _sysconfigdata, it
> speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14%
> from the slowdown (27% slower vs. 41% slower). The full results are at the
> end of the email.
> 
> Now the question is whether the maintenance cost of having to rebuild
> Python for a select number of stdlib modules is enough to warrant putting
> in the effort to make this work. My guess is the best approach would be
> adding a Lib/_frozen directory where any modules that we treat like this
> would be kept to act as a reminder that you need to rebuild for them (I
> would probably move importlib/_boostrap.py as well to make this consistent).
> 
> Thoughts?

Alternatively, the modules could be compiled with Cython. That would not
only speed up the loading time but also the initialisation time and
runtime. And since you'd keep the original .py files next to the .so files,
you'd still get proper tracebacks.

Compiling the modules natively would also enable linking them right into
the interpreter core, BTW. But that would substantially increase its size.
Maybe some of them could still be worth being linked in.

Stefan