[Python-Dev] this is what happens if you freeze all the modules required for startup

Eric Snow ericsnowcurrently at gmail.com
Tue Apr 15 16:30:12 CEST 2014


On Mon, Apr 14, 2014 at 3:51 PM, Brett Cannon <bcannon at gmail.com> wrote:
> It was realized during PyCon that since we are freezing importlib we could
> now consider freezing all the modules to cut out having to stat or read them
> from disk. So for day 1 of the sprints I I decided to hack up a
> proof-of-concept to see what kind of performance gain it would get.
>
> Freezing everything except encodings.__init__, os, and _sysconfigdata, it
> speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14%
> from the slowdown (27% slower vs. 41% slower). The full results are at the
> end of the email.

Nice.  I was hoping it would be even bigger (given the hyper-focus
people put on the impact of FS-access on startup time imports), but
this is definitely a significant improvement.  I wonder then where the
remaining slowdown lies; are there any remaining low hanging fruit
elsewhere?

>
> Now the question is whether the maintenance cost of having to rebuild Python
> for a select number of stdlib modules is enough to warrant putting in the
> effort to make this work.

Yeah.  Definitely the big question.  Who cares the most about startup
time?  Would this improvement please them?  Does that alone make it
worth the increased maintenance burden?  Is that group big enough or
important enough to justify it?  At the very least it may be good for
the PR value alone, but the maintenance cost will long outlive the PR
benefit. :)

> My guess is the best approach would be adding a
> Lib/_frozen directory where any modules that we treat like this would be
> kept to act as a reminder that you need to rebuild for them (I would
> probably move importlib/_boostrap.py as well to make this consistent).

That makes sense.  I also wonder if we could accomplish the same thing
with a marker (e.g. a comment) in each related module (and leave them
where they are).  A marker would allow for easily finding the
freezable modules.

Personally, I think the speedup would be worth it if it doesn't add
significant to the maintenance burden.

-eric


More information about the Python-Dev mailing list