[New-bugs-announce] [issue44331] Generate static PyCodeObjects for faster startup

[Neil Schemenauer]

New submission from Neil Schemenauer <nas-python at arctrix.com>:

Note: This is a proof of concept and not ready for merging as is.


This is based on 'frozen_modules' from Jeethu Rao <jeethu at jeethurao.com>, via Larry Hastings.  Larry's git branch was:

        git at github.com:larryhastings/cpython.git
        not_another_use_of_the_word_frozen

Usage:

- Compile Python as normal
- Run "make regen-freeze-startup" to re-generate Python/frozenmodules_code.c
- Compile Python a second time

Changes from Larry's branch:

- Move static C code generation tool to Tools/freeze2
- Move _serializer to Modules
- Rebase on Python 3.10.0b1
- determine startup modules by running sys.executable
- use importlib.util.find_spec() to get code objects
- fix ref-counting when setting __path__
- put static frozen modules in frozen_code_objects dict
- reduce set of "bad" modules as it seems only _collections_abc needs
  exclusion
- fix the is_frozen_package() and is_frozen() functions to find
  static frozen code

It's not passing all unit tests yet but I'm somewhat hopeful there are no deep problems.  Porting the changes from 3.6 to 3.8 and then to 3.10 was not too horrible.  There was a few changes to PyGC_Head, to the PyCodeObject structure and to the runtime initialization process.  That gives me some hope that it wouldn't be too burdensome to maintain this in the long-term.  Mostly it would be updating _serialize.c to keep up with PyCodeObject changes.

Based on benchmarking with 3.8, this gives a decent speedup for startup of a trival program, e.g. python -c "True".  I measure it as taking 76% of the time.  The savings are mostly in marshal.c but there is also some importlib/filesystem overhead that's removed.

----------
messages: 395243
nosy: nascheme
priority: low
severity: normal
stage: patch review
status: open
title: Generate static PyCodeObjects for faster startup
type: performance

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44331>
_______________________________________