[Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?
larry at hastings.org
Fri Sep 14 17:27:37 EDT 2018
What follows is the text of issue 34690:
The PR is here:
I don't know if we should be discussing this here on python-dev, or on
bpo, or on Zulip, or on the soon-to-be-created Discourse. But maybe we
can talk about it somewhere!
This patch was sent to me privately by Jeethu Rao at Facebook. It's a
change they're working with internally to improve startup time. What
I've been told by Carl Shapiro at Facebook is that we have their
blessing to post it publicly / merge it / build upon it for CPython.
Their patch was written for 3.6, I have massaged it to the point where
it minimally works with 3.8.
What the patch does: it takes all the Python modules that are loaded as
part of interpreter startup and deserializes the marshalled .pyc file
into precreated objects stored as static C data. You add this .C file
to the Python build. Then there's a patch to Python itself (about 250
lines iirc) that teaches it to load modules from these data structures.
I wrote a quick dumb test harness to compare this patch vs 3.8 stock.
It runs a command line 500 times and uses time.perf_counter to time the
process. On a fast quiescent laptop I observe a 21-22% improvement:
cmdline: ['./python', '-c', 'pass']
average time 0.006302303705982922
average time 0.007969956444008858
improvement 0.20924239043734505 %
cmdline: ['./python', '-c', 'import io']
average time 0.006297688038004708
average time 0.007996319670004595
improvement 0.21242667903482038 %
The downside of the patch: for these modules it ignores the Python files
on disk--it doesn't even stat them. If you add stat calls you lose half
of the speed improvement. I believe they added a work-around, where you
can set a flag (command-line? environment variable? I don't know, I
didn't go looking for it) that tells Python "don't use the frozen
modules" and it loads all those files from disk.
I don't propose to merge the patch in its current state. I think it
would need a lot of work both in terms of "doing things the way Python
does it" as well as just code smell (the serializer is implemented in
both C and Python and jumps back and forth, also the build process for
the serialized modules is pretty tiresome).
Is it worth working on?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev