Cache an entire import object graph
Hi, I would like some feedback on the feasibility of an idea I've had: caching an entire import graph to speed up interactive apps. The idea is that tools like pytest, pip, hg, etc would be able to save and restore an import graph to cut down on initial startup time. As I think of it, this will be an opt-in system where a cache file could be stored on disk and loaded quickly and then gone through with one or two passes to relocate pointers to the present values if needed. The details of the API to be determined later. I'm thinking it would require changes to pip/setuptools to clear these caches when a package is upgraded for example. I have very limited experience with the CPython code base so I'd like to know if this is even a feasible thing to do? Best regards Anders
I encourage you to search this list (and maybe Python-dev) for a discussion (maybe a year ago or so?) about speeding up interpreter startup. In particular, there were ideas about having a bi art pre-loaded with all the standard modules that get loaded every time. The conversation petered out, but there were some good ideas on the table that could be further explored. -CHB On Mon, Sep 16, 2019 at 4:20 AM Anders Hovmöller <boxed@killingar.net> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I'd hacked together an implementation of this idea last year for Python 3.6 and even gave a talk[1] on it at last year's EuroPython. Larry Hastings was interested in this idea, so I'd sent him my patch. He seems to have ported it to Python 3.8 and it's on this issue[2] on the bug tracker. - Jeethu [1]: https://www.youtube.com/watch?v=KRqv2Bm1J18 [2]: https://bugs.python.org/issue34690 ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, September 16, 2019 12:19 PM, Anders Hovmöller <boxed@killingar.net> wrote:
Hi I'm interested in this area. Jeethu's slides are available at https://ep2018.europython.eu/media/conference/slides/faster-python-startup.p... By the way, TeX has a \dump command, which also makes it an example of an "image-based language", as in Jeethu's slides. LaTeX is an example of a \dump used to make an executable (together with a run-time library). https://tug.org/texinfohtml/web2c.html#Memory-dumps best regards Jonathan
I encourage you to search this list (and maybe Python-dev) for a discussion (maybe a year ago or so?) about speeding up interpreter startup. In particular, there were ideas about having a bi art pre-loaded with all the standard modules that get loaded every time. The conversation petered out, but there were some good ideas on the table that could be further explored. -CHB On Mon, Sep 16, 2019 at 4:20 AM Anders Hovmöller <boxed@killingar.net> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I'd hacked together an implementation of this idea last year for Python 3.6 and even gave a talk[1] on it at last year's EuroPython. Larry Hastings was interested in this idea, so I'd sent him my patch. He seems to have ported it to Python 3.8 and it's on this issue[2] on the bug tracker. - Jeethu [1]: https://www.youtube.com/watch?v=KRqv2Bm1J18 [2]: https://bugs.python.org/issue34690 ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, September 16, 2019 12:19 PM, Anders Hovmöller <boxed@killingar.net> wrote:
Hi I'm interested in this area. Jeethu's slides are available at https://ep2018.europython.eu/media/conference/slides/faster-python-startup.p... By the way, TeX has a \dump command, which also makes it an example of an "image-based language", as in Jeethu's slides. LaTeX is an example of a \dump used to make an executable (together with a run-time library). https://tug.org/texinfohtml/web2c.html#Memory-dumps best regards Jonathan
participants (4)
-
Anders Hovmöller -
Christopher Barker -
jeethu -
Jonathan Fine