Tim Delaney, 27.10.2012 22:53:
On 28 October 2012 07:40, Mark Shannon wrote:
I suspect that stating and loading the .pyc files is responsible for most of the overhead. PyRun starts up quite a lot faster thanks to embedding all the modules in the executable: http://www.egenix.com/**products/python/PyRun/http://www.egenix.com/products/python/PyRun/
Freezing all the core modules into the executable should reduce start up time.
That suggests a test to me that the Cython guys might be interested in (or may well have performed in the past). How much of the stdlib could be compiled with Cython and used during the startup process?
We have a Jenkins job set up to run the CPython test suite with a compiled stdlib:
Basically, we use pyximport as an import hook that tries to compile Python modules on import and then imports the shared library if it worked or the original Python module if it failed. A solution that explicitly runs over the stdlib and compiles it would be substantially cleaner and more stable.
I don't have numbers for Py3.4 because we currently have a hard crash in one of the tests on that platform when compiling recursively on import (likely meaning that one of the stdlib modules and/or tests would have to be excluded from compilation), but I get 434 automatically compiled stdlib modules for the latest Py2.7 branch out of 744 (excluding the test suite). And Py3.x code tends to pass as least as well through the compiler, often better.
Note that quite a number of modules are excluded accidentally because they are already imported as Python modules when Cython starts working. Compiling them explicitly would remove that limitation, maybe adding another (wild guess) 50 modules or so. Another few are not being compiled because the test module that uses them fails to compile. So missing shared libraries are not always due to failures to compile that particular Python module.
I didn't pay much attention to this part of our integration tests so far - a bit of debugging should get the Py3.4 build working.
How much of an effect would it have on startup times and these benchmarks if Cython-compiled extensions were used?
Depends on what and how much code you use. If you compile everything into one big module that "imports" all of the stdlib when it gets loaded, you'd likely loose a lot of time because it would take a while to initialise all that useless code on startup. If you keep it separate, it would likely be a lot faster because you avoid the interpreter for most of the module startup.
Most Python code runs about 30% faster when compiled, some faster, some slower. If you want better numbers, you can start optimising the code by giving Cython static type hints. I did that for difflib a while ago, for example. Changing two methods made it some 50% faster back then:
That particular module should compile without changes these days, and you can provide the type hints externally, i.e. without modifying the Python code itself.
I'm thinking here of elimination of .pyc interpretation and execution (stat calls would be similar, probably slightly higher).
CPython checks for .so files before looking for .py files and imports are absolute by default in Py3, so there should be a slight reduction in stat calls. The net result then obviously also depends on how fast your shared library loader and linker is, etc., but I doubt that that path is any slower than loading and running a .pyc file.
BTW, you'd still get nice stack traces for compiled modules as long as your .py files lie right next to your .so files.
To be clear - I'm *not* suggesting Cython become part of the required build toolchain. But *if* the Cython-compiled extensions prove to be significantly faster I'm thinking maybe it could become a semi-supported option (e.g. a HOWTO with the caveat "it worked on this particular system").