[Python-Dev] importlib is now bootstrapped (and what that means)

Stefan Behnel stefan_ml at behnel.de
Mon Apr 16 21:17:54 CEST 2012

"Martin v. Löwis", 16.04.2012 16:07:
>> We have other instances of this (e.g. the Objects/typeslots.inc file
>> is generated and checked in), but in the case of importlib, we have
>> to use the ./python binary for freezing to avoid bytecode
>> incompatibilities, which obviously is a problem if ./python isn't
>> built yet.
> As for dependencies on byte code: we could consider using Cython instead
> of freeze (not sure whether Cython would build the bootstrap correctly;
> it may need to be fixed first).

I assume that this would be needed rather early during the interpreter
startup, so you're right that there may be obstacles due to expectations in
the generated C code that may not be there yet. But I think it's worth a
try. Cython is rather predictable in its requirements given the Python
source code. Just run the module through Cython and see what it gives you.
(Note that you may need the latest github master version to make everything
work nicely in Py3.3, I keep adapting things there as I find them - our CI
server runs daily tests against the latest CPython master.)

One thing Cython does during module initialisation is to import the
builtins module using PyModule_AddModule(). Does that work already at this
point? It uses it mostly to cache builtins that the module uses. That can
be disabled, though, and I also wouldn't mind letting Cython put a
preprocessor guard into that code that would let it do something else based
on a macro that CPython defines at this point, maybe even just
Py_BUILD_CORE. We already do these things for all sorts of purposes.

And, obviously, the module init function can also be called directly
instead of running it as part of an import. That's commonly done when using
Cython modules in an embedded Python runtime.

> With that, we would get semi-readable
> source code, which should also play more nicely with hg diffs.

There's still a Cython patch hanging around on github that aims to keep the
C code from changing all that drastically on Python source code changes
(e.g. due to source line references etc.). Might be worth integrating for
something like this. There's also a switch that disables the (helpful)
reproduction of the Python code context in C code comments, in case that
gets in the way for diffs.

> On the down side, we would depend on Cython for evolving .

Right, although not as a strict dependency. The code would still work just
fine in plain Python. But it would depend on Cython for performance. And we
usually recommend to ship the generated C sources anyway to avoid a user
dependency on Cython, so this use case is quite normal.


More information about the Python-Dev mailing list