[Cython] dynamically compile and import pure python modules

Stefan Behnel stefan_ml at behnel.de
Fri Apr 27 22:16:38 CEST 2012

mark florisson, 27.04.2012 21:16:
> On 22 March 2012 20:07, Stefan Behnel wrote:
>> mark florisson, 22.03.2012 19:50:
>>> For the fused type runtime dispatch I found it very convenient to use
>>> the with statement, but that is not supported in Python 2.4. However,
>>> the compiler could dynamically compile compiler code with the compiler
>>> itself and import it (pyximport), if it is not needed to compile
>>> Cython itself. I gave it a try and it seems to work like a charm (but
>>> probably needs more testing :):
>>> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2
>> The advantages are limited, so I'm leaning towards seeing the drawbacks, of
>> which there are at least some. For one, *running* Cython (as opposed to
>> installing it) becomes more complex and involves a (much) higher first time
>> overhead. We'd also start putting shared libraries into user directories
>> without asking them first. Might be a problem on shared installations with
>> many users.
> The overhead would only be for certain python versions that try to use
> certain functionality, in this case, python2.4 and fused types. To be
> honest, the overhead isn't very large. As for compiling shared
> libraries, I don't think people will complain about shared libraries,
> that's the only way in which Python and Cython can be used.
>> Note also that Cython no longer compiles itself when installing in PyPy (at
>> all), but that would be easy to special case here (and PyPy obviously has
>> features like the "with" statement).
>> Next, I think it would tempt us to split source files into separate modules
>> just because that way we can use a specific feature in one of them because
>> it'll get compiled (and the other half is needed for bootstrapping). That
>> would be bad design.
> Possibly. In the case of fused types, the code of the fused node is
> nearly 800 lines, which is probably good to separate from the other,
> typically smaller nodes, especially considering it's kind of a
> specific feature. In my case, and I wouldn't mind limiting the
> functionality until further discussion to that case only, using the
> with statement really helps keeping track of blocks, and the resulting
> code is much more readable than it would otherwise be.

What about this deal: we remove the hard bootstrap dependency on the fused
types code (and maybe other Cython specific features) and require its
compilation at install time in Py2.4 (and maybe even 2.5). That would allow
us to use newer Python syntax (and even Cython supported syntax) there
(except for fused types, obviously). Failure to compile the module in
Python 2.4/5 at install time would then abort the installation. Bad luck
for the user, but easy to fix by installing a newer Python version.

That would give us the advantage of not needing to pollute user home
directories with shared libraries at runtime (which I would consider a very
annoying property).

Making the dependency optional can be as simple as using a try-except
conditional import at the module level and setting the module name to None
in the failure case. If we want to prevent weird error messages, we can
just ignore the import failure during Cython's own installation (by setting
a flag somewhere) and during a normal run, we use a guard that checks that
the import worked (i.e. the module is non-None) before starting the
compilation and raises an error otherwise.

We should then clearly document in the module comment that this source file
can use newer syntax while all other files cannot. Oh, and we should still
take care *not* to split our source base by the "can be compiled or not"

What do you think?


More information about the cython-devel mailing list