[Cython] dynamically compile and import pure python modules

mark florisson markflorisson88 at gmail.com
Fri Apr 27 22:38:41 CEST 2012

On 27 April 2012 21:16, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 27.04.2012 21:16:
>> On 22 March 2012 20:07, Stefan Behnel wrote:
>>> mark florisson, 22.03.2012 19:50:
>>>> For the fused type runtime dispatch I found it very convenient to use
>>>> the with statement, but that is not supported in Python 2.4. However,
>>>> the compiler could dynamically compile compiler code with the compiler
>>>> itself and import it (pyximport), if it is not needed to compile
>>>> Cython itself. I gave it a try and it seems to work like a charm (but
>>>> probably needs more testing :):
>>>> https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724d7db0ace99d89b#diff-2
>>> The advantages are limited, so I'm leaning towards seeing the drawbacks, of
>>> which there are at least some. For one, *running* Cython (as opposed to
>>> installing it) becomes more complex and involves a (much) higher first time
>>> overhead. We'd also start putting shared libraries into user directories
>>> without asking them first. Might be a problem on shared installations with
>>> many users.
>> The overhead would only be for certain python versions that try to use
>> certain functionality, in this case, python2.4 and fused types. To be
>> honest, the overhead isn't very large. As for compiling shared
>> libraries, I don't think people will complain about shared libraries,
>> that's the only way in which Python and Cython can be used.
>>> Note also that Cython no longer compiles itself when installing in PyPy (at
>>> all), but that would be easy to special case here (and PyPy obviously has
>>> features like the "with" statement).
>>> Next, I think it would tempt us to split source files into separate modules
>>> just because that way we can use a specific feature in one of them because
>>> it'll get compiled (and the other half is needed for bootstrapping). That
>>> would be bad design.
>> Possibly. In the case of fused types, the code of the fused node is
>> nearly 800 lines, which is probably good to separate from the other,
>> typically smaller nodes, especially considering it's kind of a
>> specific feature. In my case, and I wouldn't mind limiting the
>> functionality until further discussion to that case only, using the
>> with statement really helps keeping track of blocks, and the resulting
>> code is much more readable than it would otherwise be.
> What about this deal: we remove the hard bootstrap dependency on the fused
> types code (and maybe other Cython specific features) and require its
> compilation at install time in Py2.4 (and maybe even 2.5). That would allow
> us to use newer Python syntax (and even Cython supported syntax) there
> (except for fused types, obviously). Failure to compile the module in
> Python 2.4/5 at install time would then abort the installation. Bad luck
> for the user, but easy to fix by installing a newer Python version.
> That would give us the advantage of not needing to pollute user home
> directories with shared libraries at runtime (which I would consider a very
> annoying property).

I think it's fine to require compiling in the installed case (or will
that be a problem for some package managers?). In the non-installed
case with python versions smaller than needed, would you prefer a
pyximport or an error message telling you to install Cython? Because
for development you really don't want to install every time.

> Making the dependency optional can be as simple as using a try-except
> conditional import at the module level and setting the module name to None
> in the failure case. If we want to prevent weird error messages, we can
> just ignore the import failure during Cython's own installation (by setting
> a flag somewhere) and during a normal run, we use a guard that checks that
> the import worked (i.e. the module is non-None) before starting the
> compilation and raises an error otherwise.
> We should then clearly document in the module comment that this source file
> can use newer syntax while all other files cannot. Oh, and we should still
> take care *not* to split our source base by the "can be compiled or not"
> predicate.


> What do you think?
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

More information about the cython-devel mailing list