[Cython] Conditional import in pure Python mode

[Stefan Behnel]

Robert Bradshaw, 02.05.2012 08:56:
> On Tue, May 1, 2012 at 1:02 PM, Stefan Behnel wrote:
>> Francesc Alted, 01.05.2012 21:49:
>>> On 5/1/12 2:39 PM, mark florisson wrote:
>>>> On 1 May 2012 20:22, Stefan Behnel wrote:
>>>>> Stefan Behnel, 01.05.2012 21:14:
>>>>>> 2) Use math.pxd as an override for the math module. I'm not sure yet how
>>>>>> that would best be made to work, but it shouldn't be all that complex. It
>>>>>> already works (mostly?) for numpy.pxd, for example, although that's done
>>>>>> explicitly in user code.
> 
> math.pxd would be a bit trickier, as we're trying to shadow python
> functions with independent c implementations (rather than declaring
> structure to the single numpy array object and exposing c-level only
> methods. We'd need to support stuff like
> 
> double x = ...
> double y = sin(x) # fast
> cdef object f = sin # grab the builtin one?
> 
> but this is by no means insurmountable and could be really useful.

I already did that for the builtin abs() function. Works nicely so far,
although not from a .pxd but declared internally in Builtin.py.

It's not currently supported for methods (I tried it for one of the builtin
types and it seemed to require more work than I wanted to invest at that
point), but I don't think we need that here. Module level functions should
totally be enough for math.pxd.


>>>>> BTW, I think it would be helpful to make the numpy.pxd cimport automatic as
>>>>> well whenever someone does "import numpy" and friends, right?
>>>>>
>>>> I'm not sure, it means the user has to have numpy development headers.
>>>
>>> But if the user is going to compile a NumPy application, it sounds like
>>> strange to me that she should not be required to install the NumPy
>>> development headers, right?
>>
>> Let's say it's not impossible that someone uses NumPy and Cython without
>> any accelerated C level connection between the two, but it's rather
>> unlikely, given that Cython already has all dependencies that this
>> connection would require as well, except for the NumPy header files.
>>
>> So I would suggest to make the automatic override the default for any
>> module for which a .pxd file with the same fully qualified name is found in
>> the search path, and to require users to explicitly disable this feature
>> for a given module using a module level (or external) compiler directive if
>> they feel like getting slower code (or working around a bug or whatever).
> 
> There is another consideration: this can introduce unnecessary and
> potentially costly dependencies. For example, in Sage one has
> sage/rings/integer.pxd. Not everything that imports from this file
> needs c-level access to the Integer type, and requiring everything
> that imports from sage.rings.integer to be re-compiled when this file
> changes would increase the (admittedly already lengthy) re-compile, as
> well as sucking in a (chain of) un-needed declarations.

Ah, yes, I see your point. The .pxd may actually introduce substantially
more (and different) dependencies than the already compiled .so file. Just
because the import happens in Cython code and not in Python code doesn't
mean it should do different things.


> As Cython
> becomes more and more common, a similar effect could happen between
> projects as well.

Agreed. A compile time C level dependency is much more fragile and version
dependent than a Python level import. I can see a lot of cases where this
matters.


> This may be the exception rather than the rule, so perhaps it's not
> *that* bad to let opt-in be the default, i.e.
> 
> # cython: cimport_on_import = __all__

So, you would allow it to receive either a sequence of explicit module
names or "__all__" to enable it by default, right? Sounds like a reasonable
directive to me.

Stefan