Too many instantiations with fused type memoryviews
Hi, FYI: Cython instantiates fused type routines with memoryview arguments unnecessarily many times. Example: ``` ctypedef fused idx_t: short long # Cython 0.20.1 instantiates this function 128 times, # even though only 2 would be needed def fubar_128(idx_t[:] a, idx_t[:] b, idx_t[:] c, idx_t[:] d, idx_t[:] e, idx_t[:] f, idx_t[:] g): print("HALLO") def fubar_2(idx_t a, idx_t b, idx_t c, idx_t d, idx_t e, idx_t f, idx_t g): print("HULLO") ``` $ cython cymod.pyx $ cython --version Cython version 0.20.1 $ grep -c 'print("HALLO")' cymod.c 128 $ grep -c 'print("HULLO")' _cymod.c 2 The n**m explosion starts to hurt quite quickly when there are several array arguments and more than one fused type. I think this issue is also accompanied by some signature resolution bugs (I'll try to come up with an example case). Cheers, -- Pauli Virtanen
Pauli Virtanen, 08.03.2014 21:11:
FYI: Cython instantiates fused type routines with memoryview arguments unnecessarily many times.
Example: ``` ctypedef fused idx_t: short long
# Cython 0.20.1 instantiates this function 128 times, # even though only 2 would be needed def fubar_128(idx_t[:] a, idx_t[:] b, idx_t[:] c, idx_t[:] d, idx_t[:] e, idx_t[:] f, idx_t[:] g): print("HALLO")
def fubar_2(idx_t a, idx_t b, idx_t c, idx_t d, idx_t e, idx_t f, idx_t g): print("HULLO") ``` $ cython cymod.pyx $ cython --version Cython version 0.20.1 $ grep -c 'print("HALLO")' cymod.c 128 $ grep -c 'print("HULLO")' _cymod.c 2
The n**m explosion starts to hurt quite quickly when there are several array arguments and more than one fused type. I think this issue is also accompanied by some signature resolution bugs (I'll try to come up with an example case).
Yes, this is a known problem. The work-around is to ctypedef the memory view type, not just the dtype. http://thread.gmane.org/gmane.comp.python.cython.user/10437 Stefan
08.03.2014 22:40, Stefan Behnel kirjoitti: [clip]
Yes, this is a known problem. The work-around is to ctypedef the memory view type, not just the dtype.
Thanks, I missed that. Re: arguments in the previous thread --- I think it would indeed be better if memoryviews would not invoke cross-product behavior. If scalars work in one way, also arrays should work in the same way. Pauli
Pauli Virtanen, 08.03.2014 21:59:
08.03.2014 22:40, Stefan Behnel kirjoitti: [clip]
Yes, this is a known problem. The work-around is to ctypedef the memory view type, not just the dtype.
Thanks, I missed that.
Re: arguments in the previous thread --- I think it would indeed be better if memoryviews would not invoke cross-product behavior. If scalars work in one way, also arrays should work in the same way.
I'm not a fused types user myself, but it seems to me that it's worth changing this behaviour for 0.21 to better match the most common use cases. A pull request that implements this certainly won't hurt, if only to get real-world feedback on the change. The work-around for (existing) code that actually wants the cross product behaviour for memory views (for whatever reason) would then be (I guess) to use two identical typedefs with different names. This sounds acceptable, given that the cross product suggests that we are dealing with independent (i.e. different) kinds of data anyway. Stefan
Hi, 22.03.2014 12:08, Stefan Behnel kirjoitti: [clip]
Re: arguments in the previous thread --- I think it would indeed be better if memoryviews would not invoke cross-product behavior. If scalars work in one way, also arrays should work in the same way.
I'm not a fused types user myself, but it seems to me that it's worth changing this behaviour for 0.21 to better match the most common use cases. A pull request that implements this certainly won't hurt, if only to get real-world feedback on the change.
The work-around for (existing) code that actually wants the cross product behaviour for memory views (for whatever reason) would then be (I guess) to use two identical typedefs with different names. This sounds acceptable, given that the cross product suggests that we are dealing with independent (i.e. different) kinds of data anyway.
I think another argument against the cross-product behavior is that constructs such as cdef idx_t foo and x = <idx_t>y and if idx_t is np.int: # FOO else: # BAR all become ambiguous in the function body, if idx_t is instantiated in several different ways. I can take a look at this today... (Re: Sturla: I *want* a bloatware generator, but a controllable one :) -- Pauli Virtanen
22.03.2014 14:48, Pauli Virtanen kirjoitti: [clip]
I can take a look at this today...
Here's one attempt: https://github.com/cython/cython/pull/284 In the long run, I'd like to be able to write code like this: https://github.com/pv/scipy-work/blob/csparsetools-nocrud/scipy/sparse/_cspa... with fused types and have it just work. The above change gets it mostly towards the goal, apart from issues with Numpy scalars and numpy boolean arrays. I'll probably look at this later on... -- Pauli Virtanen
Pauli Virtanen <pav@iki.fi> wrote:
The n**m explosion starts to hurt quite quickly when there are several array arguments and more than one fused type. I think this issue is also accompanied by some signature resolution bugs (I'll try to come up with an example case).
I warned that fused types would be a bloatware generator. This behavior is "by design". https://mail.python.org/pipermail/cython-devel/2011-May/000670.html https://mail.python.org/pipermail/cython-devel/2011-May/000648.html Sturla
participants (3)
-
Pauli Virtanen -
Stefan Behnel -
Sturla Molden