[Cython] Utility Codes and templates

[Robert Bradshaw]

On Mon, Jul 25, 2011 at 11:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 26.07.2011 07:00:
>>
>> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote:
>>>
>>> I think that would seriously help with  moving shared C-code into
>>> cython library (.h and .so).
>>>
>>> Some things like generator class implementation and cyfunction could
>>> be move into .so file.
>>
>> Yes, that would be good.
>
> Well, *if* we find a way to make sure the additional modules get properly
> distributed. It's not obvious to me how to do that, certainly not in a
> cross-package way, and not even in a per-package way, as that would still
> require some kind of interaction with distutils and maybe even changes to
> the users' setup.py scripts (hopefully in a way that doesn't require them to
> know what exact additional modules will be generated).

This is where the ext_modules = Cython.Build.cythonize(...) pattern
could come in handy. We'd just add an extra module to the list. (The
generated .c file could also be shipped, but might have to be listed
explicitly.)

>> Of course we don't want a single .so file for
>> every tiny chunk of utility code, but one needs to be careful lumping
>> them together as well (e.g. things may compile differently depending
>> on the set of headers included in the .pyx file). To do this right I
>> we need to compile things at a higher level than module-by-module with
>> Cython.Build.cythonize or similar.
>>
>> Certainly CyFunction and the generator class are easy and high-benefit
>> targets (compared to, e.g. tuple indexing utility code).
>
> Now that we have a way to group utility code in a single file, including
> automatically resolved dependencies to other files, it should be trivial to
> take such a file and to compile it into a separate module. Cython modules
> would then trivially export their public names anyway, and utility code in C
> files could have an additional Cython wrapper file that would simply depend
> on the C file and that properly exports the C functions through a Python API
> and inter-module C-API.
>
> We could even write empty utility files that only contain dependencies, thus
> grouping together multiple utility files to one larger module.

I was thinking about stuff like how we handle complex numbers where
the pre-processor has a large role to play. Low-level stuff like this
probably belongs in a .h file rather than a .so file though. I'm also
wondering if there are any efficiency gains to be had from actually
linking the resulting .so file at compile time rather than using the
runtime inter-module C-API stuff.

- Robert