[Cython] Utility Codes and templates

[Stefan Behnel]

Robert Bradshaw, 26.07.2011 07:00:
> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote:
>> I think that would seriously help with  moving shared C-code into
>> cython library (.h and .so).
>>
>> Some things like generator class implementation and cyfunction could
>> be move into .so file.
>
> Yes, that would be good.

Well, *if* we find a way to make sure the additional modules get properly 
distributed. It's not obvious to me how to do that, certainly not in a 
cross-package way, and not even in a per-package way, as that would still 
require some kind of interaction with distutils and maybe even changes to 
the users' setup.py scripts (hopefully in a way that doesn't require them 
to know what exact additional modules will be generated).


> Of course we don't want a single .so file for
> every tiny chunk of utility code, but one needs to be careful lumping
> them together as well (e.g. things may compile differently depending
> on the set of headers included in the .pyx file). To do this right I
> we need to compile things at a higher level than module-by-module with
> Cython.Build.cythonize or similar.
>
> Certainly CyFunction and the generator class are easy and high-benefit
> targets (compared to, e.g. tuple indexing utility code).

Now that we have a way to group utility code in a single file, including 
automatically resolved dependencies to other files, it should be trivial to 
take such a file and to compile it into a separate module. Cython modules 
would then trivially export their public names anyway, and utility code in 
C files could have an additional Cython wrapper file that would simply 
depend on the C file and that properly exports the C functions through a 
Python API and inter-module C-API.

We could even write empty utility files that only contain dependencies, 
thus grouping together multiple utility files to one larger module.

Stefan