[Cython] Utility Codes and templates

Robert Bradshaw robertwb at math.washington.edu
Tue Jul 26 07:00:59 CEST 2011

On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2011/7/26 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 25 July 2011 12:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Vitja Makarov, 25.07.2011 10:25:
>>>>> 2011/7/25 Stefan Behnel<stefan_ml at behnel.de>:
>>>>>> Vitja Makarov, 25.07.2011 08:41:
>>>>>>> 2011/7/23 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
>>>>>>>> <markflorisson88 at gmail.com>    wrote:
>>>>>>>>> For my work on the _memview branch (and also on fused types) I noticed
>>>>>>>>> that UtilityCodes started weighing heavily on me in their current
>>>>>>>>> form, so I wrote a little loader in the _memview branch:
>>>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in
>>>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both
>>>>>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>>>>>>> This sounds like it could be a nice way to organize our UtilityCode
>>>>>>>> snippets. So far we haven't really needed any more templating than
>>>>>>>> simple substitution, but for what you're doing I can see this being
>>>>>>>> quite handy. This may also provide a more flexible way forward for
>>>>>>>> supporting multiple backends.
>>>>>>>>> myutility.c
>>>>>>>>> // UtilityProto: MyUtility
>>>>>>>>> header code here
>>>>>>>>> // UtilityCode: MyUtility
>>>>>>>>> implementation code here
>>>>>>>>> You can add as many other utilities as you like to the same file. You
>>>>>>>>> can then load it using
>>>>>>>>>    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>>>>>> I agree with you that having multiple related, named snippets in same
>>>>>>>> file is worthwhile. What about
>>>>>>>> ////////////////////// MyUtility.proto ///////////////////////////
>>>>>>>> and
>>>>>>>> ############ MyCyUtility ##############
>>>>>>>> so the chunks are easy to see.
>>>>>>> C++ comments looks ugly. May be it's better to have something like this:
>>>>>>> /* UtilityCode:  MyUtility.proto */
>>>>>>> and
>>>>>>> # UtilityCode: MyCyUtility
>>>>>>> That's also pretty easy to parse
>>>>>> For the parser, it makes no difference. For a human, it does. A big fat
>>>>>> marker like "/////////////////////" is hard to miss, whereas a tiny one
>>>>>> like
>>>>>> "/* ... */" is easily overlooked within a longer piece of code.
>>>>> Ok, but how many slashes should be there?
>>>>> I'd prefer something like:
>>>>> ########## UtilityCode: MyUtility ################
>>>>> For both C and Cython
>>>> I don't think the parser should care, and I don't think we should put much
>>>> thought into this kind of detail, either. I'd suggest making the separator
>>>> lines at most 78 characters long, and putting as many slashes or hashes in
>>>> there as are needed to fill up the lines after writing the text content.
>>>> What's important to the parser is that there's more than one hash/slash at
>>>> start and end, and something parsable in between. The rest is up to those
>>>> who write the file.
>>> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either
>>> side (symmetry is not enforced).
>> I don't even think we need a max.
>>> It's now 'MyUtility' and
>>> 'MyUtility.proto'. If there's no objection to the ini-style header
>>> (with requirements and other metadata possibly), then I'll implement
>>> that one of these days.
>> One drawback with the ini style is that it detaches the metadata from
>> the code block itself (and require duplicating their names).
> Are we going to move all the existing utility code away from python code?

Yes, though I don't see a huge hurry to get rid of the old style.

> I think that would seriously help with  moving shared C-code into
> cython library (.h and .so).
> Some things like generator class implementation and cyfunction could
> be move into .so file.

Yes, that would be good. Of course we don't want a single .so file for
every tiny chunk of utility code, but one needs to be careful lumping
them together as well (e.g. things may compile differently depending
on the set of headers included in the .pyx file). To do this right I
we need to compile things at a higher level than module-by-module with
Cython.Build.cythonize or similar.

Certainly CyFunction and the generator class are easy and high-benefit
targets (compared to, e.g. tuple indexing utility code).

- Robert

More information about the cython-devel mailing list