[Cython] Utility Codes and templates

[Stefan Behnel]

mark florisson, 22.07.2011 15:07:
> On 22 July 2011 14:38, Stefan Behnel wrote:
>> mark florisson, 22.07.2011 13:45:
>>>
>>> On 22 July 2011 13:10, Stefan Behnel wrote:
>>>>
>>>> mark florisson, 22.07.2011 12:12:
>>>>>
>>>>> For my work on the _memview branch (and also on fused types) I noticed
>>>>> that UtilityCodes started weighing heavily on me in their current
>>>>> form, so I wrote a little loader in the _memview branch:
>>>>>
>>>>>
>>>>>
>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>>>
>>>>> The idea is simple: you put your utility codes in Cython/Utility in
>>>>> .pyx, .c, .h files etc, and then load them. It works for both
>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>>>>
>>>>> myutility.c
>>>>>
>>>>> // UtilityProto: MyUtility
>>>>> header code here
>>>>>
>>>>> // UtilityCode: MyUtility
>>>>> implementation code here
>>>>>
>>>>> You can add as many other utilities as you like to the same file. You
>>>>> can then load it using
>>>>>
>>>>>      UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>>
>>>> Why not have exactly one per file? They can't (or at least shouldn't) be
>>>> interdependent anyway, since they're always loaded and injected
>>>> separately.
>>>> Having one per file makes it easy to take the file name and grep for it.
>>>
>>> Putting them in one file does not make them interdependent
>>
>> I meant the opposite direction. They shouldn't depend on each other anyway,
>> so there's no need to put them in one file.
>
> Say we have one struct definition, and two utilities that depend on
> it, but not directly on one another. I want all of them in the same
> file, because they are *related*, just not all dependent on each
> other.

Ok, I see the use case now. How do you mark the dependencies within the 
file? And how do you mark dependencies towards other code in other files? 
I.e. how do you map the parameters that UtilityCode() current takes into 
the file format?

You sure don't want to keep those in the Python code that injects (or 
loads) the utility code, do you?


>>> Once the first utility is loaded from the
>>> file, it loads all the utilities as strings from the file and caches
>>> them.
>>
>> Premature optimisation? ;)
>
> Not at all, I have one file with multiple templates, and you can't
> seek to their starting position. Why should you re-read the file every
> time you load just one of them?

You seem to be subscribed to misinterpreting me. ;)

*Iff* you accept that multiple implementations should be in one file, then 
this approach makes sense. I was merely suggesting that the performance is 
not a reason by itself for putting multiple implementations into one file.


>>> You don't want to enforce one utility per file as many utilities
>>> are pretty small, and you want to group utilities based on their
>>> purpose.
>>
>> It substantially complicates the file handling, though.
>
> Complicates? In what way?

In the sense that you have to do something to split the file into separate 
sections before being able to apply the templating engine to it. So you 
have several levels of marker semantics in the file format, one to separate 
the implementations, one to separate the parts of an implementation, and 
one level for the templating language. I call that "complicating the file 
handling".

However, given that you always need to express metadata in some way (e.g. 
for dependencies), I don't think you can do better than with two levels 
anyway, so a third won't hurt *that* much.


>>>>> Of course you can pass in any other arguments, like proto_block, name,
>>>>> etc. You can additionally pass it a dict for formatting (for both the
>>>>> prototypes and the implementation)
>>>>
>>>> Dict? Why not keyword arguments?
>>>
>>> Because (Cython)UtilityCode already takes a bunch of keyword arguments
>>
>> But none that would still matter when you put the code into an external
>> file. IMHO, the file should be as self contained as possible, with the
>> exception of externally provided parameters and (obviously) the decision
>> where the resulting utility code will eventually be injected.
>
> No, it matters. The load() method takes any additional keyword
> arguments that are passed to (Cython)UtilityCode. e.g. you might want
> to pass in the variable 'name' or 'init' into your template, but those
> are already taken.

But why would you need to pass those into the UtilityCode instantiation? As 
I said, the templates should be as self-contained as possible, just as they 
are today, as written in the Python code. Passing "init" as a parameter to 
UtilityCode() doesn't make sense to me. (not sure what exactly the "name" 
parameter is there for, though)

Stefan