[Cython] Utility Codes and templates

mark florisson markflorisson88 at gmail.com
Fri Jul 22 16:45:26 CEST 2011

On 22 July 2011 16:31, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 22.07.2011 16:11:
>> On 22 July 2011 15:54, Stefan Behnel wrote:
>>> mark florisson, 22.07.2011 15:07:
>>>> On 22 July 2011 14:38, Stefan Behnel wrote:
>>>>> mark florisson, 22.07.2011 13:45:
>>>>>> On 22 July 2011 13:10, Stefan Behnel wrote:
>>>>>>> mark florisson, 22.07.2011 12:12:
>>>>>>>> For my work on the _memview branch (and also on fused types) I
>>>>>>>> noticed
>>>>>>>> that UtilityCodes started weighing heavily on me in their current
>>>>>>>> form, so I wrote a little loader in the _memview branch:
>>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in
>>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both
>>>>>>>> prototypes and implementations, for UtilityCode and
>>>>>>>> CythonUtilityCode:
>>>>>>>> myutility.c
>>>>>>>> // UtilityProto: MyUtility
>>>>>>>> header code here
>>>>>>>> // UtilityCode: MyUtility
>>>>>>>> implementation code here
>>>>>>>> You can add as many other utilities as you like to the same file.
>>>>>>>> You
>>>>>>>> can then load it using
>>>>>>>>     UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>>>>> Why not have exactly one per file? They can't (or at least shouldn't)
>>>>>>> be
>>>>>>> interdependent anyway, since they're always loaded and injected
>>>>>>> separately.
>>>>>>> Having one per file makes it easy to take the file name and grep for
>>>>>>> it.
>>>>>> Putting them in one file does not make them interdependent
>>>>> I meant the opposite direction. They shouldn't depend on each other
>>>>> anyway,
>>>>> so there's no need to put them in one file.
>>>> Say we have one struct definition, and two utilities that depend on
>>>> it, but not directly on one another. I want all of them in the same
>>>> file, because they are *related*, just not all dependent on each
>>>> other.
>>> Ok, I see the use case now. How do you mark the dependencies within the
>>> file? And how do you mark dependencies towards other code in other files?
>>> I.e. how do you map the parameters that UtilityCode() current takes into
>>> the
>>> file format?
>>> You sure don't want to keep those in the Python code that injects (or
>>> loads)
>>> the utility code, do you?
>> That's what I currently do. You load the utilities with the normal
>> arguments, except that the implementation and prototypes are
>> automatically fetched from a utility code file. So you do
>> my_utility = UtilityCode.load("MyUtility", proto_block='...',
>> requires=[my_other_utility])
> Ah, ok, I hadn't noticed that in your commit. That doesn't make much sense.
> It's the implementation that knows what it depends on, not the code that
> uses it. It's not uncommon for utility code to be used in multiple places,
> it's bad to require all of them to agree on the same configuration for that
> code. That's why I'm saying that the external file should be self-contained
> in this regard. The dependencies must be explicit and right where the code
> is.
Point taken, it was mainly implemented this way because it was easiest
and still premature. I'll support at least the requires clause.

>> It wouldn't be hard to support requirements (and other options) in
>> utility code files though. Do you think we should?
> Sure. The code that injects utility code should only have to know its name
> and what cnames (functions etc.) it provides, and shouldn't have to know
> anything about its implementation details. Exactly as it works now.

This is about loading the utility code, not about injecting them. You
usually assign them to global variables, just like before.

>>>>>> You don't want to enforce one utility per file as many utilities
>>>>>> are pretty small, and you want to group utilities based on their
>>>>>> purpose.
>>>>> It substantially complicates the file handling, though.
>>>> Complicates? In what way?
>>> In the sense that you have to do something to split the file into
>>> separate
>>> sections before being able to apply the templating engine to it. So you
>>> have
>>> several levels of marker semantics in the file format, one to separate
>>> the
>>> implementations, one to separate the parts of an implementation, and one
>>> level for the templating language. I call that "complicating the file
>>> handling".
>>> However, given that you always need to express metadata in some way (e.g.
>>> for dependencies), I don't think you can do better than with two levels
>>> anyway, so a third won't hurt *that* much.
>> Actually, the prototype and the implementation do not need to be
>> grouped. i.e. it is valid to first list all prototypes and then all
>> implementations.
> I understand that, and it makes sense to support that.
>> So yeah, there would be two levels instead of one,
> Well, three instead of two. But I see how you want to mingle the third level
> into the second. That should work.
>> but that's what you want, because you want different parameters for
>> different utilities in the file, and you may load them at different
>> times or even multiple times with different parameters.
> Right, that's why it would be easier with separate files.

Unless you think passing in a filename is hard, separating the
utilities in different files doesn't make it easier in any way. For my
utilities I simply created a function load_memview_cy_utility() that
eats the name and parameters and loads it from the right file. I think
this two-line function is well-worth the convenience of the grouping.

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

More information about the cython-devel mailing list