[Cython] Utility Codes and templates

mark florisson markflorisson88 at gmail.com
Fri Jul 22 15:07:18 CEST 2011

On 22 July 2011 14:38, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 22.07.2011 13:45:
>> On 22 July 2011 13:10, Stefan Behnel wrote:
>>> mark florisson, 22.07.2011 12:12:
>>>> For my work on the _memview branch (and also on fused types) I noticed
>>>> that UtilityCodes started weighing heavily on me in their current
>>>> form, so I wrote a little loader in the _memview branch:
>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>> The idea is simple: you put your utility codes in Cython/Utility in
>>>> .pyx, .c, .h files etc, and then load them. It works for both
>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>>> myutility.c
>>>> // UtilityProto: MyUtility
>>>> header code here
>>>> // UtilityCode: MyUtility
>>>> implementation code here
>>>> You can add as many other utilities as you like to the same file. You
>>>> can then load it using
>>>>     UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>> Why not have exactly one per file? They can't (or at least shouldn't) be
>>> interdependent anyway, since they're always loaded and injected
>>> separately.
>>> Having one per file makes it easy to take the file name and grep for it.
>> Putting them in one file does not make them interdependent
> I meant the opposite direction. They shouldn't depend on each other anyway,
> so there's no need to put them in one file.

Say we have one struct definition, and two utilities that depend on
it, but not directly on one another. I want all of them in the same
file, because they are *related*, just not all dependent on each
For instance, I have a memviewslice (which is actually a struct) with
different "methods". Depending on what methods are used they are
included. I certainly want all of them in the same file, but as
separate utilities.

>> Once the first utility is loaded from the
>> file, it loads all the utilities as strings from the file and caches
>> them.
> Premature optimisation? ;)

Not at all, I have one file with multiple templates, and you can't
seek to their starting position. Why should you re-read the file every
time you load just one of them? But sure, I haven't timed it. But I'm
sure it will make a difference if I put it on a network share.

>> You don't want to enforce one utility per file as many utilities
>> are pretty small, and you want to group utilities based on their
>> purpose.
> It substantially complicates the file handling, though.

Complicates? In what way? It means I don't have a gazillion files, but
just a MemoryView.c and a MemoryView.pyx (actually I renamed the
former to avoid confusion, as people may think it's the cythonization
of the .pyx). That way I can easily oversee the code, and people will
have no trouble locating it.

>> You may still want to pass
>> the filename as you might want a .c and a corresponding .h (or a .pyx
>> and a corresponding .pxd), or simply one .pyx and one .c, one for the
>> Cython utility code and one for the C utility code.
> Do you have a use case for that? For example, when would you need a .h file?
> AFAICT, that's unprecedented in Cython. Plus, I would expect that files with
> different extensions would also have different names as they contain
> different things.
>> Of course, we could introduce a method load_utility() which takes just
>> "MyUtility" and loads it from MyUtility.*, whichever one it finds
>> first.
> I'm not talking about convenience functionality here, it's more of a basic
> design question.
>>>> Of course you can pass in any other arguments, like proto_block, name,
>>>> etc. You can additionally pass it a dict for formatting (for both the
>>>> prototypes and the implementation)
>>> Dict? Why not keyword arguments?
>> Because (Cython)UtilityCode already takes a bunch of keyword arguments
> But none that would still matter when you put the code into an external
> file. IMHO, the file should be as self contained as possible, with the
> exception of externally provided parameters and (obviously) the decision
> where the resulting utility code will eventually be injected.

No, it matters. The load() method takes any additional keyword
arguments that are passed to (Cython)UtilityCode. e.g. you might want
to pass in the variable 'name' or 'init' into your template, but those
are already taken.

>>> I'd prefer an interface like this:
>>>  UtilityCode.load_from_file("myutility",
>>>          some_format_arg="somename", some_repeat_arg = 2)
>>> That would automatically look up the corresponding files
>>>    Cython/UtilityCode/myutility.{pyx,c,cpp,h}
>>> and take whichever it finds (first), then run the template engine on it.
>>> I don't think we need to support separate arguments for prototype and
>>> code
>>> section, they can just use different names. Keep it simple.
>> Sure, that's also fine.
> Well, it's a much simpler interface to start with.
>>>> It will return a UtilityCode instance ready for use.
>>>> You can also simply retrieve a utility code as a string, where it
>>>> returns (proto, implementation).
>>>> As debated before, an actual template library would be really
>>>> convenient. Dag and I had a discussion about it and he suggested
>>>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3,
>>>> and is pure-python. It supports all the good things like iteration,
>>>> template inheritance, etc. Now I'm not sure whether it supports python
>>>> 2.3 as it doesn't compile on my system, but it does support 2.4
>>>> (confirmation for 2.3 would be appreciated). On a side note, I'd be
>>>> perfectly happy to drop support for 2.3, it's kind of a chore.
>>>> The documentation for Tempita can be found here:
>>>> http://pythonpaste.org/tempita/
>>>> That way we might rid ourselves of a lot of code.putln() and move
>>>> those to template utilities instead (or at least prevent writing more
>>>> of those). What do you guys think?
>>> I'm fine with using a template engine for the more involved cases (which
>>> are
>>> rare enough). However, I'd prefer not adding a new dependency, but just
>>> shipping a tiny single-module engine with Cython, e.g. Templite or
>>> pyratemp
>>> (just found them, never used them).
>> It's far from rare, you could then use it not only for utility codes,
>> but for general code generation.
> At least I find it rare. I never felt the need to use a template engine,
> simply because most of the generated code is either static (utility code) or
> so highly parametrised that it's better to generate the code
> programmatically than to use a template. There are exceptions, sure, but I
> consider them exceptions, and thus rare.
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

More information about the cython-devel mailing list