[Cython] [cython-users] malloc vs PyMem_malloc

[mark florisson]

On 10 December 2011 22:00, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 10.12.2011 21:44:
>>
>> On 10 December 2011 20:39, mark florisson wrote:
>>
>>> On 10 December 2011 19:16, Robert Bradshaw wrote:
>>>>
>>>> On this note, a useful pattern is
>>>>
>>>> try:
>>>>    x = malloc(...)
>>>> finally:
>>>>    free(x)
>>>>
>>>> It could be nice to encapsulate this in a context manager.
>
>
> Absolutely.
>
>
>
>>> I think I'd prefer variable-sized arrays that would always get
>>> deallocated on exit of the function
>
>
> Why? A context manager is much clearer

That is highly subjective, I think it would be harder to read and
introduce more code blocks and nesting.

> and gives users total control over
> the lifetime of the memory.
>

Yes, but very often you don't need it. And if Cython would support
declarations in blocks you'd get it for free. Supporting that
(disregarding the difficulties of that) would also be helpful in
identifying the scope and privatization rules in parallel blocks.

The thing is that a context manager would be very Cython-specific,
whereas most people are already familiar with arrays of variable size
from C or Java. Lets compare the following statements and decide which
is more aesthetically pleasing:

    cdef int array1[m]
    cdef double array2[n]

vs

    cdef int *array1
    cdef double *arrays2

    with cython.malloc(sizeof(int) * m), cython.malloc(sizeof(double)
* n) as array1, array2:
        ...

>
>>> (which could be implemented as C99
>>> variable sized arrays, with alloca or with malloc, depending on the
>>> size of the array and the availability of the respective
>>> functionalities).
>
>
> That could still be done for a context manager, just like we do with
> gil/nogil blocks today.
>

Sure (it was more of an observation than an argument).

>
>> That wouldn't tackle every use case, such as for instance mallocing
>> stuff in a parallel section (until we get declarations in blocks!),
>> but special cases can still just malloc and use try blocks, as
>> demonstrated.
>
>
> I would consider the usage of memory over the whole lifetime of a function
> the special case, not the other way round.

Yes, but the point is not where to deallocate the memory, the point is
that you very often don't care. You need it somewhere in the function,
and deallocation on return is fine (or, "at the end of the block").
Analogously, you don't 'del' your variables once you have stopped
using them.

I also gave this functionality some thought for memoryviews, e.g.

    cdef int[:m, :n] myslice # this gets you a view on a cython.array
of size m * n

> Stefan