<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body><div><div style="font-family: Calibri,sans-serif; font-size: 11pt;">My impression is that this level of optimization does and should not fall within the scope of numpy.. </div></div><div dir="ltr"><hr><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">From: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;"><a href="mailto:sturla.molden@gmail.com">Sturla Molden</a></span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">Sent: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;">9-12-2014 16:02</span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">To: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;"><a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a></span><br><span style="font-family: Calibri,sans-serif; font-size: 11pt; font-weight: bold;">Subject: </span><span style="font-family: Calibri,sans-serif; font-size: 11pt;">[Numpy-discussion] Should ndarray be a context manager?</span><br><br></div><br>I wonder if ndarray should be a context manager so we can write <br>something like this:<br><br><br> with np.zeros(n) as x:<br> [...]<br><br><br>The difference should be that __exit__ should free the memory in x (if <br>owned by x) and make x a zero size array.<br><br>Unlike the current ndarray, which does not have an __exit__ method, this <br>would give precise control over when the memory is freed. The timing of <br>the memory release would not be dependent on the Python implementation, <br>and a reference cycle or reference leak would not accidentally produce a <br>memory leak. It would allow us to deterministically decide when the <br>memory should be freed, which e.g. is useful when we work with large arrays.<br><br><br>A problem with this is that the memory in the ndarray would be volatile <br>with respect to other Python threads and view arrays. However, there are <br>dozens of other ways to produce segfaults or buffer overflows with NumPy <br>(cf. stride_tricks or wrapping external buffers).<br><br><br>Below is a Cython class that does something similar, but we would need <br>to e.g. write something like<br><br> with Heapmem(n * np.double().itemsize) as hm:<br> x = hm.doublearray<br> [...]<br><br>instead of just<br><br> with np.zeros(n) as x:<br> [...]<br><br><br>Sturla<br><br><br># (C) 2014 Sturla Molden<br><br>from cpython cimport PyMem_Malloc, PyMem_Free<br>from libc.string cimport memset<br>cimport numpy as cnp<br>cnp.init_array()<br><br><br>cdef class Heapmem:<br><br> cdef:<br> void *_pointer<br> cnp.intp_t _size<br><br> def __cinit__(Heapmem self, Py_ssize_t n):<br> self._pointer = NULL<br> self._size = <cnp.intp_t> n<br><br> def __init__(Heapmem self, Py_ssize_t n):<br> self.allocate()<br><br> def allocate(Heapmem self):<br> if self._pointer != NULL:<br> raise RuntimeError("Memory already allocated")<br> else:<br> self._pointer = PyMem_Malloc(self._size)<br> if (self._pointer == NULL):<br> raise MemoryError()<br> memset(self._pointer, 0, self._size)<br><br> def __dealloc__(Heapmem self):<br> if self._pointer != NULL:<br> PyMem_Free(self._pointer)<br> self._pointer = NULL<br><br> property pointer:<br> def __get__(Heapmem self):<br> return <cnp.intp_t> self._pointer<br><br> property doublearray:<br> def __get__(Heapmem self):<br> cdef cnp.intp_t n = self._size//sizeof(double)<br> if self._pointer != NULL:<br> return cnp.PyArray_SimpleNewFromData(1, &n,<br> cnp.NPY_DOUBLE, self._pointer)<br> else:<br> raise RuntimeError("Memory not allocated")<br><br> property chararray:<br> def __get__(Heapmem self):<br> if self._pointer != NULL:<br> return cnp.PyArray_SimpleNewFromData(1, &self._size,<br> cnp.NPY_CHAR, self._pointer)<br> else:<br> raise RuntimeError("Memory not allocated")<br><br> def __enter__(self):<br> if self._pointer != NULL:<br> raise RuntimeError("Memory not allocated")<br><br> def __exit__(Heapmem self, type, value, traceback):<br> if self._pointer != NULL:<br> PyMem_Free(self._pointer)<br> self._pointer = NULL<br><br><br><br><br><br>_______________________________________________<br>NumPy-Discussion mailing list<br>NumPy-Discussion@scipy.org<br>http://mail.scipy.org/mailman/listinfo/numpy-discussion<br></body></html>