[Numpy-discussion] nditer as a context manager

[Matti Picus]

|Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes 
an update to the nditer API, both C and python. The issue (link) is that |||sometimes nditer uses temp arrays via the "writeback" mechanism, the 
data is copied back to the original arrays "when finished". However 
"when finished" was implemented using nditer deallocation. |This 
mechanism is implicit and unclear, and relies on refcount semantics 
which do not work on non-refcount python implementations like PyPY. It 
also leads to lines of code like "iter=None" to trigger the writeback 
resolution. On the c-api level the agreed upon solution is to add a new |||`NpyIter_Close` function in C, this is to be called before 
`NpyIter_Dealloc`. The reviewers and I would like to ask the wider NumPy 
community for opinions about the proposed python-level solution: 
|turning the python nditer object into a context manager. This way 
"writeback" occurs at context manager exit via a call to 
`NpyIter_Close`, instead of like before when it occurred at nditer 
deallocation (which might not happen until much later in Pypy, and could 
be delayed by GC even in Cpython). Another solution that was rejected 
(https://github.com/numpy/numpy/pull/10184) was to add an nditer.close() 
python-level function that would not require a context manager It was 
felt that this is more error-prone since it requires users to add the 
line for each iterator created. The back-compat issues are that: 1. We 
are adding a new function to the numpy API, `NpyIter_Close` (pretty 
harmless) 2. We want people to update their C code using nditer, to call 
`NpyIter_Close` before  they call `NpyIter_Dealloc` and will start 
raising a deprecation warning if misuse is detected 3. We want people to 
update their Python code to use the nditer object as a context manager, 
and will warn if they do not. We tried to minimize back-compat issues, 
in the sense that old code (which didn't work in PyPy anyway) will still 
work, although it will now emit deprecation warnings. In the future we 
also plan to raise an error if an nditer is used in Python without a 
context manager (when it should have been). For C code, we plan to leave 
the deprecation warning in place probably forever, as we can only detect 
the deprecated behavior in the deallocator, where exceptions cannot be 
raised. Anybody who uses nditers should take a look and please reply if 
it seems the change will be too painful. For more details, please see 
the updated docs in that PR Matti (and reviewers) |