[Numpy-discussion] nditer as a context manager (reformatted?)

[Matti Picus]

Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes 
an update to the nditer API, both C and python. The issue 
(https://github.com/numpy/numpy/issues/9714) is that sometimes nditer 
uses temp arrays via the "writeback" mechanism, the data is copied back 
to the original arrays "when finished". However "when finished" was 
implemented using nditer deallocation.

This mechanism is implicit and unclear, and relies on refcount semantics 
which do not work on non-refcount python implementations like PyPY. It 
also leads to lines of code like "iter=None" to trigger the writeback 
resolution.

On the c-api level the agreed upon solution is to add a new 
`NpyIter_Close` function in C, this is to be called before 
`NpyIter_Dealloc`.

The reviewers and I would like to ask the wider NumPy community for 
opinions about the proposed python-level solution: turning the python 
nditer object into a context manager. This way "writeback" occurs at 
context manager exit via a call to `NpyIter_Close`, instead of like 
before when it occurred at `nditer` deallocation (which might not happen 
until much later in Pypy, and could be delayed by GC even in Cpython).

Another solution that was rejected 
(https://github.com/numpy/numpy/pull/10184) was to add an nditer.close() 
python-level function that would not require a context manager It was 
felt that this is more error-prone since it requires users to add the 
line for each iterator created.

The back-compat issues are that:

1. We are adding a new function to the numpy API, `NpyIter_Close` 
(pretty harmless)

2. We want people to update their C code using nditer, to call 
`NpyIter_Close` before  they call `NpyIter_Dealloc` and will start 
raising a deprecation warning if misuse is detected

3. We want people to update their Python code to use the nditer object 
as a context manager, and will warn if they do not.

We tried to minimize back-compat issues, in the sense that old code 
(which didn't work in PyPy anyway) will still work, although it will now 
emit deprecation warnings. In the future we also plan to raise an error 
if an nditer is used in Python without a context manager (when it should 
have been). For C code, we plan to leave the deprecation warning in 
place probably forever, as we can only detect the deprecated behavior in 
the deallocator, where exceptions cannot be raised.

Anybody who uses nditers should take a look and please reply if it seems 
the change will be too painful.

For more details, please see the updated docs in that PR

Matti (and reviewers)