Given the lack of objections, we are probably going forward with this change to nditer. Anyone who uses nditers may have to update their code slightly if they want to avoid deprecation warnings, but otherwise old nditer code should work for a long time from now. Allan On 03/22/2018 01:43 PM, Matti Picus wrote:
Hello all, PR #9998 (https://github.com/numpy/numpy/pull/9998/) proposes an update to the nditer API, both C and python. The issue (https://github.com/numpy/numpy/issues/9714) is that sometimes nditer uses temp arrays via the "writeback" mechanism, the data is copied back to the original arrays "when finished". However "when finished" was implemented using nditer deallocation.
This mechanism is implicit and unclear, and relies on refcount semantics which do not work on non-refcount python implementations like PyPY. It also leads to lines of code like "iter=None" to trigger the writeback resolution.
On the c-api level the agreed upon solution is to add a new `NpyIter_Close` function in C, this is to be called before `NpyIter_Dealloc`.
The reviewers and I would like to ask the wider NumPy community for opinions about the proposed python-level solution: turning the python nditer object into a context manager. This way "writeback" occurs at context manager exit via a call to `NpyIter_Close`, instead of like before when it occurred at `nditer` deallocation (which might not happen until much later in Pypy, and could be delayed by GC even in Cpython).
Another solution that was rejected (https://github.com/numpy/numpy/pull/10184) was to add an nditer.close() python-level function that would not require a context manager It was felt that this is more error-prone since it requires users to add the line for each iterator created.
The back-compat issues are that:
1. We are adding a new function to the numpy API, `NpyIter_Close` (pretty harmless)
2. We want people to update their C code using nditer, to call `NpyIter_Close` before they call `NpyIter_Dealloc` and will start raising a deprecation warning if misuse is detected
3. We want people to update their Python code to use the nditer object as a context manager, and will warn if they do not.
We tried to minimize back-compat issues, in the sense that old code (which didn't work in PyPy anyway) will still work, although it will now emit deprecation warnings. In the future we also plan to raise an error if an nditer is used in Python without a context manager (when it should have been). For C code, we plan to leave the deprecation warning in place probably forever, as we can only detect the deprecated behavior in the deallocator, where exceptions cannot be raised.
Anybody who uses nditers should take a look and please reply if it seems the change will be too painful.
For more details, please see the updated docs in that PR
Matti (and reviewers) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion