[Numpy-discussion] Status of NumPy and Python 3.3

[Ondřej Čertík]

On Fri, Aug 3, 2012 at 8:03 AM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
> On Mon, Jul 30, 2012 at 5:00 PM, Ronan Lamy <ronan.lamy at gmail.com> wrote:
>> Le lundi 30 juillet 2012 à 11:07 -0700, Ondřej Čertík a écrit :
>>> On Mon, Jul 30, 2012 at 10:04 AM, Ronan Lamy <ronan.lamy at gmail.com> wrote:
>>> > Le lundi 30 juillet 2012 à 17:10 +0100, Ronan Lamy a écrit :
>>> >> Le lundi 30 juillet 2012 à 04:57 +0100, Ronan Lamy a écrit :
>>> >> > Le lundi 30 juillet 2012 à 02:00 +0100, Ronan Lamy a écrit :
>>> >> >
>>> >> > >
>>> >> > > Anyway, I managed to compile (by blanking
>>> >> > > numpy/distutils/command/__init__.py) and to run the tests. I only see
>>> >> > > the 2 pickle errors from your latest gist. So that's all good!
>>> >> >
>>> >> > And the cause of these errors is that running the test suite somehow
>>> >> > corrupts Python's internal cache of bytes objects, causing the
>>> >> > following:
>>> >> > >>> b'\x01XXX'[0:1]
>>> >> > b'\xbb'
>>> >>
>>> >> The culprit is test_pickle_string_overwrite() in test_regression.py. The
>>> >> test actually tries to check for that kind of problem, but on Python 3,
>>> >> it only manages to trigger it without detecting it. Here's a simple way
>>> >> to reproduce the issue:
>>> >>
>>> >> >>> a = numpy.array([1], 'b')
>>> >> >>> b = pickle.loads(pickle.dumps(a))
>>> >> >>> b[0] = 77
>>> >> >>> b'\x01  '[0:1]
>>> >> b'M'
>>> >>
>>> >> Actually, this problem is probably quite old: I can see it in 1.6.1 w/
>>> >> Python 3.2.3. 3.3 only makes it more visible.
>>> >>
>>> >> I'll open an issue on GitHub ASAP.
>>> >>
>>> > https://github.com/numpy/numpy/issues/370
>>>
>>> Thanks Ronan, nice work!
>>>
>>> Since you looked into this -- do you know a way to fix this? (Both
>>> NumPy and the test.)
>>
>> Pauli found out how to fix the code, so I'll try to send a PR tonight.
>
>
> So this PR is now in and the issue is fixed.
>
> As far as swapping the unicode issues, I finally understand what is
> going on and I posted my current understanding into the Python tracker
> issue (http://bugs.python.org/issue15540) which was recently created
> for this same issue:
>
> http://bugs.python.org/msg167280
>
> but it was determined that it is not a bug in Python so it is closed
> now. Finally, I have submitted a reworked version of my patch here:
>
> https://github.com/numpy/numpy/pull/372
>
> It implements things in a clean way.

Final update: the patch is in, so NumPy now passes all tests in Python 3.3.

There seems to be a better way to support unicode and that is
discussed in another thread.

Ondrej