Mailman 3 Unreliable crash when converting using numpy.asarray via C buffer interface - NumPy-Discussion

newer
ANN: SfePy 2021.1

Unreliable crash when converting using numpy.asarray via C buffer interface

older
Japanese translation of numpy.org...

Friedrich Romstedt

26 Jan 2021 26 Jan '21

8:48 a.m.

Hi, This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this. The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes:: image = <... Image production ...> ar = numpy.asarray(image) However, when I say:: image = <... Image production ...> print("---") ar = numpy.asarray(image) the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface. The extension type permits reading the pixel values by a method; copying them over by a Python loop works fine. I am ``Py_INCREF``-ing the producer in the C++ buffer view creation function properly. The shapes and strides of the buffer view are ``delete[]``-ed upon releasing the buffer; avoiding this does not prevent the crash. I am catching ``std::exception`` in the view creation function; no such exception occurs. The shapes and strides are allocated by ``new Py_ssize_t[2]``, so they will survive the view creation function. I spent some hours trying to figure out what I am doing wrong. Maybe someone has an idea about this? I double-checked each line of code related to this problem and couldn't find any mistake. Probabaly I am not looking at the right aspect. Best, Friedrich

Show replies by thread

Friedrich Romstedt

1 Feb 1 Feb

7:57 a.m.

Hi, Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt :

...

[...] The following Python code crashes::

image = <... Image production ...> ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...> print("---") ar = numpy.asarray(image)

the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface.

[...]

Does anyone have an idea about this? By the way, I noticed that this mailing list turned pretty quiet, am I missing something? For completeness, the abovementioned "crash" shows up as just a premature exit of the program. There is no error message whatsoever. The buffer view producing function raises Exceptions properly when something goes wrong; also notice that this code completes without error when the ``print("---")`` statement is in action. So I presume the culprit lies somewhere on the C level. I can only guess that it might be some side-effect unknown to me. Best, Friedrich

Hameer Abbasi

8:03 a.m.

Hey Friedrich, If you can produce an MVCE that would be really helpful, along with your hardware and environment. Without that, it isn’t possible to be of much help. https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports Best Regards, Hameer Abbasi -- Sent from Canary (https://canarymail.io)

...

On Dienstag, Jan. 26, 2021 at 9:49 AM, Friedrich Romstedt wrote: Hi,

This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this.

The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes::

image = <... Image production ...> ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...> print("---") ar = numpy.asarray(image)

the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface.

The extension type permits reading the pixel values by a method; copying them over by a Python loop works fine. I am ``Py_INCREF``-ing the producer in the C++ buffer view creation function properly. The shapes and strides of the buffer view are ``delete[]``-ed upon releasing the buffer; avoiding this does not prevent the crash. I am catching ``std::exception`` in the view creation function; no such exception occurs. The shapes and strides are allocated by ``new Py_ssize_t[2]``, so they will survive the view creation function.

I spent some hours trying to figure out what I am doing wrong. Maybe someone has an idea about this? I double-checked each line of code related to this problem and couldn't find any mistake. Probabaly I am not looking at the right aspect.

Best, Friedrich _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Matti Picus

8:46 a.m.

On 2/1/21 9:57 AM, Friedrich Romstedt wrote:

...

Hi,

Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt :

...
[...] The following Python code crashes::

image = <... Image production ...> ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...> print("---") ar = numpy.asarray(image)

the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface.

[...] Does anyone have an idea about this? By the way, I noticed that this mailing list turned pretty quiet, am I missing something?

For completeness, the abovementioned "crash" shows up as just a premature exit of the program. There is no error message whatsoever. The buffer view producing function raises Exceptions properly when something goes wrong; also notice that this code completes without error when the ``print("---")`` statement is in action. So I presume the culprit lies somewhere on the C level. I can only guess that it might be some side-effect unknown to me.

Best, Friedrich

It is very hard to help you from this description. It may be a refcount problem, it may be a buffer protocol problem, it may be something else. Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here). A few things you might want to check: - Make sure you give instructions how to build your project for Linux, since most of the people on this list do not use windows. - There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies. - numpy.asarray has a number of strategies to convert instances, which one is it using? Matti

Friedrich Romstedt

4 Feb 4 Feb

8:07 a.m.

Hello Matti, Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus :

...

[...]

It is very hard to help you from this description. It may be a refcount problem, it may be a buffer protocol problem, it may be something else.

Yes, indeed!

...

Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01 I boiled it down considerably, compared to the program where I stumbled upon the problem. In the abovementioned repo, you find a Python test script in the `test/` folder. Therein, a single `print` statement can be used to trigger or to avoid the error. On Linux, I get a somewhat more precise description than just from the premature exit on Windows: It is a segfault. Certainly it is still asked quite much to skim through my source code, however, I hope that I trimmed it down sufficiently.

...

- Make sure you give instructions how to build your project for Linux, since most of the people on this list do not use windows.

The code reproducing the segfault can be compiled by `$ python3 setup.py install`, both on Windows as well as on Linux.

...

- There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

...

- numpy.asarray has a number of strategies to convert instances, which one is it using?

I've tried to read about this, but coudn't find anything. What are these different strategies? Many thanks in advance, Friedrich

Friedrich Romstedt

15 Feb 15 Feb

9:12 a.m.

Hi, Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt :

...

Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus :

...
Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01

Last week I updated my example code to be more slim. There now exists a single-file extension module: https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin.... The corresponding test program https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-... crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out. My hope to be able to fix my error myself by reducing the code to reproduce the problem has not been fulfillled. I feel that the abovementioned test code is short enough to ask for help with it here. Any hint on how I could solve my problem would be appreciated very much. There are some points which were not clarified yet; I am citing them below. So far, Friedrich

...

...
- There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

...
- numpy.asarray has a number of strategies to convert instances, which one is it using?

I've tried to read about this, but couldn't find anything. What are these different strategies?

Sebastian Berg

3:54 p.m.

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

...

Hi,

Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt :

...
Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus < matti.picus@gmail.com>:

...
Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01

Last week I updated my example code to be more slim. There now exists a single-file extension module: https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin... . The corresponding test program https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-... crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out.

My hope to be able to fix my error myself by reducing the code to reproduce the problem has not been fulfillled. I feel that the abovementioned test code is short enough to ask for help with it here. Any hint on how I could solve my problem would be appreciated very much.

I have tried it out, and can confirm that using debugging tools (namely valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.) Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy. Cheers, Sebastian

...

There are some points which were not clarified yet; I am citing them below.

So far, Friedrich

...
...
- There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

...
- numpy.asarray has a number of strategies to convert instances, which one is it using?

I've tried to read about this, but couldn't find anything. What are these different strategies?

NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Lev Maximov

5:49 p.m.

Hi Friedrich, Try adding view->suboffsets = NULL; view->internal = NULL; to Image_getbuffer Best regards, Lev On Mon, Feb 15, 2021 at 10:57 PM Sebastian Berg wrote:

...

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

...
Hi,

Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt :

...
Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus < matti.picus@gmail.com>:

...
Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01

Last week I updated my example code to be more slim. There now exists a single-file extension module:

https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin...

...
. The corresponding test program

https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-...

...
crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out.

My hope to be able to fix my error myself by reducing the code to reproduce the problem has not been fulfillled. I feel that the abovementioned test code is short enough to ask for help with it here. Any hint on how I could solve my problem would be appreciated very much.

I have tried it out, and can confirm that using debugging tools (namely valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy.

Cheers,

Sebastian

...
There are some points which were not clarified yet; I am citing them below.

So far, Friedrich

...
...
- There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

...
- numpy.asarray has a number of strategies to convert instances, which one is it using?

I've tried to read about this, but couldn't find anything. What are these different strategies?

NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Mansour Moufid

16 Feb 16 Feb

12:35 a.m.

On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt wrote:

...

Hi,

This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this.

The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes::

image = <... Image production ...> ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...> print("---") ar = numpy.asarray(image)

the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface.

Maybe a dereference bug. Try setting pointers to NULL after freeing, something like this: delete[] view->shape; view->shape = NULL; delete[] view->strides; view->strides = NULL; ... delete[] self->data; self->data = NULL;

Mansour Moufid

12:47 a.m.

On Mon, Feb 15, 2021 at 7:35 PM Mansour Moufid wrote:

...

On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt wrote:

...
Hi,

This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this.

The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes::

image = <... Image production ...> ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...> print("---") ar = numpy.asarray(image)

the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface.

Maybe a dereference bug.

Try setting pointers to NULL after freeing, something like this:

delete[] view->shape; view->shape = NULL; delete[] view->strides; view->strides = NULL;

...

delete[] self->data; self->data = NULL;

Sorry for two messages in a row, I just noticed: I don't see the type's tp_free member defined? You can set it to PyObject_Free in Init_ImageType: ImageType.tp_free = PyObject_Free; See here: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_free

Friedrich Romstedt

10 a.m.

Hello again, Am Mo., 15. Feb. 2021 um 16:57 Uhr schrieb Sebastian Berg :

...

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

...
Last week I updated my example code to be more slim. There now exists a single-file extension module: https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin... . The corresponding test program https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-... crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out.

I have tried it out, and can confirm that using debugging tools (namely valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy.

...

From running ``PYTHONMALLOC=malloc valgrind python3 2021-01-11_0909.py`` (with the preceding call of ``print`` in :file:`2021-01-11_0909.py` commented out) I found a few things:

- The call might or might not succeed. It doesn't always lead to a segfault. - "at 0x4A64A73: ??? (in /usr/lib/libpython3.9.so.1.0), called by 0x4A64914: PyMemoryView_FromObject (in /usr/lib/libpython3.9.so.1.0)", a "Conditional jump or move depends on uninitialised value(s)". After one more block of valgrind output ("Use of uninitialised value of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0)"), it finally leads either to "Invalid read of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0) [...] Address 0x1 is not stack'd, malloc'd or (recently) free'd", resulting in a segfault, or just to another "Use of uninitialised value of size 8 at 0x48EEA15: ??? (in /usr/lib/libpython3.9.so.1.0)", after which the program completes successfully. - All this happens within "PyMemoryView_FromObject". So I can only guess that the "uninitialised value" is compared to 0x0, and when it is different (e.g. 0x1), it leads via "Address 0x1 is not stack'd, malloc'd or (recently) free'd" to the segfault observed. I suppose I need to compile Python and numpy myself to see the debug symbols instead of the "???" marks? Maybe even with ``-O0``? Furthermore, the shared object belonging to my code isn't involved directly in any way, so the segfault possibly has to do with some data I am leaving "uninitialised" at the moment. Thanks for the other replies as well; for the moment I feel that going the valgrind way might teach me how to debug errors of this kind myself. So far, Friedrich

Lev Maximov

10:48 a.m.

I've reproduced the error you've described and got rid of it without valgrind. Those two lines are enough to avoid the segfault. But feel free to find it yourself :) Best regards, Lev On Tue, Feb 16, 2021 at 5:02 PM Friedrich Romstedt < friedrichromstedt@gmail.com> wrote:

...

Hello again,

Am Mo., 15. Feb. 2021 um 16:57 Uhr schrieb Sebastian Berg :

...
On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

...
Last week I updated my example code to be more slim. There now exists a single-file extension module:

https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin...

...
...
. The corresponding test program

https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-...

...
crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out.

I have tried it out, and can confirm that using debugging tools (namely valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy.

From running ``PYTHONMALLOC=malloc valgrind python3 2021-01-11_0909.py`` (with the preceding call of ``print`` in :file:`2021-01-11_0909.py` commented out) I found a few things:

- The call might or might not succeed. It doesn't always lead to a segfault. - "at 0x4A64A73: ??? (in /usr/lib/libpython3.9.so.1.0), called by 0x4A64914: PyMemoryView_FromObject (in /usr/lib/libpython3.9.so.1.0)", a "Conditional jump or move depends on uninitialised value(s)". After one more block of valgrind output ("Use of uninitialised value of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0)"), it finally leads either to "Invalid read of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0) [...] Address 0x1 is not stack'd, malloc'd or (recently) free'd", resulting in a segfault, or just to another "Use of uninitialised value of size 8 at 0x48EEA15: ??? (in /usr/lib/libpython3.9.so.1.0)", after which the program completes successfully. - All this happens within "PyMemoryView_FromObject".

So I can only guess that the "uninitialised value" is compared to 0x0, and when it is different (e.g. 0x1), it leads via "Address 0x1 is not stack'd, malloc'd or (recently) free'd" to the segfault observed.

I suppose I need to compile Python and numpy myself to see the debug symbols instead of the "???" marks? Maybe even with ``-O0``?

Furthermore, the shared object belonging to my code isn't involved directly in any way, so the segfault possibly has to do with some data I am leaving "uninitialised" at the moment.

Thanks for the other replies as well; for the moment I feel that going the valgrind way might teach me how to debug errors of this kind myself.

So far, Friedrich _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Friedrich Romstedt

11:40 a.m.

Hi Lev, Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov :

...

I've reproduced the error you've described and got rid of it without valgrind. Those two lines are enough to avoid the segfault.

Okay, good to know, I'll try it! Thanks for looking into it.

...

But feel free to find it yourself :)

Yes :-D Best wishes, Friedrich

Sebastian Berg

4 p.m.

On Tue, 2021-02-16 at 12:40 +0100, Friedrich Romstedt wrote:

...

Hi Lev,

Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov < lev.maximov@gmail.com>:

...
I've reproduced the error you've described and got rid of it without valgrind. Those two lines are enough to avoid the segfault.

Okay, good to know, I'll try it! Thanks for looking into it.

Yeah, sorry if I was too fuzzy. Your error was random, and checking valgrind in that case is often helpful and typically quick (it runs slow, but not much preparation needed). Especially because you reported it succeeding sometimes, where "uninitialized" might help, although I guess a `gdb` backtrace in the crash case might have been just as clear. With debugging symbols in Python (a full debug build makes sense), it mentioned "suboffsets" in a function name for me (maybe when a crash happened), a debug Python will also default to a debug malloc: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONMALLOC Which would not have been very useful here, but could be if you access a Python object after it was free'd for example. Uninitialized + "suboffsets" seemed fairly clear, but I may have underestimated it alot because I recognize "suboffsets" for buffers immediately. Cheers, Sebastian

...

...
But feel free to find it yourself :)

Yes :-D

Best wishes, Friedrich _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Friedrich Romstedt

29 Mar 29 Mar

7:52 a.m.

Hi Matti, Sebastian and Lev, Am Mo., 15. Feb. 2021 um 18:50 Uhr schrieb Lev Maximov :

...

Try adding view->suboffsets = NULL; view->internal = NULL; to Image_getbuffer

finally I got it working easily using Lev's pointer cited above. I didn't follow the valgrind approach furthermore, since I found it likely that it'd produce the same finding. This is just to let you know; I applied the fix several weeks ago. Many thanks, Friedrich

Lev Maximov

8:10 a.m.

I'm glad you sorted it out as the subject line sounded quite horrifying ) Best regards, Lev On Mon, Mar 29, 2021 at 2:54 PM Friedrich Romstedt < friedrichromstedt@gmail.com> wrote:

...

Hi Matti, Sebastian and Lev,

Am Mo., 15. Feb. 2021 um 18:50 Uhr schrieb Lev Maximov < lev.maximov@gmail.com>:

...
Try adding view->suboffsets = NULL; view->internal = NULL; to Image_getbuffer

finally I got it working easily using Lev's pointer cited above. I didn't follow the valgrind approach furthermore, since I found it likely that it'd produce the same finding.

This is just to let you know; I applied the fix several weeks ago.

Many thanks, Friedrich _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Gunter Meissner

5:21 p.m.

Aloha Numpy Community, I am just writing a book on "How to Cheat in Statistics - And get Away with It". I noticed there is no built-in syntax for the 'Adjusted R-squared' in any library (do correct me if I am wrong) I think it would be a good idea to program it. The math is straight forward, I can provide it if desired. Thank you, Gunter On Mon, Feb 15, 2021 at 5:56 AM Sebastian Berg wrote:

...

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

...
Hi,

Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt :

...
Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus < matti.picus@gmail.com>:

...
Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01

Last week I updated my example code to be more slim. There now exists a single-file extension module:

https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bugIhunti... https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntin...

...
. The corresponding test program

https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-...

...
crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out.

My hope to be able to fix my error myself by reducing the code to reproduce the problem has not been fulfillled. I feel that the abovementioned test code is short enough to ask for help with it here. Any hint on how I could solve my problem would be appreciated very much.

I have tried it out, and can confirm that using debugging tools (namely valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy.

Cheers,

Sebastian

...
There are some points which were not clarified yet; I am citing them below.

So far, Friedrich

...
...
- There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

...
- numpy.asarray has a number of strategies to convert instances, which one is it using?

I've tried to read about this, but couldn't find anything. What are these different strategies?

NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

-- Gunter Meissner, PhD University of Hawaii Adjunct Professor of MathFinance at Columbia University and NYU President of Derivatives Software www.dersoft.com CEO Cassandra Capital Management www.cassandracm.com CV: www.dersoft.com/cv.pdf Email: meissner@hawaii.edu Tel: USA (808) 779 3660

1123

Age (days ago)

1185

Last active (days ago)

List overview

Download

16 comments

7 participants

participants (7)

Friedrich Romstedt
Gunter Meissner
Hameer Abbasi
Lev Maximov
Mansour Moufid
Matti Picus
Sebastian Berg

Unreliable crash when converting using numpy.asarray via C buffer interface

tags

participants (7)