pytest segfault, not with -v

[Marco Sulla]

I know how to check the refcounts, but I don't know how to check the
memory usage, since it's not a program, it's a simple library. Is
there not a way to check inside Python the memory usage? I have to use
a bash script (I'm on Linux)?

On Sat, 20 Nov 2021 at 19:00, MRAB <python at mrabarnett.plus.com> wrote:
>
> On 2021-11-20 17:40, Marco Sulla wrote:
> > Indeed I have introduced a command line parameter in my bench.py
> > script that simply specifies the number of times the benchmarks are
> > performed. This way I have a sort of segfault checker.
> >
> > But I don't bench any part of the library. I suppose I have to create
> > a separate script that does a simple loop for all the cases, and
> > remove the optional parameter from bench. How boring.
> > PS: is there a way to monitor the Python consumed memory inside Python
> > itself? In this way I could also trap memory leaks.
> >
> I'm on Windows 10, so I debug in Microsoft Visual Studio. I also have a
> look at the memory usage in Task Manager. If the program uses more
> memory when there are more iterations, then that's a sign of a memory
> leak. For some objects I'd look at the reference count to see if it's
> increasing or decreasing for each iteration when it should be constant
> over time.
>
> > On Sat, 20 Nov 2021 at 01:46, MRAB <python at mrabarnett.plus.com> wrote:
> >>
> >> On 2021-11-19 23:44, Marco Sulla wrote:
> >> > On Fri, 19 Nov 2021 at 20:38, MRAB <python at mrabarnett.plus.com> wrote:
> >> >>
> >> >> On 2021-11-19 17:48, Marco Sulla wrote:
> >> >> > I have a battery of tests done with pytest. My tests break with a
> >> >> > segfault if I run them normally. If I run them using pytest -v, the
> >> >> > segfault does not happen.
> >> >> >
> >> >> > What could cause this quantical phenomenon?
> >> >> >
> >> >> Are you testing an extension that you're compiling? That kind of problem
> >> >> can occur if there's an uninitialised variable or incorrect reference
> >> >> counting (Py_INCREF/Py_DECREF).
> >> >
> >> > Ok, I know. But why can't it be reproduced if I do pytest -v? This way
> >> > I don't know which test fails.
> >> > Furthermore I noticed that if I remove the __pycache__ dir of tests,
> >> > pytest does not crash, until I re-ran it with the __pycache__ dir
> >> > present.
> >> > This way is very hard for me to understand what caused the segfault.
> >> > I'm starting to think pytest is not good for testing C extensions.
> >> >
> >> If there are too few Py_INCREF or too many Py_DECREF, it'll free the
> >> object too soon, and whether or when that will cause a segfault will
> >> depend on whatever other code is running. That's the nature of the
> >> beast: it's unpredictable!
> >>
> >> You could try running each of the tests in a loop to see which one
> >> causes a segfault. (Trying several in a loop will let you narrow it down
> >> more quickly.)
> >>
> >> pytest et al. are good for testing behaviour, but not for narrowing down
> >> segfaults.
> >
> --
> https://mail.python.org/mailman/listinfo/python-list