From mhroncok at redhat.com  Sat Jun 15 19:23:24 2024
From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=C4=8Dok?=)
Date: Sun, 16 Jun 2024 01:23:24 +0200
Subject: [Cython] How to see why distutils-powered tests failed?
In-Reply-To: <f4ad21af-0f9e-4190-be39-c239e7863840@redhat.com>
References: <f4ad21af-0f9e-4190-be39-c239e7863840@redhat.com>
Message-ID: <ccd1697d-26a7-4ec0-81bf-a13db9c074fb@redhat.com>

Hi. Apparently, I was not properly subscribed to the list when I sent this, 
trying again:

-------- Forwarded Message --------
Subject: How to see why distutils-powered tests failed?
Date: Mon, 13 May 2024 12:10:20 +0200
From: Miro Hron?ok <mhroncok at redhat.com>
To: cython-devel at python.org

Hello,

I've recently encountered the following test failure when I wanted to run the 
upstream Cython testusite on Fedora CI.

======================================================================
ERROR: runTest (__main__.CythonRunTestCase.runTest)
[-1] compiling (cpp/cy2/pythran) and running numpy_pythran
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", 
line 266, in link
     self.spawn(linker + ld_args)
   File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", 
line 1041, in spawn
     spawn(cmd, dry_run=self.dry_run, **kwargs)
   File "/usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", 
line 70, in spawn
     raise DistutilsExecError(
distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File 
"/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
line 1550, in run
     ext_so_path = self.runCompileTest()
                   ^^^^^^^^^^^^^^^^^^^^^
   File 
"/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
line 1163, in runCompileTest
     return self.compile(
            ^^^^^^^^^^^^^
   File 
"/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
line 1482, in compile
     so_path = self.run_distutils(test_directory, module, workdir, incdir)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
line 1378, in run_distutils
     build_extension.run()
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
line 345, in run
     self.build_extensions()
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
line 467, in build_extensions
     self._build_extensions_serial()
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
line 493, in _build_extensions_serial
     self.build_extension(ext)
   File 
"/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
line 637, in build_extension
     _build_ext.build_extension(self, ext)
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
line 572, in build_extension
     self.compiler.link_shared_object(
   File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", 
line 752, in link_shared_object
     self.link(
   File 
"/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", 
line 268, in link
     raise LinkError(msg)
distutils.errors.LinkError: command '/usr/bin/g++' failed with exit code 1



There were 4 failures like this (all pythran_numpy related).



I was clueless as to why is this failing. I was trying to make the tests tell 
me more.

After long digging I ended up modifying 
/usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py in a test 
container directly to always make this conditional False:

https://github.com/pypa/setuptools/blob/v69.1.0/setuptools/_distutils/spawn.py#L61

This turned:

   distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit 
code 1

Into:

   distutils.errors.DistutilsExecError: command ['/usr/bin/g++', ...] failed 
with exit code 1

Then, I was able to rerun the failed g++ command manually to see the actual 
linker error (missing flexiblas).

However, the experience was far from ideal.

Would it be possible (and desired) to modify the tests to actually print out 
the compiler/linker errors when they happen?

Or perhaps this already is possible, but I am not doing it correctly? (I tried 
--no-capture, but no dice.)

Thanks,
-- 
Miro Hron?ok
-- 
Phone: +420777974800
IRC: mhroncok


From dw-git at d-woods.co.uk  Thu Jun 20 16:19:37 2024
From: dw-git at d-woods.co.uk (da-woods)
Date: Thu, 20 Jun 2024 21:19:37 +0100
Subject: [Cython] How to see why distutils-powered tests failed?
In-Reply-To: <ccd1697d-26a7-4ec0-81bf-a13db9c074fb@redhat.com>
References: <f4ad21af-0f9e-4190-be39-c239e7863840@redhat.com>
 <ccd1697d-26a7-4ec0-81bf-a13db9c074fb@redhat.com>
Message-ID: <7c654bbc-a66b-468f-8224-f549c9a205fd@d-woods.co.uk>

Hi Miro,

Sorry for the slightly slow reply.

When I tried to check this I can definitely see compile errors in the 
error output from the test. It looks like you're getting a link error 
though.? I'm just going to do a PR to try to fix that. I'll ping you on 
GitHub when that's done.

David




On 16/06/2024 00:23, Miro Hron?ok wrote:
> Hi. Apparently, I was not properly subscribed to the list when I sent 
> this, trying again:
>
> -------- Forwarded Message --------
> Subject: How to see why distutils-powered tests failed?
> Date: Mon, 13 May 2024 12:10:20 +0200
> From: Miro Hron?ok <mhroncok at redhat.com>
> To: cython-devel at python.org
>
> Hello,
>
> I've recently encountered the following test failure when I wanted to 
> run the upstream Cython testusite on Fedora CI.
>
> ======================================================================
> ERROR: runTest (__main__.CythonRunTestCase.runTest)
> [-1] compiling (cpp/cy2/pythran) and running numpy_pythran
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", 
> line 266, in link
> ??? self.spawn(linker + ld_args)
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", 
> line 1041, in spawn
> ??? spawn(cmd, dry_run=self.dry_run, **kwargs)
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", 
> line 70, in spawn
> ??? raise DistutilsExecError(
> distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed 
> with exit code 1
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> ? File 
> "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
> line 1550, in run
> ??? ext_so_path = self.runCompileTest()
> ????????????????? ^^^^^^^^^^^^^^^^^^^^^
> ? File 
> "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
> line 1163, in runCompileTest
> ??? return self.compile(
> ?????????? ^^^^^^^^^^^^^
> ? File 
> "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
> line 1482, in compile
> ??? so_path = self.run_distutils(test_directory, module, workdir, incdir)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ? File 
> "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
> line 1378, in run_distutils
> ??? build_extension.run()
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
> line 345, in run
> ??? self.build_extensions()
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
> line 467, in build_extensions
> ??? self._build_extensions_serial()
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
> line 493, in _build_extensions_serial
> ??? self.build_extension(ext)
> ? File 
> "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", 
> line 637, in build_extension
> ??? _build_ext.build_extension(self, ext)
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", 
> line 572, in build_extension
> ??? self.compiler.link_shared_object(
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", 
> line 752, in link_shared_object
> ??? self.link(
> ? File 
> "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", 
> line 268, in link
> ??? raise LinkError(msg)
> distutils.errors.LinkError: command '/usr/bin/g++' failed with exit 
> code 1
>
>
>
> There were 4 failures like this (all pythran_numpy related).
>
>
>
> I was clueless as to why is this failing. I was trying to make the 
> tests tell me more.
>
> After long digging I ended up modifying 
> /usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py in a 
> test container directly to always make this conditional False:
>
> https://github.com/pypa/setuptools/blob/v69.1.0/setuptools/_distutils/spawn.py#L61 
>
>
> This turned:
>
> ? distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed 
> with exit code 1
>
> Into:
>
> ? distutils.errors.DistutilsExecError: command ['/usr/bin/g++', ...] 
> failed with exit code 1
>
> Then, I was able to rerun the failed g++ command manually to see the 
> actual linker error (missing flexiblas).
>
> However, the experience was far from ideal.
>
> Would it be possible (and desired) to modify the tests to actually 
> print out the compiler/linker errors when they happen?
>
> Or perhaps this already is possible, but I am not doing it correctly? 
> (I tried --no-capture, but no dice.)
>
> Thanks,