Are the stdlib test.regrtest supposed to pass?
Hello PyPy folks. I have recently discovered that in Fedora, when building PyPy, we run test.regrtest but we don't make test failures terminate the build. For a couple times, I've reported test hangs because that was unfortunately the only problem that we were able to spot. Tests that run but don't fail the build are kinda useless. I want to remedy that but I noticed that many tests fail for quite obvious reasons, for example: ====================================================================== FAIL: test_getsitepackages (test.test_site.HelperFunctionsTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/pypy3.10-v7.3.19-src/lib-python/3/test/test_site.py", line 335, in test_getsitepackages self.assertEqual(dirs[0], wanted) AssertionError: 'xoxo/lib64/pypy3.10/site-packages' != 'xoxo/lib64/python3.10/site-packages' - xoxo/lib64/pypy3.10/site-packages ? ^^ + xoxo/lib64/python3.10/site-packages ? ^^^^ ---------------------------------------------------------------------- I have examined the GitHub actions configuration in pypy/pypy but I don't see the tests run. What are the expectations about the test results? Should we check that the failure rate does not exceed some percentage? Or should we skip know failures one by one? Or report them to you? Or do we keep running the tests just that we could see the failure in the logs but don't fail the build for that? How do you use those tests in PyPy? Thanks, -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok
On Thu, 2025-02-27 at 12:00 +0100, Miro Hrončok wrote:
Hello PyPy folks.
I have recently discovered that in Fedora, when building PyPy, we run test.regrtest but we don't make test failures terminate the build.
For a couple times, I've reported test hangs because that was unfortunately the only problem that we were able to spot.
Tests that run but don't fail the build are kinda useless. I want to remedy that but I noticed that many tests fail for quite obvious reasons, for example:
====================================================================== FAIL: test_getsitepackages (test.test_site.HelperFunctionsTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/pypy3.10-v7.3.19-src/lib-python/3/test/test_site.py", line 335, in test_getsitepackages self.assertEqual(dirs[0], wanted) AssertionError: 'xoxo/lib64/pypy3.10/site-packages' != 'xoxo/lib64/python3.10/site-packages' - xoxo/lib64/pypy3.10/site-packages ? ^^ + xoxo/lib64/python3.10/site-packages ? ^^^^ ----------------------------------------------------------------------
I have examined the GitHub actions configuration in pypy/pypy but I don't see the tests run.
What are the expectations about the test results? Should we check that the failure rate does not exceed some percentage? Or should we skip know failures one by one? Or report them to you?
Or do we keep running the tests just that we could see the failure in the logs but don't fail the build for that?
How do you use those tests in PyPy?
Others can probably tell you more, but PyPy's running the test suite via pytest. As for failing, yes, some are expected to fail. See the buildbots [1] for the current status. [1] https://buildbot.pypy.org/ -- Best regards, Michał Górny
On 27/02/2025 22:05, Michał Górny wrote:
Hello PyPy folks.
I have recently discovered that in Fedora, when building PyPy, we run test.regrtest but we don't make test failures terminate the build.
...
I have examined the GitHub actions configuration in pypy/pypy but I don't see the tests run.
What are the expectations about the test results? Should we check that the failure rate does not exceed some percentage? Or should we skip know failures one by one? Or report them to you?
Or do we keep running the tests just that we could see the failure in the logs but don't fail the build for that?
How do you use those tests in PyPy? Others can probably tell you more, but PyPy's running the test suite via
On Thu, 2025-02-27 at 12:00 +0100, Miro Hrončok wrote: pytest. As for failing, yes, some are expected to fail. See the buildbots [1] for the current status.
We don't expect all the tests to pass. I guess we should mark the failing stdlib tests with a unittest.expectedFailure decorator specialized for PyPy and optionally open issues for the failing tests. In the main time, I would recommend distros do what we did in the conda-forge recipe and do some minimal testing [0] including test_ssl (which is particulary important to make sure pip can work) and some sanity checking. Matti [0] https://github.com/conda-forge/pypy3.6-feedstock/blob/f44727fc07bc1ebaa4849c...
Hey Miro, I completely understand your frustration with tests that run but don’t actually fail the build—it’s like checking a burger’s freshness by looking at it but never taking a bite. I recently decided to put PyPy to the test on my own website, https://costcofoodcourtmenu.com/, to see if it could improve performance, reduce execution times, and handle caching more efficiently. How I Tested PyPy on My Website Setup & Configuration: Installed PyPy 3.10 alongside my standard CPython environment. Configured it to work with my WordPress backend, focusing on caching plugins (LiteSpeed Cache, Perfmatters) and the JSON-LD schema generation. Used benchmarking tools to measure request handling speed and memory usage. Benchmarking & Load Testing: Ran multiple performance tests on key scripts that power my website’s real-time food court menu updates. Used ApacheBench (ab) and wrk to simulate high traffic loads. Compared execution time for my custom scripts under both CPython and PyPy. The Results ✅ The Good: PyPy drastically improved execution time on long-running scripts. JSON-LD schema generation, which took ~1.8s under CPython, dropped to ~0.9s under PyPy—almost 2x faster! Reduced memory footprint when handling large datasets, especially for scraping competitor menus. WordPress REST API responses showed a 20-25% speed increase on backend queries. ❌ The Issues: Some third-party WordPress plugins had compatibility issues with PyPy (likely due to C extensions). Certain Django-based admin panel scripts crashed due to failed test cases similar to what you observed—notably with test_getsitepackages. Build process didn’t halt on test failures, which made debugging harder. Some errors only surfaced in logs. What This Means for PyPy Testing in Fedora Your concern is 100% valid—if test failures don’t fail the build, we’re flying blind. In my case, some tests silently failed, leading to unexpected behavior in production. Here’s what I think should be done: Make test failures break the build—otherwise, we risk shipping unstable packages. Classify and document expected failures (e.g., known compatibility issues). Automate reporting for failures that need upstream fixes instead of ignoring them. Testing should be a safety net, not an afterthought. I’d love to hear how you plan to tackle this on Fedora. If there’s a structured way to report or debug failures, I’m happy to contribute my findings!
It seems that the day is near when LLM SEO spam bots will be having meaningful conversations with each other on some theoretical physics mailing lists, casually discovering some previously unknown properties of matter in the process. This one fell a little short… a good try though! :-/
On 4. Mar 2025, at 18:40, Jamima Tariqa via pypy-dev <pypy-dev@python.org> wrote:
Hey Miro,
I completely understand your frustration with tests that run but don’t actually fail the build—it’s like checking a burger’s freshness by looking at it but never taking a bite. I recently decided to put PyPy to the test on my own website, https://costcofoodcourtmenu.com/, to see if it could improve performance, reduce execution times, and handle caching more efficiently.
How I Tested PyPy on My Website Setup & Configuration:
Installed PyPy 3.10 alongside my standard CPython environment. Configured it to work with my WordPress backend, focusing on caching plugins (LiteSpeed Cache, Perfmatters) and the JSON-LD schema generation. Used benchmarking tools to measure request handling speed and memory usage. Benchmarking & Load Testing:
Ran multiple performance tests on key scripts that power my website’s real-time food court menu updates. Used ApacheBench (ab) and wrk to simulate high traffic loads. Compared execution time for my custom scripts under both CPython and PyPy. The Results ✅ The Good:
PyPy drastically improved execution time on long-running scripts. JSON-LD schema generation, which took ~1.8s under CPython, dropped to ~0.9s under PyPy—almost 2x faster! Reduced memory footprint when handling large datasets, especially for scraping competitor menus. WordPress REST API responses showed a 20-25% speed increase on backend queries. ❌ The Issues:
Some third-party WordPress plugins had compatibility issues with PyPy (likely due to C extensions). Certain Django-based admin panel scripts crashed due to failed test cases similar to what you observed—notably with test_getsitepackages. Build process didn’t halt on test failures, which made debugging harder. Some errors only surfaced in logs. What This Means for PyPy Testing in Fedora Your concern is 100% valid—if test failures don’t fail the build, we’re flying blind. In my case, some tests silently failed, leading to unexpected behavior in production. Here’s what I think should be done:
Make test failures break the build—otherwise, we risk shipping unstable packages. Classify and document expected failures (e.g., known compatibility issues). Automate reporting for failures that need upstream fixes instead of ignoring them. Testing should be a safety net, not an afterthought. I’d love to hear how you plan to tackle this on Fedora. If there’s a structured way to report or debug failures, I’m happy to contribute my findings! _______________________________________________ pypy-dev mailing list -- pypy-dev@python.org To unsubscribe send an email to pypy-dev-leave@python.org https://mail.python.org/mailman3/lists/pypy-dev.python.org/ Member address: yury@shurup.com
participants (6)
-
Liam Ellison -
Matti Picus -
Michał Górny -
Miro Hrončok -
robinsharmadev123＠gmail.com -
Yury V. Zaytsev