Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example. So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac. Adrian

On Sun, 2021-02-21 at 19:12 +0100, Christian Heimes wrote:

On 21/02/2021 13.47, glaubitz@debian.org wrote:

...

Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example.

So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac.

A --enable-unstable-platforms configure flag is my peace offer to meet you half way. You get a simple way to enable builds on untested platforms and we can clearly communicate that some OS and hardware platforms are not supported.

Does that imply accepting non-intrusive patches for platform support, or at least not proactively removing it? -- Best regards, Michał Górny

On 2/21/21 9:24 PM, Gregory P. Smith wrote:

On Sun, Feb 21, 2021 at 10:15 AM Christian Heimes wrote:

...

...

Rust doesn't keep any user from building Rust for Tier 2 or Tier 3

On 21/02/2021 13.47, glaubitz@debian.org wrote: platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example.

...

So if you want to copy Rust's mechanism, you should just leave it as is

and not claim that users are being confused because "m68k" shows up in configure.ac.

A --enable-unstable-platforms configure flag is my peace offer to meet you half way. You get a simple way to enable builds on untested platforms and we can clearly communicate that some OS and hardware platforms are not supported.

I personally wouldn't want to maintain such a check in autoconf, but it'll be an isolated thing on its own, that if you or someone else creates, will do its job and not bother the rest of us.

I think just publishing our list of (1) supported, (2) best-effort non-release-blocker quasi-supported, and (3) explicitly unsupported in a policy doc is sufficient. But it's not like any of us are going to stop someone from codifying that in configure.ac to require a flag.

agreed with (1) and(2). I don't like a negative list, as this will be incomplete at any time for both *-linux and *-non{linux,win,mac}. Looking at another project like GCC https://gcc.gnu.org/buildstat.html used to collect information about successful builds on various platforms and architectures. But as you can see, the web page isn't updated in recent times. GCC also doesn't keep an explicit list of the lesser supported platforms: https://gcc.gnu.org/gcc-11/criteria.html Matthias

There are zero technical reasons for what you are planning here. You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder. I could understand the maintenance burden argument if there was actually any maintenance burden, there isn't. The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux"). I have explained that these assumptions are not correct and that downstreams do actively use Python in ways that upstream no longer considers supported, yet you want to make changes to make everything for downstreams harder. I have not seen any other upstream project that is so bothered about a few lines of autoconf code. All other upstream projects I have worked with, be it Rust, OpenJDK, abseil-cpp and so on: None of them had problems when I sent patches to improve the code on more architectures. But in Python, it's suddenly a problem and you start long discussions about a few lines of autoconf code that pose zero maintenance burden. I absolutely don't get it. Adrian

On 2/22/21 12:30 PM, Victor Stinner wrote:

On Mon, Feb 22, 2021 at 8:19 AM wrote:

...

There are zero technical reasons for what you are planning here.

Multiple core developers explained how it's a maintenance burden. It has been explained in multiple different ways.

Well, that doesn't mean these statements are correct. Please don't assume that the people you are talking to are inexperienced developers, we aren't. Downstream distribution maintainers certainly have enough experience with project maintenance to be able to asses whether your claims are valid or not.

...

You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder.

"Making life harder" sounds to me like oh, maybe supporting one additional platform is not free and comes with a cost. This cost is something called the "maintenance burden".

Please explain to me how guarding some platforms with *additional* lines autoconf is helping to reduce maintenance burden for the upstream project.

My question is if Python wants to pay this cost, or if we want transfering the maintenance burden to people who actually care about these legacy platforms and architectures.

Your position is: Python must pay this price. My position is: Python should not.

No, my position is that such changes should have valid technical reasons which is simply not the case. You're not helping your point if you are basing your arguments on incorrect technical assumptions.

Honestly, if it's just a few lines, it will be trivial for you to maintain a downstream patch and I'm not sure why we even need this conversation. If it's more than a few lines, well, again, we come back to the problem of the real maintenance burden.

This argument goes both ways. The code we are talking about here are just a few lines of autoconf which are hardly touched during normal development work. And the architecture-mapping you have in [1] is probably not even needed (CC @jrtc27).

...

The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux").

I'm talking about 16-bit memory alignment which causes SIGBUS if it's not respected on m68k. For example, unicodeobject.c requires special code just for this arch:

/* * Issue #17237: m68k is a bit different from most architectures in * that objects do not use "natural alignment" - for example, int and * long are only aligned at 2-byte boundaries. Therefore the assert() * won't work; also, tests have shown that skipping the "optimised * version" will even speed up m68k. */ #if !defined(__m68k__) (...)

Such issue is hard to guess when you write code and usually only spot it while actually running the code on such architecture.

This is the only such place in the code where there is an extra section for m68k that I could find. And the bug was fixed by Andreas Schwab [2], so another downstream maintainer which was my point earlier in the discussion. We downstreams care about the platform support, hence we keep it working. Thanks, Adrian

[1] https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e... [2] https://github.com/python/cpython/commit/8b0e98426dd0e1fde93715256413bc70775...

-- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

On 22 Feb 2021, at 12:40, Michał Górny wrote:

...

I'm talking about 16-bit memory alignment which causes SIGBUS if it's not respected on m68k.

I don't understand why you consider this to be a problem. After all, x86 has stronger (32-bit) alignment requirements, so m68k is actually less likely to break.

On x86 you can make unaligned access to memory. Alignment is a nice to have for performance. But on m68k you MUST align or you get a SIGBUS. Barry

Barry Scott wrote:

...

On 22 Feb 2021, at 12:40, Michał Górny mgorny@gentoo.org wrote: I'm talking about 16-bit memory alignment which causes SIGBUS if it's not respected on m68k. I don't understand why you consider this to be a problem. After all, x86 has stronger (32-bit) alignment requirements, so m68k is actually less likely to break. On x86 you can make unaligned access to memory. Alignment is a nice to have for performance. But on m68k you MUST align or you get a SIGBUS. Barry

That is not the problem. Many architectures (SPARC, PA-RISC, IA-64, ...) do not natively support unaligned accesses. The problem is in fact that m68k's alignment requirements are *weaker* than x86 *and they are exposed to C*. On x86, the compiler still aligns ints to 4 byte boundaries, but on m68k they are only aligned to 2 byte boundaries. This means that assert(p % sizeof(int) == 0) does not work. The code in question is #ifdef'ed out for three reasons: 1. The assert is overly strict. This is trivially fixed by changing the SIZEOF_SIZE_T to ALIGNOF_SIZE_T. 2. The `if` uses (and code within the `if` relies on it using) SIZEOF_SIZE_T to check the alignment. 3. The code is potentially slower on m68k than the simple byte-by-byte loop, though I don't think anyone's going to complain about slight performance regressions on m68k if it comes from cleaning up the code, and I imagine the supposed performance hit came from not fixing 2 properly (i.e. there was a bunch of extra code that wasn't able to be used that often due to overly-strict requirements in the algorithm). I have filed https://github.com/python/cpython/pull/24624 to fix all these, though the first smaller commit is the only one strictly required for correctness on m68k (and any other architecture that chooses/has chosen to make the same ill-advised choices in its ABI), whilst the second larger one makes minor changes to the algorithm (that should not affect performance on any supported architecture other than m68k), and the many copies of it, in order to also cope with ALIGNOF_SIZE_T < SIZEOF_SIZE_T. This thus improves m68k support whilst removing m68k-specific hacks and making the code less reliant on implementation-defined behaviour, i.e. is how portability patches are _meant_ to be.

On 2/21/21 7:12 PM, Christian Heimes wrote:

On 21/02/2021 13.47, glaubitz@debian.org wrote:

...

Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example.

So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac.

A --enable-unstable-platforms configure flag is my peace offer to meet you half way.

Making a "peace offer" is the confession of having started a war. I don't know why you are on this crusade, but I doubt it is the lonely "confused user" cited in https://bugs.python.org/issue43179. Others pointed out in this thread that it's not the first time of such unfriendly behavior. Even others might also see it as some kind of harassment. Matthias

On Sun, 2021-02-21 at 13:35 +0100, Christian Heimes wrote:

On 21/02/2021 13.13, Victor Stinner wrote:

...

Hi,

I propose to actively remove support for *legacy* platforms and architectures which are not supported by Python according to PEP 11 rules: hardware no longer sold and end-of-life operating systems. The removal should be discussed on a case by case basis, but I would like to get an agreement on the overall idea first. Hobbyists wanting to support these platforms/archs can continue to support them with patches maintained outside Python. For example, I consider that the 16-bit m68k architecture is legacy, whereas the OpenBSD platform is still actively maintained. [...] Python has different kinds of platform and architecture supports. In practice, I would say that we have:

* (1) Fully supported. Platform/architecture used by core developers and have at least one working buildbot worker: fully supported. Since core developers want to use Python on their machine, they fix issues as soon as they notice them. Examples: x86-64 on Linux, Windows and macOS.

* (2) Best effort. Platform/architecture which has a buildbot worker usually not used by core developers. Regressions (buildbot failures) are reported to bugs.python.org, if someone investigates and provides a fix, the fix is merged. But there is usually no "proactive" work to ensure that Python works "perfectly" on these platforms. Example: FreeBSD/x86-64.

* (3) Not (officially) supported. We enter the blurry grey area. There is no buildbot worker, no core dev use it, but Python contains code specific to these platforms/architectures. Example: 16-bit m68k and 31-bit s390 architectures, OpenBSD.

The Rust programming language has 3 categories of Platform Support, the last one is :

Thanks Victor!

(short reply, I'm heading out)

I'm +1 in general for your proposal. I also like the idea to adopt Rust's platform support definition.

For 3.10 I propose to add a configure option to guard builds on unsupported / unstable platforms. My draft PR https://github.com/python/cpython/pull/24610/commits/f8d2d56757a9cec7ae4dc72... implements a checker for unsupported platforms and adds a --enable-unstable-platforms flag. Configuration on unsupported platforms fails unless users explicitly opt-in.

The checker serves two purposes:

1) It gives users an opportunity to provide full PEP 11 support (buildbot, engineering time) for a platform.

Does that mean that if someone offers to run the build bot for a minor platform and do the necessary maintenance to keep it working, they will be able to stay? How much maintenance is actually expected, i.e. is it sufficient to maintain CPython in a 'good enough' working state to resolve major bugs blocking real usage on these platforms? -- Best regards, Michał Górny

On Sun, Feb 21, 2021 at 1:07 PM Gregory P. Smith wrote:

...

I'm +1 in general for your proposal. I also like the idea to adopt

...

Rust's platform support definition.

+1, but see below.

The main thing from a project maintenance perspective is for platforms to not become a burden to other code maintainers. PRs need to be reviewed. Every #if/#endif in code is a cognitive burden. So being a minor platform can come with unexpected breakages that need fixing due to other changes made in the codebase that did not pay attention to the platform. As we cannot expect everyone working on code to care about anything beyond the tier-1 fully supported platforms, buildbot or not.

I think #ifdef's are a big part of the problem, actually. IIRC, the Linux kernel has been reducing/eliminating #ifdef's and instead using different .c's for each platform's platform-specific code.

On Sun, 2021-02-21 at 13:04 -0800, Gregory P. Smith wrote:

The main thing from a project maintenance perspective is for platforms to not become a burden to other code maintainers.  PRs need to be reviewed. Every #if/#endif in code is a cognitive burden.  So being a minor platform can come with unexpected breakages that need fixing due to other changes made in the codebase that did not pay attention to the platform.  As we cannot expect everyone working on code to care about anything beyond the tier-1 fully supported platforms, buildbot or not.

I have to disagree -- the support code (even if any is actually necessary) does not have to be a burden. Generally 'hobbyists' don't have a problem that the support for their platform becomes broken accidentally, or even deliberately because it blocks something else. They understand that others don't have hardware, time or motivation to maintain support for their platform properly. They themselves have limited time to work on it. So it's entirely fine for things to break occasionally, and they provide fixes as their time permits. They don't ask others to maintain their code. There's no real maintenance burden involved. In fact, this whole thread feels like removing 80%-complete translations from a program because they 'burden developers' and confuse users. Even if the translations are not actively updated and degenerates with strings changing, some users find them helpful. -- Best regards, Michał Górny

Hello, On Mon, 22 Feb 2021 09:20:46 +0100 Michał Górny wrote:

On Sun, 2021-02-21 at 13:04 -0800, Gregory P. Smith wrote:

...

The main thing from a project maintenance perspective is for platforms to not become a burden to other code maintainers.  PRs need to be reviewed. Every #if/#endif in code is a cognitive burden.  So being a minor platform can come with unexpected breakages that need fixing due to other changes made in the codebase that did not pay attention to the platform.  As we cannot expect everyone working on code to care about anything beyond the tier-1 fully supported platforms, buildbot or not.

I have to disagree -- the support code (even if any is actually necessary) does not have to be a burden. Generally 'hobbyists' don't have a problem that the support for their platform becomes broken accidentally, or even deliberately because it blocks something else. They understand that others don't have hardware, time or motivation to maintain support for their platform properly.

That's the problem CPython developers have - they like to remind they're volunteers (it's not too far-fetched to say they do that work as a hobby), but they also want to play big corporate types who with a flick of a wrist shut down systems and throw users in the cold. It all reminds another recent drama with the "cryptography" package: https://github.com/pyca/cryptography/issues/5771 . And it's not surprising that some people on that thread is also here. Likewise, it's not surprising that some Debian people are on this thread, with a clear message, just as they have in https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e (which happened to be linked from the ticket above). There's a conflict between 2 forms of volunteering. Old-school, still practiced by Debian, where people volunteer to *maintain* things for the benefit of other people. And new-style, along the lines "Hereby I volunteer to smash some Rust in the face of unsuspecting users" or "I volunteer to pull the rag under the feet of users of some systems". -- Best regards, Paul mailto:pmiscml@gmail.com

IIRC I suggested earlier that buildsbots should be integrated into the PR workflow in order to make it the contributor's rather than a core dev's burden to fix any breakages that result from their changes. On 22.02.2021 14:20, Victor Stinner wrote:

On Sun, Feb 21, 2021 at 8:57 PM Michał Górny wrote:

...

...

The checker serves two purposes:

1) It gives users an opportunity to provide full PEP 11 support (buildbot, engineering time) for a platform. Does that mean that if someone offers to run the build bot for a minor platform and do the necessary maintenance to keep it working, they will be able to stay? How much maintenance is actually expected, i.e. is it sufficient to maintain CPython in a 'good enough' working state to resolve major bugs blocking real usage on these platforms? Maintaining a buildbot doesn't mean to look at it every 6 months. It means getting emails multiple times per month about real bug which must be fixed. My main annoyance is that every single buildbot failure sends me an email, and I'm overwhelmed by emails (I'm not only getting emails from buildbots ;-)).

Python has already a long list of buildbot workers (between 75 and 100, I'm not sure of the exact number) and they require a lot of attention. Over the last 5 years, there is basically only Pablo Galindo and me who pay attention to them.

I fear that if more buildbots are added, Pablo and me will be the only ones to look at ones. FYI if nobody looks at buildbots, they are basically useless. They only waste resources.

To have an idea of the existing maintenance burden, look at emails sent to: https://mail.python.org/archives/list/buildbot-status@python.org/

Every single email is basically a problem. There are around 110 emails over the last 30 years: 3.6 email/day in average. When a bug is identified, it requires an investigation which takes between 5 minutes and 6 months depending on the bug. I would say 2 hours in average. Sometimes, if the investigation is too long, we simply revert the change.

The buildbot configuration also requires maintenance. For example, 14 commits have been pushed since January 1st: https://github.com/python/buildmaster-config/commits/master

Multiple buildbots are "unstable": tests are failing randomly. Again, each failure means a new email. For example, test_asyncio likes to fail once every 10 runs (coarse average, I didn't check exactly). multiprocessing tests, tests using network like imaplib or nntplib, and some other tests fail randomly. Some tests just fail because just once, the buildbot became slower.

People have many ideas to automate bug triage from emails, but so far, nobody came with a concrete working solution, and so emails are still read manually one by one. Also, almost nobody is trying to fix tests which are failing randomly.

For example, I called multiple times for help to fix test_asyncio, so far it's still randomly every day:

* 2020: https://mail.python.org/archives/list/python-dev@python.org/message/Y7I5ADXA... * 2019: https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGE...

By the way, these random failures are not only affecting buildbots, but also CIs run on pull requests. It's *common* that these failures prevent to merge a pull request and require manual actions to be able to merge the PR (usually, re-run all CIs).

I'm not talking about exotic platforms with very slow hardware, but platorms like Linux/x86-64 with "fast" hardware.

I expect more random errors on exotic platforms. For example, I reported a crash on AIX one year ago, and nobody fixed it so far. I pushed a few fixes for that crash, but it's not enough to fully fix it: https://bugs.python.org/issue40068

I pushed AIX fixes only because I was annoyed by getting buildbot emails about AIX failures. Sometimes, I just turn off emails from AIX.

Since there is no proactive work on fixing AIX issues, I would even prefer to *remove* the AIX buildbots.

Victor

-- Regards, Ivan

On 2/22/2021 6:58 AM, Victor Stinner wrote:

On Mon, Feb 22, 2021 at 12:51 PM Ivan Pozdeev via Python-Dev wrote:

...

IIRC I suggested earlier that buildsbots should be integrated into the PR workflow in order to make it the contributor's rather than a core dev's burden to fix any breakages that result from their changes.

The problems is that many 'breakages' do NOT result from the PR changes. For IDLE patches, EVERY CI failure other than in test_idle is a bogus failure. My productivity is already degraded by the current blind automated rules. My point here is that machines literally have no judgment, and that automation is not the panacea that people think. Writing good rules is often hard to impossible, and blind application of inadequate rules leads to bad results.

Some buildbot worker take 2 to 3 hours per build. Also, it would not scale. Buildbots are not fast enough to handle the high number of PR and PR updates.

When there is clear relationship between a buildbot failure and a merged PR, a comment is added automatically showing the failed test and explanation how to investigate the issue.

This is false and an example of the above comment. The notice is sent whenever any buildbot test fails after a merge, regardless of whether there is any relationship or not. There often (usually?) is not.

It's there for 2 years thanks to Pablo, and so far, I rarely saw developers paying attention to these failures. They just ignore it.

Every one of those big, bold, ugly notices I have received has been a false positive. I believe they are usually due to one of the flaky tests (asyncio or otherwise) that should have been disabled but have not been. Of course people ignore routinely false positives. If there is an expert for the module whose test file failed, that person should be the target of the notice. If a PR or merge breaks test_idle, I would like to know. Actually, because I am conscientious, and because there was one instance years ago where test_idle passed CI and failed on one buildbot, and because the notice lists which test file failed (in among the noise), I do check that line before deleting the notice.

I'm not trying to blame anyone. Contributing to Python requires a lot of free time. I'm only trying to explain in length what does the "maintenance burden" mean in practice, since some people are pretending that supporting a platform is free.

Blind automated blind rules are also not free. -- Terry Jan Reedy

This, and an earlier letter about the burned of a release manager, confirmes my suspicion that a large share of "maintenance burden" comes from unresolved process issues. E.g. looks like Victor had been unconvinced that unreliable tests are a serious issue and cost the team so much (including himself since they cause ppl to ignore his buildbot issue reports right back) and had blocked the effort to disable them or whatnot (this is just my guess but Terry's message suggests that he was against resolving it). I hope this convinces him and will let the team clear this issue at last! Likewise, half of the bullet points in https://mail.python.org/archives/list/python-dev@python.org/message/LJ3E2UQB... comes from either ppl trying to bypass the process, or the release manager doing others' jobs that should've already been done had the process been followed. On 22.02.2021 19:48, Terry Reedy wrote:

On 2/22/2021 6:20 AM, Victor Stinner wrote:

...

To have an idea of the existing maintenance burden, look at emails sent to: https://mail.python.org/archives/list/buildbot-status@python.org/

Every single email is basically a problem. There are around 110 emails over the last 30 years:

30 days, not years.  The 20 visible messages are all have title f"Buildbot worker {worker} missing".  All the messages on a day are sent at the same time.  Replace with a daily "Builbot workers currently missing"?  The individual messages seem useless except possibly if sent to individual buildbot owners. ...

...

Multiple buildbots are "unstable": tests are failing randomly. Again, each failure means a new email. For example, test_asyncio likes to fail once every 10 runs (coarse average, I didn't check exactly).

I strongly feel that the individual repeatedly failing tests (only 3 or 4 I think) should be disabled and the asyncio people notified.  As I remember, you once insisted on keeping routinely failing tests.

...

By the way, these random failures are not only affecting buildbots, but also CIs run on pull requests. It's *common* that these failures prevent to merge a pull request and require manual actions to be able to merge the PR (usually, re-run all CIs).

For backports, it means closing the backport and reopening by re-adding a backport label to the master PR.  This is why I really really want the repeatedly failing tests disabled.

-- Regards, Ivan

On 2/22/21, 1:39 PM, "Steve Dower" wrote: On 2/22/2021 5:18 PM, Matthias Klose wrote: > On 2/21/21 1:13 PM, Victor Stinner wrote: > Document what is supported, be inclusive about anything else. Don't make a > distinction yet between legacy and upcoming new architectures. I agree with this, and I don't see any reason why we shouldn't just use the list of stable buildbot platforms as the "supported" list. That makes it really clear what the path is to push support onto upstream (join up and bring a buildbot with you), and also means that we've got a physically restricted set of machines to prove work before doing a release. Actively blocking anything at all seems unnecessary at the source/build level. That's for pre-built binaries and other conveniences. Cheers, Steve _______________________________________________ +1 to this. I use a few unsupported platforms, not as a hobby, but in my work. I generally don't require ALL the things in python to work on these platforms, so even if I were to contribute a buildbot for an obscure (but definitely not "hobby") platform, it's implied that I would also need to fix test failures in modules I don't use. I generally need to provide some reason for me to use my employer's time, so I can't really justify fixing test failures in code the company doesn't use. I think these users are just asking that what currently works not be broken intentionally, and that follows the spirit of Autoconf, to test if something works and enable if it when it does. There are also other ways to reduce the burden of maintaining a large number of platforms, such as abstracting away the OS services. The other thing that crept into this thread was the mention of test that intermittently fail. That's a huge problem because it suggests that applications will sometimes fail. I have usually seen these sort of issues because of 1) Uninitialized memory being used (read) 2) Threading problems 3) Resources used (files, networking, daemons) but unavailable 4) memory mis-management (buffer overrun that doesn't cause a crash) #3 is probably best fixed by testing for resources and skipping when unavailable The others are problems in the code, and can be fixed with clang sanitizers, but Without some routine running of them, those sorts of problems will reappear.

On 2/22/21, 4:06 PM, "Antoine Pitrou" wrote: On Mon, 22 Feb 2021 19:50:43 +0000 Rob Boehne wrote: > > The other thing that crept into this thread was the mention of test that intermittently fail. > That's a huge problem because it suggests that applications will sometimes fail. > I have usually seen these sort of issues because of > 1) Uninitialized memory being used (read) > 2) Threading problems > 3) Resources used (files, networking, daemons) but unavailable > 4) memory mis-management (buffer overrun that doesn't cause a crash) > > #3 is probably best fixed by testing for resources and skipping when unavailable 5) Poor quality POSIX support in the target platform. The Python test suite is actually quite demanding in this regard (except on Windows). Regards Antoine. Antoine: I find this a bit of a surprise, can you give an example of poor POSIX support leading to intermittent test failures? Thanks, Rob Boehne

...

Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements.

Actually, fewer. Most architectures have alignof(x) == sizeof(x) for all the primitive types, but m68k is more relaxed and caps alignof(x) at 2. This means that assert((p & sizeof(long)) == 0) is too strict, and should instead just be assert((p & alignof(long)) == 0), which is always correct, rather than relying on implementation-defined alignment requirements. In autoconf there's AC_CHECK_ALIGNOF just as there is AC_CHECK_SIZEOF, the result of which should be used for the alignment check instead. That's the portable way to do it (and would allow the removal of the #ifdef).

Michał Górny wrote:

...

Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements. Actually, fewer. Most architectures have alignof(x) == sizeof(x) for all the primitive types, but m68k is more relaxed and caps alignof(x) at 2. This means that assert((p & sizeof(long)) == 0) is too strict, and should instead just be assert((p & alignof(long)) == 0), which is always correct, rather than relying on implementation-defined alignment requirements. In autoconf there's AC_CHECK_ALIGNOF just as there is AC_CHECK_SIZEOF, the result of which should be used for the alignment check instead. That's the portable way to do it (and would allow the removal of the #ifdef). I agree, except that -- as I mentioned elsewhere -- the #ifdef was added because the x86 optimization hack is actually slower on m68k. I suspect

On Mon, 2021-02-22 at 19:27 +0000, Jessica Clarke wrote: that if more benchmarking was made, it might turn out that #ifdef should actually disable it on more platforms.

I think it's more complicated than that. The code in question didn't just have a bogus assert, it actually relied on ALIGNOF_SIZE_T == SIZEOF_SIZE_T in order to work, but without much good reason other than being written in a poor style. I suspect that the slowdown was seen because the strictness of the `if` on m68k meant the optimised version wasn't used that often but was still sitting there using up space, and time to evaluate the branch, plus potentially the various consequences of additional register pressure, and that the performance hit goes away once the algorithm is fixed to be more general (in such a way that other architectures shouldn't see any performance hits, and possibly even a slight improvement). I've done this in https://github.com/python/cpython/pull/24624. Plus, even if m68k is slightly slower, who cares? It still works, better to have working clean code than hacky slightly-faster code.