Move support of legacy platforms/architectures outside Python
Hi, I propose to actively remove support for *legacy* platforms and architectures which are not supported by Python according to PEP 11 rules: hardware no longer sold and end-of-life operating systems. The removal should be discussed on a case by case basis, but I would like to get an agreement on the overall idea first. Hobbyists wanting to support these platforms/archs can continue to support them with patches maintained outside Python. For example, I consider that the 16-bit m68k architecture is legacy, whereas the OpenBSD platform is still actively maintained. I already know that there will be a strike back: "oh no, you must continue to support my architecture" and "their existing code should stay and doesn't cost anything to maintain". Python is maintained by volunteers, the majority is contributing in their free time, so people are free to use their free time as they want. You cannot ask core developers to support your favorite *legacy* platform/architecture if they don't want to. In short, I propose to move maintenance of the legacy platforms/archs outside Python: people are free to continue support them as patches. -- Concrete example: Christian Heimes proposed to drop support for 31-bit s390 Linux: https://bugs.python.org/issue43179 The lack of clear definition on how a platform is supported or not confuses users who consider that their favorite platform/arch is supported, whereas core developers don't want to support it since it would be too much work. In fact, the PEP 11 has clear and explicit rules: https://www.python.org/dev/peps/pep-0011/#supporting-platforms A platform is only considered as supported if the following two conditions are met: 1) a core developer needs to volunteer to maintain platform-specific code 2) a stable buildbot must be provided Last October, I proposed to drop Solaris support (bpo-42173). Jakub Kulik stepped in and proposed some Solaris patches, so I abandoned my idea. But I still don't see any running Solaris buildbot worker, and there is still no core developer volunteer to maintain Solaris support. It's unclear to me if Python has "best effort" support for Solaris, of if Solaris is "not supported". -- Over the years, Python was ported to tons of platforms and CPU architectures. It didn't matter if the platform or the architecture was commonly used or not. 30 years later, Python still has the code for many legacy platforms and architectures. Some hardware is no longer sold but kept alive by hobbyists, especially members of retro computing groups. Some Linux distributions like Gentoo and Debian are trying to support most architectures which are supported by these hobbyist groups, whereas some other distributions like Ubuntu are limited to a few platforms. For example, Ubuntu 20.4.2 LTS server supports 4 architectures: x86-64, AArch64 (ARM), POWER and s390x. I guess that the difference between Debian and Ubuntu is that Ubuntu is a Canonical product, Canonical sells professional support and so cannot support too many architectures. Each architecture support requires to build all packages on it, tests the packages, have experts who fix issues specific to this arch, etc. -- Python has different kinds of platform and architecture supports. In practice, I would say that we have: * (1) Fully supported. Platform/architecture used by core developers and have at least one working buildbot worker: fully supported. Since core developers want to use Python on their machine, they fix issues as soon as they notice them. Examples: x86-64 on Linux, Windows and macOS. * (2) Best effort. Platform/architecture which has a buildbot worker usually not used by core developers. Regressions (buildbot failures) are reported to bugs.python.org, if someone investigates and provides a fix, the fix is merged. But there is usually no "proactive" work to ensure that Python works "perfectly" on these platforms. Example: FreeBSD/x86-64. * (3) Not (officially) supported. We enter the blurry grey area. There is no buildbot worker, no core dev use it, but Python contains code specific to these platforms/architectures. Example: 16-bit m68k and 31-bit s390 architectures, OpenBSD. The Rust programming language has 3 categories of Platform Support, the last one is : "Tier 3 platforms are those which the Rust codebase has support for, but which are not built or tested automatically, and may not work. Official builds are not available." https://doc.rust-lang.org/nightly/rustc/platform-support.html Rust Tier 3 looks like our "Not supported" category. Maybe we should have a better definition in the PEP 11 of our 3 support levels. There is also a 4th category: platforms/archs which are really not supported, like they legacy ones for which we removed the code :-) Examples: BeOS, MacOS 9, platforms with no thread support, etc. -- There is also a "Stable Buildbot" category used by the "Release Status" web page: https://buildbot.python.org/all/#/release_status There is no clear rule why is a worker is added to that list or not. For example, there are two FreeBSD workers which runs FreeBSD CURRENT: the FreeBSD development branch. I don't think that it should be declared as "stable". Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 21/02/2021 13.13, Victor Stinner wrote:
Thanks Victor! (short reply, I'm heading out) I'm +1 in general for your proposal. I also like the idea to adopt Rust's platform support definition. For 3.10 I propose to add a configure option to guard builds on unsupported / unstable platforms. My draft PR https://github.com/python/cpython/pull/24610/commits/f8d2d56757a9cec7ae4dc72... implements a checker for unsupported platforms and adds a --enable-unstable-platforms flag. Configuration on unsupported platforms fails unless users explicitly opt-in. The checker serves two purposes: 1) It gives users an opportunity to provide full PEP 11 support (buildbot, engineering time) for a platform. 2) It gives us the leverage to remove a flagged platform in the future or refuse support on BPO. Christian
Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example. So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac. Adrian
On 21/02/2021 13.47, glaubitz@debian.org wrote:
Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example.
So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac.
A --enable-unstable-platforms configure flag is my peace offer to meet you half way. You get a simple way to enable builds on untested platforms and we can clearly communicate that some OS and hardware platforms are not supported.
On Sun, Feb 21, 2021 at 10:15 AM Christian Heimes <christian@python.org> wrote:
I personally wouldn't want to maintain such a check in autoconf, but it'll be an isolated thing on its own, that if you or someone else creates, will do its job and not bother the rest of us. I think just publishing our list of (1) supported, (2) best-effort non-release-blocker quasi-supported, and (3) explicitly unsupported in a policy doc is sufficient. But it's not like any of us are going to stop someone from codifying that in configure.ac to require a flag. -gps
On 2/21/21 9:24 PM, Gregory P. Smith wrote:
agreed with (1) and(2). I don't like a negative list, as this will be incomplete at any time for both *-linux and *-non{linux,win,mac}. Looking at another project like GCC https://gcc.gnu.org/buildstat.html used to collect information about successful builds on various platforms and architectures. But as you can see, the web page isn't updated in recent times. GCC also doesn't keep an explicit list of the lesser supported platforms: https://gcc.gnu.org/gcc-11/criteria.html Matthias
On Sun, Feb 21, 2021 at 12:28 PM Gregory P. Smith <greg@krypto.org> wrote:
If we add a compile flag to explicitly make people have to realize they are running on an unsupported platform then I think it should be a negation against an allowlist versus a blocklist to be more explicit about what we would take PRs for.
I like the idea of making PEP 11 list what platforms *are* supported in some way, and being off that list means you're not. I also like the idea of having a tier 1 that will block a release and a tier 2 where we will accept PRs but it will in no way block releases. I also think who ends up on either of those tiers should be an SC problem. Based on how heated this thread has gotten there's obviously some emotional connection for some folks when it comes to whether a platform is supported or not and how that is handled. In that sense, letting the SC take on the burden of saying "no" makes sense. That doesn't mean PEP 11 wouldn't still list out the *minimum* requirements to add a platform, but I don't think it should be an automatic thing simply because a machine was donated and a hand was raised as e.g. #ifdefs have a cognitive cost. So, my suggestion is to update PEP 11 to: - List platforms that can block releases as a "tier 1" supported platform - List platforms that are best effort as "tier 2" and which will never hold up a release - All other platforms will need to manage their patchset externally and we will not accept PRs that are specifically for them - Specify what the *minimum* requirements are to add support for a platform to either tier - Have the SC manage what platforms can end up in what tier (and we can publish guidelines like conditional tier 2 to prove support exists, what is required to graduate to tier 1, removal from tiers, etc.)
There are zero technical reasons for what you are planning here. You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder. I could understand the maintenance burden argument if there was actually any maintenance burden, there isn't. The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux"). I have explained that these assumptions are not correct and that downstreams do actively use Python in ways that upstream no longer considers supported, yet you want to make changes to make everything for downstreams harder. I have not seen any other upstream project that is so bothered about a few lines of autoconf code. All other upstream projects I have worked with, be it Rust, OpenJDK, abseil-cpp and so on: None of them had problems when I sent patches to improve the code on more architectures. But in Python, it's suddenly a problem and you start long discussions about a few lines of autoconf code that pose zero maintenance burden. I absolutely don't get it. Adrian
On Mon, Feb 22, 2021 at 8:19 AM <glaubitz@debian.org> wrote:
There are zero technical reasons for what you are planning here.
Multiple core developers explained how it's a maintenance burden. It has been explained in multiple different ways.
You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder.
"Making life harder" sounds to me like oh, maybe supporting one additional platform is not free and comes with a cost. This cost is something called the "maintenance burden". My question is if Python wants to pay this cost, or if we want transfering the maintenance burden to people who actually care about these legacy platforms and architectures. Your position is: Python must pay this price. My position is: Python should not. Honestly, if it's just a few lines, it will be trivial for you to maintain a downstream patch and I'm not sure why we even need this conversation. If it's more than a few lines, well, again, we come back to the problem of the real maintenance burden.
The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux").
I'm talking about 16-bit memory alignment which causes SIGBUS if it's not respected on m68k. For example, unicodeobject.c requires special code just for this arch: /* * Issue #17237: m68k is a bit different from most architectures in * that objects do not use "natural alignment" - for example, int and * long are only aligned at 2-byte boundaries. Therefore the assert() * won't work; also, tests have shown that skipping the "optimised * version" will even speed up m68k. */ #if !defined(__m68k__) (...) Such issue is hard to guess when you write code and usually only spot it while actually running the code on such architecture. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2/22/21 12:30 PM, Victor Stinner wrote:
Well, that doesn't mean these statements are correct. Please don't assume that the people you are talking to are inexperienced developers, we aren't. Downstream distribution maintainers certainly have enough experience with project maintenance to be able to asses whether your claims are valid or not.
Please explain to me how guarding some platforms with *additional* lines autoconf is helping to reduce maintenance burden for the upstream project.
No, my position is that such changes should have valid technical reasons which is simply not the case. You're not helping your point if you are basing your arguments on incorrect technical assumptions.
This argument goes both ways. The code we are talking about here are just a few lines of autoconf which are hardly touched during normal development work. And the architecture-mapping you have in [1] is probably not even needed (CC @jrtc27).
This is the only such place in the code where there is an extra section for m68k that I could find. And the bug was fixed by Andreas Schwab [2], so another downstream maintainer which was my point earlier in the discussion. We downstreams care about the platform support, hence we keep it working. Thanks, Adrian
[1] https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e... [2] https://github.com/python/cpython/commit/8b0e98426dd0e1fde93715256413bc70775...
-- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On Mon, 2021-02-22 at 12:30 +0100, Victor Stinner wrote:
I don't understand why you consider this to be a problem. After all, x86 has stronger (32-bit) alignment requirements, so m68k is actually less likely to break.
Unless I'm reading something wrong, the code is disabled not because it m68k is broken (as your comment seems to imply) but because the assert is wrong and because the code turned out to be slower. That said, I wonder if this 'optimized' path has been actually benchmarked on other supported platforms. -- Best regards, Michał Górny
Barry Scott wrote:
That is not the problem. Many architectures (SPARC, PA-RISC, IA-64, ...) do not natively support unaligned accesses. The problem is in fact that m68k's alignment requirements are *weaker* than x86 *and they are exposed to C*. On x86, the compiler still aligns ints to 4 byte boundaries, but on m68k they are only aligned to 2 byte boundaries. This means that assert(p % sizeof(int) == 0) does not work. The code in question is #ifdef'ed out for three reasons: 1. The assert is overly strict. This is trivially fixed by changing the SIZEOF_SIZE_T to ALIGNOF_SIZE_T. 2. The `if` uses (and code within the `if` relies on it using) SIZEOF_SIZE_T to check the alignment. 3. The code is potentially slower on m68k than the simple byte-by-byte loop, though I don't think anyone's going to complain about slight performance regressions on m68k if it comes from cleaning up the code, and I imagine the supposed performance hit came from not fixing 2 properly (i.e. there was a bunch of extra code that wasn't able to be used that often due to overly-strict requirements in the algorithm). I have filed https://github.com/python/cpython/pull/24624 to fix all these, though the first smaller commit is the only one strictly required for correctness on m68k (and any other architecture that chooses/has chosen to make the same ill-advised choices in its ABI), whilst the second larger one makes minor changes to the algorithm (that should not affect performance on any supported architecture other than m68k), and the many copies of it, in order to also cope with ALIGNOF_SIZE_T < SIZEOF_SIZE_T. This thus improves m68k support whilst removing m68k-specific hacks and making the code less reliant on implementation-defined behaviour, i.e. is how portability patches are _meant_ to be.
Hello! On 2/22/21 12:30 PM, Victor Stinner wrote:
Just as a heads-up: There is a PR by Jessica Clarke now [1] which gets rid of this architecture-specific #ifdef. I think this is a good approach as it gets rid of one of your complaining points. I have already verified that these changes don't break on 32-bit PowerPC, 64-bit SPARC and, of course, M68k. Thanks, Adrian
-- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On 2/21/21 7:12 PM, Christian Heimes wrote:
Making a "peace offer" is the confession of having started a war. I don't know why you are on this crusade, but I doubt it is the lonely "confused user" cited in https://bugs.python.org/issue43179. Others pointed out in this thread that it's not the first time of such unfriendly behavior. Even others might also see it as some kind of harassment. Matthias
(Since my first reply here somehow got lost, I'm posting this again). Rust doesn't prevent anyone from building Tier 2 or Tier 3 targets. There is no limitation for "legacy" or "deprecated" targets. Any target can be built and any target can be selected by the Rust compiler for cross-compliation. So, if you want to mimic the behavior of the Rust compiler, you would obviously not guard any architectures or platforms in configure.ac. I am objecting this change as it does not reduce code complexity (on the contrary) and only causes more maintenance burden for downstream distributions. Thanks, Adrian
On Sun, 2021-02-21 at 13:35 +0100, Christian Heimes wrote:
Does that mean that if someone offers to run the build bot for a minor platform and do the necessary maintenance to keep it working, they will be able to stay? How much maintenance is actually expected, i.e. is it sufficient to maintain CPython in a 'good enough' working state to resolve major bugs blocking real usage on these platforms? -- Best regards, Michał Górny
On Sun, Feb 21, 2021 at 12:03 PM Michał Górny <mgorny@gentoo.org> wrote:
Definitely start with this. This level of effort to maintain the minor platform support in-tree may be able to keep an otherwise neglected minor platform in second tier non-release-blocker best effort status. Having a buildbot, at least provides visibility, and sometimes PR authors will be sympathetic to easy edits that keep something working that way (no guarantee). The main thing from a project maintenance perspective is for platforms to not become a burden to other code maintainers. PRs need to be reviewed. Every #if/#endif in code is a cognitive burden. So being a minor platform can come with unexpected breakages that need fixing due to other changes made in the codebase that did not pay attention to the platform. As we cannot expect everyone working on code to care about anything beyond the tier-1 fully supported platforms, buildbot or not. Example: I consider many of the BSDs and the Solaris derivatives to be in this state. (non specific here, i don't even know which ones we claim are supported or not without going and reading whatever policy docs we might or might not have today - Victor alludes to this state of the world). We tend to accept patches when someone offers them. Occasionally we have a core dev who actually runs one of them. But most of us don't go out of the way ourselves to try and keep changes we make working there. We expect interested parties to jump in when something isn't working right. And will generally do related PR reviews/merges if they're not burdensome. An example of the above happening recently is VxWorks support via https://bugs.python.org/issue31904. -gps
Yes, but all this doesn't apply here. We're literally talking about a few lines of autoconf and preprocessor defines [1]. In fact, the change to remove "s390 support" (in quotes because these few lines don't really can be considered platform-specific code), introduces more lines than it removes because of the comments. Adrian
On Sun, 2021-02-21 at 13:04 -0800, Gregory P. Smith wrote:
I have to disagree -- the support code (even if any is actually necessary) does not have to be a burden. Generally 'hobbyists' don't have a problem that the support for their platform becomes broken accidentally, or even deliberately because it blocks something else. They understand that others don't have hardware, time or motivation to maintain support for their platform properly. They themselves have limited time to work on it. So it's entirely fine for things to break occasionally, and they provide fixes as their time permits. They don't ask others to maintain their code. There's no real maintenance burden involved. In fact, this whole thread feels like removing 80%-complete translations from a program because they 'burden developers' and confuse users. Even if the translations are not actively updated and degenerates with strings changing, some users find them helpful. -- Best regards, Michał Górny
On 22.02.2021 11:20, Michał Górny wrote:
But there is. As was pointed to above, the extra legacy code makes making _any_ changes in the corresponding part of the file more difficult -- because one has to think how to combine the changes with that code. E.g.: at which side of that code to place the changes; if it's a part of a block statement, where in the block statement to place the changes; if I want to change the program structure, how to incorporate that code into the new structure -- all made much more difficult because I cannot be sure what changes would and wouldn't break the "legacy" code. So, by your logic, it would be "blocking" any change to that file and will have to be removed anyway the first time we change that file. Now, if that's the case -- why should we spent time and effort tracking which files were "cleansed" and which weren't (which we'll have to do because such cleansing is an unrelated change in a PR so a PR's author should know whether they need to look out to do any "cleansing" as well as making the change that does what they want) if we can cleanse them all at once and be done with it?
-- Regards, Ivan
Hello, On Mon, 22 Feb 2021 09:20:46 +0100 Michał Górny <mgorny@gentoo.org> wrote:
That's the problem CPython developers have - they like to remind they're volunteers (it's not too far-fetched to say they do that work as a hobby), but they also want to play big corporate types who with a flick of a wrist shut down systems and throw users in the cold. It all reminds another recent drama with the "cryptography" package: https://github.com/pyca/cryptography/issues/5771 . And it's not surprising that some people on that thread is also here. Likewise, it's not surprising that some Debian people are on this thread, with a clear message, just as they have in https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e (which happened to be linked from the ticket above). There's a conflict between 2 forms of volunteering. Old-school, still practiced by Debian, where people volunteer to *maintain* things for the benefit of other people. And new-style, along the lines "Hereby I volunteer to smash some Rust in the face of unsuspecting users" or "I volunteer to pull the rag under the feet of users of some systems". -- Best regards, Paul mailto:pmiscml@gmail.com
On Sun, Feb 21, 2021 at 8:57 PM Michał Górny <mgorny@gentoo.org> wrote:
Maintaining a buildbot doesn't mean to look at it every 6 months. It means getting emails multiple times per month about real bug which must be fixed. My main annoyance is that every single buildbot failure sends me an email, and I'm overwhelmed by emails (I'm not only getting emails from buildbots ;-)). Python has already a long list of buildbot workers (between 75 and 100, I'm not sure of the exact number) and they require a lot of attention. Over the last 5 years, there is basically only Pablo Galindo and me who pay attention to them. I fear that if more buildbots are added, Pablo and me will be the only ones to look at ones. FYI if nobody looks at buildbots, they are basically useless. They only waste resources. To have an idea of the existing maintenance burden, look at emails sent to: https://mail.python.org/archives/list/buildbot-status@python.org/ Every single email is basically a problem. There are around 110 emails over the last 30 years: 3.6 email/day in average. When a bug is identified, it requires an investigation which takes between 5 minutes and 6 months depending on the bug. I would say 2 hours in average. Sometimes, if the investigation is too long, we simply revert the change. The buildbot configuration also requires maintenance. For example, 14 commits have been pushed since January 1st: https://github.com/python/buildmaster-config/commits/master Multiple buildbots are "unstable": tests are failing randomly. Again, each failure means a new email. For example, test_asyncio likes to fail once every 10 runs (coarse average, I didn't check exactly). multiprocessing tests, tests using network like imaplib or nntplib, and some other tests fail randomly. Some tests just fail because just once, the buildbot became slower. People have many ideas to automate bug triage from emails, but so far, nobody came with a concrete working solution, and so emails are still read manually one by one. Also, almost nobody is trying to fix tests which are failing randomly. For example, I called multiple times for help to fix test_asyncio, so far it's still randomly every day: * 2020: https://mail.python.org/archives/list/python-dev@python.org/message/Y7I5ADXA... * 2019: https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGE... By the way, these random failures are not only affecting buildbots, but also CIs run on pull requests. It's *common* that these failures prevent to merge a pull request and require manual actions to be able to merge the PR (usually, re-run all CIs). I'm not talking about exotic platforms with very slow hardware, but platorms like Linux/x86-64 with "fast" hardware. I expect more random errors on exotic platforms. For example, I reported a crash on AIX one year ago, and nobody fixed it so far. I pushed a few fixes for that crash, but it's not enough to fully fix it: https://bugs.python.org/issue40068 I pushed AIX fixes only because I was annoyed by getting buildbot emails about AIX failures. Sometimes, I just turn off emails from AIX. Since there is no proactive work on fixing AIX issues, I would even prefer to *remove* the AIX buildbots. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Mon, Feb 22, 2021 at 12:51 PM Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
IIRC I suggested earlier that buildsbots should be integrated into the PR workflow in order to make it the contributor's rather than a core dev's burden to fix any breakages that result from their changes.
Some buildbot worker take 2 to 3 hours per build. Also, it would not scale. Buildbots are not fast enough to handle the high number of PR and PR updates. When there is clear relationship between a buildbot failure and a merged PR, a comment is added automatically showing the failed test and explanation how to investigate the issue. It's there for 2 years thanks to Pablo, and so far, I rarely saw developers paying attention to these failures. They just ignore it. I'm not trying to blame anyone. Contributing to Python requires a lot of free time. I'm only trying to explain in length what does the "maintenance burden" mean in practice, since some people are pretending that supporting a platform is free. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2/22/2021 6:58 AM, Victor Stinner wrote:
The problems is that many 'breakages' do NOT result from the PR changes. For IDLE patches, EVERY CI failure other than in test_idle is a bogus failure. My productivity is already degraded by the current blind automated rules. My point here is that machines literally have no judgment, and that automation is not the panacea that people think. Writing good rules is often hard to impossible, and blind application of inadequate rules leads to bad results.
This is false and an example of the above comment. The notice is sent whenever any buildbot test fails after a merge, regardless of whether there is any relationship or not. There often (usually?) is not.
Every one of those big, bold, ugly notices I have received has been a false positive. I believe they are usually due to one of the flaky tests (asyncio or otherwise) that should have been disabled but have not been. Of course people ignore routinely false positives. If there is an expert for the module whose test file failed, that person should be the target of the notice. If a PR or merge breaks test_idle, I would like to know. Actually, because I am conscientious, and because there was one instance years ago where test_idle passed CI and failed on one buildbot, and because the notice lists which test file failed (in among the noise), I do check that line before deleting the notice.
Blind automated blind rules are also not free. -- Terry Jan Reedy
On 2/22/2021 6:20 AM, Victor Stinner wrote:
30 days, not years. The 20 visible messages are all have title f"Buildbot worker {worker} missing". All the messages on a day are sent at the same time. Replace with a daily "Builbot workers currently missing"? The individual messages seem useless except possibly if sent to individual buildbot owners. ...
I strongly feel that the individual repeatedly failing tests (only 3 or 4 I think) should be disabled and the asyncio people notified. As I remember, you once insisted on keeping routinely failing tests.
For backports, it means closing the backport and reopening by re-adding a backport label to the master PR. This is why I really really want the repeatedly failing tests disabled. -- Terry Jan Reedy
This, and an earlier letter about the burned of a release manager, confirmes my suspicion that a large share of "maintenance burden" comes from unresolved process issues. E.g. looks like Victor had been unconvinced that unreliable tests are a serious issue and cost the team so much (including himself since they cause ppl to ignore his buildbot issue reports right back) and had blocked the effort to disable them or whatnot (this is just my guess but Terry's message suggests that he was against resolving it). I hope this convinces him and will let the team clear this issue at last! Likewise, half of the bullet points in https://mail.python.org/archives/list/python-dev@python.org/message/LJ3E2UQB... comes from either ppl trying to bypass the process, or the release manager doing others' jobs that should've already been done had the process been followed. On 22.02.2021 19:48, Terry Reedy wrote:
-- Regards, Ivan
On Sun, Feb 21, 2021 at 4:16 AM Victor Stinner <vstinner@python.org> wrote:
FWIW, BeOS may be resurfacing somewhat, with https://en.wikipedia.org/wiki/Haiku_(operating_system) - it's on a second beta now.
On 2/21/21 1:13 PM, Victor Stinner wrote:
In short, I propose to move maintenance of the legacy platforms/archs outside Python: people are free to continue support them as patches.
I'm not attached to s390. But I doubt that the proposed patch lowers any "maintenance burden". https://github.com/python/cpython/pull/24534.diff touches a few files - configure.ac: PLATFORM_TRIPLET is undefined with your patch, changing the extension suffix. I don't think that maintaining such a delta outside the Python tree would be good practice. I am committed to maintain these definitions, and would like to ask not to remove *any* of these. These definitions come from https://wiki.debian.org/Multiarch/Tuples. - Modules/_ctypes/libffi_osx/ffi.c: Removing a macro usage in a file specific to MacOS X. Removal doesn't help, otherwise might create a patch conflict, if the MacOS X maintainers should ever decide to pull in a new libffi upstream version. - Lib/test/test_sysconfig.py: Doesn't simplify the code, just makes a bit more unfriendly. I became curious about a s390 build, and checked it with a GCC multilib build as available on at least Debian, OpenSuse, RHEL and Ubuntu. Builds fine with CC="gcc -m31" CXX="g++ -m31" ./configure && make && make test. No test failures, although I didn't build all extensions using externals libs. Funny thing, your proposed patch doesn't make any difference in the test results, so I'm not sure why you spent any volunteer time on that patch. With the threat^Wannouncement in issue 43179 to do similar removals for alpha, hppa and m68k, you'll likely step onto more toes, and again not removing any burden.
If a platform stops working, fine. Actively breaking a platform in some form: Not ok from my point of view. Python still is ubiquitous on almost all platforms (linux and non-linux ones), and can be built with mostly generic dependencies. Maybe libffi is an exception, however even these maintainers didn't remove s390 support (https://sourceware.org/libffi/). And I was very happy to build Python on a proprietary *nix platform five years ago, and having a working Python.
In fact, the PEP 11 has clear and explicit rules: https://www.python.org/dev/peps/pep-0011/#supporting-platforms
PEP 11 is not fit for that. You're not trying to remove support for the Linux platform, you're addressing a specific architecture. Maybe PEP 11 should be updated. As seen with S390, there's not much architecture specific code. If I see issues on the Linux platform for different architectures, these usually are: (1) 32bit vs 64bit (2) big endian vs. little endian (3) architecture specific alignment requirements, sometimes resulting in degraded performance on unaligned accesses, sometimes in bus errors when running 32bit code on 64 kernels. Looking at https://pythondev.readthedocs.io/platforms.html (yes, that gets the platform/architecture distinction right), - (1) and (2) are covered as "well supported" platforms, although having just x86* might lead to some x86-isms. - (1), (2) and (3) are covered as "best effort support" with aarch64 (64/LE), powerpc64le (64/LE), s390x (64/BE), and arm-linux-gnueabi* as 32/LE having bus errors on unaligned accesses with 64bit kernels. Unless CPython stops supporting these platforms above, I don't see little value of removing that tiny amount of architecture specific code. Document what is supported, be inclusive about anything else. Don't make a distinction yet between legacy and upcoming new architectures. Apparently the cryptography project made the decision to rely on a build tool which is not ubiquitously available anymore. If CPython sees the need for such restrictions in the future, I'd like to propose to use the same deprecation process for modules/functions, e.g. if a new non-ubiquitous tool X is needed to build/run, announce it for 3.10, and only make use of it in 3.11. Matthias
Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements.
On 2/22/2021 5:18 PM, Matthias Klose wrote:
I agree with this, and I don't see any reason why we shouldn't just use the list of stable buildbot platforms as the "supported" list. That makes it really clear what the path is to push support onto upstream (join up and bring a buildbot with you), and also means that we've got a physically restricted set of machines to prove work before doing a release. Actively blocking anything at all seems unnecessary at the source/build level. That's for pre-built binaries and other conveniences. Cheers, Steve
On 2/22/21, 1:39 PM, "Steve Dower" <steve.dower@python.org> wrote: On 2/22/2021 5:18 PM, Matthias Klose wrote: > On 2/21/21 1:13 PM, Victor Stinner wrote: > Document what is supported, be inclusive about anything else. Don't make a > distinction yet between legacy and upcoming new architectures. I agree with this, and I don't see any reason why we shouldn't just use the list of stable buildbot platforms as the "supported" list. That makes it really clear what the path is to push support onto upstream (join up and bring a buildbot with you), and also means that we've got a physically restricted set of machines to prove work before doing a release. Actively blocking anything at all seems unnecessary at the source/build level. That's for pre-built binaries and other conveniences. Cheers, Steve _______________________________________________ +1 to this. I use a few unsupported platforms, not as a hobby, but in my work. I generally don't require ALL the things in python to work on these platforms, so even if I were to contribute a buildbot for an obscure (but definitely not "hobby") platform, it's implied that I would also need to fix test failures in modules I don't use. I generally need to provide some reason for me to use my employer's time, so I can't really justify fixing test failures in code the company doesn't use. I think these users are just asking that what currently works not be broken intentionally, and that follows the spirit of Autoconf, to test if something works and enable if it when it does. There are also other ways to reduce the burden of maintaining a large number of platforms, such as abstracting away the OS services. The other thing that crept into this thread was the mention of test that intermittently fail. That's a huge problem because it suggests that applications will sometimes fail. I have usually seen these sort of issues because of 1) Uninitialized memory being used (read) 2) Threading problems 3) Resources used (files, networking, daemons) but unavailable 4) memory mis-management (buffer overrun that doesn't cause a crash) #3 is probably best fixed by testing for resources and skipping when unavailable The others are problems in the code, and can be fixed with clang sanitizers, but Without some routine running of them, those sorts of problems will reappear.
On 2/22/21, 4:06 PM, "Antoine Pitrou" <antoine@python.org> wrote: On Mon, 22 Feb 2021 19:50:43 +0000 Rob Boehne <robb@datalogics.com> wrote: > > The other thing that crept into this thread was the mention of test that intermittently fail. > That's a huge problem because it suggests that applications will sometimes fail. > I have usually seen these sort of issues because of > 1) Uninitialized memory being used (read) > 2) Threading problems > 3) Resources used (files, networking, daemons) but unavailable > 4) memory mis-management (buffer overrun that doesn't cause a crash) > > #3 is probably best fixed by testing for resources and skipping when unavailable 5) Poor quality POSIX support in the target platform. The Python test suite is actually quite demanding in this regard (except on Windows). Regards Antoine. Antoine: I find this a bit of a surprise, can you give an example of poor POSIX support leading to intermittent test failures? Thanks, Rob Boehne
On Tue, 23 Feb 2021 16:45:27 +0000 Rob Boehne <robb@datalogics.com> wrote:
Hmm, I don't have any precise examples off the top of my head (that was several years ago), but issues mixing pthreads, signals and/or fork come to mind. OpenBSD was quite bad at that especially. Regards Antoine.
Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements.
Actually, fewer. Most architectures have alignof(x) == sizeof(x) for all the primitive types, but m68k is more relaxed and caps alignof(x) at 2. This means that assert((p & sizeof(long)) == 0) is too strict, and should instead just be assert((p & alignof(long)) == 0), which is always correct, rather than relying on implementation-defined alignment requirements. In autoconf there's AC_CHECK_ALIGNOF just as there is AC_CHECK_SIZEOF, the result of which should be used for the alignment check instead. That's the portable way to do it (and would allow the removal of the #ifdef).
On Mon, 2021-02-22 at 19:27 +0000, Jessica Clarke wrote:
I agree, except that -- as I mentioned elsewhere -- the #ifdef was added because the x86 optimization hack is actually slower on m68k. I suspect that if more benchmarking was made, it might turn out that #ifdef should actually disable it on more platforms. -- Best regards, Michał Górny
Michał Górny wrote:
I think it's more complicated than that. The code in question didn't just have a bogus assert, it actually relied on ALIGNOF_SIZE_T == SIZEOF_SIZE_T in order to work, but without much good reason other than being written in a poor style. I suspect that the slowdown was seen because the strictness of the `if` on m68k meant the optimised version wasn't used that often but was still sitting there using up space, and time to evaluate the branch, plus potentially the various consequences of additional register pressure, and that the performance hit goes away once the algorithm is fixed to be more general (in such a way that other architectures shouldn't see any performance hits, and possibly even a slight improvement). I've done this in https://github.com/python/cpython/pull/24624. Plus, even if m68k is slightly slower, who cares? It still works, better to have working clean code than hacky slightly-faster code.
On 21/02/2021 13.13, Victor Stinner wrote:
Thanks Victor! (short reply, I'm heading out) I'm +1 in general for your proposal. I also like the idea to adopt Rust's platform support definition. For 3.10 I propose to add a configure option to guard builds on unsupported / unstable platforms. My draft PR https://github.com/python/cpython/pull/24610/commits/f8d2d56757a9cec7ae4dc72... implements a checker for unsupported platforms and adds a --enable-unstable-platforms flag. Configuration on unsupported platforms fails unless users explicitly opt-in. The checker serves two purposes: 1) It gives users an opportunity to provide full PEP 11 support (buildbot, engineering time) for a platform. 2) It gives us the leverage to remove a flagged platform in the future or refuse support on BPO. Christian
Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example. So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac. Adrian
On 21/02/2021 13.47, glaubitz@debian.org wrote:
Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 platforms. There is no separate configure guard. All platforms that Rust can build for, are always enabled by default. No one in Rust keeps anyone from cross-compiling code for sparc64 or powerpcspe, for example.
So if you want to copy Rust's mechanism, you should just leave it as is and not claim that users are being confused because "m68k" shows up in configure.ac.
A --enable-unstable-platforms configure flag is my peace offer to meet you half way. You get a simple way to enable builds on untested platforms and we can clearly communicate that some OS and hardware platforms are not supported.
On Sun, Feb 21, 2021 at 10:15 AM Christian Heimes <christian@python.org> wrote:
I personally wouldn't want to maintain such a check in autoconf, but it'll be an isolated thing on its own, that if you or someone else creates, will do its job and not bother the rest of us. I think just publishing our list of (1) supported, (2) best-effort non-release-blocker quasi-supported, and (3) explicitly unsupported in a policy doc is sufficient. But it's not like any of us are going to stop someone from codifying that in configure.ac to require a flag. -gps
On 2/21/21 9:24 PM, Gregory P. Smith wrote:
agreed with (1) and(2). I don't like a negative list, as this will be incomplete at any time for both *-linux and *-non{linux,win,mac}. Looking at another project like GCC https://gcc.gnu.org/buildstat.html used to collect information about successful builds on various platforms and architectures. But as you can see, the web page isn't updated in recent times. GCC also doesn't keep an explicit list of the lesser supported platforms: https://gcc.gnu.org/gcc-11/criteria.html Matthias
On Sun, Feb 21, 2021 at 12:28 PM Gregory P. Smith <greg@krypto.org> wrote:
If we add a compile flag to explicitly make people have to realize they are running on an unsupported platform then I think it should be a negation against an allowlist versus a blocklist to be more explicit about what we would take PRs for.
I like the idea of making PEP 11 list what platforms *are* supported in some way, and being off that list means you're not. I also like the idea of having a tier 1 that will block a release and a tier 2 where we will accept PRs but it will in no way block releases. I also think who ends up on either of those tiers should be an SC problem. Based on how heated this thread has gotten there's obviously some emotional connection for some folks when it comes to whether a platform is supported or not and how that is handled. In that sense, letting the SC take on the burden of saying "no" makes sense. That doesn't mean PEP 11 wouldn't still list out the *minimum* requirements to add a platform, but I don't think it should be an automatic thing simply because a machine was donated and a hand was raised as e.g. #ifdefs have a cognitive cost. So, my suggestion is to update PEP 11 to: - List platforms that can block releases as a "tier 1" supported platform - List platforms that are best effort as "tier 2" and which will never hold up a release - All other platforms will need to manage their patchset externally and we will not accept PRs that are specifically for them - Specify what the *minimum* requirements are to add support for a platform to either tier - Have the SC manage what platforms can end up in what tier (and we can publish guidelines like conditional tier 2 to prove support exists, what is required to graduate to tier 1, removal from tiers, etc.)
There are zero technical reasons for what you are planning here. You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder. I could understand the maintenance burden argument if there was actually any maintenance burden, there isn't. The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux"). I have explained that these assumptions are not correct and that downstreams do actively use Python in ways that upstream no longer considers supported, yet you want to make changes to make everything for downstreams harder. I have not seen any other upstream project that is so bothered about a few lines of autoconf code. All other upstream projects I have worked with, be it Rust, OpenJDK, abseil-cpp and so on: None of them had problems when I sent patches to improve the code on more architectures. But in Python, it's suddenly a problem and you start long discussions about a few lines of autoconf code that pose zero maintenance burden. I absolutely don't get it. Adrian
On Mon, Feb 22, 2021 at 8:19 AM <glaubitz@debian.org> wrote:
There are zero technical reasons for what you are planning here.
Multiple core developers explained how it's a maintenance burden. It has been explained in multiple different ways.
You are inflating a few lines of autoconf into a "platform support", so you have a reason to justify adding multiple lines of extra autoconf codes to make life for downstream distributions harder.
"Making life harder" sounds to me like oh, maybe supporting one additional platform is not free and comes with a cost. This cost is something called the "maintenance burden". My question is if Python wants to pay this cost, or if we want transfering the maintenance burden to people who actually care about these legacy platforms and architectures. Your position is: Python must pay this price. My position is: Python should not. Honestly, if it's just a few lines, it will be trivial for you to maintain a downstream patch and I'm not sure why we even need this conversation. If it's more than a few lines, well, again, we come back to the problem of the real maintenance burden.
The thing is you made assumptions about how downstream distributions use Python without doing some research first ("16-bit m68k-linux").
I'm talking about 16-bit memory alignment which causes SIGBUS if it's not respected on m68k. For example, unicodeobject.c requires special code just for this arch: /* * Issue #17237: m68k is a bit different from most architectures in * that objects do not use "natural alignment" - for example, int and * long are only aligned at 2-byte boundaries. Therefore the assert() * won't work; also, tests have shown that skipping the "optimised * version" will even speed up m68k. */ #if !defined(__m68k__) (...) Such issue is hard to guess when you write code and usually only spot it while actually running the code on such architecture. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2/22/21 12:30 PM, Victor Stinner wrote:
Well, that doesn't mean these statements are correct. Please don't assume that the people you are talking to are inexperienced developers, we aren't. Downstream distribution maintainers certainly have enough experience with project maintenance to be able to asses whether your claims are valid or not.
Please explain to me how guarding some platforms with *additional* lines autoconf is helping to reduce maintenance burden for the upstream project.
No, my position is that such changes should have valid technical reasons which is simply not the case. You're not helping your point if you are basing your arguments on incorrect technical assumptions.
This argument goes both ways. The code we are talking about here are just a few lines of autoconf which are hardly touched during normal development work. And the architecture-mapping you have in [1] is probably not even needed (CC @jrtc27).
This is the only such place in the code where there is an extra section for m68k that I could find. And the bug was fixed by Andreas Schwab [2], so another downstream maintainer which was my point earlier in the discussion. We downstreams care about the platform support, hence we keep it working. Thanks, Adrian
[1] https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e... [2] https://github.com/python/cpython/commit/8b0e98426dd0e1fde93715256413bc70775...
-- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On Mon, 2021-02-22 at 12:30 +0100, Victor Stinner wrote:
I don't understand why you consider this to be a problem. After all, x86 has stronger (32-bit) alignment requirements, so m68k is actually less likely to break.
Unless I'm reading something wrong, the code is disabled not because it m68k is broken (as your comment seems to imply) but because the assert is wrong and because the code turned out to be slower. That said, I wonder if this 'optimized' path has been actually benchmarked on other supported platforms. -- Best regards, Michał Górny
Barry Scott wrote:
That is not the problem. Many architectures (SPARC, PA-RISC, IA-64, ...) do not natively support unaligned accesses. The problem is in fact that m68k's alignment requirements are *weaker* than x86 *and they are exposed to C*. On x86, the compiler still aligns ints to 4 byte boundaries, but on m68k they are only aligned to 2 byte boundaries. This means that assert(p % sizeof(int) == 0) does not work. The code in question is #ifdef'ed out for three reasons: 1. The assert is overly strict. This is trivially fixed by changing the SIZEOF_SIZE_T to ALIGNOF_SIZE_T. 2. The `if` uses (and code within the `if` relies on it using) SIZEOF_SIZE_T to check the alignment. 3. The code is potentially slower on m68k than the simple byte-by-byte loop, though I don't think anyone's going to complain about slight performance regressions on m68k if it comes from cleaning up the code, and I imagine the supposed performance hit came from not fixing 2 properly (i.e. there was a bunch of extra code that wasn't able to be used that often due to overly-strict requirements in the algorithm). I have filed https://github.com/python/cpython/pull/24624 to fix all these, though the first smaller commit is the only one strictly required for correctness on m68k (and any other architecture that chooses/has chosen to make the same ill-advised choices in its ABI), whilst the second larger one makes minor changes to the algorithm (that should not affect performance on any supported architecture other than m68k), and the many copies of it, in order to also cope with ALIGNOF_SIZE_T < SIZEOF_SIZE_T. This thus improves m68k support whilst removing m68k-specific hacks and making the code less reliant on implementation-defined behaviour, i.e. is how portability patches are _meant_ to be.
Hello! On 2/22/21 12:30 PM, Victor Stinner wrote:
Just as a heads-up: There is a PR by Jessica Clarke now [1] which gets rid of this architecture-specific #ifdef. I think this is a good approach as it gets rid of one of your complaining points. I have already verified that these changes don't break on 32-bit PowerPC, 64-bit SPARC and, of course, M68k. Thanks, Adrian
-- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On 2/21/21 7:12 PM, Christian Heimes wrote:
Making a "peace offer" is the confession of having started a war. I don't know why you are on this crusade, but I doubt it is the lonely "confused user" cited in https://bugs.python.org/issue43179. Others pointed out in this thread that it's not the first time of such unfriendly behavior. Even others might also see it as some kind of harassment. Matthias
(Since my first reply here somehow got lost, I'm posting this again). Rust doesn't prevent anyone from building Tier 2 or Tier 3 targets. There is no limitation for "legacy" or "deprecated" targets. Any target can be built and any target can be selected by the Rust compiler for cross-compliation. So, if you want to mimic the behavior of the Rust compiler, you would obviously not guard any architectures or platforms in configure.ac. I am objecting this change as it does not reduce code complexity (on the contrary) and only causes more maintenance burden for downstream distributions. Thanks, Adrian
On Sun, 2021-02-21 at 13:35 +0100, Christian Heimes wrote:
Does that mean that if someone offers to run the build bot for a minor platform and do the necessary maintenance to keep it working, they will be able to stay? How much maintenance is actually expected, i.e. is it sufficient to maintain CPython in a 'good enough' working state to resolve major bugs blocking real usage on these platforms? -- Best regards, Michał Górny
On Sun, Feb 21, 2021 at 12:03 PM Michał Górny <mgorny@gentoo.org> wrote:
Definitely start with this. This level of effort to maintain the minor platform support in-tree may be able to keep an otherwise neglected minor platform in second tier non-release-blocker best effort status. Having a buildbot, at least provides visibility, and sometimes PR authors will be sympathetic to easy edits that keep something working that way (no guarantee). The main thing from a project maintenance perspective is for platforms to not become a burden to other code maintainers. PRs need to be reviewed. Every #if/#endif in code is a cognitive burden. So being a minor platform can come with unexpected breakages that need fixing due to other changes made in the codebase that did not pay attention to the platform. As we cannot expect everyone working on code to care about anything beyond the tier-1 fully supported platforms, buildbot or not. Example: I consider many of the BSDs and the Solaris derivatives to be in this state. (non specific here, i don't even know which ones we claim are supported or not without going and reading whatever policy docs we might or might not have today - Victor alludes to this state of the world). We tend to accept patches when someone offers them. Occasionally we have a core dev who actually runs one of them. But most of us don't go out of the way ourselves to try and keep changes we make working there. We expect interested parties to jump in when something isn't working right. And will generally do related PR reviews/merges if they're not burdensome. An example of the above happening recently is VxWorks support via https://bugs.python.org/issue31904. -gps
Yes, but all this doesn't apply here. We're literally talking about a few lines of autoconf and preprocessor defines [1]. In fact, the change to remove "s390 support" (in quotes because these few lines don't really can be considered platform-specific code), introduces more lines than it removes because of the comments. Adrian
On Sun, 2021-02-21 at 13:04 -0800, Gregory P. Smith wrote:
I have to disagree -- the support code (even if any is actually necessary) does not have to be a burden. Generally 'hobbyists' don't have a problem that the support for their platform becomes broken accidentally, or even deliberately because it blocks something else. They understand that others don't have hardware, time or motivation to maintain support for their platform properly. They themselves have limited time to work on it. So it's entirely fine for things to break occasionally, and they provide fixes as their time permits. They don't ask others to maintain their code. There's no real maintenance burden involved. In fact, this whole thread feels like removing 80%-complete translations from a program because they 'burden developers' and confuse users. Even if the translations are not actively updated and degenerates with strings changing, some users find them helpful. -- Best regards, Michał Górny
On 22.02.2021 11:20, Michał Górny wrote:
But there is. As was pointed to above, the extra legacy code makes making _any_ changes in the corresponding part of the file more difficult -- because one has to think how to combine the changes with that code. E.g.: at which side of that code to place the changes; if it's a part of a block statement, where in the block statement to place the changes; if I want to change the program structure, how to incorporate that code into the new structure -- all made much more difficult because I cannot be sure what changes would and wouldn't break the "legacy" code. So, by your logic, it would be "blocking" any change to that file and will have to be removed anyway the first time we change that file. Now, if that's the case -- why should we spent time and effort tracking which files were "cleansed" and which weren't (which we'll have to do because such cleansing is an unrelated change in a PR so a PR's author should know whether they need to look out to do any "cleansing" as well as making the change that does what they want) if we can cleanse them all at once and be done with it?
-- Regards, Ivan
Hello, On Mon, 22 Feb 2021 09:20:46 +0100 Michał Górny <mgorny@gentoo.org> wrote:
That's the problem CPython developers have - they like to remind they're volunteers (it's not too far-fetched to say they do that work as a hobby), but they also want to play big corporate types who with a flick of a wrist shut down systems and throw users in the cold. It all reminds another recent drama with the "cryptography" package: https://github.com/pyca/cryptography/issues/5771 . And it's not surprising that some people on that thread is also here. Likewise, it's not surprising that some Debian people are on this thread, with a clear message, just as they have in https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e (which happened to be linked from the ticket above). There's a conflict between 2 forms of volunteering. Old-school, still practiced by Debian, where people volunteer to *maintain* things for the benefit of other people. And new-style, along the lines "Hereby I volunteer to smash some Rust in the face of unsuspecting users" or "I volunteer to pull the rag under the feet of users of some systems". -- Best regards, Paul mailto:pmiscml@gmail.com
On Sun, Feb 21, 2021 at 8:57 PM Michał Górny <mgorny@gentoo.org> wrote:
Maintaining a buildbot doesn't mean to look at it every 6 months. It means getting emails multiple times per month about real bug which must be fixed. My main annoyance is that every single buildbot failure sends me an email, and I'm overwhelmed by emails (I'm not only getting emails from buildbots ;-)). Python has already a long list of buildbot workers (between 75 and 100, I'm not sure of the exact number) and they require a lot of attention. Over the last 5 years, there is basically only Pablo Galindo and me who pay attention to them. I fear that if more buildbots are added, Pablo and me will be the only ones to look at ones. FYI if nobody looks at buildbots, they are basically useless. They only waste resources. To have an idea of the existing maintenance burden, look at emails sent to: https://mail.python.org/archives/list/buildbot-status@python.org/ Every single email is basically a problem. There are around 110 emails over the last 30 years: 3.6 email/day in average. When a bug is identified, it requires an investigation which takes between 5 minutes and 6 months depending on the bug. I would say 2 hours in average. Sometimes, if the investigation is too long, we simply revert the change. The buildbot configuration also requires maintenance. For example, 14 commits have been pushed since January 1st: https://github.com/python/buildmaster-config/commits/master Multiple buildbots are "unstable": tests are failing randomly. Again, each failure means a new email. For example, test_asyncio likes to fail once every 10 runs (coarse average, I didn't check exactly). multiprocessing tests, tests using network like imaplib or nntplib, and some other tests fail randomly. Some tests just fail because just once, the buildbot became slower. People have many ideas to automate bug triage from emails, but so far, nobody came with a concrete working solution, and so emails are still read manually one by one. Also, almost nobody is trying to fix tests which are failing randomly. For example, I called multiple times for help to fix test_asyncio, so far it's still randomly every day: * 2020: https://mail.python.org/archives/list/python-dev@python.org/message/Y7I5ADXA... * 2019: https://mail.python.org/archives/list/python-dev@python.org/message/R7X6NKGE... By the way, these random failures are not only affecting buildbots, but also CIs run on pull requests. It's *common* that these failures prevent to merge a pull request and require manual actions to be able to merge the PR (usually, re-run all CIs). I'm not talking about exotic platforms with very slow hardware, but platorms like Linux/x86-64 with "fast" hardware. I expect more random errors on exotic platforms. For example, I reported a crash on AIX one year ago, and nobody fixed it so far. I pushed a few fixes for that crash, but it's not enough to fully fix it: https://bugs.python.org/issue40068 I pushed AIX fixes only because I was annoyed by getting buildbot emails about AIX failures. Sometimes, I just turn off emails from AIX. Since there is no proactive work on fixing AIX issues, I would even prefer to *remove* the AIX buildbots. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Mon, Feb 22, 2021 at 12:51 PM Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:
IIRC I suggested earlier that buildsbots should be integrated into the PR workflow in order to make it the contributor's rather than a core dev's burden to fix any breakages that result from their changes.
Some buildbot worker take 2 to 3 hours per build. Also, it would not scale. Buildbots are not fast enough to handle the high number of PR and PR updates. When there is clear relationship between a buildbot failure and a merged PR, a comment is added automatically showing the failed test and explanation how to investigate the issue. It's there for 2 years thanks to Pablo, and so far, I rarely saw developers paying attention to these failures. They just ignore it. I'm not trying to blame anyone. Contributing to Python requires a lot of free time. I'm only trying to explain in length what does the "maintenance burden" mean in practice, since some people are pretending that supporting a platform is free. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2/22/2021 6:58 AM, Victor Stinner wrote:
The problems is that many 'breakages' do NOT result from the PR changes. For IDLE patches, EVERY CI failure other than in test_idle is a bogus failure. My productivity is already degraded by the current blind automated rules. My point here is that machines literally have no judgment, and that automation is not the panacea that people think. Writing good rules is often hard to impossible, and blind application of inadequate rules leads to bad results.
This is false and an example of the above comment. The notice is sent whenever any buildbot test fails after a merge, regardless of whether there is any relationship or not. There often (usually?) is not.
Every one of those big, bold, ugly notices I have received has been a false positive. I believe they are usually due to one of the flaky tests (asyncio or otherwise) that should have been disabled but have not been. Of course people ignore routinely false positives. If there is an expert for the module whose test file failed, that person should be the target of the notice. If a PR or merge breaks test_idle, I would like to know. Actually, because I am conscientious, and because there was one instance years ago where test_idle passed CI and failed on one buildbot, and because the notice lists which test file failed (in among the noise), I do check that line before deleting the notice.
Blind automated blind rules are also not free. -- Terry Jan Reedy
On 2/22/2021 6:20 AM, Victor Stinner wrote:
30 days, not years. The 20 visible messages are all have title f"Buildbot worker {worker} missing". All the messages on a day are sent at the same time. Replace with a daily "Builbot workers currently missing"? The individual messages seem useless except possibly if sent to individual buildbot owners. ...
I strongly feel that the individual repeatedly failing tests (only 3 or 4 I think) should be disabled and the asyncio people notified. As I remember, you once insisted on keeping routinely failing tests.
For backports, it means closing the backport and reopening by re-adding a backport label to the master PR. This is why I really really want the repeatedly failing tests disabled. -- Terry Jan Reedy
This, and an earlier letter about the burned of a release manager, confirmes my suspicion that a large share of "maintenance burden" comes from unresolved process issues. E.g. looks like Victor had been unconvinced that unreliable tests are a serious issue and cost the team so much (including himself since they cause ppl to ignore his buildbot issue reports right back) and had blocked the effort to disable them or whatnot (this is just my guess but Terry's message suggests that he was against resolving it). I hope this convinces him and will let the team clear this issue at last! Likewise, half of the bullet points in https://mail.python.org/archives/list/python-dev@python.org/message/LJ3E2UQB... comes from either ppl trying to bypass the process, or the release manager doing others' jobs that should've already been done had the process been followed. On 22.02.2021 19:48, Terry Reedy wrote:
-- Regards, Ivan
On Sun, Feb 21, 2021 at 4:16 AM Victor Stinner <vstinner@python.org> wrote:
FWIW, BeOS may be resurfacing somewhat, with https://en.wikipedia.org/wiki/Haiku_(operating_system) - it's on a second beta now.
On 2/21/21 1:13 PM, Victor Stinner wrote:
In short, I propose to move maintenance of the legacy platforms/archs outside Python: people are free to continue support them as patches.
I'm not attached to s390. But I doubt that the proposed patch lowers any "maintenance burden". https://github.com/python/cpython/pull/24534.diff touches a few files - configure.ac: PLATFORM_TRIPLET is undefined with your patch, changing the extension suffix. I don't think that maintaining such a delta outside the Python tree would be good practice. I am committed to maintain these definitions, and would like to ask not to remove *any* of these. These definitions come from https://wiki.debian.org/Multiarch/Tuples. - Modules/_ctypes/libffi_osx/ffi.c: Removing a macro usage in a file specific to MacOS X. Removal doesn't help, otherwise might create a patch conflict, if the MacOS X maintainers should ever decide to pull in a new libffi upstream version. - Lib/test/test_sysconfig.py: Doesn't simplify the code, just makes a bit more unfriendly. I became curious about a s390 build, and checked it with a GCC multilib build as available on at least Debian, OpenSuse, RHEL and Ubuntu. Builds fine with CC="gcc -m31" CXX="g++ -m31" ./configure && make && make test. No test failures, although I didn't build all extensions using externals libs. Funny thing, your proposed patch doesn't make any difference in the test results, so I'm not sure why you spent any volunteer time on that patch. With the threat^Wannouncement in issue 43179 to do similar removals for alpha, hppa and m68k, you'll likely step onto more toes, and again not removing any burden.
If a platform stops working, fine. Actively breaking a platform in some form: Not ok from my point of view. Python still is ubiquitous on almost all platforms (linux and non-linux ones), and can be built with mostly generic dependencies. Maybe libffi is an exception, however even these maintainers didn't remove s390 support (https://sourceware.org/libffi/). And I was very happy to build Python on a proprietary *nix platform five years ago, and having a working Python.
In fact, the PEP 11 has clear and explicit rules: https://www.python.org/dev/peps/pep-0011/#supporting-platforms
PEP 11 is not fit for that. You're not trying to remove support for the Linux platform, you're addressing a specific architecture. Maybe PEP 11 should be updated. As seen with S390, there's not much architecture specific code. If I see issues on the Linux platform for different architectures, these usually are: (1) 32bit vs 64bit (2) big endian vs. little endian (3) architecture specific alignment requirements, sometimes resulting in degraded performance on unaligned accesses, sometimes in bus errors when running 32bit code on 64 kernels. Looking at https://pythondev.readthedocs.io/platforms.html (yes, that gets the platform/architecture distinction right), - (1) and (2) are covered as "well supported" platforms, although having just x86* might lead to some x86-isms. - (1), (2) and (3) are covered as "best effort support" with aarch64 (64/LE), powerpc64le (64/LE), s390x (64/BE), and arm-linux-gnueabi* as 32/LE having bus errors on unaligned accesses with 64bit kernels. Unless CPython stops supporting these platforms above, I don't see little value of removing that tiny amount of architecture specific code. Document what is supported, be inclusive about anything else. Don't make a distinction yet between legacy and upcoming new architectures. Apparently the cryptography project made the decision to rely on a build tool which is not ubiquitously available anymore. If CPython sees the need for such restrictions in the future, I'd like to propose to use the same deprecation process for modules/functions, e.g. if a new non-ubiquitous tool X is needed to build/run, announce it for 3.10, and only make use of it in 3.11. Matthias
Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements.
On 2/22/2021 5:18 PM, Matthias Klose wrote:
I agree with this, and I don't see any reason why we shouldn't just use the list of stable buildbot platforms as the "supported" list. That makes it really clear what the path is to push support onto upstream (join up and bring a buildbot with you), and also means that we've got a physically restricted set of machines to prove work before doing a release. Actively blocking anything at all seems unnecessary at the source/build level. That's for pre-built binaries and other conveniences. Cheers, Steve
On 2/22/21, 1:39 PM, "Steve Dower" <steve.dower@python.org> wrote: On 2/22/2021 5:18 PM, Matthias Klose wrote: > On 2/21/21 1:13 PM, Victor Stinner wrote: > Document what is supported, be inclusive about anything else. Don't make a > distinction yet between legacy and upcoming new architectures. I agree with this, and I don't see any reason why we shouldn't just use the list of stable buildbot platforms as the "supported" list. That makes it really clear what the path is to push support onto upstream (join up and bring a buildbot with you), and also means that we've got a physically restricted set of machines to prove work before doing a release. Actively blocking anything at all seems unnecessary at the source/build level. That's for pre-built binaries and other conveniences. Cheers, Steve _______________________________________________ +1 to this. I use a few unsupported platforms, not as a hobby, but in my work. I generally don't require ALL the things in python to work on these platforms, so even if I were to contribute a buildbot for an obscure (but definitely not "hobby") platform, it's implied that I would also need to fix test failures in modules I don't use. I generally need to provide some reason for me to use my employer's time, so I can't really justify fixing test failures in code the company doesn't use. I think these users are just asking that what currently works not be broken intentionally, and that follows the spirit of Autoconf, to test if something works and enable if it when it does. There are also other ways to reduce the burden of maintaining a large number of platforms, such as abstracting away the OS services. The other thing that crept into this thread was the mention of test that intermittently fail. That's a huge problem because it suggests that applications will sometimes fail. I have usually seen these sort of issues because of 1) Uninitialized memory being used (read) 2) Threading problems 3) Resources used (files, networking, daemons) but unavailable 4) memory mis-management (buffer overrun that doesn't cause a crash) #3 is probably best fixed by testing for resources and skipping when unavailable The others are problems in the code, and can be fixed with clang sanitizers, but Without some routine running of them, those sorts of problems will reappear.
On 2/22/21, 4:06 PM, "Antoine Pitrou" <antoine@python.org> wrote: On Mon, 22 Feb 2021 19:50:43 +0000 Rob Boehne <robb@datalogics.com> wrote: > > The other thing that crept into this thread was the mention of test that intermittently fail. > That's a huge problem because it suggests that applications will sometimes fail. > I have usually seen these sort of issues because of > 1) Uninitialized memory being used (read) > 2) Threading problems > 3) Resources used (files, networking, daemons) but unavailable > 4) memory mis-management (buffer overrun that doesn't cause a crash) > > #3 is probably best fixed by testing for resources and skipping when unavailable 5) Poor quality POSIX support in the target platform. The Python test suite is actually quite demanding in this regard (except on Windows). Regards Antoine. Antoine: I find this a bit of a surprise, can you give an example of poor POSIX support leading to intermittent test failures? Thanks, Rob Boehne
On Tue, 23 Feb 2021 16:45:27 +0000 Rob Boehne <robb@datalogics.com> wrote:
Hmm, I don't have any precise examples off the top of my head (that was several years ago), but issues mixing pthreads, signals and/or fork come to mind. OpenBSD was quite bad at that especially. Regards Antoine.
Example: 16-bit m68k no, it's a 32bit platform with extra alignment requirements.
Actually, fewer. Most architectures have alignof(x) == sizeof(x) for all the primitive types, but m68k is more relaxed and caps alignof(x) at 2. This means that assert((p & sizeof(long)) == 0) is too strict, and should instead just be assert((p & alignof(long)) == 0), which is always correct, rather than relying on implementation-defined alignment requirements. In autoconf there's AC_CHECK_ALIGNOF just as there is AC_CHECK_SIZEOF, the result of which should be used for the alignment check instead. That's the portable way to do it (and would allow the removal of the #ifdef).
On Mon, 2021-02-22 at 19:27 +0000, Jessica Clarke wrote:
I agree, except that -- as I mentioned elsewhere -- the #ifdef was added because the x86 optimization hack is actually slower on m68k. I suspect that if more benchmarking was made, it might turn out that #ifdef should actually disable it on more platforms. -- Best regards, Michał Górny
Michał Górny wrote:
I think it's more complicated than that. The code in question didn't just have a bogus assert, it actually relied on ALIGNOF_SIZE_T == SIZEOF_SIZE_T in order to work, but without much good reason other than being written in a poor style. I suspect that the slowdown was seen because the strictness of the `if` on m68k meant the optimised version wasn't used that often but was still sitting there using up space, and time to evaluate the branch, plus potentially the various consequences of additional register pressure, and that the performance hit goes away once the algorithm is fixed to be more general (in such a way that other architectures shouldn't see any performance hits, and possibly even a slight improvement). I've done this in https://github.com/python/cpython/pull/24624. Plus, even if m68k is slightly slower, who cares? It still works, better to have working clean code than hacky slightly-faster code.
participants (17)
-
Antoine Pitrou -
Barry Scott -
Brett Cannon -
Christian Heimes -
Dan Stromberg -
glaubitz＠debian.org -
Gregory P. Smith -
Ivan Pozdeev -
Jessica Clarke -
John Paul Adrian Glaubitz -
Matthias Klose -
Michał Górny -
Paul Sokolovsky -
Rob Boehne -
Steve Dower -
Terry Reedy -
Victor Stinner