Is clang testing necessary?
Hello,
I am curious about the decision to have Travis-CI tests with both clang and gcc. clang is a compiler which strives to be very compatible with gcc. I would understand testing under OS X with clang in addition to Linux with gcc, but I'm skeptical we gain much from testing both gcc and clang under Linux. Also, it lengthens CI times, as clang produces slower code in debug mode than gcc does (because we use "-Og" on gcc).
Or perhaps we could switch the clang build to produce a non-debug interpreter (and therefore a faster test suite)?
Regards
Antoine.
On Fri, 24 Mar 2017 at 06:43 Antoine Pitrou <antoine@python.org> wrote:
Hello,
I am curious about the decision to have Travis-CI tests with both clang and gcc.
It was when I was trying to maximize Travis usefulness before the migration. It's also because we as a project have found bugs in both compilers previously (I know I personally reported a bug that affected both of them once upstream).
clang is a compiler which strives to be very compatible with gcc. I would understand testing under OS X with clang in addition to Linux with gcc, but I'm skeptical we gain much from testing both gcc and clang under Linux. Also, it lengthens CI times, as clang produces slower code in debug mode than gcc does (because we use "-Og" on gcc).
Is there an equivalent flag for clang we should be using?
Or perhaps we could switch the clang build to produce a non-debug interpreter (and therefore a faster test suite)?
Another option is if we drop a compiler from a debug build we can use it for the coverage run so it's at least being exercised somewhere.
If we drop either compiler I would vote for dropping gcc actually as the compiler messages are much worse on gcc compared to clang. I guess it comes down to whether we think CI should be used as a fallback for a failing test suite or if we think CI might help contributors if they happen to have a bug in their C code that they simply didn't notice (e.g. thought they made an innocuous fix that didn't warrant a rebuild).
Even if for most changes, testing gcc and clang is useless, I'm in favor of keeping both for the CI. It doesn't cost much, and it's not that uncommon that Clang catchs bugs or warnings, than GCC doesn't.
The question is more who checks for compiler warnings in the CI :-) Currently, you have to dig into logs, it's not really convenient.
Sometimes, I see compiler warnings when I read buildbots stdout. I'm trying to fix them in a irregular basis :-)
Victor
On 25 March 2017 at 02:24, Brett Cannon <brett@python.org> wrote:
Another option is if we drop a compiler from a debug build we can use it for the coverage run so it's at least being exercised somewhere.
If we drop either compiler I would vote for dropping gcc actually as the compiler messages are much worse on gcc compared to clang. I guess it comes down to whether we think CI should be used as a fallback for a failing test suite or if we think CI might help contributors if they happen to have a bug in their C code that they simply didn't notice (e.g. thought they made an innocuous fix that didn't warrant a rebuild).
Catching "forgot to run make before running the tests" is a useful role, but in that case I'd expect most folks to go "oops" and do a local build, rather than trying to debug compile errors solely from the CI logs (except for the cross-platform case, where something works on Windows but not on *nix, and vice-versa).
However, from the point of view of making it easier for Windows devs to debug *nix debug errors, it probably makes more sense to use clang for the main *nix test run, and then use gcc to do the coverage run.
That setup would give:
- all 3 default compilers running in CI (MSVC in Appveyor, clang for the main Travis tests, gcc for the coverage run)
- Windows devs getting the friendlier clang error messages when they're trying to debug a cross-platform compilation problem
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 25/03/2017 à 16:27, Nick Coghlan a écrit :
However, from the point of view of making it easier for Windows devs to debug *nix debug errors, it probably makes more sense to use clang for the main *nix test run, and then use gcc to do the coverage run.
What's the problem exactly? Situations where MSVC is more lenient than either gcc or clang? Usually it's the reverse :-)
That setup would give:
- all 3 default compilers running in CI (MSVC in Appveyor, clang for the main Travis tests, gcc for the coverage run)
- Windows devs getting the friendlier clang error messages when they're trying to debug a cross-platform compilation problem
- Travis builds are longer because gcc benefits from "-Og" which clang doesn't.
Regards
Antoine.
On 26 March 2017 at 03:54, Antoine Pitrou <antoine@python.org> wrote:
Le 25/03/2017 à 16:27, Nick Coghlan a écrit :
However, from the point of view of making it easier for Windows devs to debug *nix debug errors, it probably makes more sense to use clang for the main *nix test run, and then use gcc to do the coverage run.
What's the problem exactly? Situations where MSVC is more lenient than either gcc or clang? Usually it's the reverse :-)
I'm mainly thinking of situations where the Windows way of doing something is different from the *nix way of doing it (e.g. I hit one in the other direction recently, where setenv becomes something like SetEnvironmentVarW on the Windows side of things).
While those are generally pretty straightforward complaints about undefined symbols, it doesn't hurt to favour the compiler that's known for easier to read messages.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I've gone ahead and dropped the gcc test build and switched the coverage test to gcc. If we find in time that clang's speed is really an issue compared to its better error messages we can swap the compilers' roles.
On Sun, 26 Mar 2017 at 04:31 Nick Coghlan <ncoghlan@gmail.com> wrote:
On 26 March 2017 at 03:54, Antoine Pitrou <antoine@python.org> wrote:
Le 25/03/2017 à 16:27, Nick Coghlan a écrit :
However, from the point of view of making it easier for Windows devs to debug *nix debug errors, it probably makes more sense to use clang for the main *nix test run, and then use gcc to do the coverage run.
What's the problem exactly? Situations where MSVC is more lenient than either gcc or clang? Usually it's the reverse :-)
I'm mainly thinking of situations where the Windows way of doing something is different from the *nix way of doing it (e.g. I hit one in the other direction recently, where setenv becomes something like SetEnvironmentVarW on the Windows side of things).
While those are generally pretty straightforward complaints about undefined symbols, it doesn't hurt to favour the compiler that's known for easier to read messages.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Antoine Pitrou -
Brett Cannon -
Nick Coghlan -
Victor Stinner