On Wed, 30 Jun 2010 09:00:09 PDT Bill Janssen wrote:

So, my question then is, why are these skips "unexpected"? Seems to me that if this is the case, this test will never run on any platform.

You can change the value of the "usepty" option in your buildbot.tac. (you will also have to restart the buildslave process) Regards Antoine.

On 06:46 pm, exarkun@twistedmatrix.com wrote:

On 04:44 pm, solipsis@pitrou.net wrote:

...

On Wed, 30 Jun 2010 09:00:09 PDT Bill Janssen wrote:

...

So, my question then is, why are these skips "unexpected"? Seems to me that if this is the case, this test will never run on any platform.

You can change the value of the "usepty" option in your buildbot.tac. (you will also have to restart the buildslave process)

But don't do this. The usepty option is completely unrelated to the suggestion I was making. Flipping it to True will only cause other things to break and have no impact on this test.

Ah, sorry. I confused myself. The option is related. But it will also break other things, so I still would recommend looking for other solutions. Jean-Paul

On 05:24 am, martin@v.loewis.de wrote:

...

Seems to work fine. So this I don't understand. Any ideas, anyone?

Didn't we discuss this before? The buildbot slave has no controlling terminal anymore, hence it cannot open /dev/tty. If you are curious, just patch your checkout to output the exact errno (e.g. to stdout), and trigger a build through the web.

Could the test be rewritten (or supplemented) to use a pty? Most or perhaps all of the same operations should be supported. Jean-Paul

On 04:26 pm, janssen@parc.com wrote:

exarkun@twistedmatrix.com wrote:

...

Could the test be rewritten (or supplemented) to use a pty? Most or perhaps all of the same operations should be supported.

Buildbot seems to be explicitly not using a PTY. From the the top of the test output:

make buildbottest in dir /Users/buildbot/buildarea/trunk.parc-leopard-1/build (timeout 1800 secs) watching logfiles {} argv: ['make', 'buildbottest'] [...] closing stdin using PTY: False

This output is telling you that the build slave isn't giving the child processes it creates a pty. What I had in mind was writing the test to create a new pty, instead of trying to use the controlling tty. So basically, the two things are completely unrelated and this buildbot configuration isn't hurting anything (and in fact is likely helping quite a few things, so I suggest leaving it alone).

I believe this is specified by the build master.

This test seems to work on Ubuntu and FreeBSD, though.

That's interesting. I wonder if those slaves are able to open /dev/tty for some reason? The slave is supposed to detach from the controlling terminal when it daemonizes. There could be a bug in that code, I suppose, or the slaves could be running without daemonization for some reason. The operators would have to tell us about that, I think. Or, another possibility is that /dev/tty doesn't work how I expect it to and on Ubuntu and FreeBSD it can be opened even if you don't have a controlling terminal. Hopefully not, though. Jean-Paul

On 05:29 pm, martin@v.loewis.de wrote:

Am 30.06.2010 13:32, schrieb exarkun@twistedmatrix.com:

...

On 05:24 am, martin@v.loewis.de wrote:

...

...

Seems to work fine. So this I don't understand. Any ideas, anyone?

Didn't we discuss this before? The buildbot slave has no controlling terminal anymore, hence it cannot open /dev/tty. If you are curious, just patch your checkout to output the exact errno (e.g. to stdout), and trigger a build through the web.

Could the test be rewritten (or supplemented) to use a pty? Most or perhaps all of the same operations should be supported.

I'm not sure. It uses TIOCGPGRP, basically to establish that ioctl can also put results into a Python array (IIUC). This goes back to http://bugs.python.org/555817

Somebody rewriting it would need to make sure the original test purpose is still met.

Absolutely. And even so, it may still make sense to run the test against both /dev/tty and a pty (or whatever subset of those things can be acquired in the testing environment). You can do a TIOCGPGRP on a new pty (created by os.openpty) but it produces somewhat less interesting results than doing it on /dev/tty. FIONREAD might be a nice alternative. It produces interesting (ie, non- zero) values in an easily predictable/controllable way (it tells you how many bytes are in the read buffer). Jean-Paul

On Wed, Jun 30, 2010 at 09:03, Bill Janssen wrote:

Martin v. Löwis wrote:

...

...

Seems to work fine.  So this I don't understand.  Any ideas, anyone?

Didn't we discuss this before?

Possibly, but I don't recall doing so.

...

The buildbot slave has no controlling terminal anymore, hence it cannot open /dev/tty. If you are curious, just patch your checkout to output the exact errno (e.g. to stdout), and trigger a build through the web.

So, why is skipping this test "unexpected"?  I see "x86 Tiger" is also showing this as an unexpected skip.  Should I just add it to the list of expected skips on Darwin?  Actually, will it run on any platform?

The whole "unexpected" skipping is somewhat of a mess. In an ideal situation modules that are optionally built should be allowed to skip, and on a per-platform basis certain OS-specific tests (whether they be exclusive to a specific OS or run on all OSs except Windows) should be skipped. Otherwise any import failure should be a test failure. The "unexpected" test skipping was meant to solve both of these situations, but in an imperfect way. My PSF grant proposal to work on Python full-time for two to three months after my Ph.D. is complete (assuming the PSF gives me the grant this would start most likely in November or December) includes cleaning up the test suite and this would be the first thing I tackle.

Martin v. Löwis wrote:

...

The whole "unexpected" skipping is somewhat of a mess. In an ideal situation modules that are optionally built should be allowed to skip,

While this may be the wide-spread interpretation, it is definitely *not* the original intention of the feature.

When Tim Peters added it, he wanted it to tell him whether he did the Windows build correctly, INCLUDING ALL OPTIONAL PACKAGES that can possibly work on Windows. If you try to generalize this beyond Windows, then the only skips that are expected are the ones for tests that absolutely cannot work on the platform - i.e. Unix tests on Windows, and Windows tests on Unix. Otherwise, if you can get it to pass by installing additional software, Tim did *not* mean this to be an expected skip.

Perfectly reasonable, good to know. So on my OS X buildbots I should update gdb, tcl/tk, and readline, so that those tests can run. Probably be good to put a note in the regrtest.py comments to this effect, as I don't see a PEP about testing or buildbots. Bill

...

When Tim Peters added it, he wanted it to tell him whether he did the Windows build correctly, INCLUDING ALL OPTIONAL PACKAGES that can possibly work on Windows. If you try to generalize this beyond Windows, then the only skips that are expected are the ones for tests that absolutely cannot work on the platform - i.e. Unix tests on Windows, and Windows tests on Unix. Otherwise, if you can get it to pass by installing additional software, Tim did *not* mean this to be an expected skip.

Interesting. Do you use it that way when you make the Windows build?

You want a honest answer? I usually don't run the test suite on Windows, and trust that the packaging will tell me if an extension module failed to build (and otherwise trust that if the setup worked for the release candidate, it will also work for the final release). Independent of that, I also decided to entirely ignore the notion of expected skipped test (so even if I would run the test suite, I wouldn't bother if one was reported as skipped). Regards, Martin

Brett Cannon wrote:

On Wed, Jun 30, 2010 at 15:01, "Martin v. Löwis" wrote:

...

...

...

When Tim Peters added it, he wanted it to tell him whether he did the Windows build correctly, INCLUDING ALL OPTIONAL PACKAGES that can possibly work on Windows. If you try to generalize this beyond Windows, then the only skips that are expected are the ones for tests that absolutely cannot work on the platform - i.e. Unix tests on Windows, and Windows tests on Unix. Otherwise, if you can get it to pass by installing additional software, Tim did *not* mean this to be an expected skip. Interesting. Do you use it that way when you make the Windows build? You want a honest answer?

Of course. You're amongst friends. =)

...

I usually don't run the test suite on Windows, and trust that the packaging will tell me if an extension module failed to build (and otherwise trust that if the setup worked for the release candidate, it will also work for the final release).

Seems reasonable.

...

Independent of that, I also decided to entirely ignore the notion of expected skipped test (so even if I would run the test suite, I wouldn't bother if one was reported as skipped).

Yeah, I remember you bringing this up years ago. I think that "unexpected" is a bad term and should be renamed to reflect the fact that the test was skipped because an optional third-party package was not included.

A "dependency skip", perhaps?

Or simply output why a certain test was skipped at the end of the test suite run (I know it is outputted when the individual test is skipped, but it would be easier to read that info at the end during the summary of results).

That would be a good idea, I think. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/

On Wed, Jun 30, 2010 at 14:52, Nick Coghlan wrote:

On Thu, Jul 1, 2010 at 5:53 AM, "Martin v. Löwis" wrote:

...

When Tim Peters added it, he wanted it to tell him whether he did the Windows build correctly, INCLUDING ALL OPTIONAL PACKAGES that can possibly work on Windows. If you try to generalize this beyond Windows, then the only skips that are expected are the ones for tests that absolutely cannot work on the platform - i.e. Unix tests on Windows, and Windows tests on Unix. Otherwise, if you can get it to pass by installing additional software, Tim did *not* mean this to be an expected skip.

Note that it works this way on Linux as well. On Kubuntu (for example) you need another half dozen or so additional *-dev packages installed to avoid unexpected test skips.

So it isn't that it's "unexpected", it's that a dependency is missing. So it seems the terminology needs to get tweaked.

On Wed, Jun 30, 2010 at 15:20, Nick Coghlan wrote:

On Thu, Jul 1, 2010 at 7:55 AM, Brett Cannon wrote:

...

So it isn't that it's "unexpected", it's that a dependency is missing. So it seems the terminology needs to get tweaked.

More that the phrase "expected skip" isn't clearly defined and people sometimes guess wrong as to what it means. As Martin pointed out, there are two possible meanings: "will never work on this OS" and "won't work with just the base OS install". Currently, the "expected skip" list is based purely on the former, but developers occasionally interpret it as the latter (as Bill did in this case).

I will note that the first list is much easier to keep up to date, since the latter may vary significantly based on vendor decisions as to what they install by default (a fairly significant factor in the Linux and *BSD worlds).

Adding "(Were all optional modules built successfully?)" to the end of the "skips were unexpected" line in the regrtest output may be enough to eliminate the confusion.

Probably. So I would still want to shift the test-specific info into the tests instead of regrtest and raise a subclass od SkippedTest (or whatever the name of the exception is) to signify that there is a difference. This would also do away with the possibility of having a test get silently skipped by an ImportError even though the module should definitely be available (didn't that bite you once, or was that someone else?).

[Barry Warsaw]

As an aside, I wonder if it would make sense to create an Ubuntu/Debian package that essentially just depended on all those -dev packages. That way you could install this python-uber-dev package and get everything you needed to build a sumo Python.

I don’t know. Any Debian advanced user wanting to build a package can just run “aptitude build-dep python2.6”. Aptitude has the nice property of asking for a confirmation, so it’s even possible to cherry-pick from the lists of deps (press “e” at the confirmation prompt). E.g., I choose readline but not tk. So it seems to me that the information and the command are already here.