[Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as

[Andrew Bennetts]

On Fri, Oct 07, 2011 at 08:27:01AM +1100, Cameron Simpson wrote:
[…]
> | >> running buildbot tests as root does not reflect the experience of
> | >> non-root users. It seems some tests need to be run both ways just for
> | >> correctness testing.
> | 
> | (except I'd say "all", not "some")
> 
> No. Terry is right and you are ... not. Most tests need no special
> privileges - they're testing language/library semantics that do not
> depend on the system facilities much, and when they do they should work
> for unprivileged users.

You could also say that most tests that work on Linux work on FreeBSD
too, so when they work on Linux they should work for FreeBSD too… so why
bother running tests on FreeBSD at all?  The reason is because the
assumptions behind that “should” are wrong frequently enough to make it
worth running tests in both environments.

Like Glyph, I think that “running as root” is sufficiently different
environment to “running as typical user” (in terms of how POSIX-like
systems behave w.r.t. to things like permissions checks) to make it
worthwhile to regularly run the whole test suite as root.

> HOWEVER, the whole suite should not be _tested_ as root because the code
> being testing is by definition untrusted.

No, that just means you shouldn't trust *root*.  Which is where a VM is
a very useful tool.  You can have the “as root” environment for your
tests without the need to have anything important trust it.

[…]
> Root _is_ special, within the host and with scope to misbehave beyond
> the host.
> 
> 1: The permission system does _not_ behave the same for root as for
>    other users.

Those are arguments *for* running tests as root!

> 2: Root _can_ corrupt things anywhere in the system (within the VM, of
>    course, but the builtbot is a subset of it). A normal unprivileged user

This appears to be a key error in your logic.  There's no fundamental
reason why “tests run as root inside a VM” must necessarily imply
“buildbot process is run inside that same VM and is therefore vulnerable
to code in that test run.”

It may be more convenient to deploy it that way, but I'm sure it's
possible to have a buildslave configured to e.g. start a pristine VM
(from a snapshot with all the necessary build dependencies installed) and
via SSH copy the the source into it, build it, run it, and report the
results.  The VM could be fully isolated from the real network and
filesystem etc if you like.

Given that it is certainly possible to run tests as root about as
securely as running them without root, do you still feel it is not worth
running the tests as root?

> The prudent way to run the buildbots, especially if they cycle (refetch
> newer codebase, rebuilt, retest) instead of (scrub VM, reinstall,
> install built system, etc) is:
> 
>   - a user to fetch source and dispatch builds
>   - possibly a distinct user to run the builds
>   - definitely a distinct user to run the test suite

If we're talking prudence, then surely s/user/VM/ is even better :)

-Andrew.