certification (Brainbench)

[Alex Martelli]

Dr. David Mertz wrote:
        ...
> I don't necessarily have anything against testing in general.  It is
> possible--at least to a first approximation--to learn something about
> people's skill levels by means of tests.  And I wouldn't even go so far
> as to suppose that a complete novice would get the same result as Alex
> Martelli on the Brainbench exam.  It's not as good as it *should* be, but
> it's not completely random either.

Actually, given that Brainbench's "big idea" is _unsupervised_ tests,
said "complete novice" might well turn out to get exactly the same
results as me -- if he or she paid me enough, or was a friend of mine,
and assuming for the sake of argument that I didn't feel particularly
honor-bound to uphold that certificate's integrity.  The "honor system"
is a nice thing, but if a potential employer trusts a potential
employee enough to assume he or she didn't just hire an expert or have
a friend take the test, why don't they trust him or her enough to
just state out his or her level of expertise?

I guess this specific issue might be finessed by having the
_employer_ assign personnel to supervise the test-taking (this, I
gather, isn't the normal way Brainbench tests are taken, but it
would seem feasible).  This would also let the employer set the
environment to however they expect the prospective employee to
have to "perform under pressure": some people's performance might
be impaired by having to respect a dress-code, others' by not
being allowed to smoke, etc -- if each prospective employee can
arrange his or her own favourite test-taking conditions, this may
be meaningful for a prospective teleworking situation, but likely
not for many other job siuations.

A worse issue I have with most tests (supervised or not) is their
"quiz" nature.  Some people are very effective at taking quizzes (I,
for one), and the correlation between that and the ability to
perform useful work, while not null, isn't necessarily all that
strong either.  For Python, at least, I think a few simple,
well-specified programming tasks (one also requiring design,
one working to a pre-specified design, one about fixing a bug
in a given piece of code, another about adding functionality
to another given piece of code) might well be more meaningful.

Quizes are great for fun -- and some technologies don't really
lend themselves well to directly testing a candidate's performance
on somewhat meaningful tasks -- but Python does, so why not take
advantage of that?


Alex