[Python-Dev] Stable buildbots

[David Bolen]

Trent Nelson <trent at snakebite.org> writes:

> That's interesting.  (That kill_python.exe doesn't kill the wedged
> processes, but pskill does.)  kill_python is pretty simple, it just
> calls TerminateProcess() after acquiring a handle with the relevant
> PROCESS_TERMINATE access right.  (...)
>
> Are you calling pskill with the -t flag? i.e. kill process and all
> dependents?  That might be the ticket, especially if killing the child
> process that wedged select() is waiting on causes it to return, and
> thus, makes it killable.

Nope, just "pskill python_d".  Haven't bothered to check the pskill
source but I'm assuming it's just a basic TerminateProcess. Ideally my
quickest workaround would just be to replace the kill_python in the
buildbot tools script with that command but of course they could get
updated on checkouts and I'm not arguing it's generally appropriate enough
to belong in the source.

I suspect the problem may be on the "identify which process to kill"
rather than the "kill it" part, but it's definitely going to take time
to figure that out for sure.  While the approach kill_python takes is
much more appropriate, since we don't currently have multiple builds
running simultaneously (and for me the machines are dedicated as build
slaves, so I won't be having my own python_d), a more blanket kill
operation is safe enough.

> Otherwise, if it happens again, can you try kill_python.exe first,
> then pskill, and confirm if the former fails but the latter succeeds?

Yeah, I've got a temporary tree with a built-binary around, but still
have to make sure of the right way to run it manually in a way that it
will do the identification right (which I think also means I need to
figure out from which build tree the hung process started).  Up until
now, typically when I've found a hung setup, the rest of the build
tree which originally applied to that process has been cleaned.

I definitely sympathize with Martin's position though - it wasn't the
simplest tool to write (and I still have some email from him about the
week+ it took just to test the process identification part remotely
through buildbots at the time), so I regret not jumping right in to
try to fix it.  But it's just way more effort than typing "pskill
python_d", at least with my current availability.

-- David