Well I think I found the culprit.  I knew it must be something simple.  And it was.

Our pydisutils.cfg had both an index-url and find_links, both with the same simple url.

When I removed the find_links, all the time sinks went away.  What did take 3hr is now back to 24min.  /facepalm

fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools.  I also thought that setuptools just used pip, but now I see it's actually easy-install, which I didn't realize was a different thing from pip.

Good luck on https://github.com/pypa/setuptools/issues/917