[New-bugs-announce] [issue29265] test suite is attempting to spawn 258 child processes to run tests

[Brian Vandenberg]

New submission from Brian Vandenberg:

I'm attempting to build python 3.6.0 on sparc/solaris 10.  After the initial configure/compile complete I ran "make test" and I see:

$ make test
running build
running build_ext
(...)
running build_scripts
copying and adjusting (...)
changing mode of (...)
renaming (...)
(...)
Run tests in parallel using 258 child processes


I'm fairly sure the issue stems from the fact that each core on the machine has 8 "threads" and there's 32 cores (for a total of 256 virtual cores).

Each core can execute 8 parallel tasks only in very specific circumstances.  It's intended for use by things like lapack/atlas where you might be doing many computations on the same set of data.

Outside of these more restricted circumstances each core can only handle 2 parallel tasks (or so I gathered from the documentation), so at best this machine could handle 64 backgrounded jobs though I normally restrict my builds to the actual core count or less.

The most common way to get a "realistic" core count on these machines from shell scripts is:

$ core_count=`kstat -m cpu_info | grep core_id | sort -u | wc -l`

... though I'm not sure how the test suite is determining the core count.  I didn't see any mention of "kstat" anywhere.

----------
components: Tests
messages: 285430
nosy: phantal
priority: normal
severity: normal
status: open
title: test suite is attempting to spawn 258 child processes to run tests
type: behavior
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29265>
_______________________________________