Hi all, not sure if this is the best place to ask for this. If not, please advise on the correct place. Since the numpy wheels internally use openBLAS, operations can be implicitly multithreaded directly by openBLAS. This, of course, can clash with multithreading or parallel processing. The recommended practice in this case is to set export OPENBLAS_NUM_THREADS=1 in the environment. However, I would like to be able to adjust this directly in my python code. Is there a way to control this directly through Python, whether through numpy or not? Thanks for your time! -- Jean-Christophe Houde
os.environ can be used to change environment variables from within python. https://docs.python.org/2/library/os.html#os.environ I do not know when openBLAS is reading the environment variables though. Changing a value while your python process is running might be to late. Jean-Christophe Houde writes:
Hi all,
not sure if this is the best place to ask for this. If not, please advise on the correct place.
Since the numpy wheels internally use openBLAS, operations can be implicitly multithreaded directly by openBLAS.
This, of course, can clash with multithreading or parallel processing. The recommended practice in this case is to set
export OPENBLAS_NUM_THREADS=1
in the environment. However, I would like to be able to adjust this directly in my python code.
Is there a way to control this directly through Python, whether through numpy or not?
Thanks for your time!
On 28 Sep 2017, at 3:37 pm, Max Linke
os.environ can be used to change environment variables from within python.
https://docs.python.org/2/library/os.html#os.environ
I do not know when openBLAS is reading the environment variables though. Changing a value while your python process is running might be to late.
It should use the value that is set at the time a BLAS routine is called. At least I can confirm that this works analogously within Fortran programs setting the *NUM_THREADS variables at runtime. HTH Derek
This came up for Biopython recently (someone using our
library on a cluster ran into thread limits triggered by the
importing of NumPy), and suggested something like this:
import os
try:
os.environ["OMP_NUM_THREADS"] = "1"
import numpy
finally:
del os.environ["OMP_NUM_THREADS"]
Or MKL_NUM_THREADS, or apparently also it might
be OPENBLAS_NUM_THREADS as well:
https://github.com/biopython/biopython/pull/1401
Peter
On Thu, Sep 28, 2017 at 2:26 PM, Jean-Christophe Houde
Hi all,
not sure if this is the best place to ask for this. If not, please advise on the correct place.
Since the numpy wheels internally use openBLAS, operations can be implicitly multithreaded directly by openBLAS.
This, of course, can clash with multithreading or parallel processing. The recommended practice in this case is to set
export OPENBLAS_NUM_THREADS=1
in the environment. However, I would like to be able to adjust this directly in my python code.
Is there a way to control this directly through Python, whether through numpy or not?
Thanks for your time!
-- Jean-Christophe Houde
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (4)
-
Derek Homeier
-
Jean-Christophe Houde
-
Max Linke
-
Peter Cock