![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
Hi, NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface: numpy.distutils.system_info.get_info("blas_opt") But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output: Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 -L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy. Is there a way to access to silence that output? Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface. thanks Frédéric
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien <nouiz@nouiz.org> wrote:
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line. Chuck
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
I found a line like this in the file: numpy/distutils/fcompiler/__init__.py I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called. The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything. I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr. What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4. Fred On Fri, Nov 15, 2013 at 12:40 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it. David
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
If it don't change, currently it mean that each process that use Theano and use BLAS will have that printed: Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 Not very nice. As told, in Theano we do that and it work on Mac, Linux and Windows in 32 and 64 bits. Yes, we have our own wrapper around it to handle windows. But as it is already done and tested in those cases, I think it is a good idea to do it. Do you think that it should be tested in other environment? thanks Frédéric p.s. There is also warning printed, but I can hide them without change in NumPy. p.p.s The first line isn't yet removed in my local tests, so maybe more is needed. On Fri, Nov 15, 2013 at 2:28 PM, David Cournapeau <cournape@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:39 PM, Frédéric Bastien <nouiz@nouiz.org> wrote:
I don't worry much about unix-like environments, windows with 'native' compilers is my main worry. Currently, building scipy with VS 2008 and ifort causes weird issues because scipy recently gained lots of files to be compiled, and sometimes, for some reason, the compilers crash. See: http://mail.scipy.org/pipermail/scipy-dev/2013-August/019169.html Now, if using subprocess fixes those, that alone would be a big +1 from me in its favor. Do you have the patch somewhere I can look at for testing ? David
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau <cournape@gmail.com> wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <
charlesr.harris@gmail.com> wrote: trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it. It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes. -- Robert Kern
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern <robert.kern@gmail.com> wrote:
Indeed, it does not have to be, but that's useful for debugging compilation issues (not so much for numpy itself, but for some packages which have files that takes a very long time to build, like scipy.sparsetools or bottleneck). That's a minor point compared to the potential issues when building on windows, though. David
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
I didn't forgot this, but I got side tracked. Here is the Theano code I would like to try to use to replace os.system: https://github.com/Theano/Theano/blob/master/theano/misc/windows.py But I won't be able to try this before next week. Fred On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau <cournape@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
Hi, After more investigation, I found that there already exist a way to suppress those message on posix system. So I reused it in the PR. That way, it was faster, but prevent change in that area. So there is less change of breaking other syste: https://github.com/numpy/numpy/pull/4081 But it remove the stdout when we run this command: numpy.distutils.system_info.get_info("blas_opt") But during compilation, we still have the info about what is found: atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib64/atlas -lptf77blas -lptcblas -latlas -o _configtest success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/opt/lisa/os_v2/common/Canopy_64bit/User/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib'] NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries lapack_atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/lib64/atlas numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] Frédéric On Fri, Nov 22, 2013 at 4:26 PM, Frédéric Bastien <nouiz@nouiz.org> wrote:
![](https://secure.gravatar.com/avatar/cd77a1586454ca2d9ee66c9ef292dba0.jpg?s=120&d=mm&r=g)
Since there was opposition to just removing the output all the time, I've added a new parameter, 'verbosity' that can be set to 0 to hide the output. This unfortunately requires a bit of code churn and changes the interface to get_info() (but in a backwards-compatible way). I came up with another option, which is to reuse system_info.verbosity to control whether that output is shown/hidden. It is already documented as doing so even if that is mostly ignored in the code. I've made another PR https://github.com/numpy/numpy/pull/4387 to show what I mean. This would supersede #4081 if accepted. 2013-11-27 13:44 GMT-05:00 Frédéric Bastien <nouiz@nouiz.org>:
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien <nouiz@nouiz.org> wrote:
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line. Chuck
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
I found a line like this in the file: numpy/distutils/fcompiler/__init__.py I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called. The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything. I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr. What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4. Fred On Fri, Nov 15, 2013 at 12:40 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it. David
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
If it don't change, currently it mean that each process that use Theano and use BLAS will have that printed: Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 Not very nice. As told, in Theano we do that and it work on Mac, Linux and Windows in 32 and 64 bits. Yes, we have our own wrapper around it to handle windows. But as it is already done and tested in those cases, I think it is a good idea to do it. Do you think that it should be tested in other environment? thanks Frédéric p.s. There is also warning printed, but I can hide them without change in NumPy. p.p.s The first line isn't yet removed in my local tests, so maybe more is needed. On Fri, Nov 15, 2013 at 2:28 PM, David Cournapeau <cournape@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:39 PM, Frédéric Bastien <nouiz@nouiz.org> wrote:
I don't worry much about unix-like environments, windows with 'native' compilers is my main worry. Currently, building scipy with VS 2008 and ifort causes weird issues because scipy recently gained lots of files to be compiled, and sometimes, for some reason, the compilers crash. See: http://mail.scipy.org/pipermail/scipy-dev/2013-August/019169.html Now, if using subprocess fixes those, that alone would be a big +1 from me in its favor. Do you have the patch somewhere I can look at for testing ? David
![](https://secure.gravatar.com/avatar/764323a14e554c97ab74177e0bce51d4.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau <cournape@gmail.com> wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <
charlesr.harris@gmail.com> wrote: trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it. It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes. -- Robert Kern
![](https://secure.gravatar.com/avatar/59bdb3784070f0a6836aca9ee03ad817.jpg?s=120&d=mm&r=g)
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern <robert.kern@gmail.com> wrote:
Indeed, it does not have to be, but that's useful for debugging compilation issues (not so much for numpy itself, but for some packages which have files that takes a very long time to build, like scipy.sparsetools or bottleneck). That's a minor point compared to the potential issues when building on windows, though. David
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
I didn't forgot this, but I got side tracked. Here is the Theano code I would like to try to use to replace os.system: https://github.com/Theano/Theano/blob/master/theano/misc/windows.py But I won't be able to try this before next week. Fred On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau <cournape@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/082c34b6c1544c177cb7a66609360089.jpg?s=120&d=mm&r=g)
Hi, After more investigation, I found that there already exist a way to suppress those message on posix system. So I reused it in the PR. That way, it was faster, but prevent change in that area. So there is less change of breaking other syste: https://github.com/numpy/numpy/pull/4081 But it remove the stdout when we run this command: numpy.distutils.system_info.get_info("blas_opt") But during compilation, we still have the info about what is found: atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib64/atlas -lptf77blas -lptcblas -latlas -o _configtest success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/opt/lisa/os_v2/common/Canopy_64bit/User/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib'] NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries lapack_atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/lib64/atlas numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] include_dirs = ['/usr/include'] Frédéric On Fri, Nov 22, 2013 at 4:26 PM, Frédéric Bastien <nouiz@nouiz.org> wrote:
![](https://secure.gravatar.com/avatar/cd77a1586454ca2d9ee66c9ef292dba0.jpg?s=120&d=mm&r=g)
Since there was opposition to just removing the output all the time, I've added a new parameter, 'verbosity' that can be set to 0 to hide the output. This unfortunately requires a bit of code churn and changes the interface to get_info() (but in a backwards-compatible way). I came up with another option, which is to reuse system_info.verbosity to control whether that output is shown/hidden. It is already documented as doing so even if that is mostly ignored in the code. I've made another PR https://github.com/numpy/numpy/pull/4387 to show what I mean. This would supersede #4081 if accepted. 2013-11-27 13:44 GMT-05:00 Frédéric Bastien <nouiz@nouiz.org>:
participants (5)
-
Arnaud Bergeron
-
Charles R Harris
-
David Cournapeau
-
Frédéric Bastien
-
Robert Kern