Hi,
NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface:
numpy.distutils.system_info.get_info("blas_opt")
But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 -L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas
I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy.
Is there a way to access to silence that output?
Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface.
thanks
Frédéric
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien nouiz@nouiz.org wrote:
Hi,
NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface:
numpy.distutils.system_info.get_info("blas_opt")
But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
-L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas
I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy.
Is there a way to access to silence that output?
Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface.
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line.
Chuck
On Fri, Nov 15, 2013 at 10:31 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien nouiz@nouiz.org wrote:
Hi,
NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface:
numpy.distutils.system_info.get_info("blas_opt")
But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
-L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas
I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy.
Is there a way to access to silence that output?
Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface.
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line.
In numpy/distutils/__init__.py line 886 try changing log.set_verbosity(-2) to log.set_verbosity(2)
Chuck
On Fri, Nov 15, 2013 at 10:40 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Fri, Nov 15, 2013 at 10:31 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien nouiz@nouiz.orgwrote:
Hi,
NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface:
numpy.distutils.system_info.get_info("blas_opt")
But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
-L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas
I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy.
Is there a way to access to silence that output?
Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface.
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line.
In numpy/distutils/__init__.py line 886 try changing log.set_verbosity(-2) to log.set_verbosity(2)
OK, that isn't right.
Chuck
I found a line like this in the file:
numpy/distutils/fcompiler/__init__.py
I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called.
The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything.
I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr.
What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4.
Fred
On Fri, Nov 15, 2013 at 12:40 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Fri, Nov 15, 2013 at 10:31 AM, Charles R Harris charlesr.harris@gmail.com wrote:
On Fri, Nov 15, 2013 at 8:12 AM, Frédéric Bastien nouiz@nouiz.org wrote:
Hi,
NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface:
numpy.distutils.system_info.get_info("blas_opt")
But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
-L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas
I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy.
Is there a way to access to silence that output?
Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface.
Looks like the problem is in numpy/distutils/exec_command.py and numpy/distutils/log.py. In particular, it looks like a logging problem and I'd guess it may be connected to the debug logs. Also, looks like numpy.distutils.log inherits from distutils.log, which may be obsolete. You might get some control of the log with an environment variable, but the function itself looks largely undocumented. That said, it should probably be printing to stderror when run from the command line.
In numpy/distutils/__init__.py line 886 try changing log.set_verbosity(-2) to log.set_verbosity(2)
Chuck
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Nov 15, 2013 at 11:05 AM, Frédéric Bastien nouiz@nouiz.org wrote:
I found a line like this in the file:
numpy/distutils/fcompiler/__init__.py
I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called.
The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything.
I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr.
What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4.
Numpy 1.8 doesn't support python 2.4 in any case, so that isn't a problem ;)
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Chuck
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Fri, Nov 15, 2013 at 11:05 AM, Frédéric Bastien nouiz@nouiz.orgwrote:
I found a line like this in the file:
numpy/distutils/fcompiler/__init__.py
I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called.
The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything.
I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr.
What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4.
Numpy 1.8 doesn't support python 2.4 in any case, so that isn't a problem ;)
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
David
If it don't change, currently it mean that each process that use Theano and use BLAS will have that printed:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
Not very nice.
As told, in Theano we do that and it work on Mac, Linux and Windows in 32 and 64 bits. Yes, we have our own wrapper around it to handle windows. But as it is already done and tested in those cases, I think it is a good idea to do it.
Do you think that it should be tested in other environment?
thanks
Frédéric
p.s. There is also warning printed, but I can hide them without change in NumPy. p.p.s The first line isn't yet removed in my local tests, so maybe more is needed.
On Fri, Nov 15, 2013 at 2:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Fri, Nov 15, 2013 at 11:05 AM, Frédéric Bastien nouiz@nouiz.org wrote:
I found a line like this in the file:
numpy/distutils/fcompiler/__init__.py
I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called.
The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything.
I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr.
What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4.
Numpy 1.8 doesn't support python 2.4 in any case, so that isn't a problem ;)
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
David
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Nov 15, 2013 at 7:39 PM, Frédéric Bastien nouiz@nouiz.org wrote:
If it don't change, currently it mean that each process that use Theano and use BLAS will have that printed:
Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64
Not very nice.
As told, in Theano we do that and it work on Mac, Linux and Windows in 32 and 64 bits. Yes, we have our own wrapper around it to handle windows. But as it is already done and tested in those cases, I think it is a good idea to do it.
Do you think that it should be tested in other environment?
I don't worry much about unix-like environments, windows with 'native' compilers is my main worry. Currently, building scipy with VS 2008 and ifort causes weird issues because scipy recently gained lots of files to be compiled, and sometimes, for some reason, the compilers crash.
See: http://mail.scipy.org/pipermail/scipy-dev/2013-August/019169.html
Now, if using subprocess fixes those, that alone would be a big +1 from me in its favor. Do you have the patch somewhere I can look at for testing ?
David
thanks
Frédéric
p.s. There is also warning printed, but I can hide them without change in NumPy. p.p.s The first line isn't yet removed in my local tests, so maybe more is needed.
On Fri, Nov 15, 2013 at 2:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris charlesr.harris@gmail.com wrote:
On Fri, Nov 15, 2013 at 11:05 AM, Frédéric Bastien nouiz@nouiz.org wrote:
I found a line like this in the file:
numpy/distutils/fcompiler/__init__.py
I changed the -2 to 2, but it didn't change anything. In fact, this line wasn't called.
The fct set_verbosity() is called only once, with the value of 0. The default value set at import. If I change that to 2 or -2, it do not change anything.
I think that the problem is related to the exec_command as you told. It seam it call it in a way that don't take the stdout/stderr set in Python. So I can't redirect it. The problem is that is use os.system() and we can't redirect its stdout/stderr.
What about replacing the os.system call to subprocess.Popen? This would allow us to catch the stdout/stderr. We use this call in Theano and it is compatible with python 2.4.
Numpy 1.8 doesn't support python 2.4 in any case, so that isn't a
problem
;)
Sure, give it a shot. Looks like subprocess.Popen was intended to
replace
os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is
very
fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
David
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <
charlesr.harris@gmail.com> wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to
replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so
reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes.
-- Robert Kern
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern robert.kern@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris <
charlesr.harris@gmail.com> wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to
replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so
reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes.
Indeed, it does not have to be, but that's useful for debugging compilation issues (not so much for numpy itself, but for some packages which have files that takes a very long time to build, like scipy.sparsetools or bottleneck).
That's a minor point compared to the potential issues when building on windows, though.
David
I didn't forgot this, but I got side tracked. Here is the Theano code I would like to try to use to replace os.system:
https://github.com/Theano/Theano/blob/master/theano/misc/windows.py
But I won't be able to try this before next week.
Fred
On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern robert.kern@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris charlesr.harris@gmail.com wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes.
Indeed, it does not have to be, but that's useful for debugging compilation issues (not so much for numpy itself, but for some packages which have files that takes a very long time to build, like scipy.sparsetools or bottleneck).
That's a minor point compared to the potential issues when building on windows, though.
David
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi,
After more investigation, I found that there already exist a way to suppress those message on posix system. So I reused it in the PR. That way, it was faster, but prevent change in that area. So there is less change of breaking other syste:
https://github.com/numpy/numpy/pull/4081
But it remove the stdout when we run this command:
numpy.distutils.system_info.get_info("blas_opt")
But during compilation, we still have the info about what is found:
atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c':
/* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; }
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -fPIC
compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib64/atlas -lptf77blas -lptcblas -latlas -o _configtest success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/opt/lisa/os_v2/common/Canopy_64bit/User/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib'] NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries lapack_atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/lib64/atlas numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
Frédéric
On Fri, Nov 22, 2013 at 4:26 PM, Frédéric Bastien nouiz@nouiz.org wrote:
I didn't forgot this, but I got side tracked. Here is the Theano code I would like to try to use to replace os.system:
https://github.com/Theano/Theano/blob/master/theano/misc/windows.py
But I won't be able to try this before next week.
Fred
On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern robert.kern@gmail.com wrote:
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris charlesr.harris@gmail.com wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing so reliably on every platform (cough - windows - cough) is not completely trivial. You also have to handle buffered output, etc... That code is very fragile, so this would be quite a lot of testing to change, and I am not sure it worths it.
It doesn't have to be "real time". Just use .communicate() and print out the stdout and stderr to their appropriate streams after the subprocess finishes.
Indeed, it does not have to be, but that's useful for debugging compilation issues (not so much for numpy itself, but for some packages which have files that takes a very long time to build, like scipy.sparsetools or bottleneck).
That's a minor point compared to the potential issues when building on windows, though.
David
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Since there was opposition to just removing the output all the time, I've added a new parameter, 'verbosity' that can be set to 0 to hide the output. This unfortunately requires a bit of code churn and changes the interface to get_info() (but in a backwards-compatible way).
I came up with another option, which is to reuse system_info.verbosity to control whether that output is shown/hidden. It is already documented as doing so even if that is mostly ignored in the code.
I've made another PR https://github.com/numpy/numpy/pull/4387 to show what I mean.
This would supersede #4081 if accepted.
2013-11-27 13:44 GMT-05:00 Frédéric Bastien nouiz@nouiz.org:
Hi,
After more investigation, I found that there already exist a way to suppress those message on posix system. So I reused it in the PR. That way, it was faster, but prevent change in that area. So there is less change of breaking other syste:
https://github.com/numpy/numpy/pull/4081
But it remove the stdout when we run this command:
numpy.distutils.system_info.get_info("blas_opt")
But during compilation, we still have the info about what is found:
atlas_blas_threads_info: Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS customize Gnu95FCompiler Found executable /usr/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config compiling '_configtest.c':
/* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; }
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -O2 -fPIC
compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib64/atlas -lptf77blas -lptcblas -latlas -o _configtest success! removing: _configtest.c _configtest.o _configtest Setting PTATLAS=ATLAS FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
FOUND: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = c define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/opt/lisa/os_v2/common/Canopy_64bit/User/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib'] NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries lapack_atlas not found in /opt/lisa/os_v2/common/Canopy_64bit/User/lib libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64 libraries lapack_atlas not found in /usr/local/lib64 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/lib64/atlas numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
FOUND: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] language = f77 define_macros = [('ATLAS_INFO', '"\"3.8.3\""')] include_dirs = ['/usr/include']
Frédéric
On Fri, Nov 22, 2013 at 4:26 PM, Frédéric Bastien nouiz@nouiz.org wrote:
I didn't forgot this, but I got side tracked. Here is the Theano code I would like to try to use to replace os.system:
https://github.com/Theano/Theano/blob/master/theano/misc/windows.py
But I won't be able to try this before next week.
Fred
On Fri, Nov 15, 2013 at 5:49 PM, David Cournapeau cournape@gmail.com
wrote:
On Fri, Nov 15, 2013 at 7:41 PM, Robert Kern robert.kern@gmail.com
wrote:
On Fri, Nov 15, 2013 at 7:28 PM, David Cournapeau cournape@gmail.com wrote:
On Fri, Nov 15, 2013 at 6:21 PM, Charles R Harris charlesr.harris@gmail.com wrote:
Sure, give it a shot. Looks like subprocess.Popen was intended to replace os.system in any case.
Except that output is not 'real time' with straight Popen, and doing
so
reliably on every platform (cough - windows - cough) is not
completely
trivial. You also have to handle buffered output, etc... That code
is very
fragile, so this would be quite a lot of testing to change, and I am
not
sure it worths it.
It doesn't have to be "real time". Just use .communicate() and print
out
the stdout and stderr to their appropriate streams after the subprocess finishes.
Indeed, it does not have to be, but that's useful for debugging
compilation
issues (not so much for numpy itself, but for some packages which have
files
that takes a very long time to build, like scipy.sparsetools or
bottleneck).
That's a minor point compared to the potential issues when building on windows, though.
David
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (5)
-
Arnaud Bergeron -
Charles R Harris -
David Cournapeau -
Frédéric Bastien -
Robert Kern