Freezing Numeric Python code problem under Linux

Hi, i am having troubles freezing some python code with call the Numeric 17.1.1 module. Here is an example of what happens under Linux Red Hat 6.2 and Python 2.0: $ cat t.py import Numeric a = Numeric.ones([10,10]) b = a * 10 print b $ python ~/Python-2.0/Tools/freeze/freeze.py -o tt t.py ... $ cd tt $ ./t Traceback (most recent call last): File "t.py", line 1, in ? File "/scisoft/python/lib/python2.0/site-packages/Numeric/Numeric.py", line 79, in ? import multiarray ImportError: /scisoft/python/lib/python2.0/site-packages/Numeric/multiarray.so: undefined symbol: _Py_NoneStruct The very same code, under Solaris 2.6 and Python 2.0 works just fine. Code which does not use the Numeric package freeze just fine under Linux, so I think that this point to some problem/incompatibility of Numeric with freeze.py. Does anybody have a suggestion, or a work-around?? Nor

When I initialize an array, I use a = ones(shape)*initial_val But, I am wondering if Numpy has more efficient way. For example, a = array(initial_value, shape) Peter

Daehyok Shin wrote:
I don't know if it's any more efficient (what you have is pretty fast already), but another option is to use resize:
shape = (3,4)
initial_val = 5.0
resize(initial_val,shape)
array([[ 5., 5., 5., 5.], [ 5., 5., 5., 5.], [ 5., 5., 5., 5.]])
-Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------

Comparing the performance, resize() seems to be more effient than ones(). Daehyok Shin ----- Original Message ----- From: "Chris Barker" <cbarker@jps.net> Cc: <numpy-discussion@lists.sourceforge.net> Sent: Wednesday, November 29, 2000 11:13 AM Subject: Re: [Numpy-discussion] Initialization of array?

"DS" == Daehyok Shin <sdhyok@email.unc.edu> writes:
DS> When I initialize an array, I use a = ones(shape)*initial_val DS> But, I am wondering if Numpy has more efficient way. For example, DS> a = array(initial_value, shape) Looking at the definition of "ones": def ones(shape, typecode='l', savespace=0): """ones(shape, typecode=Int, savespace=0) returns an array of the given dimensions which is initialized to all ones. """ return zeros(shape, typecode, savespace)+array(1, typecode) It looks like you could try a=zeros(shape)+initial_val instead. Hm.. I might do some experimenting. Rob -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========

Initialization on huge arrays is frequent operations in scientific programming. It must be efficient as much as possible. So, I was surprisized to see the inner codes of ones() in Numpy. It maybe use calloc() rather than malloc() in C level, then for(..) for addition. Why not use malloc() and for(...) simultaneously in C level with a command such as: a = arrray(1,shape=(10000,10000)) Daehyok Shin ----- Original Message ----- From: "Rob W. W. Hooft" <rob@hooft.net> To: "Daehyok Shin" <sdhyok@email.unc.edu> Cc: <numpy-discussion@lists.sourceforge.net> Sent: Wednesday, November 29, 2000 1:00 PM Subject: Re: [Numpy-discussion] Initialization of array?

I had another look at the definition of "ones" and of another routine I frequently use: arange. It appears that even without rewriting them in C, some speedup can be achieved: - in ones(), the + 1 should be done "in place", saving about 15%, more if you run out of processor cache: amigo[167]~%3% /usr/local/bin/python test_ones.py Numeric.ones 10 -> 0.098ms Numeric.ones 100 -> 0.103ms Numeric.ones 1000 -> 0.147ms Numeric.ones 10000 -> 0.830ms Numeric.ones 100000 -> 11.900ms Numeric.zeros 10 -> 0.021ms Numeric.zeros 100 -> 0.022ms Numeric.zeros 1000 -> 0.026ms Numeric.zeros 10000 -> 0.290ms Numeric.zeros 100000 -> 4.000ms Add inplace 10 -> 0.091ms Add inplace 100 -> 0.094ms Add inplace 1000 -> 0.127ms Add inplace 10000 -> 0.690ms Add inplace 100000 -> 8.100ms Reshape 1 10 -> 0.320ms Reshape 1 100 -> 0.436ms Reshape 1 1000 -> 1.553ms Reshape 1 10000 -> 12.910ms Reshape 1 100000 -> 141.200ms Also notice that zeros() is 4-5 times faster than ones(), so it may pay to reimplement ones in C as well (it is used in indices() and arange()). The "resize 1" alternative is much slower. - in arange, additional 10% can be saved by adding brackets around (start+(stop-stop)) (in addition to the gain by the faster "ones"): amigo[168]~%3% /usr/local/bin/python test_arange.py Numeric.arange 10 -> 0.390ms Numeric.arange 100 -> 0.410ms Numeric.arange 1000 -> 0.670ms Numeric.arange 10000 -> 4.100ms Numeric.arange 100000 -> 59.000ms Optimized 10 -> 0.340ms Optimized 100 -> 0.360ms Optimized 1000 -> 0.580ms Optimized 10000 -> 3.500ms Optimized 100000 -> 48.000ms Regards, Rob Hooft -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========

I'd also try assignment in place: def ones(shape, typecode='l', savespace=0): a = zeros(shape, typecode, savespace) a[len(shape)*[slice(0, None)]] = 1 return a Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

KH> I had another look at the definition of "ones" and of another routine KH> I frequently use: arange. It appears that even without rewriting them KH> in C, some speedup can be achieved: KH> KH> - in ones(), the + 1 should be done "in place", saving about 15%, more KH> if you run out of processor cache: KH> I'd also try assignment in place: KH> def ones(shape, typecode='l', savespace=0): KH> a = zeros(shape, typecode, savespace) KH> a[len(shape)*[slice(0, None)]] = 1 KH> return a This is even faster, but it is better to write "a[...] = 1", because your manual calculation of "..." gives a large overhead for small arrays. On another machine this time: Numeric.ones 10 -> 0.254ms Numeric.ones 100 -> 0.268ms Numeric.ones 1000 -> 0.340ms Numeric.ones 10000 -> 1.960ms Numeric.ones 100000 -> 29.300ms Numeric.zeros 10 -> 0.055ms Numeric.zeros 100 -> 0.059ms Numeric.zeros 1000 -> 0.068ms Numeric.zeros 10000 -> 0.430ms Numeric.zeros 100000 -> 9.800ms Add inplace 10 -> 0.246ms Add inplace 100 -> 0.255ms Add inplace 1000 -> 0.312ms Add inplace 10000 -> 1.270ms Add inplace 100000 -> 18.100ms Assign inplace 10 -> 0.192ms Assign inplace 100 -> 0.201ms Assign inplace 1000 -> 0.242ms Assign inplace 10000 -> 1.010ms Assign inplace 100000 -> 16.300ms Reshape 1 10 -> 0.842ms Reshape 1 100 -> 1.175ms Reshape 1 1000 -> 4.100ms Reshape 1 10000 -> 35.100ms Reshape 1 100000 -> 368.600ms Rob -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========

When I initialize an array, I use a = ones(shape)*initial_val But, I am wondering if Numpy has more efficient way. For example, a = array(initial_value, shape) Peter

Daehyok Shin wrote:
I don't know if it's any more efficient (what you have is pretty fast already), but another option is to use resize:
shape = (3,4)
initial_val = 5.0
resize(initial_val,shape)
array([[ 5., 5., 5., 5.], [ 5., 5., 5., 5.], [ 5., 5., 5., 5.]])
-Chris -- Christopher Barker, Ph.D. cbarker@jps.net --- --- --- http://www.jps.net/cbarker -----@@ -----@@ -----@@ ------@@@ ------@@@ ------@@@ Water Resources Engineering ------ @ ------ @ ------ @ Coastal and Fluvial Hydrodynamics ------- --------- -------- ------------------------------------------------------------------------ ------------------------------------------------------------------------

Comparing the performance, resize() seems to be more effient than ones(). Daehyok Shin ----- Original Message ----- From: "Chris Barker" <cbarker@jps.net> Cc: <numpy-discussion@lists.sourceforge.net> Sent: Wednesday, November 29, 2000 11:13 AM Subject: Re: [Numpy-discussion] Initialization of array?

"DS" == Daehyok Shin <sdhyok@email.unc.edu> writes:
DS> When I initialize an array, I use a = ones(shape)*initial_val DS> But, I am wondering if Numpy has more efficient way. For example, DS> a = array(initial_value, shape) Looking at the definition of "ones": def ones(shape, typecode='l', savespace=0): """ones(shape, typecode=Int, savespace=0) returns an array of the given dimensions which is initialized to all ones. """ return zeros(shape, typecode, savespace)+array(1, typecode) It looks like you could try a=zeros(shape)+initial_val instead. Hm.. I might do some experimenting. Rob -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========

Initialization on huge arrays is frequent operations in scientific programming. It must be efficient as much as possible. So, I was surprisized to see the inner codes of ones() in Numpy. It maybe use calloc() rather than malloc() in C level, then for(..) for addition. Why not use malloc() and for(...) simultaneously in C level with a command such as: a = arrray(1,shape=(10000,10000)) Daehyok Shin ----- Original Message ----- From: "Rob W. W. Hooft" <rob@hooft.net> To: "Daehyok Shin" <sdhyok@email.unc.edu> Cc: <numpy-discussion@lists.sourceforge.net> Sent: Wednesday, November 29, 2000 1:00 PM Subject: Re: [Numpy-discussion] Initialization of array?

I had another look at the definition of "ones" and of another routine I frequently use: arange. It appears that even without rewriting them in C, some speedup can be achieved: - in ones(), the + 1 should be done "in place", saving about 15%, more if you run out of processor cache: amigo[167]~%3% /usr/local/bin/python test_ones.py Numeric.ones 10 -> 0.098ms Numeric.ones 100 -> 0.103ms Numeric.ones 1000 -> 0.147ms Numeric.ones 10000 -> 0.830ms Numeric.ones 100000 -> 11.900ms Numeric.zeros 10 -> 0.021ms Numeric.zeros 100 -> 0.022ms Numeric.zeros 1000 -> 0.026ms Numeric.zeros 10000 -> 0.290ms Numeric.zeros 100000 -> 4.000ms Add inplace 10 -> 0.091ms Add inplace 100 -> 0.094ms Add inplace 1000 -> 0.127ms Add inplace 10000 -> 0.690ms Add inplace 100000 -> 8.100ms Reshape 1 10 -> 0.320ms Reshape 1 100 -> 0.436ms Reshape 1 1000 -> 1.553ms Reshape 1 10000 -> 12.910ms Reshape 1 100000 -> 141.200ms Also notice that zeros() is 4-5 times faster than ones(), so it may pay to reimplement ones in C as well (it is used in indices() and arange()). The "resize 1" alternative is much slower. - in arange, additional 10% can be saved by adding brackets around (start+(stop-stop)) (in addition to the gain by the faster "ones"): amigo[168]~%3% /usr/local/bin/python test_arange.py Numeric.arange 10 -> 0.390ms Numeric.arange 100 -> 0.410ms Numeric.arange 1000 -> 0.670ms Numeric.arange 10000 -> 4.100ms Numeric.arange 100000 -> 59.000ms Optimized 10 -> 0.340ms Optimized 100 -> 0.360ms Optimized 1000 -> 0.580ms Optimized 10000 -> 3.500ms Optimized 100000 -> 48.000ms Regards, Rob Hooft -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========

I'd also try assignment in place: def ones(shape, typecode='l', savespace=0): a = zeros(shape, typecode, savespace) a[len(shape)*[slice(0, None)]] = 1 return a Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------

KH> I had another look at the definition of "ones" and of another routine KH> I frequently use: arange. It appears that even without rewriting them KH> in C, some speedup can be achieved: KH> KH> - in ones(), the + 1 should be done "in place", saving about 15%, more KH> if you run out of processor cache: KH> I'd also try assignment in place: KH> def ones(shape, typecode='l', savespace=0): KH> a = zeros(shape, typecode, savespace) KH> a[len(shape)*[slice(0, None)]] = 1 KH> return a This is even faster, but it is better to write "a[...] = 1", because your manual calculation of "..." gives a large overhead for small arrays. On another machine this time: Numeric.ones 10 -> 0.254ms Numeric.ones 100 -> 0.268ms Numeric.ones 1000 -> 0.340ms Numeric.ones 10000 -> 1.960ms Numeric.ones 100000 -> 29.300ms Numeric.zeros 10 -> 0.055ms Numeric.zeros 100 -> 0.059ms Numeric.zeros 1000 -> 0.068ms Numeric.zeros 10000 -> 0.430ms Numeric.zeros 100000 -> 9.800ms Add inplace 10 -> 0.246ms Add inplace 100 -> 0.255ms Add inplace 1000 -> 0.312ms Add inplace 10000 -> 1.270ms Add inplace 100000 -> 18.100ms Assign inplace 10 -> 0.192ms Assign inplace 100 -> 0.201ms Assign inplace 1000 -> 0.242ms Assign inplace 10000 -> 1.010ms Assign inplace 100000 -> 16.300ms Reshape 1 10 -> 0.842ms Reshape 1 100 -> 1.175ms Reshape 1 1000 -> 4.100ms Reshape 1 10000 -> 35.100ms Reshape 1 100000 -> 368.600ms Rob -- ===== rob@hooft.net http://www.hooft.net/people/rob/ ===== ===== R&D, Nonius BV, Delft http://www.nonius.nl/ ===== ===== PGPid 0xFA19277D ========================== Use Linux! =========
participants (5)
-
Chris Barker
-
Daehyok Shin
-
Konrad Hinsen
-
Nor Pirzkal
-
rob@hooft.net