Mailman 3 April 2011 - NumPy-Discussion

random number generator, entropy and pickling
by Gael Varoquaux 25 Apr '11

25 Apr '11

Hi there, We are courrently having a discussion on the scikits learn mailing list about which patterns to adopt for random number generation. One thing that is absolutely clear is that making the stream of random numbers reproducible is critical. We have several objects that can serve as random variate generators. So far, we instanciate these objects with a optional seed or PRNG argument, as in: def __init__(self, prng=None): if prng is None: prng = np.random self.prng = prng The problem with this pattern is that np.random doesn't pickle, and therefore the objects do not pickle by default. A bit of pickling magic would solve this, but we'd rather avoid it. We thought that we could simply have a PRNG per object, as in: def __init__(self, prng=None): if prng is None: prng = np.random.RandomState() self.prng = prng I don't like this option, because it means that with a given pieve of code, setting the seed of numpy's PRNG isn't enough to make it reproducible. I couldn't retrieve a handle on a picklable instance for the global PRNG. The only option I can see would be to use the global numpy PRNG to seed an instance specific RandomState, as in: def __init__(self, prng=None): if prng is None: prng = np.random.RandomState(np.random.random()) self.prng = prng That way seeding the global PRNG really does control the full random number generation. I am wondering if it would have an adverse consequence on the entropy of the stream of random numbers. Does anybody have suggestions? Advices? Cheers, Gael

3 5

Re: [Numpy-discussion] [ANN]: OpenPiv. A python package for PIV image analysis
by Alex Liberzon 25 Apr '11

25 Apr '11

Dear Fabrice, Let's take this discussion off-line, contact us directly through http://github.com/alexlib or to e-mail. It's a coincidence but I just finished wrapping the same library using SWIG :). Such a pity I didn't know about your work. I also believe that we talk about the same thing as I did it also to work with PIVMAT and was in touch with its author. We would be happy to use your stuff and include it as a part of OpenPIV. I'm not sure how much can we go into the 3D stuff (although my second big thing is managing the 3D-PTV http://ptv.origo.ethz.ch) and we do have tools for 3D analysis. I'd be very happy to release everything we have in our backyards, Matlab, C, Python, to bring it all together to a serious skikit or similar project based on Numpy/Scipy and Cython for the good of all fluid mechanics experimentalists. Such packages exist actually for the CFD data, I am not aware of the open source and free ones. Would you join our team? Best regards, Alex Liberzon Turbulence Structure Laboratory [http://www.eng.tau.ac.il/turbulencelab] School of Mechanical Engineering Tel Aviv University Le dimanche 24 avril 2011 ? 16:53 +0200, Davide a ?crit : > Hi all, > I am pleased to announce the re-birth of OpenPiv, a python package for > the analysis of PIV images. PIV is a well established optical > measurement technique for fluid flows. The package has the goal of > provide an implementation of state-of-the-art processing algorithms, and > compete with commercial closed-source software packages. > OpenPiv is in alpha state and several thing has to be done. However, the > code is already usable and a pure python implementation of the standard > cross-correlation algorithm is already available. > OpenPiv needs developers and contributors, helping in with code and > documentation. If you can provide some, and if you are interested in > becoming a core developer, please drop us an email at openpiv-develop at > lists dot sourceforge dot net. A draft website can be found at > www.openpiv.net/python. > > Thanks for your attention, > > Davide Lasagna Hi Davide Investigations in fluid mechanics lead me to build a few months a wrapper on the I/O functions from Davis (LaVision), a commercial solution used to handle the complete procedure from camera calibration to postprocessing velocity fields. I had to postprocess previous measurements stored in Davis format to a deeper analysis than their software allows, so that I needed a tool able to read data from their (almost documented) format. The solution I found googling were not satisfying at the moment. For example, PIVMAT [2] only worked for 2D fields and we were dealing with 3D flows and Stereoscopic PIV. After speaking with it author, it appears that it would require too much work to support the third component in a decent way. The ctypes-based wrappers are available here https://launchpad.net/libim7 https://github.com/FabricioS/libim7 It is a quite dumb piece of code, but it allows to extract images or velocity fields from binary files (extensions im7, imx and vc7), avoiding some of the traps in the extraction of the data. I would be happy if you consider useful this tool within OpenPIV. It would also be possible to change the ctype dependance to cython... Best regards [1] http://www.lavision.de/en/products/davis.php [2] http://www.fast.u-psud.fr/pivmat/ -- Fabrice Silva

1 0

using numpy to save arrays in a loop
by Zhaoru Zhang 25 Apr '11

25 Apr '11

Hi guys, I have a time loop and at each time step I get a 1-d array and want to save it in the data file 'data.dat '(or whatever suffix), so finally i got . I made this by using the python 'write' function, but I' thinking about if there is any numpy function that can realize this? Thanks. Zhaoru

2 1

Array as Variable using "from cdms2 import MV2 as MV"
by dileep kunjaai 25 Apr '11

25 Apr '11

Dear sir, I am have 2 mxn numpy array say "obs" & "fcst". I have to calculate * sum of squre of (obs[i, j]-fcst[i, j])* using from cdms2 import MV2 as MV in CDAT without using "for" loop. For example: obs= [0.6 1.1 0.02 0.2 0.2 0.8 0. 0. 0.4 0.8 0.5 5.5 1.5 0.5 1.5 3.5 0.5 1.5 5.0 2.6 5.1 4.1 3.2 2.3 1.5 4.4 0.9 1.5 2. 2.3 1.1 1.1 1.5 12.6 1.3 2.2 12 1.7 1.6 15 1.9 1.5 0.9 2.5 5.5 ] fcst= [0.7 0.1 0.2 0.2 0.2 0.3 0.8 0. 0. 0. 0.5 0.5 0.5 0.5 0.5 0.7 1. 1.5 2. 2.6 5.1 4.1 3.2 2.3 1.5 0.7 1. 1.5 2. 2.3 1.1 1.1 1.1 12.7 1.3 2.2 2. 1.7 1.6 1.5 1.9 1.5 0.9 0.5 7.5] here "obs" and "fcst" are numpy array I give obs=MV.array(obs) fcst=MV.array(fcst) Then it become sbst=obs-fcst >> subst= [[ -0.1 1. -0.18 0. 0. ] [ 0.5 -0.8 0. 0.4 0.8 ] [ 0. 5. 1. 0. 1. ] [ 2.8 -0.5 0. 3. 0. ] [ 0. 0. 0. 0. 0. ] [ 3.7 -0.1 0. 0. 0. ] [ 0. 0. 0.4 -0.1 0. ] [ 0. 10. 0. 0. 13.5 ] [ 0. 0. 0. 2. -2. ]] But i dont know how to find sum of squre of each term....(Actually my aim is to finding MEAN SQUARED ERROR) Thanking you................... -- DILEEPKUMAR. R J R F, IIT DELHI

3 3

[ANN]: OpenPiv. A python package for PIV image analysis
by Davide 25 Apr '11

25 Apr '11

Hi all, I am pleased to announce the re-birth of OpenPiv, a python package for the analysis of PIV images. PIV is a well established optical measurement technique for fluid flows. The package has the goal of provide an implementation of state-of-the-art processing algorithms, and compete with commercial closed-source software packages. OpenPiv is in alpha state and several thing has to be done. However, the code is already usable and a pure python implementation of the standard cross-correlation algorithm is already available. OpenPiv needs developers and contributors, helping in with code and documentation. If you can provide some, and if you are interested in becoming a core developer, please drop us an email at openpiv-develop at lists dot sourceforge dot net. A draft website can be found at www.openpiv.net/python. Thanks for your attention, Davide Lasagna

2 1

cdms2.tvariable.TransientVariable to float variable
by dileep kunjaai 25 Apr '11

25 Apr '11

Dear sir, How to change cdms2.tvariable.TransientVariable to float. Actually this "cdms2.tvariable.TransientVariable" obtained from using from cdms2 import MV2 Thanking you -- DILEEPKUMAR. R J R F, IIT DELHI

1 1

Segfault using "fromstring" and reading variable length string
by Gökhan Sever 24 Apr '11

24 Apr '11

Hello, Given this piece of code (I can provide the meg file off-the list for those who wants to reproduce the error) import numpy as np f = open("a08A0122.341071.meg", "rb") dt = np.dtype([('line1', '|S80'), ('line2', np.object_), ('line3', '|S80'), ('line4', '|S80'), ('line5', '|S80'), ('line6', '|S2'), ('line7', np.int32, 2000), ('line8', '|S2'), ('line9', np.int32, 2000), ('line10', '|S2')]) k = np.fromstring(f.read(dt.itemsize), dt)[0] Accessing k causes a "Segmentation fault (core dumped)" and kills my python and IPython sessions immediately. I actually know that the culprit is "np.object_" in this case. The original was as ('line2', '|S81') however those meg files (mix of text and binary content) have a funny habit of switching from 80 characters to 81 (including "/r/n" chars). I was testing if I could create a variable length string dtype, which seems not possible. Little more info: that line2 has time stamps, one of which is in the form of 22:34:59.999. I have seen in the file that 22:34:59.999 was originally written as 22:34:59.1000 which causes that extra character flow. (Interestingly, millisecond should cycle from 0-999 and overflow at 999 instead of 1000 which to me indicates a slight bug) Because of this reason, I can't read the whole content of those meg files since somewhere in the middle fromstring attempts reading a shifted (erroneous) content. Should I go fix that millisecond overflow first or is there an alternative way to approach this problem? Thanks ================================================================================ Platform : Linux-2.6.35.12-88.fc14.x86_64-x86_64-with-fedora-14-Laughlin Python : ('CPython', 'tags/r27', '82500') IPython : 0.10 NumPy : 2.0.0.dev-2e96d91 ================================================================================ -- Gökhan

4 5

assert and allclose
by josef.pktd＠gmail.com 24 Apr '11

24 Apr '11

Some observations about allclose I have a test case where I would like to impose absolute (logical_)and relative errors (Pdb) pdf_r array([ 2.97341655e-90, 3.68756271e-24, 2.01840159e-07, 3.98942280e-01, 4.83941449e-01, 1.07981933e-01]) (Pdb) pdf_st array([ 3.09721604e-90, 3.69697466e-24, 2.01872148e-07, 3.98942181e-01, 4.83941207e-01, 1.07982122e-01]) (Pdb) pdf_st - pdf_r array([ 1.23799491e-91, 9.41194691e-27, 3.19891525e-11, -9.95821123e-08, -2.41764579e-07, 1.89015140e-07]) (Pdb) (pdf_st - pdf_r) / pdf_r array([ 4.16354347e-02, 2.55234897e-03, 1.58487551e-04, -2.49615338e-07, -4.99574028e-07, 1.75043301e-06]) allclose just uses the sum of abolute and relative errors, which I think is binding in the wrong way, for what I want. (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-4, atol=1e-8) True the relative and absolute errors are to tight on one side and too loose on the other side (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-2, atol=0) False (Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-8) False tighter bounds for both relative and absolute errors if I impose both separately (Pdb) np.allclose(pdf_st, pdf_r, rtol=0, atol=1e-6) True (Pdb) np.allclose(pdf_st, pdf_r, rtol=1e-1, atol=0) True Are there any alternatives to this? (or am I too tired) (test cases are for statistical distributions, where all values are smaller than one, and some values can be very small) Josef

2 1

Python crashes while printing a huge numpy array (Windows 1.6b2)
by Till Stensitzki 23 Apr '11

23 Apr '11

Hello, due a mistake i found the following bug: import numpy as np x=np.zeros((262144, 262144)) print x.shape print x The last line crashes the python interpreter without any feedback. Windows 7, Python 2.6.5, Numpy 1.6b2

7 14

status for 1.6.0
by Ralf Gommers 23 Apr '11

23 Apr '11

Hi, The list of open issues for 1.6.0 is down to a handful: - f2py segfault on Ubuntu reported by David (David, did you get any further with this?) - #1801: test_noncentral_f failure on win64 - #1798: type promotion regression (needs review, https://github.com/numpy/numpy/pull/73) - API docs for new parameters to ufuncs (full list: 'casting', 'out', 'order', 'subok', 'dtype'), - API docs, some docstrings and update of NEP for datetime I'm planning to release the first release candidate next weekend if these are solved. As far as I know no one is working on the doc issues right now. It would be great if someone could step up and do this. Thanks, Ralf

5 10