Mailman 3 January 2012 - SciPy-User

Re: [SciPy-User] FortranFile and large records
by Vincent Favre-Nicolin Jan. 20, 2012

Jan. 20, 2012

Le 20/01/2012 11:25, Sturla Molden a écrit : > Why? The FortranFile cookbook receipe is based on fundamentally wrong > assumptions. If it works it is by pure accident. I suggest it is removed > from the wiki. Even if it is not a complete solution, at least it allows many people to read their files. If there is another solution to import more fortran record file into numpy, then fine ! But until then, this code remains a very good help to anyone who does not want to bother writing a fortran program just to read fortran data (even if it does not work with data from /every/ f compiler). Vincent

1 0

FortranFile and large records
by Vincent Favre-Nicolin Jan. 20, 2012

Jan. 20, 2012

Hi, This week I had to import large data from a Fortran 'record' file. For this I naturally used FortranFile (http://www.scipy.org/Cookbook/FortranIO/FortranFile). Unfortunately, it turned out that when a record is larger than 2,147,483,639 bytes (a bit short of 2^31), the 'record' where the data is stored is split into sub-records. This took me some time to figure out, since I don't use Fortran that much, and the only symptom was a negative size for the record... Which I originally interpreted as a type (signed/unsigned) or byteswapping issue.... Anyway, once I got what was going wrong (from Intel's documentation), I modified the "readRecord" function in FortranFile to be able to read a complete record when such a case is encountered. The corresponding code is below (not very efficient memorywise but working as long as you have >=24Gb of memory, to perform directly a readReals()). I'm not sure if there is another way to do this (in numpy), hopefully it will be useful for someone else. Neil if you think it is OK, maybe the wiki page can be updated. And thanks again for FortranFile which is really useful ! Vincent def readRecord(self): """Read a single fortran record""" l = self._read_check() data_str='' while l<0: # Handle sub-records, for lengths>2,147,483,639 bytes # cf: Intel Fortran Compiler User and Reference Guides, page 217-218 # (doc number 304970-006US) data_str+=self._read_exactly(abs(l)) check_size = self._read_check() if abs(check_size) != abs(l): raise IOError('Error reading record from data file') l = self._read_check() data_str += self._read_exactly(abs(l)) check_size = self._read_check() if abs(check_size) != abs(l): raise IOError('Error reading record from data file') return data_str -- Vincent Favre-Nicolin http://inac.cea.fr CEA/Grenoble Institut Nanosciences & Cryogénie Laboratoire SP2M/Nano-structures et Rayonnement Synchrotron 17, rue des Martyrs 38054 Grenoble Cedex 9 - France Université Joseph Fourier http://www.ujf-grenoble.fr tél: (+33) 4 38 78 95 40 fax: (+33) 4 38 78 51 38

3 2

pandas 0.7.0 release candidate
by Wes McKinney Jan. 20, 2012

Jan. 20, 2012

I just tagged and uploaded a release candidate for pandas 0.7.0. This is a huge release including a ton of new and improved functionality as well as a host of bug fixes. It also includes a handful of API changes that veteran pandas users will want to pay close attention to. Barring major issues I'll make the final release around this time next week. Please beat on it and report any issues on GitHub: Repository: http://github.com/wesm/pandas Issue tracker: http://github.com/wesm/pandas/issues What's new in 0.7.0: http://pandas.sourceforge.net/whatsnew.html Windows Installers and Sources: http://pypi.python.org/pypi/pandas Still some lingering documentation items that will get finished over the next week. The official pandas repository will be migrating to the newly formed PyData GitHub organization shortly after the 0.7.0 release. best, Wes

2 1

a reference: signal splines
by josef.pktd＠gmail.com Jan. 18, 2012

Jan. 18, 2012

I'm still looking at smoothing splines. This reference looks good for equal spaced data (which I'm not really interested in). Includes matlab code in paper. (as aside: The spline code in scipy.signal has a single test, as far as I have seen.) http://www.sciencedirect.com/science/article/pii/S0167947308005045 A fast compact algorithm for cubic spline smoothing Computational Statistics & Data Analysis, Volume 53, Issue 4, 15 February 2009, Pages 932–940 ''' Abstract An efficient algorithm is presented for computing discrete or continuous cubic smoothing splines with uniformly spaced and uniformly weighted measurements. The algorithm computes both the spline values and the generalized cross-validation score. Execution time and memory use are reduced by carefully exploiting the problem’s rich structure. The frequency domain properties of the steady-state cubic spline smoother are also examined. Table 2. Execution time and memory use. n, r Our algorithm MATLAB spline toolbox 105, 2 36 ms/4.8 MB 1.00 s/19.2 MB 106, 2 367 ms/48 MB 12.2 s/192 MB 105, 10 117 ms/11.2 MB 1.51 s/25.6 MB 106, 10 1.19 s/112 MB 17.8 s/256 MB ''' Josef

1 0

Creating smaller .mat files
by Jaidev Deshpande Jan. 18, 2012

Jan. 18, 2012

Hi I use the savemat function in the scipy.io module quite often. Suppose you create a .mat file through Python. Most often this file is unnecessarily large. Even more interesting - when you open that file into a MATLAB workspace >> load myfile.mat And then save it back, >> save myfile.mat the size reduces dramatically! (Even though the data types and precision are kept the same) Why does this happen? If this means we can do something to reduce array sizes in the Python workspace (since MATLAB can recreate the same data from a much smaller size-on-disk), it would be a great advantage when dealing with large data. Thanks

3 2

OT: wikipedia
by josef.pktd＠gmail.com Jan. 18, 2012

Jan. 18, 2012

maybe north america only, I'm not able to get a quick summary of Savitzky–Golay_smoothing http://en.wikipedia.org/wiki/Savitzky%E2%80%93Golay_smoothing_filter Josef

4 4

Information about the Numerical Stability of Scipy/Numpy
by Marcel Caraciolo Jan. 18, 2012

Jan. 18, 2012

Hi all, I am studying scipy and numpy and I decide to write some routines with those packages for a paper submission in a scientific congress. The problem is the validation of the results and experiments I have to show that the libraries that I used (in this case Scipy and Numpy) provides numerical stability, otherwise the chances that my article be approved will be decreased. Any further information in docs or the website about this topic ? Regards, -- Marcel Pinheiro Caraciolo M.S.C. Candidate at CIN/UFPE http://www.mobideia.com http://aimotion.blogspot.com/

3 2

what happened to scipy.integrate.Inf?
by Mark Bakker Jan. 17, 2012

Jan. 17, 2012

Hello List, After I updated to version 0.10, scipy.integrate.Inf disappeared (but it is still in the docs of, for example, quad). Can I simply use numpy.inf? Should the docs be updated? Thanks, Mark

2 1

[PhD position] Modeling the sustainability of the value chain of wood in Switzerland using Python/Scipy (German & English required)
by Christopher Mutel Jan. 17, 2012

Jan. 17, 2012

Dear all- Please find attached an announcement for a PhD position modeling the sustainability of the wood value chain in Switzerland at ETH Zurich. The PhD will build upon an existing model for life cycle assessment that uses NumPy and SciPy extensively. This position is still open. Note that some knowledge of German and English is required for this position. Yours, -Chris -- ############################ Chris Mutel Ökologisches Systemdesign - Ecological Systems Design Institut f.Umweltingenieurwissenschaften - Institute for Environmental Engineering ETH Zürich - HIF C 44 - Schafmattstr. 6 8093 Zürich Telefon: +41 44 633 71 45 - Fax: +41 44 633 10 61 ############################

1 0

convolution density kernel
by Johannes Radinger Jan. 17, 2012

Jan. 17, 2012

Hello, I'd like to apply a 'convolution' on a probability density kernel. Like in: http://graphics.stanford.edu/courses/cs178/applets/convolution.html where f=g (for step 1) In my case the function (base and that for the convolution) is: def pdf(x,sigma_stat,sigma_mob): return (p * stats.norm.pdf(x, loc=m, scale=sigma_stat) + (1-p) * stats.norm.pdf(x, loc=m, scale=sigma_mob)) this are actually two superimposed normal distributions. Due to computational problems I think it easier to sample finely as I just want to represent the result using matplotlib. Furthermore I'd like to do this convolution computation several times, where the output of the last convolution is the input (f) of the next step (function (g) stays the same original one). Is there any easy applicable function in scipy to to this convolution? As I am not that familiar with scipy hopefully someone can help me in this case. Best regards, Johannes -- NEU: FreePhone - 0ct/min Handyspartarif mit Geld-zurück-Garantie! Jetzt informieren: http://www.gmx.net/de/go/freephone

2 1