Mailman 3 October 2014 - SciPy-User

Graphics of Scipy-Stack Core Packages and dependencies
by Sebastian Oct. 18, 2014

Oct. 18, 2014

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, For a presentation - giving an overview of the scientific stack - I'm looking for a graphics that shows the core packages (numpy, scipy, pandas, matplotlib, and possibly cython, pytables etc.) and how they are related. So: Numpy as basic package at the top and then pandas and scipy, matplotlib below of scipy. Does anyone know if such a graphics does already exist? If not, I will create one and appreciate your help for the details. I also think of adding details of third-pary requirements (the LAPACK, ATLAS etc.). best, Sebastian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) iQIcBAEBAgAGBQJUQMp1AAoJEBn0X+vcm0Y7HekQAIL9O2rNL1CVozEI4gWx+4CH PQTPGfM5YcDuYVSNng8GSBmcDAsHLs59Be6AS7beCpd8JiW7rQJtOzv4D4T9Ao9l hIwtEblF9lC3C86+D9BPJW9+9yDTocTeaoy+/Uubs9oskA+/D5xiRIgBgFjwoebX GLxlZ4ASygatPAA03SSwIfiB2Ch44PSd4vhQgLDAR2u7S7V1wPbXUllI1fVInJi1 rOs719IN/64+cB5GeCLJPhZZclSOawDIULI3YIfBLTSEiqjmr9vLYRlQu+5Iy8BB DjEAqF/ocXwKMak8fsl9Xpuz52ErPw3VWGj+984V6mYUAa/jQMDIReWrRdLHJZd4 idDP/78QNkcDID3kr1DoP2NqB0Y5kssed4zynsKbQ5IRKBMgazWKBRsort3HOeSZ bL32CCSKEa+gTUJw0tKskYvnSPRIuWwZ1vwkLbDpwyZUU/BUap6GIlK+WLpmi+Jt 61k+yxIO/y612yRZ3ruIee0ZMzv5NpdkvPQEeQ/rC36zHbOHDVU36OD9UrwL9jPz ErI85QRSSQ/EKbT17m2tN8afQ7WVG0H/9J6/EooFvucM3p3oorhtXjVVEZD73iwY oWur/rLOXR2+Ckm7NFc28zO9hkq4MorX1NMFyK/GIS3rABBNd4GXtElIfHqOiBUn EO2rJ3C+YWOZVlnd1qRb =JEZY -----END PGP SIGNATURE-----

1 1

BFGS precision vs least_sq
by Andrew Nelson Oct. 17, 2014

Oct. 17, 2014

Dear scipy users, I am using BFGS as a final 'polishing' minimizer after using another minimization technique. I thought I would check how it performs against a known non-linear regression standard, http://www.itl.nist.gov/div898/strd/nls/data/thurber.shtml. I discovered that BFGS doesn't get to the absolute known minimum for this least squares problem, even if the starting point is almost the best solution. Rather, it finishes with a chi2 of 5648.6 against the lowest chi2 minimum of 5642.7 (using scipy.optimize.leastsq). The problem seems to be 'precision loss', as reported by fmin_bfgs. I have tried adjusting the gtol and epsilon options for fmin_bfgs, but did not do any better. Ok, so the two chi2 are only out by 0.1%, but I would like to know why BFGS can't get to the same minimum as leastsq. The following code illustrates the issue: <python> import numpy as np from scipy.optimize import fmin_bfgs, leastsq a = np.asarray([80.574, 84.248, 87.264, 87.195, 89.076, 89.608, 89.868, 90.101, 92.405, 95.854, 100.696, 101.06, 401.672, 390.724, 567.534, 635.316, 733.054, 759.087, 894.206, 990.785, 1090.109, 1080.914, 1122.643, 1178.351, 1260.531, 1273.514, 1288.339, 1327.543, 1353.863, 1414.509, 1425.208, 1421.384, 1442.962, 1464.350, 1468.705, 1447.894, 1457.628]) b = np.asarray([-3.067, -2.981, -2.921, -2.912, -2.840, -2.797, -2.702, -2.699, -2.633, -2.481, -2.363, -2.322, -1.501, -1.460, -1.274, -1.212, -1.100, -1.046, -0.915, -0.714, -0.566, -0.545, -0.400, -0.309, -0.109, -0.103, 0.010, 0.119, 0.377, 0.790, 0.963, 1.006, 1.115, 1.572, 1.841, 2.047, 2.200]) def chi2(x, *args): vec = x[0] + x[1] * b + x[2] * b ** 2 + x[3] * b ** 3 vec /= 1 + x[4] * b + x[5] * b ** 2 + x[6] * b ** 3 return sum((vec - a)**2) startX = np.array([1.28861830e+03, 1.49582373e+03, 5.86557938e+02, 7.60498878e+01, 9.69658594e-01, 3.99904927e-01, 5.03732685e-02]) #epsilon = np.sqrt(np.finfo(np.float64).eps) * np.max(np.abs(startX), 0.1) x_BFGS = fmin_bfgs(chi2, startX) def res(x, *args): vec = x[0] + x[1] * b + x[2] * b ** 2 + x[3] * b ** 3 vec /= 1 + x[4] * b + x[5] * b ** 2 + x[6] * b ** 3 return (vec - a) x_leastsq = leastsq(res, startX, full_output=True)[0] print "Chi2 BFGS:", chi2(x_BFGS), "Chi2 leastsq", chi2(x_leastsq) </python> I'm using the master branch of scipy with these tests. regards, Andrew. -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 3

odeint - Excess work done on this call
by Farid Afandiyev Oct. 16, 2014

Oct. 16, 2014

Hello! I'm newbie both in calculus and python/scipy so I apologize if this question is too dumb. I'm trying to model flow between two pressure vessels. Let's say we have two points and a link between them like this [Vc_1, P_1]=====A====[Vc_2, P_2] Vc_1, Vc_2 are constants volumes of the nodes and P_1, P_2 varying pressure of the nodes respectively. I end up writing differential questions below. Never mind physical meaning I just want to get math correct. dP_1/dt = dVa/dt * 1/(Vc_1 * B) dP_2/dt = dVa/dt * 1/(Vc_2 * B) Here B is compressibility. dVa/dt = A * K * sqrt(P_1 - P_2) dVa is amount of "flow" between nodes. K is some constant coefficient and A is link "weight". (P_1-P_2) can change sign so I've adjusted this in the software. Below is python program that I wrote to evaluate this. #!/usr/bin/env python import math from scipy.integrate import odeint from time import time import numpy B_compressibility = 0.0000033 #water compressibility K = 0.747871759938 #coefficient Vc_1 = 20 Vc_2 = 50 A = 0.01 P_1 = 4000 P_2 = 2000 def deriv(state, t): _P_1 = state[0] _P_2 = state[2] diff_P = _P_1 - _P_2 flow_direction = math.copysign(1, diff_P) dVa = flow_direction * A * K * math.sqrt(abs(diff_P)) dP_1 = -(dVa/Vc_1)/B_compressibility dP_2 = (dVa/Vc_2)/B_compressibility return [dP_1, -dVa, dP_2, dVa] if __name__ == '__main__': Va_1 = Vc_1 * P_1 * B_compressibility Va_2 = Vc_2 * P_2 * B_compressibility odeIterations = 10 timeperiod = numpy.linspace(0.0,1.0, odeIterations) initial_state = [P_1, Va_1, P_2, Va_2] t0 = time() state_array = odeint(deriv, initial_state, timeperiod) t1 = time() print 'runtime %fs' %(t1-t0) state = state_array[odeIterations-1] print state_array P_1 = state[0] Va_1 = state[1] P_2 = state[2] Va_2 = state[3] Below is output from program Excess work done on this call (perhaps wrong Dfun type). Run with full_output = 1 to get quantitative information. runtime 0.157000s [[ 4.00000000e+03 2.64000000e-01 2.00000000e+03 3.30000000e-01] [ 3.49242034e+03 2.30499743e-01 2.20303186e+03 3.63500257e-01] [ 3.09580400e+03 2.04323064e-01 2.36167840e+03 3.89676936e-01] [ 2.81015098e+03 1.85469965e-01 2.47593961e+03 4.08530035e-01] [ 2.63546127e+03 1.73940444e-01 2.54581549e+03 4.20059556e-01] [ 2.57173487e+03 1.69734501e-01 2.57130605e+03 4.24265499e-01] [ 2.57142857e+03 1.69714286e-01 2.57142857e+03 4.24285714e-01] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00] [ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]] lsoda-- at current t (=r1), mxstep (=i1) steps taken on this call before reaching tout In above message, I1 = 500 In above message, R1 = 0.6110315150411E+00 Odeint gives correct results up to 7th line and then something goes seriously wrong. I have searched in google and it looks like I'm not the only one who struggles with scipy. Everybody suggests increasing mxstep but that doesn't solve my problem. In addition it slows down the method significantly. Somebody suggested reducing accuracy but I don't know how to do that. Also I don't need super accuracy from the odeint. Couple of digits after dot is more than enough for me. Any help is greatly appreciated! Kind regards, Farid.

3 5

-KPIC flag compilation error
by Evan Brown Oct. 15, 2014

Oct. 15, 2014

Hi there I am trying to compile SciPy from source on Solaris 11. I have downloaded the latest version of the SciPi source code - scipy-0.14.0. I am running gcc 4.8.2 and gfortran and Python 2.6. When I run: python setup.py build in the SciPy source directory, the process runs for a while before exiting with the following error: --------------------- building 'rootfind' library compiling C sources C compiler: /usr/lib/python2.6/pycc -DNDEBUG -KPIC creating build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros compile options: '-I/usr/lib/python2.6/vendor-packages/numpy/core/include -c' pycc: scipy/optimize/Zeros/brenth.c gcc: error: unrecognized command line option ‘-KPIC’ gcc: error: unrecognized command line option ‘-KPIC’ error: Command "/usr/lib/python2.6/pycc -DNDEBUG -KPIC -I/usr/lib/python2.6/vendor-packages/numpy/core/include -c scipy/optimize/Zeros/brenth.c -o build/temp.solaris-2.11-i86pc-2.6/scipy/optimize/Zeros/brenth.o" failed with exit status 1 ------------------------ My question is: how do I remove the apparently illegal -KPIC flag which is being passed to gcc? I can't find the location in the source or setup.py file where this flag is passed to the compiler. Any help would be much appreciated. Best wishes Evan

2 1

Re: [SciPy-User] odeint - Excess work done on this cal
by Alex Liberzon Oct. 13, 2014

Oct. 13, 2014

Apparently, the problem is with the time step, see the linked IPython notebook http://nbviewer.ipython.org/gist/alexlib/5220b92868c800fd1700

2 1

Hilbert transform
by Tanguanzhong Oct. 13, 2014

Oct. 13, 2014

Dear all, I have made a test for Hilbert transform for PCM data. Suggest "pcmRawData" is PCM audio data and collect by pyaudio. After Hilbert transform, "pcmData" has 90 degree phase shift relative to "pcmRawData". Because phase shift for PCM audio is not sensitive to Human-being. So I guess I should not find the difference for "pcmData" and "pcmRawData". But on the contrary, the test result is the audio data of "pcmData" is not good to hear. There are some distortions during the scipy Hilbert transform. Could anyone can give some advice for this? Is any problem for apply Hilbert transform for audio by Scipy? This is some code example. from scipy import fftpack ... pcmData = fftpack.hilbert(pcmRawData)

2 2

Numerical Recipes and scipy
by Peter Parente Oct. 12, 2014

Oct. 12, 2014

Hi all, I have a question about SciPy and its relation to the code from Numerical Recipes (http://www.nr.com/licenses/redistribute.html) I can see that both the SciPy FAQ and code comments today both contain references to the book. I noticed that in 2008 there was a discussion about removing SciPy code derived from the book to avoid infringing on the NR license. ( http://mail.scipy.org/pipermail/scipy-dev/2008-December/010446.html) I see that some code was identified and possibly replaced, but there is no distinct end to the activity. Does anyone have an understanding they can share about where SciPy stands today with respect to NR and its license terms? Thanks, Pete (I'm sorry if this is a dupe. I had a problem posting from my subscribed email address.)

6 9

scipy.ndimage.filters.median_filter with NAN values
by Shahar Barak Oct. 11, 2014

Oct. 11, 2014

I have a 3D array, in which some values are Nan (masked). I am using scipy.ndimage.filters.median_filter to smooth the data. Is there a way to have scipy.ndimage.filters.median_filter not include the NAN values in its median calculation? Thanks, Shahar

3 2

ANN: Pandas 0.15.0 Release Candiate 1
by Jeff Reback Oct. 7, 2014

Oct. 7, 2014

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.15.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues> We will be releasing officially in 1-2 weeks or so. This is a major release from 0.14.1 and includes a number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. Highlights include: - Drop support for numpy < 1.7.0 - The Categorical type was integrated as a first-class pandas type - New scalar type Timedelta, and a new index type TimedeltaIndex - New DataFrame default display for df.info() to include memory usage - New datetimelike properties accessor .dt for Series - Split indexing documentation into Indexing and Selecting Data and MultiIndex / Advanced Indexing - Split out string methods documentation into Working with Text Data - read_csv will now by default ignore blank lines when parsing - API change in using Indexes in set operations - Internal refactoring of the Index class to no longer sub-class ndarray - dropping support for PyTables less than version 3.0.0, and numexpr less than version 2.1 Here are the full whatsnew and documentation links: v0.15.0 Whatsnew <http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew.html> v0.15.0 Documentation Page <http://pandas.pydata.org/pandas-docs/version/0.15.0/> Source tarballs, and windows builds are available here: Pandas v0.15.0rc1 Release <https://github.com/pydata/pandas/releases> A big thank you to everyone who contributed to this release! Jeff

1 0

progress in scipy, special and stats
by josef.pktd＠gmail.com Oct. 6, 2014

Oct. 6, 2014

Just a general thanks. Small things that make the life of a developer easier and improve numerical precision in corner cases. I just went over some statsmodels code that was written maybe 5 years ago with clumsy or missing 0*log(0) = 0 handling. scipy.special has been and is getting nice little (*) functions that handle some corner cases automatically and numerically in a more stable way. scipy.stats is one of the beneficiaries. Thanks to several maintainer/developers and contributors. ---- (*) little compared to the heavy stuff in scipy.special, but also useful (apropos heavy: 5 papers mention that there is no closed form solution for the pmf of a discrete distribution but we can calculate it using a recursion. The 6th article mentions it can be calculated with the modified Bessel function of the second kind. Why didn't they say so right away. I'd rather use scipy.special than figuring out how to do a numerically stable recursion.) Josef Watching from the sidelines. BTW: PIG or P-IG is not a good name for a discrete distribution if we care about google searchability.

1 0