Mailman 3 August 2013 - SciPy-User

Stick (line segments) percolation algorithm - graph theory?
by Josè Luis Mietta 23 Aug '13

23 Aug '13

Hi experts! I wrote an algorithm for study stick percolation (i.e.: networks between line segments that intersect between them). In my algorithm N sticks (line segments) are created inside a rectanglar box of sides 'b' and 'h' and then, one by one, the algorithm explores the intersection between all line segments. This is a Monte Carlo simulation, so the 'experiment' is executed many times (no less than 100 times). Writen like that, very much RAM is consumed: array_x1=uniform.rvs(loc=-b/2,scale=b,size=N) array_y1=uniform.rvs(loc=-h/2,scale=h,size=N) array_x2=uniform.rvs(loc=-b/2,scale=b,size=N) array_y2=uniform.rvs(loc=-h/2,scale=h,size=N) M =np.zeros([N,N]) foru inxrange(100): ---->This'100'isthe number of experiments. forj inxrange(N): ifj>0: x_A1B1 =array_x2[j]-array_x1[j] y_A1B1 =array_y2[j]-array_y1[j] x_A1A2 =array_x1[0:j]-array_x1[j] y_A1A2 =array_y1[0:j]-array_y1[j] x_A2A1 =-1*x_A1A2 y_A2A1 =-1*y_A1A2 x_A2B2 =array_x2[0:j]-array_x1[0:j] y_A2B2 =array_y2[0:j]-array_y1[0:j] x_A1B2 =array_x2[0:j]-array_x1[j] y_A1B2 =array_y2[0:j]-array_y1[j] x_A2B1 =array_x2[j]-array_x1[0:j] y_A2B1 =array_y2[j]-array_y1[0:j] p1 =x_A1B1*y_A1A2 -y_A1B1*x_A1A2 p2 =x_A1B1*y_A1B2 -y_A1B1*x_A1B2 p3 =x_A2B2*y_A2B1 -y_A2B2*x_A2B1 p4 =x_A2B2*y_A2A1 -y_A2B2*x_A2A1 condition_1=p1*p2 condition_2=p3*p4 fork inxrange (j): ifcondicion_1[k]<=0andcondicion_2[k]<=0: M[j,k]=1 ifj+1<N+4: x_A1B1 =array_x2[j]-array_x1[j] y_A1B1 =array_y2[j]-array_y1[j] x_A1A2 =array_x1[j+1:]-array_x1[j] y_A1A2 =array_y1[j+1:]-array_y1[j] x_A2A1 =-1*x_A1A2 y_A2A1 =-1*y_A1A2 x_A2B2 =array_x2[j+1:]-array_x1[j+1:] y_A2B2 =array_y2[j+1:]-array_y1[j+1:] x_A1B2 =array_x2[j+1:]-array_x1[j] y_A1B2 =array_y2[j+1:]-array_y1[j] x_A2B1 =array_x2[j]-array_x1[j+1:] y_A2B1 =array_y2[j]-array_y1[j+1:] p1 =x_A1B1*y_A1A2 -y_A1B1*x_A1A2 p2 =x_A1B1*y_A1B2 -y_A1B1*x_A1B2 p3 =x_A2B2*y_A2B1 -y_A2B2*x_A2B1 p4 =x_A2B2*y_A2A1 -y_A2B2*x_A2A1 condicion_1=p1*p2 condicion_2=p3*p4 fork inxrange (N-j-1): ifcondicion_1[k]<=0andcondicion_2[k]<=0: M[j,k+j+1]=1 Here, the element Mij=1 if stick i intersect stick j and Mij=0 if not. How can i optimize my algorithm? Graph theory is usefull in this case? Waiting for your answers. Thanks a lot! Best regards

1 0

ANN: Spyder v2.2.3 released
by Pierre Raybaut 21 Aug '13

21 Aug '13

Hi all, On the behalf of Spyder's development team ( http://code.google.com/p/spyderlib/people/list), I'm pleased to announce that Spyder v2.2.3 has been released and is available for Windows XP/Vista/7/8, GNU/Linux and MacOS X: http://code.google.com/p/spyderlib/. This is a maintenance release of the v2.2 branch which is the last release to support Python 2.5: * Spyder 2.2 supports Python 2.5 to 2.7 * Spyder 2.3 will support Python 2.7 to 3.3 * (Spyder 2.3.0dev6 is also available as of today: this is an experimental release but quite stable which already supports Python 3) See also https://code.google.com/p/spyderlib/downloads/list. Since v2.2.2: * Several bug have been fixed (see changelog for further details: https://code.google.com/p/spyderlib/wiki/ChangeLog) * New features have been added like full support for a MATLAB-like cell mode (see "Run" menu) and the Optional Dependencies dialog box (see menu "?") which gives the user a status of Spyder's dependencies Spyder is a free, open-source (MIT license) interactive development environment for the Python language with advanced editing, interactive testing, debugging and introspection features. Originally designed to provide MATLAB-like features (integrated help, interactive console, variable explorer with GUI-based editors for dictionaries, NumPy arrays, ...), it is strongly oriented towards scientific computing and software development. Thanks to the `spyderlib` library, Spyder also provides powerful ready-to-use widgets: embedded Python console (example: http://packages.python.org/guiqwt/_images/sift3.png), NumPy array editor (example: http://packages.python.org/guiqwt/_images/sift2.png), dictionary editor, source code editor, etc. Description of key features with tasty screenshots can be found at: http://code.google.com/p/spyderlib/wiki/Features Don't forget to follow Spyder updates/news: * on the project website: http://code.google.com/p/spyderlib/ * and on our official blog: http://spyder-ide.blogspot.com/ Last, but not least, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment! (http://code.google.com/p/spyderlib/wiki/NoteForContributors) Enjoy! -Pierre

1 0

possible bug in OpenCV python module (cv2)
by ha 21 Aug '13

21 Aug '13

When I run: img = uint8([[0,255,255,0],[0,255,255,0],[0,255,255,0],[0,255,255,0]]) aaa,bbb = cv2.findContours(img,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) img I get the following output: array([[ 0, 0, 0, 0], [ 0, 2, 130, 0], [ 0, 2, 130, 0], [ 0, 0, 0, 0]], dtype=uint8) This does not seem to me as the expected behavior, as the input image has changed. Should this be treated as a bug? What's your experience with cv2 module? Is this the right place to discuss such problems?

3 2

Re: [SciPy-User] Maximum file size for savemat?
by Michal Romaniuk 20 Aug '13

20 Aug '13

Hi, With some further work, I found that the file produced by SciPy isn't actually correct. Matlab can read it but at some point in the array the rest of the data is just zeros. I'm surprised that SciPy doesn't throw an error when writing data that is too big... Are there any good alternatives to .mat files? (Preferably something that Matlab could read too.) PyTables? The data consists of one large array (around 9GB), one list containing a few smaller arrays, and a few other arrays and scalars. Thanks, Michal > Hi, > >> Hi, >> >> On Mon, Aug 19, 2013 at 7:44 AM, Michal Romaniuk >> <michal.romaniuk06(a)imperial.ac.uk> wrote: >>> Hi, >>> >>> I'm saving a large batch of data using savemat and although I get no >>> errors, the files produced are not readable for either matlab or scipy. >>> Is there a limit on file size? >> >> Ah - yes there is - the individual matrices in the mat file cannot be >> larger than 4GB. Is it possible you hit this limit? >> >> Sorry, I only realized this when Richard Llewellyn pointed this out a >> couple of weeks ago on the list: >> >> http://scipy-user.10969.n7.nabble.com/SciPy-User-scipy-io-loadmat-throws-Ty… >> >> The current scipy code has an error message for matrices that are too large. >> >> Cheers, >> >> Matthew > > Well, I managed to work around the problem to some extent by setting > do_compression=True. Now Matlab can read those files (so they must be > valid to some extent) but SciPy can't (even though they were written > with SciPy). > > I get this error: > > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio.pyc in > loadmat(file_name, mdict, appendmat, **kwargs) > 173 variable_names = kwargs.pop('variable_names', None) > 174 MR = mat_reader_factory(file_name, appendmat, **kwargs) > --> 175 matfile_dict = MR.get_variables(variable_names) > 176 if mdict is not None: > 177 mdict.update(matfile_dict) > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio5.pyc in > get_variables(self, variable_names) > 290 continue > 291 try: > --> 292 res = self.read_var_array(hdr, process) > 293 except MatReadError, err: > 294 warnings.warn( > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio5.pyc in > read_var_array(self, header, process) > 253 `process`. > 254 ''' > --> 255 return self._matrix_reader.array_from_header(header, > process) > 256 > 257 def get_variables(self, variable_names=None): > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in > scipy.io.matlab.mio5_utils.VarReader5.array_from_header > (scipy/io/matlab/mio5_utils.c:5401)() > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in > scipy.io.matlab.mio5_utils.VarReader5.array_from_header > (scipy/io/matlab/mio5_utils.c:4849)() > > PATH/lib/python2.6/site-packages/scipy/io/matlab/mio5_utils.so in > scipy.io.matlab.mio5_utils.VarReader5.read_real_complex > (scipy/io/matlab/mio5_utils.c:5602)() > > ValueError: total size of new array must be unchanged > > > > The size of the main array is about 9 GB before compression, but the > compressed files are less than 500 MB and closer to 400 MB. There are > some other arrays in the file too but they are much smaller. > > Any ideas on how I could get SciPy to read this data back? Right now I > can only think of storing the data in single precision format... > > Thanks, > Michal

3 2

At.: question about refresh numpy array in a for-cycle
by Josè Luis Mietta 19 Aug '13

19 Aug '13

Hi experts! Im writing a code with a numpy array L, the numpy matrix M and the next script: for x in L: for l in srange(N): z= l in L if z is False and M[x,l] != 0: L=np.append(L,l) here, in the end of the cycle, new elements are incorporated to the array 'L'. I want these new elements be considered as 'x' index in the cycle. When I execute the script I see that only the 'originals' elements of L are considered as 'x'. How can i fix it? Waiting for your answers. Thanks a lot!

2 2

Maximum file size for savemat?
by Michal Romaniuk 19 Aug '13

19 Aug '13

Hi, I'm saving a large batch of data using savemat and although I get no errors, the files produced are not readable for either matlab or scipy. Is there a limit on file size? Thanks, Michal

2 1

sparse.linalg.splu can I use option Fact=SamePattern
by John Floyd 17 Aug '13

17 Aug '13

I am using scipy in a time variable flow model. The matrix pattern does not change over time, so the matrix reordering does not need to be recalculated each timestep. SuperLU supports this with the Fact=SamePattern option - can it be done using the python interface - if so please give an example. Thanks John John Floyd Water, Wetlands and Coasts Science Office of Environment and Heritage Department of Premier and Cabinet john.floyd(a)environment.nsw.gov.au <mailto:e@environment.nsw.gov.au> Ph: 02-9995-5656 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- This email is intended for the addressee(s) named and may contain confidential and/or privileged information. If you are not the intended recipient, please notify the sender and then delete it immediately. Any views expressed in this email are those of the individual sender except where the sender expressly and with authority states them to be the views of the Office of Environment and Heritage, NSW Department of Premier and Cabinet. PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL

2 1

constrained least square fitting using scipy.optimize.fmin_slsqp() function
by sudipta sinha 17 Aug '13

17 Aug '13

Hi All, I am facing a problem for constrained linear least square fitting. In my case the matrix equation looks like [Y]nX1=[X]nXm[P]mX1, where Y and P are vectors and X is a matrix and n, m are dimension of the matrix. Further, there is a equality constraint on P which is Sum(P(i))=0.0. How do I proceed to solve that? Which function of python is suitable for this? I saw few of discussion on scipy.optimize.fmin_slsqp() function but the implementation of this function is not very straightforward. Therefore, I need your help. I am new in SCIPY. Please help me out in this regard.

3 2

[ANN] statsmodels version 0.5.0 released
by Skipper Seabold 14 Aug '13

14 Aug '13

Hi all, I'm happy to announce on behalf of the statsmodels development team the release of 0.5.0. It's been a little over a year in the making. The release includes over 2000 commits from 38 authors. All users are encouraged to upgrade to this version. Highlights ======== * Support for fitting models via R-like formulas * ANOVA modeling * Multivariate kernel density estimators * Nonparametric regression * Empirical likelihood estimators * Quantile regression * Negative binomial maximum likelihood estimator * l1-penalized discrete choice models * New and improved graphics * Power and sample size calculations * And much more Full Release Notes: http://statsmodels.sourceforge.net/stable/release/version0.5.html Issues Closed: http://statsmodels.sourceforge.net/stable/release/github-stats-0.5.html#iss… Installers: https://pypi.python.org/pypi/statsmodels Thanks to ======== git log v0.4.3..v0.5.0 --format='* %aN ' | sort -u * Ana Martinez Pardo * anov * avishaylivne * Bruno Rodrigues * Carl Vogel * Chad Fulton * Christian Prinoth * Daniel B. Smith * dengemann * Dieter Vandenbussche * Dougal Sutherland * Enrico Giampieri * evelynmitchell * George Panterov * Grayson * Jan Schulz * Jeff Reback * Josef Perktold * Justin Grana * langmore * Matthew Brett * Nathaniel J. Smith * otterb * padarn * Paul Hobson * Pietro Battiston * Ralf Gommers * Richard T. Guy * Robert Cimrman * Skipper Seabold * Thomas Haslwanter * timmie * Tom Augspurger * Trent Hauck * tylerhartley * Vincent Arel-Bundock * VirgileFritsch * Zhenya Let me know if you want to update your Mailmap name. What is it ======== Statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available for different types of data and each estimator. Researchers across fields may find that statsmodels fully meets their needs for statistical computing and data analysis in Python. Dependencies =========== Required dependencies: python >= 2.6 (including Python 3.x) numpy >= 1.5.1 scipy >= 0.7 pandas >= 0.7.1 patsy >= 0.1.0 Build dependencies: cython >= 0.15.1 (If building from github repo) C compiler Optional dependencies: matplotlib >= 1.0.0 : needed for plotting sphinx >= 1.0.0 : needed to build the docs ipython >= 1.0 : needed to build the notebook examples nose >= 1.0.0 : needed to run the tests Links ====== Documentation: http://statsmodels.sourceforge.net/ Mailing List: https://groups.google.com/forum/#!forum/pystatsmodels PyPi: https://pypi.python.org/pypi/statsmodels Github: https://github.com/statsmodels/statsmodels/ Bug Tracker: https://github.com/statsmodels/statsmodels/issues Cheers, Skipper

1 0

Read file with comma decimal separator
by Florian Lindner 14 Aug '13

14 Aug '13

Hello, I have a file that used comma as a decimal separator. How can I read a file like that using loadtxt or genfromtxt ? Thanks, Florian

5 5