Mailman 3 March 2010 - SciPy-User

where to put plots in advice.mechanicalkern ?
by denis 17 Mar '10

17 Mar '10

Where should plots and other pngs refd from notes in advice.mechanicalkern.com be put ? Thanks, cheers -- denis

6 13

usemask kwd in timeseries.tsfromtxt missing?
by Skipper Seabold 16 Mar '10

16 Mar '10

The docs say that usemask is a kwd arg, but it's not implemented. Is this an oversight or are the docs wrong and the TimeSeries object must be a masked array? Using recent trunk. In [1]: import scikits.timeseries as ts In [2]: ts.__version__ Out[2]: '0.91.3' Skipper

3 6

Re: [SciPy-User] Maximum file size for .npz format?
by David Warde-Farley 16 Mar '10

16 Mar '10

Hey Francesc, On 2010-03-15, at 4:16 PM, Francesc Alted <faltet(a)pytables.org> wrote: > Good! So, I should start by looking at how it achieves that then :) Actually I did build a 4-way universal binary of HDF5. Long story short, because of some quirks of the build process for HDF5 it isn't possible to add some -arch flags to CFLAGS -- you actually have to build it separately for each architecture and then do something like make install DESTDIR=path/for/arch and then stitch them together manually with otool. I'm also not sure of the correct way, if there is one, to handle "hdf5.settings" -- it's unclear to me whether programs ever look at this at build time/ runtime, or if it's just there for the user's convenience. An installer for the binaries I made is at http://www.cs.toronto.edu/~dwf/mirror/hdf5-1.8.4-quad.pkg I can write up some instructions if that would be helpful. David

4 4

Potential KDTree memory leak
by Pierre Lamon 16 Mar '10

16 Mar '10

Hi all, the memory increases at each iteration of the following program: from numpy import ndarray from scipy.spatial import KDTree class C(ndarray): pass pts = [C((2,)) for i in range(1000)] for i in xrange(10000): tree = KDTree(pts) It does not occur when using the base class ndarray instead. I've only tested on a windows machine. Any idea what happens? Of course this is a dummy program tailored to emphasis the problem. But as I'm using KDTree intensively in my application (both creation and query) and I need to fix this memory leak. Any help is greatly appreciated, ~Pierre

3 4

3D Interpolation For Uneven Data?
by Joseph Smidt 16 Mar '10

16 Mar '10

I have 3D data *not* equally-spaced. Is there a routine in scipy that will interpolate this data? It seems like the only higher dimensional routine I can find requires even;y spaced data? http://www.scipy.org/Cookbook/Interpolation. Thanks. Joseph Smidt -- ------------------------------------------------------------------------ Joseph Smidt <josephsmidt(a)gmail.com> Physics and Astronomy 4129 Frederick Reines Hall Irvine, CA 92697-4575 Office: 949-824-3269

3 2

Correlation coefficient of large arrays
by Vincent Davis 16 Mar '10

16 Mar '10

I have an array 10 observations of 230,000 variables and what to find the correlation coefficient between each variable. numpy.corrcef(data) works except I can only do it with about 30,000 variables at a time. numpy.corrcef(data[:30000]). It uses up a lot of memory. Is there a better way? *Vincent Davis 720-301-3003 * vincent(a)vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis>

2 6

Re: [SciPy-User] [R] Correlation coefficient of large data sets
by Vincent Davis 16 Mar '10

16 Mar '10

> > @ Dennis With 35000 variables at a time, the storage is under 20 Gb; you'd have to compute about 50 such chunks to get the entire matrix. Is there a way to calculate a column or row of the correlation matrix one at a time? I ma looking how including an additional set of observation effect the correlation. For example if I have variables a,b,c,d..... and set of observations 1-10 if the correlation is calculated for obs 1-5, I then add observations 6-10 and what to know the average effect of this on the correlation of c with (a,b,,d,e.....). So I only need a column or a row at a time. Just not clear to me how I would do this. @Joshua Wiley cor(my.data) # calculate the correlation matrix between all variables > (columns) of my.data > *Vincent Davis 720-301-3003 * vincent(a)vincentdavis.net my blog <http://vincentdavis.net> | LinkedIn<http://www.linkedin.com/in/vincentdavis> On Tue, Mar 16, 2010 at 12:06 AM, Joshua Wiley <jwiley.psych(a)gmail.com>wrote: > I think what you have done should be fine. read.table() will return a > data frame, which cor() can handle happily. For example: > > my.data <- read.table("file.csv", header = TRUE, row.names = 1, > sep=",", strip.white = TRUE) # assign your data to "my.data" > > cor(my.data) # calculate the correlation matrix between all variables > (columns) of my.data > > What happens if you try that? >

1 0

[ANN] EuroPython 2010 registration and talk submissions now open!
by Ciarán Mooney 16 Mar '10

16 Mar '10

EuroPython 2010 - 17th to 24th July 2010 ---------------------------------------- EuroPython is a conference for the Python programming language community, including the Django, Zope and Plone communities. It is aimed at everyone in the Python community, of all skill levels, both users and programmers. Last year's conference was the largest open source conference in the UK and one of the largest community organised software conferences in Europe. This year EuroPython will be held from the 17th to 24th July in Birmingham, UK. It will include over 100 talks, tutorials, sprints and social events. Registration ------------ Registration is open now at: http://www.europython.eu/registration/ For the best registration rates, book as soon as you can! Extra Early Bird closes soon, after which normal Early Bird rate will apply until 10th May Talks, Activities and Events ---------------------------- Do you have something you wish to present at EuroPython? You want to give a talk, run a tutorial or sprint? Go to http://www.europython.eu/talks/cfp/ for information and advice! Go to http://wiki.europython.eu/Sprints to plan a sprint! Help Us Out ----------- EuroPython is run by volunteers, like you! We could use a hand, and any contribution is welcome. Go to http://wiki.europython.eu/Helping to join us! Go to http://www.europython.eu/contact/ to contact us directly! Sponsors -------- Sponsoring EuroPython is a unique opportunity to affiliate with this prestigious conference and to reach a large number of Python users from computing professionals to academics, from entrepreneurs to motivated and well-educated job seekers. http://www.europython.eu/sponsors/ Spread the Word --------------- We are a community-run not-for-profit conference. Please help to spread the word by distributing this announcement to colleagues, project mailing lists, friends, your blog, Web site, and through your social networking connections. Take a look at our publicity resources: http://wiki.europython.eu/Publicity General Information ------------------- For more information about the conference, please visit the official site: http://www.europython.eu/ Looking forward to see you! The EuroPython Team

1 0

Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes
by David Baddeley 15 Mar '10

15 Mar '10

Here's the code, hope it's useful. cheers, David --- On Tue, 16/3/10, Charles Vejnar <ce(a)vejnar.eu> wrote: > From: Charles Vejnar <ce(a)vejnar.eu> > Subject: Re: [SciPy-User] Shared memory using multiprocessing.sharedctypes > To: "SciPy Users List" <scipy-user(a)scipy.org> > Cc: "David Baddeley" <david_baddeley(a)yahoo.com.au> > Received: Tuesday, 16 March, 2010, 4:31 AM > Hi David, > > I am interested. Can you please show us your code ? > > BW, > Charles > > On Monday 15 March 2010, David Baddeley wrote: > > Thanks! > > > > Will have to give it a spin. Unfortunately (or > fortunately - as it was > > quite instructive from a learning perspective) I > got bored and already > > knocked up my own version. It delegates all the > memory management to > > multiprocessing so weighs in pretty light at > only 80 lines of pure python > > & works well on 64 bit systems. Just in case > anyone wants yet another > > shared memory array implementation. > > > > cheers, > > David > > > > > > ----- Original Message ---- > > From: Christopher Lee-Messer <chris.leemesser(a)gmail.com> > > To: scipy-user(a)scipy.org > > Sent: Sun, 14 March, 2010 5:45:08 AM > > Subject: [SciPy-User] Shared memory using > multiprocessing.sharedctypes > > > > David Baddeley wrote:: > > > I'd like shared memory numpy arrays for some code > I'm trying to > > > parallelise. The solution should ideally be cross > platform. I remember > > > some earlier discussion about this and, as far as > I recall Sturla had a > > > working implementation which was (unfortunately) > linux only (please > > > correct me if I'm wrong). > > > > > > Looking a bit further, > multiprocessing.sharedctypes seems to have all the > > > stuff needed to handle the low level bits on at > least linux and windows, > > > and it seems as though it should be relatively > easy to cobble something > > > together on top of this which let you use shared > memory numpy arrays > > > relatively transparently. This would have the > advantage that all the > > > platform specific nasty stuff is being handled by > the multiprocessing > > > module, which as it's now part of the standard > library, should hopefully > > > be well maintained & has a good chance of > being ported to additional > > > platforms > > > > > > Before I jump in and try writing such a wrapper - > essentially a helper > > > function for creating shared numpy arrays (easy) > and some pickle handlers > > > to make sure what comes out the other end looks > like an array (would need > > > to think/read a bit more, but should be doable) I > wanted to make sure > > > that there's not something out there already > & that I'm not going to be > > > reinventing the wheel. > > > > [Note, I posted this via google groups originally > before realizing > > that it wasn't the official list. Sorry if you get > multiple versions. > > -chris] > > > > Hi David, > > I was just in the same boat. I checked out > Sturla's a few weeks ago > > and tried it out on windows xp and now linux > (32bit, python 2.6 with > > numpy 1.3 (Pythonxy 2.6 with mingw) and python2.5 with > numpy 1.2.1) > > > > It's working very well for me in soft-realtime data > acquisition > > application with sizes of 1 to 800KB. > > > > In case it saves you time, I recently threw together > some packaging > > for the code and put it on bitbucket as a > numpy-sharedmem repository > > in case it might save me or someone else some time. > > > > http://bitbucket.org/cleemesser/numpy-sharedmem/ > > > > I don't know if Sturla, Gael Varoquaux, or Robert Kern > are continuing > > to work on these, but I plan to add testing and save > results on > > different platforms as my app gets used in different > computers in the > > lab. Nadav Horesh mentioned that there are some > limitations on the > > size of the arrays for "large" arrays and Sturla > mentioned that there > > may be issue on 64bit systems. > > -Chris > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > > > > > > > > > _______________________________________________ > > SciPy-User mailing list > > SciPy-User(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-user > > >

2 1

Maximum file size for .npz format?
by Jose Gomez-Dans 15 Mar '10

15 Mar '10

Hi! I need to save a fairly large set of arrays to disk. I have saved it using numpy.savez, and the resulting file is around 11Gb (yes, I did say fairly large ;D). When I try to load it using numpy.load, the zipfile module compains about BadZipfile: Bad magic number for file header I can't open it with the normal zip utility present on the system, but it could be that it's barfing about files being larger than 2Gb. Is there some file limit for npzs? Is there anyway I can recover the data (I guess I could try decompressing the file with 7z and extracting the individual npy files?) Thanks! Jose

17 31