Mailman 3 September 2007 - SciPy-User

stats module accessibility
by Alan G Isaac Sept. 27, 2007

Sept. 27, 2007

Why not allow direct use of subpackages of scipy? That is, if I want to ``describe`` a series, why not allow:: scipy.stats.describe(myseries) instead of:: from scipy import stats stats.describe(myseries) or:: from scipy.stats import describe describe(myseries) I am sure this is a naive question about package structure, but I'd like to understand. Thank you, Alan Isaac

4 4

ANN: maskedarray
by Pierre GM Sept. 27, 2007

Sept. 27, 2007

All, The latest version of maskedarray has just been released on the scipy SVN sandbox. This version fixes the inconsistencies in filling (see below) and introduces some minor modifications for optimization purposes (see below as well). Many thanks to Eric Firing and Matt Knox for the fruitful discussions at the origin of this release! In addition, a bench.py file has been introduced, to compare the speed of numpy.ma and maskedarray. Once again, thanks to Eric for his first draft. Please feel free to try it and send me some feedback. Modifications: * Consistent filling ! In numpy.ma, the division of array A by array B works in several steps: - A is filled w/ 0 - B is filled w/ 1 - A/B is computed - the output mask is updated as the combination of A.mask, B.mask and the domain mask (B==0) The problems with this approach are that (i) it's not useful to fill A and B beforehand if the values will be masked anyway; (ii) nothing prevents infs to show up, as the domain is taken into account at the end only. In this latest version of maskedarray, the same division is decomposed as: - a copy of B._data is filled with 1 with the domain (B==0) - the division of A._data by this copy is computed - the output mask is updated as the combination of A.mask, B.mask and the domain mask (B==0). Prefilling on the domain avoids the presence of nans/infs. However, this comes with the price of making some functions and methods slower than their numpy.ma counterparts, as you'll be able to observe for sqrt and log with the bench.py file. An alternative would be to avoid filling at all, at the risk of leaving nans and infs. * masked_invalid / fix_invalid Two new functions are introduced. masked_invalid(x) masks x where x is nan or inf. fix_invalid(x) returns (a copy of) x, where invalid values (nans & infs) are replaced by fill_value. * No mask shrinking Following Paul Dubois and Sasha's example, I eventually had to get rid of the semi-automatic shrinking of the mask in __getitem__, which appeared to be a major bottleneck. In other words, one can end up with an array full of False instead of nomask, which may slow things down a bit. You can force a mask back to nomask with the new shrink_mask method. *_sharedmask Here again, I followed Paul and Sasha's ideas and reintroduce the _sharedmask flag to prevent inadequate propagation of the mask. When creating a new array with x=masked_array(data, mask=m), x._mask is initially a reference to m and x._sharedmask is True. When x is modified, x._mask is copied to prevent a propagation back to m.

1 0

array.tofile stops writing to file
by J. K. Sept. 27, 2007

Sept. 27, 2007

Hi, I am using SciPy 0.6.0 with NumPy 1:1.03-1ubuntu2 on Python 2.5.1.5ubuntu3. I am using arrays (about 120 rows with 7 columns) to store data and calculate stuff with the stored values. To be able to cross check the calculations, I wanted to store the arrays in files. I tried the different approaches: - Array.tofile(FileName, sep='\t', format = "%s") - Iteration over array k = 0 for row in Koeff: Filename.write(array[k, 0], array[k, 1], array[k, 3]) k = k + 1 and similar ways. Now the problem is, the write process just stops in the middle of a number. In the same loop I use to write the files I can print all those lines on screen, the arrays are complete. When using array.tofile, the same thing happens. I can print them on screen, but the file just ends in the middle of a value (always the same). On the other hand, I can write to file (not using arrays) without problems, the files can be as large as I want them to be. Is there something terribly wrong with the way I use it or is it a bug in numpy? If you need more detail, please tell me Thank you for your time and help, Jack Kinch

2 2

Ipython history and editing
by Jose Unpingco Sept. 26, 2007

Sept. 26, 2007

One of the greatest things about Ipython is the ability to use one's favorite editor to edit previous entires as in > %edit _i18 which would edit the 18th input in the history. The annoying thing is that the so-edited command is not placed on the history, instead, the unuseful 22: _ip.magic("edit _i18") is put there. I would like the output of the post-edited command to show in the %hist list. By the way, I know I can get the post-edited output using Out[22], but I would really like to see it on the %hist list. Is this possible? Please contact me if you have questions or need more information. Thanks! Jose Unpingco, Ph.D. (619)553-2922

2 1

Stack two matrices on top of each other?
by Jose Unpingco Sept. 26, 2007

Sept. 26, 2007

a=randn(2,3) bmat([[a],[a]]) # stack them side-by-side But, this doesn't work with the semicolon: bmat([[a];[a]]) # not work, although it would be natural Is there another way to stack two matrices on top of each other? I know I can do bmat('a ; a') but I have a list of matrices I would like to stack and doing it this way means that I would have to assign a variable to each one of matrices like a0=A[0] a1=A[1] .. aN=A[N] which is what I'm doing now, but would prefer not to. Please contact me if you have questions or need more information. Thanks! Jose Unpingco, Ph.D. (619)553-2922

3 2

Dickey-Fuller test
by Alan G Isaac Sept. 25, 2007

Sept. 25, 2007

Nobody responded to my appeal for SciPy based or Python based unit root tests, so in case anyone cares, I cleaned up this Dickey-Fuller test (tested against EViews output): http://econpy.googlecode.com/svn/trunk/pytrix/unitroot.py Cheers, Alan Isaac

1 0

NIPY
by Alan G Isaac Sept. 24, 2007

Sept. 24, 2007

What is the relationship between NIPY and SciPy? It seems a classic SciKit, but it is not in SciKits. Anyway, does the old NIPY models and algorithms modules still exist somewhere? Thank you, Alan Isaac

2 1

A first proposal for dataset organization
by David Cournapeau Sept. 24, 2007

Sept. 24, 2007

Hi there, A few months ago, we started to discuss about various issues about dataset for numpy/scipy. In the context of my Summer Of Code for machine learning tools in python, I had the possibility to tackle concretely the issue. Before announcing a first alpha version of my work, I would like to gather comments, critics about the following proposal for dataset organization. The following proposal is also available in svn: http://projects.scipy.org/scipy/scikits/browser/trunk/learn/scikits/learn/d… Dataset for scipy: design proposal ================================== One of the thing numpy/scipy is missing now is a set of datasets, available for demo, courses, etc. For example, R has a set of dataset available at the core. The expected usage of the datasets are the following: - machine learning: eg the data contain also class information (discrete or continuous) - descriptive statistics - others ? That is, a dataset is not only data, but also some meta-data. The goal of this proposal is to propose common practices for organizing the data, in a way which is both straightforward, and does not prevent specific usage of the data. Organization ------------ A preliminary set of datasets is available at the following address: http://projects.scipy.org/scipy/scikits/browser/trunk/learn/scikits/learn/d… Each dataset is a directory and defines a python package (e.g. has the __init__.py file). Each package is expected to define the function load, returning the corresponding data. For example, to access datasets data1, you should be able to do: >>> from datasets.data1 import load >>> d = load() # -> d contains the data. load can do whatever it wants: fetching data from a file (python script, csv file, etc...), from the internet, etc... Some special variables must be defined for each package, containing a python string: - COPYRIGHT: copyright informations - SOURCE: where the data are coming from - DESCHOSRT: short description - DESCLONG: long description - NOTE: some notes on the datasets. Format of the data ------------------ Here, I suggest a common practice for the returned value by the load function. Instead of using classes to provide meta-data, I propose to use a dictionnary of arrays, with some values mandatory. The key goals are: - for people who just want the data, there is no extra burden ("just give me the data !" MOTO). - for people who need more, they can easily extract what they need from the returned values. More high level abstractions can be built easily from this model. - all possible dataset should fit into this model. - In particular, I want to be able to be able to convert our dataset to Orange Dataset representation (or other machine learning tool), and vice-versa. For the datasets to be useful in the learn scikits, which is the project which initiated this datasets package, the data returned by load has to be a dict with the following conventions: - 'data': this value should be a record array containing the actual data. - 'label': this value should be a rank 1 array of integers, contains the label index for each sample, that is label[i] should be the label index of data[i]. If it contains float values, it is used for regression instead. - 'class': a record array such as class[i] is the class name. In other words, this makes the correspondance label name > label index. As an example, I use the famouse IRIS dataset: the dataset contains 3 classes of flowers, and for each flower, 4 measures (called attributes in machine learning vocabulary) are available (sepal width and length, petal width and length). In this case, the values returned by load would be: - 'data': a record array containing all the flowers' measurements. For descriptive statistics, that's all you may need. You can easily find the attributes from the dtype (a function to find the attributes is also available: it returns a list of the attributes). - 'labels': an array of integers (for class information) or float (for regression). each class is encoded as an integer, and labels[i] returns this integer for the sample i. - 'class': a record array, which returns the integer code for each class. For example, class['Iris-versicolor'] will return the integer used in label, and all samples i such as label[i] == class['Iris-versicolor'] are of the class 'Iris-versicolor'. This contains enough information to get all useful information through introspection and simple functions. I already implemented a small module to do basic things such as: - selecting only a subset of all samples. - selecting only a subset of the attributes (only sepal length and width, for example). - selecting only the samples of a given class. - small summary of the dataset. This is implemented in less than 100 lines, which tends to show that the above design is not too simplistic. Remaining problems: ------------------- I see mainly two big problems: - if the dataset is big and cannot fit into memory, what kind of API do we want to avoid loading all the data in memory ? Can we use memory mapped arrays ? - Missing data: I thought about subclassing both record arrays and masked arrays classes, but I don't know if this is feasable, or even makes sense. I have the feeling that some Data mining software use Nan (for example, weka seems to use float internally), but this prevents them from representing integer data. Current implementation ---------------------- An implementation following the above design is available in scikits.learn.datasets. If you installed scikits.learn, you can execute the file learn/utils/attrselect.py, which shows the information you can easily extract for now from this model. Also, once the above problems are solved, an arff converter will be available: arff is the format used by WEKA, and many datasets are available at this format: http://weka.sourceforge.net/wekadoc/index.php/en:ARFF_%283.5.4%29 http://www.cs.waikato.ac.nz/ml/weka/index_datasets.html Note ---- Although the datasets package emerged from the learn package, I try to keep it independant from everything else, that is once we agree on the remaining problems and where the package should go, it can easily be put elsewhere without too much trouble. cheers, David

7 11

JOB: Data Process Developers at Gemini Observatory
by James Turner Sept. 23, 2007

Sept. 23, 2007

Hi everyone, Gemini Observatory is advertising again for developers to work on Python-based data reduction software. If you're interested in applying, please see the instructions at the links below. It would be great to see applications from some developers here ;-). Apologies for the late notice (applications are due on 1 October). http://www.gemini.edu/jobs/#41 http://members.aas.org/JobReg/JobDetailPage.cfm?JobID=23796 Thanks! James.

1 0

New scipy 0.6.0 uploads
by Jarrod Millman Sept. 22, 2007

Sept. 22, 2007

There is a new scipy-0.6.0.tar.gz on the sourceforge page, which contains the missing scipy/linalg/src/fblaswrap_veclib_c.c. There is also now a scipy-0.6.0-py2.4-win32.egg and scipy-0.6.0-py2.5-win32.egg. Enjoy, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

1 0