Mailman 3 May 2009 - SciPy-User

mathworks fileexchange with BSD license
by josef.pktd＠gmail.com 25 May '09

25 May '09

I just saw that mathworks fileexchange includes now a license statement and most of the recent code, I looked at, has a BSD license attached. This makes adapting code for the use with scipy much easier. Last time when I asked an author of a script about the license, I only got a "free for non-commercial use", now it is under BSD. Josef

1 0

eigenvector values (negative where it should be positive)
by Carlos "Guâno" Grohmann 25 May '09

25 May '09

Hello all I'm working on some structural geology data, using numpy (I'm following some class notes, so I can check my results). I have a set of directional data (azimuth/dip): 12 42 18 40 22 48 15 30 10 42 20 30 First I read the data and create a matrix with the direction cosines like this: #direction cosines relative to axis oriented north, east and down # phi = longitude = azimuth (dip direction) # theta = latitude = dip # xi = cos(theta[i])*cos(phi[i]) # yi = cos(theta[i])*sin(phi[i]) # zi = sin(theta[i]) # Tmat = orientation matrix T # Tmat = sum(xi2) sum(xi.yi) sum(xi.zi) # sum(yi.xi) sum(yi2) sum(yi.zi) # sum(zi.xi) sum(zi.yi) sum(zi2) This is the matrix: [[ 3.34172131 0.96327612 2.73061427] [ 0.96327612 0.29736701 0.78834422] [ 2.73061427 0.78834422 2.36091168]] So far so good, but according to my example, the eigenvectors should look like: Vector 1 Vector 2 Vector 3 X 0.749 -0.590 -0.300 Y 0.217 -0.210 0.953 Z 0.626 0.779 0.029 and I have this: [[-0.74913585 -0.59037777 0.30041565] [-0.21679731 -0.21002264 -0.95335692] [-0.62593482 0.77932315 -0.02934318]] So, the values are OK, but the negative signs I don't understand. any ideas are welcome TIA Carlos -- Carlos Henrique Grohmann - Geologist D.Sc. a.k.a. Guano - Linux User #89721 ResearcherID: A-9030-2008 carlos dot grohmann at gmail dot com http://www.igc.usp.br/pessoais/guano/ _________________ "Good morning, doctors. I have taken the liberty of removing Windows 95 from my hard drive." --The winning entry in a "What were HAL's first words" contest judged by 2001: A SPACE ODYSSEY creator Arthur C. Clarke Can’t stop the signal.

2 1

scipy.interpolate spline class names
by Erik Tollerud 23 May '09

23 May '09

I use the splines in scipy.interpolate quite a bit, and I particularly like the *UnivariateSpline and *BivariateSpline wrapper classes. However, I cannot for the life of me work out what gives with the names and documentation... As far as I can tell, the univariate splines are as follows: UnivariateSpline : A spline where the number of knots is chosen using the "smoothing factor" s LSQUnivariateSpline: A spline where the knots are explicitly specified InterpolatedUnivariateSpline: A spline with s=0 or t=[] (e.g. passes through all the fitting points) The documentation just says the second two "just have less error checking"... aren't they for very different purposes? And while I recognize that name changes at this stage might be uncalled for, the names are somewhat misleading, too... shouldn't they be "SmoothUnivariateSpline","KnotUnivariateSpline", and "InterpolatedUnivariateSpline" or something like that? It also seems there are similar versions for the *BivariateSpline classes, although it's unclear to me exactly what the raw BivariateSpline class does as compared to the SmoothBivariateSpline (and the RectBivariateSpline, at least, makes sense)

5 9

using UnivariateSpline
by David Warde-Farley 23 May '09

23 May '09

I must be crazy, but how does one actually USE UnivariateSpline, etc. to do interpolation? How do I evaluate the spline at other data after it's fit? There seems to be no "evaluate" method or equivalent to splev. Thanks, David

5 6

improving docs for classes was: Re: scipy.interpolate spline class names
by josef.pktd＠gmail.com 23 May '09

23 May '09

I'm changing the thread title. On Fri, May 22, 2009 at 9:03 PM, <josef.pktd(a)gmail.com> wrote: > On Fri, May 22, 2009 at 8:24 PM, Pauli Virtanen <pav(a)iki.fi> wrote: >> Fri, 22 May 2009 19:22:14 -0400, josef.pktd wrote: >> [clip] >>> There is also still the open question how we get the information of the >>> docstrings in class.__init__ into the sphinx docs. >> >> The Numpy docstring standard dictated that the __init__ method should be >> documented in the main class docstring. >> >> I don't personally like this very much. Maybe we need to revise this? >> >> Anyway, the Sphinx dev version contains an improved version of >> autosummary that has features that could be used to address this. >> >> *** >> >> So I'd suggest currently just making a separate hand-written page for the >> interpolation class docs, making appropriate use of the autoclass:: and >> automethod:: directives. >> >> The main documentation page interpolate.rst could then contain the >> corresponding autosummary directives without the :toctree: argument. >> >> -- >> Pauli Virtanen > > Pauli, > do you have an example how to do this? when I tried autoclass and > automethod in the doc editor then it didn't produce the intended > results. > > For example for the KroghInterpolator: > Given last years discussion a lot of information was put into the __init__ > > What I would find very helpful would be if the link for > KroghInterpolator in > http://docs.scipy.org/scipy/docs/scipy-docs/interpolate.rst/ leads to > the full autodocs of the class, with all or selected automethods. I > would prefer one page per class for the class based modules such as > the interpolator classes. > > I find the docs very well structured and accessible for functions but > in many cases it doesn't provide a good structure for classes. > > If you have an example for how this can be done, then I could fix > parts of the docs. The numpy docstring doesn't really define the > structure for the sphinx documentation, or does it. > > I still appreciate the htmhelp files for windows a lot. It's very > useful to have instantaneous search and access to the docs. That's why > it's bugging me when the information is not in the docs, even though > it can be accessed with >>> help(classname). But currently the docs > don't produce the same result. > some more on the current status of the documentation of classes: The __call__ method also seems to be a second class citizen, not only in interpolate. I assume it is not automatically included in the autodoc. I just checked scipy.stats.rv_continuous and stats.kde.gaussian_kde. rv_continuous mentions the __call__ method in the class docstring, but there is no further reference, stats.kde.gaussian_kde is silent about it's __call__ methods Also the class docstrings don't link to or list the methods. For example rv_continuous lists all the important methods but there are no links. Additional to the classes in scipy, I checked numpy.DataSource and numpy.random.RandomState, and the help page for them is pretty uninformative, I don't know and cannot link to any of their methods. compare this for example with help(numpy.DataSource) I hope that by the end of this years gsoc, we will be adding many stats.model classes. So a good documentation pattern for classes would be very helpful for us. And if we allow for different array subclasses in the __init__ method, then we need the facility to document them. A similar example is scipy.signal.lti which allows 3 different constructors, see the doc page at http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lti.html#s… I don't know whether this just needs some additional rst files, changes in the automatic document creation or in the docstring standard. Of course there is the way of writing full rst docs as in http://docs.scipy.org/numpy/source/numpy/doc/source/reference/arrays.ndarra…, but that looks like a lot of additional work, which might not happen very soon. So I think that for small to medium sized classes we should find a way to create automatic class documentation including special methods __init__ and __call__ and maybe others, if these methods have a special meaning. __init__ is useful if we want to describe the constructor in more detail instead of copying all information into the class docs or to separate rst files. I'm still stuck getting members or methods added to an autoclass directive: http://docs.scipy.org/scipy/docs/scipy-docs/interpolate_UnivariateSpline.rs… Josef

1 0

TimeSeries concatenate
by Robert Ferrell 23 May '09

23 May '09

I'm having trouble with concatenate in scikits.timeseries. I can't find (timeseries) concatenate documented, but I did find an example from a few years ago (http://projects.scipy.org/scipy/changeset/3570), so maybe this isn't supported. Here's what I get when I try to use the timeseries concatenate. In [82]: import scikits.timeseries as ts In [83]: ts1 = ts.time_series(data=[1,2,3], start_date=ts.now('d')) In [84]: ts2 = ts.time_series(data=[4,5,6], start_date=ts.now('d')+1) In [85]: ts.concatenate(ts1, ts2) --------------------------------------------------------------------------- <type 'exceptions.AttributeError'> Traceback (most recent call last) /Users/Shared/Develop/Financial/LakMerc/Mercury/tests/<ipython console> in <module>() /Library/Frameworks/Python.framework/Versions/2.5.2001/lib/python2.5/ site-packages/scikits/timeseries/tseries.py in concatenate(series, axis, remove_duplicates, fill_missing) 1950 """ 1951 # Get the common frequency, raise an error if incompatibility -> 1952 common_f = _compare_frequencies(*series) 1953 # Concatenate the order of series 1954 sidx = np.concatenate([np.repeat(i,len(s)) /Library/Frameworks/Python.framework/Versions/2.5.2001/lib/python2.5/ site-packages/scikits/timeseries/tseries.py in _compare_frequencies(*series) 284 frequencies. 285 """ --> 286 unique_freqs = np.unique([x.freqstr for x in series]) 287 try: 288 common_freq = unique_freqs.item() <type 'exceptions.AttributeError'>: 'numpy.int32' object has no attribute 'freqstr'

2 3

Re: [SciPy-user] Fitting an arbitrary distribution
by David Baddeley 22 May '09

22 May '09

Thanks for the prompt replies! I guess what I was meaning was that the PDF / histogram was the sum or multiple Gaussians/normal distibutions. Sorry about the ambiguity. I've had a quick look at the Em package and mixture models, and while my problem is similar they might be a little more general. I guess I should describe the problem in a bit more detail - I'm measuring the length of an objects which can be built up from multiple unit cells. The measured size distribution is thus multimodal, and I want to extract both the unit size and the fraction of objects having each number of unit cells. This makes the problem much more constrained than what is dealt with in the Em package. So far I've tried overriding rv_continuous to create a distribution which roughly matches - but haven't been able to fit this. cheers, David ----- Original Message ---- From: "scipy-user-request(a)scipy.org" <scipy-user-request(a)scipy.org> To: scipy-user(a)scipy.org Sent: Friday, 22 May, 2009 2:44:37 PM Subject: SciPy-user Digest, Vol 69, Issue 43 Send SciPy-user mailing list submissions to scipy-user(a)scipy.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.scipy.org/mailman/listinfo/scipy-user or, via email, send a message with subject or body 'help' to scipy-user-request(a)scipy.org You can reach the person managing the list at scipy-user-owner(a)scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-user digest..." Today's Topics: 1. Fitting an arbitrary distribution (David Baddeley) 2. Re: Fitting an arbitrary distribution (David Cournapeau) 3. Re: Inconsistent function calls? (Ivo Maljevic) 4. Re: Fitting an arbitrary distribution (josef.pktd(a)gmail.com) 5. Re: Fitting an arbitrary distribution (josef.pktd(a)gmail.com) 6. Re: Fitting an arbitrary distribution (David Cournapeau) 7. Re: Fitting an arbitrary distribution (josef.pktd(a)gmail.com) 8. Re: Fitting an arbitrary distribution (josef.pktd(a)gmail.com) ---------------------------------------------------------------------- Message: 1 Date: Thu, 21 May 2009 18:47:00 -0700 (PDT) From: David Baddeley <david_baddeley(a)yahoo.com.au> Subject: [SciPy-user] Fitting an arbitrary distribution To: scipy-user(a)scipy.org Message-ID: <36002.66689.qm(a)web33005.mail.mud.yahoo.com> Content-Type: text/plain; charset=utf-8 Hi all, I want to fit an arbitrary distribution (in this case the sum of multiple Gaussians) to some measured data and was wondering if anyone could give me any pointers as to the best way of doing this. I'd like to avoid fitting to a histogram if possible. How do the .fit() methods of the various distributions under scipy.stats do it? My first thought would be to compare the cumulative distribution of my data with that of the model distibution using something like the kolmogorov-smirnov metric (maximum absolute distance between the curves) and to minimize this using optimize.fmin. Is this the right way to do it? Or is there an easier way? thanks in advance, David ------------------------------ Message: 2 Date: Fri, 22 May 2009 10:58:06 +0900 From: David Cournapeau <david(a)ar.media.kyoto-u.ac.jp> Subject: Re: [SciPy-user] Fitting an arbitrary distribution To: David Baddeley <david_baddeley(a)yahoo.com.au>, SciPy Users List <scipy-user(a)scipy.org> Message-ID: <4A1606AE.1030008(a)ar.media.kyoto-u.ac.jp> Content-Type: text/plain; charset=ISO-8859-1 David Baddeley wrote: > Hi all, > > I want to fit an arbitrary distribution (in this case the sum of multiple Gaussians) to some measured data and was wondering if anyone could give me any pointers as to the best way of doing this. I'd like to avoid fitting to a histogram if possible. How do the .fit() methods of the various distributions under scipy.stats do it? My first thought would be to compare the cumulative distribution of my data with that of the model distibution using something like the kolmogorov-smirnov metric (maximum absolute distance between the curves) and to minimize this using optimize.fmin. Is this the right way to do it? Or is there an easier way? That's a complex topic in general, there is no best answer, it depends on your case, and what you intend to do with the estimated distribution. In the case of a sum of mutiple Gaussians, the more commonly used name for this model is mixture models, and there is a vast range of possible techniques for fitting a dataset to this model. There is a package in scikits.learn to use the so-called Expectation Maximization algorithm to estimate the maximum likelihood of such models http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/em/ You can have an overview on the wiki page: http://en.wikipedia.org/wiki/Mixture_model cheers, David ------------------------------ Message: 3 Date: Thu, 21 May 2009 22:17:42 -0400 From: Ivo Maljevic <ivo.maljevic(a)gmail.com> Subject: Re: [SciPy-user] Inconsistent function calls? To: SciPy Users List <scipy-user(a)scipy.org> Message-ID: <826c64da0905211917u15ec1567g72547e6cff117535(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Sorry Christopher, I thought since they are used for the same purpose, and have similar syntax (http://www.scipy.org/NumPy_for_Matlab_Users says ``MATLAB? and NumPy/SciPy have a lot in common``), that SciPy looks more like Matlab than any other programing language (excluding Octave and other Matlab clones). As for everything else you wrote, I already said that I don`t have any problem with using SciPy the way it is. Ivo 2009/5/21 Christopher Barker <Chris.Barker(a)noaa.gov> > Ivo Maljevic wrote: > > why bother to make something that looks like matlab, > > who ever said numpy "looks like matlab", any more than it look s like > any number of other programming environments... > > > Matplotlib does a pretty good job at replicating > > matlab plot functions, at least at the level I need it to. > > Because is was designed exactly to do that -- but I think MPL's Matlab > replicating has been a hindrance, rather than a help, to a good API. > However, is has been a help to its adoption. > > You may have noticed that over the years MPL is moving away from matlab, > toward a more pythonic API. > > Personally, I like python so much more than Matlab exactly for these > differences (and so many more). I suppose it's tough if you switch back > and forth, but I haven't touched Matlab in years. > > It is rand() that is inconsistent, and that is an accident of history. > > > what ones([3,3]) does, the same way random.rand(3,3) does, > > well, rand() is a convenience function, and doesn't take a bunch of > other parameters. In fact, it's listed under "Compatibility functions", > and is really a wrapper for: > > numpy.random.uniform, which takes a shape argument. > > > the reason why I included that error message in my previous message > > is because I think it is completely non-helpful. > > That's another issue -- non-helpful error messages do show up a lot -- > in that case, if the user had typed: > > np.zeros(3, dtype=3) > > the error message would make sense. If you can suggest a better message, > patches are always welcome. > > -Chris > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker(a)noaa.gov > _______________________________________________ > SciPy-user mailing list > SciPy-user(a)scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-user >

3 3

Fitting an arbitrary distribution
by David Baddeley 22 May '09

22 May '09

Hi all, I want to fit an arbitrary distribution (in this case the sum of multiple Gaussians) to some measured data and was wondering if anyone could give me any pointers as to the best way of doing this. I'd like to avoid fitting to a histogram if possible. How do the .fit() methods of the various distributions under scipy.stats do it? My first thought would be to compare the cumulative distribution of my data with that of the model distibution using something like the kolmogorov-smirnov metric (maximum absolute distance between the curves) and to minimize this using optimize.fmin. Is this the right way to do it? Or is there an easier way? thanks in advance, David

4 7

Inconsistent function calls?
by Ivo Maljevic 22 May '09

22 May '09

Just as I was making up an example for the block diagonal matrix question, I remembered the old problem I had with consistency of nympy functions. If you want to generate a random number matrix, you can make the same call as with matlab: rand(2,2) for 2x2 matrix, randn(1,5) for 1x5 etc. but if you want to generate ones or zeros matrices, you cannot say ones(3,3), you have to write ones([3,3]) or zeros([3,3]) (note the extra brackets). It is not a big deal, but it seems a bit inconsistent for me.

10 24

fmin using spherical bounds
by ElMickerino 21 May '09

21 May '09

Hello Fellow SciPythonistas, I have a seemingly simple task: minimize a function inside a (hyper)sphere in parameter space. Unfortunately, I can't seem to make fmin_cobyla do what I'd like it to do, and after reading some of the old messages posted to this forum, it seems that fmin_cobyla will actually wander outside of the allowed regions of parameter space as long as it smells a minimum there (with some appropriate hand-waving). The function I'd like to minimize is only defined in this hypersphere (well, hyperellipsoid, but I do some linear algebra), so ideally I'd use something like fmin_bounds to strictly limit where the search can occur, but it seems that fmin_bounds can only handle rectangular bounds. fmin_cobyla seems to be happy to simply ignore the constraints I give it (and yes, I've got print statements that make it clear that it is wandering far, far outside of the allowed region of parameter space). Is there a simple way to use fmin_bounds with a bound of the form: x^2 + y^2 + z^2 + .... <= 1.0 ? or more generally: transpose(x).M.x <= 1.0 where x is a column vector and M is a positive definite matrix? It seems very bizarre that fmin_cobyla is perfectly happy to wander very, very far outside of where it should be. Thanks very much, Michael -- View this message in context: http://www.nabble.com/fmin-using-spherical-bounds-tp23654947p23654947.html Sent from the Scipy-User mailing list archive at Nabble.com.

4 6