Mailman 3 August 2009 - SciPy-User

[ANN] scikits.statsmodels 0.1.0b1 release
by Skipper Seabold 01 Sep '09

01 Sep '09

Hello all, As many of you know, Josef and I have been working hard to get a release ready for general consumption for the models code (now statsmodels). Well, we're happy to announce that a (very) beta release is ready. Please, read on. Background ========== The statsmodels code was started by Jonathan Taylor and was formerly included as part of scipy. It was taken up to be tested, corrected, and extended as part of the Google Summer of Code 2009. What it is ========== We are now releasing the efforts of the last few months under the scikits namespace as scikits.statsmodels. Statsmodels is a pure python package that requires numpy and scipy. It offers a convenient interface for fitting parameterized statistical models with growing support for displaying univariate and multivariate summary statistics, regression summaries, and (postestimation) statistical tests. Main Feautures ============== * regression: Generalized least squares (including weighted least squares and least squares with autoregressive errors), ordinary least squares. * glm: Generalized linear models with support for all of the one-parameter exponential family distributions. * rlm: Robust linear models with support for several M-estimators. * datasets: Datasets to be distributed and used for examples and in testing. There is also a sandbox which contains code for generalized additive models (untested), mixed effects models, cox proportional hazards model (both are untested and still dependent on the nipy formula framework), generating descriptive statistics, and printing table output to ascii, latex, and html. None of this code is considered "production ready". Where to get it =============== Development branches will be on LaunchPad. This is where to go to get the most up to date code in the trunk branch. Experimental code will also be hosted here in different branches. <https://code.launchpad.net/statsmodels> Source download of stable tags will be on SourceForge. <https://sourceforge.net/projects/statsmodels/> or PyPi: <http://pypi.python.org/pypi/scikits.statsmodels/> License ======= Simplified BSD Documentation ============= The official documentation is hosted on SourceForge. <http://statsmodels.sourceforge.net/> The sphinx docs are currently undergoing a lot of work. They are not yet comprehensive, but should get you started. There is some more information on the project blog, which will be updated as we make progress on the code. <http://scipystats.blogspot.com> Discussion and Development ========================== All chatter will take place on the or scipy-user mailing list. We are very interested in receiving feedback about usability, suggestions for improvements, and bug reports via the mailing list or the bug tracker at <https://bugs.launchpad.net/statsmodels>. Cheers, The Current statsmodels maintainers Josef Perktold and Skipper Seabold

1 0

Re: [SciPy-User] bug in ndimage grey_dilation?
by Tony S Yu 31 Aug '09

31 Aug '09

On Aug 30, 2009, at 1:00 PM, scipy-user-request(a)scipy.org wrote: > Message: 3 > Date: Sun, 30 Aug 2009 12:14:27 -0400 > From: Zachary Pincus <zachary.pincus(a)yale.edu> > Subject: Re: [SciPy-User] bug in ndimage grey_dilation? > To: SciPy Users List <scipy-user(a)scipy.org> > Message-ID: <E5ED36AA-3097-4144-8821-C6AEDAF99BBC(a)yale.edu> > Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes > >>> Any other morphological filters that have this inconsistency? >> >> This seems to be the only filter affected with this inconsistency b/ >> c all other filters delegate their handling of size/footprint/ >> structure arguments to filters._min_or_max_filter (which handles the >> scalar sizes without problems). > > Presumably there's some reason that grey_opening doesn't delegate this > as do the rest, but this is a bit of a surprise. Maybe it could? > > Zach It turns out that all the code before the call to filters._min_or_max_filter handles a shift of the origin parameter when the size/footprint/structure has even numbered sides (e.g. a 2x2 footprint has no real "center" to place the origin). For some reason, the grey_erosion function works just fine without any special argument handling, but grey_dilation does need it (I should know, I tried removing all of it). Here's a test script to test grey_dilation and grey_erosion against binary_dilation and binary_erosion, respectively: Without any changes to scipy 0.7.1, the only failures are for dilation using scalar size arguments or structure arguments (dilation with a structure will pass if you also give it a size tuple). Note, the erosion tests all pass, as well. After the fix below, all tests pass. Note, however, that I only test the degenerate case, i.e. when the structure is binary. Also, in the patch, if both a structure and footprint are specified, then size will equal the shape of the structure---maybe this should be reversed. Finally, I'm only testing 2D arrays, so this may need more testing... Let me know what you think, -Tony Index: morphology.py =================================================================== --- morphology.py (revision 5911) +++ morphology.py (working copy) @@ -339,21 +339,16 @@ value when mode is equal to 'constant'. """ if structure is not None: - structure = numpy.asarray(structure) - structure = structure[tuple([slice(None, None, -1)] * - structure.ndim)] - if footprint is not None: - footprint = numpy.asarray(footprint) - footprint = footprint[tuple([slice(None, None, -1)] * - footprint.ndim)] - input = numpy.asarray(input) + size = numpy.asarray(structure).shape + elif footprint is not None: + size = numpy.asarray(footprint).shape + elif size is None: + raise RuntimeError("size, footprint, or structure must be specified") + sizes = _ni_support._normalize_sequence(size, input.ndim) origin = _ni_support._normalize_sequence(origin, input.ndim) for ii in range(len(origin)): origin[ii] = -origin[ii] - if footprint is not None: - sz = footprint.shape[ii] - else: - sz = size[ii] + sz = sizes[ii] if not sz & 1: origin[ii] -= 1 return filters._min_or_max_filter(input, size, footprint, structure,

1 0

Problem with scipy.special.chebyt
by Kristian Hans Sandberg 31 Aug '09

31 Aug '09

When I type import scipy.special scipy.special.chebyt(12)(-0.5) in IPython, IPython crashes. (If I put this snippet in a file and run it outside IPython, Python also aborts, so it's not a problem with IPython.) Can anyone else reproduce this? I'm using Python 2.6.1 and scipy 0.7.0 on Mandriva 2009.1, 64-bit. Thanks! Kristian Kristian Sandberg, Ph.D. Dept. of Applied Mathematics and The Boulder Laboratory for 3-D Electron Microscopy of Cells University of Colorado at Boulder Campus Box 526 Boulder, CO 80309-0526, USA Phone: (303) 492 0593 (work) (303) 499 4404 (home) (303) 547 6290 (cell) Home page: http://amath.colorado.edu/faculty/sandberg

8 27

Re: [SciPy-User] bug in ndimage grey_dilation?
by Tony S Yu 30 Aug '09

30 Aug '09

On Aug 29, 2009, at 9:48 PM, scipy-user-request(a)scipy.org wrote: > Message: 1 > Date: Sat, 29 Aug 2009 15:30:59 -0400 > From: Zachary Pincus <zachary.pincus(a)yale.edu> > Subject: Re: [SciPy-User] bug in ndimage grey_dilation? > To: SciPy Users List <scipy-user(a)scipy.org> > Message-ID: <C36F4C5B-58CA-40A7-87C4-4485490140D6(a)yale.edu> > Content-Type: text/plain; charset=US-ASCII; format=flowed > > That looks good and reasonable! Oops! I really didn't test my fix. This patch causes a failure if footprint is specified without specifying a size (which should be allowed). Also, the function seems to fail (both before and after the patch) if the structure is specified without specifying a size (which the docstring suggests is allowed). I'll work on another patch, but it may require me to understand the code a bit better (I don't even understand the difference between footprint and structure at this point). (Also, ``if not sz & 1``---at the bottom of the diff---seems really weird to me). > > Any other morphological filters that have this inconsistency? This seems to be the only filter affected with this inconsistency b/c all other filters delegate their handling of size/footprint/structure arguments to filters._min_or_max_filter (which handles the scalar sizes without problems). -Tony > > Zach > > > On Aug 29, 2009, at 1:00 PM, Tony S Yu wrote: > >> ndimage.grey_dilation doesn't handle scalar size arguments (in >> contrast to other filters). For example: >> >>>>> ndimage.grey_dilation(np.ones((5,5)), size=2) >> TypeError: 'int' object is unsubscriptable >> >> Below is a simple patch which seems to fix the issue (I basically >> copied the behavior of ndimage.filters._min_or_max_filter). The diff >> below is taken against the 0.7.x branch. >> >> I guess this may not technically be a bug since filters in >> morphology.py don't specify they can handle scalar size arguments >> (filters in filters.py do document this behavior). In any case, this >> fix would make the API more consistent. >> >> Cheers, >> -Tony >> >> =================================================================== >> --- morphology.py (revision 5911) >> +++ morphology.py (working copy) >> @@ -347,13 +347,14 @@ >> footprint = footprint[tuple([slice(None, None, -1)] * >> footprint.ndim)] >> input = numpy.asarray(input) >> + sizes = _ni_support._normalize_sequence(size, input.ndim) >> origin = _ni_support._normalize_sequence(origin, input.ndim) >> for ii in range(len(origin)): >> origin[ii] = -origin[ii] >> if footprint is not None: >> sz = footprint.shape[ii] >> else: >> - sz = size[ii] >> + sz = sizes[ii] >> if not sz & 1: >> origin[ii] -= 1 >> return filters._min_or_max_filter(input, size, footprint, >> structure, >

2 1

Ваш лист потрапив у папку "Спам". Ви можете "проштовхнути" його одним кліком.
by dmitrey＠mail.ua 30 Aug '09

30 Aug '09

(see english version below) Адреса Dmitrey Kroshko <dmitrey(a)mail.ua>, SciPy Users List <scipy-user(a)scipy.org> відсутня у моїй адресній книзі і тому Вашого листа було поміщено у папку "Спам". Щоб перенести ("проштовхнути") його у "Вхідні" кликніть на наступний лінк і введіть код, зображений на малюнку: http://www.dmitrey.mail.ua/confirm/1251643845.H305026P3954.mx.mail.ua Цю операцію достатньо виконати один раз. Після цього всі листи, надсілані з адреси Dmitrey Kroshko <dmitrey(a)mail.ua>, SciPy Users List <scipy-user(a)scipy.org>, будуть відразу направлятися у "Вхідні" без додаткових перевірок. Для відправлення мені повідомлень Ви також можете скористатися моєю персональною сторінкою (візиткою) за адресою: http://www.dmitrey.mail.ua Цей спосіб більш зручний для випадків коли звичайний e-mail клієнт недоступний (наприклад в інтернет-кафе). З повагою, сподіваюсь на розуміння Dmitrey Kroshko dmitrey(a)mail.ua -------------------------- english ------------------------------- My antispam system detected your message as possible SPAM. To confirm that your message is not a spam use this link: http://www.dmitrey.mail.ua/confirm/1251643845.H305026P3954.mx.mail.ua After that your address Dmitrey Kroshko <dmitrey(a)mail.ua>, SciPy Users List <scipy-user(a)scipy.org> will be added to my address book and I'll be able to read your message. This operation should be done only once. After that you will be able to send me e-mail messages from Dmitrey Kroshko <dmitrey(a)mail.ua>, SciPy Users List <scipy-user(a)scipy.org> as usual. With the respect, I hope for the understanding Dmitrey Kroshko dmitrey(a)mail.ua

1 0

ANN: FuncDesigner 0.15RC2, DerApproximator 0.15RC2, OpenOpt 0.25RC2
by Dmitrey Kroshko 30 Aug '09

30 Aug '09

Hi all, (sorry if you get the message for twice - there were some problems occured) I'm glad to inform you about new packages - FuncDesigner and DerApproximator. They have been extracted from OpenOpt into standalone Python modules. FuncDesigner is a convenient tool for building functions and getting their derivatives via Automatic differentiation ( http://en.wikipedia.org/wiki/Automatic_differentiation ), not to be confused with numerical differentiation via finite-differences derivatives approximation and symbolic differentiation provided by Maxima, SymPy etc. So, you can optimize models coded in FuncDesigner by OpenOpt w/o providing user-supplied derivatives. For more details see http://openopt.org/FuncDesigner DerApproximator is a small yet important package for getting/checking derivatives (currently only 1st ones), extracted from OpenOpt framework to be standalone Python module. It is required by FuncDesigner (for obtaining derivatives of oofuns beyond standard set without routines to yield them directly) and some OpenOpt solvers (when there are some functions without user-supplied derivatives). See http://openopt.org/DerApproximator for more details. For more info about these RCs read http://forum.openopt.org/viewtopic.php?id=131 Regards, D.

1 0

symmetric lil_matrix
by T J 30 Aug '09

30 Aug '09

In constructing a scipy.sparse.lil_matrix, is there any built-in functionality to support symmetric matrices? >>> m = sparse.lil_matrix((3,3)) >>> m[1,2] = 3 >>> m <3x3 sparse matrix of type '<type 'numpy.float64'>' with 1 stored elements in LInked List format> >>> m[2,1] = m[1,2] <3x3 sparse matrix of type '<type 'numpy.float64'>' with 2 stored elements in LInked List format> Was wondering if m[2,1] could be automatically added if one declared the lil_matrix as symmetric. Also, the data structure says it has 2 stored elements. Is it actually storing two elements? My hope is that additional "sparseness" could be gained if it is known that the matrix is symmetric.

2 1

Monotonic Piecewise Cubic Hermite Interpolator (Matlab pchip() equivalent).
by Chris Michalski 30 Aug '09

30 Aug '09

Recently, I had a need for a monotonic piece-wise cubic Hermite interpolator. Matlab provides the function "pchip" (Piecewise Cubic Hermite Interpolator), but when I Googled I didn't find any Python equivalent. I tried "interp1d()" from scipy.interpolate but this was a standard cubic spline using all of the data - not a piece-wise cubic spline. This type of interpolator wasn't up to my needs because it tends to ring between points where there are sharp transitions because it isn't monotonic and it uses all of the data - which makes the interpolator "miss" sharp breaks in the data. I had access to Matlab documentation, so I spent a some time tracing through the code to figure out how I might write a Python duplicate. This was an massive exercise in frustration and a potent reminder on why I love Python and use Matlab only under duress. I find typical Matlab code is poorly documented (if at all) and that apparently includes the code included in their official releases. I also find Matlab syntax “dated” and the code very difficult to “read”. Wikipedia to the rescue: http://en.wikipedia.org/wiki/Cubic_Hermite_spline http://en.wikipedia.org/wiki/Monotone_cubic_interpolation Not to be deterred, these two very well written Wikipedia entries explained in simple language how to compute the interpolant. Hats off to whoever wrote these entries – they are excellent. The result was a surprising small amount of code considering the Matlab code was approaching 10 pages of incomprehensible code. Again - strong evidence that things are just better in Python... The resulting code: along with a matplotlib plot showing the results: As the plot shows, the "pchip" does preserve monotonicity compared to a "straight" piecewise cubic Hermite interpolation. In addition, both of the piecewise interpolators result in less "ringing" away from transitions because they only consider the data points in the immediate vicinity of the desired output point. Hope this is helpful in filling the void that several have asked about.

1 0

SciPy09 Video page direct links
by Eric Carlson 30 Aug '09

30 Aug '09

Hello, I had some issues searching for SciPy 09 video files this morning. Although the search was broken, at the same time it was still possible to link directly to the video pages. The attached basic html file (no javascript or images) has the tutorial and conference schedule and direct links to corresponding videos. You should be able to save the file and open locally. Cheers, Eric Carlson

7 11

bug in ndimage grey_dilation?
by Tony S Yu 30 Aug '09

30 Aug '09

ndimage.grey_dilation doesn't handle scalar size arguments (in contrast to other filters). For example: >>> ndimage.grey_dilation(np.ones((5,5)), size=2) TypeError: 'int' object is unsubscriptable Below is a simple patch which seems to fix the issue (I basically copied the behavior of ndimage.filters._min_or_max_filter). The diff below is taken against the 0.7.x branch. I guess this may not technically be a bug since filters in morphology.py don't specify they can handle scalar size arguments (filters in filters.py do document this behavior). In any case, this fix would make the API more consistent. Cheers, -Tony =================================================================== --- morphology.py (revision 5911) +++ morphology.py (working copy) @@ -347,13 +347,14 @@ footprint = footprint[tuple([slice(None, None, -1)] * footprint.ndim)] input = numpy.asarray(input) + sizes = _ni_support._normalize_sequence(size, input.ndim) origin = _ni_support._normalize_sequence(origin, input.ndim) for ii in range(len(origin)): origin[ii] = -origin[ii] if footprint is not None: sz = footprint.shape[ii] else: - sz = size[ii] + sz = sizes[ii] if not sz & 1: origin[ii] -= 1 return filters._min_or_max_filter(input, size, footprint, structure,

2 1