Mailman 3 March 2015 - SciPy-User

Pandas v0.16.0 release candidate 1
by Jeff Reback 13 Mar '15

13 Mar '15

Hi, I'm pleased to announce the availability of the first release candidate of Pandas 0.16.0. Please try this RC and report any issues here: Pandas Issues <https://github.com/pydata/pandas/issues> We will be releasing officially in 1 week or so. This is a major release from 0.15.2 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version. - Highlights include: - DataFrame.assign method, see *here* <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0160…> - Series.to_coo/from_coo methods to interact with scipy.sparse, see *here* <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0160…> - Backwards incompatible change to Timedelta to conform the .seconds attribute with datetime.timedelta, see *here* <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0160…> - Changes to the .loc slicing API to conform with the behavior of .ix see *here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#indexing-chan…>* - Changes to the default for ordering in the Categorical constructor, see *here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0160…>* Here are the full whatsnew and documentation links: v0.16.0 Whatsnew <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html> Source tarballs, windows builds, and mac wheels are available here: Pandas v0.16.0rc1 Release <https://github.com/pydata/pandas/releases> A big thank you to everyone who contributed to this release! Jeff

2 1

np.corrcoef ddof is redundant?
by Alistair Miles 13 Mar '15

13 Mar '15

Hi, I'm trying to calculate correlation coefficients and looking at the np.corrcoef function. It has bias and ddof arguments, however when I try different values of ddof with test data the results are always the same, i.e., changing ddof has no effect. From some back-of-the-envelope algebra I reckon the n/(n-ddof) normalisations should get cancelled out when calculating correlation coefficients from a covariance matrix, and therefore the ddof (and bias) arguments to np.corrcoef are redundant. I'd be very grateful if someone could verify this is true or tell me if I've missed something. Thanks, Alistair -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Web: http://purl.org/net/aliman Email: alimanfoo(a)gmail.com Tel: +44 (0)1865 287721

4 9

Robust fitting of an exponential distribution subpopulation
by Antonino Ingargiola 12 Mar '15

12 Mar '15

Hi to the list, I'm seeking the advise of the scientific python community to solve the following fitting problem. Both suggestions on the methodology and on particular software packages are appreciated. I often encounter the need to fit a sample containing a (dominant) exponentially-distributed sub-population. Mostly the non-exponential samples (from an unknown distribution) are distributed close to the origin of the exponential distribution, therefore a simple approach I used so far is selecting all the samples higher than a threshold and fitting the exponential "tail" with MLE. The problem is that the choice of the threshold is somewhat arbitrary and moreover there can be a small set of outlier on the extreme right-side of the distribution that would bias the MLE fit. To improve the accuracy, I'm thinking of using (if necessary implementing) some kind of robust fitting procedure. For example using a scheme in which the outlier are identified by putting a threshold on the residual and then this threshold is optimized using some "goodness of fit" cost function. If this approach reasonable? I am surely not the first to tackle this problem, so I would appreciated some suggestion and specific pointers to help me getting started. Thank you, Antonio

5 7

fitting discrete probability distributions to data
by c 12 Mar '15

12 Mar '15

hi, i have some data: A) a 1d array (dimensions 1x50), made by summing the columns of a 2d array (dimensions ~20k x 50). B) a 1D array that is just a particular row of that 2d array i need to fit a sum of 2 negative binomial distributions to A), and to fit a single negative binomial distrib. to B). i have spent a while now reading the documentation for numpy.stats and the statsmodel package and various stack overflow posts, etc.. but i do not yet understand how to go about fitting a discrete probability distribution to a vector of data. specific subquestions: - do i need to load data in as a pandas df? an ndarray? does it not matter? - i understand endog and exog in the context of the examples given in the docs (where you have one column that you want to use to predict some other column) but not what they should be in the case where i basically am trying to fit a curve to the normalized histogram of my data - if someone can explain how to fit with statsmodels' "Negative Binomial ( http://statsmodels.sourceforge.net/devel/generated/statsmodels.discrete.dis…) that would be a good start. but i do also need to know how to fit to a sum of two of these, or possibly a sum of two other discrete distributions - is the patsy formula syntax relevant here? i have never used R and could not find an example of the "R-like" syntax that is similar enough to my use case to parse how it works - honestly i don't know what i'm doing, please help! if these questions reveal grave ignorance, or are not directly relevant enough to scipy for this mailing list, i apologize and thanks for bearing with me. i barely know how to flip a coin, this stuff is new to me. thanks a lot c

2 3

Experimental Design with Python
by memilanuk 11 Mar '15

11 Mar '15

Hello there, Is anyone on the list involved with experimental design, aka 'design of experiments', aka 'DoE'? https://en.wikipedia.org/wiki/Design_of_experiments http://www.itl.nist.gov/div898/handbook/pri/pri.htm I'm having a very hard time finding much if anything about DoE involving Python, which I kind of thought would be a natural fit given the utility of scipy, etc.. R has an entire task view page on CRAN (http://cran.r-project.org/web/views/ExperimentalDesign.html). Is there some particular reason Python would be unsuitable for this sort of endeavor, or something to explain the utter lack of interest? I've found a few packages (pyDOE, pyDesign, experimentator) but nothing like the hype surrounding 'machine learning'. I'm looking at using it to optimize something (for a hobby), and the commercial stats packages such as Minitab, JMP, Stat-ease, etc. are pretty much out-of-reach. At this point I'm leaning towards using R; though I'm more comfortable overall with Python, I'm not really at a point to where I feel comfortable 'rolling my own' for this sort of thing just yet. Thanks, Monte -- Shiny! Let's be bad guys. Reach me @ memilanuk (at) gmail dot com

4 3

soft limiter function
by Neal Becker 10 Mar '15

10 Mar '15

I'm looking for a parameterized set of functions, similar to logistic, where a parameter determines the 'sharpness' of the transition from the linear region to the flat region. I need to keep all the same scaling and derivative near the origin - so like a family of logistic functions that would overlay near the origin, but would become increasingly sharp limiters as the parameter was varied. In the limit, would approach the ideal limiter x |x<1| y = { 1 x > 1 -1 x < -1 -- Those who fail to understand recursion are doomed to repeat it

4 6

Package recommendations for *efficiently* making good looking animations, or alternatively, how can I "predraw" matplotlib animation frames?
by Brian Merchant 09 Mar '15

09 Mar '15

Hi all, Right now, I am at the simulation stage in my modelling. I need to explore possible behaviours, in some parameter sets. In order to do this exploration, it is helpful to visualize the data via animation. I started with matplotlib, but am disappointed with how long it takes to produce an animation using my data. It is also difficult for me to navigate matplotlib's documentation. Anyway, let me explain what I am doing right now. Currently, my custom matplotlib animation (i.e. a class that inherit's from animation.TimedAnimation) class's `_draw_frame` method does this: 1. get line data for timestep `n` 2. update data of each line of interest in the axes with `set_data` (lets say for the sake of an example (`ax_line1.set_data(datax, datay`, where `ax_line1, = ax.plot(...)`) 3. make a list of each line that was updated, and set `self._drawn_artists` to that (so in the example if two lines of interest are there, then we update them `self._drawn_artists = [ax_line1, ax_line2]` Is there a way I can set the data in multiple copies of the line artists before `_draw_frame` is called? Then, all that `_draw_frame` has to do is fetch the predrawn lines, and update the drawn artist list with that. That means that I set the artist data in parallel, if I have the data already, and thus produce an animation faster? I tried doing `copy.deepcopy` of the artists, but got an error telling me that I cannot do so (`TransformationNode` has to be frozen). What other options do I have? At this point, I am totally open to exploring other packages apart from matplotlib, that may be recommended. Kind regards, Brian

4 3

[ANN] GSOC project for data / geometry visual analysis needs student with technical matplotlib proficiency
by Rob Clewley 06 Mar '15

06 Mar '15

Hi, One final reminder that I am a mentor for two Google Summer of Code projects that involve extensive matplotlib GUI development for python scientific software. The INCF (incf.org) is sponsoring two GSOC projects that will directly benefit PyDSTool (http://pydstool.sf.net), a math modeling toolbox for science and engineering that some of you will know. One, if not both, of the projects will involve significant technical usage of MPL's 2D GUI widgets but, more excitingly, to build new types of model visualization tools over MPL. MPL has been chosen to maximize platform independence and minimize reliance on additional user installation of third party libraries, and is well suited for the prototyping of application front ends for research code. http://incf.org/gsoc/2015/proposals/#-span--span----nbsp---span---span--spa… High level knowledge of math and the theoretical principles of cell biology are beneficial but not required. Applications can begin right away! Read more about the process here: http://www.google-melange.com/gsoc/homepage/google/gsoc2015 Thanks, Rob

1 0

ANN: BayesPy 0.3 released
by Jaakko Luttinen 05 Mar '15

05 Mar '15

Dear all, I am pleased to announce that BayesPy 0.3 has been released. BayesPy provides tools for variational Bayesian inference. The user can easily constuct conjugate exponential family models from nodes and run approximate posterior inference. BayesPy aims to be efficient and flexible enough for experts but also accessible for casual users. ----------------------------------------------------------------------- This release adds several state-of-the-art VB features. Below is a list of significant new features in this release: * Gradient-based optimization of the nodes by using either the Euclidean or Riemannian/natural gradient. This enables, for instance, the Riemannian conjugate gradient method. * Collapsed variational inference to improve the speed of learning. * Stochastic variational inference to improve scalability. * Pattern search to improve the speed of learning. * Deterministic annealing to improve robustness against initializations. * Gaussian Markov chains can use input signals. More details about the new features can be found here: http://www.bayespy.org/user_guide/advanced.html ------------------------------------------------------ PyPI: https://pypi.python.org/pypi/bayespy/0.3 Git repository: https://github.com/bayespy/bayespy Documentation: http://www.bayespy.org/ Best regards, Jaakko

1 0

ANN: xray v0.4 released
by Stephan Hoyer 03 Mar '15

03 Mar '15

I'm pleased to announce a major release of xray, v0.4. xray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures. Our goal is to provide a pandas-like and pandas-compatible toolkit for analytics on multi-dimensional arrays, rather than the tabular data for which pandas excels. Our approach adopts the Common Data Model for self-describing scientific data in widespread use in the Earth sciences: xray.Dataset is an in-memory representation of a netCDF file. Documentation: http://xray.readthedocs.org/ GitHub: https://github.com/xray/xray Highlights of this release: * Automatic alignment of index labels in arithmetic and when combining arrays or datasets. * Aggregations like mean now skip missing values by default. * Relaxed equality rules in concat and merge for variables with equal value(s) but different shapes. * New drop method for dropping variables or index labels. * Support for reindexing with a fill method like pandas. For more details, read the full release notes: http://xray.readthedocs.org/en/stable/whats-new.html Best, Stephan

1 0