Mailman 3 April 2012 - NumPy-Discussion

(no subject)
by Dinesh Vadhia 28 Apr '12

28 Apr '12

http://www.cibingru.net/orndkty/566502.html

1 0

ANN: statsmodels 0.4.0
by josef.pktd＠gmail.com 27 Apr '12

27 Apr '12

We are pleased to announce the release of statsmodels 0.4.0. The big changes in this release are that most models can now be used with Pandas dataframes, and that we dropped the scikits namespace. Importing scikits.statsmodels is still possible but will be removed in the future. Pandas is now a required dependency. For more changes including some breaks in backwards compatibility see below. Josef and Skipper What it is ========== Statsmodels is a Python package that provides a complement to scipy for statistical computations including descriptive statistics and estimation and inference for statistical models. Documentation for the 0.4 version is currently at http://statsmodels.sourceforge.net/devel/ Main Changes and Additions in 0.4.0 ----------------------------------- * Added pandas dependency. * Cython source is built automatically if cython and compiler are present * Support use of dates in timeseries models * Improved plots - Violin plots - Bean Plots - QQ Plots * Added lowess function * Support for pandas Series and DataFrame objects. Results instances return pandas objects if the models are fit using pandas objects. * Full Python 3 compatibility * Fix bugs in genfromdta. Convert Stata .dta format to structured array preserving all types. Conversion is much faster now. * Improved documentation * Models and results are pickleable via save/load, optionally saving the model data. * Kernel Density Estimation now uses Cython and is considerably faster. * Diagnostics for outlier and influence statistics in OLS * Added El Nino Sea Surface Temperatures dataset * Numerous bug fixes * Internal code refactoring * Improved documentation including examples as part of HTML * ... *Changes that break backwards compatibility* * Deprecated scikits namespace. The recommended import is now:: import statsmodels.api as sm * model.predict methods signature is now (params, exog, ...) where before it assumed that the model had been fit and omitted the params argument. (this removed circularity between models and results instances) * For consistency with other multi-equation models, the parameters of MNLogit are now transposed. * tools.tools.ECDF -> distributions.ECDF * tools.tools.monotone_fn_inverter -> distributions.monotone_fn_inverter * tools.tools.StepFunction -> distributions.StepFunction Main Features ============= * linear regression models: Generalized least squares (including weighted least squares and least squares with autoregressive errors), ordinary least squares. * glm: Generalized linear models with support for all of the one-parameter exponential family distributions. * discrete: regression with discrete dependent variables, including Logit, Probit, MNLogit, Poisson, based on maximum likelihood estimators * rlm: Robust linear models with support for several M-estimators. * tsa: models for time series analysis - univariate time series analysis: AR, ARIMA - vector autoregressive models, VAR and structural VAR - descriptive statistics and process models for time series analysis * nonparametric : (Univariate) kernel density estimators * datasets: Datasets to be distributed and used for examples and in testing. * stats: a wide range of statistical tests - diagnostics and specification tests - goodness-of-fit and normality tests - functions for multiple testing - various additional statistical tests * iolib - Tools for reading Stata .dta files into numpy arrays. - printing table output to ascii, latex, and html * miscellaneous models * sandbox: statsmodels contains a sandbox folder with code in various stages of developement and testing which is not considered "production ready". This covers among others Mixed (repeated measures) Models, GARCH models, general method of moments (GMM) estimators, kernel regression, various extensions to scipy.stats.distributions, panel data models, generalized additive models and information theoretic measures. Where to get it =============== The master branch on GitHub is the most up to date code https://www.github.com/statsmodels/statsmodels Source download of release tags are available on GitHub https://github.com/statsmodels/statsmodels/tags Binaries and source distributions are available from PyPi http://pypi.python.org/pypi/statsmodels/

1 0

SciPy 2012 - The Eleventh Annual Conference on Scientific Computing with Python
by Warren Weckesser 26 Apr '12

26 Apr '12

SciPy 2012, the eleventh annual Conference on Scientific Computing with Python, will be held July 16–21, 2012, in Austin, Texas. At this conference, novel scientific applications and libraries related to data acquisition, analysis, dissemination and visualization using Python are presented. Attended by leading figures from both academia and industry, it is an excellent opportunity to experience the cutting edge of scientific software development. The conference is preceded by two days of tutorials, during which community experts provide training on several scientific Python packages. Following the main conference will be two days of coding sprints. We invite you to give a talk or present a poster at SciPy 2012. The list of topics that are appropriate for the conference includes (but is not limited to): - new Python libraries for science and engineering; - applications of Python in solving scientific or computational problems; - high performance, parallel and GPU computing with Python; - use of Python in science education. Specialized Tracks Two specialized tracks run in parallel to the main conference: - High Performance Computing with Python Whether your algorithm is distributed, threaded, memory intensive or latency bound, Python is making headway into the problem. We are looking for performance driven designs and applications in Python. Candidates include the use of Python within a parallel application, new architectures, and ways of making traditional applications execute more efficiently. - Visualization They say a picture is worth a thousand words--we’re interested in both! Python provides numerous visualization tools that allow scientists to show off their work, and we want to know about any new tools and techniques out there. Come show off your latest graphics, whether it’s an old library with a slick new feature, a new library out to challenge the status quo, or simply a beautiful result. Domain-specific Mini-symposia Mini-symposia on the following topics are also being organized: - Computational bioinformatics - Meteorology and climatology - Astronomy and astrophysics - Geophysics Talks, papers and posters We invite you to take part by submitting a talk or poster abstract. Instructions are on the conference website: <http://conference.scipy.org/scipy2012/papers.php> http://conference.scipy.org/scipy2012/talks.php<http://conference.scipy.org/scipy2012/papers.php> <http://conference.scipy.org/scipy2012/papers.php> Selected talks are included as papers in the peer-reviewed conference proceedings, to be published online. Tutorials Tutorials will be given July 16–17. We invite instructors to submit proposals for half-day tutorials on topics relevant to scientific computing with Python. See http://conference.scipy.org/scipy2012/tutorials.php<http://conference.scipy.org/scipy2011/tutorials.php> <http://conference.scipy.org/scipy2011/tutorials.php> for information about submitting a tutorial proposal. To encourage tutorials of the highest quality, the instructor (or team of instructors) is given a $1,000 stipend for each half day tutorial. Student/Community Scholarships We anticipate providing funding for students and for active members of the SciPy community who otherwise might not be able to attend the conference. See http://conference.scipy.org/scipy2012/student.php<http://conference.scipy.org/scipy2011/student.php> <http://conference.scipy.org/scipy2011/student.php> for scholarship application guidelines. Be a Sponsor The SciPy conference could not run without the generous support of the institutions and corporations who share our enthusiasm for Python as a tool for science. Please consider sponsoring SciPy 2012. For more information, see http://conference.scipy.org/scipy2012/sponsor/index.php Important dates: Monday, April 30: Talk abstracts and tutorial proposals due. Monday, May 7: Accepted tutorials announced. Monday, May 13: Accepted talks announced. Monday, June 18: Early registration ends. (Price increases after this date.) Sunday, July 8: Online registration ends. Monday-Tuesday, July 16 - 17: Tutorials Wednesday-Thursday, July 18 - July 19: Conference Friday-Saturday, July 20 - July 21: Sprints We look forward to seeing you all in Austin this year! The SciPy 2012 Team http://conference.scipy.org/scipy2012/organizers.php

1 1

(no subject)
by Andre Martel 26 Apr '12

26 Apr '12

What would be the best way to remove the maximum from a cube and "collapse" the remaining elements along the z-axis ? For example, I want to reduce Cube to NewCube: >>> Cube array([[[ 13, 2, 3, 42], [ 5, 100, 7, 8], [ 9, 1, 11, 12]], [[ 25, 4, 15, 1], [ 17, 30, 9, 20], [ 21, 2, 23, 24]], [[ 1, 2, 27, 28], [ 29, 18, 31, 32], [ -1, 3, 35, 4]]]) NewCube array([[[ 13, 2, 3, 1], [ 5, 30, 7, 8], [ 9, 1, 11, 12]], [[ 1, 2, 15, 28], [ 17, 18, 9, 20], [ -1, 2, 23, 4]]]) I tried with argmax() and then roll() and delete() but these all work on 1-D arrays only. Thanks.

3 3

linalg.lstsq
by Till Stensitzki 25 Apr '12

25 Apr '12

Hello, is there weighted version of linalg.lstsq available? In my case, b is a (N,K) matrix, so i can't use manual scaling of x and b. greetings Till

2 1

Masked Arrays in NumPy 1.x
by Travis Oliphant 24 Apr '12

24 Apr '12

Hey all, I've been waiting for Mark Wiebe to arrive in Austin where he will spend several weeks, but I also know that masked arrays will be only one of the things he and I are hoping to make head-way on while he is in Austin. Nevertheless, we need to make progress on the masked array discussion and if we want to finalize the masked array implementation we will need to finish the design. I've caught up on most of the discussion including Mark's NEP, Nathaniel's NEP and other writings and the very-nice mailing list discussion that included a somewhat detailed discussion on the algebra of IGNORED. I think there are some things still to be decided. However, I think some things are pretty clear: 1) Masked arrays are going to be fundamental in NumPy and these should replace most people's use of numpy.ma. The numpy.ma code will remain as a compatibility layer 2) The reality of #1 and NumPy's general philosophy to date means that masked arrays in NumPy should support the common use-cases of masked arrays (including getting and setting of the mask from the Python and C-layers). However, the semantic of what the mask implies may change from what numpy.ma uses to having a True value meaning selected. 3) There will be missing-data dtypes in NumPy. Likely only a limited sub-set (string, bytes, int64, int32, float32, float64, complex64, complex32, and object) with an API that allows more to be defined if desired. These will most likely use Mark's nice machinery for managing the calculation structure without requiring new C-level loops to be defined. 4) I'm still not sure about whether the IGNORED concept is necessary or not. I really like the separation that was emphasized between implementation (masks versus bit-patterns) and operations (propagating versus non-propagating). Pauli even created another dimension which I don't totally grok and therefore can't remember. Pauli? Do you still feel that is a necessary construction? But, do we need the IGNORED concept to indicate what amounts to different default key-word arguments to functions that operate on NumPy arrays containing missing data (however that is represented)? My current weak view is that it is not really necessary. But, I could be convinced otherwise. I think the good news is that given Mark's hard-work and Nathaniel's follow-up we are really quite far along. I would love to get Nathaniel's opinion about what remains un-done in the current NumPy code-base. I would also appreciate knowing (from anyone with an interest) opinions of items 1-4 above and anything else I've left out. Thanks, -Travis

7 10

[ANN] Optimization with categorical variables, disjunctive (and other logical) constraints
by Dmitrey 24 Apr '12

24 Apr '12

hi all, free solver interalg for global nonlinear optimization with specifiable accuracy now can handle categorical variables, disjunctive (and other logical) constraints, thus making it available to solve GDP, possibly in multiobjective form. There are ~ 2 months till next OpenOpt release, but I guess someone may find it useful for his purposes right now. See here for more details. Regards, D.

1 0

A 1.6.2 release?
by Charles R Harris 24 Apr '12

24 Apr '12

Hi All, Given the amount of new stuff coming in 1.7 and the slip in it's schedule, I wonder if it would be worth putting out a 1.6.2 release with fixes for einsum, ticket 1578, perhaps some others. My reasoning is that the fall releases of Fedora, Ubuntu are likely to still use 1.6 and they might as well use a somewhat fixed up version. The downside is located and backporting fixes is likely to be a fair amount of work. A 1.7 release would be preferable, but I'm not sure when we can make that happen. Chuck

4 13

NEP mask code and the 1.7 release
by Nathaniel Smith 23 Apr '12

23 Apr '12

We need to decide what to do with the NA masking code currently in master, vis-a-vis the 1.7 release. While this code is great at what it is, we don't actually have consensus yet that it's the best way to give our users what they want/need -- or even an appropriate way. So we need to figure out how to release 1.7 without committing ourselves to supporting this design in the future. Background: what does the code currently in master do? -------------------------------------------- It adds 3 pointers at the end of the PyArrayObject struct (which is better known as the numpy.ndarray object). These new struct members, and some accessors for them, are exposed as part of the public API. There are also a few additions to the Python-level API (mask= argument to np.array, skipna= argument to ufuncs, etc.) What does this mean for compatibility? ------------------------------------------------ The change in the ndarray struct is not as problematic as it might seem, compatibility-wise, since Python objects are almost always referred to by pointers. Since the initial part of the struct will continue to have the same memory layout, existing source and binary code that works with PyArrayObject *pointers* will continue to work unchanged. One place where the actual struct size matters is for any C-level ndarray subclasses, which will have their memory layout change, and thus will need to be recompiled. (Python-level ndarray subclasses will have their memory layout change as well -- e.g., they will have different __dictoffset__ values -- but it's unlikely that any existing Python code depends on such details.) What if we want to change our minds later? ------------------------------------------------------- For the same reasons as given above, any new code which avoids referencing the new struct fields referring to masks, or using the new masking APIs, will continue to work even if the masking is later removed. Any new code which *does* refer to the new masking APIs, or references the fields directly, will break if masking is later removed. Specifically, source will fail to compile, and existing binaries will silently access memory that is past the end of the PyArrayObject struct, which will have unpredictable consequences. (Most likely segfaults, but no guarantees.) This applies even to code which simply tries to check whether a mask is present. So I think the preconditions for leaving this code as-is for 1.7 are that we must agree: * We are willing to require a recompile of any C-level ndarray subclasses (do any exist?) * We are willing to make absolutely no guarantees about future compatibility for code which uses APIs marked "experimental" * We are willing for this breakage to occur in the form of random segfaults * We are okay with the extra 3 pointers worth of memory overhead on each ndarray Personally I can live with all of these if everyone else can, but I'm nervous about reducing our compatibility guarantees like that, and we'd probably need, at a minimum, a flashier EXPERIMENTAL sign than we currently have. (Maybe we should resurrect the weasels ;-) [1]) [1] http://mail.scipy.org/pipermail/numpy-discussion/2012-March/061204.html Any other options? ------------------------ Alternative 1: The obvious other option is to go through and move all the strictly mask-related code out of master and into a branch. Presumably this wouldn't include all the infrastructure that Mark added, since a lot of it is e.g. shared with where=, and that would stay. Even so, this would be a big and possibly time-consuming change. Alternative 2: After auditing the code a bit, the cleanest third option I can think of is: 1. Go through and make sure that all numpy-internal access to the new maskna fields happens via the accessor functions. (This patch would produce no functionality change.) 2. Move the accessors into some numpy-internal header file, so that user code can't call them. 3. Remove the mask= argument to Python-level ndarray constructors, remove the new maskna_ fields from PyArrayObject, and modify the accessors so that they always return NULL, 0, etc., as if the array does not have a mask. This would make 1.7 completely compatible with 1.6 API and ABI-wise. But it would also be a minimal code change, leaving the mask-related code paths in place but inaccessible. If we decided to re-enable them, it would just be matter of reverting steps (3) and (2). The main downside I see with this approach is that leaving a bunch of inaccessible code paths lying around might make it harder to maintain 1.7 as a "long term support" release. I'm personally willing to implement either of these changes. Or perhaps there's another option that I'm not thinking of! -- Nathaniel

4 7

documentation bug: Matrix library page not populated
by Alan G Isaac 23 Apr '12

23 Apr '12

http://docs.scipy.org/doc/numpy/reference/routines.matlib.html#module-numpy… promises a list of functions that does not appear (at the moment, anyway). Alan Isaac

4 5