Mailman 3 February 2014 - NumPy-Discussion

Re: [Numpy-discussion] Overlapping time series
by RayS 11 Feb '14

11 Feb '14

> > On 11.02.2014 14:08, Daniele Nicolodi wrote: > >> Hello, > >> > >> I have two time series (2xN dimensional arrays) recorded on the same > >> time basis, but each with it's own dead times (and start and end > >> recording times). I would like to obtain two time series containing > >> only the time overlapping segments of the data. > >> > >> Does numpy or scipy offer something that may help in this? > >> > >> I can imagine strategies about how to approach the problem, but none > >> that would be efficient. Ideas? or? st1 = 5 >>> np.concatenate((a1[:,st1:],a2[:,:a1.shape[1]-st1])) array([[ 5, 6, 7, 8, 9], [15, 16, 17, 18, 19], [ 5, 6, 7, 8, 9], [15, 16, 17, 18, 19]]) - Ray

1 0

Inheriting from ndarray Was: deprecate numpy.matrix
by Alexander Belopolsky 11 Feb '14

11 Feb '14

On Mon, Feb 10, 2014 at 11:31 AM, Nathaniel Smith <njs(a)pobox.com> wrote: > And in the long run, I > think the goal is to move people away from inheriting from np.ndarray. > This is music to my ears, but what is the future of numpy.ma? I understand that numpy.oldnumeric.ma (the older version written without inheritance) has been deprecated and slated to be removed in 1.9. I also have seen some attempts to bring ma functionality into the core ndarray object, but those have not been successful as far as I can tell. In general, what is the future of inheriting from np.ndarray?

3 2

[pydata] ANN: pandas 0.13.1 released
by Jeff Reback 09 Feb '14

09 Feb '14

> Hello, > > This is a minor release from 0.13.0 and includes a small number of API changes, several new features, enhancements, and > performance improvements along with a large number of bug fixes. > > We recommend that all users upgrade to this version. > > Highlights include: > > - Added infer_datetime_format keyword to read_csv/to_datetime to allow speedups for homogeneously formatted datetimes. > - Will intelligently limit display precision for datetime/timedelta formats. > - Enhanced Panel apply() method. > - Suggested tutorials in new Tutorials section. > - Our pandas ecosystem is growing, We now feature related projects in a new Pandas Ecosystem section. > - Much work has been taking place on improving the docs, and a new Contributing section has been added. > > v0.13.1 Whatsnew Page > http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#v0-13-1-febru… > > v0.13.1 Documentation Page > http://pandas.pydata.org/pandas-docs/stable/ > > Pleas visit here for the source tarball: > https://github.com/pydata/pandas/releases/tag/v0.13.1 > > Windows binaries are available from Christoph Gohlke's collection: > http://www.lfd.uci.edu/~gohlke/pythonlibs/#pandas > > tarballs and windows binaries are available on PyPi: > https://pypi.python.org/pypi/pandas > > We are looking forward to a next planned release of v0.14.0 in about three months time. > > Some things that we would like to include: > > - A big upgrade to SQL to/from interop with support for all major DBs, leveraging SQLAlchemy. > - Template-based displays for dataframes, with conditional formatting and roll-your-own output generation. > - Reduced memory dataframe construction from known-length iterators. > - Your PRs. > > Thanks > > The Pandas Team > > > Contributors to the 0.13.1 release > > $ git log v0.12.1..v0.13.1 --pretty='%aN##%s' | grep -v 'Merge pull' | grep -Po '^[^#]+' | sort | uniq -c | sort -rn > > 146 y-p > 97 jreback > 14 Joris Van den Bossche > 8 Phillip Cloud > 8 Andy Hayden > 6 unutbu > 4 Skipper Seabold > 3 TomAugspurger > 3 Jeff Tratner > 3 DSM > 3 Douglas McNeil > 3 Dan Birken > 3 Chapman Siu > 2 Tom Augspurger > 2 Naveen Michaud-Agrawal > 2 Michael Schatzow > 2 Kieran O'Mahony > 2 Jacob Schaer > 2 Doran Deluz > 2 danielballan > 2 Clark Fitzgerald > 2 chapman siu > 2 Caleb Epstein > 2 Brad Buran > 2 Andrew Burrows > 2 Alex Rothberg > 1 Spencer Lyon > 1 Roman Pekar > 1 Patrick O'Keeffe > 1 mwaskom > 1 lexual > 1 Julia Evans > 1 John McNamara > 1 Jan Wagner > 1 immerrr > 1 Guillaume Gay > 1 George Kuan > 1 Felix Lawrence > 1 Elliot S > 1 Draen Luanin > 1 Douglas Rudd > 1 David Wolever > 1 davidshinn > 1 david > 1 Daniel Waeber > 1 Chase Albert > 1 bwignall > 1 bmu > 1 Bjorn Arneson > 1 Alok Singhal > 1 akittredge > 1 acorbe > > -- > You received this message because you are subscribed to the Google Groups "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an email to pydata+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

1 0

create numerical arrays from strings
by Alan G Isaac 09 Feb '14

09 Feb '14

NumPy matrix construction includes as a convenience feature the construction of matrices with a Matlab-like syntax. E.g., np.mat('1 2;3 4'). Is it correct that this syntax is not supported for direct (i.e., not using `mat`) ndarray creation? You may ask, where would this possibly matter? The answer: in the undergraduate classroom. Compare np.mat('1 2; 3 4') to np.array([[1, 2], [3, 4]]) for readability and intimidation factor. Little things matter when getting started with students who lack programming background. Thanks, Alan Isaac

4 3

MKL and OpenBLAS
by Dinesh Vadhia 07 Feb '14

07 Feb '14

This conversation gets discussed often with Numpy developers but since the requirement for optimized Blas is pretty common these days, how about distributing Numpy with OpenBlas by default? People who don't want optimized BLAS or OpenBLAS can then edit the site.cfg file to add/remove. I can never remember if Numpy comes with Atlas by default but either way, if using MKL is not feasible because of its licensing issues then Numpy has to be re-compiled with OpenBLAS (for example). Why not make it easier for developers to use Numpy with an in-built optimized Blas. Btw, just in case some folks from Intel are listening: how about releasing MKL binaries for all platforms for developers to do with it what they want ie. free. You know it makes sense!

16 39

Re: [Numpy-discussion] create numerical arrays from strings
by David Goldsmith 07 Feb '14

07 Feb '14

Date: Thu, 6 Feb 2014 08:42:38 -0800 > From: Chris Barker <chris.barker(a)noaa.gov> > Subject: Re: [Numpy-discussion] create numerical arrays from strings > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > < > CALGmxEKVNQok6wtY-jbjzgaeU5ewHh1_FLmSQXJsUJfcLExh2w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > 1) so use np.mat ! > To elaborate on this (because I, for one, was not aware that mat supported this API, and, significantly, the fact that it does, does not appear in its docstring: import numpy as np help(np.mat) Help on function asmatrix in module numpy.matrixlib.defmatrix: asmatrix(data, dtype=None) Interpret the input as a matrix. Unlike `matrix`, `asmatrix` does not make a copy if the input is already a matrix or an ndarray. Equivalent to ``matrix(data, copy=False)``. Parameters ---------- data : array_like Input data. Returns ------- mat : matrix `data` interpreted as a matrix. Examples -------- >>> x = np.array([[1, 2], [3, 4]]) >>> m = np.asmatrix(x) >>> x[0,0] = 5 >>> m matrix([[5, 2], [3, 4]]) ) However, we do have: a=np.mat('1 2;3 4') a matrix([[1, 2], [3, 4]]) b = np.array(a) b array([[1, 2], [3, 4]]) and so, as we should expect: c=np.array(np.mat('1 2;3 4')) c array([[1, 2], [3, 4]]) So the substance of the utility function Stefan suggests is one line: def numstr2numarr(in): """ 'in' is a matlab-style array containing strings for the numerical array entries """ return np.array(np.mat(in)) In essence, numpy "almost" provides the API you're asking for. DG

2 1

[ANN] Summer School "Advanced Scientific Programming in Python" in Split, Croatia
by Tiziano Zito 06 Feb '14

06 Feb '14

Advanced Scientific Programming in Python ========================================= a Summer School by the G-Node and the Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture (FESB), University of Split Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques, incorporating theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python is assumed. Participants without any prior experience with Python should work through the proposed introductory materials before the course. Date and Location ================= September 8—13, 2014. Split, Croatia Preliminary Program =================== Day 0 (Mon Sept 8) — Best Programming Practices • Best Practices for Scientific Computing • Version control with git and how to contribute to Open Source with github • Object-oriented programming & design patterns Day 1 (Tue Sept 9) — Software Carpentry • Test-driven development, unit testing & quality assurance • Debugging, profiling and benchmarking techniques • Advanced Python I: idioms, useful built-in data structures, generators Day 2 (Wed Sept 10) — Scientific Tools for Python • Advanced NumPy • The Quest for Speed (intro): Interfacing to C with Cython • Programming in teams Day 3 (Thu Sept 11) — The Quest for Speed • Writing parallel applications in Python • Python 3: why should I care • Programming project Day 4 (Fri Sept 12) — Efficient Memory Management • When parallelization does not help: the starving CPUs problem • Advanced Python II: decorators and context managers • Programming project Day 5 (Sat Sept 13) — Practical Software Development • Programming project • The Pelita Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects. Applications ============ You can apply on-line at http://python.g-node.org Applications must be submitted before 23:59 UTC, May 1, 2014. Notifications of acceptance will be sent by June 1, 2014. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate is usually around 20%. Prerequisites: You are supposed to know the basics of Python to participate in the lectures. You are encouraged to go through the introductory material available on the website. Faculty ======= • Francesc Alted, Continuum Analytics Inc., USA • Pietro Berkes, Enthought Inc., UK • Kathryn D. Huff, Department of Nuclear Engineering, University of California - Berkeley, USA • Zbigniew Jędrzejewski-Szmek, Krasnow Institute, George Mason University, USA • Eilif Muller, Blue Brain Project, École Polytechnique Fédérale de Lausanne, Switzerland • Rike-Benjamin Schuppner, Technologit GbR, Germany • Nelle Varoquaux, Centre for Computational Biology Mines ParisTech, Institut Curie, U900 INSERM, Paris, France • Stéfan van der Walt, Applied Mathematics, Stellenbosch University, South Africa • Niko Wilbert, TNG Technology Consulting GmbH, Germany • Tiziano Zito, Institute for Theoretical Biology, Humboldt-Universität zu Berlin, Germany Organized by Tiziano Zito (head) and Zbigniew Jędrzejewski-Szmek for the German Neuroinformatics Node of the INCF (Germany), Lana Periša for the Numerical and applied mathematics group, FESB, University of Split (Croatia), Ivana Kajić from the Bernstein Center for Computational Neuroscience Berlin (Germany), Ivana Balažević from the Technical University Berlin (Germany), and Filip Petkovski from IN2 Ltd. Skopje (Macedonia). Website: http://python.g-node.org Contact: python-info(a)g-node.org

1 0

ANN: PyTables 3.1.0 released
by Antonio Valentino 05 Feb '14

05 Feb '14

=========================== Announcing PyTables 3.1.0 =========================== We are happy to announce PyTables 3.1.0. This is a feature release. The upgrading is recommended for users that are running PyTables in production environments. What's new ========== Probably the most relevant changes in this release are internal improvements like the new node cache that is now compatible with the upcoming Python 3.4 and the registry for open files has been deeply reworked. The caching feature of file handlers has been completely dropped so now PyTables is a little bit more "thread friendly". New, user visible, features include: - a new lossy filter for HDF5 datasets (EArray, CArray, VLArray and Table objects). The *quantization* filter truncates floating point data to a specified precision before writing to disk. This can significantly improve the performance of compressors (many thanks to Andreas Hilboll). - support for the H5FD_SPLIT HDF5 driver (thanks to simleo) - all new features introduced in the Blosc_ 1.3.x series, and in particular the ability to leverage different compressors within Blosc_ are now available in PyTables via the blosc filter (a big thank you to Francesc) - the ability to save/restore the default value of :class:`EnumAtom` types Also, installations of the HDF5 library that have a broken support for the *long double* data type (see the `Issues with H5T_NATIVE_LDOUBLE`_ thread on the HFG5 forum) are detected by PyTables 3.1.0 and the corresponding features are automatically disabled. Users that need support for the *long double* data type should ensure to build PyTables against an installation of the HDF5 library that is not affected by the bug. .. _`Issues with H5T_NATIVE_LDOUBLE`: http://hdf-forum.184993.n3.nabble.com/Issues-with-H5T-NATIVE-LDOUBLE-tt4026… As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.1.0 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

Re: [Numpy-discussion] striding through arbitrarily large files
by RayS 05 Feb '14

05 Feb '14

At 12:11 PM 2/5/2014, Richard Hattersley wrote: >On 4 February 2014 15:01, RayS ><<mailto:rays@blue-cove.com>rays(a)blue-cove.com> wrote: >I was struggling with methods of reading large disk files into >numpy efficiently (not FITS or .npy, just raw files of IEEE floats >from numpy.tostring()). When loading arbitrarily large files it >would be nice to not bother reading more than the plot can display >before zooming in. There apparently are no built in methods that >allow skipping/striding... > > >Since you mentioned the plural "files", are your datasets entirely >contained within a single file? If not, you might be interested in >Biggus >(<https://pypi.python.org/pypi/Biggus>https://pypi.python.org/pypi/Biggus). >It's a small pure-Python module that lets you "glue-together" arrays >(such as those from smmap) into a single arbitrarily large virtual >array. You can then step over the virtual array and it maps it back >to the underlying sources. > >Richard ooh, that might help they are individual GB files from medical trial studies I see there are some examples about https://github.com/SciTools/biggus/wiki/Sample-usage http://nbviewer.ipython.org/gist/pelson/6139282 Thanks!

1 0

Re: [Numpy-discussion] striding through arbitrarily large files
by RayS 05 Feb '14

05 Feb '14

At 07:35 AM 2/4/2014, Julian Taylor wrote: >On Tue, Feb 4, 2014 at 4:27 PM, RayS ><<mailto:rays@blue-cove.com>rays(a)blue-cove.com> wrote: >At 07:09 AM 2/4/2014, you wrote: > >On 04/02/2014 16:01, RayS wrote: > > > I was struggling with methods of reading large disk files into numpy > > > efficiently (not FITS or .npy, just raw files of IEEE floats from > > > numpy.tostring()). When loading arbitrarily large files it would be nice > > > to not bother reading more than the plot can display before zooming in. > > > There apparently are no built in methods that allow skipping/striding... > > > >If you mmap the data file with np.memmap() you can access the data in a > >strided way through the numpy array interface and the OS will handle the > >scheduling of the reads from the disc. > > > >Note however if that the data samples you need are quite dense, there is > >no real advantage in doing this because the OS will have to read a whole > >page anyway for each read. > >Thanks Daniele, I'll be trying mmap with Python64. With 32 bit the >mmap method throws MemoryError with 2.5GB files... >The idea is that we allow the users to inspect the huge files >graphically, then they can "zoom" into regions of interest and then >load a ~100 MB en block for the usual spectral analysis. > >memory maps are limited to the size of the available address space >(31 bits with sign), so you would have to slide them, see e.g. the >smmap module. >But its not likely this is going to be much faster than a loop with >explicit seeks depending on the sparseness of the data. memory maps >have relatively high overheads at the operating system level. yes, very sparse data - ~4k out of 50 million I hadn't tried smmap, https://github.com/Byron/smmap, thanks - Ray

2 1