Mailman 3 April 2016 - NumPy-Discussion

f2py: ram usage
by Vasco Gervasi 11 Apr '16

11 Apr '16

Hi all, I am trying to write some code to do calculation onto an array: for each row I need to do some computation and have a number as return. To speed up the process I wrote a fortran subroutine that is called from python [using f2py] for each row of the array, so the input of this subroutine is a row and the output is a number. This method works but I saw some speed advantage if I pass the entire array to fortran and then, inside fortran, call the subroutine that does the math; so in this case I pass an array and return a vector. But I noticed that when python pass the array to fortran, the array is copied and the RAM usage double. Is there a way to "move" the array to fortran, I don't care if the array is lost after the call to fortran. The pyd module is generated using: python f2py.py -c --opt="-ffree-form -Ofast" -m F2PYMOD F2PYMOD.f90 Thanks Vasco

3 3

Preliminary schedule for 1.12
by Charles R Harris 09 Apr '16

09 Apr '16

Hi All, As we are trying out an accelerated release schedule this year it is time to start thinking of the 1.12 release. My current plan is to release a Numpy 1.11.1 at the end of the month with a few small fixups. Numpy 1.11.0 looks to have been one of our more successful releases and there are currently only three fixups in 1.11.x, none of which are major, so I think we can just release with no betas or release candidates unless something terrible turns up. After that release I'm looking to branch 1.12.x in early to mid May aiming at a final sometime in late July or early August. The main thing I think we must have in 1.12 is `__numpy_ufunc__`, so unless someone else wants to resurrect that topic I will do so myself starting sometime next week. I don't think a lot of work is needed to finish things up, Nathaniel's PR #6001 <https://github.com/numpy/numpy/pull/6001> is a good start and with the addition of some opt out code that adheres to the Python convention should provide a solution we can all live with. Others may disagree, which is why we are still discussing the topic at this late date, but I'm hopeful. If there are other PRs or issues that folks feel need to be in 1.12.x, please reply to this post. Chuck

1 0

ANN: bcolz 1.0.0 (final) released
by Francesc Alted 07 Apr '16

07 Apr '16

============================= Announcing bcolz 1.0.0 final ============================= What's new ========== Yeah, 1.0.0 is finally here. We are not introducing any exciting new feature (just some optimizations and bug fixes), but bcolz is already 6 years old and it implements most of the capabilities that it was designed for, so I decided to release a 1.0.0 meaning that the format is declared stable and that people can be assured that future bcolz releases will be able to read bcolz 1.0 data files (and probably much earlier ones too) for a long while. Such a format is fully described at: https://github.com/Blosc/bcolz/blob/master/DISK_FORMAT_v1.rst Also, a 1.0.0 release means that bcolz 1.x series will be based on C-Blosc 1.x series (https://github.com/Blosc/c-blosc). After C-Blosc 2.x (https://github.com/Blosc/c-blosc2) would be out, a new bcolz 2.x is expected taking advantage of shiny new features of C-Blosc2 (more compressors, more filters, native variable length support and the concept of super-chunks), which should be very beneficial for next bcolz generation. Important: this is a final release and there are no important known bugs there, so this is recommended to be used in production. Enjoy! For a more detailed change log, see: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst For some comparison between bcolz and other compressed data containers, see: https://github.com/FrancescAlted/DataContainersTutorials specially chapters 3 (in-memory containers) and 4 (on-disk containers). Also, if it happens that you are in Madrid during this weekend, you can drop by my tutorial and talk: http://pydata.org/madrid2016/schedule/ See you! What it is ========== *bcolz* provides columnar and compressed data containers that can live either on-disk or in-memory. Column storage allows for efficiently querying tables with a large number of columns. It also allows for cheap addition and removal of column. In addition, bcolz objects are compressed by default for reducing memory/disk I/O needs. The compression process is carried out internally by Blosc, an extremely fast meta-compressor that is optimized for binary data. Lastly, high-performance iterators (like ``iter()``, ``where()``) for querying the objects are provided. bcolz can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr optimizes the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, since the carray/ctable containers can be disk-based, and it is possible to use them for seamlessly performing out-of-memory computations. bcolz has minimal dependencies (NumPy), comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Together, bcolz and the Blosc compressor, are finally fulfilling the promise of accelerating memory I/O, at least for some real scenarios: http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/queryi… Other users of bcolz are Visualfabriq (http://www.visualfabriq.com/) , Quantopian (https://www.quantopian.com/) and Scikit-Allel ( https://github.com/cggh/scikit-allel) which you can read more about by pointing your browser at the links below. * Visualfabriq: * *bquery*, A query and aggregation framework for Bcolz: * https://github.com/visualfabriq/bquery * Quantopian: * Using compressed data containers for faster backtesting at scale: * https://quantopian.github.io/talks/NeedForSpeed/slides.html * Scikit-Allel * Provides an alternative backend to work with compressed arrays * https://scikit-allel.readthedocs.org/en/latest/model/bcolz.html Resources ========= Visit the main bcolz site repository at: http://github.com/Blosc/bcolz Manual: http://bcolz.blosc.org Home of Blosc compressor: http://blosc.org User's mail list: bcolz(a)googlegroups.com http://groups.google.com/group/bcolz License is the new BSD: https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt Release notes can be found in the Git repository: https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst ---- **Enjoy data!** -- Francesc Alted

1 0

ANN: python-blosc 1.3.1
by Francesc Alted 07 Apr '16

07 Apr '16

============================= Announcing python-blosc 1.3.1 ============================= What is new? ============ This is an important release in terms of stability. Now, the -O1 flag for compiling the included C-Blosc sources on Linux. This represents slower performance, but fixes the nasty issue #110. In case maximum speed is needed, please `compile python-blosc with an external C-Blosc library < https://github.com/Blosc/python-blosc#compiling-with-an-installed-blosc-lib… )>`_. Also, symbols like BLOSC_MAX_BUFFERSIZE have been replaced for allowing backward compatibility with python-blosc 1.2.x series. For whetting your appetite, look at some benchmarks here: https://github.com/Blosc/python-blosc#benchmarking For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc/blob/master/RELEASE_NOTES.rst More docs and examples are available in the documentation site: http://python-blosc.blosc.org What is it? =========== Blosc (http://www.blosc.org) is a high performance compressor optimized for binary data. It has been designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call. Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. python-blosc (http://python-blosc.blosc.org/) is the Python wrapper for the Blosc compression library, with added functions (`compress_ptr()` and `pack_array()`) for efficiently compressing NumPy arrays, minimizing the number of memory copies during the process. python-blosc can be used to compress in-memory data buffers for transmission to other machines, persistence or just as a compressed cache. There is also a handy tool built on top of python-blosc called Bloscpack (https://github.com/Blosc/bloscpack). It features a commmand line interface that allows you to compress large binary datafiles on-disk. It also comes with a Python API that has built-in support for serializing and deserializing Numpy arrays both on-disk and in-memory at speeds that are competitive with regular Pickle/cPickle machinery. Sources repository ============== The sources and documentation are managed through github services at: http://github.com/Blosc/python-blosc ---- **Enjoy data!** -- Francesc Alted

1 0

ANN: numexpr 2.5.2 released
by Francesc Alted 07 Apr '16

07 Apr '16

========================= Announcing Numexpr 2.5.2 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== This is a maintenance release shaking some remaining problems with VML (it is nice to see how Anaconda VML's support helps raising hidden issues). Now conj() and abs() are actually added as VML-powered functions, preventing the same problems than log10() before (PR #212); thanks to Tom Kooij. Upgrading to this release is highly recommended. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

[ANN] Summer School "Advanced Scientific Programming in Python" in Reading, UK, September 5—11, 2016
by Tiziano Zito 07 Apr '16

07 Apr '16

Advanced Scientific Programming in Python ========================================= a Summer School by the G-Node, and the Centre for Integrative Neuroscience and Neurodynamics, School of Psychology and Clinical Language Sciences, University of Reading, UK Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists have been trained to use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques and best practices which are standard in the industry, but especially tailored to the needs of a programming scientist. Lectures are devised to be interactive and to give the students enough time to acquire direct hands-on experience with the materials. Students will work in pairs throughout the school and will team up to practice the newly learned skills in a real programming project — an entertaining computer game. We use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. We show how clean language design, ease of extensibility, and the great wealth of open source libraries for scientific computing and data visualization are driving Python to become a standard tool for the programming scientist. This school is targeted at Master or PhD students and Post-docs from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. Basic knowledge of Python and of a version control system such as git, subversion, mercurial, or bazaar is assumed. Participants without any prior experience with Python and/or git should work through the proposed introductory material before the course. We are striving hard to get a pool of students which is international and gender-balanced. You can apply online: https://python.g-node.org Application deadline: 23:59 UTC, May 15, 2016. Be sure to read the FAQ before applying. Participation is for free, i.e. no fee is charged! Participants however should take care of travel, living, and accommodation expenses by themselves. Travel grants may be available. Date & Location =============== September 5—11, 2016. Reading, UK Program ======= - Best Programming Practices • Best practices for scientific programming • Version control with git and how to contribute to open source projects with GitHub • Best practices in data visualization - Software Carpentry • Test-driven development • Debugging with a debuggger • Profiling code - Scientific Tools for Python • Advanced NumPy - Advanced Python • Decorators • Context managers • Generators - The Quest for Speed • Writing parallel applications • Interfacing to C with Cython • Memory-bound problems and memory profiling • Data containers: storage and fast access to large data - Practical Software Development • Group project Preliminary Faculty =================== • Francesc Alted, freelance consultant, author of PyTables, Spain • Pietro Berkes, Enthought Inc., Cambridge, UK • Zbigniew Jędrzejewski-Szmek, Krasnow Institute, George Mason University, Fairfax, VA, USA • Eilif Muller, Blue Brain Project, École Polytechnique Fédérale de Lausanne, Switzerland • Juan Nunez-Iglesias, Victorian Life Sciences Computation Initiative, University of Melbourne, Australia • Rike-Benjamin Schuppner, Institute for Theoretical Biology, Humboldt-Universität zu Berlin, Germany • Bartosz Teleńczuk, European Institute for Theoretical Neuroscience, CNRS, Paris, France • Stéfan van der Walt, Berkeley Institute for Data Science, UC Berkeley, CA, USA • Nelle Varoquaux, Centre for Computational Biology Mines ParisTech, Institut Curie, U900 INSERM, Paris, France • Tiziano Zito, freelance consultant, Germany Organizers ========== For the German Neuroinformatics Node of the INCF (G-Node) Germany: • Tiziano Zito, freelance consultant, Germany • Zbigniew Jędrzejewski-Szmek, Krasnow Institute, George Mason University, Fairfax, USA • Jakob Jordan, Institute of Neuroscience and Medicine (INM-6), Forschungszentrum Jülich GmbH, Germany For the Centre for Integrative Neuroscience and Neurodynamics, School of Psychology and Clinical Language Sciences, University of Reading UK: • Etienne Roesch, Centre for Integrative Neuroscience and Neurodynamics, University of Reading, UK Website: https://python.g-node.org Contact: python-info(a)g-node.org

1 0

mtrand.c update 1.11 breaks my crappy code
by Neal Becker 06 Apr '16

06 Apr '16

I have C++ code that tries to share the mtrand state. It unfortunately depends on the layout of RandomState which used to be: struct __pyx_obj_6mtrand_RandomState { PyObject_HEAD rk_state *internal_state; PyObject *lock; }; But with 1.11 it's: struct __pyx_obj_6mtrand_RandomState { PyObject_HEAD struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab; rk_state *internal_state; PyObject *lock; PyObject *state_address; }; So 1. Why the change? 2. How can I write portable code?

3 6

Build NumPy against MATLAB BLAS & Lapack
by Marmaduke Woodman 06 Apr '16

06 Apr '16

hi I'm trying to provide a Python numerical library to colleagues who are MATLAB users, since recent MATLAB versions include an official method for calling Python [1], and I'm struggling to build NumPy which is compatible with MATLAB's libraries. My current site.cfg is here https://gist.github.com/maedoc/a41cb253011ad55edacca560a66dff76 I build from Git master with BLAS=none LAPACK=none ATLAS=none python setup.py install Currently, most of NumPy works, except linear algebra calls, like pinv, which segfault inside the MKL: np.__config__.show() shows NumPy thinks it linked against OpenBLAS with a CBLAS ABI, not a standard Fortran BLAS api. Is there a way to force the NumPy build to ignore OpenBLAS and assume it's calling the standard Fortran style BLAS functions? Thanks in advance for any comments or suggestions, Marmaduke [1] http://fr.mathworks.com/help/matlab/call-python-libraries.html

1 0

Multidimension array access in C via Python API
by mpc 05 Apr '16

05 Apr '16

Hello, is there a C-API function for numpy that can implement Python's multidimensional indexing? For example, if I had a 2d array: PyArrayObject * M; and an index int i; how do I extract the i-th row M[i,:] or i-th column M[:,i]? Ideally it would be great if it returned another PyArrayObject* object (not a newly allocated one, but whose data will point to the correct memory locations of M). I've searched everywhere in the API documentation, Google, and SO, but no luck. Any help is greatly appreciated. Thank you. -Matthew -- View this message in context: http://numpy-discussion.10968.n7.nabble.com/Multidimension-array-access-in-… Sent from the Numpy-discussion mailing list archive at Nabble.com.

8 20

rational custom dtype example
by Steve Mitchell 04 Apr '16

04 Apr '16

I have noticed a few issues with the "rational" custom C dtype example. 1. It doesn't build on Windows. I managed to tweak it to build. Mainly, the MSVC9 compiler is C89. 2. A few tests don't pass on Windows, due to integer sizes. 3. The copyswap and copyswapn routines don't do in-place swapping if src == NULL, as specified in the docs. http://docs.scipy.org/doc/numpy-1.10.0/reference/c-api.types-and-structures… --Steve

2 1