Mailman 3 October 2015 - NumPy-Discussion

when did column_stack become C-contiguous?
by josef.pktd＠gmail.com 20 Oct '15

20 Oct '15

>>> np.column_stack((np.ones(10), np.ones(10))).flags C_CONTIGUOUS : True F_CONTIGUOUS : False >>> np.__version__ '1.9.2rc1' on my notebook which has numpy 1.6.1 it is f_contiguous I was just trying to optimize a loop over variable adjustment in regression, and found out that we lost fortran contiguity. I always thought column_stack is for fortran usage (linalg) What's the alternative? column_stack was one of my favorite commands, and I always assumed we have in statsmodels the right memory layout to call the linalg libraries. ("assumed" means we don't have timing nor unit tests for it.) Josef

4 12

correct sizeof for ndarray
by Jason Newton 20 Oct '15

20 Oct '15

Hi folks, I noticed an unexpected behavior of itemsize for structures with offsets that are larger than that of a packed structure in memory. This matters when parsing in memory structures from C and some others (recently and HDF5/h5py detail got me for a bit). So what is the correct way to get "sizeof" a structure? AFAIK this is the size of the last item + it's offset. If this doesn't exist... shouldn't it? Thanks, Jason

2 1

Behavior of numpy.copy with sub-classes
by Jonathan Helmus 20 Oct '15

20 Oct '15

In GitHub issue #3474, a number of us have started a conversation on how NumPy's copy function should behave when passed an instance which is a sub-class of the array class. Specifically, the issue began by noting that when a MaskedArray is passed to np.copy, the sub-class is not passed through but rather a ndarray is returned. I suggested adding a "subok" parameter which controls how sub-classes are handled and others suggested having the function call a copy method on duck arrays. The "subok" parameter is implemented in PR #6509 as an example. Both of these options would change the API of numpy.copy and possibly break backwards compatibility. Do others have an opinion of how np.copy should handle sub-classes? For a concrete example of this behavior and possible changes, what type should copy_x be in the following snippet: import numpy as np x = np.ma.array([1,2,3]) copy_x = np.copy(x) Cheers, - Jonathan Helmus

4 3

numpy 1.10.1 reduce operation on recarrays
by josef.pktd＠gmail.com 20 Oct '15

20 Oct '15

was there a change with reduce operations with recarrays in 1.10 or 1.10.1? Travis shows a new test failure in the statsmodels testsuite with 1.10.1: ERROR: test suite for <class 'statsmodels.base.tests.test_data.TestRecarrays'> File "/home/travis/miniconda/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/base/data.py", line 131, in _handle_constant const_idx = np.where(self.exog.ptp(axis=0) == 0)[0].squeeze() TypeError: cannot perform reduce with flexible type Sorry for asking so late. (statsmodels is short on maintainers, and I'm distracted) statsmodels still has code to support recarrays and structured dtypes from the time before pandas became popular, but I don't think anyone is using them together with statsmodels anymore. Josef

3 7

Making datetime64 timezone naive
by Stephan Hoyer 20 Oct '15

20 Oct '15

As has come up repeatedly over the past few years, nobody seems to be very happy with the way that NumPy's datetime64 type parses and prints datetimes in local timezones. The tentative consensus from last year's discussion was that we should make datetime64 timezone naive, like the standard library's datetime.datetime: http://thread.gmane.org/gmane.comp.python.numeric.general/57184 That makes sense to me, and it's exactly what I'd like to see happen for NumPy 1.11. Here's my PR to make that happen: https://github.com/numpy/numpy/pull/6453 As a temporary measure, we still will parse datetimes that include a timezone specification by converting them to UTC, but will issue a DeprecationWarning. This is important for a smooth transition, because at the very least I suspect the "Z" modifier for UTC is widely used. Another option would be to preserve this conversion indefinitely, without any deprecation warning. There's one (slightly) contentious API decision to make: What should we do with the numpy.datetime_to_string function? As far as I can tell, it was never documented as part of the NumPy API and has not been used very much or at all outside of NumPy's own test suite, but it is exposed in the main numpy namespace. If we can remove it, then we can delete and simplify a lot more code related to timezone parsing and display. If not, we'll need to do a bit of work so we can distinguish between the string representations of timezone naive and UTC. Best, Stephan

8 25

[Feature Suggestion]More comparison functions for floating point numbers
by cy18 19 Oct '15

19 Oct '15

I think these would be useful and easy to implement. greater_close(a, b) = greater_equal(a, b) | isclose(a, b) less_close(a, b) = less_equal(a, b) | isclose(a, b) greater_no_close = greater(a, b) & ~isclose(a, b) less_no_close = less(a, b) & ~isclose(a, b) The results are element-wise, just like the original functions. I'm not sure if it is useful enough to be a part of numpy. If so, I will try to implement them and make a pull request.

3 3

Workshop tonight, expect GitHub activity
by Jaime Fernández del Río 19 Oct '15

19 Oct '15

Hi all, As mentioned a few weeks ago, I am organizing a "Become an Open Source Contributor" workshop tonight, for the Data Science Student Society at UCSD. During this morning I will be creating a few ridiculously simple issues, e.g. "missing space, arrayobject --> array object", for participants to work on as part of the workshop. So there may also be a surge in simple PRs starting at around 7 PM PST. Please, bear with us. And refrain from fixing those issues if you are not a workshop participant! Thanks, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

1 1

Re: [Numpy-discussion] [SciPy-Dev] Setting up a dev environment with conda
by Luke Zoltan Kelley 19 Oct '15

19 Oct '15

Thanks for the help Nathaniel --- but building via `./runtests.py` is failing in the same way. Hopefully Numpy-discussion can help me out. I'm able to build using `python setup.py build_ext --inplace` but both trying to run `python setup.py install` or `./runtests.py` leads to the following error: (numpy-py27)daedalus-2:numpy lzkelley$ ./runtests.py Building, see build.log... Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 264, in <module> setup_package() File "setup.py", line 248, in setup_package from numpy.distutils.core import setup File "/Users/lzkelley/Programs/public/numpy/numpy/distutils/__init__.py", line 21, in <module> from numpy.testing import Tester File "/Users/lzkelley/Programs/public/numpy/numpy/testing/__init__.py", line 14, in <module> from .utils import * File "/Users/lzkelley/Programs/public/numpy/numpy/testing/utils.py", line 17, in <module> from numpy.core import float32, empty, arange, array_repr, ndarray File "/Users/lzkelley/Programs/public/numpy/numpy/core/__init__.py", line 59, in <module> test = Tester().test File "/Users/lzkelley/Programs/public/numpy/numpy/testing/nosetester.py", line 180, in __init__ if raise_warnings is None and '.dev0' in np.__version__: AttributeError: 'module' object has no attribute '__version__' Build failed! Has anyone seen something like this before? Thanks! Luke

4 5

dot product: large speed difference metween seemingly indentical operations
by Nadav Horesh 18 Oct '15

18 Oct '15

The functions dot, matmul and tensordot performs the same on a MxN matrix multiplied by length N vector, but very different if the matrix is replaced by a PxQxN array. Why? In [3]: a = rand(1000000,3) In [4]: a1 = a.reshape(1000,1000,3) In [5]: w = rand(3) In [6]: %timeit a.dot(w) 100 loops, best of 3: 3.47 ms per loop In [7]: %timeit a1.dot(w) # Very slow! 10 loops, best of 3: 25.5 ms per loop In [8]: %timeit a@w 100 loops, best of 3: 3.45 ms per loop In [9]: %timeit a1@w 100 loops, best of 3: 6.77 ms per loop In [10]: %timeit tensordot(a,w,1) 100 loops, best of 3: 3.44 ms per loop In [11]: %timeit tensordot(a1,w,1) 100 loops, best of 3: 3.41 ms per loop BTW, this is not a corner case, since PxQx3 arrays represent RGB images. Nadav

1 0

Re: [Numpy-discussion] [SciPy-Dev] Setting up a dev environment with conda
by Nathaniel Smith 17 Oct '15

17 Oct '15

Hi Luke, For day-to-day development and testing of numpy, I don't bother with either inplace builds *or* installing it -- I just use the magical "runtests.py" script that you'll find in the root of your git checkout. E.g., to build and then test the (possibly modified) source in your current checkout, just do: ./runtests.py That's all. This builds into a hidden directory and then sets up the correct PYTHONPATH before running the tests etc. -- you don't have to worry about any of it, it's magic. There are also lots of options, see ./runtests.py --help. Try adding -j for multi-core builds, or you can specify arbitrary options to pass to nose, or you can run it under gdb (there's an example in --help), or if you just want an interactive shell to futz around in manually instead of running the test suite then try passing --ipython. BTW, numpy has its own mailing list at numpy-discussion(a)scipy.org (CC'ed), which is where numpy development discussions usually take place -- this list is more for scipy-the-package itself. There's lots of overlap in readership between the two lists, but numpy-discussion will probably give you quicker and more useful answers to questions like this in general :-) -n On Sat, Oct 17, 2015 at 4:03 PM, Luke Zoltan Kelley <lkelley(a)cfa.harvard.edu> wrote: > Thanks Nathan, I'll try that. Both without the inplace build, I'll have to > rebuild and install everytime I want to test something, right? > > On Oct 17, 2015, at 6:42 PM, Nathan Woods <charlesnwoods(a)gmail.com> wrote: > > My best guess is to nuke and reclone Numpy, then do setup.py install without > the inplace build. What you're doing seems like it should work, though, so > I'm not sure what's going on. > > Nathan Woods > > On Oct 17, 2015, at 3:59 PM, Luke Zoltan Kelley <lkelley(a)cfa.harvard.edu> > wrote: > > When trying to do that is when I get the error I described in the OP. i.e. > I get an error when trying to install. > > > On Oct 17, 2015, at 5:57 PM, Andrew Nelson <andyfaff(a)gmail.com> wrote: > > It would get installed into whatever conda environment you had activated. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)scipy.org > https://mail.scipy.org/mailman/listinfo/scipy-dev > -- Nathaniel J. Smith -- http://vorpus.org

1 0