Mailman 3 September 2014 - NumPy-Discussion

Custom dtypes without C -- or, a standard ndarray-like type
by Stephan Hoyer 24 Sep '14

24 Sep '14

pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet. Most of these look like numpy arrays but with custom dtypes and type specific methods/properties. But clearly nobody is particularly excited about writing the the C necessary to implement custom dtypes [3]. Nor is do we need the ndarray ABI. In many cases, writing C may not actually even be necessary for performance reasons, e.g., categorical can be fast enough just by wrapping an integer ndarray for the internal storage and using vectorized operations. And even if it is necessary, I think we'd all rather write Cython than C. It's great for pandas to write its own ndarray-like wrappers (*not* subclasses) that work with pandas, but it's a shame that there isn't a standard interface like the ndarray to make these arrays useable for the rest of the scientific Python ecosystem. For example, pandas has loads of fixes for np.datetime64, but nobody seems to be up for porting them to numpy (I doubt it would be easy). I know these sort of concerns are not new, but I wish I had a sense of what the solution looks like. Is anyone actively working on these issues? Does the fix belong in numpy, pandas, blaze or a new project? I'd love to get a sense of where things stand and how I could help -- without writing any C :). Thanks, Stephan [1] https://github.com/pydata/pandas/pull/7217 [2] https://github.com/geopandas/geopandas/issues/166 [3] https://github.com/numpy/numpy-dtypes

10 11

Changes to the generalized functions.
by Charles R Harris 24 Sep '14

24 Sep '14

Hi All, The question has come up as the whether of not to treat the new gufunc behavior as a bug fix, keeping the old constructor name, or have a different constructor. Keeping the name makes life easier as we don't need to edit the code where numpy currently uses gufuncs, but is risky if some third party depends on the old behavior. The gufuncs have been part of numpy since the 1.3 release, and google doesn't turn up any uses that I can see apart from repeats of numpy code. We can also make fixes if needed during the 1.10 beta release cycle. Even so, it is a bit of a risk. To spread the blame, if any, please weigh in on the following. 1. Yes, it is a bug, keep the current name and fix the behavior. 2. No, we need to be conservative and use a new function. Chuck

2 3

Building numpy with OpenMP support
by John 24 Sep '14

24 Sep '14

I am in the process of trying to build numpy with OpenMP support but have had several issues. Has anyone else built it with success that could offer some guidance in what needs to be passed at build time. For reference I am using Atlas 3.10.2 built with OpenMP as well (-F alg -fopenmp) Thanks, JB

1 0

PR #5109 - interpolation function for polar coordinates
by Saullo Castro 24 Sep '14

24 Sep '14

Dear all, today I've submitted a pull request: https://github.com/numpy/numpy/pull/5109 in order to include an interpolation function for angular coordinates, since using np.interp for this purpose is cumbersome. I kindly ask for your feedback and/or questions. Greetings, Saullo

1 0

All numpy-f2py tests failed
by Romu Hu 24 Sep '14

24 Sep '14

Hi, I'm using python27-numpy-f2py-1.7.1-9.el6.x86_64 from RHEL6, the package has a test directory "/usr/lib64/python2.7/site-packages/numpy/f2py/tests", when I run unittest in the directory, all 358 testcases fail: # cd /usr/lib64/python2.7/site-packages/numpy/f2py/tests # python27 -m unittest discover -v ====================================================================== ERROR: test_c_copy_in_from_23casttype (test_array_from_pyobj.test_BOOL_gen) ---------------------------------------------------------------------- Traceback (most recent call last): File "<string>", line 5, in setUp File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 126, in __new__ obj._init(name) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 132, in _init self.type_num = getattr(wrap,'NPY_'+self.NAME) AttributeError: 'NoneType' object has no attribute 'NPY_BOOL' ====================================================================== ERROR: test_c_in_from_23casttype (test_array_from_pyobj.test_BOOL_gen) ---------------------------------------------------------------------- Traceback (most recent call last): File "<string>", line 5, in setUp File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 126, in __new__ obj._init(name) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 132, in _init self.type_num = getattr(wrap,'NPY_'+self.NAME) AttributeError: 'NoneType' object has no attribute 'NPY_BOOL' ...... ...... ====================================================================== ERROR: test_optional_from_23seq (test_array_from_pyobj.test_USHORT_gen) ---------------------------------------------------------------------- Traceback (most recent call last): File "<string>", line 5, in setUp File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 126, in __new__ obj._init(name) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 132, in _init self.type_num = getattr(wrap,'NPY_'+self.NAME) AttributeError: 'NoneType' object has no attribute 'NPY_USHORT' ====================================================================== ERROR: test_optional_from_2seq (test_array_from_pyobj.test_USHORT_gen) ---------------------------------------------------------------------- Traceback (most recent call last): File "<string>", line 5, in setUp File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 126, in __new__ obj._init(name) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 132, in _init self.type_num = getattr(wrap,'NPY_'+self.NAME) AttributeError: 'NoneType' object has no attribute 'NPY_USHORT' ====================================================================== ERROR: test_optional_none (test_array_from_pyobj.test_USHORT_gen) ---------------------------------------------------------------------- Traceback (most recent call last): File "<string>", line 5, in setUp File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 126, in __new__ obj._init(name) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 132, in _init self.type_num = getattr(wrap,'NPY_'+self.NAME) AttributeError: 'NoneType' object has no attribute 'NPY_USHORT' ====================================================================== ERROR: test_in_out (test_array_from_pyobj.test_intent) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 270, in test_in_out assert_equal(str(intent.in_.out),'intent(in,out)') File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 67, in __getattr__ return self.__class__(self.intent_list+[name]) File "/opt/rh/python27/root/usr/lib64/python2.7/site-packages/numpy/f2py/tests/test_array_from_pyobj.py", line 62, in __init__ flags |= getattr(wrap,'F2PY_INTENT_'+i.upper()) AttributeError: 'NoneType' object has no attribute 'F2PY_INTENT_IN' ---------------------------------------------------------------------- Ran 358 tests in 0.047s FAILED (errors=358) It seems that all tests fail because the 'wrap' variable is None. Am I using a wrong way to run the tests? Any idea? Thanks Romu

2 4

Dataframe memory info printing
by Jeff Reback 23 Sep '14

23 Sep '14

For the 0.15.0 release of pandas (coming 2nd week of oct), we are going to include memory info printing: see here: https://github.com/pydata/pandas/pull/7619 This will be controllable by an option display.memory_usage. My question to the community should this be by default True, e.g. show the memory usage (this only applies to using df.info()) There is really no performance impact here. Pls let us know! thanks Jeff >>> df.info(memory_usage=True)<class 'pandas.core.frame.DataFrame'>Int64Index: 10000000 entries, 0 to 9999999Data columns (total 5 columns):date datetime64[ns]float float64int int64smallint int16string objectdtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)memory usage: 324.2 MB

1 0

ANN: xray 0.3 released
by Stephan Hoyer 22 Sep '14

22 Sep '14

I'm pleased to announce the v0.3 release for xray, N-D labeled arrays and datasets in Python. xray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures, Series and DataFrame: the xray DataArray and Dataset. Our goal is to provide a pandas-like and pandas-compatible toolkit for analytics on multi-dimensional arrays, rather than the tabular data for which pandas excels. Our approach adopts the Common Data Model for self-describing scientific data in widespread use in the Earth sciences (e.g., netCDF and OPeNDAP): xray.Dataset is an in-memory representation of a netCDF file. Documentation: http://xray.readthedocs.org Code: http://github.com/xray/xray Mailing list: xray-dev(a)googlegroups.com Comments, feedback and contributions are all very welcome! Cheers, Stephan

1 0

(no subject)
by Petr Viktorin 18 Sep '14

18 Sep '14

Hello, Over at Python-ideas, there is a thread [0] about the following discrepancy: >>> numpy.array(float('inf')) // 1 inf >>> float('inf') // 1 nan There are reasons for either result, but I believe it would be very nice if either Python or Numpy changed, so they would give the same value. If any of you have reasons to defend Numpy's (or Python's) choice, or otherwise want to chime in, please post there. Thanks, Petr Viktorin [0] https://mail.python.org/pipermail/python-ideas/2014-September/029365.html

5 7

Re: [Numpy-discussion] float('inf') // 1 = ?
by Petr Viktorin 18 Sep '14

18 Sep '14

Sorry for the lack of subject before. On Thu, Sep 18, 2014 at 6:48 PM, Benjamin Root <ben.root(a)ou.edu> wrote: > My vote is that NumPy is correct here. I see no reason why >>>> float('inf') / 1 > and >>>> float('inf') // 1 > > should return different results. I recommend reading the python-ideas thread; there are some arguments for both sides. > On Thu, Sep 18, 2014 at 12:31 PM, Petr Viktorin <encukou(a)gmail.com> wrote: >> >> Hello, >> Over at Python-ideas, there is a thread [0] about the following >> discrepancy: >> >> >>> numpy.array(float('inf')) // 1 >> inf >> >>> float('inf') // 1 >> nan >> >> There are reasons for either result, but I believe it would be very >> nice if either Python or Numpy changed, so they would give the same >> value. >> If any of you have reasons to defend Numpy's (or Python's) choice, or >> otherwise want to chime in, please post there. >> >> Thanks, >> Petr Viktorin >> >> >> [0] >> https://mail.python.org/pipermail/python-ideas/2014-September/029365.html >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

2 2

float('inf') // 1 = ?
by Petr Viktorin 18 Sep '14

18 Sep '14

(Apologies for the lack of subject earlier) Hello, Over at Python-ideas, there is a thread [0] about the following discrepancy: >>> numpy.array(float('inf')) // 1 inf >>> float('inf') // 1 nan There are reasons for either result, but I believe it would be very nice if either Python or Numpy changed, so they would give the same value. If any of you have reasons to defend Numpy's (or Python's) choice, or otherwise want to chime in, please post there. Thanks, Petr Viktorin [0] https://mail.python.org/pipermail/python-ideas/2014-September/029365.html

1 0