Mailman 3 October 2011 - NumPy-Discussion

Nice float -> integer conversion?
by Matthew Brett Nov. 2, 2011

Nov. 2, 2011

Hi, Have I missed a fast way of doing nice float to integer conversion? By nice I mean, rounding to the nearest integer, converting NaN to 0, inf, -inf to the max and min of the integer range? The astype method and cast functions don't do what I need here: In [40]: np.array([1.6, np.nan, np.inf, -np.inf]).astype(np.int16) Out[40]: array([1, 0, 0, 0], dtype=int16) In [41]: np.cast[np.int16](np.array([1.6, np.nan, np.inf, -np.inf])) Out[41]: array([1, 0, 0, 0], dtype=int16) Have I missed something obvious? See y'all, Matthew

4 8

Float128 integer comparison
by Matthew Brett Nov. 2, 2011

Nov. 2, 2011

Hi, Continuing the exploration of float128 - can anyone explain this behavior? >>> np.float64(9223372036854775808.0) == 9223372036854775808L True >>> np.float128(9223372036854775808.0) == 9223372036854775808L False >>> int(np.float128(9223372036854775808.0)) == 9223372036854775808L True >>> np.round(np.float128(9223372036854775808.0)) == np.float128(9223372036854775808.0) True Thanks for any pointers, Best, Matthew

3 3

float64 / int comparison different from float / int comparison
by Matthew Brett Nov. 2, 2011

Nov. 2, 2011

Hi, I just ran into this confusing difference between np.float and np.float64: In [8]: np.float(2**63) == 2**63 Out[8]: True In [9]: np.float(2**63) > 2**63-1 Out[9]: True In [10]: np.float64(2**63) == 2**63 Out[10]: True In [11]: np.float64(2**63) > 2**63-1 Out[11]: False In [16]: np.float64(2**63-1) == np.float(2**63-1) Out[16]: True I believe values above 2*52 are all represented as integers in float64. http://matthew-brett.github.com/pydagogue/floating_point.html Is this this int64 issue that came up earlier in float128 comparison? Why the difference between np.float and np.float64? Thanks for any insight, Matthew

3 5

Reason behind C_ABI_VERSION so high
by Sandro Tosi Nov. 1, 2011

Nov. 1, 2011

Hello, in Debian we're trying to define a way to handle numpy transitions more smoothly (you can read the proposal, if interested, at http://bugs.debian.org/643873). In order to do that, we'd like to use the C_API_VERSION and C_ABI_VERSION values; while for C_API_VERSION we can see it's a quite small value, with a clear history at ./numpy/core/code_generators/cversions.txt , we don't have quite clear why C_ABI_VERSION is such a high value and how it would be incremented. Could you please shine some light on it? can we, f.e., just take 6 for API and 9 for ABI and be sure we're seeing them incremented to 7 and 10 (respectively) when needed? C_ABI_VERSION is incremented in a different way? Thanks, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi

2 2

NumPy nogil API
by mark florisson Nov. 1, 2011

Nov. 1, 2011

Hello, First, I'd like to report a bug. It seems ndarray does not implement tp_traverse or tp_clear, so if you have a reference cycle in an ndarray with dtype object none of those objects will ever be collected. Secondly, please bear with me, I'm not a NumPy expert, but would it be possible to have NumPy export an API that can be called without the GIL? In the upcoming Cython release 0.16 (soon to be released) we will have what is called typed memoryviews [1], which allow you to obtain a typed view on any PEP 3118 buffer, and it allows you to do a lot more things without the GIL. e.g. it allows you to pass these views around, transpose them, slice them (in the same way as in NumPy but slightly more restricted, it doesn't support masks and such), index them etc without the GIL. However, there is a lot of good functionality in NumPy that is simply not accessible without going through a Python and GIL layer, only to have NumPy (possibly) release the GIL. So instead we could have an API that takes a C-level ndarray view descriptor (shape/strides/ndim (possibly suboffsets), base type description) that would do the actual work (and perhaps doesn't even need to know anything about Python) and won't need the GIL (this wouldn't work for the dtype object, of course). The Py_buffer struct comes to mind, but format strings are hard to deal with. It would be the caller's responsibility to do any necessary synchronization. This API would simply be wrapped by a Python API that gets this "view" from the PyArrayObject, and which may or may not decide to release the GIL, and call the nogil API. This wrapping API could even be written easily in Cython. As for exceptions and error handling, there are many ways we can think of doing this without requiring the GIL. One of the reasons that I think this is important is that when you're using cython.parallel [2] you don't want to hold the gil, but you do want your NumPy goodies. Cython re-implements a very small subset of NumPy to get you the core functionality, but to get back to NumPy world you have to acquire the GIL, convert your memoryview slice to an ndarray (using the buffer interface through numpy.asarray()) and then have NumPy operate on that. It's a pain to write and it's terrible for performance. Even if you forget the GIL part, there's still the (expensive and explicit) conversion. In general I think there might be many advantages to such functionality other than for Cython. There shouldn't really be a reason to tie NumPy only to the CPython platform. Anyway, what do you guys think, does this make any sense? Mark [1]: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src… [2]: http://docs.cython.org/src/userguide/parallelism.html

5 11

np.in1d() capacity limit?
by Grové Oct. 31, 2011

Oct. 31, 2011

I have picked up a strange limit to np.in1d(): ---------- b Out[100]: array(['2007-01-01T02:30:00+0200', '2007-01-01T03:00:00+0200', '2007-01-01T03:30:00+0200', ..., '2008-01-01T01:00:00+0200', '2008-01-01T01:30:00+0200', '2008-01-01T02:00:00+0200'], dtype='datetime64[s]') b.shape Out[101]: (17520,) a = b[0:42] np.in1d(b,a) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/grove/<ipython-input-103-a7d9d12b1353> in <module>() ----> 1 np.in1d(b,a) /usr/local/lib/python2.6/dist-packages/numpy/lib/arraysetops.pyc in in1d(ar1, ar2, assume_unique) 338 # here. The values from the first array should always come before 339 # the values from the second array. --> 340 order = ar.argsort(kind='mergesort') 341 sar = ar[order] 342 equal_adj = (sar[1:] == sar[:-1]) TypeError: requested sort not available for type But this works: a = b[0:41] np.in1d(b,a) Out[105]: array([ True, True, True, ..., False, False, False], dtype=bool) --------- In other words the limit seems to be 41 elements for a. Is this a bug or am I getting something wrong? Grové

2 2

problem in running test(nose) with numpy/scipy
by akshar bhosale Oct. 31, 2011

Oct. 31, 2011

hi, i have installed numpy (1.6.0) and scipy (0.9). , nose version is 1.0 i have intel cluster toolkit installed on my system. (11/069 version and mkl 10.3). i have machine having intel xeon processor and rhel 5.2 x86_64 platform. i have installed it with intel compilers. when i execute numpy.test and scipy.test, it hangs . numpy.test(verbose=3) Running unit tests for numpy NumPy version 1.6.0 NumPy is installed in /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy Python version 2.6 (r26:66714, May 29 2011, 15:10:47) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] nose version 1.0.0 nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext'] nose.selector: INFO: /home/akshar/Python-2.6/lib/ python2.6/site-packages/numpy/core/multiarray.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/scalarmath.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/umath.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/multiarray_tests.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/umath_tests.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/fft/fftpack_lite.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/random/mtrand.so is executable; skipped test_api.test_fastCopyAndTranspose ... ok test_arrayprint.TestArrayRepr.test_nan_inf ... ok test_str (test_arrayprint.TestComplexArray) ... ok Ticket 844. ... ok test_blasdot.test_blasdot_used ... ok test_blasdot.test_dot_2args ... ok test_blasdot.test_dot_3args ... ok test_blasdot.test_dot_3args_errors ... ok test_creation (test_datetime.TestDateTime) ... ok test_creation_overflow (test_datetime.TestDateTime) ... ok test_divisor_conversion_as (test_datetime.TestDateTime) ... ok test_divisor_conversion_bday (test_datetime.TestDateTime) ... ok test_divisor_conversion_day (test_datetime.TestDateTime) ... ok test_divisor_conversion_fs (test_datetime.TestDateTime) ... ok test_divisor_conversion_hour (test_datetime.TestDateTime) ... ok test_divisor_conversion_minute (test_datetime.TestDateTime) ... ok test_divisor_conversion_month (test_datetime.TestDateTime) ... ok test_divisor_conversion_second (test_datetime.TestDateTime) ... ok test_divisor_conversion_week (test_datetime.TestDateTime) ... ok test_divisor_conversion_year (test_datetime.TestDateTime) ... ok test_hours (test_datetime.TestDateTime) ... ok test_from_object_array (test_defchararray.TestBasic) ... ok test_from_object_array_unicode (test_defchararray.TestBasic) ... ok test_from_string (test_defchararray.TestBasic) ... ok test_from_string_array (test_defchararray.TestBasic) ... ok test_from_unicode (test_defchararray.TestBasic) ... ok test_from_unicode_array (test_defchararray.TestBasic) ... ok test_unicode_upconvert (test_defchararray.TestBasic) ... ok test_it (test_defchararray.TestChar) ... ok test_equal (test_defchararray.TestComparisons) ... ok test_greater (test_defchararray.TestComparisons) ... ok test_greater_equal (test_defchararray.TestComparisons) ... ok test_less (test_defchararray.TestComparisons) ... ok test_less_equal (test_defchararray.TestComparisons) ... ok test_not_equal (test_defchararray.TestComparisons) ... ok test_equal (test_defchararray.TestComparisonsMixed1) ... ok test_greater (test_defchararray.TestComparisonsMixed1) ... ok test_greater_equal (test_defchararray.TestComparisonsMixed1) ... ok test_less (test_defchararray.TestComparisonsMixed1) ... ok test_less_equal (test_defchararray.TestComparisonsMixed1) ... ok test_not_equal (test_defchararray.TestComparisonsMixed1) ... ok test_equal (test_defchararray.TestComparisonsMixed2) ... ok test_greater (test_defchararray.TestComparisonsMixed2) ... ok test_greater_equal (test_defchararray.TestComparisonsMixed2) ... ok test_less (test_defchararray.TestComparisonsMixed2) ... ok test_less_equal (test_defchararray.TestComparisonsMixed2) ... ok test_not_equal (test_defchararray.TestComparisonsMixed2) ... ok test_count (test_defchararray.TestInformation) ... ok test_endswith (test_defchararray.TestInformation) ... ok test_find (test_defchararray.TestInformation) ... ok test_index (test_defchararray.TestInformation) ... ok test_isalnum (test_defchararray.TestInformation) ... ok test_isalpha (test_defchararray.TestInformation) ... ok test_isdigit (test_defchararray.TestInformation) ... ok test_islower (test_defchararray.TestInformation) ... ok test_isspace (test_defchararray.TestInformation) ... ok test_istitle (test_defchararray.TestInformation) ... ok test_isupper (test_defchararray.TestInformation) ... ok test_len (test_defchararray.TestInformation) ... ok test_rfind (test_defchararray.TestInformation) ... ok test_rindex (test_defchararray.TestInformation) ... ok test_startswith (test_defchararray.TestInformation) ... ok test_capitalize (test_defchararray.TestMethods) ... ok test_center (test_defchararray.TestMethods) ... ok test_decode (test_defchararray.TestMethods) ... ok test_encode (test_defchararray.TestMethods) ... ok test_expandtabs (test_defchararray.TestMethods) ... ok test_isdecimal (test_defchararray.TestMethods) ... ok test_isnumeric (test_defchararray.TestMethods) ... ok test_join (test_defchararray.TestMethods) ... ok test_ljust (test_defchararray.TestMethods) ... ok test_lower (test_defchararray.TestMethods) ... ok test_lstrip (test_defchararray.TestMethods) ... ok test_partition (test_defchararray.TestMethods) ... ok test_replace (test_defchararray.TestMethods) ... ok test_rjust (test_defchararray.TestMethods) ... ok test_rpartition (test_defchararray.TestMethods) ... ok test_rsplit (test_defchararray.TestMethods) ... ok test_rstrip (test_defchararray.TestMethods) ... ok test_split (test_defchararray.TestMethods) ... ok test_splitlines (test_defchararray.TestMethods) ... ok test_strip (test_defchararray.TestMethods) ... ok test_swapcase (test_defchararray.TestMethods) ... ok test_title (test_defchararray.TestMethods) ... ok test_upper (test_defchararray.TestMethods) ... ok test_add (test_defchararray.TestOperations) ... ok Ticket #856 ... ok test_mul (test_defchararray.TestOperations) ... ok test_radd (test_defchararray.TestOperations) ... ok test_rmod (test_defchararray.TestOperations) ... ok test_rmul (test_defchararray.TestOperations) ... ok test_broadcast_error (test_defchararray.TestVecString) ... ok test_invalid_args_tuple (test_defchararray.TestVecString) ... ok test_invalid_function_args (test_defchararray.TestVecString) ... ok test_invalid_result_type (test_defchararray.TestVecString) ... ok test_invalid_type_descr (test_defchararray.TestVecString) ... ok test_non_existent_method (test_defchararray.TestVecString) ... ok test_non_string_array (test_defchararray.TestVecString) ... ok test1 (test_defchararray.TestWhitespace) ... ok test_dtype (test_dtype.TestBuiltin) ... ok Only test hash runs at all. ... ok test_metadata_rejects_nondict (test_dtype.TestMetadata) ... ok test_metadata_takes_dict (test_dtype.TestMetadata) ... ok test_nested_metadata (test_dtype.TestMetadata) ... ok test_no_metadata (test_dtype.TestMetadata) ... ok test1 (test_dtype.TestMonsterType) ... ok test_different_names (test_dtype.TestRecord) ... ok test_different_titles (test_dtype.TestRecord) ... ok Test whether equivalent record dtypes hash the same. ... ok Test if an appropriate exception is raised when passing bad values to ... ok Test whether equivalent subarray dtypes hash the same. ... ok Test whether different subarray dtypes hash differently. ... ok Test some data types that are equal ... ok Test some more complicated cases that shouldn't be equal ... ok Test some simple cases that shouldn't be equal ... ok test_single_subarray (test_dtype.TestSubarray) ... ok test_einsum_errors (test_einsum.TestEinSum) ... ok test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... It hangs here..

3 2

Process-shared memory allocation per default?
by Dag Sverre Seljebotn Oct. 31, 2011

Oct. 31, 2011

This comes out of a long discussion on the Cython list. Following Mark's success with the shared memory parallelism, the question is: Where to take Cython's capabilities for parallelism further? One thing that's been up now and then is that we could basically use something like: - multiprocessing (to get rid of any GIL issues) - allocate all NumPy arrays in process-shared memory; passing NumPy arrays between processes happens by "picling views". This can be done with current NumPy by using a seperate constructor, e.g., a = sharedmem_zeros((3, 3)) However, construction of the array feels like the wrong place to make this decision. It is really when it is sent to another process the decision should be made. If all NumPy arrays are allocated in shared memory per default, one could do shared_queue.put(a.shared()) and shared() would wrap a in something that pickled a shared memory pointer rather than the data (and unpickled directly to the NumPy array). I just find this *a lot* more convenient than the tedious business of making sure the memory is allocated in the right way everywhere. Any downsides to doing this? (Additional overhead for small arrays perhaps?) - On the Cython end, parallelism could then both be supported by low-level message passing using ZeroMQ (possibly with syntax candy for sending typed messages), or with another multiprocessing backend to the current prange which requires that any memoryviews worked on are allocated in shared memory. I'm just looking for feedback here. I don't have cycles in terms of implementation; the point is that what NumPy users and devs are thinking about this could direct the further discussion of parallelism within Cython. Dag Sverre

1 0

consensus (was: NA masks in the next numpy release?)
by Nathaniel Smith Oct. 30, 2011

Oct. 30, 2011

On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant <oliphant(a)enthought.com> wrote: > I think Nathaniel and Matthew provided very > specific feedback that was helpful in understanding other perspectives of a > difficult problem. In particular, I really wanted bit-patterns > implemented. However, I also understand that Mark did quite a bit of work > and altered his original designs quite a bit in response to community > feedback. I wasn't a major part of the pull request discussion, nor did I > merge the changes, but I support Charles if he reviewed the code and felt > like it was the right thing to do. I likely would have done the same thing > rather than let Mark Wiebe's work languish. My connectivity is spotty this week, so I'll stay out of the technical discussion for now, but I want to share a story. Maybe a year ago now, Jonathan Taylor and I were debating what the best API for describing statistical models would be -- whether we wanted something like R's "formulas" (which I supported), or another approach based on sympy (his idea). To summarize, I thought his API was confusing, pointlessly complicated, and didn't actually solve the problem; he thought R-style formulas were superficially simpler but hopelessly confused and inconsistent underneath. Now, obviously, I was right and he was wrong. Well, obvious to me, anyway... ;-) But it wasn't like I could just wave a wand and make his arguments go away, no matter how annoying and wrong-headed I thought they were... I could write all the code I wanted but no-one would use it unless I could convince them it's actually the right solution, so I had to engage with him, and dig deep into his arguments. What I discovered was that (as I thought) R-style formulas *do* have a solid theoretical basis -- but (as he thought) all the existing implementations *are* broken and inconsistent! I'm still not sure I can actually convince Jonathan to go my way, but, because of his stubbornness, I had to invent a better way of handling these formulas, and so my library[1] is actually the first implementation of these things that has a rigorous theory behind it, and in the process it avoids two fundamental, decades-old bugs in R. (And I'm not sure the R folks can fix either of them at this point without breaking a ton of code, since they both have API consequences.) -- It's extremely common for healthy FOSS projects to insist on consensus for almost all decisions, where consensus means something like "every interested party has a veto"[2]. This seems counterintuitive, because if everyone's vetoing all the time, how does anything get done? The trick is that if anyone *can* veto, then vetoes turn out to actually be very rare. Everyone knows that they can't just ignore alternative points of view -- they have to engage with them if they want to get anything done. So you get buy-in on features early, and no vetoes are necessary. And by forcing people to engage with each other, like me with Jonathan, you get better designs. But what about the cost of all that code that doesn't get merged, or written, because everyone's spending all this time debating instead? Better designs are nice and all, but how does that justify letting working code languish? The greatest risk for a FOSS project is that people will ignore you. Projects and features live and die by community buy-in. Consider the "NA mask" feature right now. It works (at least the parts of it that are implemented). It's in mainline. But IIRC, Pierre said last time that he doesn't think the current design will help him improve or replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring this feature in favor of his library pandas' current hacky NA support. Members of the neuroimaging crowd are saying that the memory overhead is too high and the benefits too marginal, so they'll stick with NaNs. Together these folk a huge proportion of the this feature's target audience. So what have we actually accomplished by merging this to mainline? Are we going to be stuck supporting a feature that only a fraction of the target audience actually uses? (Maybe they're being dumb, but if people are ignoring your code for dumb reasons... they're still ignoring your code.) The consensus rule forces everyone to do the hardest and riskiest part -- building buy-in -- up front. Because you *have* to do it sooner or later, and doing it sooner doesn't just generate better designs. It drastically reduces the risk of ending up in a huge trainwreck. -- In my story at the beginning, I wished I had a magic wand to skip this annoying debate and political stuff. But giving it to me would have been a bad idea. I think that's went wrong with the NA discussion in the first place. Mark's an excellent programmer, and he tried his best to act in the good of everyone in the project -- but in the end, he did have a wand like that. He didn't have that sense that he *had* to get everyone on board (even the people who were saying dumb things), or he'd just be wasting his time. He didn't ask Pierre if the NA design would actually work for numpy.ma's purposes -- I did. You may have noticed that I do have some ideas for about how NA support should work. But my ideas aren't really the important thing. The alter-NEP was my attempt to find common ground between the different needs people were bringing up, so we could discuss whether it would work for people or not. I'm not wedded to anything in it. But this is a complicated issue with a lot of conflicting interests, and we need to find something that actually does work for everyone (or as large a subset as is practical). So here's what I think we should do: 1) I will submit a pull request backing Mark's NA work out of mainline, for now. (This is more or less done, I just need to get it onto github, see above re: connectivity) 2) I will also put together a new branch containing that work, rebased against current mainline, so it doesn't get lost. (Ditto.) 3) And we'll decide what to do with it *after* we hammer out a design that the various NA-supporting groups all find convincing. Or at least a design for some of the less controversial pieces (like the 'where=' ufunc argument?), get those merged, and then iterate incrementally. What do you all think? And in any case, thanks for reading, -- Nathaniel [1] https://github.com/charlton/charlton [2] For example, this is written into the Apache voting procedure: https://www.apache.org/foundation/voting.html (it's the "code modification" rules that are relevant). And as usual, Karl Fogel has more useful discussion: http://producingoss.com/en/consensus-democracy.html (see esp. the "When to vote" section, which is entirely about how to avoid voting)

13 72

Is distributing GPL + exception dll in the windows installer ok
by David Cournapeau Oct. 30, 2011

Oct. 30, 2011

Hi, While testing the mingw gcc 3.x -> 4.x migration, I realized that some technical requirements in gcc 4.x have potential license implications. In short, it is more difficult now than before to statically link gcc-related runtimes into numpy/scipy. I think using the DLL is safer and better, but it means the windows installers will contain GPL code. My understanding is that this is OK because the code in question is GPL + exception, meaning the usual GPL requirements only apply to those runtimes, and that's ok ? cheers, David

2 2