Mailman 3 September 2016 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

automatically avoiding temporary arrays
by Julian Taylor 27 Feb '17

27 Feb '17

hi, Temporary arrays generated in expressions are expensive as the imply extra memory bandwidth which is the bottleneck in most numpy operations. For example: r = a + b + c creates the b + c temporary and then adds a to it. This can be rewritten to be more efficient using inplace operations: r = b + c r += a This saves some memory bandwidth and can speedup the operation by 50% for very large arrays or even more if the inplace operation allows it to be completed completely in the cpu cache. The problem is that inplace operations are a lot less readable so they are often only used in well optimized code. But due to pythons refcounting semantics we can actually do some inplace conversions transparently. If an operand in python has a reference count of one it must be a temporary so we can use it as the destination array. CPython itself does this optimization for string concatenations. In numpy we have the issue that we can be called from the C-API directly where the reference count may be one for other reasons. To solve this we can check the backtrace until the python frame evaluation function. If there are only numpy and python functions in between that and our entry point we should be able to elide the temporary. This PR implements this: https://github.com/numpy/numpy/pull/7997 It currently only supports Linux with glibc (which has reliable backtraces via unwinding) and maybe MacOS depending on how good their backtrace is. On windows the backtrace APIs are different and I don't know them but in theory it could also be done there. A problem is that checking the backtrace is quite expensive, so should only be enabled when the involved arrays are large enough for it to be worthwhile. In my testing this seems to be around 180-300KiB sized arrays, basically where they start spilling out of the CPU L2 cache. I made a little crappy benchmark script to test this cutoff in this branch: https://github.com/juliantaylor/numpy/tree/elide-bench If you are interested you can run it with: python setup.py build_ext -j 4 --inplace ipython --profile=null check.ipy At the end it will plot the ratio between elided and non-elided runtime. It should get larger than one around 180KiB on most cpus. If no one points out some flaw in the approach, I'm hoping to get this into the next numpy version. cheers, Julian

11 22

Added atleast_nd, request for clarification/cleanup of atleast_3d
by Joseph Fox-Rabinovitz 27 Oct '16

27 Oct '16

Hi, I have generalized np.atleast_1d, np.atleast_2d, np.atleast_3d with a function np.atleast_nd in PR#7804 (https://github.com/numpy/numpy/pull/7804). As a result of this PR, I have a couple of questions about `np.atleast_3d`. `np.atleast_3d` appears to do something weird with the dimensions: If the input is 1D, it prepends and appends a size-1 dimension. If the input is 2D, it appends a size-1 dimension. This is inconsistent with `np.atleast_2d`, which always prepends (as does `np.atleast_nd`). - Is there any reason for this behavior? - Can it be cleaned up (e.g., by reimplementing `np.atleast_3d` in terms of `np.atleast_nd`, which is actually much simpler)? This would be a slight API change since the output would not be exactly the same. Thanks, -Joe

11 25

Vendorize tempita
by Charles R Harris 01 Oct '16

01 Oct '16

Hi All, There is a PR <https://github.com/numpy/numpy/pull/8096> to vendorize tempita. This removes tempita as a dependency and simplifies some things. Feedback on this step is welcome. One question is whether the package should be renamed to something like `npy_tempita`, as otherwise installed tempita, if any has priority. Thoughts? Chuck

5 10

ANN: SfePy 2016.3
by Robert Cimrman 30 Sep '16

30 Sep '16

I am pleased to announce release 2016.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: http://groups.google.com/group/sfepy-devel Git (source) repository, issue tracker: http://github.com/sfepy/sfepy Highlights of this release -------------------------- - Python 3 support - testing with Travis CI - new classes for homogenized coefficients - using argparse instead of optparse For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Cheers, Robert Cimrman --- Contributors to this release in alphabetical order: Robert Cimrman Jan Heczko Thomas Kluyver Vladimir Lukes

1 0

Using library-specific headers
by Pavlyk, Oleksandr 29 Sep '16

29 Sep '16

Suppose I would like to take advantage of some functions from MKL in numpy C source code, which would require to use #include "mkl.h" Ideally this include line must not break the build of numpy when MKL is not present, so my initial approach was to use #if defined(SCIPY_MKL_H) #include "mkl.h" #endif Unfortunately, this did not work when building with gcc on a machine where MKL is present on default LD_LIBRARY_PATH, because then the distutils code was setting SCIPY_MKL_H preprocessor variable, even though mkl headers are not on the C_INCLUDE_PATH. What is the preferred solution to include an external library header to ensure that code-base continues to build in most common cases? One approach I can think of is to set a preprocessor variable, say HAVE_MKL_HEADERS in numpy/core/includes/numpy/config.h depending on an outcome of building of a simple _configtest.c using config.try_compile(), like it is done in numpy/core/setup.py Is there a simpler, or a better way? Thank you, Oleksandr

4 4

PR 8053 np.random.multinomial tolerance param
by Alex Beloi 27 Sep '16

27 Sep '16

Hello, Pull Request: https://github.com/numpy/numpy/pull/8053 I would like to expose a tolerance parameter for the function numpy.random.multinomial. The function `multinomial(n, pvals, size=None)` correctly raises exception when `sum(pvals) > 1 + 1e-12` as these values should sum to 1. However, other libraries often cannot or do not guarantee such level of precision. Specifically, I have encountered issues with tensorflow function tf.nn.softmax, which is expected to output a tensor whose values sum to 1, but often with precision of only 1e-8. I propose to expose the `1e-12` tolerance to a non-negative float parameter with default value `1e-12`. Alex

2 2

Fwd: [NumFOCUS Projects] Job Posting
by Ralf Gommers 27 Sep '16

27 Sep '16

Hi all, The below job posting may be of interest. It's actually a key role for a quite exciting program, could really have an impact on a large set of scientific computing projects. Cheers, Ralf ---------- Forwarded message ---------- From: Leah Silen <leah(a)numfocus.org> Date: Thu, Sep 22, 2016 at 6:05 AM Subject: [NumFOCUS Projects] Job Posting To: projects(a)numfocus.org The last project email contained information on both the Summit and the Sustainability Program Job Posting. In case you missed it, we wanted to resend the job posting info. Please share with those in your circles. Best, Leah NumFOCUS Projects Director NumFOCUS is seeking to hire a full-time Projects Director to develop and run a sustainability incubator program for NumFOCUS fiscally sponsored open source projects. This is the first program of its kind, with opportunity for enormous impact on the open source ecosystem. The Projects Director will work with NumFOCUS fiscally sponsored Open Source projects to develop appropriate business development strategies and opportunities to engage with industry. The learnings from this program will be public, meaning it has the potential to change how all open source projects are managed. NumFOCUS is 501(c)3 whose mission is to promote sustainable high-level programming languages, open code development, and reproducible scientific research. They accomplish this mission through their educational programs and events as well as through fiscal sponsorship of open source data science projects. They aim to increase collaboration and communication within the scientific computing community. The Projects Director position is initially funded for 2 years through a grant from the Sloan Foundation. The incumbent will be hired as an employee of NumFOCUS. Review of applications will begin October 10th, and the position will remain open until filled. Job description The Projects Director will lead efforts to identify skills and perspectives that Open Source Project leads need to establish sustainability models and seek opportunities for funding and will consult with NumFOCUS projects and disseminate outcomes. Responsibilities Include: - Organize a ‘business bootcamp’ workshop, coach project leads and lead an Advisory Board for these business development efforts - Develop business and financial plans, improve communication, build industry relationships, and broadly develop strategies for fostering these relationships. - Lead efforts to identify skills and perspectives that Open Source Project leads need to establish sustainability models and seek opportunities for funding - Using materials from workshops and other resources, develop and disseminate an Open Source Nurturing and Actionable Practices Kit (OSNAP Kit) to provide guidance to open source projects outside of NumFOCUS - Establish assessment strategies to measure impact and help NumFOCUS open-source projects develop industry relationships Qualifications As this role provides a business, operations and marketing perspective to Open Source project leads, the candidate should have: - Preferred: 4 years experience in business development, operations and/or marketing - Strong oral and written communication skills - Teaching experience of some capacity, e.g. teaching at universities or creating and running workshops - Demonstrated ability to interact with a broad range of people from technical leads to industry leaders - Experience working with open communities, e.g. open source software, hardware or makers, - A rich network of experts in business and technology to draw ideas and collaboration from. Salary NumFOCUS offers a competitive salary commensurate with experience and a comprehensive benefits package. Applications To apply, please submit your resume/CV and cover letter (up to two pages) to NumFOCUS at: projects_director(a)numfocus.org See the posting at: http://www.numfocus.org/blog/ projects-director-job-posting --- Leah Silen Executive Director, NumFOCUS leah(a)numfocus.org 512-222-5449 <http://www.numfocus.org/> <https://twitter.com/NumFOCUS> <https://www.facebook.com/NumFOCUS/> -- You received this message because you are subscribed to the Google Groups "Fiscally Sponsored Project Representatives" group. To unsubscribe from this group and stop receiving emails from it, send an email to projects+unsubscribe(a)numfocus.org. To post to this group, send email to projects(a)numfocus.org. Visit this group at https://groups.google.com/a/numfocus.org/group/projects/ . To view this discussion on the web visit https://groups.google.com/a/ numfocus.org/d/msgid/projects/CALWv6u%2BkwhK_RrFq7bCfMq9mB36o5FjGJjBBGo0yJ1 jzPLsmvw%40mail.gmail.com <https://groups.google.com/a/numfocus.org/d/msgid/projects/CALWv6u%2BkwhK_Rr…> .

1 0

Parsing a file with dates to datetim64
by Florian Lindner 26 Sep '16

26 Sep '16

Hey, I have a file: ;;;;;;;;;Eintrittsdatum;;; ;;;;;;;;;04.03.16;;10,00 €;genehmigt ;;;;;;;;;04.03.16;;10,00 €;genehmigt which I try to parse using def dateToNumpyDate(s): s = s.decode("utf-8") ret = datetime.datetime.strptime(s, "%d.%m.%y").isoformat() return ret def generateMembers(): members = np.genfromtxt("test_CSC_Mitglieder.csv", dtype = { "names" : ["EntryDate"], "formats" : ['datetime64[D]'] }, converters = { 9 : dateToNumpyDate }, skip_header = 1, delimiter = ";", usecols = (9)) count = members.shape[0] y = np.linspace(1, count, count) print(members) print(members.dtype) plt.plot(members["EntryDate"], y) plt.show() but the datatype was ignored homehow, <U19 instead of the datetime64 ['2016-03-04T00:00:00' '2016-03-04T00:00:00'] <U19 CSC.py:76: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future plt.plot(members["EntryDate"], y) Traceback (most recent call last): File "CSC.py", line 82, in <module> generateMembers() File "CSC.py", line 76, in generateMembers plt.plot(members["EntryDate"], y) IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices I also tried to print the return of dateToNumpbyData: 2016-03-04T00:00:00 2016-03-04T00:00:00 2016-03-04T00:00:00 Is there a problem with the dtype argument? Thanks, Florian

1 0

NumPy-Discussion September 2016