Mailman 3 April 2013 - NumPy-Discussion

type conversion question
by K.-Michael Aye April 19, 2013

April 19, 2013

I don't understand why sometimes a direct assignment of a new dtype is possible (but messes up the values), and why at other times a seemingly harmless upcast (in my potentially ignorant point of view) is not possible. So, maybe a direct assignment of a new dtype is actually never a good idea? (I'm asking), and one should always go the route of newarray= array(oldarray, dtype=newdtype), but why then sometimes the upcast provides an error and forbids it and sometimes not? Examples: In [140]: slope.read_center_window() In [141]: slope.data.dtype Out[141]: dtype('float32') In [142]: slope.data[1,1] Out[142]: 10.044398 In [143]: val = slope.data[1,1] In [144]: slope.data.dtype='float64' In [145]: slope.data[1,1] Out[145]: 586.98938070189865 #----- #Here, the value of data[1,1] has completely changed (and so has the rest of the array), and no error was given. # But then... #---- In [146]: val.dtype Out[146]: dtype('float32') In [147]: val Out[147]: 10.044398 In [148]: val.dtype='float64' --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-148-52a373a41cac> in <module>() ----> 1 val.dtype='float64' AttributeError: attribute 'dtype' of 'numpy.generic' objects is not writable === end of code So why is there an error in the 2nd case, but no error in the first case? Is there a logic to it? Thanks, Michael

3 3

Index Parsing redo
by Sebastian Berg April 18, 2013

April 18, 2013

Hey, so I ignored trying to redo MapIter (likely it is lobotomized at this time though). But actually got a working new index parsing (still needs cleanup, etc.). Also some of the fast paths are not yet put back. For most pure integer indices it got a bit slower, if it actually gets too much one could re-add such special cases I guess... If anyone wants to take a look, it is here: https://github.com/seberg/numpy/compare/rewrite-index-parser The tests run through fine with the exception of the matrix item setting [1] and changes in errors. Certainly many errors still need to be made indexing specific. If anyone is interested in messing with it, give me a ping for direct access. Polishing such a thing up (if deemed right) is a lot of work and I am not sure I will find the time soon. Regards, Sebastian [1] That is because subclass item setting (for non-fancy, non-scalar result and non-single integer indices) is really tmp = subclass.view(np.ndarray) tmp[index] = values` which needs to be put back in. The whole matrix indexing business probably needs some extra functionality in the core to get right I guess.

1 0

contributing to numpy
by andrew giessel April 18, 2013

April 18, 2013

First, my apologies if this isn't the right forum for this question- I looked for a dev list, but couldn't really find it. I have a small method I'd like to contribute to numpy, ideally as a method on ndarrays and a general function in the numpy namespace. I found it on a stackoverflow thread, and it is a generator that yields slices of a multidimensional array over a specified axis, which is convenient for use in list comprehensions and loops. https://gist.github.com/andrewgiessel/5400659 I've forked the numpy source and am familar with git/pull requests/etc but the code base is a bit overwhelming. I have 2 questions: 1) Is there a document which gives an overview of the numpy source and perhaps a tutorial on the best way to add methods/functions? 2) is there a current way to do this in numpy? The only iterator related stuff I found in a brief search last night was for essentially looping over all elements of an array, one by one. best, Andrew ps: please feel free to contribute to the gist! -- Andrew Giessel, PhD Department of Neurobiology, Harvard Medical School 220 Longwood Ave Boston, MA 02115 ph: 617.432.7971 email: andrew_giessel(a)hms.harvard.edu

3 3

datetime64 1970 issue
by Bob Nnamtrop April 18, 2013

April 18, 2013

I am curious if others have noticed an issue with datetime64 at the beginning of 1970. First: In [144]: (np.datetime64('1970-01-01') - np.datetime64('1969-12-31')) Out[144]: numpy.timedelta64(1,'D') OK this look fine, they are one day apart. But look at this: In [145]: (np.datetime64('1970-01-01 00') - np.datetime64('1969-12-31 00')) Out[145]: numpy.timedelta64(31,'h') Hmmm, seems like there are 7 extra hours? Am I missing something? I don't see this at any other year. This discontinuity makes it hard to use the datetime64 object without special adjustment in ones code. I assume this a bug? Thanks, Bob ps I'm using the most recent anaconda release on mac os x 10.6.8 which includes numpy 1.7.0. pss It would be most handy if datetime64 had a constructor of the form np.datetime64(year,month,day,hour,min,sec) where these inputs were numpy arrays and the output would have the same shape as the input arrays (but be of type datetime64). The hour,min,sec would be optional. Scalar inputs would be broadcast to the size of the array inputs, etc. Maybe this is a topic for another post.

8 19

Position at Brookhaven National Laboratory..
by Scott Collis April 18, 2013

April 18, 2013

Good morning Numpy list! A colleague of mine at Brookhaven National Laboratory (Long Island, NY) is hiring a "Senior Applications Analyst". One of the desired languages is Python, and as a strong collaborator with this group I can attest anyone with good python skills will do well there… Here is the description: Brookhaven National Laboratory's Atmospheric Science Division currently has a Full-Time opportunity for a Senior Applications Analyst. The major duties and responsibilities include the development of meteorological products which add value to and enhance the scientific usability of data (particularly from millimeter-wavelength radar systems) collected by the Department of Energy's Atmospheric Radiation Measurement (ARM) program. Required Knowledge, Skills and Abilities: Bachelor’s degree with at least one year experience or equivalent experience (2:1 / work:education). Expertise in high-level programming languages (including one or more of C, IDL, Python, and Matlab). Experience working in a Linux/Unix environment. Attention to detail is essential, as is the ability to work well as part of a team. Strong written and oral communication skills. Preferred Knowledge, Skills, and Abilities:A background meteorology, radar applications or another physical science. and here are the instructions: To be considered for this position, apply online at www.bnl.gov and click Jobs, then click Search Job List and apply to job #16385 Any questions give me a yell and I can put you in contact with them.. and please feel free to disseminate! Cheers, Scott

1 0

NumPy 1.7.0 w/ MKL available on PiCloud
by John Riley April 18, 2013

April 18, 2013

Hello, I'm happy to announce that NumPy 1.7.0 is now available on PiCloud [1]. Previously, NumPy 1.6.2 was the most up-to-date version available by default. While any user could always create a custom environment, and install the latest version themselves [2], we've decided to address the issue directly by having the latest NumPy available as part of the public environment "/picloud/science". Using that environment will give you access to 1.7.0, and we plan to have it track the latest version of popular scientific packages including SciPy. Note also that NumPy has been compiled with the Intel Math Kernel (MKL) library, which greatly accelerates certain operations (for ex. numpy.dot), especially on the hyper-threading-enabled f2 core. If you're unfamiliar with how to use NumPy on PiCloud, please see our documentation [3] [4]. Hope this helps! [1] http://www.picloud.com [2] http://docs.picloud.com/**environment.html<http://docs.picloud.com/environment.html> [3] http://docs.picloud.com/howto/pyscientifictools.html [4] http://docs.picloud.com/howto/primer.html Best Regards, John -- John Riley PiCloud, Inc.

1 0

Time Zones and datetime64
by Chris Barker - NOAA Federal April 18, 2013

April 18, 2013

Recent discussion has made it clear that the timezone handling in the current (numpy1.7) version of datetime64 is broken. Below is a discussion of some possible solutions, hopefully including most of the comments made on the recent thread on this list. http://mail.scipy.org/pipermail/numpy-discussion/2013-April/066038.html The intent it that with a bit more discussion (focused, in this thread at least) on the time zone issues, rather than other DateTIme64 issues, we can start a new datetime64 NEP. Background: =============================== The current version (numpy 1.7) of datetime64 appears to handle timezones in the following ways: datetime64s are assumed to be in UTC internally. Time zone translation is done on I/O -- i.e creating a new datetime64 and outputting to text format or as a datetime.datetime object. When creating a datetime64 from an ISO string, the timezone info in the string is respected. If there is no timezone info in the string, the system time zone (locale setting) is used. On output (i.e.converting to text: __str__ and __repr__) the system locale is used to set the timezone. In [9]: np.datetime64('2013-04-08T12:00:00Z') Out[9]: numpy.datetime64('2013-04-08T05:00:00-0700') In [10]: np.datetime64('2013-04-08T12:00:00') Out[10]: numpy.datetime64('2013-04-08T12:00:00-0700') However, if a datetime,datetime is used without a tzinfo object (the common case, as no tzinfo objects are provided with the python stdlib), the timezone is assumed to be UTC: In [13]: dt Out[13]: datetime.datetime(2013, 4, 8, 12, 0) In [14]: np.datetime64(dt) Out[14]: numpy.datetime64('2013-04-08T05:00:00.000000-0700') which can give some odd results, as it's different if you convert the datetime object to a iso string first: In [15]: np.datetime64(dt.isoformat()) Out[15]: numpy.datetime64('2013-04-08T12:00:00-0700') Converting from a datetime64 to a datetime object uses the UTC time (the internal representation with no offset). Issues with the current configuration: =============================== Using the locale time zone is a long standing tradition, and used by the C standard library time functions. However, it is almost always NOT what one wants in a typical numpy application. When working with Scientific (and financial) datasets, the time zone of the data at hand is likely to have nothing to do with the timezone of the computer the code is running on. Also, with cloud computing and web applications, the time zone of the machine on which the code is running is irrelevant to the user. A number of early-adopters of datetime64 have found that they have needed to wrap creating and use of datetime64 arrays to override the timezone behavior. The current implementation may be natural for some interactive use, but that's often not the case, and is particularly problematic when datetime.datetime.now() gives locale lime, but with no time zone info, so numpy actually appears to shift it. In [19]: datetime.datetime.now().isoformat() Out[19]: '2013-04-08T12:05:26.157475' In [20]: np.datetime64(datetime.datetime.now()) Out[20]: numpy.datetime64('2013-04-08T05:05:45.813027-0700') This is really ugly -- and regardless if we like what the std lib does, we need to deal with it. The python standard library datetime implementation uses "naive" datetimes by default, with the provision for an optional "tzinfo" object, so that the user can supply timezone info if desired, However, the library does not provide any tzinfo objects out of the box. This is because timezones are messy, complicated, and change over time, and the core python devs did not want to be in the position of maintaining that code. There is a third party "pytz" package (http://pytz.sourceforge.net/) that provides a pretty complete implementation of time zone handling for those that need it. Note also that in the current implementation, The busday functions just operate on datetime64[D]. There is no timezone interaction there -- which makes it very hard for them to be useful, as it's a bit tricky to make sure your datetime64 arrays are in the correct time zone for your application. In fact, they are assuming that datetime64 is time zone naive, even though the I/O functions assume locale time. Proposed Alternatives: ====================== Principles: ------------------ * Partial time zone handling is probably worse than none. * The library should never apply the locale timezone (or any other) unless explicitly requested by the user. * It should be possible (and easy) to use "naive" datetime64 arrays. 1) A naive datetime64: ==================== This would follow, more or less, the python stdlib datetime approach (with no tzinfo object) -- ignore timezones entirely. This model leaves all time zone handling up to the user. In general, when working with this method, applications either: use UTC everywhere, or use "local time" everywhere, where local time is simply "all data is in the same time zone" and it's up to the user (or lib code) to make sure that's correct. Issues: ------------------ The main issue with a naive datetime64 is what to do on creation if time zone information is supplied (i.e in a ISO 8601 string, or datetime object with non-None tzinfo). Options include: - ignore the TZ info - raise an exception if there is a TZ adjustment (other than UTC, 00:00, or Z) I propose that we raise an exception, unless there is a way to pass an optional flag in to request timezone conversion. note that the stdlib datetime package does not provide an ISO8601 parsing function, so it has ignored the issue. There is also the issue of what to provide on output/conversion. I propose: - a datetime object with no tzinof - ISO8601 with no tz adjustment np.datetime_as_string() could be used with options to allow the user to request a time zone offset. 1) UTC-only ==================== This would be very similar to a naive datetime64 -- there are no timezone adjustments with pure UTC -- and would be similar to the current implementation, except for I/O: On conversion to datetime64, time zone offset would be respected, if it exists in the ISO8601 string or the datetime object has a tzinfo attribute. The value would be stored in UTC. If there is no timezone info in the input string or datetime objects, UTC is assumed. On output -- UTC is used, no offset computed. Issues: ------------------ The ISO string produced on output would logically contain the "Z" flag to indicate UTC. This may confuse some apps that really expect a naive datetime. If there were a way to pass in a flag to create ISO strings indicating the time zone, that would be perfect, probably using np.datetime_as_string() 3) Optional time zone support ========================== This would follow the standard library approach -- provide a hook for a tzinfo object -- and if there, handle properly. This would allow one to mix and match datetime64s that are in different time zones, etc. issues: ---------------- The biggest issue is that to be useful, you'd need a comprehensive tzinfo package. pytz provides one, but then you'd need to go through the python layer for every item in an array -- killing performance. However, perhaps that would be worth it for those that need it, and folks that need performance could use naive datetime64s. There apparently is also a datetime library in boost that has a nice timezone object which could be used as inspiration for an equivalent in NumPy. That could be a lot of work, though. 3) Full time zone support ========================== This would be similar to the above, except that every datetime64 array would be required to carry time zone info. This would probably be reasonable, as one could use UTC everywhere if you wanted the simple case. But it would require that a comprehensive tzinfo package be included with numpy -- likely something we don't want to have to maintain (even if someone wants to built it in the first place) issues: ----------- We would still want ways to input/output naive datetimes -- some app simply don't want to deal with all this! As I (Chris Barker) am not in a postion to implement anything, I advocate the simplest possible approach -- which I think is Naive datetime and/or UTC only. But if people want more, and someone wants to implement it, great! Please add your comment, and maybe we'll get a NEP together. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

9 17

Gsoc : Performance parity between numpy arrays and Python scalars
by Arink Verma April 17, 2013

April 17, 2013

Hello everyone I am Arink, computer science student and open source enthusiastic. This year I am interested to work on project "Performance parity between numpy arrays and Python scalars"[1]. I tried to adobt rald's work on numpy1.7[2] (which was done for numpy1.6 [3]). Till now by avoiding a) the uncessary Checking for floating point errors which is slow, b) unnecessarily Creation / destruction of scalar array types I am getting the speedup by ~ 1.05 times, which marginal offcourse. As in project's describtion it is mention that ufunc look up code is slow and inefficient. Few questions 1. Does it has to check every single data type possible until if finds the best match for the data that the operation is being performed on, or is there better way to find the best possible match? 2. If yes, so where are bottle-necks? Is the checks for proper data types are very expensive? [1]http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas [2]https://github.com/arinkverma/numpy/compare/master...gsoc_performance [3]http://article.gmane.org/gmane.comp.python.numeric.general/52480 -- Arink Computer Science and Engineering Indian Institute of Technology Ropar www.arinkverma.in

2 1

Adding gufuncs
by Charles R Harris April 17, 2013

April 17, 2013

There is a pull request <https://github.com/numpy/numpy/pull/3220>for work adding linear algebra support as generalized ufuncs. The result is that many of the linear algebra routines can now be applied to stacks of matrices. Another new feature is support for float32 versions of the routines. Some work has also gone into porting the current linalg package to use the new routines. The work isn't finished, the new and old libraries {blas, lapack}_lite libraries should probably be united and the error handling could maybe use more polish, but I'm inclined to put the PR in at this point. Although some things may break, I think it needs to be out there to gather feedback and testing. Thoughts? Chuck

2 1

Finding the same value in List
by Happyman April 17, 2013

April 17, 2013

Hello, I have had encountered some problem while I was trying to create the following code which finds number of the same values and indexes in an array or list. Here is the code: y = [ 1, 12, 3, 3, 5, 1, 1, 34, 0, 0, 1, 5] OR y = array( [ 1, 12, 3, 3, 5, 1, 1, 34, 0, 0, 1, 5 ] ) b = [ [ item for item in range( len(y) ) if y[ item ] == y[ j ] ] for j in range(0, len( y ) ) ] answer: [ [ 0, 5, 6, 10], [1], [2, 3], [2, 3], [4, 11], [0, 5, 6, 10], [0, 5, 6, 10], [7], [8, 9], [8, 9], [0, 5, 6, 10], [4, 11] ] The result I want to get is ,not that shown above as an answer, that I want to calculate the number of the same values and their indexes in not repeated way as well. For example, '1' - 4, index: '0, 5, 6, 10' '12' - 1, index: '1' '3' - 2, index: '2, 3' '5' - 2, index: '4, 11' '34' - 1, index: '7' '0' - 2, index: '8, 9' Any answer would be appreciated.. --

4 8