Mailman 3 July 2012 - NumPy-Discussion

Problem w/ Win installer
by David Goldsmith July 19, 2012

July 19, 2012

Hi, folks! Having a problem w/ the Windows installer; first, the "back-story": I have both Python 2.7 and 3.2 installed. When I run the installer and click next on the first dialog, I get the message that I need Python 2.7, which was not found in my registry. I ran regedit and searched for Python and get multiple hits on both Python 2.7 and 3.2. So, precisely which registry key has to have the value Python 2.7 for the installer to find it? Thanks! OlyDLG

2 1

indexed increment (was Re: Pull Requests I'm planning to merge)
by Nathaniel Smith July 19, 2012

July 19, 2012

On Thu, Jul 19, 2012 at 6:04 AM, Frédéric Bastien <nouiz(a)nouiz.org> wrote: > On Tue, Jul 17, 2012 at 10:20 PM, Travis Oliphant <travis(a)continuum.io> wrote: >> I will hold off on this one, but only because you raised your objection to -1 (instead of -0.5). But, I disagree very strongly with you that it is "sub-optimal" in the context of NumPy. I actually think it is a very reasonable solution. He has re-used the MapIter code correctly (which is the only code that *does* fancy indexing). The author has done a lot of work to support multiple data-types and satisfy an often-requested feature in a very reasonable way. Your objection seems to be that you would prefer that it were a method on ufuncs. I don't think this will work without a *major* refactor. I'm not sure it's even worth it then either. > > Due to Travis comments, I expect the "better" solution won't be done > in the short time. So blocking a feature requeted frequently because > we could do sometimes in an unknow futur something better is not a > good idea. (if I get it wrong on the expectation, please, correct me!) Really I'm making two points: 1- this design smells just "off" enough that we should discuss it to figure out whether or not it's actually the best way forward. 2- such discussions are too important to just skip whenever a release is imminent. there will be another release... > Also, if we decide to change the interface, we could do it for NumPy > 2, so I don't see a need for an infinit time support of this API. The "NumPy 2" idea is not very realistic IMHO, and shouldn't be used as an excuse for being sloppy. We're never going to be able to go through and make all the APIs perfect in one release cycle. Especially if we keep deferring all hard questions until then. > So in conclusion, can we say that when John will have make is PR work > with all dtype, we merge it? Well, we'll come up with something :-). (Assuming that people stay interested, which seems likely since this is such a common problem people have.) But I'm not sure what. Making the PR work with all dtypes is exactly what Travis is arguing is too much work. > Also, I understood that it won't be > included in NumPy 1.7. Is that right? Yes. -n

2 1

Pull Requests I'm planning to merge
by Travis Oliphant July 19, 2012

July 19, 2012

I would like to merge the following pull requests sometime today: * 326 -- inplace increment function * 325 -- prefer gfortran on OSX and Linux * 192 -- meshgrid enhancements * 327 -- restore commas and update C-API doc a bit * 352 -- simplifying case for insert and adding tests (#2028) * 350 -- return view for subset of fields. Also, I would like to do * 356 -- with Charles's edited fix in the comments (Bug #808) Are there concerns with the above list, or do others have PR they would especially like to see merged... Thanks, -Travis

4 4

Proposed change in genfromtxt(..., comments='#', names=True) behaviour
by Paul Natsuo Kishimoto July 18, 2012

July 18, 2012

Hello everyone, I am a longtime NumPy user, and I just filed my first contribution to the code as pull request to fix what I felt was a bug in the behaviour of genfromtxt() https://github.com/numpy/numpy/pull/351 It turns out this alters existing behaviour that some people may depend on, so I was encouraged to raise the issue on this list to see what the consensus was. This behaviour happens in the specific situation where: * Comments are used in the file (the default comment character is '#', which I'll use here), AND * The kwarg names=True is given. In this case, genfromtxt() is supposed to read an initial row containing the names of the columns and return an array with a structured dtype. Currently, these options work with a file like (Example #1): # gender age weight M 21 72.100000 F 35 58.330000 M 33 21.99 …but NOT with a file like (Example #2): # here is a general file comment # it is spread over multiple lines gender age weight M 21 72.100000 F 35 58.330000 M 33 21.99 …genfromtxt() believes the column names are 'here', 'is', 'a', etc., and thinks all of the columns are strings because 'gender', 'age' and 'weight' are not numbers. This is because genfromtxt() (after skipping a number of lines as specified in the optional kwarg skip_header) will use the *first* line it encounters to produce column names. If that line contains a comment character, genfromtxt() discards everything *up to and including* the comment character, and tries to use the content *after* the comment character as headers (Example 3): gender age weight # wrong column names M 21 72.100000 F 35 58.330000 M 33 21.99 …the resulting column names are 'wrong', 'column' and 'names'. My proposed change was that, if the first (or any subsequent) line contains a comment character, it should be treated as an *actual comment*, and discarded along with anything that follows it on the line. In Example 2, the result would be that the first two lines appear empty (no text before '#'), and the third line ("gender age weight") is used for column names. In Example 3, the result would be that "gender age weight" is used for column names while "# wrong column names" is ignored. BUT! In Example 1, the result would be that the first line appears empty, and "M 21 72.100000" are used for column names. In other words, this change would do away with the previous behaviour where the very first commented line was (magically?) treated not as a comment but instead as column headers. This might break some existing code. On the positive side, it would allow the user to be more liberal with the format of input files (Example 4): # here is a general file comment # the columns in this table are gender age weight # here is a comment on the header line # following this line are the data M 21 72.100000 F 35 58.330000 # here is a comment on a data line M 33 21.99 I feel that this is a better/more flexible behaviour for genfromtxt(), but—as stated—I am interested in your thoughts. Cheers, -- Paul Natsuo Kishimoto SM candidate, Technology & Policy Program (2012) Research assistant, http://globalchange.mit.edu https://paul.kishimoto.name +1 617 302 6105

6 17

"Symbol table not found" compiling numpy from git repository on Windows
by John Salvatier July 18, 2012

July 18, 2012

Hello, I'm trying to compile numpy on Windows 7 using the command: "python setup.py config --compiler=mingw32 build" but I get an error about a symbol table not found. Anyone know how to work around this or what to look into? building library "npymath" sources Building msvcr library: "C:\Python26\libs\libmsvcr90.a" (from C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll) objdump.exe: C:\Windows\winsxs\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_750b37ff97f4f68b\msvcr90.dll: File format not recognized Traceback (most recent call last): File "setup.py", line 214, in <module> setup_package() File "setup.py", line 207, in setup_package configuration=configuration ) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\core.py", line 186, in setup return old_setup(**new_attr) File "C:\Python26\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build.py", line 37, in run old_build.run(self) File "C:\Python26\lib\distutils\command\build.py", line 134, in run self.run_command(cmd_name) File "C:\Python26\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 152, in run self.build_sources() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy\core\setup.py", line 646, in get_mathlib_info st = config_cmd.try_link('int main(void) { return 0;}') File "C:\Python26\lib\distutils\command\config.py", line 257, in try_link self._check_compiler() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\command\config.py", line 45, in _check_compiler old_config._check_compiler(self) File "C:\Python26\lib\distutils\command\config.py", line 107, in _check_compiler dry_run=self.dry_run, force=1) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\ccompiler.py", line 560, in new_compiler compiler = klass(None, dry_run, force) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 94, in __init__ msvcr_success = build_msvcr_library() File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 362, in build_msvcr_library generate_def(dll_file, def_file) File "C:\Users\jsalvatier\workspace\numpy\numpy\distutils\mingw32ccompiler.py", line 282, in generate_def raise ValueError("Symbol table not found") ValueError: Symbol table not found Thank you, John

4 5

numpy.complex
by Molinaro Céline July 18, 2012

July 18, 2012

Hello, In [2]: numpy.real(arange(3)) Out[2]: array([0, 1, 2]) In [3]: numpy.complex(arange(3)) TypeError: only length-1 arrays can be converted to Python scalars Are there any reasons why numpy.complex doesn't work on arrays? Should it be bug reported? Thanks for your help C. Molinaro

3 2

Lazy imports again
by Charles R Harris July 18, 2012

July 18, 2012

Hi All, Working lazy imports would be useful to have. Ralf is opposed to the idea because it caused all sorts of problems on different platforms when it was tried in scipy. I thought I'd open the topic for discussion so that folks who had various problems/solutions could offer input and the common experience could be collected in one place. Perhaps there is a solution that actually works. Ideas? Chuck

7 9

Would a patch with a function for incrementing an array with advanced indexing be accepted?
by John Salvatier July 17, 2012

July 17, 2012

Hello, If you increment an array using advanced indexing and have repeated indexes, the array doesn't get repeatedly incremented, http://comments.gmane.org/gmane.comp.python.numeric.general/50291. I wrote a C function that does incrementing with repeated indexes correctly. The branch is here (https://github.com/jsalvatier/numpy see the last two commits). Would a patch with a cleaned up version of a function like this be accepted into numpy? I'm not experienced writing numpy C code so I'm sure it still needs improvement. If you compile and install that branch, you can test the code using: from numpy import * from numpy.core.multiarray import index_increment a = arange(12).reshape((3,4)) index = ([1,1,2,0], [0,0,2,3]) vals = [50,50., 30.,16] b = index_increment(a, index, vals) print b """ should print out: [[ 0. 1. 2. 19.] [ 104. 5. 6. 7.] [ 8. 9. 40. 11.]] """ Cheers, John

3 11

py3 datetime woes (was Re: Code Freeze for NumPy 1.7)
by Nathaniel Smith July 17, 2012

July 17, 2012

On Sun, Jul 15, 2012 at 5:32 PM, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > > > On Sun, Jul 15, 2012 at 5:57 PM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> On Sun, Jul 15, 2012 at 1:08 PM, Ralf Gommers >> <ralf.gommers(a)googlemail.com> wrote: >> > >> > >> > On Sun, Jul 15, 2012 at 12:45 AM, Travis Oliphant <travis(a)continuum.io> >> > wrote: >> >> >> >> >> >> Hey all, >> >> >> >> We are nearing a code-freeze for NumPy 1.7. Are there any last-minute >> >> changes people are wanting to push into NumPy 1.7? We should discuss >> >> them >> >> as soon as possible. >> >> >> >> I'm proposing a code-freeze at midnight UTC on July 18th (7:00pm CDT on >> >> July 17th). This will allow the creation of beta releases of NumPy on >> >> the >> >> 18th of July. This is a few days later than originally hoped for --- >> >> largely >> >> due to unexpected travel schedules of Ondrej and I, but it does give >> >> people >> >> a few more days to get patches in. Of course, we will be able to apply >> >> bug-fixes to the 1.7.x branch once the tag is made. >> > >> > >> > What about the tickets still open for 1.7.0 >> > (http://projects.scipy.org/numpy/report/3)? There are a few important >> > ones >> > left. >> > >> > These I would consider blockers: >> > - #2108 Datetime failures with MinGW >> >> Is there a description anywhere of what the problem actually is here? >> I looked at the ticket, which referred to a PR, and it's hard to work >> out from the PR discussion what the actual remaining test failures are >> -- and there definitely doesn't seem to be any description of the >> underlying problem. (Something about working 64-bit time_t on windows >> being difficult depending on the compiler used?) > > There's a lot more discussion on > http://projects.scipy.org/numpy/ticket/1909 > https://github.com/numpy/numpy/pull/156 > https://github.com/numpy/numpy/pull/161. > > The issue is that for MinGW 3.x some _s / _t functions seem to be missing. > And we don't yet support MinGW 4.x. > > Current issues can be seen from the last test log on our Windows XP buildbot > (June 29, > http://buildbot.scipy.org/builders/Windows_XP_x86/builds/1124/steps/shell_1…): > > ====================================================================== > ERROR: test_datetime_arange (test_datetime.TestDateTime) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "C:\buildbot\numpy\b11\numpy-install\Lib\site-packages\numpy\core\tests\test_datetime.py", > line 1351, in test_datetime_arange > assert_raises(ValueError, np.arange, np.datetime64('today'), > OSError: Failed to use '_localtime64_s' to convert to a local time ...I don't understand how this is even building if the functions are missing. Well, anyway. MSVC provides both _s and regular versions of localtime and friends. Mingw only provides the regular version. It looks like there are two difference between the regular and _s versions of these functions: 1) The _s version reports errors by calling the "invalid parameter handler" instead of just returning an error code. (I.e., by default, if you try to work with a date that's out of range, then your program will just abort(). Very friendly.)[1] 2) MSVC whines if you use the regular version Basically AFAICT this is an interface designed to make it easier for lousy programmers to write broken code that will still limp along more successfully than it otherwise would. There's a legitimate role for such interfaces, but I'm not sure numpy is it -- I think we can write actual error handling code? The classic localtime() interface AFAICT works perfectly -- it's even threadsafe, unlike most unix versions [2]. I think the best solution here is to just switch to using localtime() everywhere, and tell MSVC to shut up by using a #pragma. Here's an example of how to do this taken from the Boost developer guidelines [3]: #if defined(_MSC_VER) #pragma warning(push) // Save warning settings. #pragma warning(disable : 4996) // Disable deprecated localtime/gmtime warning. #endif ... #if defined(_MSC_VER) #pragma warning(pop) // Restore warnings to previous state. #endif (Yes, Boost's canonical example of how you control warnings on MSVC is specifically for letting you use localtime() in piece -- I didn't add that. Of course this could be hidden in a macro or helper function... npy_localtime() or something.) Or alternatively one can just #define _CRT_SECURE_NO_DEPRECATE to kill this warning in general. The Boost developer's general advice on this particular warning is "Unless you strongly believe that the 'secure' versions are useful, suppress."[3] [1] http://msdn.microsoft.com/en-us/library/a442x3ye%28v=vs.80%29.aspx , and note the "community content" at the bottom. [2] http://old.nabble.com/Re%3A-Need-localtime_s-p13088250.html [3] https://svn.boost.org/trac/boost/wiki/Guidelines/WarningsGuidelines -n

2 1

Code Freeze for NumPy 1.7
by Travis Oliphant July 16, 2012

July 16, 2012

Hey all, We are nearing a code-freeze for NumPy 1.7. Are there any last-minute changes people are wanting to push into NumPy 1.7? We should discuss them as soon as possible. I'm proposing a code-freeze at midnight UTC on July 18th (7:00pm CDT on July 17th). This will allow the creation of beta releases of NumPy on the 18th of July. This is a few days later than originally hoped for --- largely due to unexpected travel schedules of Ondrej and I, but it does give people a few more days to get patches in. Of course, we will be able to apply bug-fixes to the 1.7.x branch once the tag is made. If you have a pull-request that is not yet applied and would like to discuss it for inclusion, the time to do it is now. Best, -Travis

13 23