Jarrod Millman wrote:

Hello,

This is a reminder that 1.1.1rc1 will be tagged tonight. Chuck is planning to spend some time today fixing a few final bugs on the 1.1.x branch. If anyone else is planning to commit anything to the 1.1.x branch today, please let me know immediately. Obviously now is not the time to commit anything to the branch that could break anything, so please be extremely careful if you have to touch the branch.

Once the release is tagged, Chris and David will create binary installers for both Windows and Mac. Hopefully, this will give us an opportunity to have much more widespread testing before releasing 1.1.1 final at the end of the month.

Can I get anyone to look at this patch for loadtext()? I was trying to use loadtxt() today to read in some text data, and I had a problem when I specified a dtype that only contained as many elements as in columns in usecols. The example below shows the problem: import numpy as np import StringIO data = '''STID RELH TAIR JOE 70.1 25.3 BOB 60.5 27.9 ''' f = StringIO.StringIO(data) names = ['stid', 'temp'] dtypes = ['S4', 'f8'] arr = np.loadtxt(f, usecols=(0,2),dtype=zip(names,dtypes), skiprows=1) With current 1.1 (and SVN head), this yields: IndexError Traceback (most recent call last) /home/rmay/<ipython console> in <module>() /usr/lib64/python2.5/site-packages/numpy/lib/io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 309 for j in xrange(len(vals))] 310 if usecols is not None: --> 311 row = [converterseq[j](vals[j]) for j in usecols] 312 else: 313 row = [converterseq[j](val) for j,val in enumerate(vals)] IndexError: list index out of range ----------------------------------------- I've added a patch that checks for usecols, and if present, correctly creates the converters dictionary to map each specified column with converter for the corresponding field in the dtype. With the attached patch, this works fine:

arr array([('JOE', 25.300000000000001), ('BOB', 27.899999999999999)], dtype=[('stid', '|S4'), ('temp', '

Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma

Thanks. I wouldn't have ordinarily pushed so much, but I wanted to hit the bugfix release. Ryan David Huard wrote:

Ryan, I committed your patch to the trunk and added a test for it from your failing example.

Jarrod, though I'm also wary to touch the branch so late, the patch is minor and I don't see how it could break something that was not already broken.

David

2008/7/20 Ryan May mailto:rmay31@gmail.com>:

Jarrod Millman wrote:

Hello,

This is a reminder that 1.1.1rc1 will be tagged tonight. Chuck is planning to spend some time today fixing a few final bugs on the 1.1.x branch. If anyone else is planning to commit anything to the 1.1.x branch today, please let me know immediately. Obviously now is not the time to commit anything to the branch that could break anything, so please be extremely careful if you have to touch the branch.

Once the release is tagged, Chris and David will create binary installers for both Windows and Mac. Hopefully, this will give us an opportunity to have much more widespread testing before releasing 1.1.1 final at the end of the month.

Can I get anyone to look at this patch for loadtext()?

I was trying to use loadtxt() today to read in some text data, and I had a problem when I specified a dtype that only contained as many elements as in columns in usecols. The example below shows the problem:

import numpy as np import StringIO data = '''STID RELH TAIR JOE 70.1 25.3 BOB 60.5 27.9 ''' f = StringIO.StringIO(data) names = ['stid', 'temp'] dtypes = ['S4', 'f8'] arr = np.loadtxt(f, usecols=(0,2),dtype=zip(names,dtypes), skiprows=1)

With current 1.1 (and SVN head), this yields:

IndexError Traceback (most recent call last)

/home/rmay/<ipython console> in <module>()

/usr/lib64/python2.5/site-packages/numpy/lib/io.pyc in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack) 309 for j in xrange(len(vals))] 310 if usecols is not None: --> 311 row = [converterseq[j](vals[j]) for j in usecols] 312 else: 313 row = [converterseq[j](val) for j,val in enumerate(vals)]

IndexError: list index out of range -----------------------------------------

I've added a patch that checks for usecols, and if present, correctly creates the converters dictionary to map each specified column with converter for the corresponding field in the dtype. With the attached patch, this works fine:

>arr array([('JOE', 25.300000000000001), ('BOB', 27.899999999999999)], dtype=[('stid', '|S4'), ('temp', '

Thanks, Ryan

-- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma

-- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org mailto:Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

------------------------------------------------------------------------

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma

I'm in favor of putting it in. Pierre has also made some fixes to masked arrays. I think that is about the end of it for 1.1.1. However, if anyone is running Python 2.3 it would be helpful if you could test the release candidate as the buildbots are all 2.4 or 2.5.

Oh yes. I don't have access to Python 2.3, so I haven't been able to check whether one dictionary update in MaskedArray.reshape would work properly (on 1.1.x, it does on 1.2).

On Mon, Jul 21, 2008 at 6:59 PM, Charles R Harris wrote:

I'm in favor of putting it in. Pierre has also made some fixes to masked arrays. I think that is about the end of it for 1.1.1. However, if anyone is running Python 2.3 it would be helpful if you could test the release candidate as the buildbots are all 2.4 or 2.5.

Hey Chuck, Let's commit the changes to the 1.1.x branch and I can tag a 1.1.1rc2 tomorrow night (in about 24 hours). Is that enough time for everyone to get there final fixes in? I will ask Chris and David to wait until I tag 1.1.1rc2 to create new binaries (assuming they haven't done so all ready). Thanks, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

On Tuesday 22 July 2008 01:10:30 Charles R Harris wrote:

Hmm... 70 errors. Pretty much all of them of this sort:

NameError: global name 'actual_dtype' is not defined

OK, that's a problem with numpy.ma.testutils. r5496 should fix that

Pierre, I suggest you go to python.org and install python-2.3.7 instead of shooting blind. It's pretty easy if you're running linux, just be sure to end with make altinstall in case your distro has python installed in /usr/local.

If you don't mind, I'd do that as a last resort. The problem wasn't a Python version, just a mistake on the version of numpy being installed. I tested 1.2.0 when I was dealing with 1.1.x. So, I'm back on a 1.1.x, hopefully not for long.

On Mon, Jul 21, 2008 at 11:33 PM, Pierre GM wrote:

...

Pierre, I suggest you go to python.org and install python-2.3.7 instead of shooting blind. It's pretty easy if you're running linux, just be sure to end with make altinstall in case your distro has python installed in /usr/local.

I forgot to mention that my reluctance to install another Python version comes from the distro I run on: Gentoo is unforgiving with the slightest issue with Python, and I can't really afford the time to install Kubuntu on my only machine...

But I thought Gentoo was for uber geeks? And testing the cpu cooler. Anyway, the tests pass now, thanks. Chuck

On Tue, Jul 22, 2008 at 12:10 AM, Pierre GM wrote:

On Tuesday 22 July 2008 02:06:35 Charles R Harris wrote:

...

But I thought Gentoo was for uber geeks? And testing the cpu cooler.

Turns out that I'm not one. It's educational, though, but the older I get, the less compiling kernels and tweaking OS corresponds to my idea of fun.

...

Anyway, the tests pass now, thanks.

You're quite welcome, and many apologies for the noise and waste of time.

Looks like you shouldn't use NumpyTestCase for the 1.2 test, however. /usr/lib/python2.5/unittest.py:507: DeprecationWarning: NumpyTestCase will be removed in the next release; please update your code to use nose or unittest return self.suiteClass(map(testCaseClass, testCaseNames)) /usr/lib/python2.5/site-packages/numpy/ma/tests/test_extras.py:330: DeprecationWarning: NumpyTestCase will be removed in the next release; please update your code to use nose or unittest NumpyTestCase.__init__(self, *args, **kwds) Chuck

On Tue, Jul 22, 2008 at 12:39 AM, Pierre GM wrote:

On Tuesday 22 July 2008 02:27:38 Charles R Harris wrote:

...

Looks like you shouldn't use NumpyTestCase for the 1.2 test, however.

Looks like Alan updated it for me, the tests look OK on 1.2.

Interestingly, I segfault when running python -c "import numpy; numpy.test()" (NumPy version 1.2.0.dev5496)

I'm testing 1.2.0.dev5497 and still see the deprecation warnings, they are up near the beginning. I don't get a segfault, either. I wonder what that is about? Chuck

On Mon, Jul 21, 2008 at 8:40 PM, Stéfan van der Walt wrote:

2008/7/20 Ryan May :

...

...

arr array([('JOE', 25.300000000000001), ('BOB', 27.899999999999999)], dtype=[('stid', '|S4'), ('temp', '

The code in SVN still breaks for more complicated dtypes, such as:

np.dtype([('x', int), ('y', [('t', int), ('s', float)])])

Please find attached a patch which addresses the issue (it passes all unit tests).

This bit is illegal syntax in Python 2.3 X.append(tuple(conv(val) for (conv, val) in zip(converterseq, vals))) So this isn't going to work for 1.1.1 Chuck

On Tue, Jul 22, 2008 at 1:06 AM, Pierre GM wrote:

On Tuesday 22 July 2008 02:58:25 Charles R Harris wrote:

...

I'm testing 1.2.0.dev5497 and still see the deprecation warnings, they are up near the beginning.

I'm afraid you have some outdated files lingering somewhere, there's no NumpyTestCase in the sources of numpy.ma

...

I don't get a segfault, either. I wonder what that is about?

Looks like it's coming from linalg. Now, which one ?

This is a new thing, I take it. Looks like a good time to wait for morning ;) The deprecation warning went away when I deleted the numpy site-package and reinstalled. Grrr, I should know better by now. Chuck

On Tue, Jul 22, 2008 at 2:45 AM, Stéfan van der Walt wrote:

2008/7/22 Charles R Harris :

...

This bit is illegal syntax in Python 2.3

X.append(tuple(conv(val) for (conv, val) in zip(converterseq, vals)))

So this isn't going to work for 1.1.1

That's easy to fix. New patch attached.

Yep, but I wanted you to look it over and find all the problems ;) Could you post a patch against current mainline svn, which already has your previous patch applied? I'm also curious why we need tuples, are we using these values as hashes someplace. Chuck

2008/7/22 Stéfan van der Walt :

...

Could you post a patch against current mainline svn, which already has your previous patch applied? I'm also curious why we need tuples, are we using these values as hashes someplace.

Applied. The reason we need to use tuples is because

Should these changes be back-ported to the 1.1.1 branch? Stéfan

On Tue, Jul 22, 2008 at 11:44 AM, Charles R Harris wrote:

OK, the only other change since 1.1.0 is using np.x in the doctests, which doesn't look like a big problem. I wonder what the status of that is in 1.1.x? Alan?

All the changes I made for that were in the trunk after 1.1.1 was tagged, so the 1.1.1 test behavior should be completely under the old framework's rules. I could probably backport the np.x context to rundocs() (off the top of my head I can't remember if there was any mechanism to run all the doctests in NumPy), but it might just be easier to add "import numpy as np" to the modules containing doctests that use np.x. I can help with that if need be, just let me know.

On Tue, Jul 22, 2008 at 9:58 AM, Alan McIntyre wrote:

On Tue, Jul 22, 2008 at 11:44 AM, Charles R Harris wrote:

...

OK, the only other change since 1.1.0 is using np.x in the doctests, which doesn't look like a big problem. I wonder what the status of that is in 1.1.x? Alan?

All the changes I made for that were in the trunk after 1.1.1 was tagged, so the 1.1.1 test behavior should be completely under the old framework's rules. I could probably backport the np.x context to rundocs() (off the top of my head I can't remember if there was any mechanism to run all the doctests in NumPy), but it might just be easier to add "import numpy as np" to the modules containing doctests that use np.x. I can help with that if need be, just let me know.

If we do a 1.1.2 it might be handy for future backports if rundocs() has the np.x context. For the moment I'm tempted to just leave them as they don't look any worse than than they were, i.e., assuming from numpy import * Thanks, Chuck

On Tue, Jul 22, 2008 at 10:23 AM, Pierre GM wrote:

On Tuesday 22 July 2008 12:05:22 Charles R Harris wrote:

...

Failed importing /usr/local/lib/python2.3/site-packages/numpy/ma/tests/test_mrecords.py: invalid syntax (mrecords.py, line 245)

Charles, Can you import numpy.ma.mrecords ? And we're talking about the 1.1.x branch, right ?

I fixed it, Pierre. You can't do (_[1] for _ in ddtype.descr) to get a tuple. Actually, I think this should fail in mainline also. Alan, does nose show import failures? Chuck

On Tuesday 22 July 2008 12:56:11 Charles R Harris wrote:

Well, it produces a generator object in python2.5, which zip accepts. I don't know in which Python version this feature was added.

Likely 2.4, as it works on my machine (else I would have found about it at one point or another...)

On Tue, Jul 22, 2008 at 12:48 PM, Charles R Harris wrote:

Actually, I think this should fail in mainline also. Alan, does nose show import failures?

...

...

import numpy.ma.mrecords Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.3/site-packages/numpy/__init__.py",

I've seen talk of it swallowing some exceptions, but I'm not sure of the specifics. Are you referring to "import numpy.ma.mrecords" under Python 2.3, NumPy trunk? Here's what I get: Python 2.3.7 (#1, Jul 14 2008, 22:34:29) [GCC 4.1.2 (Gentoo 4.1.2 p1.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. line 107, in ? import ma File "/usr/local/lib/python2.3/site-packages/numpy/ma/__init__.py", line 14, in ? import core File "/usr/local/lib/python2.3/site-packages/numpy/ma/core.py", line 113, in ? max_filler.update([(k, -np.inf) for k in [np.float32, np.float64]]) AttributeError: keys

On Tue, Jul 22, 2008 at 1:00 PM, Alan McIntyre wrote:

I've seen talk of it swallowing some exceptions, but I'm not sure of the specifics. Are you referring to "import numpy.ma.mrecords" under Python 2.3, NumPy trunk?

When I run numpy/numpy/ma/tests/test_mrecords.py from the trunk with python2.3 (and this should be running under nose), the import fails and appears to be handled correctly. That's with nose 0.11.0 locally, though; I'll try it with an older version to see if it changes.

On Tue, Jul 22, 2008 at 10:52 AM, Pierre GM wrote:

On Tuesday 22 July 2008 12:48:20 Charles R Harris wrote:

...

I fixed it, Pierre. You can't do

(_[1] for _ in ddtype.descr)

...

to get a tuple.

OK, thx for that. AAMOF, lines 243-245 should be: self._fill_value = np.array(tuple(fillval), dtype=[(_[0], _[1]) for _ in ddtype.descr])

I guess 2.3 choked on the generator instead of the list.

I just replaced () by []. The 1.1.x version is now dt = zip(ddtype.names, [s[1] for s in ddtype.descr]) Which should have the same effect, although not done as slickly. Chuck

On Tue, Jul 22, 2008 at 1:00 PM, Pierre GM wrote:

On Tuesday 22 July 2008 13:00:08 Alan McIntyre wrote:

...

I've seen talk of it swallowing some exceptions, but I'm not sure of the specifics. Are you referring to "import numpy.ma.mrecords" under Python 2.3, NumPy trunk? Here's what I get:

Hold on a second, what are we taling about ? If it's the trunk, it's Numpy1.2, and Python2.3 shouldn't be supported, right ?

I've been told NumPy 1.2 doesn't have to support Python 2.3, so I hope that's right. :)

If it's the 1.1.x branch, then I'm in trouble. Please don't tell me I'm in trouble.

No, I was just trying NumPy trunk against Python 2.3 based on something Charles said; sorry for the confusion.

On Tue, Jul 22, 2008 at 10:09 AM, Alan McIntyre wrote:

I've been told NumPy 1.2 doesn't have to support Python 2.3, so I hope that's right. :)

Yes that's correct. NumPy 1.2 requires at least Python 2.4 -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

On Tuesday 22 July 2008 13:41:04 Charles R Harris wrote:

The missing ctypes shouldn't be a problem. I don't know what to make of the f2py import failure or if it matters.

/extgen/py_support.py:284: parent_container_options.update(prefix='"\\n\\n:Parameters:\\n"\n" ') ./extgen/py_support.py:287: parent_container_options.update(prefix='"\\n\\n:Optional parameters: \\n"\n" ') ./extgen/py_support.py:290: parent_container_options.update(prefix='"\\n\\n:Extra parameters:\\n"\n" ') ./extgen/py_support.py:293: parent_container_options.update(prefix='"\\n\\n:Returns:\\n"\n" ', so, we can't use that for Python 2.3, we need to use a syntax like parent_container_options.update({'prefix':blahblahblah})

On Tue, Jul 22, 2008 at 11:46 AM, Robert Kern wrote:

On Tue, Jul 22, 2008 at 12:41, Charles R Harris wrote:

...

So, the following oddities remain for Python2.3 :

Failed importing numpy.f2py.lib.extgen: update() takes no keyword

...

<snip> ctypes is not available on this python: skipping the test (import error was: ctypes is not available.) No distutils available, skipping test.

The missing ctypes shouldn't be a problem. I don't know what to make of

arguments the

...

f2py import failure or if it matters.

Everything in numpy/f2py/lib/ is the unfinished G3 version of f2py, whose development has moved to Launchpad. It has been removed from the trunk, and errors in it can be ignored for 1.1.x.

...

Nor do I understand what No distutils available implies. Anyone have ideas?

It's another ctypes skipped test. numpy/tests/test_ctypes.py:TestLoadLibrary.check_basic2().

OK, thanks. Can I take it that we are good to go as far as the tests show? Chuck

On Tue, Jul 22, 2008 at 12:03 PM, Robert Kern wrote:

On Tue, Jul 22, 2008 at 12:57, Charles R Harris wrote:

...

On Tue, Jul 22, 2008 at 11:46 AM, Robert Kern

wrote:

...

...

On Tue, Jul 22, 2008 at 12:41, Charles R Harris wrote:

...

So, the following oddities remain for Python2.3 :

Failed importing numpy.f2py.lib.extgen: update() takes no keyword arguments <snip> ctypes is not available on this python: skipping the test (import

error

...

...

was: ctypes is not available.) No distutils available, skipping test.

The missing ctypes shouldn't be a problem. I don't know what to make of the f2py import failure or if it matters.

Everything in numpy/f2py/lib/ is the unfinished G3 version of f2py, whose development has moved to Launchpad. It has been removed from the trunk, and errors in it can be ignored for 1.1.x.

...

Nor do I understand what No distutils available implies. Anyone have ideas?

It's another ctypes skipped test. numpy/tests/test_ctypes.py:TestLoadLibrary.check_basic2().

OK, thanks. Can I take it that we are good to go as far as the tests show?

I think so.

Although... would it be a problem to just remove the f2py stuff? That would get rid of one confusing message. Chuck

On Tue, Jul 22, 2008 at 11:20 AM, Charles R Harris wrote:

Although... would it be a problem to just remove the f2py stuff? That would get rid of one confusing message.

+1 -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/

On Tue, Jul 22, 2008 at 15:07, Charles R Harris wrote:

On Tue, Jul 22, 2008 at 12:51 PM, Robert Kern wrote:

...

On Tue, Jul 22, 2008 at 13:20, Charles R Harris wrote:

...

Although... would it be a problem to just remove the f2py stuff? That would get rid of one confusing message.

Probably not. See r5347 and r5348 for what I had to do on the trunk.

OK, done. I think I'll replace the ctypes warnings with a simple "S" if no one objects.

Sure. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Tuesday 22 July 2008 17:10:57 Charles R Harris wrote:

Runs on both Python 2.5.1 and 2.3.7 here. I'll run the buildbots.

Thx a lot, I'll close the ticket.

Any more changes in the pipeline?

I hope not. I wasn't especially thrilled to get a ticket at the very last minute, but this one was definitely worth fixing ASAP...

On Tue, Jul 22, 2008 at 3:04 PM, Pierre GM wrote:

On Tuesday 22 July 2008 17:10:57 Charles R Harris wrote:

...

Runs on both Python 2.5.1 and 2.3.7 here. I'll run the buildbots.

Thx a lot, I'll close the ticket.

Any more changes in the pipeline? Chuck

On Tue, Jul 22, 2008 at 3:22 PM, Pierre GM wrote:

On Tuesday 22 July 2008 17:24:27 Alan McIntyre wrote:

...

1.1.x runs with no errors against 2.3.7, 2.4.5, and 2.5.3 for me on Linux.

Thx Alan ! I didn't expect the fix to crash anything, but better safe than sorry. _____

Apropos nothing much, but I can definitely recommend gcc 4.3. It's *much* faster. So much so that I wasn't even sure the build was working correctly ;) YMMV. Chuck