1.6.0b1 half float buffer bug?
I'm working on getting support for 16-bit floats into cython and have run into what seems like a numpy bug (I'm a new contributor to both projects, so bear with me ;). https://github.com/wickedgrey/numpy/commit/29f9f1b709cc2c346b8514859c58a761d... Adding NPY_HALF to the switch statement on line 350 of numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it needs to be spelled NPY_FLOAT16 or something similar), but I have no idea how to test the change. Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request? Thanks, Eli
Fri, 25 Mar 2011 10:00:24 -0700, Eli Stevens (Gmail) wrote:
I'm working on getting support for 16-bit floats into cython and have run into what seems like a numpy bug (I'm a new contributor to both projects, so bear with me ;).
https://github.com/wickedgrey/numpy/ commit/29f9f1b709cc2c346b8514859c58a761df80f031
Adding NPY_HALF to the switch statement on line 350 of numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it needs to be spelled NPY_FLOAT16 or something similar), but I have no idea how to test the change.
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec. What can be done instead is exporting the half-float items (and any other non-3118 types) as plain bytes. I think this is the way to go. On the Cython side, you'd need to detect when you are working with Numpy arrays, and get the half-float type information from the Numpy dtype rather than from the exported buffer. -- Pauli Virtanen
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
Fri, 25 Mar 2011 10:00:24 -0700, Eli Stevens (Gmail) wrote:
I'm working on getting support for 16-bit floats into cython and have run into what seems like a numpy bug (I'm a new contributor to both projects, so bear with me ;).
https://github.com/wickedgrey/numpy/ commit/29f9f1b709cc2c346b8514859c58a761df80f031
Adding NPY_HALF to the switch statement on line 350 of numpy/core/src/multiarray/buffer.c doesn't seem controversial (unless it needs to be spelled NPY_FLOAT16 or something similar), but I have no idea how to test the change.
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118. -Mark
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes. Eli
On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com>wrote:
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes.
That's true, but explaining the hoops being jumped through to support a new data type across multiple Python plugins could help evolve the spec if necessary, and they may have some good suggestions. -Mark
On Fri, Mar 25, 2011 at 12:28 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) < wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes.
That's true, but explaining the hoops being jumped through to support a new data type across multiple Python plugins could help evolve the spec if necessary, and they may have some good suggestions.
Last I looked, support for PEP 3118 in Python 3 was stuck in the doldrums. Might be worth another ping. Chuck
On Fri, Mar 25, 2011 at 11:28 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) < wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes.
That's true, but explaining the hoops being jumped through to support a new data type across multiple Python plugins could help evolve the spec if necessary, and they may have some good suggestions.
Thinking about that a bit more, maybe a convention added to PEP 3118 for custom types would be useful. For float16, labeling it as 'H' makes sense, then it could be followed by metadata identifying the custom type, like 'H{float16}'. -Mark
On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes.
Actually, thinking about this more, if the PEP gets updated with a backwards-compatible change, numpy could support the newer version, even on older versions of Python that don't officially, right? How quickly could something like the 'e' type be added to the PEP? Eli
On Mon, Mar 28, 2011 at 10:45 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com>wrote:
On Fri, Mar 25, 2011 at 11:14 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:35 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
That said, I think starting a discussion with the Python core developers about the float16 type is worthwhile. There might be interest in supporting the float16 type in the struct interface, something that would be required as part of amending PEP 3118.
That's something that wouldn't see production until Python 3.3, right? I'm imagining that the buffer spec in use wouldn't change during the 2.7.x series, unless it's unusually accepting of changes.
Actually, thinking about this more, if the PEP gets updated with a backwards-compatible change, numpy could support the newer version, even on older versions of Python that don't officially, right?
How quickly could something like the 'e' type be added to the PEP?
You've got to get the attention and buy-in from the right people for this to happen. I found this with a brief search: http://www.python.org/dev/peps/pep-0001/#reporting-pep-bugs-or-submitting-pe... -Mark
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
What can be done instead is exporting the half-float items (and any other non-3118 types) as plain bytes. I think this is the way to go.
What about as uint16s? Won't plain bytes have endian issues? case NPY_HALF: if (_append_char(str, 'H')) return -1; break; Something like that? I am also still curious about where unit tests for things like this should go.
On the Cython side, you'd need to detect when you are working with Numpy arrays, and get the half-float type information from the Numpy dtype rather than from the exported buffer.
Hrm, okay. I'll have to follow up with the Cython folks about that. I hope that my fears about that ending up being a sweeping change are unfounded (having already gone through and made the changes to unilaterally add support for 'e' buffer types, it seems like it's baked in; we'll see). :-/ Thanks, Eli
On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request?
I've gotten a bit of positive feedback to adding the 'e' type to the struct module on the python-ideas list (per my understanding, not before python 3.3, but I don't think that should hinder adoption in other libraries), so I'd like to ask again about unit testing a change like this. Can anyone offer some advice for where to start? Also, what kind of timeframe / cutoff am I looking at to get this into 1.6.0 or 1.6.x? Thanks, Eli
On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com>wrote:
On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request?
I've gotten a bit of positive feedback to adding the 'e' type to the struct module on the python-ideas list (per my understanding, not before python 3.3, but I don't think that should hinder adoption in other libraries), so I'd like to ask again about unit testing a change like this. Can anyone offer some advice for where to start?
The tests for the buffer protocol are in numpy/core/tests/test_multiarray.py, starting at line 1847. It does some round-trip testing of all the types through the memoryview object. You'll also need to change the _dtype_from_pep3118 in numpy/core/_internal.py, called from the _descriptor_from_pep3118_format function, so that NumPy can accept the half type from a buffer as well. Also, what kind of timeframe / cutoff am I looking at to get this into
1.6.0 or 1.6.x?
Since the changes to NumPy are pretty small, I think it should be ok to get into 1.6.0. How to handle it is up to Ralf, however, and it would probably be best to get it into beta 2 which I believe he would like to release over the weekend (please chime in to correct me). -Mark
On Thu, Mar 31, 2011 at 7:06 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request?
I've gotten a bit of positive feedback to adding the 'e' type to the struct module on the python-ideas list (per my understanding, not before python 3.3, but I don't think that should hinder adoption in other libraries), so I'd like to ask again about unit testing a change like this. Can anyone offer some advice for where to start?
The tests for the buffer protocol are in numpy/core/tests/test_multiarray.py, starting at line 1847. It does some round-trip testing of all the types through the memoryview object. You'll also need to change the _dtype_from_pep3118 in numpy/core/_internal.py, called from the _descriptor_from_pep3118_format function, so that NumPy can accept the half type from a buffer as well.
Also, what kind of timeframe / cutoff am I looking at to get this into 1.6.0 or 1.6.x?
Since the changes to NumPy are pretty small, I think it should be ok to get into 1.6.0. How to handle it is up to Ralf, however, and it would probably be best to get it into beta 2 which I believe he would like to release over the weekend (please chime in to correct me).
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment. It would need some testing with other libraries that use the numpy buffer protocol. Also, reading back the thread, Pauli seemed to disagree with this. Ralf
On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:
On Wed, Mar 30, 2011 at 4:42 PM, Eli Stevens (Gmail) < wickedgrey@gmail.com> wrote:
On Fri, Mar 25, 2011 at 10:00 AM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Can anyone please give me some suggestions on how to go about writing a unit test for this? Or should I just submit a pull request?
I've gotten a bit of positive feedback to adding the 'e' type to the struct module on the python-ideas list (per my understanding, not before python 3.3, but I don't think that should hinder adoption in other libraries), so I'd like to ask again about unit testing a change like this. Can anyone offer some advice for where to start?
The tests for the buffer protocol are in numpy/core/tests/test_multiarray.py, starting at line 1847. It does some round-trip testing of all the types through the memoryview object. You'll also need to change the _dtype_from_pep3118 in numpy/core/_internal.py, called from the _descriptor_from_pep3118_format function, so that NumPy can accept the half type from a buffer as well.
Also, what kind of timeframe / cutoff am I looking at to get this into 1.6.0 or 1.6.x?
Since the changes to NumPy are pretty small, I think it should be ok to get into 1.6.0. How to handle it is up to Ralf, however, and it would
On Thu, Mar 31, 2011 at 7:06 PM, Mark Wiebe <mwwiebe@gmail.com> wrote: probably
be best to get it into beta 2 which I believe he would like to release over the weekend (please chime in to correct me).
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment. It would need some testing with other libraries that use the numpy buffer protocol. Also, reading back the thread, Pauli seemed to disagree with this.
My reading of Pauli's thoughts was that putting it in unilaterally is undesirable, something I definitely agree with. I think with Eli doing the legwork of getting input and acceptance from the relevant parties, we should help him out as much as possible. Not getting this change into 1.6 makes the Cython support much more difficult because of their design based around the buffer protocol. Looking at the thread on python-users, I see mostly concern that the type be standard or have precedent elsewhere, which as an IEEE standard it definitely satisfies. The other question is on the chosen letter - we picked 'e' as 'h' was taken and it is close to 'f' and 'd', the other float types. If a letter change is deemed necessary, it would be good to get that into 1.6 as well, since this kind of change is easy now, but much more difficult later. For testing, Eli implementing the Cython support seems like a good start to me. Another nice test would be to get PIL's OpenEXR support to export a half buffer to NumPy. -Mark
Thu, 31 Mar 2011 11:15:08 -0700, Mark Wiebe wrote: [clip]
My reading of Pauli's thoughts was that putting it in unilaterally is undesirable, something I definitely agree with. I think with Eli doing the legwork of getting input and acceptance from the relevant parties, we should help him out as much as possible.
Yes, this was the main concern. However, reading the thread on python-ideas, apart from the choice of the letter there does not seem to be much resistance, especially as we have a volunteer for implementing any changes required. :)
Not getting this change into 1.6 makes the Cython support much more difficult because of their design based around the buffer protocol.
Looking at the thread on python-users, I see mostly concern that the type be standard or have precedent elsewhere, which as an IEEE standard it definitely satisfies. The other question is on the chosen letter - we picked 'e' as 'h' was taken and it is close to 'f' and 'd', the other float types. If a letter change is deemed necessary, it would be good to get that into 1.6 as well, since this kind of change is easy now, but much more difficult later.
The buffer interface format string does not need to match what Numpy internally uses (although that would be clearer), so changing that later on, if the Python devs strongly disagree with the choice made, maybe does not have *too* much inertia. However, it could be good to come to an agreement soon, if possible. Pauli
On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment.
If I miss the 1.6.0b2 cutoff, would a change like this be appropriate for 1.6.1, or will it need to wait until 1.7.0? I wish I'd known that half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd have started working on this a few weeks earlier. :)
It would need some testing with other libraries that use the numpy buffer protocol.
I am currently working on a patch for cython to accept half-floats. ATM, it is able to accept the data from numpy (though it does not yet handle the bit-fiddling to expand the values into a float32 on read, nor collapse it from a float32 back into a float16 on write). The code change to numpy was literally a one-line change to add NPY_HALF to a switch statement (though I haven't tried to import half floats yet). Is one other library enough, or will I need to find another library to patch as well? I think that OpenCL would probably be my next choice, since I think that we'll eventually want float16 support there too.
Also, reading back the thread, Pauli seemed to disagree with this.
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
I'm currently in the process of submitting a patch to CPython to update the struct module to handle half-floats. So far, the response on python-ideas has been generally positive (a couple +1s from people with commit bits, GVR has chimed in on the thread with an aside (but not an objection), etc.). Unless I'm misunderstanding the objection, making half-floats an official part of the spec (the discussion on python-ideas explicitly notes that numpy driving force behind the change) would satisfy Pauli's concern. Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today. Thanks! Eli
On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment.
If I miss the 1.6.0b2 cutoff, would a change like this be appropriate for 1.6.1, or will it need to wait until 1.7.0? I wish I'd known that half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd have started working on this a few weeks earlier. :)
There likely will not be a 1.7.0, if things with 2.0 go according to plan. It probably wouldn't be a very appropriate change for 1.6.1.
It would need some testing with other libraries that use the numpy buffer protocol.
I am currently working on a patch for cython to accept half-floats. ATM, it is able to accept the data from numpy (though it does not yet handle the bit-fiddling to expand the values into a float32 on read, nor collapse it from a float32 back into a float16 on write). The code change to numpy was literally a one-line change to add NPY_HALF to a switch statement (though I haven't tried to import half floats yet). Is one other library enough, or will I need to find another library to patch as well? I think that OpenCL would probably be my next choice, since I think that we'll eventually want float16 support there too.
If it works with Cython I think that's enough. With other libraries I did not mean that you should write patches for those, just some testing to make sure the change is 100% backwards compatible.
Also, reading back the thread, Pauli seemed to disagree with this.
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
I'm currently in the process of submitting a patch to CPython to update the struct module to handle half-floats. So far, the response on python-ideas has been generally positive (a couple +1s from people with commit bits, GVR has chimed in on the thread with an aside (but not an objection), etc.). Unless I'm misunderstanding the objection, making half-floats an official part of the spec (the discussion on python-ideas explicitly notes that numpy driving force behind the change) would satisfy Pauli's concern.
Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today.
Sounds good. Cheers, Ralf
On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today.
Sounds good.
CPython issue here: http://bugs.python.org/issue11734 My numpy changes so far: https://github.com/wickedgrey/numpy/compare/23965b6b665985390d7a...maintenan...
From my last commit message: "Note that my changes to _internal were mostly of the "make it work" variety; I'm not really understanding the wider scope. Very much needs review by wiser eyes."
I'd also appreciate some feedback on the unit test coverage; the test_roundtrip_half function uncovered a problem that the other two didn't (which surprised me), and I'd like to make sure that there aren't more gaps in the tests that I've written. Thanks for all the help! Eli
On Thu, Mar 31, 2011 at 8:12 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com>wrote:
On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today.
Sounds good.
CPython issue here: http://bugs.python.org/issue11734
My numpy changes so far:
https://github.com/wickedgrey/numpy/compare/23965b6b665985390d7a...maintenan...
From my last commit message: "Note that my changes to _internal were mostly of the "make it work" variety; I'm not really understanding the wider scope. Very much needs review by wiser eyes."
I'd also appreciate some feedback on the unit test coverage; the test_roundtrip_half function uncovered a problem that the other two didn't (which surprised me), and I'd like to make sure that there aren't more gaps in the tests that I've written.
Your changes look fine to me, and I think the tests are reasonable as is. I would suggest tidying up your commit messages a bit, creating an enhancement bug in the bug tracker, and making a pull request. Some info about the pull requests is here: http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html -Mark
Thanks for all the help! Eli _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Apr 1, 2011 at 5:28 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
Your changes look fine to me, and I think the tests are reasonable as is. I would suggest tidying up your commit messages a bit, creating an enhancement bug in the bug tracker, and making a pull request. Some info about the pull requests is here: http://docs.scipy.org/doc/numpy/dev/gitwash/development_workflow.html
Done and done. :) http://projects.scipy.org/numpy/ticket/1789 https://github.com/numpy/numpy/pull/68 Now to figure out how I can speed things up on the CPython side of things... Cheers, Eli
On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:
On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment.
If I miss the 1.6.0b2 cutoff, would a change like this be appropriate for 1.6.1, or will it need to wait until 1.7.0? I wish I'd known that half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd have started working on this a few weeks earlier. :)
There likely will not be a 1.7.0, if things with 2.0 go according to plan. It probably wouldn't be a very appropriate change for 1.6.1.
It would need some testing with other libraries that use the numpy buffer protocol.
I am currently working on a patch for cython to accept half-floats. ATM, it is able to accept the data from numpy (though it does not yet handle the bit-fiddling to expand the values into a float32 on read, nor collapse it from a float32 back into a float16 on write). The code change to numpy was literally a one-line change to add NPY_HALF to a switch statement (though I haven't tried to import half floats yet). Is one other library enough, or will I need to find another library to patch as well? I think that OpenCL would probably be my next choice, since I think that we'll eventually want float16 support there too.
If it works with Cython I think that's enough. With other libraries I did not mean that you should write patches for those, just some testing to make sure the change is 100% backwards compatible.
Also, reading back the thread, Pauli seemed to disagree with this.
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
I'm currently in the process of submitting a patch to CPython to update the struct module to handle half-floats. So far, the response on python-ideas has been generally positive (a couple +1s from people with commit bits, GVR has chimed in on the thread with an aside (but not an objection), etc.). Unless I'm misunderstanding the objection, making half-floats an official part of the spec (the discussion on python-ideas explicitly notes that numpy driving force behind the change) would satisfy Pauli's concern.
Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today.
Sounds good.
I've committed the patch adding 'e' to the buffer protocol and back-ported it to the 1.6.x branch. This leaves the testing with Cython and getting consensus from CPython about the letter to use in the struct module/buffer interface.
-Mark
On Sat, Apr 2, 2011 at 12:39 PM, Mark Wiebe <mwwiebe@gmail.com> wrote:
On Thu, Mar 31, 2011 at 11:45 AM, Ralf Gommers < ralf.gommers@googlemail.com> wrote:
On Thu, Mar 31, 2011 at 8:37 PM, Eli Stevens (Gmail) <wickedgrey@gmail.com> wrote:
On Thu, Mar 31, 2011 at 10:40 AM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
It would definitely need to get into beta 2, and even then I'm a little hesitant to push in such a change at the last moment.
If I miss the 1.6.0b2 cutoff, would a change like this be appropriate for 1.6.1, or will it need to wait until 1.7.0? I wish I'd known that half floats were going to be in 1.6.0 (the bug tracker says 2.0); I'd have started working on this a few weeks earlier. :)
There likely will not be a 1.7.0, if things with 2.0 go according to plan. It probably wouldn't be a very appropriate change for 1.6.1.
It would need some testing with other libraries that use the numpy buffer protocol.
I am currently working on a patch for cython to accept half-floats. ATM, it is able to accept the data from numpy (though it does not yet handle the bit-fiddling to expand the values into a float32 on read, nor collapse it from a float32 back into a float16 on write). The code change to numpy was literally a one-line change to add NPY_HALF to a switch statement (though I haven't tried to import half floats yet). Is one other library enough, or will I need to find another library to patch as well? I think that OpenCL would probably be my next choice, since I think that we'll eventually want float16 support there too.
If it works with Cython I think that's enough. With other libraries I did not mean that you should write patches for those, just some testing to make sure the change is 100% backwards compatible.
Also, reading back the thread, Pauli seemed to disagree with this.
On Fri, Mar 25, 2011 at 10:21 AM, Pauli Virtanen <pav@iki.fi> wrote:
The buffer interface cannot be used to export the half-float types, since the type is not specified in PEP 3118. Numpy cannot unilaterally add nonstandard format codes to the spec.
I'm currently in the process of submitting a patch to CPython to update the struct module to handle half-floats. So far, the response on python-ideas has been generally positive (a couple +1s from people with commit bits, GVR has chimed in on the thread with an aside (but not an objection), etc.). Unless I'm misunderstanding the objection, making half-floats an official part of the spec (the discussion on python-ideas explicitly notes that numpy driving force behind the change) would satisfy Pauli's concern.
Next up for me is to get a patch onto the CPython issue tracker, but as soon as I have that done, I'll start working on adding unit tests to my numpy fork. I anticipate being able to get that done today.
Sounds good.
I've committed the patch adding 'e' to the buffer protocol and back-ported it to the 1.6.x branch. This leaves the testing with Cython and getting consensus from CPython about the letter to use in the struct module/buffer interface.
I didn't see any usable suggestions for an alternative to 'e' and not much resistance, so I expect 'e' will stick. Chuck
On Sat, Apr 2, 2011 at 11:39 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:
I've committed the patch adding 'e' to the buffer protocol and back-ported it to the 1.6.x branch.
Awesome! Thanks a bunch. :)
This leaves the testing with Cython and getting consensus from CPython about the letter to use in the struct module/buffer interface.
I've done some informal poking around using half-floats with my patched numpy and Cython and things seem to be okay; let me know what you're looking for in more detail and I'd be happy to do the legwork. :) Cheers, Eli
participants (5)
-
Charles R Harris -
Eli Stevens (Gmail) -
Mark Wiebe -
Pauli Virtanen -
Ralf Gommers