Fixing numpy 1.4.0 ABI breakage, and a plea for self-contained, small commits
Hi, I have investigated further the ABI issues reported for numpy 1.4.0. I can confirm that we have broken the ABI for 1.4.0 compared to 1.3.0 (besides the "trivial" cython issue). The good new is that I have found the issue, the bad news is that I don't know how to (cleanly) fix it. The problem was caused by the new datetime support, in particular the structure PyArray_ArrFuncs which has been modified in a ABI-incompatible way (the first member cast is bigger because NTYPES is bigger). I don't know how to fix this cleanly - the only solution I can see is to to split cast into two parts, the first part the same size as before and the second part at the end of the structure, with a double-case test everytime the cast member is accessed inside the relevant functions.... As an aside, I would like to reiterate my advice for *small* commits. It took me nearly 2 hours to find this because the offending commit was > 4000 LOC, and it would have been very easy to find this were the code committed as a set of small self-contained commits. thanks, David
On Tue, Jan 26, 2010 at 9:24 PM, David Cournapeau <david@silveregg.co.jp>wrote:
Hi,
I have investigated further the ABI issues reported for numpy 1.4.0. I can confirm that we have broken the ABI for 1.4.0 compared to 1.3.0 (besides the "trivial" cython issue). The good new is that I have found the issue, the bad news is that I don't know how to (cleanly) fix it.
The problem was caused by the new datetime support, in particular the structure PyArray_ArrFuncs which has been modified in a ABI-incompatible way (the first member cast is bigger because NTYPES is bigger).
Hmm, nasty. I don't like that structure anyway, it should be a pointer to a structure, or somehow not there in the first place. Yeah, it's a catastrophic "solution". Probably the only compatible fixes are: 1) remove the new function, 2) put it at the end of the enclosing structure, 3) live with the ABI breakage. The last is the easiest way to go for us, if not for others. The first solves the problem, but pretty much vitiates the datetime work. And moving the function leads to all sorts of nasty work arounds and code fixes. Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm... Chuck
On Tue, Jan 26, 2010 at 9:50 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Tue, Jan 26, 2010 at 9:24 PM, David Cournapeau <david@silveregg.co.jp>wrote:
Hi,
I have investigated further the ABI issues reported for numpy 1.4.0. I can confirm that we have broken the ABI for 1.4.0 compared to 1.3.0 (besides the "trivial" cython issue). The good new is that I have found the issue, the bad news is that I don't know how to (cleanly) fix it.
The problem was caused by the new datetime support, in particular the structure PyArray_ArrFuncs which has been modified in a ABI-incompatible way (the first member cast is bigger because NTYPES is bigger).
Hmm, nasty. I don't like that structure anyway, it should be a pointer to a structure, or somehow not there in the first place. Yeah, it's a catastrophic "solution". Probably the only compatible fixes are: 1) remove the new function, 2) put it at the end of the enclosing structure, 3) live with the ABI breakage. The last is the easiest way to go for us, if not for others. The first solves the problem, but pretty much vitiates the datetime work. And moving the function leads to all sorts of nasty work arounds and code fixes.
Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm...
Thinking a bit more, for 1.4.1 I think we should just remove the function. Chuck
Charles R Harris wrote:
Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm...
That's something that we have to do at some point if we care about ABI (I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance). Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc... David
On Tue, Jan 26, 2010 at 10:02 PM, David Cournapeau <david@silveregg.co.jp>wrote:
Charles R Harris wrote:
Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm...
That's something that we have to do at some point if we care about ABI (I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance).
Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc...
NumPy 2.0 is going to be a *lot* of work. And I've been thinking about it lately, mostly because I was looking over the same code where you found this problem. What I didn't know was how public the code was. Good find, BTW. One thought was to start by separating out the ufuncs and their dependency on ndarrays. But then I looked at the new buffer interface and it just won't do as a replacement, no complex numbers, etc. Maybe it can be extended. Anyway, if we make a move it needs to be well planned. Chuck
Charles R Harris wrote:
On Tue, Jan 26, 2010 at 10:02 PM, David Cournapeau <david@silveregg.co.jp <mailto:david@silveregg.co.jp>> wrote:
Charles R Harris wrote:
> > Whatever we do, it would be good to figure out some way to avoid this > problem in the future. We could hide access to the array, for instance. > But again, that would require a lot of other code mods. Hmm...
That's something that we have to do at some point if we care about ABI (I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance).
Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc...
NumPy 2.0 is going to be a *lot* of work. And I've been thinking about it lately, mostly because I was looking over the same code where you found this problem. What I didn't know was how public the code was. Good find, BTW.
One thought was to start by separating out the ufuncs and their dependency on ndarrays. But then I looked at the new buffer interface and it just won't do as a replacement, no complex numbers, etc. Maybe it can be extended. Anyway, if we make a move it needs to be well planned.
Huh? The PEP 3118 buffer format strings "Zf", "Zd", "Zg" are respectively complex float, double, long double. Any other reasons PEP 3118 can't be used? Not saying I believe there isn't, I'm just curious... Dag Sverre
On Wed, Jan 27, 2010 at 1:48 AM, Dag Sverre Seljebotn < dagss@student.matnat.uio.no> wrote:
Charles R Harris wrote:
On Tue, Jan 26, 2010 at 10:02 PM, David Cournapeau <david@silveregg.co.jp <mailto:david@silveregg.co.jp>> wrote:
Charles R Harris wrote:
> > Whatever we do, it would be good to figure out some way to avoid this > problem in the future. We could hide access to the array, for instance. > But again, that would require a lot of other code mods. Hmm...
That's something that we have to do at some point if we care about
ABI
(I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance).
Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc...
NumPy 2.0 is going to be a *lot* of work. And I've been thinking about it lately, mostly because I was looking over the same code where you found this problem. What I didn't know was how public the code was. Good find, BTW.
One thought was to start by separating out the ufuncs and their dependency on ndarrays. But then I looked at the new buffer interface and it just won't do as a replacement, no complex numbers, etc. Maybe it can be extended. Anyway, if we make a move it needs to be well
planned. Huh? The PEP 3118 buffer format strings "Zf", "Zd", "Zg" are respectively complex float, double, long double.
Any other reasons PEP 3118 can't be used? Not saying I believe there isn't, I'm just curious...
I wasn't looking at the PEP, I was looking at the python 3.x documentation which claims that the type strings used the same notation as the struct module. const char *formatA *NULL* terminated string in struct<http://docs.python.org/library/struct.html#module-struct>module style syntax giving the contents of the elements available through the buffer. If this is *NULL*, "B" (unsigned bytes) is assumed.I assumed that the PEP would be more compatible since Travis put it together and that it was changed on the journey to python inclusion. It could also be the case that the python documentation isn't correct ;) But if we go over to a buffer interface we need to use what was in the PEP. Chuck
On Tue, Jan 26, 2010 at 11:02 PM, David Cournapeau <david@silveregg.co.jp> wrote:
Charles R Harris wrote:
Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm...
That's something that we have to do at some point if we care about ABI (I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance).
Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc...
David
Numpy 1.5? :-) That was an incredible effort! My understanding is that a minor numpy release should not break the ABI and a major release is required when there is an ABI breakage.Thus, this ABI change must be in numpy 2.0 and not allowed in the numpy 1.x series unless the changes can not be 'easily' made in a way that does not break the 1.x series ABI. Alternatively, just acknowledge the fact as a unintended consequence and move on - which has happened before in numpy for a similar situation (see links below). Recent comments appeared in David's thread 'Going toward time-based release ?' http://article.gmane.org/gmane.comp.python.numeric.general/21368 Especially Robert's and Jarrod's responses in the sub-thread: http://article.gmane.org/gmane.comp.python.numeric.general/21378 Hopefully some users of the numpy ABI can provide some feedback on their needs. Just my 2 cents, Bruce
Bruce Southey wrote:
On Tue, Jan 26, 2010 at 11:02 PM, David Cournapeau <david@silveregg.co.jp> wrote:
Charles R Harris wrote:
Whatever we do, it would be good to figure out some way to avoid this problem in the future. We could hide access to the array, for instance. But again, that would require a lot of other code mods. Hmm... That's something that we have to do at some point if we care about ABI (I think we should care - expecting people to recompile all the extensions for a new version of numpy is a big hindrance).
Assuming python 1.5 will have py3k support, I was wondering about starting working on NumPy 2.0, with massive changes to the C API so that we can avoid this problem in the future: no more "naked" structures, much cleaner/leaner headers to avoid accidental reliance on specific private binary layouts, etc...
David
Numpy 1.5? :-)
That was an incredible effort!
My understanding is that a minor numpy release should not break the ABI and a major release is required when there is an ABI breakage.
That's the usual practice, but not in numpy (neither in python BTW). The ABI is regularly broken in minor releases.
Hopefully some users of the numpy ABI can provide some feedback on their needs.
Breaking the ABI simply means that every single package using the C API will have to be recompiled. This means for example that every windows/mac os x binary out there is broken (including scipy binaries). It is particularly painful if you need to have some packages with numpy 1.3.0 and some with 1.4.0. The idea is that in the current state, keeping the ABI is incredibly difficult, so we would need to severly change the C code (in backward incompatible ways, i.e. it would require to break the A*P*I as well) to control those issues later. Guido explicitly asked not to break compatibility while staying under py3k, so we should try to do it once numpy has been ported to py3k (e.g. if numpy 1.5 still is not py3k compatible, do a 1.6 before a 2.0 - iterate if necessary :) ). David
On 1/27/2010 7:57 PM, David Cournapeau wrote:
Guido explicitly asked not to break compatibility while staying under py3k, so we should try to do it once numpy has been ported to py3k (e.g. if numpy 1.5 still is not py3k compatible, do a 1.6 before a 2.0 - iterate if necessary:) ).
This sounds very different than http://www.artima.com/weblogs/viewpost.jsp?thread=227041 Can you provide a link? Thanks, Alan Isaac
Alan G Isaac wrote:
On 1/27/2010 7:57 PM, David Cournapeau wrote:
Guido explicitly asked not to break compatibility while staying under py3k, so we should try to do it once numpy has been ported to py3k (e.g. if numpy 1.5 still is not py3k compatible, do a 1.6 before a 2.0 - iterate if necessary:) ).
This sounds very different than http://www.artima.com/weblogs/viewpost.jsp?thread=227041
Maybe my English is broken, as I meant exactly the same as in Guido's post: do not break API (C API here) while porting to py3k. Making the NumPy C API robust to changes wo constantly breaking the ABI will require heavy changes to C structures and how they are exposed to 3rd parties. It is impossible to do without breaking the C API. cheers, David
On 1/27/2010 7:57 PM, David Cournapeau wrote:
Guido explicitly asked not to break compatibility while staying under py3k, so we should try to do it once numpy has been ported to py3k (e.g. if numpy 1.5 still is not py3k compatible, do a 1.6 before a 2.0 - iterate if necessary:) ).
Alan G Isaac wrote:
This sounds very different than http://www.artima.com/weblogs/viewpost.jsp?thread=227041
On 1/27/2010 8:28 PM, David Cournapeau wrote:
Maybe my English is broken, as I meant exactly the same as in Guido's post: do not break API (C API here) while porting to py3k. Making the NumPy C API robust to changes wo constantly breaking the ABI will require heavy changes to C structures and how they are exposed to 3rd parties. It is impossible to do without breaking the C API.
My reading is: do not see py3k as an opportunity for API breakage. So if breakage is know to be necessary, do it now in a forward looking way, so that it will not be necessary after moving to py3k. Quoting from http://www.artima.com/weblogs/viewpost.jsp?thread=227041 : "If you have make API changes, do them before you port to 3.0" I thought you were saying the opposite of that ... ? fwiw, Alan
Alan G Isaac wrote:
On 1/27/2010 7:57 PM, David Cournapeau wrote:
Guido explicitly asked not to break compatibility while staying under py3k, so we should try to do it once numpy has been ported to py3k (e.g. if numpy 1.5 still is not py3k compatible, do a 1.6 before a 2.0 - iterate if necessary:) ).
Alan G Isaac wrote:
This sounds very different than http://www.artima.com/weblogs/viewpost.jsp?thread=227041
On 1/27/2010 8:28 PM, David Cournapeau wrote:
Maybe my English is broken, as I meant exactly the same as in Guido's post: do not break API (C API here) while porting to py3k. Making the NumPy C API robust to changes wo constantly breaking the ABI will require heavy changes to C structures and how they are exposed to 3rd parties. It is impossible to do without breaking the C API.
My reading is: do not see py3k as an opportunity for API breakage.
Yup.
So if breakage is know to be necessary, do it now in a forward looking way, so that it will not be necessary after moving to py3k.
Quoting from http://www.artima.com/weblogs/viewpost.jsp?thread=227041 : "If you have make API changes, do them before you port to 3.0"
Ah, that's the misunderstanding: I think you focus on before vs after, but that's not the most important point. The full quote is " If you have make API changes, do them before you port to 3.0 -- release a version with the new API for Python 2.5, or 2.6 if you must. (Or do it later, after you've released a port to 3.0 without adding new features.)" What matters is not to do it at the same time, so that porting 3rd party code with 2to3 is possible (that's the bolded text). Since the py3k port is already underway, it seems natural to me to first release a py3k compatible release, and then a new numpy with incompatible API. OTOH, one could make the argument that releasing the API would avoid having to port numpy "twice" (first to py3k with say numpy 1.5.0, then to the new API for numpy 2.0). But I am not sure it is a big change in practice ? David
On 1/27/2010 8:56 PM, David Cournapeau wrote:
one could make the argument that releasing the API would avoid having to port numpy "twice" (first to py3k with say numpy 1.5.0, then to the new API for numpy 2.0). But I am not sure it is a big change in practice ?
OK, I misunderstood: I thought you were proposing to change the API *only* for the py3k NumPy, effectively leaving the earlier Pythons orphaned. Sorry for the mistake. Alan
Alan G Isaac wrote:
On 1/27/2010 8:56 PM, David Cournapeau wrote:
one could make the argument that releasing the API would avoid having to port numpy "twice" (first to py3k with say numpy 1.5.0, then to the new API for numpy 2.0). But I am not sure it is a big change in practice ?
OK, I misunderstood: I thought you were proposing to change the API *only* for the py3k NumPy, effectively leaving the earlier Pythons orphaned.
Ah, inddeed. Given the current adoption of py3k for libraries that matter to us, that would be insane :) David
participants (5)
-
Alan G Isaac -
Bruce Southey -
Charles R Harris -
Dag Sverre Seljebotn -
David Cournapeau