Hi, I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed - some exceptions constructions are not allowed I think a first step would be to add a compatibility module to remove any use of print/reduce in the numpy codebase (should not be too difficult). For reduce, we could import functools from python code (functools is not available in python 2.4). Unless someone has an objection, I will import functools code, make it compile for python 2.4 and above, and remove all builtin reduce calls in numpy. Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years). cheers, David
Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip]
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed
Print is not removed, just changed to a function. So, print("foo") is valid code in 2.X and 3.0. OTOH, I don't think there are any valid use cases for "print" in Numpy code anyway (maybe apart from tests).
- some exceptions constructions are not allowed
Some more: - xrange is no more - dict.iteritems / dict.iterkeys / etc are no more etc.
Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years).
Can't we just assume Python 2.6 and 3.0 users have the appropriate versions of Nose installed? Or is the old Nose version going to disappear? This would limit our responsibility to just ensuring that our Nose extensions work with all relevant Nose versions. -- Pauli Virtanen
On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip]
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed
Print is not removed, just changed to a function. So,
print("foo")
Yes, as reduce, they are still available, but not as builtins anymore. But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
is valid code in 2.X and 3.0. OTOH, I don't think there are any valid use cases for "print" in Numpy code anyway (maybe apart from tests).
Yes, maybe. I will look at removing them completely.
Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years).
Can't we just assume Python 2.6 and 3.0 users have the appropriate versions of Nose installed? Or is the old Nose version going to disappear?
The problem is that nose 1.0 and py3k nose won't be compatible, and I don't really know how much different they will be. It is still not clear whether we will have to change our unit tests, for example. David
On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau <cournape@gmail.com>wrote:
On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip]
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed
Print is not removed, just changed to a function. So,
print("foo")
Yes, as reduce, they are still available, but not as builtins anymore. But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
I think the point is that you can just change it to print("yoyo") which will work in both python 2.x and 3.x. The parenthesis are just extraneous in python 2.x. Now, the more complicated uses of print won't be as easy to change, but I'm not sure how prevalent their use is. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma Sent from Norman, Oklahoma, United States
Ryan May wrote:
On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau <cournape@gmail.com But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
I think the point is that you can just change it to print("yoyo") which will work in both python 2.x and 3.x.
I think he meant: print "yoyo", a which can not be translated by adding parens:
print "yoyo", a yoyo [ 1. 1. 1.] print ("yoyo", a) ('yoyo', array([ 1., 1., 1.]))
I suppose we can write something like: def new_print(*args): print (" ".join([str(s) for s in args]))
new_print("yoyo", a) yoyo [ 1. 1. 1.]
Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Thu, Apr 23, 2009 at 11:23 AM, Christopher Barker <Chris.Barker@noaa.gov>wrote:
Ryan May wrote:
On Thu, Apr 23, 2009 at 9:52 AM, David Cournapeau <cournape@gmail.com But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
I think the point is that you can just change it to print("yoyo") which will work in both python 2.x and 3.x.
I think he meant:
print "yoyo", a
which can not be translated by adding parens:
print "yoyo", a yoyo [ 1. 1. 1.] print ("yoyo", a) ('yoyo', array([ 1., 1., 1.]))
I suppose we can write something like:
def new_print(*args): print (" ".join([str(s) for s in args]))
new_print("yoyo", a) yoyo [ 1. 1. 1.]
Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5)
-Chris
Good point. We could just borrow the implementation from 2.6 and in fact just import print from future on 2.6. Just a thought... Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma Sent from Norman, Oklahoma, United States
Christopher Barker wrote:
Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5)
That's actually how it works as far as I can tell. The thing with removing those print is that we can do it without too much trouble. As long as we cannot actually test any py3k code, warnings from python 2.6 is all we can get. I think we should aim at getting "something" which builds and runs (even if does not go further than import stage), so we can gradually port. For now, porting py3k is this huge thing that nobody can work on for say one hour. I would like to make sure we get at that stage, so that many people can take part of it, instead of the currently quite few people who are deeply intimate with numpy. David
David Cournapeau wrote:
Christopher Barker wrote:
Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5)
That's actually how it works as far as I can tell. The thing with removing those print is that we can do it without too much trouble. As long as we cannot actually test any py3k code, warnings from python 2.6 is all we can get.
I think we should aim at getting "something" which builds and runs (even if does not go further than import stage), so we can gradually port. For now, porting py3k is this huge thing that nobody can work on for say one hour. I would like to make sure we get at that stage, so that many people can take part of it, instead of the currently quite few people who are deeply intimate with numpy.
One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems. Cython already contains the start of an implementation of this on NumPy's behalf (though it would need to be improved as one does no longer know the characteristics of the client but must implement the whole API). I think it would be better if I spend my time on Cython, but I'm willing to mentor anybody who'd like to do this in NumPy as I've worked a lot with PEP 3118. (OTOH Travis wrote that PEP :-) ). -- Dag Sverre
On Fri, Apr 24, 2009 at 4:49 PM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
David Cournapeau wrote:
Christopher Barker wrote:
Though I'm a bit surprised that that's not how the print function is written in the first place (maybe it is in py3k -- I'm testing on 2.5)
That's actually how it works as far as I can tell. The thing with removing those print is that we can do it without too much trouble. As long as we cannot actually test any py3k code, warnings from python 2.6 is all we can get.
I think we should aim at getting "something" which builds and runs (even if does not go further than import stage), so we can gradually port. For now, porting py3k is this huge thing that nobody can work on for say one hour. I would like to make sure we get at that stage, so that many people can take part of it, instead of the currently quite few people who are deeply intimate with numpy.
One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems.
Yes, this could be done independently. I am not familiar with PEP 3118; from the python-dev ML, it looks like the current buffer API has some serious shortcomings, I don't whether this implies to numpy or not. Do you have more on this ? Another relatively independent thing is to be able to bootstrap our build from py3k. At least, distutils and the code for bootstrapping, so that we can then run 2to3 on the source code from distutils. Not being able to bootstrap our build process under py3k from distutils sounds too much of a pain. The only real alternative I could see is to have two codebases, because 2to3 does not seem able to convert numpy.distutils 100 % automatically. David
David Cournapeau wrote:
On Fri, Apr 24, 2009 at 4:49 PM, Dag Sverre Seljebotn
One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems.
Yes, this could be done independently. I am not familiar with PEP 3118; from the python-dev ML, it looks like the current buffer API has some serious shortcomings, I don't whether this implies to numpy or not. Do you have more on this ?
Not sure what you refer to ... I'll just write more and hope it answers your question. The difference with PEP 3118 is that many more memory models are supported (beyond 1D contiguous). All of NumPy's strided arrays are very easy to expose (after all, Travis Oliphant did the PEP), with no information lost (i.e. the dtype, shape and strides can be communicated). This means that one should in Python 3/2.6+ be able to use other CPython libraries (like image libraries etc.) in a seamless fashion with NumPy arrays without those libraries having to know about NumPy as such; they can simply request a strided view of the data through PEP 3118. To support clients one mainly has to copy out information that is already there into a struct when requested. The one small challenge is creating a format string for the buffer dtype (which is incompatible with the current string representations of dtype). In addition it would be natural to act as a client, so that calling "np.array(obj)" (and/or np.view?) would acquire the data through PEP 3118. There is a class of buffers which doesn't fit in NumPy's memory model (e.g. pointers to rows of a matrix) and for which a copy would have to be made, but a lot of them could be used through a direct view as well. Dag Sverre
On Mon, May 4, 2009 at 1:24 AM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
David Cournapeau wrote:
On Fri, Apr 24, 2009 at 4:49 PM, Dag Sverre Seljebotn
One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems.
Yes, this could be done independently. I am not familiar with PEP 3118; from the python-dev ML, it looks like the current buffer API has some serious shortcomings, I don't whether this implies to numpy or not. Do you have more on this ?
Not sure what you refer to ...
http://mail.python.org/pipermail/python-dev/2009-April/088211.html Thank you for those information. I don't understand what is meant by "not implemented for multi-dimensional array", and the consequences for numpy. Does it mean that PEP 3118 is not fully implemented ? Is the status of the buffer interface the same for python 2.6 and python 3 ?
In addition it would be natural to act as a client, so that calling "np.array(obj)" (and/or np.view?) would acquire the data through PEP 3118.
Yes, it would help making sure we implement the interface correctly for once :) I am almost done having a numpy.distutils which can bootstrap itself to the point of converting the rest of the python code to py3k. With the buffer interface, this should enable moving forward in a piecewise manner. thank you, David
David Cournapeau wrote:
On Mon, May 4, 2009 at 1:24 AM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
David Cournapeau wrote:
On Fri, Apr 24, 2009 at 4:49 PM, Dag Sverre Seljebotn
One thing somebody *could* work on rather independently for some hours is proper PEP 3118 support, as that is available in Python 2.6+ as well and could be conditionally used on those systems.
Yes, this could be done independently. I am not familiar with PEP 3118; from the python-dev ML, it looks like the current buffer API has some serious shortcomings, I don't whether this implies to numpy or not. Do you have more on this ?
Not sure what you refer to ...
http://mail.python.org/pipermail/python-dev/2009-April/088211.html
Thank you for those information.
I don't understand what is meant by "not implemented for multi-dimensional array", and the consequences for numpy. Does it mean that PEP 3118 is not fully implemented ? Is the status of the buffer interface the same for python 2.6 and python 3 ?
The "memoryview" is not implemented on 2.6, but that's just a utility for being able to acquire a buffer and inspect it from Python-space. From Cython or C one still has access. I think this just refers to there not being any multidimensional consumers nor exporters in the standard library. So from the point of view of the standard library it is a bit useless; but it is not if one uses 3rd party libraries. The API itself is working fine, and you can e.g. export a multidimensional buffer and use it in Cython (defined __getbuffer__ for a cdef class and then access it through classname[dtype, ndim=...]), under both 2.6 and 3.0. Dag Sverre
On Thu, Apr 23, 2009 at 09:52, David Cournapeau <cournape@gmail.com> wrote:
On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip]
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed
Print is not removed, just changed to a function. So,
print("foo")
Yes, as reduce, they are still available, but not as builtins anymore. But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
Doesn't the 2to3 tool do this conversion for you? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Thu, Apr 23, 2009 at 15:39, Robert Kern <robert.kern@gmail.com> wrote:
On Thu, Apr 23, 2009 at 09:52, David Cournapeau <cournape@gmail.com> wrote:
On Thu, Apr 23, 2009 at 11:20 PM, Pauli Virtanen <pav@iki.fi> wrote:
Thu, 23 Apr 2009 22:38:21 +0900, David Cournapeau kirjoitti: [clip]
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to
keep
one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed
Print is not removed, just changed to a function. So,
print("foo")
Yes, as reduce, they are still available, but not as builtins anymore. But replacing print is not as easy as reduce. Things like print "yoyo", a do not work, for example.
Doesn't the 2to3 tool do this conversion for you?
It does for Print at least. conditional compiling for C would work. I could see extending it to the python code as well. I don't know how feasible this is, but have a script that runs at installation and cuts out blocks of code that aren't compatible with the version it's being installed on. for example, run the script on a file like below, and keep only what is useable by the version installed. import foo ###py2x.begin import py2xbar.py ###py2x.end ###py3k.begin import py3kbar.py ###py3k.end -just my .02 -Ross
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Robert Kern wrote:
Doesn't the 2to3 tool do this conversion for you?
I have not seen it handling everything. It does not handle reduce, for example: print reduce(lambda x, y: x+y, [1, 2, 3]) is translated to print(reduce(lambda x, y: x + y, [1, 1, 1])) I guess an alternative to a compat module would be to extend 2to3, or maybe mixing the two approaches. My understanding is that 2to3 is "optimized" to get a single codebase for python 2.6 and 3.x, but we want to be compatible below (>=2.4). David
David Cournapeau wrote:
Hi,
I looked more in detail on what would be needed to port numpy to py3k. In particular, I was interested at the possible strategies to keep one single codebase for both python 2.x and python 3.x. The first step is to remove all py3k warnings reported by python 2.6. A couple of recurrent problems - reduce is removed in py3k - print is removed - some exceptions constructions are not allowed
I think a first step would be to add a compatibility module to remove any use of print/reduce in the numpy codebase (should not be too difficult). For reduce, we could import functools from python code (functools is not available in python 2.4). Unless someone has an objection, I will import functools code, make it compile for python 2.4 and above, and remove all builtin reduce calls in numpy.
Another problem is related to nose: there is an experimental branch for nose which supports py3k, but nose <= 1.0 will not support py3k. The nose author intend to support py3k in a version > 1.0, at which point he will only support python 2.6 and above. I don't know what to do with this (include our own nose in numpy for version < 2.6 - giving up support for python < 2.6 does not sound like a realistic option in the next few years).
cheers,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, It is good news that nose has some Py3K support. Hopefully all NumPy tests will work with all versions of nose as Python 2.6 is meant to be backwards compatible. I agree that a single code base should be used if possible however there are a lot of C code changes required as well as Python code changes: http://www.scipy.org/Python3k http://jarrodmillman.blogspot.com/2009/01/when-will-numpy-and-scipy-migrate-... Are the C code changes independent of the warnings provided by Python 2.6 -3 option and eventually the 2to3 tool? Bruce
On Thu, Apr 23, 2009 at 11:21 PM, Bruce Southey <bsouthey@gmail.com> wrote:
I agree that a single code base should be used if possible however there are a lot of C code changes required as well as Python code changes: http://www.scipy.org/Python3k http://jarrodmillman.blogspot.com/2009/01/when-will-numpy-and-scipy-migrate-...
Yes, the C code is most likely the big work. That's another reason why I have started to split the C code into smaller files which don't include each other - we can port one file after the other. And even though porting C code is obviously more difficult, handling thing with one codebase should be easier in C with conditional compilation, because it is "just" API changes, not language changes.
Are the C code changes independent of the warnings provided by Python 2.6 -3 option and eventually the 2to3 tool?
I don't think there is any support for C code with the 2to3 tool. The following website has interesting links: http://pvergain.wordpress.com/py3k/ In particular, this one experience from Martin Loewis on porting some codebase with C: http://mail.python.org/pipermail/python-porting/2008-December/000010.html David
participants (9)
-
Bruce Southey -
Christopher Barker -
Dag Sverre Seljebotn -
David Cournapeau -
David Cournapeau -
Pauli Virtanen -
Robert Kern -
ross smith -
Ryan May