Hello, At one time Guido mentioned adding a built-in product() function to cover some of the remaining use cases of the built-in reduce(). I don't know if this function is still wanted or needed, but I've created an implementation with tests and documentation at http://bugs.python.org/issue1093 . If it is still wanted, could someone review it and give me feedback on it? Thanks, -- ===== --Ryan E. Freckleton
>> At one time Guido mentioned adding a built-in product() function to >> cover some of the remaining use cases of the built-in reduce(). Martin> What is the use case for product()? As I recall, there were basically two uses of reduce(), to sum a series or (less frequently) to take the product of a series. sum() obviously takes care of the first use case. product() would take care of the second. Skip
Actually, if you use Google code search, you'll find that multiplying the numbers in a list doesn't have much use at all. After summing numbers, joining strings is by far the most common usage -- which is much better done with the str.join() method. (PS. I rejected the issue; product() was proposed and rejected when sum() was originally proposed and accepted, and I don't see anything to change my mind.) On 9/3/07, skip@pobox.com <skip@pobox.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Ryan Freckleton wrote:
At one time Guido mentioned adding a built-in product() function to cover some of the remaining use cases of the built-in reduce().
Speaking of such things, I was thinking the other day that it might be useful to have somewhere in the stdlib a full set of functions for doing elementwise operations and reductions on the built-in array type. This would make it possible for one to do efficient bulk arithmetic when the need arises from time to time without having to pull in a heavyweight dependency such as Numeric or numpy. -- Greg
On 9/3/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
But what's the point, given that numpy already exists? Wouldn't you just be redoing the work that numpy has already done? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
But what's the point, given that numpy already exists? Wouldn't you just be redoing the work that numpy has already done?
Sometimes I just want to do something simple like adding two vectors together, and it seems unreasonable to add the whole of numpy as a dependency just to get that. Currently Python has built-in ways of doing arithmetic, and built-in ways of storing arrays of numbers efficiently, but no built-in way of doing arithmetic on arrays of numbers efficiently. I'd like to see some of the core machinery of numpy moved into the Python stdlib, and numpy refactored so that it builds on that. Then there wouldn't be duplication. -- Greg
Greg Ewing wrote:
Concur. Array processing would be a very practical addition to the standard library. It's used extensively in engineering, finance, and the sciences. It looks like they may find room in the OLPC XO for key subsets of NumPy and Matplotlib. They want it both as a teaching resource and to optimize their software suite as a whole. If they're successful, we'll have a lot of young pythoneers expecting this functionality. # Steve
On 9/4/07, Steven H. Rogers <steve@shrogers.com> wrote:
I still don't see why the standard library needs to be weighed down with a competitor to numpy. Including a subset of numpy was considered in the past, but it's hard to decide on the right subset. In the end it was decided that numpy is too big to become a standard library. Given all the gyrations it has gone through I definitely believe this was the right decision. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I still don't see why the standard library needs to be weighed down with a competitor to numpy.
The way to get things done efficiently with an interpreted language is for the language or its libraries to provide primitives that work on large chunks of data at once, and can be combined in flexible ways. Python provides many such primitives for working with strings -- the string methods, regexps, etc. But it doesn't provide *any* for numbers, and that strikes me as an odd gap in functionality. What I have in mind would be quite small, so it wouldn't "weigh down" the stdlib. You could think of it as an extension to the operator module that turns it into something useful. :-) And, as I said, if it's designed so that numpy can build on it, then it needn't be competing with numpy.
Including a subset of numpy was considered in the past, but it's hard to decide on the right subset.
What I'm thinking of wouldn't be a "subset" of numpy, in the sense that it wouldn't necessarily share any of the numpy API from the Python perspective. All it would provide is the minimum necessary primitives to get the grunt work done. I'm thinking of having a bunch of functions like add_elementwise(src1, src2, dst, start, chunk, stride) where src1, src2 and dst are anything supporting the new buffer protocol. That should be sufficient to support something with a numpy-like API, I think. -- Greg
By all means do write up a PEP -- it's hard to generalize from that one example. On 9/4/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 9/4/07, Steven H. Rogers <steve@shrogers.com> wrote:
What makes 3.0 so special? Additions to the stdlib can be considered at any feature release. Frankly, 3.0 is already so loaded with new features (and removals) that I'm not sure it's worth pile this onto it. That said, I would much rather argue with a detailed PEP than with yet another suggestion that we do something. I am already doing enough -- it's up for some other folks to get together and produce a proposal. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
The 3.x compatibility break (however alleviated by 2to3) makes a nice clean cutoff. The numpy that works on Pythons 3.x would essentially be a port from the current numpy. Consequently, we could modify the numpy for Pythons 3.x to always rely on the stdlib API to build on top of. We couldn't do that for the version targeted to Pythons 2.x because we could only rely on its presence for 2.6+. I don't mind maintaining two versions of numpy, one for Python 2.x and one for 3.x, but I don't care to maintain three. I invite Greg and Steven and whoever else is interested to discuss ideas for the PEP on numpy-discussion. I'm skeptical, seeing what currently has been suggested, but some more details could easily allay that. http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 9/4/07, Robert Kern <robert.kern@gmail.com> wrote:
I just had a discussion with Glyph "Twisted" Lefkowitz about this. He warns that if every project using Python uses 3.0's incompatibility as an excuse to make their own library/package/project incompatible as well, we will end up with total pandemonium (my paraphrase). I think he has a good point -- we shouldn't be injecting any more instability into the world than absolutely necessary. In any case, the rift is more likely to be between 2.5 and 2.6, since 2.6 will provide backports of most 3.0 features (though without some of the accompanying cleanups, in order to also provide strong backwards compatibility). To be honest, I also doubt the viability of designing and implementing something that would satisfy Greg Ewing's goals *and* be stable enough in the standard library, in under a year. But as I said before, I don't see much point in arguing much further until I see the PEP. I may yet be convinced, but it will have to be a good design and a well-argued proposal. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I agree. I didn't mean to imply that the 3.x version of numpy would be incompatible to users of it, just that the codebase that implements it will be different, whether it is automatically or manually translated. Of course, if the API is introduced in 3.(x>0), we end up with the same problem I wanted to avoid. Ah well. See you on the flip side of the PEP. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Martin v. Löwis wrote:
Travis has actually been working on this off-and-on for the last couple of years, including mentoring an SoC project last year. I believe PEP 3118 (the revised buffer protocol) was one of the major outcomes - rather than having yet-another-array-type to copy data to and from in order to use different libraries, the focus moved to permitting better interoperability amongst the array types that already exist. Once we support better interoperability at the data storage level, it will actually become *more* useful to have a simple multi-dimensional array type in the standard library as you could easily pass those objects to functions from more powerful array manipulation libraries as your needs become more complicated. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
Travis has actually been working on this off-and-on for the last couple of years,
Well, yes, but that's concentrating on a different aspect of things -- the data storage. My proposal concerns what you can *do* with the data, independent of the way it's stored. My idea and Travis's would complement each other, I think. -- Greg
Martin v. Löwis wrote:
I think this requires a PEP, and explicit support from the NumPy people.
Someone who knows more about numpy's internals would be needed to figure out what the details should be like in order to be usable by numpy. But I could write a PEP about how what I have in mind would look from the Python level. -- Greg
>> At one time Guido mentioned adding a built-in product() function to >> cover some of the remaining use cases of the built-in reduce(). Martin> What is the use case for product()? As I recall, there were basically two uses of reduce(), to sum a series or (less frequently) to take the product of a series. sum() obviously takes care of the first use case. product() would take care of the second. Skip
Actually, if you use Google code search, you'll find that multiplying the numbers in a list doesn't have much use at all. After summing numbers, joining strings is by far the most common usage -- which is much better done with the str.join() method. (PS. I rejected the issue; product() was proposed and rejected when sum() was originally proposed and accepted, and I don't see anything to change my mind.) On 9/3/07, skip@pobox.com <skip@pobox.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Ryan Freckleton wrote:
At one time Guido mentioned adding a built-in product() function to cover some of the remaining use cases of the built-in reduce().
Speaking of such things, I was thinking the other day that it might be useful to have somewhere in the stdlib a full set of functions for doing elementwise operations and reductions on the built-in array type. This would make it possible for one to do efficient bulk arithmetic when the need arises from time to time without having to pull in a heavyweight dependency such as Numeric or numpy. -- Greg
On 9/3/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
But what's the point, given that numpy already exists? Wouldn't you just be redoing the work that numpy has already done? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
But what's the point, given that numpy already exists? Wouldn't you just be redoing the work that numpy has already done?
Sometimes I just want to do something simple like adding two vectors together, and it seems unreasonable to add the whole of numpy as a dependency just to get that. Currently Python has built-in ways of doing arithmetic, and built-in ways of storing arrays of numbers efficiently, but no built-in way of doing arithmetic on arrays of numbers efficiently. I'd like to see some of the core machinery of numpy moved into the Python stdlib, and numpy refactored so that it builds on that. Then there wouldn't be duplication. -- Greg
Greg Ewing wrote:
Concur. Array processing would be a very practical addition to the standard library. It's used extensively in engineering, finance, and the sciences. It looks like they may find room in the OLPC XO for key subsets of NumPy and Matplotlib. They want it both as a teaching resource and to optimize their software suite as a whole. If they're successful, we'll have a lot of young pythoneers expecting this functionality. # Steve
On 9/4/07, Steven H. Rogers <steve@shrogers.com> wrote:
I still don't see why the standard library needs to be weighed down with a competitor to numpy. Including a subset of numpy was considered in the past, but it's hard to decide on the right subset. In the end it was decided that numpy is too big to become a standard library. Given all the gyrations it has gone through I definitely believe this was the right decision. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I still don't see why the standard library needs to be weighed down with a competitor to numpy.
The way to get things done efficiently with an interpreted language is for the language or its libraries to provide primitives that work on large chunks of data at once, and can be combined in flexible ways. Python provides many such primitives for working with strings -- the string methods, regexps, etc. But it doesn't provide *any* for numbers, and that strikes me as an odd gap in functionality. What I have in mind would be quite small, so it wouldn't "weigh down" the stdlib. You could think of it as an extension to the operator module that turns it into something useful. :-) And, as I said, if it's designed so that numpy can build on it, then it needn't be competing with numpy.
Including a subset of numpy was considered in the past, but it's hard to decide on the right subset.
What I'm thinking of wouldn't be a "subset" of numpy, in the sense that it wouldn't necessarily share any of the numpy API from the Python perspective. All it would provide is the minimum necessary primitives to get the grunt work done. I'm thinking of having a bunch of functions like add_elementwise(src1, src2, dst, start, chunk, stride) where src1, src2 and dst are anything supporting the new buffer protocol. That should be sufficient to support something with a numpy-like API, I think. -- Greg
By all means do write up a PEP -- it's hard to generalize from that one example. On 9/4/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 9/4/07, Steven H. Rogers <steve@shrogers.com> wrote:
What makes 3.0 so special? Additions to the stdlib can be considered at any feature release. Frankly, 3.0 is already so loaded with new features (and removals) that I'm not sure it's worth pile this onto it. That said, I would much rather argue with a detailed PEP than with yet another suggestion that we do something. I am already doing enough -- it's up for some other folks to get together and produce a proposal. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
The 3.x compatibility break (however alleviated by 2to3) makes a nice clean cutoff. The numpy that works on Pythons 3.x would essentially be a port from the current numpy. Consequently, we could modify the numpy for Pythons 3.x to always rely on the stdlib API to build on top of. We couldn't do that for the version targeted to Pythons 2.x because we could only rely on its presence for 2.6+. I don't mind maintaining two versions of numpy, one for Python 2.x and one for 3.x, but I don't care to maintain three. I invite Greg and Steven and whoever else is interested to discuss ideas for the PEP on numpy-discussion. I'm skeptical, seeing what currently has been suggested, but some more details could easily allay that. http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 9/4/07, Robert Kern <robert.kern@gmail.com> wrote:
I just had a discussion with Glyph "Twisted" Lefkowitz about this. He warns that if every project using Python uses 3.0's incompatibility as an excuse to make their own library/package/project incompatible as well, we will end up with total pandemonium (my paraphrase). I think he has a good point -- we shouldn't be injecting any more instability into the world than absolutely necessary. In any case, the rift is more likely to be between 2.5 and 2.6, since 2.6 will provide backports of most 3.0 features (though without some of the accompanying cleanups, in order to also provide strong backwards compatibility). To be honest, I also doubt the viability of designing and implementing something that would satisfy Greg Ewing's goals *and* be stable enough in the standard library, in under a year. But as I said before, I don't see much point in arguing much further until I see the PEP. I may yet be convinced, but it will have to be a good design and a well-argued proposal. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I agree. I didn't mean to imply that the 3.x version of numpy would be incompatible to users of it, just that the codebase that implements it will be different, whether it is automatically or manually translated. Of course, if the API is introduced in 3.(x>0), we end up with the same problem I wanted to avoid. Ah well. See you on the flip side of the PEP. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Martin v. Löwis wrote:
Travis has actually been working on this off-and-on for the last couple of years, including mentoring an SoC project last year. I believe PEP 3118 (the revised buffer protocol) was one of the major outcomes - rather than having yet-another-array-type to copy data to and from in order to use different libraries, the focus moved to permitting better interoperability amongst the array types that already exist. Once we support better interoperability at the data storage level, it will actually become *more* useful to have a simple multi-dimensional array type in the standard library as you could easily pass those objects to functions from more powerful array manipulation libraries as your needs become more complicated. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
Nick Coghlan wrote:
Travis has actually been working on this off-and-on for the last couple of years,
Well, yes, but that's concentrating on a different aspect of things -- the data storage. My proposal concerns what you can *do* with the data, independent of the way it's stored. My idea and Travis's would complement each other, I think. -- Greg
Martin v. Löwis wrote:
I think this requires a PEP, and explicit support from the NumPy people.
Someone who knows more about numpy's internals would be needed to figure out what the details should be like in order to be usable by numpy. But I could write a PEP about how what I have in mind would look from the Python level. -- Greg
participants (8)
-
"Martin v. Löwis" -
Greg Ewing -
Guido van Rossum -
Nick Coghlan -
Robert Kern -
Ryan Freckleton -
skip＠pobox.com -
Steven H. Rogers