On 12/06/2013 14:10, Nathaniel Smith wrote:

It looks like we've gotten a bit confused and need to untangle something. There's a PR to add new functions 'np.filled' and 'np.filled_like': https://github.com/numpy/numpy/pull/2875 And there was a discussion about this on the list back in January: http://thread.gmane.org/gmane.comp.python.numeric.general/52763

I think a reasonable summary of the opinions in the thread are: - This functionality is great, ... - ...but we can't call it 'np.filled' because there's also 'np.ma.filled' which does something else... - ...but there really aren't any better names... - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'

There where the additional proposal (mostly neglected on the original thread) to add the 'fill' optional parameter to the array constructor: np.ndarray(shape, fill=value) has an obvious meaning to me. I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does. However, I don't know if adding a np.ndarray_like() constructor would make much sense. Cheers, Daniele

On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele@grinta.net> wrote:

There where the additional proposal (mostly neglected on the original thread) to add the 'fill' optional parameter to the array constructor:

np.ndarray(shape, fill=value)

has an obvious meaning to me.

I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does.

It predates numpy and the numpy.ndarray type.

However, I don't know if adding a np.ndarray_like() constructor would make much sense.

Also that. -- Robert Kern

On 12/06/2013 14:29, Robert Kern wrote:

On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele@grinta.net> wrote:

...

There where the additional proposal (mostly neglected on the original thread) to add the 'fill' optional parameter to the array constructor:

np.ndarray(shape, fill=value)

has an obvious meaning to me.

I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does.

It predates numpy and the numpy.ndarray type.

I see, but I'm surprised by the fact that np.empty() is not an alias for np.ndarray() neither is implemented in term of np.ndarray(). Since the constructor exists and it would be the "most pythonic" way of constructing an ndarray object I don't see why it's usage should not be encouraged.

...

However, I don't know if adding a np.ndarray_like() constructor would make much sense.

Also that.

I don't understand if you are saying that adding the fill parameter to np.ndarray() is a good or a bad idea. If it is a bad idea, what are the drawbacks compared to the other two proposed solutions? Cheers, Daniele

On Wed, Jun 12, 2013 at 1:47 PM, Daniele Nicolodi <daniele@grinta.net> wrote:

On 12/06/2013 14:29, Robert Kern wrote:

...

On Wed, Jun 12, 2013 at 1:26 PM, Daniele Nicolodi <daniele@grinta.net> wrote:

...

There where the additional proposal (mostly neglected on the original thread) to add the 'fill' optional parameter to the array constructor:

np.ndarray(shape, fill=value)

has an obvious meaning to me.

I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does.

It predates numpy and the numpy.ndarray type.

I see, but I'm surprised by the fact that np.empty() is not an alias for np.ndarray() neither is implemented in term of np.ndarray().

Since the constructor exists and it would be the "most pythonic" way of constructing an ndarray object I don't see why it's usage should not be encouraged.

...

...

However, I don't know if adding a np.ndarray_like() constructor would make much sense.

Also that.

I don't understand if you are saying that adding the fill parameter to np.ndarray() is a good or a bad idea.

My point there was that the ones()/ones_like(), zeros()/zeros_like() pairings are useful and extending that to empty()/empty_like() is a good thing. Dropping empty() in favor of ndarray() breaks that symmetry, and ndarray_like() doesn't make sense as a name.

If it is a bad idea, what are the drawbacks compared to the other two proposed solutions?

I think it has more drawbacks than the others. ndarray() is quite low-level. There is very little intelligence in it, which is a good thing. It is used in places where that low-level capability is necessary, e.g. in the memmap implementation where it is constructing an ndarray object that points to specific memory. array(), empty_like(), ones(), etc. are relatively high level and have more intelligence built into them. The "filled" functionality, however you spell it, is part of the latter category. Implementing it in the low level part interferes with ndarray()'s current low-level uses. You *could* drop the current empty() in favor ndarray() if you wanted to as empty() is not so smart and the defaults for the other ndarray() arguments are sensible. But if you extend the functionality of empty() to implement filled(), then that no longer holds. -- Robert Kern

On Wed, Jun 12, 2013 at 10:18 AM, Nathaniel Smith <njs@pobox.com> wrote:

On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.)

Maybe you could unpack this, as I seem to remember this was the option with the most support previously.

Indeed it was, which is why I brought it up :-).

I'm not sure what more there is to unpack, though. It's just... offensive to every sense of API design I have, I don't know how to explain more than I have. I speculate that it's only attraction is that it showed up at the end of a 50 email thread and offered the promise of ending things, but I don't know.

Well, here's maybe another way of getting at the ugliness.

Here's the current doc page listing all the options for creating an array -- a very useful reference, esp. while learning: http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html

Now imagine a new version of this page, if we add 'filled'. There will be a list at the top with functions named: empty filled ones zeros It's immediately obvious what all of these things do, and how they differ from each other, and in which situation you might want each, just from the names, even before you read the one-line synopses. Even more so if you know about the existence of np.fill(). The synopses for 'ones' and 'zeros' don't even have to change, they already use the word 'filled' to describe what they do. It all just works.

Now imagine a different new version of this page, if we overload 'empty' to add a fill= option. I don't even know how we document that on this page. The list will remain: empty ones zeros So there will be no clue there how you get an array filled with NaNs or whatever, or even any hint that it's possible. Then there's the prose on the right. Right now the synopsis for 'empty' is: Return a new array of given shape and type, without initializing entries. I guess we change this to Return a new array of given shape and type, without initializing entries, OR return a new array of given shape and type, with values initialized to some specific value. ? IMO probably the single best criterion for judging whether your API is good, is whether you can write clean and pretty docs for it. This fails that test horribly...

We probably should advertise the ndarray constructor more, and possibly make it more generally useful, but the current situation for better or worse is that we've spent many years telling people that it's a weird low-level thing that they shouldn't use. (I didn't even know how it worked until 10 minutes ago!) Adding this functionality there means it'll still be hidden away, so it's not a great solution to the 'filled' problem, and it doesn't really move us any closer to having a coherent story on when you should use the ndarray constructor either.

So IMO the best (least bad) solution on offer is still to just add a 'filled' function, and live with the np.ma inconsistency.

Another idea (also imperfect): call the new functions `filledwith` and `filledwith_like`. Not as concise as `filled`, but the meaning is still clear, and it avoids the clash with `ma.filled`. Warren

-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 12, 2013 at 10:45 AM, Warren Weckesser <warren.weckesser@gmail.com> wrote:

On Wed, Jun 12, 2013 at 10:18 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

On Wed, Jun 12, 2013 at 1:28 PM, Matthew Brett <matthew.brett@gmail.com> wrote:

...

On Wed, Jun 12, 2013 at 1:10 PM, Nathaniel Smith <njs@pobox.com> wrote:

...

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.)

Maybe you could unpack this, as I seem to remember this was the option with the most support previously.

Indeed it was, which is why I brought it up :-).

I'm not sure what more there is to unpack, though. It's just... offensive to every sense of API design I have, I don't know how to explain more than I have. I speculate that it's only attraction is that it showed up at the end of a 50 email thread and offered the promise of ending things, but I don't know.

Well, here's maybe another way of getting at the ugliness.

Here's the current doc page listing all the options for creating an array -- a very useful reference, esp. while learning: http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html

Now imagine a new version of this page, if we add 'filled'. There will be a list at the top with functions named: empty filled ones zeros It's immediately obvious what all of these things do, and how they differ from each other, and in which situation you might want each, just from the names, even before you read the one-line synopses. Even more so if you know about the existence of np.fill(). The synopses for 'ones' and 'zeros' don't even have to change, they already use the word 'filled' to describe what they do. It all just works.

Now imagine a different new version of this page, if we overload 'empty' to add a fill= option. I don't even know how we document that on this page. The list will remain: empty ones zeros So there will be no clue there how you get an array filled with NaNs or whatever, or even any hint that it's possible. Then there's the prose on the right. Right now the synopsis for 'empty' is: Return a new array of given shape and type, without initializing entries. I guess we change this to Return a new array of given shape and type, without initializing entries, OR return a new array of given shape and type, with values initialized to some specific value. ? IMO probably the single best criterion for judging whether your API is good, is whether you can write clean and pretty docs for it. This fails that test horribly...

We probably should advertise the ndarray constructor more, and possibly make it more generally useful, but the current situation for better or worse is that we've spent many years telling people that it's a weird low-level thing that they shouldn't use. (I didn't even know how it worked until 10 minutes ago!) Adding this functionality there means it'll still be hidden away, so it's not a great solution to the 'filled' problem, and it doesn't really move us any closer to having a coherent story on when you should use the ndarray constructor either.

So IMO the best (least bad) solution on offer is still to just add a 'filled' function, and live with the np.ma inconsistency.

+2 on this. I like a good name like `filled` for a function that I expect to use regularly. it's short and descriptive. Josef

...

Another idea (also imperfect): call the new functions `filledwith` and `filledwith_like`. Not as concise as `filled`, but the meaning is still clear, and it avoids the clash with `ma.filled`.

Warren

...

-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 12, 2013 at 4:30 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

On 6/12/2013 10:49 AM, Pierre Haessig wrote:

...

I find the "docs would look nice and consistent" argument pretty convincing. +1 for np.filled.

In Maple and Mathematica, these are called constant arrays. Thus (?) another possible name is: np.constant

This came up in the previous thread as well... the problem is that 'constant' has another even stronger meaning (in computing and science!) of 'fixed', 'unchanging', 'read-only' (cf. C's 'const', and the scipy.constants module). And numpy arrays do in fact have a flag that makes them read-only. But this would not be the function that sets it. -n

On 12 June 2013 17:29, <josef.pktd@gmail.com> wrote:

+2 on this. I like a good name like `filled` for a function that I expect to use regularly. it's short and descriptive.

FTIW, I am for this one too. That is not only clear, but a name that one may try even before checking the docs. Emtpy-filled sounds too much of an oximoron to me.

On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs@pobox.com> wrote:

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. ... deprecate np.ma.filled in favor of masked_array.filled (which does exactly the same thing) and eventually switch np.ma.filled to be consistent with the new np.filled.

+1

I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does.

I had always assumed that np.ndarray() was a "low-level" interce that you really don't want to use in regular code (maybe for subclassing array...), as the docs say: """ Arrays should be constructed using `array`, `zeros` or `empty` (refer to the See Also section below). The parameters given here refer to a low-level method (`ndarray(...)`) for instantiating an array. """ Am I wrong? is there any reason )other than history to have np.empty() But in any case, I like np.filled(), as being analogous to ones(), zeros() and empty()... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 12 Jun 2013 18:20, "Ralf Gommers" <ralf.gommers@gmail.com> wrote:

On Wed, Jun 12, 2013 at 6:36 PM, Chris Barker - NOAA Federal <

chris.barker@noaa.gov> wrote:

...

On Wed, Jun 12, 2013 at 5:10 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic.

I agree.

...

... deprecate np.ma.filled

Please don't. Rather just live with the inconsistency between numpy and numpy.ma APIs. If that bothers you, just tell yourself that we'll get an NA

Sounds like we're pretty much reaching consensus. Phew. dtype at some point and that that will make numpy.ma much less important:) Oh, I do tell myself that :-). With my committer/consensus-building hat on, np.ma has users, so I want something they can live with, and was suggesting some options. For myself I don't really care what np.ma does though since I don't use it...

...

in favor

...

of masked_array.filled (which does exactly the same thing) and eventually switch np.ma.filled to be consistent with the new np.filled.

+1

...

I also don't really see why an np.empty() constructor exists, it seems to do the same thing that np.ndarray() does.

I had always assumed that np.ndarray() was a "low-level" interce that you really don't want to use in regular code (maybe for subclassing array...), as the docs say:

""" Arrays should be constructed using `array`, `zeros` or `empty` (refer to the See Also section below). The parameters given here refer to a low-level method (`ndarray(...)`) for instantiating an array. """

Am I wrong? is there any reason )other than history to have np.empty()

But in any case, I like np.filled(), as being analogous to ones(), zeros() and empty()...

I like np.filled as well. np.fill_with sounds fine too.

Grammatically, fill_with is an imperative, which suggests it needs an array to operate on; it's synonymous with just plain 'fill'. Having 'fill' and 'fill_with' as different functions with different semantics would be pretty confusing! -n

On 2013/06/12 2:10 AM, Nathaniel Smith wrote:

Hi all,

It looks like we've gotten a bit confused and need to untangle something. There's a PR to add new functions 'np.filled' and 'np.filled_like': https://github.com/numpy/numpy/pull/2875 And there was a discussion about this on the list back in January: http://thread.gmane.org/gmane.comp.python.numeric.general/52763

I think a reasonable summary of the opinions in the thread are: - This functionality is great, ... - ...but we can't call it 'np.filled' because there's also 'np.ma.filled' which does something else... - ...but there really aren't any better names...

How about 'np.initialized'?

- ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'

In the mean time the original submitter has continued puttering along polishing the original patch, and it's ready to merge... except it's still the original interface, somehow the thread discussion and the PR discussion never met up.

So, we have to decide what to do.

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.) So I think we should just merge the PR as is. The only downside is the np.ma inconsistency, but, np.ma is already inconsistent (cf. masked_array.fill versus masked_array.filled!), somewhat deprecated,

"somewhat deprecated"? Really? Since when? By whom? Replaced by what?

and AFAICT there are far more people who will benefit from a clean np.filled idiom than who actually use np.ma (and in particular its fill-value functionality). So there would be two

I think there are more np.ma users than you realize. Everyone who uses matplotlib is using np.ma at least implicitly, if not explicitly. Many of the matplotlib examples put np.ma to good use. np.ma.filled is an essential long-standing part of the np.ma API. I don't see any good rationale for generating a conflict with it, when an adequate non-conflicting alternative ('np.initialized', maybe others) exists. Eric

bad-but-IMHO-acceptable options: either live with an inconsistency between np.filled and np.ma.filled, or deprecate np.ma.filled in favor of masked_array.filled (which does exactly the same thing) and eventually switch np.ma.filled to be consistent with the new np.filled.

But, that's just my opinion.

-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Hi, On Wed, Jun 12, 2013 at 7:49 PM, Eric Firing <efiring@hawaii.edu> wrote:

On 2013/06/12 4:18 AM, Nathaniel Smith wrote:

...

Now imagine a different new version of this page, if we overload 'empty' to add a fill= option. I don't even know how we document that on this page. The list will remain: empty ones zeros

Opposite of "empty": "full". So that is another non-conflicting alternative to my earlier suggestion of "initialized".

I like 'full'. 'filledwith' is a little ugly, but still a whole lot better than creating a clash with ma.filled, Cheers, Matthew

On 2013/06/12 8:13 AM, Warren Weckesser wrote:

That's why I suggested 'filledwith' (add the underscore if you like). This also allows a corresponding masked implementation, 'ma.filledwith', without clobbering the existing 'ma.filled'.

Consensus on np.filled? absolutely not, you do not have a consensus. np.filledwith or filled_with: fine with me, maybe even with everyone--let's see. I would prefer the underscore version. Eric

On Wed, Jun 12, 2013 at 12:00 PM, Phil Hodge <hodge@stsci.edu> wrote:

On 06/12/2013 02:55 PM, Eric Firing wrote:

I would interpret np.filled as a test, asking whether the array is filled. If the function is supposed to do something related to assigning values, the name should be a verb.

or a noun: ones, zeros -- but I agree, better not an adjective. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas < aldcroft@head.cfa.harvard.edu> wrote:

On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On 2013/06/12 8:13 AM, Warren Weckesser wrote:

...

That's why I suggested 'filledwith' (add the underscore if you like). This also allows a corresponding masked implementation, 'ma.filledwith', without clobbering the existing 'ma.filled'.

Consensus on np.filled? absolutely not, you do not have a consensus.

np.filledwith or filled_with: fine with me, maybe even with everyone--let's see. I would prefer the underscore version.

+1 on np.filled_with. It's unique the meaning is extremely obvious. We do use np.ma.filled in astropy so a big -1 on deprecating that (which would then require doing numpy version checks to get the right method). Even when there is an NA dtype the numpy.ma users won't go away anytime soon.

I like np.filled_with(), but just to be devil's advocate, think of the syntax: np.filled_with((10, 24), np.nan) As I read that, I am filling the array with (10, 24), not NaNs. Minor issue, for sure, but just thought I raise that. -1 on deprecation of np.ma.filled(). -1 on np.filled() due to collision with np.ma (both conceptually and programatically). np.values() might be a decent alternative. Cheers! Ben Root

On 6/13/2013 4:36 PM, Benjamin Root wrote:

np.values() might be a decent alternative.

This could then reasonably support broadcasting from the shape of the input to the shape of the array. Alan Isaac

On Thu, Jun 13, 2013 at 5:06 PM, <josef.pktd@gmail.com> wrote:

On Thu, Jun 13, 2013 at 4:47 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On 2013/06/13 10:36 AM, Benjamin Root wrote:

...

On Thu, Jun 13, 2013 at 9:36 AM, Aldcroft, Thomas <aldcroft@head.cfa.harvard.edu <mailto:aldcroft@head.cfa.harvard.edu>> wrote:

On Wed, Jun 12, 2013 at 2:55 PM, Eric Firing <efiring@hawaii.edu <mailto:efiring@hawaii.edu>> wrote:

On 2013/06/12 8:13 AM, Warren Weckesser wrote: > That's why I suggested 'filledwith' (add the underscore if you like). > This also allows a corresponding masked implementation, 'ma.filledwith', > without clobbering the existing 'ma.filled'.

Consensus on np.filled? absolutely not, you do not have a

consensus.

...

np.filledwith or filled_with: fine with me, maybe even with everyone--let's see. I would prefer the underscore version.

+1 on np.filled_with. It's unique the meaning is extremely obvious. We do use np.ma.filled in astropy so a big -1 on deprecating that (which would then require doing numpy version checks to get the right method). Even when there is an NA dtype the numpy.ma <http://numpy.ma> users won't go away anytime soon.

I like np.filled_with(), but just to be devil's advocate, think of the syntax:

np.filled_with((10, 24), np.nan)

As I read that, I am filling the array with (10, 24), not NaNs. Minor issue, for sure, but just thought I raise that.

-1 on deprecation of np.ma.filled(). -1 on np.filled() due to collision with np.ma <http://np.ma> (both conceptually and programatically).

np.values() might be a decent alternative.

Cheers! Ben Root

Even if he is representing the devil, Ben raises a good point. To summarize, the most recent set of suggestions that seem not to have been completely shot down include:

np.filled_with((10, 24), np.nan) np.full((10, 24), np.nan) # analogous to np.empty np.values((10, 24), np.nan) # seems clear, concise np.initialized((10, 24), np.nan) # a few more characters, but # seems clear to me.

Personally, I like all of the last three better than the first.

What about: np.filled_array((10, 24), np.nan) If I just saw np.values(..) in some code I would never guess what it is doing from the name, and beyond that I would guess that I can put in multiple "values" in the second arg. np.initialized() is more obvious to me. - Tom

np.values sounds also good to me, a noun like np.ones, np.nans, np.infs, np.zeros

I don't like np.initialized because empty also initializes and array (although) an empty one.

Josef

...

Eric

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Jun 12, 2013 at 8:00 PM, Phil Hodge <hodge@stsci.edu> wrote:

I would interpret np.filled as a test, asking whether the array is filled. If the function is supposed to do something related to assigning values, the name should be a verb.

That's a plausible convention, but it's not the convention that actually exists for these sorts of functions. 'sorted' gives you an object with the property of being sorted, it doesn't test for sortedness. 'reversed' gives you an object with the property of being reversed, it doesn't test for reversedness. Tests should be named like 'is_sorted'. Conceptually, this function doesn't assign values -- for that we have np.fill(), which is a side-effecting operation that fills an existing array with some value, and does indeed have a verb as a name. The function we're talking about is like 'sorted' in that it isn't side-effecting, but instead gives you an object that has a particular property. -n

On Fri, Jun 14, 2013 at 6:18 PM, Eric Firing <efiring@hawaii.edu> wrote:

On 2013/06/14 5:15 AM, Alan G Isaac wrote:

...

On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:

...

If I just saw np.values(..) in some code I would never guess what it is doing from the name

That suggests np.fromvalues. But more important than the name I think is allowing broadcasting of the values, based on NumPy's broadcasting rules. Broadcasting a scalar is then a special case, even if it is the case that has dominated this thread.

True, but this looks to me like mission creep. All of this fuss is about replacing two lines of user code with a single line. If it can't be kept simple, both in implementation and in documentation, it shouldn't be done at all. I'm not necessarily opposed to your suggestion, but I'm skeptical.

It's another two-liner: [~] |1> x = np.empty([3,4,5]) [~] |2> x[...] = np.arange(5) [~] |3> x array([[[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]], [[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]], [[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]]]) It's wafer-thin! -- Robert Kern

On Fri, Jun 14, 2013 at 1:21 PM, Robert Kern <robert.kern@gmail.com> wrote:

On Fri, Jun 14, 2013 at 6:18 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On 2013/06/14 5:15 AM, Alan G Isaac wrote:

...

On 6/14/2013 9:27 AM, Aldcroft, Thomas wrote:

...

If I just saw np.values(..) in some code I would never guess what it is doing from the name

That suggests np.fromvalues. But more important than the name I think is allowing broadcasting of the values, based on NumPy's broadcasting rules. Broadcasting a scalar is then a special case, even if it is the case that has dominated this thread.

True, but this looks to me like mission creep. All of this fuss is about replacing two lines of user code with a single line. If it can't be kept simple, both in implementation and in documentation, it shouldn't be done at all. I'm not necessarily opposed to your suggestion, but I'm skeptical.

It's another two-liner:

[~] |1> x = np.empty([3,4,5])

[~] |2> x[...] = np.arange(5)

[~] |3> x array([[[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]],

[[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]],

[[ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.], [ 0., 1., 2., 3., 4.]]])

It's wafer-thin!

True, but wouldn't we rather want to encourage the use of broadcasting in the numerical operations rather than creating new arrays from broadcasted arrays? a = np.arange(5) + np.ones((3, 4, 5)) b = np.filled((3, 4, 5), np.arange(5)) + np.ones((3, 4, 5)) The first one is much easier to read, and is more efficient than the second (theoretical) one because it needs to create two (3, 4, 5) arrays rather than just one. That being said, one could make a similar argument against ones(), zeros(), etc. Cheers! Ben Root

On Fri, Jun 14, 2013 at 1:22 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On 2013/06/12 2:10 AM, Nathaniel Smith wrote:

...

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.) So I think we should just merge the PR as is. The only downside is the np.ma inconsistency, but, np.ma is already inconsistent (cf. masked_array.fill versus masked_array.filled!), somewhat deprecated,

"somewhat deprecated"? Really? Since when? By whom? Replaced by what?

Sorry, not trying to start a fight, just trying to summarize the situation. As far as I can tell:

Oh... (puts away iron knuckles)

Despite heroic efforts on the part of its authors, numpy.ma has a number of weird quirks (masked data can still trigger invalid value errors), misfeatures (hard versus soft masks), and just plain old pain points (ongoing issues with whether any given operation will respect or preserve the mask).

Actually, now that we have a context manager for warning capture, we could actually fix that.

It's been in deep maintenance mode for some time; we merge the occasional bug fix that people send in, and that's it. (To be fair, numpy as a whole is fairly slow-moving, but numpy.ma still gets much less attention.)

Even if there were active maintainers, no-one really has any idea how to fix any of the problems above; they're not so much bugs as intrinsic limitations of the design.

Therefore, my impression is that a majority (not all, but a majority)

of numpy developers strongly recommend against the use of numpy.ma in new projects.

Such a recommendation should be in writing in the documentation and elsewhere. Furthermore, a proper replacement would also be needed. Just simiply deprecating it without some sort of decent alternative leaves everybody in a lurch. I have high hopes for NA to be that replacement, and the sooner, the better.

I could be wrong! And I know there's nothing to really replace it. I'd like to fix that. But I think "semi-deprecated" is not an unfair shorthand for the above.

You will have to pry np.ma from my cold, dead hands! (or distract me with a sufficiently shiny alternative)

(I'll even admit that I'd *like* to actually deprecate it. But what I mean by that is, I don't think it's possible to fix it to the point where it's actually a solid/clean/robust library, so I'd like to reach a point where everyone who's currently using it is happier switching to something else and is happy to sign off on deprecating it.)

As far as many people are concerned, it is a solid, clean, robust library.

...

...

and AFAICT there are far more people who will benefit from a clean np.filled idiom than who actually use np.ma (and in particular its fill-value functionality). So there would be two

I think there are more np.ma users than you realize. Everyone who uses matplotlib is using np.ma at least implicitly, if not explicitly. Many of the matplotlib examples put np.ma to good use. np.ma.filled is an essential long-standing part of the np.ma API. I don't see any good rationale for generating a conflict with it, when an adequate non-conflicting alternative ('np.initialized', maybe others) exists.

I'm aware of that. If I didn't care about the opinions of numpy.ma users, I wouldn't go starting long and annoying mailing list threads about features that are only problematic because of their affect on numpy.ma :-).

But, IMHO given the issues with numpy.ma, our number #1 priority ought to be making numpy proper as clean and beautiful as possible; my position that started this thread is basically just that we shouldn't make numpy proper worse just for numpy.ma's sake. That's the tail wagging the dog. And this 'conflict' seems a bit overstated given that (1) np.ma.filled already has multiple names (and 3/4 of the uses in matplotlib use the method version, not the function version), (2) even if we give it a non-conflicting name, np.ma's lack of maintenance means that it'd probably be years before someone got around to actually adding a parallel function to np.ma. [Unless this thread spurs someone into submitting one just to prove me wrong ;-).]

Actually, IIRC, np.ma does some sort of auto-wrapping of numpy functions. This is why adding np.filled() would cause a namespace clobbering, I think. Cheers! Ben Root

On 2013/06/14 7:22 AM, Nathaniel Smith wrote:

On Wed, Jun 12, 2013 at 7:43 PM, Eric Firing <efiring@hawaii.edu> wrote:

...

On 2013/06/12 2:10 AM, Nathaniel Smith wrote:

...

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.) So I think we should just merge the PR as is. The only downside is the np.ma inconsistency, but, np.ma is already inconsistent (cf. masked_array.fill versus masked_array.filled!), somewhat deprecated,

"somewhat deprecated"? Really? Since when? By whom? Replaced by what?

Sorry, not trying to start a fight, just trying to summarize the situation. As far as I can tell:

Despite heroic efforts on the part of its authors, numpy.ma has a number of weird quirks (masked data can still trigger invalid value errors), misfeatures (hard versus soft masks), and just plain old pain points (ongoing issues with whether any given operation will respect or preserve the mask).

It's been in deep maintenance mode for some time; we merge the occasional bug fix that people send in, and that's it. (To be fair, numpy as a whole is fairly slow-moving, but numpy.ma still gets much less attention.)

Even if there were active maintainers, no-one really has any idea how to fix any of the problems above; they're not so much bugs as intrinsic limitations of the design.

Therefore, my impression is that a majority (not all, but a majority) of numpy developers strongly recommend against the use of numpy.ma in new projects.

I could be wrong! And I know there's nothing to really replace it. I'd like to fix that. But I think "semi-deprecated" is not an unfair shorthand for the above.

(I'll even admit that I'd *like* to actually deprecate it. But what I mean by that is, I don't think it's possible to fix it to the point where it's actually a solid/clean/robust library, so I'd like to reach a point where everyone who's currently using it is happier switching to something else and is happy to sign off on deprecating it.)

Nathaniel, I've been pondering when to bring this up again, but you did it for me, so here it is with a new title for the thread. Maybe it will be short and sweet, maybe not. I think we can agree that there is major interest in having good numpy support for one or more styles of missing/masked values. You might not agree, but I will assert that the style of support provided by np.ma is *very* useful; it serves a real purpose in working code. We do agree that np.ma has problems. It is not at all clear to me, however, that those problems cannot or should not be fixed. Even if they can't, I don't think they are so severe that it is wise to try to kill off np.ma *before* there is a good replacement. In the NA branch, an attempt was made to lay the groundwork for solid missing/masked support. I did not agree with every design aspect, but I thought it was nevertheless good as groundwork, and could be used to greatly improve np.ma, to provide a different style of support for those who require it, and perhaps to lead over the very long term to a withering away of the need for np.ma. Some of the groundwork from the NA branch survived, but most of it is sitting off to the side. Is there any way to revive this line of development? To satisfy the needs of people coming from the R world *and* of people for whom np.ma is, despite its warts, an important tool? This seems to me to be the single biggest area where numpy needs development. It looks like this problem needs dedicated resources: a grant, a major corporate effort, or both. Numpy is central to python in science, but it doesn't seem to have a corresponding level of direction and support. Eric

On Wed, Jun 12, 2013 at 8:10 AM, Nathaniel Smith <njs@pobox.com> wrote:

Hi all,

It looks like we've gotten a bit confused and need to untangle something. There's a PR to add new functions 'np.filled' and 'np.filled_like': https://github.com/numpy/numpy/pull/2875 And there was a discussion about this on the list back in January: http://thread.gmane.org/gmane.comp.python.numeric.general/52763

I think a reasonable summary of the opinions in the thread are: - This functionality is great, ... - ...but we can't call it 'np.filled' because there's also 'np.ma.filled' which does something else... - ...but there really aren't any better names... - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)'

In the mean time the original submitter has continued puttering along polishing the original patch, and it's ready to merge... except it's still the original interface, somehow the thread discussion and the PR discussion never met up.

So, we have to decide what to do.

Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.) So I think we should just merge the PR as is. The only downside is the np.ma inconsistency, but, np.ma is already inconsistent (cf. masked_array.fill versus masked_array.filled!), somewhat deprecated, and AFAICT there are far more people who will benefit from a clean np.filled idiom than who actually use np.ma (and in particular its fill-value functionality). So there would be two bad-but-IMHO-acceptable options: either live with an inconsistency between np.filled and np.ma.filled, or deprecate np.ma.filled in favor of masked_array.filled (which does exactly the same thing) and eventually switch np.ma.filled to be consistent with the new np.filled.

But, that's just my opinion.

-n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Why we need this animal whatever it is called Scientific Python @SciPyTip Create a 2x3 array filled with integer 7's: 7*np.ones((2,3), int) and I just did this yesterday, np.nan * np.ones(3) maybe I should have used np.zeros(3) / 0. Josef

Hi, On Sat, Jun 29, 2013 at 10:37 AM, Nathaniel Smith <njs@pobox.com> wrote:

So this petered off...any objections to np.full?

full and filledwith and filled_with all seem OK to me. On a meta note - anything we can do to stop threads petering off? It seems to happen rather often, Cheers, Matthew

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs@pobox.com> wrote:

So this petered off...any objections to np.full?

How about `np.inited` ? `full` doesn't sound quite right to me. Although I expect once it comes into use everyone will get used to it. <snip> Chuck

On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd@gmail.com> wrote:

On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

So this petered off...any objections to np.full?

No objection.

...

How about `np.inited` ? `full` doesn't sound quite right to me. Although I expect once it comes into use everyone will get used to it.

Is this pronounce like "ignited" ?

I guess ``__init__``-ed. Not very clear I'm afraid. Ralf Josef

German speaker

On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

On 6/29/2013 3:00 PM, Nathaniel wrote:

...

any objections to np.full?

Still curious: why isn't ``tile`` the right name? (It already exists.)

...

...

...

import numpy as np np.tile(3.0, (2,3)) array([[ 3., 3., 3.], [ 3., 3., 3.]])

If someone explained this, sorry to have missed it.

It's implemented inefficiently. It is aimed at a different use case (building up arrays from other arrays) that only incidentally, and thus poorly, fulfils this one. It has no relation to the empty()/ones()/zeros() line of functions. In particular, tile_like() would make no sense. Being aimed at a different use case, it will be more difficult to find for people who are not familiar with numpy's entire API. It will be an isolated idiom rather than a natural completion of the existing APIs. I can probably think of others if I really get going, but I think we're about at the point of diminishing returns. -- Robert Kern On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

On 6/29/2013 3:00 PM, Nathaniel wrote:

...

any objections to np.full?

Still curious: why isn't ``tile`` the right name? (It already exists.)

...

...

...

import numpy as np np.tile(3.0, (2,3)) array([[ 3., 3., 3.], [ 3., 3., 3.]])

If someone explained this, sorry to have missed it.

Alan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

-- Robert Kern

On Sat, Jun 29, 2013 at 10:53 PM, Alan G Isaac <alan.isaac@gmail.com> wrote:

...

...

On 6/29/2013 3:00 PM, Nathaniel wrote:

...

any objections to np.full?

...

On Sat, Jun 29, 2013 at 9:55 PM, Alan G Isaac wrote:

...

Still curious: why isn't ``tile`` the right name? (It already exists.)

...

...

...

...

...

import numpy as np np.tile(3.0, (2,3)) array([[ 3., 3., 3.], [ 3., 3., 3.]])

...

...

If someone explained this, sorry to have missed it.

On 6/29/2013 5:25 PM, Robert Kern wrote:

...

It's implemented inefficiently.

...

It is aimed at a different use case (building up arrays from other arrays) that only incidentally, and thus poorly, fulfils this one.

...

It has no relation to the empty()/ones()/zeros() line of functions. In particular, tile_like() would make no sense.

...

Being aimed at a different use case, it will be more difficult to find for people who are not familiar with numpy's entire API. It will be an isolated idiom rather than a natural completion of the existing APIs.

I do not understand this reply.

1. efficiency: noted in my original post, and I believe it could easily be fixed. Am I wrong?

With more special cases, sure.

2. use case: I assume this is the efficiency argument in a new guise?

No. It does explain the inefficiency, though.

3. grammar: if there were an obvious choice, this discussion would have ceased long ago: ``tile`` is as good as any for this reason,

I'm not referring to the names. ones()/zeros()/empty() are designed and documented for similar use cases. You describe them in very similar ways: "in order to make an array of a given shape and dtype filled with (1s/0s/nothing in particular), use ones()/zeros()/empty()". You should never describe tile() with a similar sentence. It happens to do something similar given a very specific corner case, but that is not what the function does and that is not what it is for.

and I don't see the problem with ``tile_like``.

It makes no sense except in the scalar case.

4. isolated idiom (i.e., 3. again): the only way around this is to use ``constant``, which is in use in other languages. This has been vetoed as confusing terminology. The problems with each other suggestion have been covered extensively in this and related threads.

Again, I'm not talking about the name or our relationship to other languages. Abusing tile() for this use case (which is *not* the use case it is designed and documented for) would make it a numpy idiom, just something that you have to be told and remember, much like the current idiom that we recommend: np.ones(shape) * value It is the complete API of tile() and the fact that it only happens to meet the use case by abusing a small corner of its API that makes it use for this an idiom. No matter what we name the function in the PR, it won't be an idiom in the way I mean here. -- Robert Kern

On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:

On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

So this petered off...any objections to np.full?

No objection.

full full full np.full full of what? full of the number it's filledwith no objections so far, from what I remember of this thread we take an empty array, fill it and then it's "full" Josef

...

...

How about `np.inited` ? `full` doesn't sound quite right to me. Although I expect once it comes into use everyone will get used to it.

Is this pronounce like "ignited" ?

I guess ``__init__``-ed. Not very clear I'm afraid.

Ralf

...

Josef German speaker

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Sat, Jun 29, 2013 at 9:23 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

On Sat, Jun 29, 2013 at 7:15 PM, <josef.pktd@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs@pobox.com> wrote:

...

So this petered off...any objections to np.full?

No objection.

full full full

np.full full of what?

full of the number it's filledwith

no objections so far, from what I remember of this thread

we take an empty array, fill it and then it's "full"

You mean "fulled"? ;)

It's really "full", if you want to put something else in there, you need to replace something that is already in there. Fortunately, arrays are mutable and not "constant" (However np.empty doesn't give you an array that's really empty.) Josef

<snip>

Chuck

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Sun, Jun 30, 2013 at 7:11 PM, Nathaniel Smith <njs@pobox.com> wrote:

On Sun, Jun 30, 2013 at 2:15 AM, <josef.pktd@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 3:00 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 6:43 PM, <josef.pktd@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 12:04 PM, Charles R Harris <charlesr.harris@gmail.com> wrote:

...

On Sat, Jun 29, 2013 at 4:37 AM, Nathaniel Smith <njs@pobox.com>

wrote:

...

...

...

So this petered off...any objections to np.full?

No objection.

full full full

Great, then let's put an end to this! Merged.

Great. Special thanks to Eric Firing for fixing some np.ma bugs in response to a "that module is not maintained" statement. Ralf