Mailman 3 July 2011 - NumPy-Discussion

NEPaNEP lessons - was: alterNEP
by Matthew Brett 05 Jul '11

05 Jul '11

Hi, These are some notes about the masking NEP discussion, in the hope that they will be useful for thinking about the NEP and other discussions in the future. This is not a discussion about the masking API. I'm trying not to mix the mask discussion with the discussion-about-the-mask-discussion. Maybe some Points of agreement =============== 1) Blame the process not the people 2) It is useful to go back over controversial incidents to see how the current process works and how it can be improved. I believe that we as a community should aim for the discussion matching the following distinctive features of successful organizations [1] A) Calm, patient, ego-free leadership styles B) Vigorous discussion followed by group agreement C) Brutal realism There was considerable disagreement and some bad feeling during the discussion. The main axis of disagreement was between the alterNEP (Nathaniel, Matthew) and NEP camps (Mark). This was and is Mark's NEP, Mark is implementing it, and so Mark is and was in charge. Mark, as the person implementing, and in charge, has full authority on the final features. Now for areas of: Fruitful disagreement =============== Here I'm going to express my opinions in the hope that it is helpful. The main area for reflection should be the discussion of the alterNEP / NEP discussion. In the spirit of the Toyota 5 whys? [2] - let's ask why? Statement: the alterNEP / NEP discussion failed and led to bad feeling. Why1: Because we the aNEP party believed were were not being heard Why2: Because, when we asked for specific feedback, we did not get it Why3: Because the NEP party had in fact already decided to go for the NEP implementation Why4: Because Mark believed that he would lose implementation time by delaying for further discussion Why5: Because there was a belief that implementation was more important than discussion I think why5 are the most important of these. My personal belief here is that we should: a) Strengthen our commitment to full and open discussion before substantial API change. b) Be careful to state the timetable for future NEP discussion. If it is short for some reason, then we should be specific about that. I believe that will have the effect of i) Strengthening our community, so that it will be clear that each person making substantial comment will be fully heard. ii) Improving our code by improving the level of discussion. I think we must be particularly careful to avoid denial-by-delay [3] or the similar denial-by-no-reply, because these are both very toxic to openness and trust in discussion. Best, Matthew [1] http://en.wikipedia.org/wiki/Good_to_Great [2] http://en.wikipedia.org/wiki/5_Whys [3] http://thinkexist.com/quotation/delay_is_the_deadliest_form_of_denial/25352…

3 3

custom atlas
by Neal Becker 05 Jul '11

05 Jul '11

I thought I'd try to speed up numpy on my fedora system by rebuilding the atlas package so it would be tuned for my machine. But when I do: rpmbuild -ba -D 'enable_native_atlas 1' atlas.spec it fails with: res/zgemvN_5000_100 : VARIATION EXCEEDS TOLERENCE, RERUN WITH HIGHER REPS. A bit of googling has not revealed a solution. Any hints?

2 5

Broadcasting shape mismatch exception
by Davide 04 Jul '11

04 Jul '11

Hi, The exception which is currently, (v1.6), raised when two non broadcastable arrays are summed is a ValueError exception. Wouldn't it be better to create a specific exception class, e.g. BroadcastError, to be more specific and give better control in exception catching? Just a suggestion, Davide

2 1

alterNEP - was: missing data discussion round 2
by Matthew Brett 02 Jul '11

02 Jul '11

Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > Anyway, it's pretty clear that in this particular case, there are two > distinct features that different people want: the missing data > feature, and the masked array feature. The more I think about it, the > less I see how they can be combined into one dessert topping + floor > wax solution. Here are three particular points where they seem to > contradict each other: ... [some proposals] In the interest of making the discussion as concrete as possible, here is my draft of an alternative proposal for NAs and masking, based on Nathaniel's comments. Writing it, it seemed to me that Nathaniel is right, that the ideas become much clearer when the NA idea and the MASK idea are separate. Please do pitch in for things I may have missed or misunderstood: ############################################### A alternative-NEP on masking and missing values ############################################### The principle of this aNEP is to separate the APIs for masking and for missing values, according to * The current implementation of masked arrays * Nathaniel Smith's proposal. This discussion is only of the API, and not of the implementation. ************** Initialization ************** First, missing values can be set and be displayed as ``np.NA, NA``:: >>> np.array([1.0, 2.0, np.NA, 7.0], dtype='NA[f8]') array([1., 2., NA, 7.], dtype='NA[<f8]') As the initialization is not ambiguous, this can be written without the NA dtype:: >>> np.array([1.0, 2.0, np.NA, 7.0]) array([1., 2., NA, 7.], dtype='NA[<f8]') Masked values can be set and be displayed as ``np.MASKED, MASKED``:: >>> np.array([1.0, 2.0, np.MASKED, 7.0], masked=True) array([1., 2., MASKED, 7.], masked=True) As the initialization is not ambiguous, this can be written without ``masked=True``:: >>> np.array([1.0, 2.0, np.MASKED, 7.0]) array([1., 2., MASKED, 7.], masked=True) ****** Ufuncs ****** By default, NA values propagate:: >>> na_arr = np.array([1.0, 2.0, np.NA, 7.0]) >>> np.sum(na_arr) NA('float64') unless the ``skipna`` flag is set:: >>> np.sum(na_arr, skipna=True) 10.0 By default, masking does not propagate:: >>> masked_arr = np.array([1.0, 2.0, np.MASKED, 7.0]) >>> np.sum(masked_arr) 10.0 unless the ``propmsk`` flag is set:: >>> np.sum(masked_arr, propmsk=True) MASKED An array can be masked, and contain NA values:: >>> both_arr = np.array([1.0, 2.0, np.MASKED, np.NA, 7.0]) In the default case, the behavior is obvious:: >>> np.sum(both_arr) NA('float64') It's also obvious what to do with ``skipna=True``:: >>> np.sum(both_arr, skipna=True) 10.0 >>> np.sum(both_arr, skipna=True, propmsk=True) MASKED To break the tie between NA and MSK, NAs propagate harder:: >>> np.sum(both_arr, propmsk=True) NA('float64') ********** Assignment ********** is obvious in the NA case:: >>> arr = np.array([1.0, 2.0, 7.0]) >>> arr[2] = np.NA TypeError('dtype does not support NA') >>> na_arr = np.array([1.0, 2.0, 7.0], dtype='NA[f8]') >>> na_arr[2] = np.NA >>> na_arr array([1., 2., NA], dtype='NA[<f8]') Direct assignnent in the masked case is magic and confusing, and so happens only via the mask:: >>> masked_array = np.array([1.0, 2.0, 7.0], masked=True) >>> masked_arr[2] = np.NA TypeError('dtype does not support NA') >>> masked_arr[2] = np.MASKED TypeError('float() argument must be a string or a number') >>> masked_arr.visible[2] = False >>> masked_arr array([1., 2., MASKED], masked=True) See y'all, Matthew

15 45

PyCon DE 2011 - Call for Proposals extended to July 15, 2011
by Mike Müller 02 Jul '11

02 Jul '11

PyCon DE 2011 - Deadline for Proposals extended to July 15, 2011 ================================================================ The deadline for talk proposals is extended to July 15, 2011. You would like to talk about your Python project to the German-speaking Python community? Just submit your proposal within the next two weeks: http://de.pycon.org/2011/speaker/ About PyCon DE 2011 ------------------- The first PyCon DE will be held October 4-9, 2011 in Leipzig, Germany. The conference language will be German. Talks in English are possible. Please contact us for details. The call for proposals is now open. Please submit your talk by June 30, 2011 online. There are two types of talks: standard talks (20 minutes + 5 minutes Q&A) and long talks (45 minutes + 10 minutes Q&A). More details about the call can be found on the PyCon DE website: http://de.pycon.org/2011/Call_for_Papers/ Since the conference language will be German, the call is in German too. PyCon DE 2011 - Neuer Einsendeschluss für Vortragsvorschläge 15.07.2011 ======================================================================= Noch bis zum 15.7.2011 kann jeder, der sich für Python interessiert, einen Vortragsvorschlag für die PyCon DE 2011 einreichen. Es gibt nur zwei Bedingungen: das Thema sollte interessant sein und etwas mit Python zu tun haben. Für die erste deutsche Python-Konferenz sind wir an einer breiten Themenpalette interessiert, die das ganze Spektrum der Entwicklung, Nutzung und Wirkung von Python zeigt. Mögliche Themen sind zum Beispiel: * Webanwendungen mit Python * Contentmanagement mit Python * Datenbankanwendungen mit Python * Testen mit Python * Systemintegration mit Python * Python für große Systeme * Python im Unternehmensumfeld * Pythonimplementierungen (Jython, IronPython, PyPy, Unladen Swallow und andere) * Python als erste Programmiersprache * Grafische Nutzerschnittstellen (GUIs) * Parallele Programmierung mit Python * Python im wissenschaftlichen Bereich (Bioinformatik, Numerik, Visualisierung und anderes) * Embedded Python * Marketing für Python * Python, Open Source und Entwickler-Gemeinschaft * Zukünftige Entwicklungen * mehr ... Ihr Themenbereich ist nicht aufgelistet, wäre aber aus Ihrer Sicht für die PyCon DE interessant? Kein Problem. Reichen Sie Ihren Vortragsvorschlag einfach ein. Auch wir können nicht alle Anwendungsbereiche von Python überschauen. Vortragstage sind vom 5. bis 7. Oktober 2011. Es gibt zwei Vortragsformate: * Standard-Vortrag -- 20 Minuten Vortrag + 5 Minuten Diskussion * Lang-Vortrag -- 45 Minuten Vortrag + 10 Minuten Diskussion Die Vortragszeit wird strikt eingehalten. Bitte testen Sie die Länge Ihres Vortrags. Lassen Sie gegebenenfalls ein paar Folien weg. Die Vortragsprache ist Deutsch. In begründeten Ausnahmefällen können Vorträge auch auf Englisch gehalten werden. Bitte fragen Sie uns dazu. Bitte reichen Sie Ihren Vortrag auf der Konferenz-Webseite http://de.pycon.org bis zum 15.07.2011 ein. Wir entscheiden bis zum 31. Juli 2011 über die Annahme des Vortrags.

1 0

broacasting question
by Thomas K Gamble 01 Jul '11

01 Jul '11

I'm trying to convert some IDL code to python/numpy and i'm having some trouble understanding the rules for boradcasting during some operations. example: given the following arrays: a = array((2048,3577), dtype=float) b = array((256,25088), dtype=float) c = array((2048,3136), dtype=float) d = array((2048,3136), dtype=float) do: a = b * c + d In IDL, the computation is done without complaint and all array sizes are preserved. In ptyhon I get a value error concerning broadcasting. I can force it to work by taking slices, but the resulting size would be a = (256x3136) rather than (2048x3577). I admit that I don't understand IDL (or python to be honest) well enough to know how it handles this to be able to replicate the result properly. Does it only operate on the smallest dimensions ignoring the larger indices leaving their values unchanged? Can someone explain this to me? -- Thomas K. Gamble tkgamble(a)windstream.net

4 9

Re: [Numpy-discussion] An NA compromise idea -- many-NA
by Pierre GM 01 Jul '11

01 Jul '11

Mask an array with NAs? You should be able to, as IGNORE<>NA. Mask an array with a view? That's sharing the data with a different mask, you should be able to, too (np.ma works like that). Sharing mask? That'd be great if we could... That way, there'd be almost nothing left to do to adapt np.ma... On Jul 1, 2011 10:49 PM, "Charles R Harris" <charlesr.harris(a)gmail.com> wrote:

3 2

Re: [Numpy-discussion] Missing/accumulating data
by Joe Harrington 01 Jul '11

01 Jul '11

Mark Wiebe <mwwiebe(a)gmail.com>: > With a non-boolean alpha mask, there's an implication of a > multiplication operator in there somewhere, but with a boolean mask, > the data can be any data whatsoever that doesn't necessarily support > any kind of blending operations. My goal in raising the point is to find a common core that supports everything. The benefit of the np.ma module is that you have traditional numerical routines like median() and mean() that now sensibly handle missing data, plus a data structure (the paired array and mask) that you can use for other things of your own devising. All that has to happen is to allow the sense of the mask to be FALSE = the data are bad, TRUE = the data are good, and allow (not require) the mask to be of any numerical type, or at least of integer type as well as boolean. I believe that with these two basic requirements, everyone's needs can be met. Note that you could still have boolean masks, and could still have the bad=TRUE, good=FALSE of the current np.ma module, if you had a flag to set in the dtype for what sense of the mask you wanted. It could default to the current behavior if that makes people happy/breaks the least code. > For the image accumulation you're describing, I would use either a > structured array with 'color' and 'weight' fields, or have the last > element of the color channel be the weight (like an RGBA image) so > adding multiple weighted images together would add both the colors > and the weights simultaneously, without requiring a ufunc extension > supporting struct dtypes. Well, yes, we can always design a new data structure that meets our needs, and write all the routines that will ever operate on them. But we don't want that. We want to add a feature to the *old* data structure (i.e., a numerical array of the basic data) that makes the standard routines handle missing data sensibly so we don't have to rewrite them to do so. --jh--

5 5

An NA compromise idea -- many-NA
by Dag Sverre Seljebotn 01 Jul '11

01 Jul '11

I propose a simple idea *for the long term* for generalizing Mark's proposal, that I hope may perhaps put some people behind Mark's concrete proposal in the short term. If key feature missing in Mark's proposal is the ability to distinguish between different reason for NA-ness; IGNORE vs. NA. However, one could conceive wanting to track a whole host of reasons: homework_grades = np.asarray([2, 3, 1, EATEN_BY_DOG, 5, SICK, 2, TOO_LAZY]) Wouldn't it be a shame to put a lot of work into NA, but then have users to still keep a seperate "shadow-array" for stuff like this? a) In this case the generality of Mark's proposal seems justified and less confusing to teach newcomers (?) b) Since Mark's proposal seems to generalize well to many NAs (there's 8 bits in the mask, and millions of available NaN-s in floating point), if people agreed to this one could leave it for later and just go on with the proposed idea. I don't think we should scetch out the above in more detail now, I don't want to distract, I just thought it something to consider to resolve the current situation... FWIW, Dag Sverre

4 10

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2
by Pierre GM 01 Jul '11

01 Jul '11

On Jul 1, 2011 7:14 PM, "Mark Wiebe" <mwwiebe(a)gmail.com> wrote: > > On Fri, Jul 1, 2011 at 10:15 AM, Nathaniel Smith <njs(a)pobox.com> wrote: >> >> On Fri, Jul 1, 2011 at 7:09 AM, Mark Wiebe <mwwiebe(a)gmail.com> wrote: >> > On Fri, Jul 1, 2011 at 6:58 AM, Matthew Brett <matthew.brett(a)gmail.com> >> > wrote: >> >> Do you see problems with the alterNEP proposal? >> > >> > Yes, I really like my design as it stands now, and the alterNEP removes a >> > lot of the abstraction and interoperability that are in my opinion the best >> > parts. I've made more updates to the NEP based on continuing feedback, which >> > are part of the pull request I want reviews for. >> > >> >> >> >> If so, what are they? >> > >> > Mainly: Reduced interoperability, more complex implementation (leading to >> > more bugs), and an unclear theoretical model for the masked part of it. >> >> Can you give any examples of situations where one would run into this >> "reduced interoperability"? I'm not sure what it means. The only >> person who has so far spoken up as needing both masking semantics and >> NA semantics -- Gary Strangman -- has said that he strongly prefers >> the alterNEP semantics *exactly because* it makes it clear *how these >> functions will interoperate.* > > > I've given examples before, but here are a few: > > 1) You're using NA dtypes. You realize you want multiple views of the same data with different choices of NA. You switch to masked arrays with a few lines of code changes. Multiple NAs? AFAIU, there's only one NA (per type) but several choices to allocate a IGNORE depending on the situation. > 2) You're using masks. You realize that you will save memory/disk space if you switch to NA dtypes, and it's possible because it turned out that while you thought you would need masking, you came up with a new algorithm that didn't require it. Ok, your IGNOREs become N'as because you want to... > 3) You're writing matplotlib, and you want to support all forms of NA-style data. You write it once instead of twice. Repeat for all other open source libraries that want to do this. You switch your NAs to IGNOREs, ok, your call again. > >> >> Can you give any examples of how the implementation would be more >> complicated? As far as I can tell there are no elements in the >> alterNEP that are not in your NEP, they mostly just expose the >> functionality differently at the top level. > > > If that is the case, then it should be easy to change to your model after the implementation is complete. I'm happy with that, these style of design choices are easier to make when you're comparing actual usage than hypotheticals. > >> Do you have a clearer theoretical model for the masked part of your >> proposal? > > > Yes, exactly the same model used for NA dtypes. > >> >> The best I've been able to extract from any of your messages >> is when you wrote "it seems to me that people wanting masked arrays >> want missing data without touching their data". But as a matter of >> English grammar, I have no idea what this means -- if you have data, >> it's not missing! > > > Ok, missing data-like functionality, which is provided by the solid theory behind the missing data. Which is a subset of 'masked/to ignore' data... >> >> It seems to me that people wanting masked data want >> to *hide* parts of their data, which seems much clearer to me and is >> the theoretical model used in the alterNEP. > > > Once you've hidden it, isn't it now missing? Only temporarily, you can revert to not hidden when needed. If a data is flagged as NA, it should never be accessible again. >> >> Note that this model >> actually predicts several of the differences between how people want >> masks to work and how people want NAs to work (e.g., their behavior >> during reduction); I > > >> >> >> Do you agree that the alterNEP proposal is easier to understand? >> > >> > No. >> >> >> >> If not, can you explain why? >> > >> > My answers to that are already scattered in the emails in various places, >> > and in the various rationales and justifications provided in the NEP. >> >> I understand the desire not to get caught up in spending all your time >> writing emails explaining things that you feel like you've already >> explained. >> >> Maybe there's an email I missed somewhere where you explain the >> conceptual model behind your NEP's semantics in a short, >> easy-to-understand way (comparable to, say, the Rationale section of >> the alterNEP). But I haven't seen it and I can't reconstruct a >> rationale for it myself (the alterNEP comes out of my attempts to do >> so!). > > > I've been repeatedly updating the NEP. In particular this "round 2" email was an attempt to clarify between the two missing data models (what's being called NA and IGNORE), and the two implementation techniques (NA bit patterns and masks). I've argued that these are completely independent from each other. > >> >> >> What do you see as the important points of difference between the NEP >> >> and the alterNEP? >> > >> > The biggest thing is the NEP supports more use cases in a clean way by >> > composition of different simpler components. It defines one clear missing >> > data abstraction, and proposes two implementations that are interchangeable >> > and can interoperate. >> >> But the two implementations in your proposal are not interchangeable! >> The whole justification for starting with a masked-based >> implementation in your proposal is that it supports unmasking via >> views; if that requirement were removed, then there would be no reason >> to bother with the masking-based implementation at all. > > > They are interchangeable 100% with regard to the missing data semantics. Views are an orthogonal feature, and it is through composition of these two features that the masks gain this power. I'll check your code, but conceptually, NAs and IGNOREs are NOT interchangeable. >> >> Well, that's not true. There are some marginal advantages in the >> special case of working with integers+NAs. But I don't think anyone's >> making that argument. >> >> > The alterNEP proposes two independent APIs, reducing >> > interoperability and so significantly increasing the amount of learning >> > required to work with both of them. This also precludes switching between >> > the two approaches without a lot of work. >> >> You can't switch between Python and C without a lot of work too, but >> that doesn't mean that they should be merged into one design... but >> they do complement each other beautifully. Just like missing data and >> masked arrays :-). > > > This last statement is why I feel like you haven't been reading my emails. I've clearly positioned masks as an implementation technique, not implying any specific semantics. > >> >> >> > The current pull request that's sitting there waiting for review does not >> > have an impact on which approach goes ahead, but the code I'm doing now >> > does. This is a fairly large project, and I don't have a great length of >> > time to do it in, so I'm not going to participate extensively in the >> > alterNEP discussion. If you want to help me, please review my code and >> > provide specific feedback on my NEP (the code review system in github is >> > great for this too, I've received some excellent feedback on the NEP that >> > way). If you want to change my mind about things, please address the >> > specific design decisions you think are problematic by specifically >> > responding to lines in the NEP, as part of code-reviewing my pull request in >> > github. >> >> I know I'm being grumpy in this email, and I apologize for that. But, >> no. I've given extensive feedback, read the list carefully, and >> thought hard about these issues, and so far you've basically just >> dismissed my concerns. (See, e.g., [1], where your response to "we >> have to choose whether it's possible to recover data after it has been >> masked/NAed/whatever" is "no we don't, it should be both possible and >> impossible", which, I mean, what?) I've done my best to express them >> clearly, in the best way I know how -- and that way is *not* line by >> line comments on your NEP, because my concerns are more fundamental >> than that. > > > I've likewise read your emails carefully, and really appreciated that you jumped in right at the beginning with a good explanation of R's missing value semantics. I think line by line comments on the NEP expressing where the fundamental problems would help us communicate better. I've tried to tease apart the distinction between the missing value abstractions and the implementation techniques, and I haven't seen the fact that you read that reflected in your emails. If you have a good reason why implementing something with masks implies certain semantics, please explain, dealing with the points that I've laid out arguing for this design choice in the latest NEP, accessible via the pull request. > >> I am of course happy to answer questions and such if there are places >> where I've been unclear. >> >> And of course it's your prerogative to decide how you want to spend >> your time (well, yours and your employer's, I guess), which forums you >> want to participate in, what code you want to write, etc. If you have >> decided that you are tired to talking about this and want to just go >> off and implement something, then good luck (and I do mean that, it >> isn't sarcasm). > > > I do want to constructively engage the community at the same time as I do the implementation, and I have a track record of producing good interfaces even when the underlying functionality is complex. I've had very positive feedback about einsum from people who deal with multiple arrays of multidimensional data and were missing an easy way to do that kind of operation. > >> But as far as I can tell right now, every single person who has >> experience with handling missing data for statistical purposes (esp. >> in R) has real concerns about your proposal, and AFAICT the community >> has very much *not* reached consensus on how these features should >> look. So I guess my question is, once you've spent your limited time >> on writing this code -- how confident are you that it will be merged? >> This isn't a threat or anything, I have no power over what gets >> merged, but -- it seems to me that there's a real chance that you'll >> do this work and then it will go down in flames, or that it will be >> merged and then the people you're trying to target will ignore it >> anyway. This is why we try to build consensus first, right? I would >> love to find some way to make everyone happy (and have been doing what >> I can on that front), but right now I am not happy, other people are >> not happy, and you're communicating that you don't think that matters. >> I'd love for that to change. > > > Building consensus is general virtually impossible, I'm for example very impressed with the C++ standards committee's success in achieving it where they have. My development process is different from what you're describing, Like with datetime, I am merging periodically, not doing one big merge at the end. There's a reason why design by committee is frowned upon. The feedback is great, but still needs to go through a very strict software design quality filter. > > -Mark > >> >> >> -- Nathaniel >> >> [1] http://mail.scipy.org/pipermail/numpy-discussion/2011-June/057274.html >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion >

1 0