<div dir="ltr"><div>Some answers to previous mails first:<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 17 Feb 2020 at 17:34, Tom Augspurger <<a href="mailto:tom.augspurger88@gmail.com" target="_blank">tom.augspurger88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">> 2) The "only one NA value is simpler" argument strikes me as a solution in search of a problem.<div><br></div><div>I don't think that's correct. I think consistently propagating NA in comparison operations is a worthwhile goal.<br></div></div><br></blockquote><div>Having a single, consistent missing value indicator across all dtypes is <i>for me</i> one of the main drivers that led me to make the pd.NA proposal. <br></div><div>From my personal experience (eg when teaching pandas to beginners), this is an existing problem that complicates things, not one that is being invented.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 10:25 AM Brock Mendel <<a href="mailto:jbrockmendel@gmail.com" target="_blank">jbrockmendel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>> Assume we introduce a new "nullable datetime" dtype that uses a mask to
track NAs, and can still have NaT in the values. In practice, this still
means that we "replace NaT with NA"</div><div><br></div><div>This strikes me as contradictory. <br></div></div></blockquote></div></blockquote><div><br></div><div>I tried to explain this in the next sentence from the original text: "(because even though NaT is still possible, I think you would mostly get
NAs as mentioned above; eg reading a file would now give NA instead of
NaT)." <br></div><div>So assuming you have a masked-array-approach for datetimes, then you can have NaT as a valid datetime value in the values part or NA due to the mask part of the array. In such a case (but this is only an assumption how the extension array <i>could</i> work!), it's the NA that is the main missing value indicator. So if you are creating such a masked datetime-like array with missing values (eg from reading a file), you will get NAs as missing values in this case in contrast to NaTs right now. Hence, in practice we would "replace NaT with NA", although you can still have NaT in the values.</div><div><br></div><div>Note I only started to explain this in response to your initial "using a mask on top of datetimelike arrays would not _replace_ NaT, but supplement it with something semantically different", but maybe I misunderstood your initial comment.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>> So do you mean: "in my opinion, we should not do this" (what I just
described above), because in practice that would mean breaking
arithmetic consistency? Or that if we want to start using NA for
datetimelike dtypes, you think "dtype-parametrized" NA values are
necessary (so you can distinguish NA[datetime] and NA[timedelta] ?)</div><div><br></div><div>I think:</div><div><br></div><div>1) pd.NA solves an _actual_ problem which is that we used to use np.nan in places (categorical, object) where np.nan was semantically misleading. <br></div><div> a) What these have in common is that they are in general non-arithmetic dtypes.</div><div> b) This is an improvement, and I'm glad you put in the effort to make it happen.</div><div> c) Trying to shoe-horn pd.NA into cases where it is semantically misleading based on the Highlander Principle is counter-productive.</div></div></blockquote></div></blockquote><div><br></div><div>With "semantically misleading", I suppose you mean that "Series[Timedelta] + pd.NA" could result both in timedelta or datetime64? <br></div><div><br></div><div>Personally, I don't think this is big problem (or at least I think a single pd.NA brings bigger benefits), but this has indeed already discussed <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>2) The "only one NA value is simpler" argument strikes me as a solution in search of a problem.</div><div> a) All the more so if you want this to supplement np.nan/pd.NaT instead of replace them.</div><div> b) *the idea of replacing vs supplementing needs to be made much more explicit/clear*</div><br></div></blockquote></div></blockquote><div><br></div><div>I thought you were actually advocating for supplementing instead of replacing in your first email ;) (but maybe you were rather trying to summarize the discussion, and not giving an opinion?)</div><div>Anyway, I will open an github issue for float dtypes to further discuss this (I think the float case is the easier to discuss, while the issues are similar to datetime with NaT).<br></div></div></div>