<div dir="ltr"><div><br></div><div>(Accidentally dropped the mailing list)</div><div><div class="gmail_quote"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 7:17 PM Brock Mendel <<a href="mailto:jbrockmendel@gmail.com" target="_blank">jbrockmendel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>> You have no problem with changing the behavior of NaT, or changing to use pd.NA?</div><div><br></div><div>If/when we get to a point where we propagate NAs in all other comparisons, I would have no problem with editing `NaT.__richcmp__` to match that convention.<br></div></div></blockquote><div><br></div><div>What are the advantages of a NaT with NA-like comparison semantics over using NA<br>(or NA[datetime])?<br><br>1. Retain dtype in array - scalar ops with a scalar NA<br>2. ...</div><div>3. Less disruptive than changing to NA<br></div><div><br>My ... could include things like `isinstance(NaT, Timestamp)` being true and<br>`NaT.<attr>` for Timestamp attributes. But those don't strike me as necessarily<br>good things. They seem sometimes useful and sometimes harmful.<br><br>The downside of changing NaT in comparison operations are<br><br>1. We're diverging from `np.NaT`. I don't know how problematic this actually is.<br>2. It's a special case. Should users need to know that datelikes use their own<br> NA value because the underlying storage is able to store them "in-band"<br> rather than as a mask? My gut reaction is "no, users shouldn't be exposed to<br> this."<br>3. Changing NaT would leave just NaN with the "always unequal in comparisons"<br> behavior.<br><br>Thus far, I see three options going forward<br><br>1. Use NaN for floats, NaT for datelikes, NA for other.<br> 1-a: Leave NaT with always unequal<br> 1-b: Change NaT to have NA-like comparison behavior<br>2. Use NA everywhere (no NaN for float, no NaT for datelike<br>3. Implement a typed `NA<T>`, where we have an `NA` per dtype.<br><br>Option 3 I think solves the array - scalar op issue. It's more complex for users<br>though hopefully not too complex? My biggest worry is that it makes the<br>implementation much more complex, though perhaps I'm being pessimistic.<br><br>On balance, I'm not sure where I come down yet. Good news: we can take time to<br>figure this out :)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 10:06 AM Tom Augspurger <<a href="mailto:tom.augspurger88@gmail.com" target="_blank">tom.augspurger88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 11:58 AM Brock Mendel <<a href="mailto:jbrockmendel@gmail.com" target="_blank">jbrockmendel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>> or changing the behavior of NaT in comparisons to be like NA.</div><div><br></div><div>Pending the kinks being worked out of pd.NA, I have no problem with that.<br></div></div></blockquote><div><br></div><div><div>You have no problem with changing the behavior of NaT, or changing to use pd.NA?</div><div><br></div><div>Is changing the defined behavior of NaT even an option? Is it defined in a spec</div><div>like NaN, or did NumPy just choose that behavior?<br></div><div><br></div><div>Assuming NaT had NA-like behavior in comparisons, what's remaining arguments for keeping NaT?</div><div>Preserving dtypes in scalar - array ops? Anything else?</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 9:55 AM Tom Augspurger <<a href="mailto:tom.augspurger88@gmail.com" target="_blank">tom.augspurger88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Is NaT defined to be unequal in all comparisons, just like NaN? I think the goal of propagating NA</div><div>requires either using NA or changing the behavior of NaT in comparisons to be like NA.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 11:50 AM Brock Mendel <<a href="mailto:jbrockmendel@gmail.com" target="_blank">jbrockmendel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>> I think consistently propagating NA in comparison operations is a worthwhile goal.</div><div><br></div><div>That's an argument for having a three-valued bool-dtype, not for replacing all other NA-like values.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 8:34 AM Tom Augspurger <<a href="mailto:tom.augspurger88@gmail.com" target="_blank">tom.augspurger88@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">> 2) The "only one NA value is simpler" argument strikes me as a solution in search of a problem.<div><br></div><div>I don't think that's correct. I think consistently propagating NA in comparison operations is a worthwhile goal.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 10:25 AM Brock Mendel <<a href="mailto:jbrockmendel@gmail.com" target="_blank">jbrockmendel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>> It's not fully clear to me what you want to say with this, so a more
detailed clarification is welcome (I mean, I understand the sentence and
remember the discussion, but don't fully understand the point being
made in context, or in what direction you think more discussion is
needed). <br></div><div><br></div><div>I don't particularly think more discussion is needed, as this is a rehash of #28095, where this horse has already been beaten to death.<br></div><div><br></div><div>As Tom noted <a href="https://github.com/pandas-dev/pandas/issues/28095#issuecomment-537501744" target="_blank">here</a>, using pd.NA in places where we currently use NaT breaks the usual identity (that we rely on A LOT)<br></div><div><br></div><div>```<code>(array + array)[0].dtype <=> (array + array[0]).dtype</code>```<br><code></code></div></div><div><br></div><div>(Yes, this holds only imperfectly for NaT because NaT serves as both NotADatetime and NotATimedelta, and I'll refer you to the decomposing horse in #28095.)<br></div><div> </div><div>Also from #28095:</div><div><br></div><div>```Series[timedelta64] * pd.NaT``` unambiguously raises, but ```Series[timedelta64] * pd.NA``` could be timedelta64</div><div><br></div><div>> Assume we introduce a new "nullable datetime" dtype that uses a mask to
track NAs, and can still have NaT in the values. In practice, this still
means that we "replace NaT with NA"</div><div><br></div><div>This strikes me as contradictory. <br></div><div><br></div><div>> So do you mean: "in my opinion, we should not do this" (what I just
described above), because in practice that would mean breaking
arithmetic consistency? Or that if we want to start using NA for
datetimelike dtypes, you think "dtype-parametrized" NA values are
necessary (so you can distinguish NA[datetime] and NA[timedelta] ?)</div><div><br></div><div>I think:</div><div><br></div><div>1) pd.NA solves an _actual_ problem which is that we used to use np.nan in places (categorical, object) where np.nan was semantically misleading. <br></div><div> a) What these have in common is that they are in general non-arithmetic dtypes.</div><div> b) This is an improvement, and I'm glad you put in the effort to make it happen.</div><div> c) Trying to shoe-horn pd.NA into cases where it is semantically misleading based on the Highlander Principle is counter-productive.</div><div><br></div><div>2) The "only one NA value is simpler" argument strikes me as a solution in search of a problem.</div><div> a) All the more so if you want this to supplement np.nan/pd.NaT instead of replace them.</div><div> b) *the idea of replacing vs supplementing needs to be made much more explicit/clear*</div><div><br></div><div>3) The "dtype-parametrized" NA did come up in #28095, but I never advocated it.<br></div><div> a) I am open to separating out a NaTimedelta (xref #24983) from pd.NaT, and don't particularly care what it is called.</div><div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 17, 2020 at 3:37 AM Joris Van den Bossche <<a href="mailto:jorisvandenbossche@gmail.com" target="_blank">jorisvandenbossche@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>> This would also imply creating a nullable float dtype and making
our datelikes use NA rather than NaT too. That seemed to be generally
OK, but wasn't discussed too much.</div><div><br></div><div>My understanding of the discussion is that using a mask on top of datetimelike arrays would not _replace_ NaT, but supplement it with something semantically different. <br></div></blockquote><div><br></div><div>Yes, if we see it similar as NaNs for floats (where NaN is a specific float value in the data array, while NAs are tracked in the mask array), then for datetimelike arrays we can do something similar. And the same discussions about to what extent to distinguish NaN and NA or whether we need to provide options that we are going to have for float dtypes, will also be relevant for datetimelike dtypes (but then for NaT and NA). <br></div><div><br></div><div>But note that in practice, I <i>think</i> that the big majority of use cases will mostly use NA and not NaT in the data (eg when reading from files that have missing data). <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Replacing NaT with NA breaks arithmetic consistency, as has been discussed ad nauseum.</div></blockquote></div><div><br></div><div>It's not fully clear to me what you want to say with this, so a more detailed clarification is welcome (I mean, I understand the sentence and remember the discussion, but don't fully understand the point being made in context, or in what direction you think more discussion is needed). <br></div><div><br></div><div>Assume we introduce a new "nullable datetime" dtype that uses a mask to track NAs, and can still have NaT in the values. In practice, this still means that we "replace NaT with NA" (because even though NaT is still possible, I think you would mostly get NAs as mentioned above; eg reading a file would now give NA instaed of NaT). </div><div>So do you mean: "in my opinion, we should not do this" (what I just described above), because in practice that would mean breaking arithmetic consistency? Or that if we want to start using NA for datetimelike dtypes, you think "dtype-parametrized" NA values are necessary (so you can distinguish NA[datetime] and NA[timedelta] ?)</div><div><br></div><div>Joris<br></div><div><br></div><div><br></div></div>
_______________________________________________<br>
Pandas-dev mailing list<br>
<a href="mailto:Pandas-dev@python.org" target="_blank">Pandas-dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/pandas-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/pandas-dev</a><br>
</blockquote></div></div>
_______________________________________________<br>
Pandas-dev mailing list<br>
<a href="mailto:Pandas-dev@python.org" target="_blank">Pandas-dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/pandas-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/pandas-dev</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</div></div></div>