Mailman 3 March 2021 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

March 2021 monthly community meeting (today, March 10, UTC 18:00)
by Joris Van den Bossche April 14, 2021

April 14, 2021

Hi all, Late notice, but the next monthly dev call is in a few hours (today, March 10th) at 18:00 UTC (12 am Central). Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. All are welcome to attend! Video Call: https://us02web.zoom.us/j/81542460994?pwd=NktGRGd4aVNYeDZhUi96cVZDeTZmdz09 Minutes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 2

Index Constructor Performance
by Brock Mendel April 2, 2021

April 2, 2021

In optimizing the non-cython groupby.apply ( https://github.com/pandas-dev/pandas/issues/40263, https://github.com/pandas-dev/pandas/pull/40171#issuecomment-789116039) code I'm finding that an awful lot of overhead is coming from Index._simple_new*. This email is about what it would take to get rid of that overhead. * Note that the particular code snippet being profiled is chosen to be worst-case for the non-cython path. It ends up creating a _lot_ of very small Index objects. We don't particularly care about this case, but I'm thinking about this as micro-optimization of code that affects just about every use case under the sun. All of the options I have in mind involve moving some of the constructors to cython. There is a tradeoff in how invasive that is vs how much perf benefit we gain from it. For a baseline, we can trim 10-13% off the benchmark linked above by implementing in cython and mixing into NumericIndex (implementation abbreviated for brevity; the full implementation is 65 lines in cython): ``` @cython.freelist(32) cdef class NumpyIndex: cdef: public ndarray _data @classmethod def _simple_new(cls, values, name=None): ... cpdef NumpyIndex _getitem_slice(cls, slice slobj): ... ``` 10-13% is pretty good, but this only affects Int64Index, UInt64Index, and Float64Index. See Appendix 1 for discussion of what it would take to extend this to other subclasses. To get much further than this would require using __cinit__, which (absent some gymnastics) would require the FooIndex.__new__ methods to behave a lot more like the existing FooIndex._simple_new methods. TL;DR: this really isn't feasible absent a) refactoring RangeIndex to not subclass Int64Index (easy) and b) breaking API changes on the constructors for affected Index subclasses (hard). Appendix 1: Extending to Other Subclasses a) mixing libindex.NumpyIndex into pd.Index doesn't work because ExtensionIndex._data is not an ndarray. AFAICT to get the performance benefit for object-dtype would require implementing a separate subclass e.g. ObjectIndex. b) RangeIndex would not benefit, but something similar could be done for it following https://github.com/cython/cython/issues/4040 (or if we basically re-implement range ourselves in cython) c) MultiIndex could be made to benefit from this by changing ._codes to be a 2D ndarray instead of a FrozenList of ndarrays. This actually would allow for some nice cleanups in MultiIndex. The downside is that the memory footprint may be bigger with mismatched level sizes. d) With modest additional effort, this can be extended to DTI/TDI/PI/CategoricalIndex. Appendix 2: __cinit__ __cinit__ gets called implicitly before __init__ or __new__, and with whatever arguments are passed to init/new, i.e. we can't do validation before passing arguments like we could with an explicit super().__init__(...) call. For NumpyIndex we _could_ define __cinit__ without breaking the world, but we wouldn't get much use out of it unless we also tightened what we accept in the constructor Appendix 3: Notes on cython-related constraints - We cannot mix a cython cdef class into pd.Index because that will break 3rd party subclasses that use object.__new__(cls) (in particular im thinking of xarray's CFDatetimeIndex) - a python class cannot inherit from two separate cython cdef classes. i.e. if we mix something into NumericIndex, that precludes mixing something else into Int64Index

2 1

welcome new pandas committer
by Jeff Reback March 30, 2021

March 30, 2021

Patrick, Your contributions and help to others is amazing. We'd love to have you as a pandas committer and would like to welcome to the core team! Here is a wiki that shows a little about how to engage as a maintainer: https://github.com/pandas-dev/pandas/wiki/Maintainer-Overview Jeff

1 0

Re: [Pandas-dev] Future of NumPy (and pandas) scalar "promotion" (and concatenate-dtype)
by Sebastian Berg March 10, 2021

March 10, 2021

On Wed, 2021-03-10 at 10:52 +0100, Ralf Gommers wrote: > On Tue, Mar 9, 2021 at 8:10 PM Sebastian Berg < > sebastian(a)sipsolutions.net> > wrote: <snip> > > This all sounds quite good and desirable to me (assuming it's > introduced > with the required care, as Wes pointed out). Sure, there are many details I am slowly hitting. E.g. maybe its just too noisy to attempt to change ufunc calls, and the NumPy comparisons (mainly `==`, but that inherits to the others) is bad, which we probably need to fix, without any real transition, etc.: https://github.com/numpy/numpy/issues/10322 > > In general the other DType will have to decide what to do by > > implementing the correct promotion/common (concatenate) DType > > rules. My > > aim is that through "common DType" + casting rules, we can also > > allow > > things code like this (which would use the identical logic): > > > > uint8.astype(AbstractFloating) -> float64 > > > > So, it isn't quite true that we blindly convert ahead of time. That > > would not always be correct, for example `timedelta64 / 2` cannot > > conver the `2` the same way `timedelta64 + 2` might attempt to do > > it. > > > > Can you elaborate on "it isn't quite true"? It seems to me like the > conversion is the same (step 1) but it's a two-step process: > > 1. `2` is treated as the weak dtype AbstractInteger > 2. The `/` operator is true division and produces floating-point > dtype as > output > > Step 2 here is true independent of whether, for `timedelta64 / x`, x > is a > Python scalar or an array. Yes, what I mean is that you cannot do it in a ufunc-agnistoic pre- processing step (in the sense of "common DType"). But rather, it has to happen in the ufunc specific promotion step, in particular, there is no generic rule that will work for timedelta64 and all ufuncs. Cheers, Sebastian PS: Anyway, we are getting beyond my initial intention of just a small probing appetite and asking for any important insights that I might be completely missing. So I will try to steer any further things back to NumPy when they come in. > > Is that correct, or am I missing some subtlety? > > Cheers, > Ralf

1 0

Future of NumPy (and pandas) scalar "promotion" (and concatenate-dtype)
by Sebastian Berg March 9, 2021

March 9, 2021

Hi all, Summary/Abstract: I am seriously exploring the idea of modifying the NumPy promotion rules to drop the current value-based logic. This would probably affect pandas in a similar way as it does NumPy, so I am wondering what your opinion is on the "value-based" logic and potential "future" logic. One of the most annoying things is likely the transition phase (see the last part about the many warnings I see in the pandas test-suit). ** Long Story: ** I am wondering about the future of type promotion in NumPy [1], but this would probably just as much affect pandas. The problem is what to do with things like: np.array([1, 2], dtype=np.uint8) + 1000 Where the result is currently upcast to a `uint16`. The rules for this are pretty arcane, however. There are a few "worse" things that probably do not affect pandas as much. That is, the above does also happen in this case: np.array([1, 2], dtype=np.uint8) + np.int64(1000) Even though int64 is explicitly typed, we just drop that information. The weirdest things are probably regarding float precision: np.array([0.3], dtype=np.float32) == 0.3 np.array([0.3], dtype=np.float32) == np.float64(0.3) Where the latter would probably go from `True` to `False` due to the limited precision of float32. (At least unless we explicitly try to counteract this for comparisons.) ** Solution: ** The basic idea right now is the following: 1. All objects with NumPy dtypes use those strictly. Scalars or 0-D arrays will have no special handling. 2. Python integers, float, and complex are considered to have a special "weak" dtype. In the above example `1000` or `0.3` would simply be force-cast to `uint8` or `float32`. (Potentiality with a warning/error for integer-rollover) 3. The "additional" rule that all function calls use `np.asarray()`, which convert Python types. That is `np.add(uint8_arr, 1000)` would return the same as `np.add(uint8_arr, np.array(1000))`, while `uint8_arr + 1000` would not! (I am not sure about this rule, it could be modified but it seems easier to limit the "special behaviour" to Python operators) I did some initial trials with such behaviour. Although without issuing transition warnings for the "weak" logic (although I expect it rarely changes the result), but issuing warnings when Point 1. probably makes a difference. To my surprise the SciPy test suite did not even Notice! The pandas test suit runs into thousands of warnings (but few or no errors). Probably mostly due to test that effectively check ufuncs with: binary_ufunc(typed_arr, 3) NumPy does that a lot in its test suite as well. Maybe we can deal with it or rethink rule 3. Cheers, Sebastian [1] I have a conundrum. I don't really want to change things right now, but I need to reimplement it. Preserving value-based logic seems tricky to do without introducing technical debt that if we want to get rid of it later anyway...

3 5

ANN: Pandas 1.2.3 Released
by Simon Hawkins March 2, 2021

March 2, 2021

Hi all, I'm pleased to announce the release of pandas 1.2.3. This is a patch release in the 1.2.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.2/whatsnew/v1.2.3.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.2.3 Or from conda-forge conda install -c conda-forge pandas==1.2.3 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0