Mailman 3 April 2021 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Using a phrase "Similar to Pandas DataFrame" in other projects
by Niranda Perera April 16, 2021

April 16, 2021

Hi all, We are working on a distributed memory parallel runtime, "Cylon" ( https://github.com/cylondata/cylon). Cylon has a different execution model compared to Dask Distributed or Ray DF. It is using the bulk synchronous parallel model for execution. So, essentially, Cylon provides a distributed DataFrame for MPI-like environments. In our upcoming release, we have added a Python DataFrame API "Similar to Pandas DataFrame". When we checked the Pandas License it has the clause, "prohibits others from using the name of the project or its contributors to promote derived products without written consent." On the other hand, we have seen third party projects listed under "Ecosystem" on the Pandas website ( https://pandas.pydata.org/community/ecosystem.html) So, we were wondering, 1. What is the process to get the "written consent" to use Pandas name in Cylon project material (documentation, GitHub, etc) 2. What are the requirements a project needs to fulfill, to be listed under the Pandas Ecosystem? Thank you Best -- Niranda Perera @n1r44 <https://twitter.com/N1R44> +1 812 558 8884 / +94 71 554 8430 <https://www.linkedin.com/in/niranda>*https://niranda.dev/ <https://niranda.dev/>*

2 2

March 2021 monthly community meeting (today, March 10, UTC 18:00)
by Joris Van den Bossche April 14, 2021

April 14, 2021

Hi all, Late notice, but the next monthly dev call is in a few hours (today, March 10th) at 18:00 UTC (12 am Central). Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. All are welcome to attend! Video Call: https://us02web.zoom.us/j/81542460994?pwd=NktGRGd4aVNYeDZhUi96cVZDeTZmdz09 Minutes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 2

ANN: Pandas 1.2.4 Released
by Simon Hawkins April 14, 2021

April 14, 2021

Hi all, I'm pleased to announce the release of pandas 1.2.4. This is a patch release in the 1.2.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.2/whatsnew/v1.2.4.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.2.4 Or from conda-forge conda install -c conda-forge pandas==1.2.4 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0

Index Constructor Performance
by Brock Mendel April 2, 2021

April 2, 2021

In optimizing the non-cython groupby.apply ( https://github.com/pandas-dev/pandas/issues/40263, https://github.com/pandas-dev/pandas/pull/40171#issuecomment-789116039) code I'm finding that an awful lot of overhead is coming from Index._simple_new*. This email is about what it would take to get rid of that overhead. * Note that the particular code snippet being profiled is chosen to be worst-case for the non-cython path. It ends up creating a _lot_ of very small Index objects. We don't particularly care about this case, but I'm thinking about this as micro-optimization of code that affects just about every use case under the sun. All of the options I have in mind involve moving some of the constructors to cython. There is a tradeoff in how invasive that is vs how much perf benefit we gain from it. For a baseline, we can trim 10-13% off the benchmark linked above by implementing in cython and mixing into NumericIndex (implementation abbreviated for brevity; the full implementation is 65 lines in cython): ``` @cython.freelist(32) cdef class NumpyIndex: cdef: public ndarray _data @classmethod def _simple_new(cls, values, name=None): ... cpdef NumpyIndex _getitem_slice(cls, slice slobj): ... ``` 10-13% is pretty good, but this only affects Int64Index, UInt64Index, and Float64Index. See Appendix 1 for discussion of what it would take to extend this to other subclasses. To get much further than this would require using __cinit__, which (absent some gymnastics) would require the FooIndex.__new__ methods to behave a lot more like the existing FooIndex._simple_new methods. TL;DR: this really isn't feasible absent a) refactoring RangeIndex to not subclass Int64Index (easy) and b) breaking API changes on the constructors for affected Index subclasses (hard). Appendix 1: Extending to Other Subclasses a) mixing libindex.NumpyIndex into pd.Index doesn't work because ExtensionIndex._data is not an ndarray. AFAICT to get the performance benefit for object-dtype would require implementing a separate subclass e.g. ObjectIndex. b) RangeIndex would not benefit, but something similar could be done for it following https://github.com/cython/cython/issues/4040 (or if we basically re-implement range ourselves in cython) c) MultiIndex could be made to benefit from this by changing ._codes to be a 2D ndarray instead of a FrozenList of ndarrays. This actually would allow for some nice cleanups in MultiIndex. The downside is that the memory footprint may be bigger with mismatched level sizes. d) With modest additional effort, this can be extended to DTI/TDI/PI/CategoricalIndex. Appendix 2: __cinit__ __cinit__ gets called implicitly before __init__ or __new__, and with whatever arguments are passed to init/new, i.e. we can't do validation before passing arguments like we could with an explicit super().__init__(...) call. For NumpyIndex we _could_ define __cinit__ without breaking the world, but we wouldn't get much use out of it unless we also tightened what we accept in the constructor Appendix 3: Notes on cython-related constraints - We cannot mix a cython cdef class into pd.Index because that will break 3rd party subclasses that use object.__new__(cls) (in particular im thinking of xarray's CFDatetimeIndex) - a python class cannot inherit from two separate cython cdef classes. i.e. if we mix something into NumericIndex, that precludes mixing something else into Int64Index

2 1