Mailman 3 July 2020 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Help speeding up altered groubpy.value_counts
by fuller.evan＠gmail.com July 28, 2020

July 28, 2020

All, I'm a new contributor to pandas and have been working to fix a couple bugs with the value_counts methods (pull request https://github.com/pandas-dev/pandas/pull/33652). I'm looking for a bit of help in maintaining speedy performance for https://github.com/DataInformer/pandas-1/blob/value_counts_normalize/pandas/ core/groupby/generic.py The SeriesGroupBy.value_counts method required significant rewrite in order to achieve correct behavior with dropna and normalize. After fixing that, I was asked to run performance tests, which unfortunately do show a significant performance hit for that method. I have been looking at how to close that gap as much as possible, but I've found only a few minor tweaks. When I do cProfile, I don't notice any clear offenders: numpy array functions are taking a lot of time total (numpy.core._multiarray_umath.implement_array_function), but I don't see any particular functions that are slow. Similarly, timeit experiments suggest that array concatenation is relatively slow, but not much different than other options like appending in the next function (e.g. I can do something like np.diff(np.nonzero(np.r_[changes, True])) or np.diff(np.nonzero(changes), append=len(changes)) there's not much of a timing difference). I have tried to do as little as possible with the multiindex, rebuilding it at the end. I would welcome any help or suggestions for how to make SeriesGroupBy.value_counts faster. Thanks, Evan Fuller

1 0

ANN: Pandas 1.1.0 released
by Tom Augspurger July 28, 2020

July 28, 2020

Hi all, I'm pleased to announce the release of pandas 1.1.0. This is a minor release which includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the whatsnew <https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.1.0 Or from conda-forge conda install -c conda-forge pandas==1.1.0 Thanks to all of the 368 contributors <https://pandas.pydata.org/docs/whatsnew/v1.1.0.html#contributors> who made this release possible. Tom

1 0

ANN: Pandas 1.1.0rc0 released
by Tom Augspurger July 19, 2020

July 19, 2020

Hi all, I'm pleased to announce that pandas 1.1.0rc0 is now available for testing. This is the first release candidate for pandas 1.1.0. The release can be installed from conda-forge conda create -n pandas-1.1.0rc0 -c conda-forge/label/pandas_rc -c conda-forge pandas=1.1.0rc0 Or from PyPI python -m pip install --pre pandas==1.1.0rc0 The release notes are available at https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html. Please report any issues with the release on the pandas issue tracker https://github.com/pandas-dev/pandas/issues! We plan to release 1.1.0 in the next few weeks. Thanks, Tom

1 1

ROADMAP: add consistent missing values for all dtypes to the roadmap
by Joris Van den Bossche July 10, 2020

July 10, 2020

Hi all, This is a heads up to the mailing list that I opened a PR with a roadmap addition: https://github.com/pandas-dev/pandas/pull/35208 The PR proposes to add the long term goal of consistent missing value handling for all dtypes (started with the discussion and implementations around pd.NA / nullable dtypes) to the roadmap. Feedback welcome! Best, Joris

1 0

July Call Follow-Up
by Brock Mendel July 9, 2020

July 9, 2020

Would keeping pyarrow optional allow us to bump the minimum version more aggressively than if it becomes required?

4 3