Mailman 3 December 2017 - Pandas-dev

Proposal to change the default number of rows for DataFrame display (lower max_rows)
by Joris Van den Bossche March 28, 2018

March 28, 2018

*[Note for those reading it on the pydata mailing list, please answer to pandas-dev(a)python.org <pandas-dev(a)python.org> to keep discussion centralised there]* Hi all, I am reposting the mail of Clemens below, but with slightly changed focus, as I think the main discussion point is about the number of rows. The proposal in https://github.com/pandas-dev/pandas/pull/17023 is to lower the default number of rows shown when displaying a Series or DataFrame from 60 to 20. Thoughts on that? Best, Joris 2017-11-28 11:57 GMT+01:00 Clemens Brunner <clemens.brunner(a)gmail.com>: > Hello! > > We're currently discussing a change in how data frames are displayed by > default in https://github.com/pandas-dev/pandas/pull/17023. There are two > proposed changes: > > (1) Set pd.options.display.max_columns=0 (previously this was set to 20). > (2) Set pd.options.display.max_rows=20 (previously this was set to 60). > > Change (1) means that the number of printed columns is adapted to fit > within the width of the terminal. If there are too many columns, ellipsis > will be shown to indicate collapsed columns in the middle of the data > frame. This doesn't work if Python is run as a Jupyter kernel (e.g. in a > Jupyter notebook or in IPython QtConsole), in which case the maximum > columns remain 20. > > Example: > ======== > import pandas as pd > import numpy as np > pd.DataFrame(np.random.rand(5, 10)) > > Output before (in a terminal with 100 chars width): > --------------------------------------------------- > 0 1 2 3 4 5 6 \ > 0 0.643979 0.690414 0.018603 0.991478 0.707534 0.376765 0.670848 > 1 0.547836 0.810972 0.054448 0.415112 0.268120 0.904528 0.839258 > 2 0.582256 0.732149 0.284208 0.405197 0.213591 0.715367 0.150106 > 3 0.197348 0.317159 0.051669 0.738405 0.821046 0.179270 0.245793 > 4 0.483466 0.583330 0.999213 0.882883 0.315169 0.045712 0.897048 > > 7 8 9 > 0 0.891467 0.494220 0.713369 > 1 0.601304 0.449880 0.266205 > 2 0.113262 0.360580 0.238833 > 3 0.798063 0.077769 0.471169 > 4 0.262779 0.530565 0.992084 > > Output after: > ------------- > 0 1 2 3 ... 6 7 > 8 9 > 0 0.673621 0.211505 0.943201 0.946548 ... 0.900453 0.612182 > 0.861933 0.710967 > 1 0.670855 0.834449 0.796273 0.785976 ... 0.609954 0.686663 > 0.684582 0.837505 > 2 0.544736 0.814827 0.352893 0.459556 ... 0.650993 0.735943 > 0.279110 0.840203 > 3 0.440125 0.554323 0.745462 0.940896 ... 0.544576 0.224175 > 0.852603 0.509837 > 4 0.225551 0.791834 0.476059 0.321857 ... 0.391165 0.423213 > 0.290683 0.954423 > > [5 rows x 10 columns] > > > Change (2) implies fewer rows are displayed before auto-hiding takes > place. I find that 60 rows almost always causes the terminal to scroll > (most terminals have between 25-40 rows), so reducing the value to 20 > increases the chance that a data frame can be observed on one terminal > page. I'm not including a before/after output since it should be easy to > imagine how this change affects the output. > > Both changes would make Pandas behave similar to R's Tidyverse (which I > really like), but this should not be the main reason why these changes are > a good idea. I mainly like them because these settings make (large) data > frames much nicer to look at. > > Note that these changes affect the default values. Of course, users are > free to change them back in their active Python session. > > Comments to both proposed changes are highly welcome (either here on the > mailing list or at https://github.com/pandas-dev/pandas/pull/17023. > > Clemens > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev >

2 3

Proposal to change PR update policy from rebase to merge
by Joris Van den Bossche Jan. 26, 2018

Jan. 26, 2018

Hi all, Currently when PRs get outdated, we often ask to "rebase and update" (although to be honest, it mainly Jeff that is doing most of this work to ping stale PRs), and rebasing is also how it is explained in the docs ( http://pandas-docs.github.io/pandas-docs-travis/contributing.html#creating-…). And also many active contributors use rebasing while working on a PR to get in sync with changes in master. I would like to propose to change this policy from rebasing to merging (= merging master in the feature branch, creating a merge commit). Some reasons for this: - I personally think this is easier to do (certainly for less experienced git users; conflicts can be easier to solve, ..) - It makes it easier to follow what has changed (certainly if we extend 'not rebasing' with 'not squash rebasing'), making it easier review - It doesn't destroy links to github PR comments - Since we squash on merge in the end, we don't care about the additional merge commits in the PR's history Thoughts on this? If we would agree on this, that would mean: update the docs + start doing it ourselves + start asking that of contributors consistently. Regards, Joris

5 9

ANN: Pandas v0.22.0 released
by Tom Augspurger Dec. 31, 2017

Dec. 31, 2017

Hi all, I'm happy to announce pandas 0.22.0 has been released. This is a major release from 0.21.1 and includes a single, API-breaking change. We recommend that all users upgrade to this version after carefully reading the release note. The only changes are: - The sum of an empty or all-*NA* Series is now 0 - The product of an empty or all-*NA* Series is now 1 - We’ve added a min_count parameter to .sum() and .prod() controlling the minimum number of valid values for the result to be valid. If fewer than min_count non-*NA* values are present, the result is *NA*. The default is 0. To return NaN, the 0.21 behavior, use min_count=1. See the pandas 0.22.0 whatsnew <https://pandas.pydata.org/pandas-docs/version/0.22.0/whatsnew.html#whatsnew…> overview for further explanation of all the places in the library this affects. - Tom --- *What is it:* pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *How to get it:* Source tarballs and windows/mac/linux wheels are available on PyPI (thanks to Christoph Gohlke for the Windows wheels, and to Matthew Brett for setting up the Mac / Linux wheels). Conda packages are available on the default and conda-forge channels. *Issues:* Please report any issues on our issue tracker: https://github.com/py data/pandas/issues

1 0

Re: [Pandas-dev] [pydata] Feedback request for return value of empty or all-NA sum (0 or NA?)
by Stephan Hoyer Dec. 21, 2017

Dec. 21, 2017

On Fri, Dec 8, 2017 at 4:17 PM Jeff Reback <jreback(a)yahoo.com> wrote: > From Stephan Hoyer <shoyer(a)gmail.com> > > > Yes, in most cases. But this isn't what skipna=True does, which is > explicitly an indication to skip NaNs. > > Here's where we differ. skipna=True does not mean, let's remove the NaN's > and then compute > the operation, rather it means, ignore the NaN's in computing the > operation. These are distinct > and the crux of NaN propagation. This is simply a practical view of things. > I think "skipping" vs "ignore in the calculation" is too subtle of a distinction to insist on users understanding from a docstring/argument name. Sure one could always mask the NaN's themselves and do anything, but again > I WILL belabor the point. Pandas > is meant to be obvious and sensible. > If nothing else, this debate should make it very clear that there is no single "obvious and sensible" answer to how an empty or all-null sum should work. If it would help, I volunteer to survey my Twitter followers about which behavior they think is obvious ;). The best we can do is consider various use cases and clearly explain our reasoning/decision, with the recognition that it is not possible to satisfy everyone. Finally, we have a very very limited response of users / developers here > (in this thread). I could be completely wrong, > but I suspect many users have been *relatively* happy with pandas choices > over the years. > Rather I would say that most users probably don't actually care about this debate either way. This is edge case behavior that doesn't come up everyday. Cheers, Stephan

5 6

ANN: pandas v0.21.1 released
by Tom Augspurger Dec. 13, 2017

Dec. 13, 2017

Hi all, I'm happy to announce pandas 0.21.1 has been released. This is a minor bug-fix release in the 0.21.x series and includes some small regression fixes, bug fixes and performance improvements. We recommend that all users upgrade to this version. Highlights include: - Temporarily restore matplotlib datetime plotting functionality. This should resolve issues for users who relied implicitly on pandas to plot datetimes with matplotlib. See here <https://pandas.pydata.org/pandas-docs/version/0.21.1/whatsnew.html#restore-…> . - Improvements to the Parquet IO functions introduced in 0.21.0. See here <http://pandas.pydata.org/pandas-docs/version/0.21.1/whatsnew.html#whatsnew-…> . See the v0.21.1 Whatsnew <http://pandas.pydata.org/pandas-docs/version/0.21.1/whatsnew.html#whatsnew-…> overview for an extensive list of all the changes for 0.21.1. - Tom *What is it:* pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *How to get it:* Source tarballs and windows/mac/linux wheels are available on PyPI (thanks to Christoph Gohlke for the windows wheels, and to Matthew Brett for setting up the mac/linux wheels). Conda packages currently building for conda forge, and already available on the default channel. *Issues:* Please report any issues on our issue tracker: https://github.com/py data/pandas/issues *Thanks to all the contributors:* A total of 46 people contributed to this release. People with a “+” by their names contributed a patch for the first time. - Aaron Critchley + - Alex Rychyk - Alexander Buchkovsky + - Alexander Michael Schade + - Chris Mazzullo - Cornelius Riemenschneider + - Dave Hirschfeld + - David Fischer + - David Stansby + - Dror Atariah + - Eric Kisslinger + - Hans + - Ingolf Becker + - Jan Werkmann + - Jeff Reback - Joris Van den Bossche - Jörg Döpfert + - Kevin Kuhl + - Krzysztof Chomski + - Leif Walsh - Licht Takeuchi - Manraj Singh + - Matt Braymer-Hayes + - Michael Waskom + - Mie~~~ + - Peter Hoffmann + - Robert Meyer + - Sam Cohan + - Sietse Brouwer + - Sven + - Tim Swast - Tom Augspurger - Wes Turner - William Ayd + - Yee Mey + - bolkedebruin + - cgohlke - derestle-htwg + - fjdiod + - gabrielclow + - gfyoung - ghasemnaddaf + - jbrockmendel - jschendel - miker985 + - topper-123

1 0

Re: [Pandas-dev] Feedback request for return value of empty or all-NA sum (0 or NA?)
by Ryan Bressler Dec. 12, 2017

Dec. 12, 2017

I posted some brief feedback on the Issue tracker and Joris asked me to weight in here with our experience. First off some numbers. We maintain about ~30k line scientific python with a team of ~6 (and growing) researchers and engineers. I've just started to audit the code base for this issue but a quick grep reveals about 170 invocations of "sum" though some of those are numpy (more on that in second). I recently tried to upgrade to pandas .21 and a large number or our unit tests failed. For now we'll stay at .20 but this incident is also causing us to discuss limiting the use of pandas in our code base. We are in the financial industry and a lot of these invocations sum monetary amounts where pd.Series([]).sum() == 0 makes sense and may even be a common occurrence especially when aggregating via groupby or similar. Ie questions like "how many total dollars of apples were sold on Tuesday" are common and often have answer 0. However, the less domain specific and perhaps more insidious way this breaks our code is that we use a mix of pandas and numpy. We tend to use pandas for dealing with mixed data types and prototyping in pandas and then using pure numpy in areas where we care about speed or need to interface with scikit learn etc. This change means that pandas and numpy collections have very similar interface and but very different behavior. Further we have this nasty behavior: >>> np.sum(pd.Series([])) nan At first glance there isn't really a clean or consistent way for us to deal with this. If it isn't reverted we're in for a lot of careful auditing and special casing. For many sections of code it may be simplest to just eliminate pandas use. We are quite strict about dependency management which will allow us to avoid the problematic versions. However, having worked in the academic research previously I'd encourage you all to minimize headaches for downstream package maintainers / users by minimizing the number of releases with this inconsistent behavior. Thanks for reading and for all your hard work: Ryan Bressler Theorem LP

1 0

Re: [Pandas-dev] [pydata] Re: Feedback request for return value of empty or all-NA sum (0 or NA?)
by Stephan Hoyer Dec. 10, 2017

Dec. 10, 2017

On Sun, Dec 10, 2017 at 12:55 PM Sam Steingold <sds(a)gnu.org> wrote: > NB: I understand and appreciate that math is not your only > consideration, but, given that your target audience (customers) are > (applied) mathematicians, you might want to consider our opinion when > making design decisions that affect us. Sam, thank you for sharing your perspective. Yes, mathematicians are part of our audience for pandas. But to be clear, they are a relatively small portion. A big part of the reason why pandas has become so popular is that it is now used by a wide range of users for data analytics, including many without formal mathematical training.

1 0

Feedback request for return value of empty or all-NA sum (0 or NA?)
by Joris Van den Bossche Dec. 10, 2017

Dec. 10, 2017

*[Note for those reading it on the pydata mailing list, please answer to pandas-dev(a)python.org <pandas-dev(a)python.org> to keep discussion centralised there]* Hi list, In pandas 0.21.0 we changed the behaviour of the sum method for empty or all-NaN Series (to consistently return NaN), see the what's note <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#sum-prod-of-a…>. This change lead to some discussion on github whether this was the right choice we made. But the reach of github is of course limited, and therefore we wanted to solicit some more feedback on the mailing list. Below is given an overview of the background of the issue and the different options. Please keep in mind that we are not really interested in theoretical reasons why one of the other option is better or more correct. Each of the options has it advantages / disadvantages in practice. But it would be very interesting to hear the consequences in actual example analysis pipelines. Best, Joris Background Before pandas 0.21.0, the behaviour of the sum of an all-NA Series depended on whether the optional bottleneck dependency was installed. This inconsistency was in place since the bottleneck 1.0.0 release (February 2015), and you can read more background on it in the github issue #9422 <https://github.com/pandas-dev/pandas/issues/9422>. With bottleneck, the sum of all-NA was zero; without bottleneck, the sum was NaN. In [2]: pd.__version__ Out[2]: '0.20.3' In [3]: pd.options.compute.use_bottleneck = True In [4]: Series([np.nan]).sum() Out[4]: 0.0 In [5]: pd.options.compute.use_bottleneck = False In [6]: Series([np.nan]).sum() Out[6]: nan The sum of an empty series was always 0, with or without bottleneck. In [7]: Series([]).sum() Out[7]: 0 For pandas 0.21, we wanted to fix this inconsistency. The return value should not depend on whether an optional dependency is installed. After a lengthy discussion, we opted for the original pandas behaviour to return NaN. As a result, also the sum of an empty Series was changed to return NaN (see the what’s new notice here <http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#sum-prod-of-a…> ): In [2]: pd.__version__ Out[2]: '0.21.0' In [3]: pd.Series([np.nan]).sum() Out[3]: nan In [4]: pd.Series([]).sum() Out[4]: nan However, after the 0.21.0 release more feedback was received about cases where this choice is not desirable, and due to this feedback, we are reconsidering the decision. Options We see three different options for the default behaviour of sum for those two cases of empty and all-NA series: 1. Empty / all-NA sum is always zero: SUM([]) = 0 and SUM([NA]) = 0 - Behaviour of pandas < 0.21 + bottleneck installed - Consistent with NumPy, R, MATLAB, etc. (given you use the variant that is NA aware: nansum for numpy, na.rm=TRUE for R, ...) 1. Empty / all-NA sum is always NA: SUM([]) = NA and SUM([NA]) = NA - The behaviour that is introduced in 0.21.0 - Consistent with SQL (although often (rightly or not) complained about) 1. Mixed behaviour: SUM([]) = 0 and SUM([NA]) = NA - Behaviour of pandas < 0.21 (without bottleneck installed) - A practicable compromise (having SUM([NA]) keep the information of NA, while SUM([]) = 0 does not introduce NAs when there were no in the data) - But somewhat inconsistent and unique to pandas ? We have to stress that each of those choices can be preferable depending on the use case and has its advantages and disadvantages. Some might be more mathematical sound, others might preserve more information about having missing data, each can be be more consistent with a certain ecosystem, … It is clear that there is no ‘best’ option for all case. While we can only choose one of those options as the default behaviour, each choice can be accompanied by new features that can make it easier for the user to opt for a different behaviour: - When choosing option 1 or 2, we can introduce a new method (eg .total()) or a keyword to .sum() (eg min_count) to obtain the other behaviour. - When choosing for option 2, we could provide a pd.zeroifna(..) to be able to convert NaN values from aggregation results into zero’s if desired (similar to COALESCE(expr, 0) in SQL)

12 30

Help replacing workflows that used DataFrame.select
by Paul Hobson Dec. 5, 2017

Dec. 5, 2017

Hey folks, I noticed that DataFrame.select is now deprecated in favor of DataFrame.loc[index.map(selector_fxn)] PR: https://github.com/pandas-dev/pandas/pull/17633 Issue: https://github.com/pandas-dev/pandas/issues/12401 I have a lot of work flows that look something like this: res = ( data.resample(freq) .agg(agg_dict) .pipe(fxn_that_adds_many_cols) .select(complex_fxn_that_selects_a_few_cols, axis='columns') ) It's not immediately clear to me how to access all of the e.g., columns in the middle or at the end of a chain of dataframe operations. Any tips? -Paul

4 10

Re: [Pandas-dev] [pydata] Feedback request for return value of empty or, all-NA sum (0 or NA?)
by Stephen Simmons Dec. 3, 2017

Dec. 3, 2017

Nat Smith wrote: > I am baffled by the idea that sum([]) would return NaN. So am I. Here are two cases that leave me confused what the intention is. Case #1 - Summing an empty integer series Not only does the answer change from 0 to NaN, but the type changes from int to float. That occurs whether skipna is True or False! > pd.Series([], dtype=int).sum() nan > pd.Series([], dtype=int).sum(skipna=True) nan > pd.Series([], dtype=int).sum(skipna=False) nan This confused me so I went back to the docstring and tried it with a float Series: > pd.Series.sum? Signature: pd.Series.sum(self, axis=None, skipna=None, level=None, numeric_only=None, **kwargs) Docstring: Return the sum of the values for the requested axis Parameters ---------- axis : {index (0)} skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA or empty, the result will be NA level : int or level name, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar numeric_only : boolean, default None Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series. I would expect skipna being True to mean we don't want NaNs affecting the sum. So why would we want NaN when the series is empty? In fact, for an empty series, skipna gives the same NaN output for both skipna=True and skipna=False: > pd.Series([], dtype=float).sum(skipna=False) nan >pd.Series([], dtype=float).sum(skipna=True) nan This looks even more weird in this case: > pd.Series([0, float('nan')], dtype=float).sum(skipna=True) 0.0 # NaN is skipped, sum is non-NaN. So far so good... So what happens with different non-empty input? > pd.Series([float('nan')], dtype=float).sum(skipna=True) nan # Skip all NaNs, get empty series to sum, so return NaN??? So if we want to avoid NaNs in our output, the skipna parameter doesn't help. For every use of sum(), we now need to separately check two special cases: - empty input - input with only NaNs I can't see how this behaviour helps anyone! Regards Stephen

1 0