Mailman 3 July 2019 - Pandas-dev

Plans for pandas 0.25.0 and pandas 1.0
by Joris Van den Bossche Nov. 12, 2019

Nov. 12, 2019

Hi all, We had some discussion about this on the in-person dev sprint end of June, and I thought it would be good to have some public record of this as well. A pandas 0.25.0 release is close (the RC was released earlier this month), see https://github.com/pandas-dev/pandas/issues/24950 For pandas 1.0, the current plan is to finally "just do it". The idea is that it should not take too long after 0.25.0, without additional major API changes (additions are fine of course) but with removing the current deprecated functionalities. Depending on how much feedback there is on 0.25.0 and on how smoothly it goes for removing deprecated stuff, we could (maybe optimistically) target September for that. Comments certainly welcome! Joris

4 5

Dataframe summit @ EuroSciPy 2019
by Marc Garcia Sept. 15, 2019

Sept. 15, 2019

Hi there, Just to let you know that at EuroSciPy 2019 (in September in Spain) we will have a dataframe summit, to stay updated and coordinate among projects replicating the pandas API (other dataframe projects are more than welcome). Maintainers from all the main projects (pandas, dask, vaex, modin, cudf and koalas) will be attending. If you want to get involved (whether you can attend the conference or not), please DM me. More info: https://github.com/python-sprints/dataframe-summit Conference website: https://www.euroscipy.org/2019/ Cheers!

4 8

Version Policy following 1.0
by Tom Augspurger Sept. 14, 2019

Sept. 14, 2019

Split from https://mail.python.org/pipermail/pandas-dev/2019-July/001030.html Following 1.0, I think we stop outright breaking APIs. I think that stability will be welcome to users. We still have to decide how we deprecate APIs. The two options are 1. Rolling deprecations: Essentially what we do today: An API is deprecated in release 1.1.0 and can be removed in (say) 1.4.0. 2. SemVer: An API may be deprecated in 1.x.0. It can be removed in 2.0.0 Do people have preferences between the two? The (dis?)advantage of Semver is that all the API-breaking changes are restricted to a single releases. With rolling deprecations, the upgrades from any 1.x to 1.y should be smoother than 1.x to 2.x. Once we choose a strategy, we may want to formalize release schedules around it. Tom

6 10

Tidelift
by Joris Van den Bossche Aug. 8, 2019

Aug. 8, 2019

Hi all, We discussed this on the last dev chat, but putting it on the mailing list for those who were not present: we are planning to contact Tidelift to enter into a sponsor agreement for Pandas. The idea is to follow what NumPy (and recently also Scipy) did to have an agreement between Tidelift and NumFOCUS instead of an individual maintainer (see their announcement mail: https://mail.python.org/pipermail/numpy-discussion/2019-April/079370.html). Blog with overview about Tidelift: https://blog.tidelift .com/how-to-start-earning-money-for-your-open-source-project-with-tidelift. We didn't discuss yet what to do specifically with those funds, that should still be discussed in the future. Cheers, Joris

10 22

Pandas Roadmap
by Tom Augspurger July 29, 2019

July 29, 2019

Hi all, In https://github.com/pandas-dev/pandas/pull/27478, we're proposing the addition of a pandas roadmap. That document describes 1. The process for adding a roadmap item 2. Seeds the roadmap with a few items we'd like to see implemented We'd welcome your feedback, especially on 1. If / how project roadmaps can be useful to you / your organization 2. Thoughts on the roadmap evolution process Feedback on the individual items is of course also welcome, though we're trying to steer discussion of those issues to dedicated issues (but not all the items have issues yet). Thanks, Tom

1 0

Re: [Pandas-dev] Version Policy following 1.0 (Marc Garcia)
by Irv Lustig July 23, 2019

July 23, 2019

All: Let me add some perspective on version numbering from having lived as a product manager in the software industry for a number of years where we had a lot of discussions on these issues. We didn't tie deprecation events to "major" or "minor" releases. When we decided to deprecate something, we said it would be deprecated in a "future" release, which was typically in 3 releases, major or minor. At ILOG, the decision on having a major (increment the number before the decimal point) was solely a marketing decision. If we felt that the changes in the new release were significant, then we would bump up the major release number. At IBM, there were some "rules" that said whether we could have a major release. So we had a lot of minor releases (from a version 12.0 up to a version 12.9 today), and we still counted each "minor" release in terms of deprecation counts. So a policy could be Announce "will be deprecated in the future" in release X Announce "will be deprecated in 2 releases" in release X+1 Announce "will be deprecated in next release" in release X+2 and include warning messages in code. Announce "has been deprecated" in release X+3 with the code removed. So it doesn't matter whether X=1.0, X+1=1.1, X+2=1.2, and X+3=1.3, or X=1.0, X+1=1.1, X+2=2.0 and X+3=2.1 Don't let the numbering decide the deprecation policy. Just call each "major" or "minor" release a "release" and use something like the "3 release" policy stated above. And if you do decide on the "3 release" policy, have some way to keep track of when each thing is going to be deprecated. I'm not sure that exists today for pandas. -Irv Lustig (Dr-Irv) On Tue, Jul 23, 2019 at 12:00 PM <pandas-dev-request(a)python.org> wrote: > Send Pandas-dev mailing list submissions to > pandas-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/pandas-dev > or, via email, send a message with subject or body 'help' to > pandas-dev-request(a)python.org > > You can reach the person managing the list at > pandas-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pandas-dev digest..." > > > Today's Topics: > > 1. Re: Version Policy following 1.0 (Marc Garcia) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 22 Jul 2019 17:25:17 +0100 > From: Marc Garcia <garcia.marc(a)gmail.com> > To: Tom Augspurger <tom.augspurger88(a)gmail.com> > Cc: Matthew Rocklin <mrocklin(a)gmail.com>, pandas-dev > <pandas-dev(a)python.org> > Subject: Re: [Pandas-dev] Version Policy following 1.0 > Message-ID: > < > CAEk5N5sNDOFnG+3-JAvANi7pGFvLH5sSvtUknNdEx0zOquaoOQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > My comment was thinking on final users, and not developers of other > packages. I would say that a company that has a large code base with many > dependencies including pandas, would prefer to be able to keep updating > pandas in its version 1.* without worrying much about breaking anything, > and plan well a 1.* to 2.* migration. > > I'm also unsure if removing deprecated things in a rolling way will cause a > faster progress. I'm more biased to forget about removing stuff most of the > time, and for major versions just remove everything. I think it would make > our life easier to not have the overhead of > https://github.com/pandas-dev/pandas/issues/6581 > > My feeling is that everything will be simpler for everyone with SemVer, and > we are the ones deciding when we release a major version, so we'll keep > deprecated stuff for as long as we want, if the main advantage of rolling > deprecations is to remove things faster. But may be in practice there are > other factors that I'm not considering. If the rest of people think rolling > deprecations will be better, I'm ok with it, I may be wrong. > > On Mon, Jul 22, 2019 at 4:54 PM Tom Augspurger <tom.augspurger88(a)gmail.com > > > wrote: > > > Thanks Matt, > > > > Marc, do you have thoughts on that? As you say, SemVer is more popular in > > absolute terms. But within our little community (NumPy, pandas, > > scikit-learn), rolling deprecations seems to be the preferred approach. > > I think there's some value in being consistent with those libraries. > > > > Joris / Wes, do you know what Arrow's policy will be after its 1.0? > > > > Tom > > > > On Sun, Jul 21, 2019 at 10:10 AM Matthew Rocklin <mrocklin(a)gmail.com> > > wrote: > > > >> Hi All, > >> > >> I hope you don't mind the intrusion of a non-pandas dev here. In my > >> opinion SemVer makes more sense for libraries with a well defined and > >> narrowly scoped API, and less sense for an API as vast as the Pandas > API. > >> > >> My ardent hope as a user is that you all will clean up and improve the > >> Pandas API continuously. While doing this work I fully expect small > bits > >> of the API to break on pretty much every release (I think that it would > be > >> hard to avoid this). My guess is that if this community adopted SemVer > >> then devs would be far more cautious about tidying things, which I think > >> would be unfortunate. As someone who is very sensitive to changes in > the > >> Pandas API I'm fully in support of the devs breaking things regularly > if it > >> means faster progress. > >> > >> Best, > >> -matt > >> > >> On Sun, Jul 21, 2019 at 2:24 AM Marc Garcia <garcia.marc(a)gmail.com> > >> wrote: > >> > >>> Personally, I think will make things easier for users if we use SemVer. > >>> Mainly because as a user I think it's somehow intuitive that I can > upgrade > >>> for example from 1.1.0 to 1.8.0 without having to edit code. But I'd > expect > >>> to have to take a closer look and see incompatibilities when I migrate > from > >>> 1.* to 2.*. > >>> > >>> Personally I don't usually know the deprecation policies of packages, > >>> and I don't expect most pandas users to know ours even now. So, the > >>> simplest the better. > >>> > >>> Also, for ourselves, I think it's easier and more efficient to forget > >>> about removing code in minors, and to do all the removals for majors. > >>> > >>> I agree this comes at the cost of bigger changes in major releases, > >>> compared to a rolling policy. But IMHO it's worth. > >>> > >>> > >>> On Fri, 19 Jul 2019, 21:58 Tom Augspurger, <tom.augspurger88(a)gmail.com > > > >>> wrote: > >>> > >>>> On Wed, Jul 17, 2019 at 4:42 PM Brock Mendel <jbrockmendel(a)gmail.com> > >>>> wrote: > >>>> > >>>>> Do we anticipate the rate of deprecations decreasing significantly? > >>>>> i.e. if right now we deprecated everything on which there is a > consensus in > >>>>> GH, would we be done for a while? > >>>>> > >>>>> If not, then I think we're better off sticking with zero-dot-*, or > >>>>> else we'll be bumping major versions really frequently. > >>>>> > >>>> > >>>> I think this is why I prefer sticking with rolling. If every release > >>>> bumps the major version number, then no release is a major release. > >>>> > >>>> So my preference would be > >>>> > >>>> 1. Formally adopt and document that we using a rolling deprecation > cycle > >>>> 2. State that deprecations will be around for `N` major releases (3?) > >>>> 3. Require that every new deprecation includes the version it'll be > >>>> enforced in. e.g. > >>>> > >>>> ``` > >>>> DataFrame.get_dtype_counts is deprecated, and will be removed in > pandas > >>>> 1.3.0. > >>>> Use DataFrame.dtypes.value_counts() instead. > >>>> ``` > >>>> > >>>> Tom > >>>> > >>>> On Wed, Jul 17, 2019 at 1:49 PM Tom Augspurger < > >>>>> tom.augspurger88(a)gmail.com> wrote: > >>>>> > >>>>>> Split from > >>>>>> https://mail.python.org/pipermail/pandas-dev/2019-July/001030.html > >>>>>> > >>>>>> Following 1.0, I think we stop outright breaking APIs. I think that > >>>>>> stability will be welcome to users. > >>>>>> > >>>>>> We still have to decide how we deprecate APIs. The two options are > >>>>>> > >>>>>> 1. Rolling deprecations: Essentially what we do today: An API is > >>>>>> deprecated in release 1.1.0 and can be removed in (say) 1.4.0. > >>>>>> 2. SemVer: An API may be deprecated in 1.x.0. It can be removed in > >>>>>> 2.0.0 > >>>>>> > >>>>>> Do people have preferences between the two? The (dis?)advantage of > >>>>>> Semver is that all the API-breaking changes are restricted to a > single > >>>>>> releases. With rolling deprecations, the upgrades from any 1.x to > 1.y > >>>>>> should be smoother than 1.x to 2.x. > >>>>>> > >>>>>> Once we choose a strategy, we may want to formalize release > schedules > >>>>>> around it. > >>>>>> > >>>>>> Tom > >>>>>> _______________________________________________ > >>>>>> Pandas-dev mailing list > >>>>>> Pandas-dev(a)python.org > >>>>>> https://mail.python.org/mailman/listinfo/pandas-dev > >>>>>> > >>>>> _______________________________________________ > >>>> Pandas-dev mailing list > >>>> Pandas-dev(a)python.org > >>>> https://mail.python.org/mailman/listinfo/pandas-dev > >>>> > >>> _______________________________________________ > >>> Pandas-dev mailing list > >>> Pandas-dev(a)python.org > >>> https://mail.python.org/mailman/listinfo/pandas-dev > >>> > >> >

1 0

The FREE Demystifying Data Science Conference is happening again! July 30-31!!!
by Metis Chicago July 20, 2019

July 20, 2019

Metis <http://thisismetis.com/> is hosting our 3rd annual *FREE *Demystifying Data Science Conference <https://www.thisismetis.com/demystifying-data-science> July 30-31 from 10am - 5pm ET! There will be 16 interactive data science talks and 6 workshops led by industry-leading speakers. Experience interactivity via real-time chat with world-wide audience, polling, and social sharing. Plus, get access to recordings of every presentation and workshop. We speak *Python*. Day 1 - July 30 will be for aspiring data scientists. Day 2 - July 31 will be for business leaders and practitioners Registration page: https://www.thisismetis.com/demystifying-data-science

1 0

Backport Policy
by Tom Augspurger July 19, 2019

July 19, 2019

For our new maintainers (and as a reminder): We have a backport branch 0.25.x. Before merging a bugfix PR that should be backported, ensure that it has the 0.25.1 milestone. The backport bot will take care of opening a PR against 0.25.x with the necessary changes. We'd like to be fairly conservative with backports to 0.25.x. Pretty much just bug fixes. Performance improvements, new features, etc. should be merged with the 1.0 milestone. And FYI, our current version on master is 0.26.0.dev0, rather than 1.0.0.dev0, in case we need to issue a 0.26 before 1.0. But the plan is still to do 1.0 for the next major release. Tom

1 0

Avoiding Dependencies on Private Modules
by Matthew Gidden July 19, 2019

July 19, 2019

Hi all, I am one of the developers of a niche scientific library, pyam [1]. We provide some standard plotting routines for users with our specific data model, many of which are shims for pandas plotting. In order to keep track of certain aspects of data-plot attributes, we utilize the `_get_standard_colors()` [2]. Before pandas 0.25.0, this lived in pandas.plotting._style. Now it lives in pandas.plotting._matplotlib.style. Of course, this change breaks our package - we know this is our fault for relying on a private module =). Therefore - I wanted to ask the list whether there is a better way to get this functionality that is provided in first-class public interfaces? For now we can hotfix the issue, but we would like to try to guarantee stability in the future. Cheers, Matt [1] https://pyam-iamc.readthedocs.io/en/latest/ [2] https://github.com/IAMconsortium/pyam/blob/master/pyam/plotting.py#L37

2 3

ANN: Pandas 0.25.0 Released
by William Ayd July 19, 2019

July 19, 2019

Hi, I am pleased to announce the release of pandas 0.25.0. This is a major release from 0.24.2 and includes a number of API changes, new features, enhancements, and performance improvements along with a large number of bug fixes. Highlights include: Dropped Python 2 support Groupby aggregation with relabeling <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html#gr…> Better repr for MultiIndex <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html#be…> Better truncated repr for Series and DataFrame <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html#sh…> Series.explode to split list-like values to rows <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html#se…> See the release notes <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html> for a full list of all the changes from 0.24.2. The release can be installed with conda using the conda-forge channel conda install pandas Or via PyPI python3 -m pip install --upgrade pandas Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Will _______________________________________________ Pandas-dev mailing list Pandas-dev(a)python.org <mailto:Pandas-dev@python.org> https://mail.python.org/mailman/listinfo/pandas-dev <https://mail.python.org/mailman/listinfo/pandas-dev>

1 0