Mailman 3 September 2021 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Proposal for consistent, clear copy/view semantics in pandas with Copy-on-Write
by Joris Van den Bossche Oct. 11, 2021

Oct. 11, 2021

*(a.k.a. getting rid of the SettingWithCopyWarning)* Hi all, As you are probably aware, it's not always straightforward to understand the copy or view semantics of indexing methods in pandas. To understand when you get a view and when not, or why you get a SettingWithCopyWarning or how to get rid of it? It's also something that has already been discussed regularly (e.g. the discussion and implementation from 2015 started by Nick Eubank at gh-10954 <https://github.com/pandas-dev/pandas/issues/10954>). Last year, we again started to discuss this, which is tracked at https://github.com/pandas-dev/pandas/issues/36195. Based on those discussions, I have a concrete proposal to change the copy/view semantics of pandas. Short summary of the proposal: 1. The result of *any* indexing operation (subsetting a DataFrame or Series in any way) or any method returning a new DataFrame, always *behaves as if it were* a copy in terms of user API. 2. We implement Copy-on-Write. This way, we can actually use views as much as possible under the hood, while ensuring the user API behaves as a copy. This addresses multiple aspects: 1) a clear and consistent user API (a clear rule: *any* subset or returned series/dataframe is *always* a copy of the original, and thus never modifies the original) and 2) improving performance by avoiding excessive copies (eg a chained method workflow would no longer return an actual data copy at each step). Longer version of this proposal: https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2th… Proof-of-concept implementation: https://github.com/pandas-dev/pandas/pull/41878 GitHub issue with relevant discussion: https://github.com/pandas-dev/pandas/issues/36195 *Since this would be a change with a large impact on users, I think it is important to get broad feedback on this*. So comments, thoughts, concerns, ideas etc are very welcome (you can comment on the google doc, answer to this email or on the github issue). Best, Joris

7 22

Definitive guide for Regex in Pandas
by Shaozhong SHI Sept. 30, 2021

Sept. 30, 2021

Dear All, I am looking for a definitive guide for Regex in Pandas. Can anyone help? Regards, David

1 0

Automated data testing, checking, validation, reporting for data assurance
by Shaozhong SHI Sept. 29, 2021

Sept. 29, 2021

There appear to be a few options for this. Has anyone tested and got experience with automated data testing, validation and reporting? Can anyone enlighten me? Regards, David

1 0

ANN: pandas v1.3.3
by Simon Hawkins Sept. 12, 2021

Sept. 12, 2021

Hi all, I'm pleased to announce the release of pandas v1.3.3. This is a patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.3/whatsnew/v1.3.3.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.3.3 Or from conda-forge conda install -c conda-forge pandas==1.3.3 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0

More cores for Azure CI
by Irv Lustig Sept. 9, 2021

Sept. 9, 2021

At yesterday's pandas dev team meeting, we talked about an ongoing issue we have had with the Azure Pipelines Windows builds timing out. After some experimentation (see https://github.com/pandas-dev/pandas/pull/43468 ), I've determined that the issue is we are only given 2 cores on Azure, and running through the tests on the virtual image hosted on some arbitrary hardware using 2 cores in less than an hour is a challenge. It also seems that there is a bit of luck as to whether the underlying machine will have 2 or 4 cores. I've seen runs with both. Jeff posted the question: > also is there a way to request 4 cores (min) from azure for these builds? I have no clue. So does anyone know how we manage the relationship with Microsoft/Azure to get resources for CI? -Irv (Dr-Irv)

2 1

welcome new committers
by Jeff Reback Sept. 4, 2021

Sept. 4, 2021

On behalf of the pandas core team, welcome: Thomas and Matthew https://github.com/lithomas1 https://github.com/mzeitlin11 We had updated your permissions to pandas commiter! We deeply value the prior contributions and hope that you can keep contributing, help triage issues, and mentor new folks.You should see a github request shortly. See this page of the docs: https://pandas.pydata.org/pandas-docs/stable/development/maintaining.html for more color on how to triage issues and merge PRs. Thanks! Jeff

2 1