Mailman 3 July 2021 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Proposal for consistent, clear copy/view semantics in pandas with Copy-on-Write
by Joris Van den Bossche Oct. 11, 2021

Oct. 11, 2021

*(a.k.a. getting rid of the SettingWithCopyWarning)* Hi all, As you are probably aware, it's not always straightforward to understand the copy or view semantics of indexing methods in pandas. To understand when you get a view and when not, or why you get a SettingWithCopyWarning or how to get rid of it? It's also something that has already been discussed regularly (e.g. the discussion and implementation from 2015 started by Nick Eubank at gh-10954 <https://github.com/pandas-dev/pandas/issues/10954>). Last year, we again started to discuss this, which is tracked at https://github.com/pandas-dev/pandas/issues/36195. Based on those discussions, I have a concrete proposal to change the copy/view semantics of pandas. Short summary of the proposal: 1. The result of *any* indexing operation (subsetting a DataFrame or Series in any way) or any method returning a new DataFrame, always *behaves as if it were* a copy in terms of user API. 2. We implement Copy-on-Write. This way, we can actually use views as much as possible under the hood, while ensuring the user API behaves as a copy. This addresses multiple aspects: 1) a clear and consistent user API (a clear rule: *any* subset or returned series/dataframe is *always* a copy of the original, and thus never modifies the original) and 2) improving performance by avoiding excessive copies (eg a chained method workflow would no longer return an actual data copy at each step). Longer version of this proposal: https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2th… Proof-of-concept implementation: https://github.com/pandas-dev/pandas/pull/41878 GitHub issue with relevant discussion: https://github.com/pandas-dev/pandas/issues/36195 *Since this would be a change with a large impact on users, I think it is important to get broad feedback on this*. So comments, thoughts, concerns, ideas etc are very welcome (you can comment on the google doc, answer to this email or on the github issue). Best, Joris

7 22

ANN: pandas v1.3.1
by Simon Hawkins July 26, 2021

July 26, 2021

Hi all, I'm pleased to announce the release of pandas v1.2.5. This is the first patch release in the 1.3.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.3/whatsnew/v1.3.1.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.3.1 Or from conda-forge conda install -c conda-forge pandas==1.3.1 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0

Re: [Pandas-dev] Proposal for consistent, clear copy/view semantics in pandas with Copy-on-Write
by Joris Van den Bossche July 17, 2021

July 17, 2021

On Fri, 16 Jul 2021 at 20:50, Irv Lustig <irv(a)princeton.com> wrote: > > Tom Augspurger <tom.augspurger88(a)gmail.com> wrote: > > I wonder if we can validate what users (new and old) *actually* expect? >> Users coming from R, which IIRC implements Copy on Write for matrices, >> might be OK with indexing always being (behaving like) a copy. >> I'm not sure what users coming from NumPy would expect, since I don't know >> how many NumPy users really understand *a**.)* when a NumPy slice is a >> view >> or copy, and *b.) *how a pandas indexing operation translates to a NumPy >> slice. >> >> > IMHO, we should concentrate on the "new" users. For my team, there is no > numpy or R background. They learn pandas, and what pandas does needs to be > really clear in behavior and documentation. I would also hazard a guess > that most pandas users are like that - pandas is the first tool they see, > not numpy or R. > > The places where I think confusion could happen are things like this with > a DataFrame df : > > 1. s = df["a"] > 2. s.iloc[3:5] = [1, 2, 3] > 3. df["a"].iloc[3:5] = [1, 2, 3] > 4. df["b"] = df["a"] > 5. df["b"].iloc[3:5] = [4, 5, 6] > 6. s2 = df["b"] > 7. df["c"] = s2 > 8. s2.iloc[3:5] = [7, 8, 9] > > As I understand it (please correct me if I'm wrong), these lines would be > interpreted as follows with the current proposal: > It's a bit different (to reiterate, with the *current* proposal, *any* indexing operation (including series selection) behaves as a copy; and also to be clear, this is one possible proposal, there are certainly other possibilities). Answering case by case: > 1. s = df["a"] > Creates a view into the DataFrame df. No copying is done at all > Indeed a view (but that's an implementation detail) 2. s.iloc[3:5] = [1, 2, 3] > Modifies the series s and the underlying DataFrame df. (copy-on-write) > Due to copy-on-write, it does *not* modify the DataFrame df. Copy-on-write means that only when s is being written to, its data get copied (so at that point breaking the view-relation with the parent df) > 3. df["a"].iloc[3:5] = [1, 2, 3] > Modifies the dataframe > This is an example of chained assignment, which in the current proposal never works (see the example in the google doc <https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2th…>). This is because chained assignment can always be written as: temp = df["a"] temp.iloc[3:5] = [1, 2, 3] and `temp` uses copy-on-write (and then it is the same example as the one above in 2.). (what you describe is the current behaviour of pandas) > 4. df["b"] = df["a"] > Copies the series from "a" to "b" > It would indeed behave as a copy, but under the hood we can actually keep this as a view (delay the copy thanks to copy-on-write). > 5. df["b"].iloc[3:5] = [4, 5, 6] > Modifies "b" in the DataFrame, but not "a" > Also doesn't modify "b" (see example 3. above), but indeed does not modify "a" > 6. s2 = df["b"] > Create a view into the DataFrame df. No copying is done at all. > Same as 1. > 7. df["c"] = s2 > Copies the series from "b" to "c" > Same as 4. > 8. s2.iloc[3:5] = [7, 8, 9] > Modifies s2, which modifies "b", but NOT "c" > Doesn't modify "b" and "c". Similar as 3. I think the challenge is explaining the sequence 6,7,8 above in comparison > to the other sequences. > So with the current proposal, the sequece 6, 7, 8 actually doesn't behave differently. But it is mainly 2 and 3 that would be quite different compared to the current pandas behaviour. > > -Irv > > > > > > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev >

1 0

Re: [Pandas-dev] Proposal for consistent, clear copy/view semantics in pandas with Copy-on-Write
by Irv Lustig July 16, 2021

July 16, 2021

Tom Augspurger <tom.augspurger88(a)gmail.com> wrote: I wonder if we can validate what users (new and old) *actually* expect? > Users coming from R, which IIRC implements Copy on Write for matrices, > might be OK with indexing always being (behaving like) a copy. > I'm not sure what users coming from NumPy would expect, since I don't know > how many NumPy users really understand *a**.)* when a NumPy slice is a view > or copy, and *b.) *how a pandas indexing operation translates to a NumPy > slice. > > IMHO, we should concentrate on the "new" users. For my team, there is no numpy or R background. They learn pandas, and what pandas does needs to be really clear in behavior and documentation. I would also hazard a guess that most pandas users are like that - pandas is the first tool they see, not numpy or R. The places where I think confusion could happen are things like this with a DataFrame df : 1. s = df["a"] 2. s.iloc[3:5] = [1, 2, 3] 3. df["a"].iloc[3:5] = [1, 2, 3] 4. df["b"] = df["a"] 5. df["b"].iloc[3:5] = [4, 5, 6] 6. s2 = df["b"] 7. df["c"] = s2 8. s2.iloc[3:5] = [7, 8, 9] As I understand it (please correct me if I'm wrong), these lines would be interpreted as follows with the current proposal: 1. Creates a view into the DataFrame df. No copying is done at all 2. Modifies the series s and the underlying DataFrame df. (copy-on-write) 3. Modifies the dataframe 4. Copies the series from "a" to "b" 5. Modifies "b" in the DataFrame, but not "a" 6. Create a view into the DataFrame df. No copying is done at all. 7. Copies the series from "b" to "c" 8. Modifies s2, which modifies "b", but NOT "c" I think the challenge is explaining the sequence 6,7,8 above in comparison to the other sequences. -Irv

1 0

ANN: pandas v1.3.0
by Simon Hawkins July 5, 2021

July 5, 2021

Hi all, The pandas team is pleased to announce the release of pandas 1.3.0. This release includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.3/whatsnew/v1.3.0.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.3.0 Or from conda-forge conda install -c conda-forge pandas==1.3.0 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0