Mailman 3 December 2021 - Pandas-dev

Re: [Pandas-dev] Proposal for consistent, clear copy/view semantics in pandas with Copy-on-Write
by Irv Lustig Dec. 19, 2021

Dec. 19, 2021

Joris: I finally had some time to study our conversation from July, reread the Google docs proposal, and I tried out the PR as well. What I'm struggling with is how we document where behavior will change. As an example, the following sequence will give different results: Current behavior: >>> df = pd.DataFrame({"a":[10,11,12,13,14], "b": [100,101,102,103,104]}) >>> df["a"].loc[2] = 112 >>> df a b 0 10 100 1 11 101 2 112 102 3 13 103 4 14 104 New behavior: (from the PR): >>> df = pd.DataFrame({"a":[10,11,12,13,14], "b": [100,101,102,103,104]}) >>> df["a"].loc[2] = 112 >>> df a b 0 10 100 1 11 101 2 12 102 3 13 103 4 14 104 But in both cases, the following works: >>> df.loc[3,"b"] = 999 >>> df a b 0 10 100 1 11 101 2 12 102 3 13 999 4 14 104 So my concern is that if you had existing code that used the pattern df["a"].loc[2] = 112 , you'd get no warning that the behavior had changed. What I don't know is how much of code in the wild assumes the current behavior. So my questions are now: 1. How will we document, in a clean and concise way, the new behavior for people with existing pandas code? 2. How can people find pandas code where the behavior will change? Can we list all patterns that would produce different results? Can we detect chained indexing with setitem calls? 3. I'm guessing there is lots of code where people use DataFrame.copy() to avoid the SettingWithCopy warning. Can they just remove those copies now and their code will work? I agree that for new users, this new way of doing things makes sense. I'm worried about how we make the transition easier for people with large code bases that use pandas. -Irv >> On Sat, 17 Jul 2021 at 20:51, Joris Van den Bossche < jorisvandenbossche(a)gmail.com> wrote: >>> >>> On Fri, 16 Jul 2021 at 20:50, Irv Lustig <irv(a)princeton.com> wrote: >>>> >>>> >>>> Tom Augspurger <tom.augspurger88(a)gmail.com> wrote: >>>> >>>>> I wonder if we can validate what users (new and old) *actually* expect? >>>>> Users coming from R, which IIRC implements Copy on Write for matrices, >>>>> might be OK with indexing always being (behaving like) a copy. >>>>> I'm not sure what users coming from NumPy would expect, since I don't know >>>>> how many NumPy users really understand *a**.)* when a NumPy slice is a view >>>>> or copy, and *b.) *how a pandas indexing operation translates to a NumPy >>>>> slice. >>>>> >>>> >>>> IMHO, we should concentrate on the "new" users. For my team, there is no numpy or R background. They learn pandas, and what pandas does needs to be really clear in behavior and documentation. I would also hazard a guess that most pandas users are like that - pandas is the first tool they see, not numpy or R. >>>> >>>> The places where I think confusion could happen are things like this with a DataFrame df : >>>> >>>> s = df["a"] >>>> s.iloc[3:5] = [1, 2, 3] >>>> df["a"].iloc[3:5] = [1, 2, 3] >>>> df["b"] = df["a"] >>>> df["b"].iloc[3:5] = [4, 5, 6] >>>> s2 = df["b"] >>>> df["c"] = s2 >>>> s2.iloc[3:5] = [7, 8, 9] >>>> >>>> As I understand it (please correct me if I'm wrong), these lines would be interpreted as follows with the current proposal: >>> >>> >>> It's a bit different (to reiterate, with the *current* proposal, *any* indexing operation (including series selection) behaves as a copy; and also to be clear, this is one possible proposal, there are certainly other possibilities). Answering case by case: >>> >>>> >>>> 1. s = df["a"] >>>> Creates a view into the DataFrame df. No copying is done at all >>> >>> >>> Indeed a view (but that's an implementation detail) >>> >>>> 2. s.iloc[3:5] = [1, 2, 3] >>>> Modifies the series s and the underlying DataFrame df. (copy-on-write) >>> >>> >>> Due to copy-on-write, it does *not* modify the DataFrame df. Copy-on-write means that only when s is being written to, its data get copied (so at that point breaking the view-relation with the parent df) >>> >>>> >>>> 3. df["a"].iloc[3:5] = [1, 2, 3] >>>> Modifies the dataframe >>> >>> >>> This is an example of chained assignment, which in the current proposal never works (see the example in the google doc). This is because chained assignment can always be written as: >>> >>> temp = df["a"] >>> temp.iloc[3:5] = [1, 2, 3] >>> >>> and `temp` uses copy-on-write (and then it is the same example as the one above in 2.). >>> >>> (what you describe is the current behaviour of pandas) >>> >>>> >>>> 4. df["b"] = df["a"] >>>> Copies the series from "a" to "b" >>> >>> >>> It would indeed behave as a copy, but under the hood we can actually keep this as a view (delay the copy thanks to copy-on-write). >>> >>>> >>>> 5. df["b"].iloc[3:5] = [4, 5, 6] >>>> Modifies "b" in the DataFrame, but not "a" >>> >>> >>> Also doesn't modify "b" (see example 3. above), but indeed does not modify "a" >>> >>>> >>>> 6. s2 = df["b"] >>>> Create a view into the DataFrame df. No copying is done at all. >>> >>> >>> Same as 1. >>> >>>> >>>> 7. df["c"] = s2 >>>> Copies the series from "b" to "c" >>> >>> >>> Same as 4. >>> >>>> >>>> 8. s2.iloc[3:5] = [7, 8, 9] >>>> Modifies s2, which modifies "b", but NOT "c" >>> >>> >>> Doesn't modify "b" and "c". Similar as 3. >>> >>>> I think the challenge is explaining the sequence 6,7,8 above in comparison to the other sequences. >>> >>> >>> So with the current proposal, the sequece 6, 7, 8 actually doesn't behave differently. But it is mainly 2 and 3 that would be quite different compared to the current pandas behaviour. >>> >>>> >>>> >>>> -Irv

2 1

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Pandas Support For TVA
by Gilreath, Dalton E Dec. 17, 2021

Dec. 17, 2021

Pandas support, You are receiving this message because a cybersecurity vulnerability has been identified and TVA is taking precautions to remedy any applications which may be affected. Pandas is currently being used by TVA. TVA is requesting confirmation on whether the Apache Log4j utility is present as part of this application. If Log4j is being utilized please provide the version and remediation steps your company plans on taking. Please respond to this email within 48 hours and advise if this file is present. If it is not, no further action is necessary except responding with confirmation that it does not exist. If it is present, please advise on if you have a patch available to remediate this issue. We must have a patch to continue use of this application. If no patch is available, the application must be turned off by December 24, 2021 to ensure TVA system security. Your information should be sent to me at degilreath(a)tva.gov<mailto:degilreath@tva.gov>. Feel free to contact me with any questions. Dalton Gilreath Manager, Product Delivery - Maximo Technology and Innovation [TVA logo] W. 423-751-8029 M. 423-785-7215 E. degilreath(a)tva.gov<mailto:degilreath@tva.gov> 1101 Market Street, Chattanooga, TN 37402 NOTICE: This electronic message transmission contains information that may be TVA SENSITIVE, TVA RESTRICTED, or TVA CONFIDENTIAL. Any misuse or unauthorized disclosure can result in both civil and criminal penalties. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the content of this information is prohibited. If you have received this communication in error, please notify me immediately by email and delete the original message.

3 2

Pandas astype() changes the class type
by Simeon Simeonov Dec. 15, 2021

Dec. 15, 2021

Hi all, I saw this behaviour and I don't know if this is a bug or feature. I don't have much experience with directly inheriting from pandas.DataFrame as I've always preferred aggregation rather than inheritance there. A working sample is pasted below. Notice how *df.astype(dtypes)* changes the type to pandas.DataFrame. Any suggestions if this is intended behaviour? import pandas as pd class DF(pd.DataFrame): @property def _constructor(self): return self.__class__ df = DF({ 'A': [1,2,3], 'B': [10,20,30], 'C': [100,200,300], }) # Type is DF a = df['A'] # type is Series ab = df[['A', 'B']] # type is DF dtypes = {'A': 'float64', 'B': 'float64', 'C': 'float64'} x = df.astype(dtypes) type(x) # type is pd.DataFrame Regards, Simeon

2 1

ANN: pandas v1.3.5
by Simon Hawkins Dec. 14, 2021

Dec. 14, 2021

Hi all, I'm pleased to announce the release of pandas v1.3.5. This is a patch release in the 1.3.x series and includes some regression fixes. We recommend that all users upgrade to this version. See the release notes <https://pandas.pydata.org/pandas-docs/version/1.3/whatsnew/v1.3.5.html> for a list of all the changes. The release can be installed from PyPI python -m pip install --upgrade pandas==1.3.5 Or from conda-forge conda install -c conda-forge pandas==1.3.5 Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible.

1 0

pandas typing stubs meeting January 7, 2022, Noon Eastern
by Irv Lustig Dec. 10, 2021

Dec. 10, 2021

I've connected with the primary author of the Microsoft supplied pandas typing stubs ( https://github.com/microsoft/python-type-stubs/tree/main/pandas) and the primary maintainers of the pandas-stubs package ( https://github.com/VirtusLab/pandas-stubs) and due to the holidays and their schedules, we will meet in the new year on Friday, January 7, 2022, Noon Eastern Time, 5PM UTC. The meeting has been added to the pandas development calendar visible at https://pandas.pydata.org/docs/development/meeting.html Join Zoom Meeting https://us02web.zoom.us/j/89675119530?pwd=emhQQXhuVy9KeHFjZTZhQ0plN3JpUT09 <https://www.google.com/url?q=https://us02web.zoom.us/j/89675119530?pwd%3Dem…> Meeting ID: 896 7511 9530 Passcode: 136973 One tap mobile +13017158592,,89675119530# US (Washington DC) +13126266799,,89675119530# US (Chicago) I will send another reminder out a couple of days before the meeting. -Irv Lustig (Dr-Irv)

1 0

Decoupling type stubs for the public API from the pandas distribution
by Irv Lustig Dec. 7, 2021

Dec. 7, 2021

I discovered this feature of typing: https://www.python.org/dev/peps/pep-0561/#stub-only-packages The idea is that for a package like pandas, we can have a separate package "pandas-stubs" that would contain the type stubs for pandas. We wouldn't have to worry about including a `py.typed` file or `.pyi` files in our standard pandas distribution - all typing for the public API would be in the separate package. That would allow pandas typing for the public API to be maintained separately (different GitHub repo). We could start by just copying over what Microsoft created at https://github.com/microsoft/python-type-stubs/tree/main/pandas and then we maintain it as a separate repo, which could be installed via pip and conda. Any thoughts on whether we should consider doing this? -Irv

3 3

December 2021 monthly community meeting (Wednesday December 8, UTC 18:00)
by Joris Van den Bossche Dec. 7, 2021

Dec. 7, 2021

Hi all, A reminder that the next monthly dev call is tomorrow (Wednesday, December 8) at 18:00 UTC (12am Central). Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. All are welcome to attend! Video Call: https://us06web.zoom.us/j/84484803210?pwd=TjUxNmcyNHcvcG9SNGJvbE53Y21GZz09 Minutes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 0

I need pandas pakge
by 高玉 Dec. 4, 2021

Dec. 4, 2021

hi pandas -dev： I want this version of the pandas package with pandas1.3.4 for Mac osx_11_inter or Mac osx_11_x86_64. The package I downloaded does not comply with pandas-1.3.4-cp310-cp310-macosx. thank you!!

1 0

Pandas_read_json produces one single columns of nested dictionary
by Shaozhong SHI Dec. 2, 2021

Dec. 2, 2021

Pandas_read_json produces one single columns of nested dictionary. How best to convert this into a dataframe? Regards, David

1 0