Mailman 3 November 2021 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

Decoupling type stubs for the public API from the pandas distribution
by Irv Lustig Dec. 7, 2021

Dec. 7, 2021

I discovered this feature of typing: https://www.python.org/dev/peps/pep-0561/#stub-only-packages The idea is that for a package like pandas, we can have a separate package "pandas-stubs" that would contain the type stubs for pandas. We wouldn't have to worry about including a `py.typed` file or `.pyi` files in our standard pandas distribution - all typing for the public API would be in the separate package. That would allow pandas typing for the public API to be maintained separately (different GitHub repo). We could start by just copying over what Microsoft created at https://github.com/microsoft/python-type-stubs/tree/main/pandas and then we maintain it as a separate repo, which could be installed via pip and conda. Any thoughts on whether we should consider doing this? -Irv

3 3

Error on using ax in plot() Pandas 1.2.3
by Mahmood Naderan Nov. 18, 2021

Nov. 18, 2021

Hi I am using the following versions >>> import matplotlib >>> print(matplotlib. __version__) 3.3.4 >>> import pandas as pd >>> print(pd.__version__) 1.2.3 >>> import sys >>> sys.version_info sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) In my code, I use axes in Pandas plot() like this (note that I omit some variables in this snippet to highlight the problem): def plot_dataframe(df, cnt, axes): plt.subplot(2, 1, 1) ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] ) def plot_kernels(my_dict2): fig,axes = plt.subplots(2,1, figsize=(20, 15)) should_plot = plot_dataframe(df, cnt, axes=axes) for ax in axes: ax.legend() plt.show() However, I get this error: Traceback (most recent call last): File "process_csv.py", line 174, in <module> plot_kernels( my_dict2 ) File "process_csv.py", line 62, in plot_kernels should_plot = plot_dataframe(df, cnt, axes=axes) File "process_csv.py", line 34, in plot_dataframe ax1 = row.plot( fontsize=font_size, linewidth=line_width, markersize=marker_size, marker='o', title='Raw values', label=cnt, ax=axes[0] ) File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_core.py", line 955, in __call__ return plot_backend.plot(data, kind=kind, **kwargs) File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py", line 61, in plot plot_obj.generate() File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 283, in generate self._adorn_subplots() File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 483, in _adorn_subplots all_axes = self._get_subplots() File "/home/mahmood/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py", line 903, in _get_subplots ax for ax in self.axes[0].get_figure().get_axes() if isinstance(ax, Subplot) AttributeError: 'NoneType' object has no attribute 'get_axes' I guess there is a mismatch between versions. Is there any workaround for that? Regards, Mahmood

1 0

API: Make silent casting behavior consistent by deprecating silent _object_-dtype casting
by Brock Mendel Nov. 10, 2021

Nov. 10, 2021

TLDR ---- We have inconsistent silent-casting vs raising logic for numpy vs EA dtypes (and inconsistencies within EA dtypes). By deprecating silently casting to *object* dtype, we can *mostly* make the behaviors match. Background ---------- A number of Series/DataFrame methods will silently cast when dealing with mismatched values. With a numpy dtype, each of the following silently cast to float64: ser = pd.Series([1, 2, 3], dtype="i8") ser.shift(1, fill_value=1.5) ser.mask([True, False, False], 1.5) ser.where([False, True, True], 1.5) ser.replace(1, 1.5) ser[0] = 1.5 ser.fillna(1.5) # <- this one doesn't cast as it is a no-op If we were to pass "foo" or a pd.Period, these would coerce to object instead of float. By contrast, similar mixed-type operations with an ExtensionDtype Series _mostly_ raise: ser2 = pd.Series(pd.period_range("2016-01-01", periods=3, freq="D")) ser2.shift(1, fill_value=1.5) # <- ValueError ser2.mask([True, False, False], 1.5) # <- ValueError ser2.where([False, True, True], 1.5) # <- ValueError ser2.fillna(1.5) # <- TypeError ser2.replace(ser2[0], 1.5) # <- coerces to object ser2[0] = 1.5 # <- coerces to object ser3 = pd.Series([pd.NA, 2, 3], dtype="Int64") ser3.shift(1, fill_value=1.5) # <- TypeError ser3.mask([True, False, False], 1.5) # <- TypeError ser3.where([False, True, True], 1.5) # <- TypeError ser3.fillna(1.5) # <- TypeError ser3.replace(ser3[0], 1.5) # <- TypeError ser3[0] = 1.5 # <- TypeError timedelta64, datetime64, and datetime64tz mostly behave like the numpy dtypes, with a few exceptions: - shift raises on mismatch - fillna raises on mismatch for timedelta64, casts for the others Categorical mostly behaves like other ExtensionDtypes, except for replace which has special logic. Goals ----- - Have matching behavior across dtypes. - Share code. Options ------- 1) Change EA (and dt64/td64) behavior to match non-EA behavior 2) Change non-EA behavior to match EA behavior (or stricter xref https://github.com/pandas-dev/pandas/issues/39584) 3) Deprecate (and eventually raise on) silent casting to _object_ dtype, allowing silent casting otherwise. Here I am advocating for option 3). The advantages as I see them: A) For numpy dtypes, we retain the most useful cases (int->float) B) Deprecates cases most likely to be unintentional (e.g. typo "2016-01-01" -> "2p16-01-01" causing a datetime64 Series to silently cast) C) For td64/dt64/dt64tz/period, the *only* silent casting is to object, so this completely gets rid of special-casing among that code D) For IntegerArray, FloatingArray, IntervalArray leaves open the option of allowing e.g. Integer->Floating casting (xref https://github.com/pandas-dev/pandas/issues/25288#issuecomment-941762174) E) Does not preclude later deciding on the stricter options in 2)

2 1

Changing `interpolation` to `method` for quantile and percentile?
by Sebastian Berg Nov. 10, 2021

Nov. 10, 2021

Hi all, at NumPy, I would like to rename quantile's `interpolation=` to `method=` and add a DeprecationWarning for it. But I realize that probably much more people use it through pandas than NumPy itself, so wondering if you have any inputs on it: https://github.com/numpy/numpy/pull/20327 The reasons are: * interpolation was a reasonable name for variations of the default method. But, we added new methods (same as R's) and for those it does not quite fit. `method` is more descriptive. * I feel quite a few users might better of using a different method (if they did not use the default). This would nudge them to review the code. * Python's `statistics.quantils` uses `method` (although we do not offer the *same* names for the methods right now!) But, if you think it is not worth the churn/noise, I could consider just sticking with `interpolation` for now, or delay giving a `DeprecationWarning`. Cheers, Sebastian

2 1

November 2021 monthly community meeting (Wednesday November 10, UTC 18:00)
by Joris Van den Bossche Nov. 9, 2021

Nov. 9, 2021

Hi all, A reminder that the next monthly dev call is tomorrow (Wednesday, November 10th) at 18:00 UTC (given the DST changes, this might now be 1 hour earlier in your local time, I think it corresponds to 12am Central). Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. All are welcome to attend! Video Call: https://us06web.zoom.us/j/84484803210?pwd=TjUxNmcyNHcvcG9SNGJvbE53Y21GZz09 Minutes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… (I probably won't be able to attend myself this time, or only briefly) Joris

1 0

Can I participate in contributing to this project(https://github.com/pandas-dev/pandas)
by 김두연 Nov. 7, 2021

Nov. 7, 2021

Hello, I'm a computer major student at Kwangwoon University in Korea. I got the task of participating in the open source project in this semester(2021/November~December). So I was looking for various open sources and found your project(pandas). I'm interested in this project, so can I participate in contributing to this project? It seems that the contribution will be mainly translating README.md to korean and correcting typos. Thanks for reading my email.

1 0

resampling limitations after release 1.0.5
by McIntosh, Jonathan Nov. 5, 2021

Nov. 5, 2021

Hello, I would like to report an issue that may or may not already be known, but that has been a bug that has prevented our project from progressing to more recent releases of pandas. We are working with datasets that extend out past the maximum timestamp date of 2262-04-11, so we generally use period ranges rather than datetime ranges. However, starting with release 1.1.0, the resample functionality no longer works with period ranges that extend past 2262-04-11. The screenshots below show an example of this phenomenon. Would it be possible to restore the resampling functionality for period ranges that extend past 2262-04-11 in future releases? Example Code Snippet: import pandas as ps from datetime import datetime print(ps.__version__) rng = ps.period_range(datetime(2000, 1, 1, 0, 0), datetime(2300, 1, 1, 0, 0), freq="1D") df = ps.DataFrame(index = rng, data = [i+1 for i in range(0,len(rng))]) resample_df = df.resample("6H").interpolate() print(resample_df.head(10)) Result when running with pandas==1.0.5: [cid:image004.png@01D7D1AA.38C99720] Result when running with pandas==1.1.1: [cid:image003.png@01D7D1A9.FBD5A5C0] Thanks, Jonathan McIntosh Water Resources Engineer 970.498.1831 | jmcintosh(a)rti.org 2950 East Harmony Road, Suite 390 | Fort Collins, CO 80528 [cid:image001.jpg@01D7D1A9.3C12EB30]

2 1