Mailman 3 June 2020 - Pandas-dev

Re: [Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Brock Mendel Dec. 17, 2021

Dec. 17, 2021

> We actually *have* prototypes: the prototype of the split-policy discussed AFAICT that is a 5 year old branch. Is there a version of this based off of master that you can show asv results for? > Also, if performance is in the end the decisive criterion, I repeat my earlier remark in this thread: we need to be clearer about what we want / expect. In principle, this is pretty much exactly what the asvs are supposed to represent. ---- You have demonstrated that you are willing to repeat yourself more than I am, to the point that I find pandas interactions more frustrating than fulfilling. I'm going to step away for a little while. On Thu, Jun 11, 2020 at 1:29 PM Daniel Scott <scottdaniel539(a)yahoo.com> wrote: > > > Sent from Yahoo Mail on Android > <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_And…> > > On Thu, Jun 11, 2020 at 3:10 PM, Joris Van den Bossche > <jorisvandenbossche(a)gmail.com> wrote: > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

2 6

ANN: Pandas 1.0.5 Released
by Simon Hawkins June 18, 2020

June 18, 2020

Hi all, I'm pleased to announce that pandas 1.0.5 is now available. This is a minor bug-fix release in the 1.0.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the full whatsnew <https://pandas.pydata.org/docs/whatsnew/v1.0.5.html> for a list of all the changes. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Thanks to all the contributors who made this release possible. - Simon

1 0

A case for a simplified (non-consolidating) BlockManager with 1D blocks
by Joris Van den Bossche June 11, 2020

June 11, 2020

Hi list, Rewriting the BlockManager based on a simpler collection of 1D-arrays is actually on our roadmap (see here <https://pandas.pydata.org/docs/dev/development/roadmap.html#block-manager-r…>), and I also touched on it in a mailing list discussion about pandas 2.0 earlier this year (see here <https://mail.python.org/pipermail/pandas-dev/2020-February/001180.html>). But since the topic came up again recently at the last online dev meeting (and also Uwe Korn who wrote a nice blog post <https://uwekorn.com/2020/05/24/the-one-pandas-internal.html> about this yesterday), I thought to do a write-up of my thoughts on why I think we should actually move towards a simpler, non-consolidating BlockManager with 1D blocks. *Simplication of the internals* It's regularly brought up as a reason to have 2D EextensionArrays (EAs) because right now we have a lot of special cases for 1D EAs in the internals. But to be clear: the additional complexity does not come from 1D EAs in itself, it comes from the fact that we have a mixture of 2D and 1D blocks. Solving this would require a consistent block dimension, and thus removing this added complexity can be done in two ways: have all 1D blocks, or have all 2D blocks. Just to say: IMO, this is not an argument in favor of 2D blocks / consolidation. Moreover, when going with all 1D blocks, we cannot only remove the added complexity from dealing with the mixture of 1D/2D blocks, we will *also* be able to reduce the complexity of dealing with 2D blocks. A BlockManager with 2D blocks is inherently more complex than with 1D blocks, as one needs to deal with proper alignment of the blocks, a more complex "placement" logic of the blocks, etc. I think we would be able to simplify the internals a lot by going with a BlockManager as a store of 1D arrays. *Performance* Performance is typically given as a reason to have consolidated, 2D blocks. And of course, certain operations (especially row-wise operations, or on dataframes with more columns as rows) will always be faster when done on a 2D numpy array under the hood. However, based on recent experimentation with this (eg triggered by the block-wise frame ops PR <https://github.com/pandas-dev/pandas/pull/32779>, and see also some benchmarks I justed posted in #10556 <https://github.com/pandas-dev/pandas/issues/10556#issuecomment-633703160> / this gist <https://gist.github.com/jorisvandenbossche/b8ae071ab7823f7547567b1ab9d4c20c>), I also think that for many operations and with decent-sized dataframes, this performance penalty is actually quite OK. Further, there are also operations that will *benefit* from 1D blocks. First, operations that now involve aligning/splitting blocks, re-consolidation, .. will benefit (e.g. a large part of the slowdown doing frame/frame operations column-wise is currently due to the consolidation in the end). And operations like adding a column, concatting (with axis=1) or merging dataframes will be much faster when no consolidation is needed. Personally, I am convinced that with some effort, we can get on-par or sometimes even better performance with 1D blocks compared to the performance we have now for those cases that 90+% of our users care about: - With limited effort optimizing the column-wise code paths in the internals, we can get a long way. - After that, if needed, we can still consider if parts of the internals could be cythonized to further improve certain bottlenecks (and actually cythonizing this will also be simpler for a simpler non-consolidating block manager). *Possibility to get better copy/view semantics* Pandas is badly known for how much it copies ("you need 10x the memory available as the size of your dataframe"), and having 1D blocks will allow us to address part of those concerns. *No consolidation = less copying.* Regularly consolidating introduces copies, and thus removing consolidation will mean less copies. For example, this would enable that you can actually add a single column to a dataframe without having to copy to the full dataframe. *Copy / view semantics* Recently there has been discussion again around whether selecting columns should be a copy or a view, and some other issues were opened with questions about views/copies when slicing columns. In the consolidated 2D block layout this will always be inherently messy, and unpredictable (meaning: depending on the actual block layout, which means in practice unpredictable for the user unaware of the block layout). Going with a non-consolidated BlockManager should at least allow us to get better / more understandable semantics around this. ------------------------------ *So what are the reasons to have 2D blocks?* I personally don't directly see reasons to have 2D blocks *for pandas itself* (apart from performance in certain row-wise use cases, and except for the fact that we have "always done it like this"). But quite likely I am missing reasons, so please bring them up. But I think there are certainly use cases where 2D blocks can be useful, but typically "external" (but nonetheless important) use cases: conversion to/from numpy, xarray, etc. A typical example that has recently come up is scikit-learn, where they want to have a cheap dataframe <-> numpy array roundtrip for use in their pipelines. However, I personally think there are possible ways that we can still accommodate for those use cases, with some effort, while still having 1D Blocks in pandas itself. So IMO this is not sufficient to warrant the complexity of 2D blocks in pandas. (but will stop here, as this mail is getting already long ..). Joris

8 31

tslibs 2.0 and non-nanosecond datetime64/timedelta64
by Brock Mendel June 2, 2020

June 2, 2020

This is a discussion of what it would take to support non-nanosecond datetime64/timedelta64 dtypes and what decisions would need to be made along the way. The implementation would probably consist of: - add a NPY_DATETIMEUNIT attribute to Timestamp and Datetime64TZDtype - for timezone-related methods: - short-term: cast to nanosecond, use existing code, cast back to other unit - longer-term: update existing code to support non-nano units directly - comb through the code for all the places where we implicitly assume nano units and update - tests, so, so many tests We could then consider de-duplication. Tick is already redundant with Timedelta, and Timestamp[H] would render Period[H] redundant. With appropriate deprecation cycle, we could rip out a bunch of code. Another possibility is to try to upstream some code to numpy, which they have recently been receptive to (#16266 <https://github.com/numpy/numpy/pull/16266>, #16363 <https://github.com/numpy/numpy/pull/16363>, #16364 <https://github.com/numpy/numpy/pull/16364>, #16352 <https://github.com/numpy/numpy/issues/16352>, <https://github.com/numpy/numpy/issues/16195>#16195 <https://github.com/numpy/numpy/issues/16195>). @rgommers tells me that trying to implement a tz-aware datetime64 dtype in numpy would be "folly, that way madness lies", but that it might be more feasible once @seberg's dtype refactor lands. More realistically short-term, if we convinced numpy to update NPY_DATETIMEUNIT to include the anchored quarter/year/week units we use for Period, we could condense a lot of confusing enum-like code. Tangentially related: with zoneinfo (PEP 615) we should consider making those our canonical tzinfos and converting any dateutil/pytz tzinfos we encounter to those. They are implemented in C, so I'm _hopeful_ we can make some of our vectorized tzconversion code unnecessary. @pganssle has suggested we implement our own tzinfos, but I'm holding out hope we can keep that upstream.

5 6

Create an aggregate/summary pandas dataframe based on overlapping dates derived from a more specific dataframe?
by Aaron June 1, 2020

June 1, 2020

Hello, Given a dateframe with trips made my employees of different companies, I am trying to generate a new dataframe with only the company names. There are trips made by employees from different companies. I am looking to combine the overlapping travel times from employees of the SAME company into a single row. If there are no overlapping travel times, then that row just transfers over as-is. When there are overlapping travel times, then the following will happen: --The name field is removed b/c that is no longer relevant (company name stays), the Depart date will be the earliest date of any of the trip dates regardless of the employee, the Return date will be the latest date of any of the trip dates regardless of the employee, the charges for the trip will be summed For example, if trips had dates 01/01/20 - 01/31/20, 01/15/20 - 02/15/20, 02/01-20 - 02/28/20, then all three would be combined. The starting date will be 1/1/20 and ending as of 2/28/20 Basically, the company was on that trip from start to finish… kinda like a relay run handing off the baton. Also, the charges will be summed for each of those trips and transferred over to the single row. I was playing around with timedelta, hierarchal indices, grouping, and sorting but had a really hard time since I am looking at date ranges instead of specific dates. Here is the starting dataframe code/output: import pandas as pd emp_trips = {'Name': ['Bob','Joe','Sue','Jack', 'Henry', 'Frank', 'Lee', 'Jack'], 'Company': ['ABC', 'ABC', 'ABC', 'HIJ', 'HIJ', 'DEF', 'DEF', 'DEF'], 'Depart' : ['01/01/2020', '01/01/2020', '01/06/2020', '01/01/2020', '05/01/2020', '01/13/2020', '01/12/2020', '01/14/2020'], 'Return' : ['01/31/2020', '02/15/2020', '02/20/2020', '03/01/2020', '05/05/2020', '01/15/2020', '01/30/2020', '02/02/2020'], 'Charges': [10.10, 20.25, 30.32, 40.00, 50.01, 60.32, 70.99, 80.87] } df = pd.DataFrame(emp_trips, columns = ['Name', 'Company', 'Depart', 'Return', 'Charges']) # Convert to date format df['Return']= pd.to_datetime(df['Return']) df['Depart']= pd.to_datetime(df['Depart']) Name Company Depart Return Charges0 Bob ABC 2020-01-01 2020-01-31 10.101 Joe ABC 2020-01-01 2020-02-15 20.252 Sue ABC 2020-01-06 2020-02-20 30.323 Jack HIJ 2020-01-01 2020-03-01 40.004 Henry HIJ 2020-05-01 2020-05-05 50.015 Frank DEF 2020-01-13 2020-01-15 60.326 Lee DEF 2020-01-12 2020-01-30 70.997 Jack DEF 2020-01-14 2020-02-02 80.87 And, here is the desired/generated dataframe: Company Depart Return Charges0 ABC 01/01/2020 02/20/2020 60.671 HIJ 01/01/2020 03/01/2020 40.002 HIJ 05/01/2020 05/05/2020 50.013 DEF 01/12/2020 02/02/2020 212.18 Thank you in advance! Aaron

1 0