Mailman 3 May 2017 - Pandas-dev

Pandas Deferred Expressions
by Phillip Cloud June 16, 2017

June 16, 2017

Hi all, I'd like to fork part of the thread from Wes's original email about the future of pandas and discuss all things deferred expressions. To start, here's Wes's original thoughts, and a response from Chris Bartak that was in a different thread. After I send this email I'm going to follow up with my own thoughts in a different email so I can address any specific concerns as well as offer up a list of advantages and disadvantages to this approach and lessons learned about building DSLs in Python. *Wes's post:* *TOPIC THREE:* I think we should start developing a "deferred pandas API" that is designed and directly developed by the pandas developer community. >From our respective experiences creating expression DSLs and other computation frameworks on top of pandas, I believe this is something where we can build something reasonable and useful. As one concrete problem this would help with: addressing some of the awkwardness around complex groupby-aggregate expressions (custom aggregations would simply be named expressions). The idea of the deferred expression API would be similar to dplyr in R: * "True" schemas (we'll have to work around pandas 0.x warts with implicit casts, etc.) * Immutable data structures / no mutation outside "amend" operations that change values by returning new objects * Less index-related stuff in this API (perhaps this is controversial, we shall see) We can create an in-memory backend for "pandas expressions" on pandas 0.x/1.0 and separately create an alternative backend using libpandas (once that is more fully baked / functional) -- this will also help provide a forcing function for implementing analytics that are required for implementing the backend. Distributed execution for us is almost certainly out of scope, and even if so we would probably want to offload onto prior art in Dask or elsewhere. So if the dask.dataframe API and the pandas expression API look different in ways that are unpleasant, we could either compile from pandas -> dask under the hood, or make API changes to make the semantics more conforming. When libpandas / pandas 2.0 is more mature we can consider building stronger out-of-core execution (plenty of prior art we can learn from here, e.g. SFrame). As far as tools to implement the deferred expression API -- I will leave this to discussion. I spent a considerable amount of time making a pandas-like expression API for SQL in Ibis (see https://github.com/cloudera/ibis/tree/master/ibis/expr) while I was at Cloudera, so there's some ideas there (like separating the "internal" AST from the "external" user expressions) that we can learn from, or fork or use some of that expression code in some way. I don't have a strong opinion as long as the expressions are as strongly-typed as possible (i.e. tables have schemas, operations have checked input and output types) and catch user errors as soon as feasible. *Chris B's response:* Deferred API Mixed thoughts about this. On the one hand, it's obviously a good thing, enables smarter execution, typing/schemas could result in much easier/safer to write code, etc. On the other hand, the pandas API is already massive and reasonably difficult to master, and it's a big ask to learn a new one. Dask is a good example of how NOT having a new API can be very valuable. All this to say I think adoption might be pretty low? Could be my own biases - coming from a "smallish data" user of pandas, I've never found the "write once, execute on different backends" argument especially compelling because I've never had the need.

3 3

Question
by Rémi DENISE May 31, 2017

May 31, 2017

To whom it may concern, I wanted to say that I really like pandas, it is the best for what I’m doing. For one of my script I had to use rpy2 and that create a matrix but I needed to have this matrix to pandas so I use "import pandas.rpy.common as com" and a message came : FutureWarning: The pandas.rpy module is deprecated and will be removed in a future version. We refer to external packages like rpy2. See here for a guide on how to port your code to rpy2: http://pandas.pydata.org/pandas-docs/stable/r_interface.html import pandas.rpy.common as com So I check how to port my code to rpy2 instead of use it. But when I compare the two methods, it’s not the same. In rpy2 the method return me a numpy array, but your method return me a pandas DataFrame which have the name of the rows and columns as I want. So my question is how can I have exactly the same result as "com.convert_robj" with rpy2 because it’s not the same now and I want that pandas keep this method that does easily what I want and rpy2 doesn’t. Thank you for your answer Rémi

2 1

Dev meeting doodle
by Joris Van den Bossche May 30, 2017

May 30, 2017

The last dev meeting has been more than a month ago, so maybe time to at least try to find a date for the next one. Therefore, a doodle! https://doodle.com/poll/dp6s7r3q2u6wcyup Fixed the doodle at 2pm EST / 8pm GMT / 11am PST, like the last meeting (if we want to try another time, eg like two meetings ago, please let it know) Enough content to discuss from Wes' mail and the 0.21 / 1.0 topics listed in the notes <https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-lico…> from last meeting (we should probably pick some in advance to discuss). Joris

1 1

Componentization
by Joris Van den Bossche May 12, 2017

May 12, 2017

2017-05-04 1:09 GMT+02:00 Wes McKinney <wesmckinn(a)gmail.com>: > > *TOPIC TWO:* We discussed this on the last dev meeting call, but I wanted > to see what others think and if there's some action items. To help with > more frequent pandas releases, particularly of subcomponents which are pure > Python, I wonder if we could move toward a release model of "pandas" as a > metapackage for a series of subcomponents which are packaged independently. > As an example > > pandas depends on > pandas_display (Display for humans) > pandas_io > pandas_plotting > pandas_core > > and so forth. I think it would be better to go with a single codebase for > this; I don't have a strong opinion about having separate release cycles, > it's more to help establish cleaner boundaries about use of private and > public APIs. Effectively the codebase is already organized like this, so > I'm not sure concretely what we would want to do around this. > > Would your idea be to have those as separate python packages, but in a single (github) repo? If we want to go in this direction, I would first like to see some more detailed practicalities, to be able to better evaluate if this would not needlessly make the contributing cycle more complex. Eg what would separate release cycles in practice mean? Would the subpackages then need to support multiple versions of the base package (at least master + latest stable, otherwise there is not really a benefit in a separate release cycle)? That would also mean more testing builds? How would it work with Travis with multiple packages in a single repo? (can it run separate tests depending on the PR?) Trying to establish cleaner boundaries between the private and public API's is good goal, but in principle we could also try to do this in the current directory layout (eg remove usage of private API's in pandas.plotting, pandas.io, ..). I am certainly not against a potential reorganisation on this front, but I would like to see some more advantages in doing so (as I am not familiar with such workflows from other project). Regards, Joris

1 0

Performance drop with v0.20 - changes between two dataframes
by Robin Fishbein May 11, 2017

May 11, 2017

I apologize in advance, I'm not sure where to ask about this, so I thought perhaps the dev list. The function below returns a tuple of dataframes to identify the rows added, changed, and deleted between two dataframes.* Adds and deletes should be an easy index comparison, so it should spend most of its time distinguishing changed rows from unchanged rows. With Python 3.6.0 and pandas 0.19.2 it runs on test files** in around 22 seconds, with get_loc and __getitem__ at the top of %prun. With pandas 0.20.1 it's unusably slow. I believe there's something about this expression— pd.Index(i for i in index_both if not df1t[i].equals(df2t[i])) —that is handled differently in 0.20. I'm not sure how, though, or what I could do to make this effective in v0.20. * I used equals() to address missing values and handle any potential dtype; if we guarantee no missing values, the solution is much easier. I transposed because I couldn't work out a better way to convert the rows to Series, allowing the use of equals().** About 54k rows, 4 index columns (all object), and 8 other columns (4 int, 3 obj, 1 datetime). Thanks!-Robin def delta(left, right, index_cols=None, suffixes=('_1', '_2'), reset_indexes=True, value_counts=True): df1 = left.copy() df2 = right.copy() if isinstance(index_cols, pd.Index): index_cols = list(index_cols) if index_cols: df1 = df1.set_index(index_cols) df2 = df2.set_index(index_cols) full = (pd.merge(df1, df2, left_index=True, right_index=True, how='outer', suffixes=suffixes, indicator=True) .astype({'_merge': object})) index_both = full[full._merge == 'both'].index df1t = df1.reindex(index_both).T df2t = df2.reindex(index_both).T index_changes = pd.Index(i for i in index_both if not df1t[i].equals(df2t[i])) if index_changes.size > 0: full.loc[index_changes, '_merge'] = 'c' mappings = { 'both': 'm', # match 'right_only': 'a', # add 'c': 'c', # change 'left_only': 'd', # delete } full._merge = full._merge.map(mappings) add = df2.reindex(full.loc[full._merge == 'a'].index) change = full.loc[full._merge == 'c'].drop('_merge', axis=1) delete = df1.reindex(full.loc[full._merge == 'd'].index) if reset_indexes: full = full.reset_index() add = add.reset_index() change = change.reset_index() delete = delete.reset_index() if value_counts: print(full._merge.value_counts()) return full, add, change, delete

2 3

ANN: pandas v0.20.1 released
by Tom Augspurger May 5, 2017

May 5, 2017

Hi all, I'm happy to announce that pandas 0.20.0 and 0.20.1 have been released. Pandas 0.20.1 contains a single additional change from 0.20.0 for backwards compatibility with projects using pandas' utils methods. See https://github.com/pandas-dev/pandas/pull/16250. The full release notes for 0.20.0 are below. This is a major release from 0.19.2 and includes a number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. See the Whatsnew file for more information: http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html We recommend that all users upgrade to this version. This release includes 897 commits over 5 months of development by 204 contributors. A big thank you to all contributors! Tom --- *## What is it:* pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *## Highlights of the 0.20.0 release include:* - new .agg() API for Series/DataFrame similar to the groupby-rolling-resample API's, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Integration with the feather-format, including a new top-level pd.read_feather() and DataFrame.to_feather() method, see here <http://pandas.pydata.org/pandas-docs/version/0.20/io.html#io-feather> - The .ix indexer has been deprecated, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Panel has been deprecated, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Addition of an IntervalIndex and Interval scalar type, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Improved user API when accessing levels in .groupby(), see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Improved support for UInt64 dtypes, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - A new orient for JSON serialization, orient='table', that uses the Table Schema spec, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Experimental support for exporting DataFrame.style formats to Excel, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Window Binary Corr/Cov operations now return a MultiIndexed DataFrame rather than a Panel, as Panel is now deprecated, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Support for S3 handling now uses s3fs, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Google BigQuery support now uses the pandas-gbq library, see here <http://pandas.pydata.org/pandas-docs/version/0.20/whatsnew.html#whatsnew-02…> - Switched the test framework to use pytest <http://docs.pytest.org/> *## How to get it:* Source tarballs and Windows / Mac / Linux wheels are available on PyPI (thanks to Christoph Gohlke for the windows wheels, and to Matthew Brett for setting up the Mac / Linux wheels) pip install --upgrade pip setuptools pip install --upgrade --upgrade-strategy=only-if-needed pandas Conda packages currently building, and will be available via the conda-forge channel (conda install pandas -c conda-forge). It will be available on the default channel soon. conda install -c conda-forge pandas *## Issues:* Please report any issues on our issue tracker: https://github.com/pydata/ pandas/issues/ *## Thanks to all the contributors:* - Adam J. Stewart - Adrian - Ajay Saxena - Akash Tandon - Albert Villanova del Moral - Aleksey Bilogur - Alexis Mignon - Amol Kahat - Andreas Winkler - Andrew Kittredge - Anthonios Partheniou - Arco Bast - Ashish Singal - Baurzhan Muftakhidinov - Ben Kandel - Ben Thayer - Ben Welsh - Bill Chambers - Brandon M. Burroughs - Brian - Brian McFee - Carlos Souza - Chris - Chris Ham - Chris Warth - Christoph Gohlke - Christoph Paulik - Christopher C. Aycock - Clemens Brunner - D.S. McNeil - DaanVanHauwermeiren - Daniel Himmelstein - Dave Willmer - David Cook - David Gwynne - David Hoffman - David Krych - Diego Fernandez - Dimitris Spathis - Dmitry L - Dody Suria Wijaya - Dominik Stanczak - Dr-Irv - Dr. Irv - Elliott Sales de Andrade - Ennemoser Christoph - Francesc Alted - Fumito Hamamura - Giacomo Ferroni - Graham R. Jeffries - Greg Williams - Guilherme Beltramini - Guilherme Samora - Hao Wu - Harshit Patni - Ilya V. Schurov - Iván Vallés Pérez - Jackie Leng - Jaehoon Hwang - James Draper - James Goppert - James McBride - James Santucci - Jan Schulz - Jeff Carey - Jeff Reback - JennaVergeynst - Jim - Jim Crist - Joe Jevnik - Joel Nothman - John - John Tucker - John W. O'Brien - John Zwinck - Jon M. Mease - Jon Mease - Jonathan Whitmore - Jonathan de Bruin - Joost Kranendonk - Joris Van den Bossche - Joshua Bradt - Julian Santander - Julien Marrec - Jun Kim - Justin Solinsky - Kacawi - Kamal Kamalaldin - Kerby Shedden - Kernc - Keshav Ramaswamy - Kevin Sheppard - Kyle Kelley - Larry Ren - Leon Yin - Line Pedersen - Lorenzo Cestaro - Luca Scarabello - Lukasz - Mahmoud Lababidi - Mark Mandel - Matt Roeschke - Matthew Brett - Matthew Roeschke - Matti Picus - Maximilian Roos - Michael Charlton - Michael Felt - Michael Lamparski - Michiel Stock - Mikolaj Chwalisz - Min RK - Miroslav Šedivý - Mykola Golubyev - Nate Yoder - Nathalie Rud - Nicholas Ver Halen - Nick Chmura - Nolan Nichols - Pankaj Pandey - Pawel Kordek - Pete Huang - Peter - Peter Csizsek - Petio Petrov - Phil Ruffwind - Pietro Battiston - Piotr Chromiec - Prasanjit Prakash - Rob Forgione - Robert Bradshaw - Robin - Rodolfo Fernandez - Roger Thomas - Rouz Azari - Sahil Dua - Sam Foo - Sami Salonen - Sarah Bird - Sarma Tangirala - Scott Sanderson - Sebastian Bank - Sebastian Gsänger - Shawn Heide - Shyam Saladi - Sinhrks - Stephen Rauch - Sébastien de Menten - Tara Adiseshan - Thiago Serafim - Thoralf Gutierrez - Thrasibule - Tobias Gustafsson - Tom Augspurger - Tong SHEN - Tong Shen - TrigonaMinima - Uwe - Wes Turner - Wiktor Tomczak - WillAyd - Yaroslav Halchenko - Yimeng Zhang - abaldenko - adrian-stepien - alexandercbooth - atbd - bastewart - bmagnusson - carlosdanielcsantos - chaimdemulder - chris-b1 - dickreuter - discort - dr-leo - dubourg - dwkenefick - funnycrab - gfyoung - goldenbull - hesham.shabana - jojomdt - linebp - manu - manuels - mattip - maxalbert - mcocdawc - nuffe - paul-mannino - pbreach - sakkemo - scls19fr - sinhrks - stijnvanhoey - the-nose-knows - themrmax - tomrod - tzinckgraf - wandersoncferreira - watercrossing - wcwagner - xgdgsc - yui-knk

1 0

Developing libpandas as a separate codebase + Componentization + Deferred "pandas expressions"
by Wes McKinney May 3, 2017

May 3, 2017

hi folks, Bit of a multi-tiered discussion, but it's all somewhat related so putting it all in one e-mail. *TOPIC ONE:* I have been thinking about how to proceed with pandas 2.0 development in a sane way with the following goals: * Delivering some incrementally valuable functionality to production pandas users (e.g. faster CSV reader, some faster algorithms). There might be faster multithreaded code we can make available via a memory layout conversion layer (NaNs to bitmaps, etc.) * Being able to install subcomponents of pandas 2.0 (like libpandas) alongside production pandas to get feedback from users, particularly around low-level data semantics (copy-on-write, etc.) * Migrating compiled code and utility algorithms out of current pandas codebase Changing the internals of Series and DataFrame is going to be a difficult process (and frankly, it would be easier to build a brand new project, but I am not going to advocate for that). But I think one way we can make things easier is by developing "libpandas" and its Python bindings as a separate codebase. What goes in libpandas? In my view: * The semantic contents of pandas._lib * New "guts" of Series and DataFrame, what I've been colloquially calling pandas.Array (Series with no Index) and pandas.Table (DataFrame with no index) * New implementations of Index, based on libpandas.Array * Computational libraries that presume a particular memory layout: pandas.core.algorithms, pandas.core.ops, pandas.core.nanops, etc. * Low-level IO code (moving data from other formats into new pandas data structures) The idea is that libpandas would (someday) be a hard dependency of pandas, and contain most or all of the compiled code in pandas. To simplify things for most contributors, we could publish nightly dev wheels or conda packages so that you can update libpandas in your dev environment and proceed with developing pure Python code. Let me know what you think. I've spent the majority of my net development time over the last year hardening Apache Arrow as a C++ library we can use in libpandas for physical columnar in-memory management, so I'm ready (now with the Arrow 0.3 release about ready to drop) to start making some more progress on this. *TOPIC TWO:* We discussed this on the last dev meeting call, but I wanted to see what others think and if there's some action items. To help with more frequent pandas releases, particularly of subcomponents which are pure Python, I wonder if we could move toward a release model of "pandas" as a metapackage for a series of subcomponents which are packaged independently. As an example pandas depends on pandas_display (Display for humans) pandas_io pandas_plotting pandas_core and so forth. I think it would be better to go with a single codebase for this; I don't have a strong opinion about having separate release cycles, it's more to help establish cleaner boundaries about use of private and public APIs. Effectively the codebase is already organized like this, so I'm not sure concretely what we would want to do around this. *TOPIC THREE:* I think we should start developing a "deferred pandas API" that is designed and directly developed by the pandas developer community. >From our respective experiences creating expression DSLs and other computation frameworks on top of pandas, I believe this is something where we can build something reasonable and useful. As one concrete problem this would help with: addressing some of the awkwardness around complex groupby-aggregate expressions (custom aggregations would simply be named expressions). The idea of the deferred expression API would be similar to dplyr in R: * "True" schemas (we'll have to work around pandas 0.x warts with implicit casts, etc.) * Immutable data structures / no mutation outside "amend" operations that change values by returning new objects * Less index-related stuff in this API (perhaps this is controversial, we shall see) We can create an in-memory backend for "pandas expressions" on pandas 0.x/1.0 and separately create an alternative backend using libpandas (once that is more fully baked / functional) -- this will also help provide a forcing function for implementing analytics that are required for implementing the backend. Distributed execution for us is almost certainly out of scope, and even if so we would probably want to offload onto prior art in Dask or elsewhere. So if the dask.dataframe API and the pandas expression API look different in ways that are unpleasant, we could either compile from pandas -> dask under the hood, or make API changes to make the semantics more conforming. When libpandas / pandas 2.0 is more mature we can consider building stronger out-of-core execution (plenty of prior art we can learn from here, e.g. SFrame). As far as tools to implement the deferred expression API -- I will leave this to discussion. I spent a considerable amount of time making a pandas-like expression API for SQL in Ibis (see https://github.com/cloudera/ibis/tree/master/ibis/expr) while I was at Cloudera, so there's some ideas there (like separating the "internal" AST from the "external" user expressions) that we can learn from, or fork or use some of that expression code in some way. I don't have a strong opinion as long as the expressions are as strongly-typed as possible (i.e. tables have schemas, operations have checked input and output types) and catch user errors as soon as feasible. This ended up being more text than I planned. If we want to discuss these things independently, feel free to send a reply with an altered subject line. Looking forward to see what everyone thinks. Thanks! Wes

2 1