Mailman 3 June 2017 - Pandas-dev

travis slowing down?
by Jeff Reback July 7, 2017

July 7, 2017

We are fairly heavy open-source users (plus we have paid additional builds) of Travis CI. Thanks for all of the hard work! We have noticed a considerable slowdown generally on builds, see here from 9 days ago: https://travis-ci.org/pandas-dev/pandas/builds/241842849 total time 2.5 hrs and compare to here (yesterday): https://travis-ci.org/pandas-dev/pandas/builds/244822108 total time 3.5 hrs we have had very minor code changes in this period. In fact I tried reverting all of this code and build times stayed elevated. I have to conclude that something in travis itself has changed. any ideas? thanks Jeff Reback pandas development team.

3 5

Pandas Deferred Expressions
by Phillip Cloud June 16, 2017

June 16, 2017

Hi all, I'd like to fork part of the thread from Wes's original email about the future of pandas and discuss all things deferred expressions. To start, here's Wes's original thoughts, and a response from Chris Bartak that was in a different thread. After I send this email I'm going to follow up with my own thoughts in a different email so I can address any specific concerns as well as offer up a list of advantages and disadvantages to this approach and lessons learned about building DSLs in Python. *Wes's post:* *TOPIC THREE:* I think we should start developing a "deferred pandas API" that is designed and directly developed by the pandas developer community. >From our respective experiences creating expression DSLs and other computation frameworks on top of pandas, I believe this is something where we can build something reasonable and useful. As one concrete problem this would help with: addressing some of the awkwardness around complex groupby-aggregate expressions (custom aggregations would simply be named expressions). The idea of the deferred expression API would be similar to dplyr in R: * "True" schemas (we'll have to work around pandas 0.x warts with implicit casts, etc.) * Immutable data structures / no mutation outside "amend" operations that change values by returning new objects * Less index-related stuff in this API (perhaps this is controversial, we shall see) We can create an in-memory backend for "pandas expressions" on pandas 0.x/1.0 and separately create an alternative backend using libpandas (once that is more fully baked / functional) -- this will also help provide a forcing function for implementing analytics that are required for implementing the backend. Distributed execution for us is almost certainly out of scope, and even if so we would probably want to offload onto prior art in Dask or elsewhere. So if the dask.dataframe API and the pandas expression API look different in ways that are unpleasant, we could either compile from pandas -> dask under the hood, or make API changes to make the semantics more conforming. When libpandas / pandas 2.0 is more mature we can consider building stronger out-of-core execution (plenty of prior art we can learn from here, e.g. SFrame). As far as tools to implement the deferred expression API -- I will leave this to discussion. I spent a considerable amount of time making a pandas-like expression API for SQL in Ibis (see https://github.com/cloudera/ibis/tree/master/ibis/expr) while I was at Cloudera, so there's some ideas there (like separating the "internal" AST from the "external" user expressions) that we can learn from, or fork or use some of that expression code in some way. I don't have a strong opinion as long as the expressions are as strongly-typed as possible (i.e. tables have schemas, operations have checked input and output types) and catch user errors as soon as feasible. *Chris B's response:* Deferred API Mixed thoughts about this. On the one hand, it's obviously a good thing, enables smarter execution, typing/schemas could result in much easier/safer to write code, etc. On the other hand, the pandas API is already massive and reasonably difficult to master, and it's a big ask to learn a new one. Dask is a good example of how NOT having a new API can be very valuable. All this to say I think adoption might be pretty low? Could be my own biases - coming from a "smallish data" user of pandas, I've never found the "write once, execute on different backends" argument especially compelling because I've never had the need.

3 3

Why is it not possible to access index entries like columns?
by Tyler Hardin June 15, 2017

June 15, 2017

Hi, Is it just a matter of implementing the lookup in the . and [] operator methods to return df.index.get_level_values() with the argument, or is there some deeper reason this can't work that I'm not seeing? E.g. I want df['date'] to do df.get_level_values('date') when there is no date column and there is a level in the index named 'date'. Presumably, this would be a secondary lookup w/ columns taking precedence -- i.e. a 'date' column would mask a 'date' index. Thanks, Tyler

3 2

Re: [Pandas-dev] [pydata] Pandas 2.0 Design Request: A more dplyr-like API
by Joris Van den Bossche June 14, 2017

June 14, 2017

2017-06-14 17:24 GMT+02:00 Paul Hobson <pmhobson(a)gmail.com>: > Just my 2 cents on indexes: > > Every time I think I'm done with them and don't need them any more, I get > into some weird situation where a complex, nested, categorical index makes > my life soooo much easier. > > I recognize that if the library and general community doesn't need them, > they can represent a significant maintenance burden. But they saved my ass > a couple of times this week. > > Stephan mentioned some ideas to make those cases where you don't need them easier (eg allow not to have an index), but there are no plans to ditch Indexes altogether (if you look at the linked issue, it speaks about "optional indexes", but Stephan's wording in the mail below was maybe a bit misleading). Joris > -paul > > On Tue, Jun 13, 2017 at 5:03 PM, Stephan Hoyer <shoyer(a)gmail.com> wrote: > >> Hi Chris, >> >> I think most of us agree with you. We've been slowly moving in this >> direction (e.g., with .assign()) and hope to do more. For example, see our speculative >> discussion <https://github.com/pandas-dev/pandas2/issues/17> concerning >> getting rid of indexes for pandas2 and a proposal for allowing indexes >> to be referenced by name >> <https://github.com/pandas-dev/pandas/issues/8162>. >> >> There are a few major obstacles here: >> 1. Coming up with concrete plans for how new APIs should work. This is >> harder than just copying dplyr, because we don't have access to >> non-standard evaluation in Python. >> 2. Figuring out how to deprecate/replace existing behavior in a minimally >> painful way, to minimize clutter of the pandas API. (Arguably, we already >> have too many methods.) >> 3. Actually implementing these changes in a consistent fashion in the >> complex pandas codebase. >> >> These are all important work, but only the last item requires actually >> writing code. Help would be appreciated on all of these. >> >> It's worth noting that some of this may actually be easier to do outside >> of pandas proper. For example, Wes and Phil have been working on a pandas >> backend to Ibis <https://github.com/ibis-project/ibis>. >> >> Best, >> Stephan >> >> On Tue, Jun 13, 2017 at 3:48 PM, Chris Said <chris.said(a)gmail.com> wrote: >> >>> Hi Pandas developers, >>> >>> I want to start by thanking all of the pandas developers for the effort >>> they've put into the project. So much of what you do is thankless, and I >>> want you to know it is really appreciated. Pandas is a huge part of my >>> day-to-day coding. >>> >>> Because I use it so much, I want to submit a request. I want somebody to >>> #MakePandasMoreLikeDplyr. To me and to almost everyone else I've talked to >>> who knows pandas and dplyr, this is more important than performance >>> improvements and arguably more important than most of the goals in the pandas >>> 2.0 design docs <https://pandas-dev.github.io/pandas2/>. >>> >>> I'm not an R guy. 95% of my work is done in pandas. But everyone I know >>> who uses pandas is constantly having to google how to do things. In >>> contrast, dplyr feels like coding at the speed of thought. In particular, >>> the combination of groupby->{mutate, summarize} is incredibly natural >>> <https://twitter.com/Chris_Said/status/715249097326768128>. It is so >>> easy to create multiple named output columns from multiple input columns. >>> That's because the definition of new columns, with reference to multiple >>> input columns, is all done inside the call to mutate / summarize. With >>> pandas, it's much more complicated and hard to remember >>> <https://gist.github.com/TomAugspurger/37097cce6a3368a8dad7cf2c8a9e5e92> >>> . >>> >>> The new transform method in 0.20 gets us part of the way there. But >>> instead of allowing users to name the output columns, it returns multi-indexed >>> columns <https://twitter.com/Chris_Said/status/861245185002323968>, >>> which for me and most other people I've talked to are unwanted >>> <https://news.ycombinator.com/item?id=14548695>. >>> >>> Thank you again for all your hard work. Just as a TL;DR: More like >>> dplyr, less injection of multi-indexes. (Could they be eliminated entirely?) >>> >>> Best, >>> Chris >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "PyData" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pydata+unsubscribe(a)googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "PyData" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pydata+unsubscribe(a)googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "PyData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pydata+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >

2 1

ANN: pandas v0.20.2 released
by Tom Augspurger June 5, 2017

June 5, 2017

I'm pleased to announce the release of pandas 0.20.2. This is a minor bug-fix release in the 0.20.x series and includes some small regression fixes, bug fixes, and performance improvements. See the Whatsnew Page <http://pandas.pydata.org/pandas-docs/version/0.20.2/whatsnew.html> to see all of the changes. We recommend that all users upgrade to this version. This was a release of 4 weeks with 67 commits by 34 authors. Tom --- *What is it* pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. *How to get it* Source tarballs and windows/mac/linux wheels are available on PyPI (thanks to Christoph Gohlke for the windows wheels, and to Matthew Brett for setting up the mac/linux wheels). Conda packages are already available via the conda-forge channel (conda install pandas -c conda-forge). It will be available on the main channel shortly. *Issues* Please report any issues on our issue tracker: https://github.com/pydata/pandas/issues *Thanks* Thanks to all the contributors: - Aaron Barber - Andrew 亮 - Becky Sweger - Christian Prinoth - Christian Stade-Schuldt - DSM - Erik Fredriksen - Hugues Valois - Jeff Reback - Jeff Tratner - JimStearns206 - John W. O'Brien - Joris Van den Bossche - JosephWagner - Keith Webber - Mehmet Ali "Mali" Akmanalp - Pankaj Pandey - Patrick Luo - Patrick O'Melveny - Pietro Battiston - RobinFiveWords - Ryan Hendrickson - SimonBaron - Tom Augspurger - WBare - bpraggastis - chernrick - chris-b1 - economy - gfyoung - jaredsnyder - keitakurita - linebp - lloydkirk

1 0