Mailman 3 November 2017 - Pandas-dev

Proposal to change PR update policy from rebase to merge
by Joris Van den Bossche Jan. 26, 2018

Jan. 26, 2018

Hi all, Currently when PRs get outdated, we often ask to "rebase and update" (although to be honest, it mainly Jeff that is doing most of this work to ping stale PRs), and rebasing is also how it is explained in the docs ( http://pandas-docs.github.io/pandas-docs-travis/contributing.html#creating-…). And also many active contributors use rebasing while working on a PR to get in sync with changes in master. I would like to propose to change this policy from rebasing to merging (= merging master in the feature branch, creating a merge commit). Some reasons for this: - I personally think this is easier to do (certainly for less experienced git users; conflicts can be easier to solve, ..) - It makes it easier to follow what has changed (certainly if we extend 'not rebasing' with 'not squash rebasing'), making it easier review - It doesn't destroy links to github PR comments - Since we squash on merge in the end, we don't care about the additional merge commits in the PR's history Thoughts on this? If we would agree on this, that would mean: update the docs + start doing it ourselves + start asking that of contributors consistently. Regards, Joris

5 9

Help replacing workflows that used DataFrame.select
by Paul Hobson Dec. 5, 2017

Dec. 5, 2017

Hey folks, I noticed that DataFrame.select is now deprecated in favor of DataFrame.loc[index.map(selector_fxn)] PR: https://github.com/pandas-dev/pandas/pull/17633 Issue: https://github.com/pandas-dev/pandas/issues/12401 I have a lot of work flows that look something like this: res = ( data.resample(freq) .agg(agg_dict) .pipe(fxn_that_adds_many_cols) .select(complex_fxn_that_selects_a_few_cols, axis='columns') ) It's not immediately clear to me how to access all of the e.g., columns in the middle or at the end of a chain of dataframe operations. Any tips? -Paul

4 10

Changing the default max_columns and max_rows
by Clemens Brunner Nov. 28, 2017

Nov. 28, 2017

Hello! We're currently discussing a change in how data frames are displayed by default in https://github.com/pandas-dev/pandas/pull/17023. There are two proposed changes: (1) Set pd.options.display.max_columns=0 (previously this was set to 20). (2) Set pd.options.display.max_rows=20 (previously this was set to 60). Change (1) means that the number of printed columns is adapted to fit within the width of the terminal. If there are too many columns, ellipsis will be shown to indicate collapsed columns in the middle of the data frame. This doesn't work if Python is run as a Jupyter kernel (e.g. in a Jupyter notebook or in IPython QtConsole), in which case the maximum columns remain 20. Example: ======== import pandas as pd import numpy as np pd.DataFrame(np.random.rand(5, 10)) Output before (in a terminal with 100 chars width): --------------------------------------------------- 0 1 2 3 4 5 6 \ 0 0.643979 0.690414 0.018603 0.991478 0.707534 0.376765 0.670848 1 0.547836 0.810972 0.054448 0.415112 0.268120 0.904528 0.839258 2 0.582256 0.732149 0.284208 0.405197 0.213591 0.715367 0.150106 3 0.197348 0.317159 0.051669 0.738405 0.821046 0.179270 0.245793 4 0.483466 0.583330 0.999213 0.882883 0.315169 0.045712 0.897048 7 8 9 0 0.891467 0.494220 0.713369 1 0.601304 0.449880 0.266205 2 0.113262 0.360580 0.238833 3 0.798063 0.077769 0.471169 4 0.262779 0.530565 0.992084 Output after: ------------- 0 1 2 3 ... 6 7 8 9 0 0.673621 0.211505 0.943201 0.946548 ... 0.900453 0.612182 0.861933 0.710967 1 0.670855 0.834449 0.796273 0.785976 ... 0.609954 0.686663 0.684582 0.837505 2 0.544736 0.814827 0.352893 0.459556 ... 0.650993 0.735943 0.279110 0.840203 3 0.440125 0.554323 0.745462 0.940896 ... 0.544576 0.224175 0.852603 0.509837 4 0.225551 0.791834 0.476059 0.321857 ... 0.391165 0.423213 0.290683 0.954423 [5 rows x 10 columns] Change (2) implies fewer rows are displayed before auto-hiding takes place. I find that 60 rows almost always causes the terminal to scroll (most terminals have between 25-40 rows), so reducing the value to 20 increases the chance that a data frame can be observed on one terminal page. I'm not including a before/after output since it should be easy to imagine how this change affects the output. Both changes would make Pandas behave similar to R's Tidyverse (which I really like), but this should not be the main reason why these changes are a good idea. I mainly like them because these settings make (large) data frames much nicer to look at. Note that these changes affect the default values. Of course, users are free to change them back in their active Python session. Comments to both proposed changes are highly welcome (either here on the mailing list or at https://github.com/pandas-dev/pandas/pull/17023. Clemens

1 0

Pandas lite
by Aivar Annamaa Nov. 19, 2017

Nov. 19, 2017

Hi! I'm going to teach an introduction to pandas to Python newbies, and I'm looking for ways to simplify the the view to the API and/or avoid some of the pitfalls. I'd like to identify a minimal set of methods/operations, which are enough for performing most common tasks with simply-indexed data (importing/exporting from csv/Excel, selecting rows and columns by index, boolean indexing of the rows, creating new columns, simple group-by and aggregations, simple plotting, maybe also simple joins) and which have minimal potential for surprises (unexpected copies, unexpected views, confusing warnings, differences with indexing with lists vs tuples etc). Maybe even allowing only "pure" transformations a la relational algebra? We could call it an opinionated and restricted usage-scheme of pandas. The students would use this subset of the API until they gain enough experience to meet the hairier face of pandas. Has anybody tried marking a subset of pandas API for some reasons? I was also thinking about how to enforce the boundaries of this subset: * Just suggest students to stick with it. * Provide a static analysis which disallows (or warns against) the operations/tricks outside the boundaries. * a wrapper library (eg. import pandaslite as pd) which wraps required pandas classes into similar classes which publish only a subset of the pandas capabilities and perform some extra checks (eg. disallow duplicates in the index). When the students grow tough enough or need more power, they would simply replace "import pandaslite as pd" with "import pandas as pd" in their code. At the moment I'm considering experimenting with the third approach. I'd be glad to hear your comments! best regards, Aivar Annamaa

3 2

Label and Milestone cleanup
by Tom Augspurger Nov. 17, 2017

Nov. 17, 2017

We ran out of time to discuss this in the dev meeting. I want to clean up our Github labels and milestones. ## Milestones I would like to have 4 real milestones: 1. "Next Major Release" for API breaking changes that we'd like to do eventually 2. "Next Release" For bugfix / non-API breaking changes that we'd like to do eventually 3. "0.22.0" (i.e. the actual next major release) for PRs that should go into a specific release 4. "0.21.1" (i.e. the actual next minor release) for PRs that should go into a specific release The main change is the "Next Release", which I hope will clear up confusion about what file the release notes should go in. I think we can remove "Interesting Issues", remove "High Level Issue Tracking", and move "won't fix" into "No action". I'm not sure about "1.0" and "2.0" and "Someday", but maybe consolidate those. ## Labels I would like to - Start tagging "Easy" issues with "good first issue". Github gives some prominence to this tag in their UI. - Remove some of the less frequently used issues like "Closed PR, Multi Dimensional, etc." - Start using the "Needs Info" tag more often for incomplete bug reports, and regularly close issues that have been tagged as "Needs Info" and not updated in more that a couple weeks. Thoughts? Objections? Tom

4 4

Online dev meeting - wednesday 15th November 6pm UTC
by Joris Van den Bossche Nov. 12, 2017

Nov. 12, 2017

Hi all, FYI, we are planning a dev meeting coming Wednesday at 6-7pm UTC. If you are interested to join, always welcome! (https://appear.in/pandas-dev) Best, Joris

1 0