Mailman 3 September 2019 - Pandas-dev

Plans for pandas 0.25.0 and pandas 1.0
by Joris Van den Bossche Nov. 12, 2019

Nov. 12, 2019

Hi all, We had some discussion about this on the in-person dev sprint end of June, and I thought it would be good to have some public record of this as well. A pandas 0.25.0 release is close (the RC was released earlier this month), see https://github.com/pandas-dev/pandas/issues/24950 For pandas 1.0, the current plan is to finally "just do it". The idea is that it should not take too long after 0.25.0, without additional major API changes (additions are fine of course) but with removing the current deprecated functionalities. Depending on how much feedback there is on 0.25.0 and on how smoothly it goes for removing deprecated stuff, we could (maybe optimistically) target September for that. Comments certainly welcome! Joris

4 5

Discourse discussion forum
by Joris Van den Bossche Nov. 4, 2019

Nov. 4, 2019

(let's use a new thread for discourse, as it is a different discussion from the website hosting I think, regardless whether OVH might also host discourse) I am not familiar enough myself with discourse to know whether multiple projects sharing a single discourse will become annoying. But indeed, that sounds as it needs some kind of hierarchical category / tagging. For pandas itself: I think I quite like the idea of having a discourse, but *if* we do that, we should think about how that fits with / replaces / adds to /... some of the other communication channels (pandas-dev mailing list, pydata mailing list, github issues, ..). Joris On Fri, 20 Sep 2019 at 13:18, Marc Garcia <garcia.marc(a)gmail.com> wrote: > I'm fine with that conceptually, but I think Discourse will make things > quite tricky to find things then. > > We already got our discourse approved, if you want to join it an > experiment with the setting. But it's the first thing I tried, and after > you join a category (project), everything feels like it's in the same place > (even if subcategories and tags exist). And I think we need at least a > clear separation between pandas/users pandas/contributors discussions. > > May be I just couldn't find the settings, let me know if you manage to get > a multi-project set up that makes sense. > > On Fri, Sep 20, 2019 at 12:07 PM Tom Augspurger < > tom.augspurger88(a)gmail.com> wrote: > >> I'd prefer to join a discourse along with NumPy, Dask, and other PyData >> or NumFOCUS projects, rather than going out on our own. >> >> On Fri, Sep 20, 2019 at 4:47 AM Marc Garcia <garcia.marc(a)gmail.com> >> wrote: >> >>> I don't know much about discourse, but why do we want to self-host it? >>> Seems like Discourse does it for free for open source projects: >>> https://free.discourse.group/ And I don't think we want another system >>> to maintain. Am I missing something? >>> >>> I applied for https://pandas.discourse.group, so we can give it a try. >>> We should have it approved and working in couple of days. >>> >>> For what I saw, Discourse has one level of categories, so I guess we >>> want one per project, so we can have categories for "Users", >>> "Contributors", "Ecosystem"... or something similar. I guess if we have a >>> single Discourse for NumFOCUS, every project will be a category, and it'll >>> be difficult to group conversations. >>> >>> If anyone already has experience with Discourse and disagrees with my >>> guesses, please let me know. >>> >>> On Wed, Sep 18, 2019 at 4:32 PM Andy Terrel <andy(a)numfocus.org> wrote: >>> >>>> Sounds great to me. Just let me know where everything goes. >>>> >>>> NumPy wants me to help host a discourse for them, maybe OVH would be a >>>> good place to do that as well, (although I would be more inclinded if it >>>> was pydata and we had pandas, scipy, and numpy on it). >>>> >>>> -- Andy >>>> >>>> On Wed, Sep 18, 2019 at 8:51 AM Tom Augspurger < >>>> tom.augspurger88(a)gmail.com> wrote: >>>> >>>>> Sounds good w.r.t crediting OVH on those pages. >>>>> >>>>> For the ASV results at pandas.pydata.org/speed (which I now notice is >>>>> currently broken for pandas), the only thing on the webserver is a >>>>> cron job doing a `git pull` from >>>>> https://github.com/asv-runner/asv-collection, from within >>>>> `/usr/share/nginx`. >>>>> >>>>> Tom >>>>> >>>>> >>>>> On Wed, Sep 18, 2019 at 8:18 AM Marc Garcia <garcia.marc(a)gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> An update on the new website infrastructure. We need to finish >>>>>> discussing the details, but OVH is happy to provide the hosting for the >>>>>> pandas infrastructure we need. >>>>>> >>>>>> My initial idea is to credit them in the page with the rest of the >>>>>> sponsors in the new website: >>>>>> https://datapythonista.github.io/pandas-web/community/team.html#institution… and >>>>>> also in the top right corner of the runnable code widgets (see for example >>>>>> where Binder is credited here: https://spacy.io/). >>>>>> >>>>>> What I'd like to ask is: >>>>>> >>>>>> 1. For the production website and docs (static content only, for the >>>>>> traffic we need): >>>>>> https://us.ovhcloud.com/products/public-cloud/object-storage >>>>>> 2. For our tools and processes, like the benchmarks, builds, CI stuff >>>>>> (temporary publish the docs for every PR,...): >>>>>> https://www.ovh.co.uk/vps/vps-ssd.xml (VPS SSD 3) >>>>>> 3. For BinderHub (runnable code in our docs, launch tutorials on >>>>>> Binder...): https://www.ovh.co.uk/public-cloud/kubernetes/ >>>>>> >>>>>> For the BinderHub, QuantStack offered help with the set up (which is >>>>>> great, because I don't know much about Binder myself, and I'm not sure if >>>>>> anyone else does or wants to take care of this). I don't think it'll be >>>>>> easy to estimate how big is the cluster we need beforehand, but I guess we >>>>>> can add things to Binder iteratively, and have more info as we grow. >>>>>> >>>>>> OVH gave us a 200 euros voucher to experiment with the different >>>>>> services. Let me know how all this sounds, and if there are no objections, >>>>>> I'll create an account and buy those services with the voucher, and I'll >>>>>> start to prototype and see how everything works. >>>>>> >>>>>> Cheers! >>>>>> >>>>>> On Tue, Aug 20, 2019 at 11:06 PM Marc Garcia <garcia.marc(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Somehow related to the work on the new website ( >>>>>>> https://github.com/pandas-dev/pandas/pull/28014), I've been >>>>>>> discussing with the Binder team, and looks like should be quite easy soon >>>>>>> (with a Sphinx extension) to make all the documentation pages runnable with >>>>>>> Binder, directly from the website (without opening the page as a Jupyter in >>>>>>> mybinder). >>>>>>> >>>>>>> While they are very happy with the idea of having this is pandas, >>>>>>> it's uncertain if the current infrastructure Binder has got, is able to >>>>>>> handle all the traffic we would send. And scikit-learn is working on it too >>>>>>> (today they added to the dev docs a link to mybinder to run the examples). >>>>>>> >>>>>>> I'm discussing with OVH (their infrastructure provider) on whether >>>>>>> they'd be happy to provide a dedicated BinderHub specific to pandas (or may >>>>>>> be we can have one for all NumFOCUS projects). We'll see how it goes, but >>>>>>> wanted to let you know, so you're updated, and in case anyone is interested >>>>>>> in participating in the discussions. Of course before any decision is made >>>>>>> I'll open a discussion here or on GitHub. >>>>>>> >>>>>>> As part of the discussion I'm also trying to get a server for the >>>>>>> website, and one for development stuff. Specfically for the dev docs >>>>>>> (including rendered docs of every PR) and the GitHub app that will generate >>>>>>> them. I guess it should be very easy to find a sponsor for these two >>>>>>> servers (in exchange of a small note in the footer of the website, or >>>>>>> something like that). >>>>>>> >>>>>>> Let me know if you have any comment, want to be involved or whatever. >>>>>>> >>>>>>> Cheers! >>>>>>> >>>>>> _______________________________________________ >>>>>> Pandas-dev mailing list >>>>>> Pandas-dev(a)python.org >>>>>> https://mail.python.org/mailman/listinfo/pandas-dev >>>>>> >>>>> >>>> >>>> -- >>>> Andy R. Terrel, PhD >>>> President >>>> NumFOCUS >>>> andy(a)numfocus.org >>>> >>> _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev >

6 14

Fwd: [NumFOCUS Projects] Round 3: NumFOCUS Small Development Grants CFP is Open
by Marc Garcia Oct. 11, 2019

Oct. 11, 2019

The new round of the NumFOCUS small development grants has been announced. Deadline for proposals is in a bit more than a month. I copy here previous ideas for proposals: - Will: A better JSON -> DataFrame parser (I think RapidJSON came up in the past) - Will: Tighter Arrow Integration(s) - Will: Various ExtensionArrays (container support comes to mind) - Brock: Improve ASV workflow Opening the discussion here. I guess to make the proposals more specific, would be good to specify: - Summary of the proposal - Amount required - If applies, who will be working on the proposal Probably worth nothing that funds do not necessarily need to be for development time, but other initiatives like training, events... I forward the relevant parts of the email from NumFOCUS. ---------- Forwarded message --------- Hello everyone, NumFOCUS is pleased to invite proposals from its sponsored and affiliated projects for targeted small development grants three times per year. This is the third and final call for proposals for 2019. There are no restrictions on what the funding can be used for: code development; documentation work; website updates; workshops and sprints; educational, sustainability, and diversity initiatives; or other types of projects. *Yes, you may re-submit a past grant proposal that was previously not chosen for funding. * For a list of all past successful proposals, see our website: https://numfocus.org/programs/sustainability#sdg Only one application may be submitted per project per grant funding cycle. Available Funding: - Up to $5,000 per proposal. Eligibility: - Any NumFOCUS Fiscally Sponsored or Affiliated project may submit one proposal on behalf of the project per grant cycle. - Proposed work must be achievable within calendar year 2019 or the first few months of 2020. - The call is open to applicants from any nationality and can be performed at any university, institute or business worldwide (US export laws permitting). Round 3 Timeline: - *27 Oct 2019: deadline for proposal submissions* - 18 Nov 2019: proposal acceptance notifications

5 6

New website infrastructure
by Marc Garcia Sept. 25, 2019

Sept. 25, 2019

Somehow related to the work on the new website ( https://github.com/pandas-dev/pandas/pull/28014), I've been discussing with the Binder team, and looks like should be quite easy soon (with a Sphinx extension) to make all the documentation pages runnable with Binder, directly from the website (without opening the page as a Jupyter in mybinder). While they are very happy with the idea of having this is pandas, it's uncertain if the current infrastructure Binder has got, is able to handle all the traffic we would send. And scikit-learn is working on it too (today they added to the dev docs a link to mybinder to run the examples). I'm discussing with OVH (their infrastructure provider) on whether they'd be happy to provide a dedicated BinderHub specific to pandas (or may be we can have one for all NumFOCUS projects). We'll see how it goes, but wanted to let you know, so you're updated, and in case anyone is interested in participating in the discussions. Of course before any decision is made I'll open a discussion here or on GitHub. As part of the discussion I'm also trying to get a server for the website, and one for development stuff. Specfically for the dev docs (including rendered docs of every PR) and the GitHub app that will generate them. I guess it should be very easy to find a sponsor for these two servers (in exchange of a small note in the footer of the website, or something like that). Let me know if you have any comment, want to be involved or whatever. Cheers!

4 12

Dataframe summit @ EuroSciPy 2019
by Marc Garcia Sept. 15, 2019

Sept. 15, 2019

Hi there, Just to let you know that at EuroSciPy 2019 (in September in Spain) we will have a dataframe summit, to stay updated and coordinate among projects replicating the pandas API (other dataframe projects are more than welcome). Maintainers from all the main projects (pandas, dask, vaex, modin, cudf and koalas) will be attending. If you want to get involved (whether you can attend the conference or not), please DM me. More info: https://github.com/python-sprints/dataframe-summit Conference website: https://www.euroscipy.org/2019/ Cheers!

4 8

DataFrame.value_counts
by Daniel Saxton Sept. 15, 2019

Sept. 15, 2019

Currently in pandas if we want to count the values for a single column of a DataFrame we would use df["a"].value_counts(), but when we want to count combinations of more than one column we (as far as I know) have to switch syntax and use df.groupby(["a", "b"]).size(). This is a little awkward code-wise and likely carries some unnecessary overhead since we don't actually need to prepare a groupby object that can handle an arbitrary calculation on the subframes. There's some evidence of this overhead in the Series case: import numpy as np import pandas as pd s = pd.Series(np.random.randint(1, 10, 10**6)) %timeit s.value_counts() # 6.74 ms ± 78.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) %timeit s.groupby(s).size() # 11.7 ms ± 189 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) I think it would be useful and more efficient if there was a DataFrame.value_counts method, which could take a required columns argument indicating the combinations over which we want to count. This seems like a common enough operation that it might be worthwhile to add this functionality, but wanted to see what other opinions there were on this. I know pandas already has a huge number of methods and it's good to resist adding more, but I would see this more as "filling out" rather than "adding to" the API.

2 1

Version Policy following 1.0
by Tom Augspurger Sept. 14, 2019

Sept. 14, 2019

Split from https://mail.python.org/pipermail/pandas-dev/2019-July/001030.html Following 1.0, I think we stop outright breaking APIs. I think that stability will be welcome to users. We still have to decide how we deprecate APIs. The two options are 1. Rolling deprecations: Essentially what we do today: An API is deprecated in release 1.1.0 and can be removed in (say) 1.4.0. 2. SemVer: An API may be deprecated in 1.x.0. It can be removed in 2.0.0 Do people have preferences between the two? The (dis?)advantage of Semver is that all the API-breaking changes are restricted to a single releases. With rolling deprecations, the upgrades from any 1.x to 1.y should be smoother than 1.x to 2.x. Once we choose a strategy, we may want to formalize release schedules around it. Tom

6 10

NumFOCUS Newsletter
by Tom Augspurger Sept. 10, 2019

Sept. 10, 2019

Hi all, NumFOCUS is asking for this month's highlights. - recent releases - upcoming events - job or hiring announcements - calls for participation My suggestions 1. Released 0.25.1 on August 22nd. 2. Preparing for 0.25.2 with Python 3.8 compatibility 3. Participated in (/ helped organized) the DataFrame summit at EuroSciPy (Marc, can you fill in details here?) 4. Published the results of the pandas user survey: http://dev.pandas.io/pandas-blog/2019-pandas-user-survey.html Anything else to add? Tom

3 2

Series.value_counts and length of series
by Vicki Brown Sept. 10, 2019

Sept. 10, 2019

Hi - I have a dataset: <class 'pandas.core.frame.DataFrame'> RangeIndex: 237061 entries, 0 to 237060 Data columns (total 23 columns): Date 237061 non-null datetime64[ns] Station Number 237061 non-null object Depth 237061 non-null float64 ... For three of the columns, I have calculated value_counts. For two of those, the result includes the length of the set; for the third, it does not. Why not? In [1]: dt = wq_df['Date'] dt_counts = dt.value_counts() In [2]: st = wq_df['Station Number'] st_counts = st.value_counts() In [3]: dp = wq_df['Depth'] dp_counts = dp.value_counts() In [4]: dt_counts Out[4]: 1969-04-10 21 ... Name: Date, Length: 1172, dtype: int64 In [5]: st_counts Out[5]: 18 16622 ... Name: Station Number, dtype: int64 In [6]: dp_counts Out[6]: 0.5 1962 ... Name: Depth, Length: 99, dtype: int64 -- Vicki Vicki Brown cfcl.com/vlb

2 2

Monthly Dev Meeting
by Tom Augspurger Sept. 9, 2019

Sept. 9, 2019

Hi all, The next pandas meeting is this Wednesday. You can subscribe to one of these calendars with the meeting info. ical: https://calendar.google.com/calendar/ical/pgbn14p6poja8a1cf2dv2jhrmg%40grou… gcal: https://calendar.google.com/calendar/embed?src=pgbn14p6poja8a1cf2dv2jhrmg%4… Tom

1 0