Mailman 3 October 2019 - Pandas-dev

ROADMAP proposal: Consistent missing value handling with new NA scalar
by Joris Van den Bossche Nov. 20, 2019

Nov. 20, 2019

Hi all, I would like to propose a revisit of missing value handling in pandas. It's already being discussed on github ( https://github.com/pandas-dev/pandas/issues/28095), but want to mention this on the mailing list as well for broader feedback. A more detailed proposal can be found here: https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB, and discussion can be found at the above github issue. A summary of the proposal is to introduce *a new NA value (singleton) for representing scalar missing values* (instead of np.nan) that can be used consistently across all data types. This could be achieved under the hood by using a mask-based approach to store the missing values on the array/series-level, but the main discussion here is about the user-facing API: the scalar NA value and the behaviour of NA in several operation. Motivation for this change: - *Consistent user interface.* Currently, the value you get back for a missing scalar (eg from scalar access s[idx]) depends on the data type (np.nan for many, but pd.NaT for datetime-likes). Some types support missing values, others don't. This proposal would ensure you get back pd.NA regardless of the dtype. - *No "mis-use" of the np.nan floating point value.* The NaN value is a specific floating point value, and not necessarily an indicator for missing values (although pandas has always used it that way). And because we also use it for other dtypes, you get back a float value for non-float dtypes, giving misleading dtype information. - *A missing value that behaves accordingly.* Our current behaviour of missing values is inherited of the np.nan behaviour. Other languages that have a NA/NULL value that is distinguished from NaN (eg Julia, SQL, R) typically have different behaviour in comparison and logical operations. For example, comparison with NA could give NA instead of False, and consequently we need to have a boolean dtype with NA support. A new NA value opens up the possibility of having such behaviour. - An "NA" scalar *matches the terminology* that is used throughout pandas in functions and argument names (isna, dropna, fillna, skipna, …). See the proposal <https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB> for more details. This has of course many consequences in the user API of pandas. Initially, it could therefore be introduced optionally (eg only in the new data types as nullable integer or string dtype). And given those pervasive changes, many eyes on it are important. *So feedback on this idea would be greatly appreciated!* Joris

2 4

Plans for pandas 0.25.0 and pandas 1.0
by Joris Van den Bossche Nov. 12, 2019

Nov. 12, 2019

Hi all, We had some discussion about this on the in-person dev sprint end of June, and I thought it would be good to have some public record of this as well. A pandas 0.25.0 release is close (the RC was released earlier this month), see https://github.com/pandas-dev/pandas/issues/24950 For pandas 1.0, the current plan is to finally "just do it". The idea is that it should not take too long after 0.25.0, without additional major API changes (additions are fine of course) but with removing the current deprecated functionalities. Depending on how much feedback there is on 0.25.0 and on how smoothly it goes for removing deprecated stuff, we could (maybe optimistically) target September for that. Comments certainly welcome! Joris

4 5

Discourse discussion forum
by Joris Van den Bossche Nov. 4, 2019

Nov. 4, 2019

(let's use a new thread for discourse, as it is a different discussion from the website hosting I think, regardless whether OVH might also host discourse) I am not familiar enough myself with discourse to know whether multiple projects sharing a single discourse will become annoying. But indeed, that sounds as it needs some kind of hierarchical category / tagging. For pandas itself: I think I quite like the idea of having a discourse, but *if* we do that, we should think about how that fits with / replaces / adds to /... some of the other communication channels (pandas-dev mailing list, pydata mailing list, github issues, ..). Joris On Fri, 20 Sep 2019 at 13:18, Marc Garcia <garcia.marc(a)gmail.com> wrote: > I'm fine with that conceptually, but I think Discourse will make things > quite tricky to find things then. > > We already got our discourse approved, if you want to join it an > experiment with the setting. But it's the first thing I tried, and after > you join a category (project), everything feels like it's in the same place > (even if subcategories and tags exist). And I think we need at least a > clear separation between pandas/users pandas/contributors discussions. > > May be I just couldn't find the settings, let me know if you manage to get > a multi-project set up that makes sense. > > On Fri, Sep 20, 2019 at 12:07 PM Tom Augspurger < > tom.augspurger88(a)gmail.com> wrote: > >> I'd prefer to join a discourse along with NumPy, Dask, and other PyData >> or NumFOCUS projects, rather than going out on our own. >> >> On Fri, Sep 20, 2019 at 4:47 AM Marc Garcia <garcia.marc(a)gmail.com> >> wrote: >> >>> I don't know much about discourse, but why do we want to self-host it? >>> Seems like Discourse does it for free for open source projects: >>> https://free.discourse.group/ And I don't think we want another system >>> to maintain. Am I missing something? >>> >>> I applied for https://pandas.discourse.group, so we can give it a try. >>> We should have it approved and working in couple of days. >>> >>> For what I saw, Discourse has one level of categories, so I guess we >>> want one per project, so we can have categories for "Users", >>> "Contributors", "Ecosystem"... or something similar. I guess if we have a >>> single Discourse for NumFOCUS, every project will be a category, and it'll >>> be difficult to group conversations. >>> >>> If anyone already has experience with Discourse and disagrees with my >>> guesses, please let me know. >>> >>> On Wed, Sep 18, 2019 at 4:32 PM Andy Terrel <andy(a)numfocus.org> wrote: >>> >>>> Sounds great to me. Just let me know where everything goes. >>>> >>>> NumPy wants me to help host a discourse for them, maybe OVH would be a >>>> good place to do that as well, (although I would be more inclinded if it >>>> was pydata and we had pandas, scipy, and numpy on it). >>>> >>>> -- Andy >>>> >>>> On Wed, Sep 18, 2019 at 8:51 AM Tom Augspurger < >>>> tom.augspurger88(a)gmail.com> wrote: >>>> >>>>> Sounds good w.r.t crediting OVH on those pages. >>>>> >>>>> For the ASV results at pandas.pydata.org/speed (which I now notice is >>>>> currently broken for pandas), the only thing on the webserver is a >>>>> cron job doing a `git pull` from >>>>> https://github.com/asv-runner/asv-collection, from within >>>>> `/usr/share/nginx`. >>>>> >>>>> Tom >>>>> >>>>> >>>>> On Wed, Sep 18, 2019 at 8:18 AM Marc Garcia <garcia.marc(a)gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> An update on the new website infrastructure. We need to finish >>>>>> discussing the details, but OVH is happy to provide the hosting for the >>>>>> pandas infrastructure we need. >>>>>> >>>>>> My initial idea is to credit them in the page with the rest of the >>>>>> sponsors in the new website: >>>>>> https://datapythonista.github.io/pandas-web/community/team.html#institution… and >>>>>> also in the top right corner of the runnable code widgets (see for example >>>>>> where Binder is credited here: https://spacy.io/). >>>>>> >>>>>> What I'd like to ask is: >>>>>> >>>>>> 1. For the production website and docs (static content only, for the >>>>>> traffic we need): >>>>>> https://us.ovhcloud.com/products/public-cloud/object-storage >>>>>> 2. For our tools and processes, like the benchmarks, builds, CI stuff >>>>>> (temporary publish the docs for every PR,...): >>>>>> https://www.ovh.co.uk/vps/vps-ssd.xml (VPS SSD 3) >>>>>> 3. For BinderHub (runnable code in our docs, launch tutorials on >>>>>> Binder...): https://www.ovh.co.uk/public-cloud/kubernetes/ >>>>>> >>>>>> For the BinderHub, QuantStack offered help with the set up (which is >>>>>> great, because I don't know much about Binder myself, and I'm not sure if >>>>>> anyone else does or wants to take care of this). I don't think it'll be >>>>>> easy to estimate how big is the cluster we need beforehand, but I guess we >>>>>> can add things to Binder iteratively, and have more info as we grow. >>>>>> >>>>>> OVH gave us a 200 euros voucher to experiment with the different >>>>>> services. Let me know how all this sounds, and if there are no objections, >>>>>> I'll create an account and buy those services with the voucher, and I'll >>>>>> start to prototype and see how everything works. >>>>>> >>>>>> Cheers! >>>>>> >>>>>> On Tue, Aug 20, 2019 at 11:06 PM Marc Garcia <garcia.marc(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Somehow related to the work on the new website ( >>>>>>> https://github.com/pandas-dev/pandas/pull/28014), I've been >>>>>>> discussing with the Binder team, and looks like should be quite easy soon >>>>>>> (with a Sphinx extension) to make all the documentation pages runnable with >>>>>>> Binder, directly from the website (without opening the page as a Jupyter in >>>>>>> mybinder). >>>>>>> >>>>>>> While they are very happy with the idea of having this is pandas, >>>>>>> it's uncertain if the current infrastructure Binder has got, is able to >>>>>>> handle all the traffic we would send. And scikit-learn is working on it too >>>>>>> (today they added to the dev docs a link to mybinder to run the examples). >>>>>>> >>>>>>> I'm discussing with OVH (their infrastructure provider) on whether >>>>>>> they'd be happy to provide a dedicated BinderHub specific to pandas (or may >>>>>>> be we can have one for all NumFOCUS projects). We'll see how it goes, but >>>>>>> wanted to let you know, so you're updated, and in case anyone is interested >>>>>>> in participating in the discussions. Of course before any decision is made >>>>>>> I'll open a discussion here or on GitHub. >>>>>>> >>>>>>> As part of the discussion I'm also trying to get a server for the >>>>>>> website, and one for development stuff. Specfically for the dev docs >>>>>>> (including rendered docs of every PR) and the GitHub app that will generate >>>>>>> them. I guess it should be very easy to find a sponsor for these two >>>>>>> servers (in exchange of a small note in the footer of the website, or >>>>>>> something like that). >>>>>>> >>>>>>> Let me know if you have any comment, want to be involved or whatever. >>>>>>> >>>>>>> Cheers! >>>>>>> >>>>>> _______________________________________________ >>>>>> Pandas-dev mailing list >>>>>> Pandas-dev(a)python.org >>>>>> https://mail.python.org/mailman/listinfo/pandas-dev >>>>>> >>>>> >>>> >>>> -- >>>> Andy R. Terrel, PhD >>>> President >>>> NumFOCUS >>>> andy(a)numfocus.org >>>> >>> _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev >

6 14

GH Issue Labels
by Brock Mendel Oct. 21, 2019

Oct. 21, 2019

After doing some issue triage on older issues, I'd like to solicit community opinions on what labels are useful. In particular: 1) Are the "Difficulty Advanced/Intermediate", "Effort High/Low/Medium", "Prio-high/low/medium" labels useful for anyone? "Prio High" I understand, but the others are mostly noise to me. 2) "Style" appears ambiguous. Some people (me) think it refers to coding style, but it also gets applied to things issues involving the Styler class. Is there a canonical answer? 3) Any objections to removing labels with 0 open Issues/PRs? 4) Issues involving pd.eval don't have a clear home. Suggestions?

6 5

ANN: Pandas 0.25.2 released!
by Tom Augspurger Oct. 20, 2019

Oct. 20, 2019

This is a minor bug-fix release in the 0.25.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. This is the first pandas release to support Python 3.8 See the full whatsnew <https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.2.html> for a list of all the changes. The release can be installed with conda from the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the pandas issue tracker <https://github.com/pandas-dev/pandas/issues>. Note that there was an issue with building the PDF documentation. It will be uploaded later. - Tom

2 2

Re: [Pandas-dev] Proposal to include Numba as a required dependency for rolling operations
by Maximilian Roos Oct. 15, 2019

Oct. 15, 2019

Hi Matthew, That looks very exciting. Have you seen Numbagg? That already has some rolling algos implemented over n-dimensions, and pandas could easily take those over 1-2 dimensions. It's authored by Stephan Hoyer. I added an exponential moving average so we could have a n-dimensional version in xarray, and basically copied the pandas implementation into numba-compatible code (and added attribution to pandas). Your implementation looks v interesting and looks flexible. To the extent we can share implementations of the underlying algos, that would be great. Max On Tue, 15 Oct 2019 at 12:00, <pandas-dev-request(a)python.org> wrote: > > ---------- Forwarded message ---------- > From: Matthew Roeschke <emailformattr(a)gmail.com> > To: pandas-dev <pandas-dev(a)python.org> > Cc: > Bcc: > Date: Mon, 14 Oct 2019 23:21:12 -0700 > Subject: [Pandas-dev] [pandas-dev] Proposal to include Numba as a required > dependency for rolling operations > Hello All, > > I have been working on a proof of concept > <https://github.com/twosigma/pandas/blob/feature/generalized_window_operatio…> that > implements rolling mean > <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.wind…> > and rolling apply > <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.wind…> > in Numba without changing the current pandas rolling API. Numba is an > attractive substitute over the current Cython implementation as > maintainable, pure Python code and potential performance improvements. > Stemming from the proof of concept, I'd like to propose to add Numba as a > required dependency in pandas 1.0 and have rolling mean dispatch to Numba. > > Please see the associated GitHub issue > <https://github.com/pandas-dev/pandas/issues/28987> for further details > and discussion. > > Thanks, > Matt > -- > Matthew Roeschke <http://mroeschke.github.io> > > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev >

1 0

[pandas-dev] Proposal to include Numba as a required dependency for rolling operations
by Matthew Roeschke Oct. 15, 2019

Oct. 15, 2019

Hello All, I have been working on a proof of concept <https://github.com/twosigma/pandas/blob/feature/generalized_window_operatio…> that implements rolling mean <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.wind…> and rolling apply <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.wind…> in Numba without changing the current pandas rolling API. Numba is an attractive substitute over the current Cython implementation as maintainable, pure Python code and potential performance improvements. Stemming from the proof of concept, I'd like to propose to add Numba as a required dependency in pandas 1.0 and have rolling mean dispatch to Numba. Please see the associated GitHub issue <https://github.com/pandas-dev/pandas/issues/28987> for further details and discussion. Thanks, Matt -- Matthew Roeschke <http://mroeschke.github.io>

1 0

Donation for GitHub Pandas-Dev
by Alex Bill Oct. 12, 2019

Oct. 12, 2019

Hello Pandas, We are a small business impressed by your open source initiative on https://github.com/pandas-dev. Our management support different open-source projects under a limited budget on a regular basis. You have made the final list.We are looking forward to supporting you for the year ahead-either through monthly or quarterly donations. Depending on what you prefer.Please let us know the payment mode that will work for you, so we may proceed. We are hoping you will accept our humble gesture. Alex Bill Blogger | Internet Marketing e: [http://amsofttech.fstracker.io/email/track/click?hash=eyJ0eXAiOiJKV1QiLCJhb… [http://amsofttech.fstracker.io/email/track/click?hash=eyJ0eXAiOiJKV1QiLCJhb…] w: NamoBOT.com Unsubscribe [http://amsofttech.fstracker.io/email/track/unsubscribe?hash=eyJ0eXAiOiJKV1Q…]

2 1

SciPy user documentation survey - please participate
by Ralf Gommers Oct. 12, 2019

Oct. 12, 2019

Hi everyone, Maja, our technical writer for Season of Docs for SciPy, created a survey specifically about how users use the SciPy docs and what they would like to see improved. She sent out the link to scipy-dev before and it was shared on Twitter. We've received some really valuable responses, and would love to get more. If you're a SciPy user, this is a very easy way to help the project! https://docs.google.com/forms/d/e/1FAIpQLSeBAO0UFKDZyKpg2XzRslsLJVHU61ugjc1… Apologies for the cross-post. We do a survey about once a decade and Maja is putting in a lot of work creating and analyzing the survey, so it's worth some more exposure and a couple of minutes of your time! Cheers, Ralf

1 0

Fwd: [NumFOCUS Projects] Round 3: NumFOCUS Small Development Grants CFP is Open
by Marc Garcia Oct. 11, 2019

Oct. 11, 2019

The new round of the NumFOCUS small development grants has been announced. Deadline for proposals is in a bit more than a month. I copy here previous ideas for proposals: - Will: A better JSON -> DataFrame parser (I think RapidJSON came up in the past) - Will: Tighter Arrow Integration(s) - Will: Various ExtensionArrays (container support comes to mind) - Brock: Improve ASV workflow Opening the discussion here. I guess to make the proposals more specific, would be good to specify: - Summary of the proposal - Amount required - If applies, who will be working on the proposal Probably worth nothing that funds do not necessarily need to be for development time, but other initiatives like training, events... I forward the relevant parts of the email from NumFOCUS. ---------- Forwarded message --------- Hello everyone, NumFOCUS is pleased to invite proposals from its sponsored and affiliated projects for targeted small development grants three times per year. This is the third and final call for proposals for 2019. There are no restrictions on what the funding can be used for: code development; documentation work; website updates; workshops and sprints; educational, sustainability, and diversity initiatives; or other types of projects. *Yes, you may re-submit a past grant proposal that was previously not chosen for funding. * For a list of all past successful proposals, see our website: https://numfocus.org/programs/sustainability#sdg Only one application may be submitted per project per grant funding cycle. Available Funding: - Up to $5,000 per proposal. Eligibility: - Any NumFOCUS Fiscally Sponsored or Affiliated project may submit one proposal on behalf of the project per grant cycle. - Proposed work must be achievable within calendar year 2019 or the first few months of 2020. - The call is open to applicants from any nationality and can be performed at any university, institute or business worldwide (US export laws permitting). Round 3 Timeline: - *27 Oct 2019: deadline for proposal submissions* - 18 Nov 2019: proposal acceptance notifications

5 6