Mailman 3 June 2019 - Pandas-dev

Tidelift
by Joris Van den Bossche Aug. 8, 2019

Aug. 8, 2019

Hi all, We discussed this on the last dev chat, but putting it on the mailing list for those who were not present: we are planning to contact Tidelift to enter into a sponsor agreement for Pandas. The idea is to follow what NumPy (and recently also Scipy) did to have an agreement between Tidelift and NumFOCUS instead of an individual maintainer (see their announcement mail: https://mail.python.org/pipermail/numpy-discussion/2019-April/079370.html). Blog with overview about Tidelift: https://blog.tidelift .com/how-to-start-earning-money-for-your-open-source-project-with-tidelift. We didn't discuss yet what to do specifically with those funds, that should still be discussed in the future. Cheers, Joris

10 22

Pandas - FOSSA Badge
by Mahak Bandi June 27, 2019

June 27, 2019

Hey Pandas Team, One of our engineers stumbled upon your project, Pandas, on Github and posted it to our Slack! I think it's an awesome project. As you finish up development, I thought it would be great to offer up FOSSA for free to your project. We ship a tool that tracks your dependencies at each commit and scans them for licenses and copyright violations. We can block PR's that bring in bad dependencies, auto-generate attribution/NOTICE files, and more. We work with a lot of open source projects that you may have heard of ( ESLint <https://github.com/eslint/eslint>, Mocha <http://github.com/mochajs/mocha>, Webpack <https://github.com/webpack/webpack>/Grunt <https://github.com/gruntjs/grunt>, Moment <https://github.com/moment/moment>, Prometheus <https://github.com/prometheus/prometheus>, etc.). We also just released a new badge that a couple other popular projects like Kubernetes and Dockers are getting on (screenshots below): [image: FOSSA_badge_example.png] [image: FOSSA_badge_example_2.png] If you're interested, I'd love to help you get something in place. Feel free to import Pandas at http://fossa.com and I can upgrade your account for free and follow up with some PRs! Best, Mahak Bandi

1 0

Old Version Number On The Website?
by Pete[r] Landwehr June 26, 2019

June 26, 2019

Hey all, My apologies if this was covered in the last dev meeting or is simply the way the website has always been & I've just failed to notice, but the central release notes column on pandas.pydata.org appears to be one major version behind the current release. The "Versions" column shows the current release as 0.24.2 and the development release as 0.25.0, but the central column with change log notes shows the latest release as v0.23.4 Final (August 3, 2018). Is there a way to get the website updated? It's a bit confusing. Best, pml

2 1

Re: [Pandas-dev] Unexpected error using pd.read_json with chunksize
by Vicki Brown June 24, 2019

June 24, 2019

On Mon, 24 Jun 2019 09:11:01 -0400 William Ayd <william.ayd(a)icloud.com> wrote >> Can you determine which line in the file is throwing the error? It doesn't seem to matter. I've pulled all three of the records in my 3-record test file out into individual files. They each fail the same way. I also tried sending one of those records directly through json.loads import json json_string = """ {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"} """ data = json.loads(json_string) type(data) where it works as expected. (I don't have the precise_float argument set. Perhaps that's failing?) - Vicki > On Jun 23, 2019, at 10:54 PM, Vicki Brown <vlb(a)cfcl.com> wrote: > > TL;DR > > I am getting an error I don't understand from my pd.read_json reader: > > ValueError: Unexpected character found when decoding 'false' > > > Details > > I am working a set of code exercises that use a Yelp reviews dataset. At this point in the exercises I am supposed to read in review.json which has one JSON record per line (and 6 million lines! ;-). > > Because the complete dataset file is enormous, the tutorial recommends using chunksize and building a json reader. When I try this, I get an error, even with my test input. > > I am befuddled. > > TESTS: > > I have smaller versions of the JSON file, with 100, 3, and 1 records, for testing. I can read all of the test files into a pandas dataframe and examine them, e.g.. > > path = 'file://localhost/Users/vlb/Learn/DSC_Intro/' > filename = path + 'yelp_dataset/review_test.json' > > # read the entire file > reviews = pd.read_json(filename, lines=True) > > I cannot read any of the test files in chunks. I get the same error. > > CODE: > > My code currently looks like this > > path 'file://localhost/Users/.../DSC_Intro/' > filename = path + 'yelp_dataset/review_test.json' > > # create a reader to read in chunks > > review_reader = pd.read_json(StringIO(filename), lines=True, chunksize=1) > > > `type(review_reader)` returns > > pandas.io.json.json.JsonReader > > as expected. However > > for chunk in review_reader: > print(chunk) > > throws an error on all test files. > > /Local/Users/vlb/anaconda3/lib/python3.7/site-packages/pandas/io/json/json.py in _parse_no_numpy(self) > 869 if orient == "columns": > 870 self.obj = DataFrame( > --> 871 loads(json, precise_float=self.precise_float), dtype=None) > 872 elif orient == "split": > 873 decoded = {str(k): v for k, v in compat.iteritems( > > ValueError: Unexpected character found when decoding 'false' > > INPUT: Sample JSON > > {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"} > > references: > https://www.yelp.com/dataset/documentation/main > https://courses.springboard.com/courses/448797 > -- Vicki Vicki Brown cfcl.com/vlb

2 2

Unexpected error using pd.read_json with chunksize
by Vicki Brown June 24, 2019

June 24, 2019

TL;DR I am getting an error I don't understand from my pd.read_json reader: ValueError: Unexpected character found when decoding 'false' Details I am working a set of code exercises that use a Yelp reviews dataset. At this point in the exercises I am supposed to read in review.json which has one JSON record per line (and 6 million lines! ;-). Because the complete dataset file is enormous, the tutorial recommends using chunksize and building a json reader. When I try this, I get an error, even with my test input. I am befuddled. TESTS: I have smaller versions of the JSON file, with 100, 3, and 1 records, for testing. I can read all of the test files into a pandas dataframe and examine them, e.g.. path = 'file://localhost/Users/vlb/Learn/DSC_Intro/' filename = path + 'yelp_dataset/review_test.json' # read the entire file reviews = pd.read_json(filename, lines=True) I cannot read any of the test files in chunks. I get the same error. CODE: My code currently looks like this path 'file://localhost/Users/.../DSC_Intro/' filename = path + 'yelp_dataset/review_test.json' # create a reader to read in chunks review_reader = pd.read_json(StringIO(filename), lines=True, chunksize=1) `type(review_reader)` returns pandas.io.json.json.JsonReader as expected. However for chunk in review_reader: print(chunk) throws an error on all test files. /Local/Users/vlb/anaconda3/lib/python3.7/site-packages/pandas/io/json/json.py in _parse_no_numpy(self) 869 if orient == "columns": 870 self.obj = DataFrame( --> 871 loads(json, precise_float=self.precise_float), dtype=None) 872 elif orient == "split": 873 decoded = {str(k): v for k, v in compat.iteritems( ValueError: Unexpected character found when decoding 'false' INPUT: Sample JSON {"review_id":"Amo5gZBvCuPc_tZNpHwtsA","user_id":"DzZ7piLBF-WsJxqosfJgtA","business_id":"qx6WhZ42eDKmBchZDax4dQ","stars":5.0,"useful":1,"funny":0,"cool":0,"text":"Our family LOVES the food here. Quick, friendly, delicious, and a great restaurant to take kids to. 5 stars!","date":"2017-03-27 01:14:37"} references: https://www.yelp.com/dataset/documentation/main https://courses.springboard.com/courses/448797

2 1

Arithmetic Proposal
by Brock Mendel June 20, 2019

June 20, 2019

I've been working on arithmetic/comparison bugs and more recently on performance problems caused by fixing some of those bugs. After trying less-invasive approaches, I've concluded a fairly big fix is called for. This is an RFC for that proposed fix. ------ In 0.24.0 we fixed some arithmetic bugs in DataFrame operations by making DataFrame arithmetic ops operate column-by-column, dispatching to the Series implementations. This led to a significant performance hit for operations on DataFrames with many columns (#24990, #26061). To restore the lost performance, we need to have these operations take place at the Block level. To prevent DataFrame behavior from diverging from Series behavior (again), we need to retain a single shared implementation. This is a proposal for how meet these two needs. Proposal: - Allow EA to support 2D arrays - Use PandasArray to back Block subclasses currently backed by ndarray - Implement arithmetic and comparison ops directly on PandasArray, then have Series, DataFrame, and Index ops pass through to the PandasArray implementations. Fixes: - Performance degradation in DataFrame ops (#24990, #26061) - The last remaining inconsistencies between Index and Series ops (#19322, #18824) - Most of the xfailing arithmetic tests - #22120: Transposing dataframe loses dtype and ExtensionArray - #24600 BUG: DataFrame[Sparse] quantile fails because SparseArray has no reshape - #23925 DataFrame Quantile Broken with Datetime Data Other Upsides: - Series constructor could dispatch to pd.array, de-duplicating a lot of code. - Easier to move to Arrow backend if Blocks are numpy-naive. - Make EA closer to a drop-in replacement for np.ndarray, necessary if we want e.g. xarray to find them directly useful (#24716, #24583) - Block/BlockManager simplifications, see below. Downsides: - Existing constructors assume 1D - Existing downstream authors assume 1D - Reduction ops (of which there aren't many) don't have axis kwarg ATM - But for PandasArray they just pass through to nanops, which already have+test the axis kwargs - For DatetimeArray/TimedeltaArray/PeriodArray, I'm the one implementing the reductions and am OK with this extra complication. Block Simplifications: - Blocks have three main attributes: values, mgr_locs, and ndim - ndim is _usually_ the same as values.ndim, the exceptions being for cases where type(values) is restricted to 1D - Without these restrictions, we can get rid of: - Block.ndim, associated kludgy ndim-checking code - numerous can-this-be-reshaped/transposed checks and special cases in Block and BlockManager code (which are buggy anyway, e.g. #23925) - With ndim gone, we can then get rid of mgr_locs! - The blocks themselves never use mgr_locs except when passing to their own constructors. - mgr_locs makes _much_ more sense as an attribute of the BlockManager - With mgr_locs gone, Block becomes just a thin wrapper around an EA Implementation Strategy: - Remove the 1D restriction - Fairly small tweak, EA subclass must define `shape` instead of `__len__`; other attrs define in terms of shape. - Define `transpose`, `T`, `reshape`, and `ravel` - With this done, several tasks can proceed in parallel: - simplifications in core.internals, as special-cases for 1D-only can be removed - implement and test arithmetic ops on PandasArray - back Blocks with PandasArray - back Index (and numeric subclasses) with PandasArray - Change DataFrame, Series, Index ops to pass through to underlying Blocks/PandasArrays

6 15

Pandas dev meeting June 20th
by Tom Augspurger June 20, 2019

June 20, 2019

Hi all, The next pandas dev meeting is June 20th (tomorrow) at 17:00 UTC / 12:00 central. All are welcome to join. Minutes: https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-lico… Hangout: meet.google.com/xbh-thar-tzw Tom

1 0

Pandas dev sprint June 27-30 @ Nashville
by Joris Van den Bossche June 11, 2019

June 11, 2019

Dear all, We are planning to do a pandas sprint end of June in Nashville (Tennessee, USA): June 27-30. We will be meeting with some of the core devs (so not a sprint to jump-start newcomers in this case), but sending this to the mailing list to invite other pandas (or related libraries) contributors. The exact planning of the sprint still needs to be discussed, but we will probably be hacking and discussing on pandas, extension arrays, next versions of pandas, etc. *So if you are interested, let me know something! *We want to keep the number of participants somewhat limited, and also need to plan the location and funding, so please state your interest *before May 30*. If you would like to participate, but not sure if you would fit at such a sprint, don't hesitate to mail me personally. Best, Joris

6 9

Panel on 0.25
by Andrés Gutiérrez June 10, 2019

June 10, 2019

Hi, what are the plans for Panel objects for the next release Pandas 0.25? Are you going to keep it or definitely drop it? if so, are you going to store it in a particular Python package? Thanks! -- *| Andrés Gutiérrez Arcia* *| andres.gutierrez.arcia(a)gmail.com <andres.gutierrez.arcia(a)gmail.com>* | https://github.com/andrsGutirrz | https://www.linkedin.com/in/andres-gutierrez-arcia

3 3