Mailman 3 February 2019 - Pandas-dev

Python library to extract tables from PDF files
by Vinayak Mehta April 30, 2019

April 30, 2019

Hello everyone! I'm a software engineer based out of New Delhi, India. I've been a long time user and have used it in countless projects and scripts! Thanks to the core developers and contributors for working on it! I recently released a Python library which lets users extract data tables out of PDF files, my first open-source library! Here's the link: https://github.com/socialcopsdev/camelot It has a similar API to the pandas read_* functions, bearing most similarity to read_html(). Like read_html(), it has a read_pdf() main interface which returns a list of pandas DataFrames for each table found in the PDF file, and contains two flavors for parsing different types of tables! I've created a comparison with other open-source PDF table extraction libraries and tools in the wiki here <https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Tab…> . I would be really grateful if you could check it out and see if its useful to you, and give me any feedback that may help me improve it, I promise if would take less than 5 minutes of your time! :) To the core devs: I was wondering if pandas would be open to accept this library as a contribution to its read_* interface? The library uses OpenCV's morphological transformations to detect lines in PDFs when flavor='lattice', which I could vendorize or re-implement. It also has two system specific dependencies which are python-tk (used by matplotlib) and ghostscript (used to convert PDF to PNG). The first one shouldn't pose a problem since pandas also uses matplotlib, and for the second one, I could look for a Python library alternative to ghostscript. Looking forward to hearing from you all! Thanks for your time! Vinayak

2 4

Remaining API changes / deprecations to do before 1.0 (wishlist)
by Joris Van den Bossche March 5, 2019

March 5, 2019

Hi all, In the summer we had the following plan: do remaining deprecations / changes for 0.24, stabilize this in 0.25, and then clean-up deprecations in 1.0. But, we need to re-evaluate if we are still on schedule for this plan. Because if there are still remaining deprecations or API changes we certainly want to do before 1.0 (which is likely I think), we probably consider still doing a couple of such changes in 0.25 (with or without an additional 0.26 between 0.25 and 1.0). But to be able to make a better idea about what we still want to do, I think it would be good to try to come up with a list of items that we would *ideally* like to do before calling it a 1.0. That should give a better idea on how long a 0.25 might take, if we want a 0.26 before doing 1.0, etc. *So, what is your whishlist of bigger API changes / deprecations you still want to do before 1.0?* What I think we are looking for are the "bigger" topics (because of their user impact and/or impossibility to do it cleanly with a warning). For example, the deprecation of a couple of esoteric functions like Series.nonzero is nice to do now, but would also not be a huge burden to deprecate it after 1.0 and keep that warning throughout 1.x. So this would certainly not be a blocker for 1.0. On the other hand, topics like deprecating the inplace keyword, or changing the return value when indexing into an index with duplicates (#9519 <https://github.com/pandas-dev/pandas/issues/9519>), deprecation of SparseDataFrame, ... to name a few, those have a much bigger impact, and could potentially be seen as a blocker (or at least an issue to try to focus on). And further, much bigger things like complete refactor of the BlockManager, are then again topics clearly out of scope for 1.0 So it the middle category kind of issues that I think we should try to list. The list will probably be too big, but if we have such a list then we can at least have a basis for discussion and try to prioritize / identify some actual blockers. Cheers Joris

3 4

DataFrame Object creation
by Bruno Xavier Feb. 11, 2019

Feb. 11, 2019

I'm new in Pandas Library and I got a doubt on DataFrame object construction. When building from a list the shape of a dataframe is 1 column with the row-count equaling the number of list elements. Unlike that, if we build it from a dictionary it's shape is 1 row with the column-count equalling the number of key-value pairs. My question is why they are different, why not made both follow the same pattern? -- *Atenciosamente, * *Bruno Xavier Leitão*

1 0

How Far do we take ExtensionArrays?
by Tom Augspurger Feb. 8, 2019

Feb. 8, 2019

This is something I've been mulling over the past few days: how much do we want ExtensionArrays to change pandas? They've been great so far at addressing some of the shortcomings of NumPy's type system, but I imagine that users will be interested in pushing things even further. For example, users have been asking for proper support for nested data. Now that we have ExtensionArrays, things are essentially solved at the memory level (by e.g. Apache Arrow). But, I imagine that the set of APIs typically used for nested data is quite different from those used for flat, tabular data pandas handles thus far. If we want to properly support nested data, what tolerance do we have for it "cluttering" the existing API? As another semi-example, users may be interested in storing some or all their data on a GPU in an ExtensionArray or arrays backed by GPU-memory. I suspect that some things work quite well currently, e.g. `Series.sort_values` will dispatch to the `ExtensionArray.argsort`, which can use a GPU-accelerated sorting algorithm. But other parts of pandas (anything in Cython, for example) won't necessarily work. How much are we willing to refactor pandas' internals to support something that's going to live outside pandas (as a GPU extension array likely would)? Finally (and this may be a topic for another day) have people thought about how 3rd-party EAs fit in with the potential block manager rewrite? IIUC, one of the goals there was a stable C API to the memory inside a DataFrame. Does anyone know how that would work with a array that doesn't (or can't) implement the buffer protocol?

4 5

ANN: Pandas 0.24.1 Released
by Tom Augspurger Feb. 5, 2019

Feb. 5, 2019

This is a minor bug-fix release in the 0.24.x series and includes some regression fixes and bug fixes. We recommend that all users upgrade to this version. See the full whatsnew <http://pandas.pydata.org/pandas-docs/version/0.24.2/whatsnew/v0.24.2.html> for a list of all the changes. The release can be installed with conda from the defaults and conda-forge channels: conda install pandas Or via PyPI: python -m pip install --upgrade pandas

2 1