Mailman 3 November 2018 - Pandas-dev

Python library to extract tables from PDF files
by Vinayak Mehta April 30, 2019

April 30, 2019

Hello everyone! I'm a software engineer based out of New Delhi, India. I've been a long time user and have used it in countless projects and scripts! Thanks to the core developers and contributors for working on it! I recently released a Python library which lets users extract data tables out of PDF files, my first open-source library! Here's the link: https://github.com/socialcopsdev/camelot It has a similar API to the pandas read_* functions, bearing most similarity to read_html(). Like read_html(), it has a read_pdf() main interface which returns a list of pandas DataFrames for each table found in the PDF file, and contains two flavors for parsing different types of tables! I've created a comparison with other open-source PDF table extraction libraries and tools in the wiki here <https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Tab…> . I would be really grateful if you could check it out and see if its useful to you, and give me any feedback that may help me improve it, I promise if would take less than 5 minutes of your time! :) To the core devs: I was wondering if pandas would be open to accept this library as a contribution to its read_* interface? The library uses OpenCV's morphological transformations to detect lines in PDFs when flavor='lattice', which I could vendorize or re-implement. It also has two system specific dependencies which are python-tk (used by matplotlib) and ghostscript (used to convert PDF to PNG). The first one shouldn't pose a problem since pandas also uses matplotlib, and for the second one, I could look for a Python library alternative to ghostscript. Looking forward to hearing from you all! Thanks for your time! Vinayak

2 4

pandas or new project
by David M Rashty Jan. 23, 2019

Jan. 23, 2019

Dear pandas team, I am a long time Stata user and I started using pandas about a year ago in order to build web applications using an in memory dataframe structure. As a business user, I've found Stata to have a key advantage over pandas that many others have also noted: much faster development time. Examples in Stata: drop myvar* // drops all columns starting with myvar keep myvar* // drops all columns except those starting with myvar reg z y x // runs the regression z = a+bx+cy + error In order to use pandas in a Stata-like fashion, I've had to monkey patch large parts of the library e.g., df = df.sdrop('myvar*') # same as above df = df.skeep('myvar*') # same as above df = df.sreg('z y x') # same as above df = df.squery('a>80 & b.str.contains("hello") & c.isin([1,2,3])') # df.query doesn't support str.contains and isin to my knowledge I put an "s" in front of my methods to mean either "stata" or "sugar". Additionally, I've built a system to: a) Automatically load new DataFrame methods into memory (no additional imports required) b) A caching system to make loading data blazing fast along with a much tighter syntax e.g., pd.read_stata('mydata.dta') (6 secs load time) vs use.mydata (0.001 secs load time after the first read from file) c) A system of column "labels" and formats to prettify various reports e.g., df.sscatter('rate score') produces a scatter plot with labels "Interest Rate, %" and "Credit Score", respectively. d) A reactive web app (using Flask/Redis) to quickly view the full DataFrame content in a browser: [cid:image001.jpg@01D44B98.6BE52AC0] Basically, I've tried to eliminate any obvious advantages Stata has over pandas. I'm potentially interested in developing this project into something bigger. Would you like me to share my work in the context of pandas or should it be a completely separate project with a different scope? Thanks, David Rashty | Flagstar Bank | Whole Loan Trading | 248-312-6692 | david.rashty(a)flagstar.com<mailto:david.rashty@flagstar.com> This e-mail may contain data that is confidential, proprietary or non-public personal information, as that term is defined in the Gramm-Leach-Bliley Act (collectively, Confidential Information). The Confidential Information is disclosed conditioned upon your agreement that you will treat it confidentially and in accordance with applicable law, ensure that such data isn't used or disclosed except for the limited purpose for which it's being provided and will notify and cooperate with us regarding any requested or unauthorized disclosure or use of any Confidential Information. By accepting and reviewing the Confidential information, you agree to indemnify us against any losses or expenses, including attorney's fees that we may incur as a result of any unauthorized use or disclosure of this data due to your acts or omissions. If a party other than the intended recipient receives this e-mail, he or she is requested to instantly notify us of the erroneous delivery and return to us all data so delivered.

4 5

pandas 2 still active?
by Nathan Heafner Nov. 29, 2018

Nov. 29, 2018

I just wanted to know if Pandas 2 is still being developed? The repo seems like its abandoned. ~Nathan

2 1

Sparse data structures in pandas: refactor - feedback welcome!
by Joris Van den Bossche Nov. 18, 2018

Nov. 18, 2018

Hi all, Distributing the message that Tom put on twitter ( https://twitter.com/TomAugspurger/status/1062718319445213184) also on the mailing lists: we are making changes to the support for sparse data in pandas, and would like to get feedback on this. To give some context: part of the internals of pandas are getting refactored based on the ExtensionArrays. This also applies to the sparse data structures: - The SparseArray is refactored to follow the ExtensionArray protocol, and this has some consequences (also impacting SparseSeries and Series holding sparse data): no longer subclassing numpy.ndarray, change in `np.asarray` behaviour, ... For more details see http://pandas-docs.github.io/pandas-docs-travis/whatsnew/v0.24.0.html#spars… . - Since a normal pandas Series and DataFrame can hold sparse data, there may be no need for the dedicated SparseSeries and SparseDataFrame subclasses. Therefore, we are planning to deprecate those subclasses, and the specific sparse functionality will be accessible on normal Series/DataFrame with the `sparse` accessor. However, this might have complications we didn't think about, so we need your feedback! See https://github.com/pandas-dev/pandas/issues/19239 and https://github.com/pandas-dev/pandas/issues/21978 for related github issues on this topic. Is you are a user of the sparse functionalities of pandas, trying out master / providing feedback is much appreciated. Best, Joris (I send it both to pydata and pandas-dev mailing lists, but please answer to pandas-dev(a)python.org)

3 4

Pandas development hangout - Thursday September 27 at 17:00 UTC
by Joris Van den Bossche Nov. 8, 2018

Nov. 8, 2018

Hi all, We're having a dev chat coming Thursday (September 27) at 9:00 Eastern / 14:00 UTC+1 / 15:00 CEST (Europe). All are welcome to attend. Hangout: https://hangouts.google.com/hangouts/_/calendar/am9yaXN2YW5kZW5ib3NzY2hlQGd… <https://hangouts.google.com/hangouts/_/calendar/am9yaXN2YW5kZW5ib3NzY2hlQGd…> Calendar invite: https://calendar.google.com/event?action=TEMPLATE&tmeid=NG12ZGhiNGp1a2liOGV… <https://calendar.google.com/event?action=TEMPLATE&tmeid=N2Z0ODh0NzdyMG1mMTF…> Agenda/Minutes: https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJl Bw5dOkVJLY-licoBmBU/edit?usp=sharing Joris

7 10