Mailman 3 December 2018 - Pandas-dev

Python library to extract tables from PDF files
by Vinayak Mehta April 30, 2019

April 30, 2019

Hello everyone! I'm a software engineer based out of New Delhi, India. I've been a long time user and have used it in countless projects and scripts! Thanks to the core developers and contributors for working on it! I recently released a Python library which lets users extract data tables out of PDF files, my first open-source library! Here's the link: https://github.com/socialcopsdev/camelot It has a similar API to the pandas read_* functions, bearing most similarity to read_html(). Like read_html(), it has a read_pdf() main interface which returns a list of pandas DataFrames for each table found in the PDF file, and contains two flavors for parsing different types of tables! I've created a comparison with other open-source PDF table extraction libraries and tools in the wiki here <https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Tab…> . I would be really grateful if you could check it out and see if its useful to you, and give me any feedback that may help me improve it, I promise if would take less than 5 minutes of your time! :) To the core devs: I was wondering if pandas would be open to accept this library as a contribution to its read_* interface? The library uses OpenCV's morphological transformations to detect lines in PDFs when flavor='lattice', which I could vendorize or re-implement. It also has two system specific dependencies which are python-tk (used by matplotlib) and ghostscript (used to convert PDF to PNG). The first one shouldn't pose a problem since pandas also uses matplotlib, and for the second one, I could look for a Python library alternative to ghostscript. Looking forward to hearing from you all! Thanks for your time! Vinayak

2 4

pandas or new project
by David M Rashty Jan. 23, 2019

Jan. 23, 2019

Dear pandas team, I am a long time Stata user and I started using pandas about a year ago in order to build web applications using an in memory dataframe structure. As a business user, I've found Stata to have a key advantage over pandas that many others have also noted: much faster development time. Examples in Stata: drop myvar* // drops all columns starting with myvar keep myvar* // drops all columns except those starting with myvar reg z y x // runs the regression z = a+bx+cy + error In order to use pandas in a Stata-like fashion, I've had to monkey patch large parts of the library e.g., df = df.sdrop('myvar*') # same as above df = df.skeep('myvar*') # same as above df = df.sreg('z y x') # same as above df = df.squery('a>80 & b.str.contains("hello") & c.isin([1,2,3])') # df.query doesn't support str.contains and isin to my knowledge I put an "s" in front of my methods to mean either "stata" or "sugar". Additionally, I've built a system to: a) Automatically load new DataFrame methods into memory (no additional imports required) b) A caching system to make loading data blazing fast along with a much tighter syntax e.g., pd.read_stata('mydata.dta') (6 secs load time) vs use.mydata (0.001 secs load time after the first read from file) c) A system of column "labels" and formats to prettify various reports e.g., df.sscatter('rate score') produces a scatter plot with labels "Interest Rate, %" and "Credit Score", respectively. d) A reactive web app (using Flask/Redis) to quickly view the full DataFrame content in a browser: [cid:image001.jpg@01D44B98.6BE52AC0] Basically, I've tried to eliminate any obvious advantages Stata has over pandas. I'm potentially interested in developing this project into something bigger. Would you like me to share my work in the context of pandas or should it be a completely separate project with a different scope? Thanks, David Rashty | Flagstar Bank | Whole Loan Trading | 248-312-6692 | david.rashty(a)flagstar.com<mailto:david.rashty@flagstar.com> This e-mail may contain data that is confidential, proprietary or non-public personal information, as that term is defined in the Gramm-Leach-Bliley Act (collectively, Confidential Information). The Confidential Information is disclosed conditioned upon your agreement that you will treat it confidentially and in accordance with applicable law, ensure that such data isn't used or disclosed except for the limited purpose for which it's being provided and will notify and cooperate with us regarding any requested or unauthorized disclosure or use of any Confidential Information. By accepting and reviewing the Confidential information, you agree to indemnify us against any losses or expenses, including attorney's fees that we may incur as a result of any unauthorized use or disclosure of this data due to your acts or omissions. If a party other than the intended recipient receives this e-mail, he or she is requested to instantly notify us of the erroneous delivery and return to us all data so delivered.

4 5

European pandas contributors meeting in London
by Marc Garcia Jan. 15, 2019

Jan. 15, 2019

Hi there, We're planning to have a meeting for the pandas contributors in Europe (obviously open to anyone who wants to fly to London). The day will probably be from the 25 to the 27 of January. On Friday afternoon we could have an event with talks and may be an interactive session with users to answer their questions and get their feedback. And Saturday and Sunday for sprints (also open to the public). Everything is yet to be confirmed, and ideas are more than welcome. Just wanted to send an early message, to see if someone else wants to participate, get involved or propose ideas (so far we discussed it with Joris and Pietro, I'm not aware of anyone else in Europe). Just get in touch. Cheers, Marc

3 5

Re: [Pandas-dev] Mailing list for Python data analytics ecosystem developers?
by Wes McKinney Jan. 4, 2019

Jan. 4, 2019

Copying the mailing list On Wed, Dec 26, 2018 at 10:58 AM Matthew Rocklin <mrocklin(a)gmail.com> wrote: > > I would be happy to participate in such a list and think that it is a good idea. > > I encourage people to reach out and ask on other PyData focused mailing lists asking for input. Numfocus, scipy-dev, numpy-dev come to mind. There are undoubtedly others though that are in this space, but not on my personal radar. > > I'll also bring up the possibility of using a technology other than e-mail. I've been interested in PyTorch's use of discouse recently. https://discuss.pytorch.org/ No strong thoughts on this though. I'm not sure that there is a good e-mail alternative today, but it would be good to improve the longevity of asynchronous group conversations. > Discourse is interesting. It seems to be used (at least in PyTorch's case) as more of a modern message board for users than a place for long-form discussions between project developers. IMHO having a cross-project developer mailing list is probably overdue -- I think we can do a better job the next couple of years coordinating (colluding?) with each other. A lot of coordination does of course in private, project-level, or other ad-hoc basis. It would help to be able to discuss ecosystem-level problems and possible solutions. > On Wed, Dec 26, 2018 at 8:47 AM Wes McKinney <wesmckinn(a)gmail.com> wrote: >> >> I sent a request to postmaster @ python.o to create >> pydata-dev(a)python.org. We can also use google groups if others prefer >> that >> >> On Tue, Dec 25, 2018 at 3:59 PM Joris Van den Bossche >> <jorisvandenbossche(a)gmail.com> wrote: >> > >> > Giving the growing ecosysten of data tools (in some way related to pandas, but not pandas itself), I am also +1 on such a list. I think that would be welcome, and not aware of anything existing. >> > >> > Joris >> > >> > Op di 25 dec. 2018 02:19 schreef Stephan Hoyer <shoyer(a)gmail.com: >> >> >> >> +1 for pydata-dev >> >> >> >> I don't think there's a list quite like this today. >> >> >> >> On Mon, Dec 24, 2018 at 6:11 PM Wes McKinney <wesmckinn(a)gmail.com> wrote: >> >>> >> >>> I'm talking about public archived communication channels >> >>> >> >>> On Mon, Dec 24, 2018, 7:57 PM William Ayd <william.ayd(a)icloud.com wrote: >> >>>> >> >>>> What do you think is missing from the existing PyData conferences? I’ve only been to the one in LA but it seemed to be somewhat in the direction of what you are asking for. >> >>>> >> >>>> > On Dec 24, 2018, at 3:02 PM, Wes McKinney <wesmckinn(a)gmail.com> wrote: >> >>>> > >> >>>> > hi folks, >> >>>> > >> >>>> > Prompted by some recent discussions I wondered what you all think >> >>>> > would be the best venue to have public discussions that involve other >> >>>> > open source projects that are generally 1 degree of separation away >> >>>> > from pandas. Sort of like "pydata-dev", or something. Is there >> >>>> > something like this already that I just missed? >> >>>> > >> >>>> > As context, I'm trying to travel less and go to fewer conferences the >> >>>> > next couple of years, and spend more time coding and writing, but I >> >>>> > still want to talk with people (asynchronously) about things, and >> >>>> > preferably in public. >> >>>> > >> >>>> > - Wes >> >>>> > _______________________________________________ >> >>>> > Pandas-dev mailing list >> >>>> > Pandas-dev(a)python.org >> >>>> > https://mail.python.org/mailman/listinfo/pandas-dev >> >>>> >> >>> _______________________________________________ >> >>> Pandas-dev mailing list >> >>> Pandas-dev(a)python.org >> >>> https://mail.python.org/mailman/listinfo/pandas-dev >> >> >> >> _______________________________________________ >> >> Pandas-dev mailing list >> >> Pandas-dev(a)python.org >> >> https://mail.python.org/mailman/listinfo/pandas-dev >> _______________________________________________ >> Pandas-dev mailing list >> Pandas-dev(a)python.org >> https://mail.python.org/mailman/listinfo/pandas-dev

7 19

Mailing list for Python data analytics ecosystem developers?
by Wes McKinney Dec. 26, 2018

Dec. 26, 2018

hi folks, Prompted by some recent discussions I wondered what you all think would be the best venue to have public discussions that involve other open source projects that are generally 1 degree of separation away from pandas. Sort of like "pydata-dev", or something. Is there something like this already that I just missed? As context, I'm trying to travel less and go to fewer conferences the next couple of years, and spend more time coding and writing, but I still want to talk with people (asynchronously) about things, and preferably in public. - Wes

4 5

Parallel group by operations in pandas
by Alexander Veysov Dec. 16, 2018

Dec. 16, 2018

Hi, I was reading Wes's *article <http://wesmckinney.com/blog/apache-arrow-pandas-internals/> *about pandas and my eye caught this: > 7. Better groupby(...).apply operations > ... due to other problems listed here, it is difficult or impossible to fully parallelize a df.groupby(...).apply(f) operation. > At some point, we will also want to improve the API for complex apply operations in pandas. I did not quite understand what was meant here. Some time ago I wrote some simple multiprocessing *wrappers <https://gist.github.com/snakers4/24cf04786224e22325f109ed15a6ac59>* for pandas, which provide almost linear increase in speed for relatively large dataframes (5-15 GB on disk in csv / feather, 100GB of RAM) when you use many CPU cores on a single machine. This is more or less like map-reduce, but on a single machine. Does this article mean something else, like the internal overhead for small operations? Because in my practice calculations in pandas are easily parallelized even w/o fancy tools like dask / vaex / pandas on ray. Many thanks for your reply! *Alexander Veysov * *m: *+7(916) 725 0871 *e:* aveysov(a)gmail.com *telegram*: *link* <https://t.me/snakers41> *blog: link <https://t.me/snakers4>* *website: **link <http://spark-in.me>*

2 1

DataFrame.to_html()
by Aschenbrener, Drew Dec. 16, 2018

Dec. 16, 2018

Hi Devs, I'm an entry level developer with University of Florida. I've spent several hours on stack-overflow and other forums, and it seems that something that pandas could easily handle is made extremely difficult. I may not have the full context of the decisions made, but maybe you could help me understand. 'DataFrame.to_html' does not provide any basic formatting options. Everywhere I've researched, the most simple solutions involve installing other packages in order to add various stylers. This seems excessive when you are trying to do something as simple as align a single column of data. I have one column of text and one column of numbers (defaults all to left align), I simply want to change the alignment of a single column, and the amount of work it takes to make that happen seems excessive. You provide table formatters, and the ability to justify labels, it seems a small step to provide the ability to justify specified columns. Is there something I am missing? I've been using python awhile, but I'm relatively new to pandas. - Drew Andrew (Drew) Aschenbrener IT Analyst | UF - Identity Services | (352) 273-1395 | Identity-Services(a)it.ufl.edu<mailto:Identity-Services@it.ufl.edu>

2 1