Mailman 3 September 2018 - Pandas-dev

Python library to extract tables from PDF files
by Vinayak Mehta April 30, 2019

April 30, 2019

Hello everyone! I'm a software engineer based out of New Delhi, India. I've been a long time user and have used it in countless projects and scripts! Thanks to the core developers and contributors for working on it! I recently released a Python library which lets users extract data tables out of PDF files, my first open-source library! Here's the link: https://github.com/socialcopsdev/camelot It has a similar API to the pandas read_* functions, bearing most similarity to read_html(). Like read_html(), it has a read_pdf() main interface which returns a list of pandas DataFrames for each table found in the PDF file, and contains two flavors for parsing different types of tables! I've created a comparison with other open-source PDF table extraction libraries and tools in the wiki here <https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Tab…> . I would be really grateful if you could check it out and see if its useful to you, and give me any feedback that may help me improve it, I promise if would take less than 5 minutes of your time! :) To the core devs: I was wondering if pandas would be open to accept this library as a contribution to its read_* interface? The library uses OpenCV's morphological transformations to detect lines in PDFs when flavor='lattice', which I could vendorize or re-implement. It also has two system specific dependencies which are python-tk (used by matplotlib) and ghostscript (used to convert PDF to PNG). The first one shouldn't pose a problem since pandas also uses matplotlib, and for the second one, I could look for a Python library alternative to ghostscript. Looking forward to hearing from you all! Thanks for your time! Vinayak

2 4

pandas or new project
by David M Rashty Jan. 23, 2019

Jan. 23, 2019

Dear pandas team, I am a long time Stata user and I started using pandas about a year ago in order to build web applications using an in memory dataframe structure. As a business user, I've found Stata to have a key advantage over pandas that many others have also noted: much faster development time. Examples in Stata: drop myvar* // drops all columns starting with myvar keep myvar* // drops all columns except those starting with myvar reg z y x // runs the regression z = a+bx+cy + error In order to use pandas in a Stata-like fashion, I've had to monkey patch large parts of the library e.g., df = df.sdrop('myvar*') # same as above df = df.skeep('myvar*') # same as above df = df.sreg('z y x') # same as above df = df.squery('a>80 & b.str.contains("hello") & c.isin([1,2,3])') # df.query doesn't support str.contains and isin to my knowledge I put an "s" in front of my methods to mean either "stata" or "sugar". Additionally, I've built a system to: a) Automatically load new DataFrame methods into memory (no additional imports required) b) A caching system to make loading data blazing fast along with a much tighter syntax e.g., pd.read_stata('mydata.dta') (6 secs load time) vs use.mydata (0.001 secs load time after the first read from file) c) A system of column "labels" and formats to prettify various reports e.g., df.sscatter('rate score') produces a scatter plot with labels "Interest Rate, %" and "Credit Score", respectively. d) A reactive web app (using Flask/Redis) to quickly view the full DataFrame content in a browser: [cid:image001.jpg@01D44B98.6BE52AC0] Basically, I've tried to eliminate any obvious advantages Stata has over pandas. I'm potentially interested in developing this project into something bigger. Would you like me to share my work in the context of pandas or should it be a completely separate project with a different scope? Thanks, David Rashty | Flagstar Bank | Whole Loan Trading | 248-312-6692 | david.rashty(a)flagstar.com<mailto:david.rashty@flagstar.com> This e-mail may contain data that is confidential, proprietary or non-public personal information, as that term is defined in the Gramm-Leach-Bliley Act (collectively, Confidential Information). The Confidential Information is disclosed conditioned upon your agreement that you will treat it confidentially and in accordance with applicable law, ensure that such data isn't used or disclosed except for the limited purpose for which it's being provided and will notify and cooperate with us regarding any requested or unauthorized disclosure or use of any Confidential Information. By accepting and reviewing the Confidential information, you agree to indemnify us against any losses or expenses, including attorney's fees that we may incur as a result of any unauthorized use or disclosure of this data due to your acts or omissions. If a party other than the intended recipient receives this e-mail, he or she is requested to instantly notify us of the erroneous delivery and return to us all data so delivered.

4 5

Pandas development hangout - Thursday September 27 at 17:00 UTC
by Joris Van den Bossche Nov. 8, 2018

Nov. 8, 2018

Hi all, We're having a dev chat coming Thursday (September 27) at 9:00 Eastern / 14:00 UTC+1 / 15:00 CEST (Europe). All are welcome to attend. Hangout: https://hangouts.google.com/hangouts/_/calendar/am9yaXN2YW5kZW5ib3NzY2hlQGd… <https://hangouts.google.com/hangouts/_/calendar/am9yaXN2YW5kZW5ib3NzY2hlQGd…> Calendar invite: https://calendar.google.com/event?action=TEMPLATE&tmeid=NG12ZGhiNGp1a2liOGV… <https://calendar.google.com/event?action=TEMPLATE&tmeid=N2Z0ODh0NzdyMG1mMTF…> Agenda/Minutes: https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJl Bw5dOkVJLY-licoBmBU/edit?usp=sharing Joris

7 10

Pandas loc function including upper limit
by Hritik Vijay Oct. 1, 2018

Oct. 1, 2018

The `loc` function includes the upper limit which is very counter intuitive. Shouldn't it follow iloc and other indexing methods and exclude upper limit (at least for integral slices) -- Regards Hritik Vijay

3 2

Indexing API [was: Pandas Sprint Recap]
by Joris Van den Bossche Sept. 27, 2018

Sept. 27, 2018

To come back to this old thread. I think when Tom said "We didn't discuss indexing much, beyond agreeing that there's work to be done, and that fixing it was too large a task for 1.0.", he mainly meant things like "fixing __getitem__" ( https://github.com/pandas-dev/pandas/issues/9595), or at least that is how I understood our discussion. And doing that is not only out of scope for 1.0 because it is too much work, but IMO also because it would be a too big API disruption. But I am opening this new thread, as the long discussion in the old thread was triggered by the indexing API questions that Pietro brought up, but then focused more on general pandas 1.0 vs 2.0 vs new code base questions. While I think the initial question about indexing still deserves attention. [Pietro] > There are many bugs - in particular, in indexing code - which might > potentially break existing code when fixed. Some of them will have non- > trivial deprecation paths/detection strategies. The first ones that > come to my mind are #18631 > <https://github.com/pandas-dev/pandas/issues/18631>, #12827 > <https://github.com/pandas-dev/pandas/issues/12827>, #9519 > <https://github.com/pandas-dev/pandas/issues/9519>. The last one, in > particular, > implies changing the result of potentially tons of calls to .loc on a > non-unique index. > That's already a nice list of issues (only the last one might be more controversion I think). Do you think those could be solved without a complete refactor of the indexing code? (because I think we can still try to fix some API warts for 1.0, a complete refactor might be less realistic) If there are other issues, it would be good to try to compile a list of indexing-related issues that we would still like to see fixed for 1.0. Joris 2018-07-13 19:45 GMT+02:00 Tom Augspurger <tom.augspurger88(a)gmail.com>: > Thanks Pietro, > > We didn't discuss indexing much, beyond agreeing that there's work to be > done, and that fixing it was too large > a task for 1.0. > > As for whether an individual issue is a bug or feature, we'll have to > continue using our judgement. I think we'll > inevitably break users' code in a 1.x release as we fix bugs. > > We'll need to discuss workflows for these large changes (e.g. ripping out > the block manager) that will be API > breaking, but may take some time to land. Keeping a separate branch in > sync is a pain, but may be the least > painful alternative. > > One thing I want to reiterate: it's not going to take another 11 years to > reach pandas 2.0 :) Just because we don't > solve indexing for 1.0 doesn't mean we won't ever be able to fix it. > > Tom > > On Fri, Jul 13, 2018 at 12:12 PM, Pietro Battiston <me(a)pietrobattiston.it> > wrote: > >> Hi Tom, >> >> first, thanks to all those who participated in the sprint, and for the >> recap. >> >> Il giorno dom, 08/07/2018 alle 16.26 -0500, Tom Augspurger ha scritto: >> > [...] >> > I've posted a document on our wiki with a summary of the topics >> > discussed. https://github.com/pandas-dev/pandas/wiki/Pandas-Sprint-(J >> > uly,-2018) >> > >> > If people have questions or comments, feel free to post here and >> > we'll clarify that document. >> >> Something that scares me - but maybe because I'm missing something >> obvious - is what exactly qualifies as "deprecation". Is it something >> which was once presented as a distinct feature and is then disabled, or >> any general change to what any API call performs (that is, anything >> requiring a deprecation cycle - that is)? >> >> There are many bugs - in particular, in indexing code - which might >> potentially break existing code when fixed. Some of them will have non- >> trivial deprecation paths/detection strategies. The first ones that >> come to my mind are #18631, #12827, #9519. The last one, in particular, >> implies changing the result of potentially tons of calls to .loc on a >> non-unique index. >> >> My view is that those (and many more, including several that will be >> found) will be best fixed through a total rewrite of indexing code >> (i.e., all code in indexing.py, and some code in internals.py), which I >> assumed would happen before 1.0, and which I certainly won't be able to >> do before 0.24.0 (September 2018). >> I'm clearly not claiming that nobody else can do it (nor that the bugs >> can necessarily only be fixed through a complete rewrite)... but since >> I did not get any feedback on >> https://github.com/pandas-dev/pandas/wiki/(Tentative)-rules-for-restruc >> turing-indexing-code >> <https://github.com/pandas-dev/pandas/wiki/(Tentative)-rules-for-restructuri…> >> ... I assume that nobody is focusing/planning to focus on this in the >> near future (or was it somehow discussed in the sprint?). >> >> I perfectly understand the desire to stop postponing 1.0 to a vague >> future, if it's just a matter of recognizing that pandas is worth >> using. >> But if it's a statement/commitment about code robustness/quality, and >> relatedly API stability... then I think we it is risky to leave the >> indexing API, and more in general the core codebase (as opposed to >> important but more lateral features such as new dtypes) out of the >> picture (e.g. out of #21894). >> >> Cheers, >> >> Pietro >> _______________________________________________ >> Pandas-dev mailing list >> Pandas-dev(a)python.org >> https://mail.python.org/mailman/listinfo/pandas-dev >> > > > _______________________________________________ > Pandas-dev mailing list > Pandas-dev(a)python.org > https://mail.python.org/mailman/listinfo/pandas-dev > >

1 0

Future Deprecation Policy
by Tom Augspurger Sept. 26, 2018

Sept. 26, 2018

Hi all, At the sprint, we touched on this, but I don't recall there being a whole lot of discussion. I wanted to confirm that we're on the same page. Briefly, I see two options: 1. SemVer Deprecations are introduced as needed. Enforcing a deprecation is a backwards-incompatible change, and so is restricted to major releases only. 2. Rolling deprecations Deprecations are introduced as needed. Deprecations are enforced N releases after they were introduce (N typically being 2-3 "major" release in practice). Do people have a preference between these two schemes? Does the fact that NumPy uses a rolling deprecation policy swing us one way or the other? I've added this to the agenda for the meeting tomorrow, but wanted to give people a chance to collect their thoughts first. Tom

3 2

Datetime (with timezone?) as extension array?
by Pietro Battiston Sept. 26, 2018

Sept. 26, 2018

Hi all, I assumed that Datetime (with timezone, or maybe in general?) was also planned to follow the extension array interface, which is related to issue https://github.com/pandas-dev/pandas/issues/19041 , to the annoying fact that datetimeindexwithtz._values returns the index itself, and also to the fact that https://pandas.pydata.org/pandas-docs/stable/extending.html currently states "Pandas itself uses the extension system for some types that aren’t built into NumPy (categorical, period, interval, datetime with timezone).", which is false. ... but I didn't find an issue for this? Did I miss it? Should I create it? Or was there a decision to leave datetimeindextz as it is, maybe for better compatibility with numpy? Pietro

4 8