PLY in stdlib (was cffi in stdlib)
From: Eli Bendersky eliben@gmail.com
I'll be the first one to admit that pycparser is almost certainly not generally useful enough to be exposed in the stdlib. So just using it as an implementation detail is absolutely fine. PLY is a more interesting question, however, since PLY is somewhat more generally useful. That said, I see all this as implementation details that shouldn't distract us from the main point of whether cffi should be added.
Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues.
Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of.
Cheers, Dave
On 27 Feb 2013, at 11:00, David Beazley dave@dabeaz.com wrote:
From: Eli Bendersky eliben@gmail.com
I'll be the first one to admit that pycparser is almost certainly not generally useful enough to be exposed in the stdlib. So just using it as an implementation detail is absolutely fine. PLY is a more interesting question, however, since PLY is somewhat more generally useful. That said, I see all this as implementation details that shouldn't distract us from the main point of whether cffi should be added.
Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion.
+1 PLY is capable and well tried-and-tested. We used it in Resolver One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Michael
LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues.
Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of.
Cheers, Dave
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/
May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Feb 27, 2013 4:31 AM, "Michael Foord" fuzzyman@voidspace.org.uk wrote:
On 27 Feb 2013, at 11:00, David Beazley dave@dabeaz.com wrote:
From: Eli Bendersky eliben@gmail.com
I'll be the first one to admit that pycparser is almost certainly not generally useful enough to be exposed in the stdlib. So just using it
as an
implementation detail is absolutely fine. PLY is a more interesting question, however, since PLY is somewhat more generally useful. That
said,
I see all this as implementation details that shouldn't distract us
from
the main point of whether cffi should be added.
Regarding the inclusion of PLY or some subcomponent of it in the
standard library, it's not an entirely crazy idea in my opinion.
+1 PLY is capable and well tried-and-tested. We used it in Resolver One
to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
-eric
On 7/13/2013 12:10 AM, Eric Snow wrote:
On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman@voidspace.org.uk
+1 PLY is capable and well tried-and-tested. We used it in Resolver
One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
Who is the maintainer and what is his opinion?
David Beasley; see earlier in this same thread:
http://mail.python.org/pipermail/python-dev/2013-February/thread.html#124389
On Fri, Jul 12, 2013 at 9:41 PM, Terry Reedy tjreedy@udel.edu wrote:
On 7/13/2013 12:10 AM, Eric Snow wrote:
On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman@voidspace.org.uk
+1 PLY is capable and well tried-and-tested. We used it in Resolver One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
Who is the maintainer and what is his opinion?
-- Terry Jan Reedy
______________________________**_________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** jdunck%40gmail.comhttp://mail.python.org/mailman/options/python-dev/jdunck%40gmail.com
On 13 Jul 2013, at 07:41, Terry Reedy tjreedy@udel.edu wrote:
On 7/13/2013 12:10 AM, Eric Snow wrote:
On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman@voidspace.org.uk
+1 PLY is capable and well tried-and-tested. We used it in Resolver
One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
Who is the maintainer and what is his opinion?
The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly.
Michael
-- Terry Jan Reedy
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.voidspace.org.uk/
May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord fuzzyman@voidspace.org.ukwrote:
On 13 Jul 2013, at 07:41, Terry Reedy tjreedy@udel.edu wrote:
On 7/13/2013 12:10 AM, Eric Snow wrote:
On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman@voidspace.org.uk
+1 PLY is capable and well tried-and-tested. We used it in Resolver
One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
Who is the maintainer and what is his opinion?
The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly.
Just because we have now had two conflicting replies on this: David is down with PLY being added, but Alex Gaynor was working on a cleanup called RPLY for RPython. Basically David said the two of them should work together to clean up PLY and then it should be good to proposing for the stdlib (e.g. there are some backwards-compatibility hacks which should be removed).
Below is David's original email on the topic from Feb 27:
-------------------------------
Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues.
Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of.
Cheers, Dave
I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).
So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.
Cheers, Dave
On Jul 13, 2013, at 8:12 AM, Brett Cannon wrote:
On Sat, Jul 13, 2013 at 4:24 AM, Michael Foord fuzzyman@voidspace.org.uk wrote:
On 13 Jul 2013, at 07:41, Terry Reedy tjreedy@udel.edu wrote:
On 7/13/2013 12:10 AM, Eric Snow wrote:
On Feb 27, 2013 4:31 AM, "Michael Foord" <fuzzyman@voidspace.org.uk
+1 PLY is capable and well tried-and-tested. We used it in Resolver
One to implement a pretty large grammar and it is (in my opinion) best of breed in the Python parser generator world. Being stable and widely used, with an "available maintainer", makes it an ideal candidate for standard library inclusion.
Is this still on the table?
Who is the maintainer and what is his opinion?
The maintainer is David Beazley and as far as I recall he has not expressed an opinion on this particular question. It would obviously need his agreement (and maintenance commitment) if it is to fly.
Just because we have now had two conflicting replies on this: David is down with PLY being added, but Alex Gaynor was working on a cleanup called RPLY for RPython. Basically David said the two of them should work together to clean up PLY and then it should be good to proposing for the stdlib (e.g. there are some backwards-compatibility hacks which should be removed).
Below is David's original email on the topic from Feb 27:
Regarding the inclusion of PLY or some subcomponent of it in the standard library, it's not an entirely crazy idea in my opinion. LALR(1) parsers have been around for a long time, are generally known to anyone who's used yacc/bison, and would be useful outside the context of cffi or pycparser. PLY has also been around for about 12 years and is what I would call stable. It gets an update about every year or two, but that's about it. PLY is also relatively small--just two files and about 4300 lines of code (much of which could probably be scaled down a bit).
The only downside to including PLY might be the fact that there are very few people walking around who've actually had to *implement* an LALR(1) parser generator. Some of the code for that is extremely hairy and mathematical. At this time, I don't think there are any bugs in it, but it's not the sort of thing that one wants to wander into casually. Also, there are some horrible hacks in PLY that I'd really like to get rid of, but am currently stuck with due to backwards compatibility issues.
Alex Gaynor has been working on a PLY variant (RPLY) geared at RPython and which has a slightly different programming interface. I'd say if we were to go down this route, he and I should work together to put together some kind of more general "parsing.lalr" package (or similar) that cleans it up and makes it more suitable as a library for building different kinds of parsing tools on top of.
Cheers, Dave
On 13 July 2013 23:26, David Beazley dave@dabeaz.com wrote:
I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).
So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.
lrparsing is a more recent entry in the LR parsing stakes: https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)
(I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist)
Cheers, Nick.
On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan ncoghlan@gmail.com wrote:
On 13 July 2013 23:26, David Beazley dave@dabeaz.com wrote:
I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).
So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.
lrparsing is a more recent entry in the LR parsing stakes: https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)
(I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist)
It seems a bit new compared to PLY's 15 years of existence to be considered in the running. Plus Russell would need to change the license from GPL.
On Jul 14, 2013, at 8:13 AM, Brett Cannon wrote:
On Sun, Jul 14, 2013 at 2:54 AM, Nick Coghlan ncoghlan@gmail.com wrote: On 13 July 2013 23:26, David Beazley dave@dabeaz.com wrote: I'm in favor of PLY going into stdlib with the caveat that there are some things about it that should probably be cleaned up and modernized. For instance, the method by which it writes the cached parsing tables needs to be cleaned up. I still think putting the LALR(1) generator code into a common library usable by both PLY/RPLY would be a useful thing to do. That code is really hairy and non-trivial to understand without something like the Dragon book nearby (and even then it's not easy).
So, if I were to make any kind of proposal, I would say, make a standard library module for just the LALR(1) generator code. If the PLY interface is needed to add pycparser or cffi to the standard library, that can be added too, but as a separate module that uses the more general LALR(1) library.
lrparsing is a more recent entry in the LR parsing stakes:https://pypi.python.org/pypi/lrparsing (although, as Russell put it in his PyCon AU lightning talk, if PLY had shown up ranked higher than 506 in his PyPI search for "parser", he probably would have just used that: http://pyvideo.org/video/2222/sunday-lightning-talks at about 2:15)
(I plan to bug Russell about putting that up on one of the DVCS hosting sites next time I see him at BrisPy - for the moment, the source is available through the tarball/sdist)
It seems a bit new compared to PLY's 15 years of existence to be considered in the running. Plus Russell would need to change the license from GPL.
I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. My impression has always been that the main interest in PLY was due to interest in seeing CFFI in the standard library. I'd say my main desire on the PLY side is that if it does go into the standard library, perhaps I could make it slightly less of mysterious black box and clean up a few bits.
Cheers, Dave
On Jul 14, 2013, at 6:32 AM, David Beazley dave@dabeaz.com wrote:
I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library.
I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others.
Raymond
On 15 July 2013 16:01, Raymond Hettinger raymond.hettinger@gmail.com wrote:
On Jul 14, 2013, at 6:32 AM, David Beazley dave@dabeaz.com wrote:
I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library.
I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others.
Agreed (and the fact it's a prerequisite for pycparsing and cffi just seals the deal).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 July 2013 07:01, Raymond Hettinger raymond.hettinger@gmail.comwrote:
I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others.
+1. Parser generators are useful tools - parsers are right on the boundary of "easy enough to understand why you'd like one, but hard enough to put you off implementing our own". And there is value in tools like this being in the stdlib as opposed to a 3rd party dependency.
Paul
participants (9)
-
Brett Cannon -
David Beazley -
Eric Snow -
Jeremy Dunck -
Michael Foord -
Nick Coghlan -
Paul Moore -
Raymond Hettinger -
Terry Reedy