Mailman 3 June 2015 - Python-ideas

PEP 432 modular bootstrap status update
by Nick Coghlan June 14, 2015

June 14, 2015

I took some time today to resync my PEP 432 modular bootstrap branch with the current default branch of CPython. While it's still very much a work in progress (with outright test failures for at least isolated mode and subinterpreter support at the moment), https://bitbucket.org/ncoghlan/cpython_sandbox/commits/5e0d1bba5d6dab7b2121… represents a significant milestone, as for the first time I was able to replace the current call in Py_Main that initialises the hash randomisation with the new … [View More]

1 0

Re: [Python-ideas] If branch merging
by Cory Beutler June 13, 2015

June 13, 2015

Thank you all for your responses. I didn't realize how much support this mailing list had. In response to several responses: It appears I have hit a soft spot with the 'as' keyword. It seems clear to me that inlining an assignment confuses scope. With any inline solution, that confusion will exist. Now, I will say that I do not like 'if aisb = a == b' because of the potential errors, as others have mentioned. A language should be written as much for the beginners as the experts, or it will … [View More]never live very long. Avoiding absentminded mistakes is always good to do. There are many other possible solutions from a comma, as in "if a == b, aisb:", to a custom language addition of a new keyword or operator. Irregardless of how inline assignment is written, the scope issue will still exist. As such, it is more important to decide if it is needed first. The fact that this idea has been brought up before means that it deserves some research. Perhaps I can do some analytics and return with more info on where it could be used and if it will actually provide any speed benefits. Ok, that was a bit of a shotgun response to many remarks. Hopefully it will suffice. Thanks again for all the feedback. I would now like to respond to Steven's response directly: On Sat, Jun 6, 2015 at 11:19 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On Sat, Jun 06, 2015 at 09:03:38PM -0600, Cory Beutler wrote: > > [...] > > This would simplify some logic expressions by allowing the merging of > > branched code. > > > > *Examples of use:* > > *Duplicate code in if-chains may be reduced:* > > # Old Version > > if a == b: > > print ('a == b') > > foo() # <-- duplicate code > > elif b == c: > > print ('b == c') > > foo() # <-- duplicate code > > elif c == d: > > print ('c == d') > > foo() # <-- duplicate code > > if a == b: > print('a == b') > elif b == c: > print('b == c') > elif c == d: > print('c == d') > foo() > > No new syntax required. > > The functionally is not the same. In your example 'foo' gets called even if none of the conditions are true. The above example only runs 'foo' if it enters one of the if-blocks. This basic layout is useful for various parsing and reading operations. It is nice to check if something fits various conditions, then after the specific handling, add on finishing details in a 'foo'-like context. > > > *Many nested 'if' statements could now be a more linear style:* > > # Old Version > > if a == b: > > print ('a == b') > > if b == c: > > print ('b == c') > > print ('end if') > > What's wrong with that code? Nesting the code like that follows the > logic of the code: the b==c test *only* occurs if a==b. > > > > # New Version > > if a == b: > > print ('a == b') > > alif b == c: > > print ('b == c') > > also: > > print ('end if') > > I consider this significantly worse. It isn't clear that the comparison > between b and c is only made if a == b, otherwise it is entirely > skipped. > > It may only be worse because you are not used to reading it. This type of syntax looks simple once you know how the pieces work. I mean, you know that having multiple if-elif statements will result in only checking conditions until one passes. The 'also' mentality would be the same, but backwards. > *Selective Branch merging:* > > One limitation of the 'also' and 'alif' keywords is the restriction to > the > > "all of the above" checking. What I mean by that is that there is no way > to > > pick and choose which branches to merge back together. When using 'also' > > and 'alif' you are catching all previous if-branches. One easy way to > solve > > this would be to allow for named branching. The most simple way to do > this > > is to save the conditions of each branch into a variable with a name. > Here > > is an example of merging only select branches together: > > # Old Version > > if a == b: > > print ('a == b') > > elif a == c: > > print ('a == c') > > elif a == d: > > print ('a == d') > > if (a == b) or (a == d): > > print ('a == b and a == d') > > That code is wrong. Was that an intentional error? The final branch > prints that a == b == d, but that's not correct, it runs when either > a == b or a == d, not just when both are true. > Yeah, that was a mistype. That is why I shouldn't program fake code late at night. It does warm my heart to see that 2 people have corrected my fake code. That means it is easy to learn and understand. > > Personally, I would write that as: > > if a == b or a == d: > if a == b: > print('a == b') > else: > print('a == d') > print('a == b or a == d') > elif a == c: > print('a == c') > > You do end up comparing a and b for equality twice, but worrying about > that is likely to be premature optimization. It isn't worth adding > syntax to the language just for the one time in a million that actually > matters. > > With that rearrangement, you could write it with 'also': if a == b: print('a == b') elif a == d: print('a == d') also: print('a == b or a == d') elif a == c: print('a == c') but that does not demonstrate the selective branch merging. It would not work so well to combine the next two 'elif' statements, as any other 'also' blocks would capture the 'a == b', 'a == d', and first 'also' branches. I guess what I mean to say is that this example is a little too dumbed down. I think you know what I am after, though. If 'a == b' is a heavy duty calculation, it would be nice to be able to store that inline. Thank you, Steven, for your objective view of things. It is has been useful to see an outside perspective. I look forward to your future input. [View Less]

13 51

slice.literal notation
by Joseph Jevnik June 12, 2015

June 12, 2015

I was told in the thread that it might be a good idea to bring this up on python discussions. Here is a link to the proposed patch and some existing comments: http://bugs.python.org/issue24379 I often find that when working with pandas and numpy I want to store slice objects in variables to pass around and re-use; however, the syntax for constructing a slice literal outside of an indexer is very different from the syntax used inside of a subscript. This patch proposes the following change: … [View More] slice.literal This would be a singleton instance of a class that looks like: class sliceliteral(object): def __getitem__(self, key): return key The basic idea is to provide an alternative constructor to 'slice' that uses the subscript syntax. This allows people to write more understandable code. Consider the following examples: reverse = slice(None, None, -1) reverse = slice.literal[::-1] all_rows_first_col = slice(None), slice(0) all_rows_first_col = slice.literal[:, 0] first_row_all_cols_but_last = slice(0), slice(None, -1) first_row_all_cols_but_last = slice.literal[0, :-1] Again, this is not intended to make the code shorter, instead, it is designed to make it more clear what the slice object your are constructing looks like. Another feature of the new `literal` object is that it is not limited to just the creation of `slice` instances; instead, it is designed to mix slices and other types together. For example: >>> slice.literal[0] 0 >>> slice.literal[0, 1] (0, 1) >>> slice.literal[0, 1:] (0, slice(1, None, None) >>> slice.literal[:, ..., ::-1] (slice(None, None, None), Ellipsis, slice(None, None, -1) These examples show that sometimes the subscript notation is much more clear that the non-subscript notation. I believe that while this is trivial, it is very convinient to have on the slice type itself so that it is quickly available. This also prevents everyone from rolling their own version that is accesible in different ways (think Py_RETURN_NONE). Another reason that chose this aproach is that it requires no change to the syntax to support. There is a second change proposed here and that is to 'slice.__repr__'. This change makes the repr of a slice object match the new literal syntax to make it easier to read. >>> slice.literal[:] slice.literal[:] >>> slice.literal[1:] slice.literal[1:] >>> slice.literal[1:-1] slice.literal[1:-1] >>> slice.literal[:-1] slice.literal[:-1] >>> slice.literal[::-1] slice.literal[::-1] This change actually affects old behaviour so I am going to upload it as a seperate patch. I understand that the change to repr much be less desirable than the addition of 'slice.literal' [View Less]

10 23

Using semantic web (RDF triples) to link replacements for modules in standard library
by anatoly techtonik June 12, 2015

June 12, 2015

I failed to bring any attention to https://bitbucket.org/techtonik/python-stdlib to crowdsource collecting data about modules in stdlib, and was even warned not to distract people in @python.org lists to work on it, so I thought about a different way to decentralize the work on standard library. We have the data about top Python modules that need a redesign http://www.google.com/moderator/#15/e=28d4&t=28d4.40 but no way to propose alternatives for each module. So, I propose to use … [View More]

1 0

Making -m work for scripts
by Ram Rachum June 12, 2015

June 12, 2015

Hi, What do you think about making `python -m whatever` work also for installed scripts and not just for modules? I need this now because I've installed pypy on Linux, and I'm not sure how to run the `nosetests` of PyPy (in contrast to the `nosetests` of the system Python.) It's sometimes a mess to find where Linux installed the scripts related with each version of Python. But if I could do `pypy -m nosetests` then I'd have a magic solution. What do you think? Thanks, Ram.

5 7

Re: [Python-ideas] Python-ideas Digest, Vol 103, Issue 49
by Cara June 9, 2015

June 9, 2015

This isn't directed at Andrew in particular but the discussion in general, because it wasn't clear to me from how everyone was using the words that the distinction Andrew mentions was clear. PEGs are a class of grammars analogous to context-free grammars, regular grammars, or (to choose a more obscure example) Boolean grammars ( http://users.utu.fi/aleokh/boolean/ ). PEGs are probably not comparable to CFGs, while Boolean grammars are a strict superset of CFGs. Like other grammars, PEGs … [View More]admit multiple parsing algorithms. As Andrew says and as far as I know, OMeta uses a top-down recursive-descent algorithm with backtracking for parsing PEGs, which is why it can go exponential on some inputs. Packrat parsing is the algorithm that Bryan Ford published at the same time as he introduced PEGs, and it can parse PEGs in linear time by using memoization instead of backtracking. However, it's not the only algorithm that can parse PEGs, and while I don't know any work on this, it seems plausible that algorithms for parsing general CFGs like GLL or Earley could be adapted to parsing PEGs. Likewise, memoization can be used to avoid backtracking in a top-down recursive-descent parser for CFGs, though it's highly unlikely that any algorithm could achieve linear time for ambiguous CFGs. > > I don't know about OMeta, but the Earley parsing algorithm is > worst-cast cubic time "quadratic time for unambiguous grammars, and > linear time for almost all LR(k) grammars". > > I don't know why you'd want to use Earley for parsing a programming > language. IIRC, it was the first algorithm that could handle rampant > ambiguity in polynomial time, but that isn't relevant to parsing > programming languages (especially one like Python, which was > explicitly designed to be simple to parse), and it isn't relevant to > natural languages if you're not still in the 1960s, except in learning > the theory and history of parsing. GLR does much better in > almost-unambiguous/almost-deterministic languages; CYK can be easily > extended with weights (which propagate sensibly, so you can use them > for a final judgment, or to heuristically prune alternatives as you > go); Valiant is easier to reason about mathematically; etc. And that's > just among the parsers in the same basic family as Earley. Do you have a source for the assertion that Earley is slower than GLR? I've heard many people say this, but I've never seen any formal comparisons made since Masaru Tomita's 1985 book, "Efficient parsing for natural language: A fast algorithm for practical systems." As far as I know, this is emphatically not true for asymptotic complexity. In 1991 Joop Leo published “A General Context-free Parsing Algorithm Running in Linear Time on Every LR(K) Grammar Without Using Lookahead," a modification to Earley's algorithm that makes it run in linear time on LR-regular grammars. The LR-regular grammars include the LR(k) grammars for all k and are in fact a strict superset of the deterministic grammars. Again, as far as I know, GLR parsers run in linear times on grammars depending on the LR table they're using, which means in most cases LR(1) or something similar. There are many variations of the GLR algorithm now, though, so it's possible there's one I don't know about that doesn't have this limitation. As for constant factors, it's possible that GLR is better than Earley. I'm reluctant to assume that Tomita's findings still hold, though, because hardware has changed radically since 1985 and because both the original versions of both GLR and Earley had bugs. While there are now versions of both algorithms that fix the bugs, the fixes may have changed the constant factors. > In fact, even if I wanted to write an amazing parser library for > Python (and I kind of do, but I don't know if I have the time), I > still don't think I'd want to suggest it as a replacement for the > parser in CPython. Writing all the backward-compat adapters and > porting the Python parser over with all its quirks intact and building > the tests to prove that it's performance and error handling were > strictly better and so on wouldn't be nearly as much fun as other > things I could do with it. I'm working on a new parser library for Python at https://github.com/ceridwen/combinators that I intend to have some of the features that have been discussed here (it can be used for scanner-less/lexer-less parsing, if one wants; a general CFG algorithm rather than some subset thereof; linear-time parsing on a reasonable subset of CFGs; integrated semantic actions; embedding in Python rather than having a separate DSL like EBNF) and some others that haven't been. The research alone has been a lot of work, and the implementation no less so. I'd barely call what I have at the moment pre-alpha. I'm exploring an implementation based on a newer general CFG parsing algorithm, GLL ( https://www.royalholloway.ac.uk/computerscience/research/csle/gllparsers.as… ), though I'd like to compare it to Earley on constant factors. Cara [View Less]

1 0

Hooking between lexer and parser
by Andrew Barnert June 8, 2015

June 8, 2015

Compiling a module has four steps: * bytes->str (based on encoding declaration or default) * str->token stream * token stream->AST * AST->bytecode You can very easily hook at every point in that process except the token stream. There _is_ a workaround: re-encode the text to bytes, wrap it in a BytesIO, call tokenize, munge the token stream, call untokenize, re-decode back to text, then pass that to compile or ast.parse. But, besides being a bit verbose and painful, that means … [View More]

12 53

difflib.SequenceMatcher quick_ratio
by floyd June 8, 2015

June 8, 2015

Hi * I use this python line quite a lot in some projects: if difflib.SequenceMatcher.quick_ratio(None, a, b) >= threshold: I realized that this is performance-wise not optimal, therefore wrote a method that will return much faster in a lot of cases by using the length of "a" and "b" to calculate the upper bound for "threshold": if difflib.SequenceMatcher.quick_ratio_ge(None, a, b, threshold): I'd say we could include it into the stdlib, but maybe it should only be a python code recipe? … [View More]

5 4

If branch merging
by Cory Beutler June 8, 2015

June 8, 2015

I recently(1 year ago) realized that 'if', 'elif', and 'else' provide easy branching for code, but there is no easy way to merge branches of code back together. One fix for this would be introduction of two new keywords: 'also' and 'alif' (also if) Here are the definitions of if-chain keywords with the addition of 'also' and 'alif': *if *- execute code if this condition is met *else *- execute code if no previous condition is met *elif *- execute code if no previous condition is met and this … [View More]condition is met *also *- execute code if any previous condition is met *alif *- execute code if any previous condition is met and this condition is met This would simplify some logic expressions by allowing the merging of branched code. *Examples of use:* *Duplicate code in if-chains may be reduced:* # Old Version if a == b: print ('a == b') foo() # <-- duplicate code elif b == c: print ('b == c') foo() # <-- duplicate code elif c == d: print ('c == d') foo() # <-- duplicate code # New Version if a == b: print ('a == b') elif b == c: print ('b == c') elif c == d: print ('c == d') also: foo() # <-- No longer duplicated *Many nested 'if' statements could now be a more linear style:* # Old Version if a == b: print ('a == b') if b == c: print ('b == c') print ('end if') # New Version if a == b: print ('a == b') alif b == c: print ('b == c') also: print ('end if') These two examples are the most common ways this will help code. I have been writing code samples using these keywords and have found that it simplifies many other things as well. It does take a bit of getting used to, though. I have found that is is best to use 'also' and 'alif' sparingly, as overuse can make some code less flexible and more confusing. *Selective Branch merging:* One limitation of the 'also' and 'alif' keywords is the restriction to the "all of the above" checking. What I mean by that is that there is no way to pick and choose which branches to merge back together. When using 'also' and 'alif' you are catching all previous if-branches. One easy way to solve this would be to allow for named branching. The most simple way to do this is to save the conditions of each branch into a variable with a name. Here is an example of merging only select branches together: # Old Version if a == b: print ('a == b') elif a == c: print ('a == c') elif a == d: print ('a == d') if (a == b) or (a == d): print ('a == b and a == d') # New Version using 'as' keyword if a == b as aisb: print ('a == b') elif a == c: print ('a == c') elif a == d as aisd: print ('a == d') alif aisd or aisb: print ('a == b and a == d') NOTE: In the old version, it may be necessary to save off the boolean expression beforehand if the variables being used are changed in the first set of conditions. This would not be required in the new version, making the total code written an even larger gap. I realize that using the 'as' keyword may not be the best. Using 'as' seems to suggest that it will only be used in the following block. An alternative to using the 'as' keyword could be assigning the 'if' to a variable like so: aisb = if a == b: This looks a bit gross to me, though. If think of a better one, I would love to see it. *Speed:* The next logical question to ask is that of speed. Will this slow down my code at all? I happily submit to you that it shouldn't. If done right, this may even speed things up by a cycle or two (yeah. I know. That is not enough to fuss over, but I view it as a side benefit.) When just using 'also' and 'alif', there should be no difference in speed. The same number of jumps and checks should be done as before. Naming branches may add an additional assignment operation, but this should be more than made up for by not having to calculate the condition more than once. There may be a few cases where this would be slower, but those can be optimized to result in old-style code. *Implementation:* I am currently learning how the python parser and lexer work in hopes of making a custom version containing these features. Because of my lack of knowledge here, I cannot say how it should be implemented in python specifically. Here is how each if code block should work in basic english: on True: 1. Execute code block 2. Jump to next 'alif', 'also', or end of if-chain on False: 1. jump to next 'elif', 'else', or end of if-chain Note: 'also' blocks can be useful in more places than just the end of a chain: if a == b: print ('a == b') elif a == c: print ('a == c') also: print ('a == b == c') else: print ('a != b and a != c') It could also be useful to have more than one 'also' in an if chain. In contrast, having an 'else' before the end is not so useful. For example, placing one before an also makes the also a little pointless: if a == b: print ('a == b') elif a == c: print ('a == c') else: print ('a != b and a != c') also: print ('This will always execute') It would therefore be good to still require that 'else' be the last item in a chain. *The End* Thank you for humoring my idea. I am new to this mailing list, so sorry if this seems out of line or something. I am currently working with ansi C to try to get something similar into the 'C' language, but it could be 20 years before anything comes of that. The gears move ofly slow over there. Anyway, I look forward to hearing your feedback. -Cory Beutler [View Less]

10 18

Re: [Python-ideas] If branch merging
by random832＠fastmail.us June 8, 2015

June 8, 2015

On Sun, Jun 7, 2015, at 21:06, Cory Beutler wrote: > Thank you all for your responses. I didn't realize how much support this > mailing list had. > > In response to several responses: > > It appears I have hit a soft spot with the 'as' keyword. I don't have an issue with the as keyword, I was just pointing out that it disguises the fact that what you're really asking for seems to be general assignment expressions, since there is no particular rationale to constrain it to … [View More]

1 0