Mailman 3 August 2019 - Python-ideas

Proposal: Use "for x = value" to assign in scope
by Peter O'Connor 09 Aug '19

09 Aug '19

Alright hear me out here: I've often found that it would be useful for the following type of expression to be condensed to a one-liner: def running_average(x_seq): averages = [] avg = 0 for t, x in enumerate(x_seq): avg = avg*t/(t+1) + x/(t+1) averages.append(avg) return averages Because really, there's only one line doing the heavy lifting here, the rest is kind of boilerplate. Then I learned about the beautiful and terrible "for x in [value]": def running_average(x_seq): return [avg for avg in [0] for t, x in enumerate(x_seq) for avg in [avg*t/(t+1) + x/(t+1)]] Many people find this objectionable because it looks like there are 3 for loops, but really there's only one: loops 0 and 2 are actually assignments. **My Proposal** What if we just officially bless this "using for as a temporary assignment" arrangement, and allow "for x=value" to mean "assign within the scope of this for". It would be identical to "for x in [value]", just more readable. The running average function would then be: def running_average(x_seq): return [avg for avg=0 for t, x in enumerate(x_seq) for avg = avg * t/(t+1) + x / (t+1)] ------ P.S. 1 I am aware of Python 3.8's new "walrus" operator, which would make it: def running_average(x_seq): avg = 0 return [avg := avg*t/(t+1) + x / (t+1) for t, x in enumerate(x_seq)] But it seems ugly and bug-prone to be initializing a in-comprehension variable OUTSIDE the comprehension. ------ P.S. 2 The "for x = value" syntax can achieve things that are not nicely achievable using the := walrus. Consider the following example (wherein we carry forward a "hidden" variable h but do not return it): y_seq = [y for h=0 for x in x_seq for y, h = update(x, h)] There's not really a nice way to do this with the walrus because you can't (as far as I understand) combine it with tuple-unpacking. You'd have to do something awkward like: yh = None, 0 y_seq, _ = zip(*(yh := update(x, yh[1]) for x in x_seq)) ------

5 4

Universal parsing library in the stdlib to alleviate security issues
by Nam Nguyen 09 Aug '19

09 Aug '19

Hello list, I sent an email to this list two or three months ago about the same idea. In that discussion, there were both skepticism and support. Since I had some time during the previous long weekend, I have made my idea more concrete and I thought I would try with the list again, after having run it through some of you privately. GOAL: To have some parsing primitives in the stdlib so that other modules in the stdlib itself can make use of. This would alleviate various security issues we have seen throughout the years. With that goal in mind, I opine that any parsing library for this purpose should have the following characteristics: #. Can be expressed in code. My opinion is that it is hard to review generated code. Code review is even more crucial in security contexts. #. Small and verifiable. This helps build trust in the code that is meant to plug security holes. #. Less evolving. Being in the stdlib has its drawback that is development velocity. The library should be theoretically sound and stable from the beginning. #. Universal. Most of the times we'll parse left-factored context-free grammars, but sometimes we'll also want to parse context-sensitive grammars such as short XML fragments in which end tags must match start tags. I have implemented a tiny (~200 SLOCs) package at https://gitlab.com/nam-nguyen/parser_compynator that demonstrates something like this is possible. There are several examples for you to have a feel of it, as well as some early benchmark numbers to consider. This is far smaller than any of the Python parsing libraries I have looked at, yet more universal than many of them. I hope that it would convert the skeptics ;). Finally, my request to the list is: Please debate on: 1) whether we want a small (even private, underscore prefixed) parsing library in the stdlib to help with tasks that are a little too complex for regexes, and 2) if yes, how should it look like? I also welcome comments (naming, uses of operator overloading, features, bikeshedding, etc.) on the above package ;). Thanks! Nam

12 31

Re: adding support for a "raw output" in JSON serializer
by Peter Otten 08 Aug '19

08 Aug '19

Richard Musil wrote: > I have found myself in an awkward situation with current (Python 3.7) JSON > module. Basically it boils down to how it handles floats. I had been hit > on this particular case: > > In [31]: float(0.6441726684570313) > Out[31]: 0.6441726684570312 > > but I guess it really does not matter. > > What matters is that I did not find a way how to fix it with the standard > `json` module. I have the JSON file generated by another program (C++ > code, which uses nlohmann/json library), which serializes one of the > floats to the value above. Then when reading this JSON file in my Python > code, I can get either decimal.Decimal object (when specifying > `parse_float=decimal.Decimal`) or float. If I use the latter the least > significant digit is lost in deserialization. > > If I use Decimal, the value is preserved, but there seems to be no way to > "serialize it back". Writing a custom serializer: > > class DecimalEncoder(json.JSONEncoder): > def default(self, o): > if isinstance(o, decimal.Decimal): > return str(o) # <- This becomes quoted in the serialized > output > return super.default(o) > > seems to only allow returning "string" value, but then serializes it as a > string! I.e. with the double quotes. What seems to be missing is an > ability to return a "raw textual representation" of the serialized object > which will not get mangled further by the `json` module. You have overwritten the wrong method: $ cat demo.py import json import decimal class DecimalEncoder(json.JSONEncoder): def encode(self, o): if isinstance(o, decimal.Decimal): return str(o) return super().encode(o) input = "0.6441726684570313" obj = json.loads(input, parse_float=decimal.Decimal) output = json.dumps(obj, cls=DecimalEncoder) assert input == output $ python3 demo.py $

1 0

For-expression/throwaway comprehension
by Eli Berkowitz 07 Aug '19

07 Aug '19

(This is my first time posting on any Python list; I've tried to search for this idea and didn't find it but if I looked in the wrong places/this has already been discussed I apologize and feel free to tell me!) Say you have a list and you want to perform some operation on each item in the list - but you don't need to store the result in a list. There are three simple ways of doing this, at least as far as I know: ([print(item)] could be any expression, just using it as an example) ``` lst = [1, 2, 3, 4] #1 for item in lst: print(item) # 2 [print(item) for item in lst] # 3 for item in lst: print(item) ``` #1 - In my opinion, this should be a one line operation so #1 is not ideal. #2 - It also shouldn't require storing results in array, to save time/memory, so #2 is out. #3 - I think #3 is just not good syntax, it seems very unpythonic to me - it breaks the norm that blocks go on their own lines. It does seem the best of the three though and I know my assessment is kind of subjective. I'm wondering if a possible alternative syntax could be a for-expression, like there's if-expressions, which always evaluates to None: ``` print(item) for item in lst ``` A more practical example of when this would be useful is extending list-2 with a modified version of list-1 - this syntax would avoid creating an intermediate list (not sure if the way lists are implemented in python removes this advantage by the way it resizes lists though). ``` lst1 = [1, 2, 3] lst2 = [4, 5, 6] lst1.append(item * 2) for item in lst1 ```

23 76

Making io.BufferedReader buffer available
by Jérôme Carretero 06 Aug '19

06 Aug '19

Hi, I pushed http.client to a scenario where I would read streaming HTTP data (yeah, without going into native async) with a select() on HTTPResponse and using its read1(). Because HTTPResponse has a fileno() we're tempted to use select() on it, which works kind-of well... but there is one edge case. While consuming the HTTP response headers, HTTPResponse performs the buffered reading of the response (made by socket.makefile()) which may leave pending buffered payload data. In that case and when no more data is coming just yet, the select() is blocking despite some data available in buffers. While it's possible to set a custom HTTP response class in http.client, it would seem interesting (but I may be lacking background) to be able to retrieve the pending buffered data using arguments to peek() or read() on a BufferedReader, without accessing the underlying stream. What do you think? Regards, -- Jérôme

1 0

Non-standard evaluation for Python
by Nima Hamidi 05 Aug '19

05 Aug '19

Hello all, I would like to briefly share my thoughts on non-standard evaluation (NSE), why this is useful, and how, potentially, it can be added to Python. In most languages, functions have access only to the *value* of their arguments, and not to the expressions that yielded those values. However, sometimes it is useful to have access to the expressions as well. For example, let plot(x, y, ...) be a function that draws the scatter plot of y against x and adds labels for the x- and y-axes. Currently, there is no way to distinguish plot(x, *z*, ...) from plot(x, *y*, ...) from within the function. As such, one needs to pass the names of the variables to the plot function to be used as the axes' labels. Using NSE, though, the function can look at the expressions in the method call and use those as the default value for labels. In R, this idea is used widely to make the syntax simpler. In the following, I sketch how I think this can be implemented: 1. Let BoundExpression be a class containing an ast.Expression as well as locals and globals dictionaries. BoundExpression can also have an eval method that evaluates its expression using its locals and globals dictionaries. 2. Let `x` be the short form for BoundExpression("x", locals(), globals()). Then, the plot function can be implemented in a way that plot(`x`, `y`) draws scatter plot of x and y *and also,* labels the axes correctly. No need to provide labels explicitly anymore. 3. A more challenging idea is to let developers decide whether their functions need NSE or not. For example, when a function is defined as def f(`x`), for any method call like f(x), the first argument should be wrapped with a BoundExpression instance. I would appreciate any feedback on these ideas. Best, Nima

5 18

Coding environments
by David Mertz 02 Aug '19

02 Aug '19

On Fri, Aug 2, 2019, 1:56 PM Chris Angelico <rosuav(a)gmail.com> wrote: > Also a bit old-school (it took me many years to learn the value of > syntax highlighting), and an educator, and I've seen students start > out with Jupyter. As an alternative to the vanilla REPL, I think it's > awesome if a little expensive on low-end machines; but as an > alternative to a text editor, it's an attractive nuisance. Yes, you > can try things out and see the results instantly, AND you can save it, > edit, rerun, etc; the cost is that debugging becomes a nightmare. But > for discoverability, incl tab completion? It's great, and I probably > should start using it more. There are definitely things for which Jupyter is perfect, and things for which it is a terrible choice. A lot of data scientists (of whom I am one, but via a winding path) don't know anything outside Jupyter. The result is that they throw rambling code that should be a script or module into a giant notebook with many unrelated cells stuck in the middle (and no way to add unit tests, code checkers and linters have poor support, execution order is not transparent, etc). That said, the interactive notebook is a huge win for many things. I recently produced a bunch of somewhat related graphs. Each sliced and diced data a little bit differently, then rendered then with with different visual options. I didn't come in with known requirements or specification, just a general sense of what I wanted to illustrate. So modify a cell, rerun the graph, repeat a hundred times. Doing that in a plain code editor with a separate image file viewer would have been WAY harder. Jupyter is right for this. On the other hand, I wound up with the same blocks of 40-ish lines of code repeated 5 times in different places, which is terrible. But at least I then took the time to move all of that to a utility module, factoring differences in behavior into some function arguments. So the notebook is small, but allows visualizing many variations without switching tool context.

1 0

PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro 02 Aug '19

02 Aug '19

I think that adding a `no` keyword as an alias for `not` would make for more readable, simple, pythonic code. Take the below: ``` if not val: do_thing_because_value_is_falsy() ``` could be (is actually understood as): ``` if no val: do_thing_because_value_is_falsy() ``` I think this PEP is a work-around for an underlying subtle issue with how the `not` operator is used. It has two use-cases: 1. as a NOT gate for producing opposite boolean values ``` opposite = not regular ``` 2. as a sort of ".is_falsy()" checker; when used with an if statement. like the first example. This PEP would make the difference between the two usecases explicit. Thoughts? Best Intentions, Daniel Okey-Okoro.

6 5

Fwd: Re: PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro 02 Aug '19

02 Aug '19

---------- Forwarded message --------- From: Daniel Okey-Okoro <danielokeyokoro(a)gmail.com> Date: Thu, Aug 1, 2019 at 1:37 PM Subject: Re: [Python-ideas] Re: PEP: add a `no` keyword as an alias for `not` To: Chris Angelico <rosuav(a)gmail.com> > not a strong enough justification for breaking any code that uses "no" in any other way. This is a very crucial point I didn't consider. ------------------------------------------------------------ What if we could lexically limit it to `if no {{everything else in the if block}}`? I think that would sufficiently protect us from unintentionally breaking people's code. ----------------------------------------------------------- > The bar for creating new keywords is extremely high I'm grateful that it is. I think python is awesome. And I trust the community/council to keep it awesome. If the idea doesn't make sense to most people then I'm happy with it never getting picked up. Just sharing an idea. On Thu, Aug 1, 2019 at 1:12 PM Chris Angelico <rosuav(a)gmail.com> wrote: > On Fri, Aug 2, 2019 at 6:09 AM Daniel Okey-Okoro > <danielokeyokoro(a)gmail.com> wrote: > > > > I think that adding a `no` keyword as an alias for `not` would make for > > more readable, simple, pythonic code. > > The bar for creating new keywords is extremely high. Simply making > your code slightly more like English is not a strong enough > justification for breaking any code that uses "no" in any other way. > > ChrisA > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/DJUKC… > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Best Intentions, Daniel Okey-Okoro. -- Best Intentions, Daniel Okey-Okoro.

2 1

Fwd: PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro 01 Aug '19

01 Aug '19

---------- Forwarded message --------- From: Daniel Okey-Okoro <danielokeyokoro(a)gmail.com> Date: Thu, Aug 1, 2019 at 1:24 PM Subject: Re: PEP: add a `no` keyword as an alias for `not` To: Calvin Spealman <cspealma(a)redhat.com> Good point Calvin, ------------------------------------------------------------ But in many cases, when people write `if not val` they're checking if the val is `None` i.e. `if val is None`, `if val == None` (so in effect it's kind of like they're checking if "something isn't there") I just did a code review. The idea popped in my head after reading code that behaved this way. ------------------------------------------------------------------- Plus, I'm suggesting an alias. In some usecases it would be better to just use `if not val`. i.e. in cases where you are strictly doing True/False assessments not Falsey ( is_empty() ) assessments e.g. None, [], "", 0 ``` Best Intentions, Daniel Okey-Okoro On Thu, Aug 1, 2019, 1:10 PM Calvin Spealman <cspealma(a)redhat.com> wrote: > I think they actually read like they would mean slightly different things, > which would make them existing as aliases confusing. > > I read `if not val` as "If val isn't true" but i would read `if no val` as > "if val does not exist" > > On Thu, Aug 1, 2019 at 4:07 PM Daniel Okey-Okoro < > danielokeyokoro(a)gmail.com> wrote: > >> I think that adding a `no` keyword as an alias for `not` would make for >> more readable, simple, pythonic code. >> >> Take the below: >> >> ``` >> if not val: >> do_thing_because_value_is_falsy() >> ``` >> >> could be (is actually understood as): >> >> ``` >> if no val: >> do_thing_because_value_is_falsy() >> ``` >> >> I think this PEP is a work-around for an underlying subtle issue with how >> the `not` operator is used. >> >> It has two use-cases: >> >> 1. as a NOT gate for producing opposite boolean values >> >> ``` >> opposite = not regular >> ``` >> >> 2. as a sort of ".is_falsy()" checker; when used with an if statement. >> >> like the first example. >> >> >> This PEP would make the difference between the two usecases explicit. >> >> Thoughts? >> >> Best Intentions, >> Daniel Okey-Okoro. >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > > -- > > CALVIN SPEALMAN > > SENIOR QUALITY ENGINEER > > cspealma(a)redhat.com M: +1.336.210.5107 > [image: https://red.ht/sig] <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > -- Best Intentions, Daniel Okey-Okoro.

1 0