Mailman 3 November 2016 - Python-ideas

Reduce/fold and scan with generator expressions and comprehensions
by Danilo J. S. Bellini 14 Nov '16

14 Nov '16

The idea is to let generator expressions and list/set comprehensions have a clean syntax to access its last output. That would allow them to be an alternative syntax to the scan higher-order function [1] (today implemented in the itertools.accumulate function), which leads to an alternative way to write a fold/reduce. It would be nice to have something like: >>> last(abs(prev - x) for x in [3, 4, 5] from prev = 2) 2 instead of a reduce: >>> from functools import reduce >>> reduce(lambda prev, x: abs(prev - x), [3, 4, 5], 2) 2 or an imperative approach: >>> prev = 2 >>> for x in [3, 4, 5]: ... prev = abs(prev - x) >>> prev 2 or getting the last from accumulate: >>> from itertools import accumulate >>> list(accumulate([2, 3, 4, 5], lambda prev, x: abs(prev - x)))[-1] 2 or... >>> [prev for prev in [2] ... for x in [3, 4, 5] ... for prev in [abs(prev - x)] ... ][-1] 2 Actually, I already wrote a solution for something similar to that: PyScanPrev [2]. I'm using bytecode manipulation to modify the generator expression and set/list comprehensions semantics to create a "scan", but it has the limitation of using only code with a valid syntax as the input, so I can't use "from" inside a generator expression / list comprehension. The solution was to put the first output into the iterable and define the "prev" name elsewhere: >>> last(abs(prev - x) for x in [2, 3, 4, 5]) 2 That line works with PyScanPrev (on Python 3.4 and 3.5) when defined in a function with a @enable_scan("prev") decorator. That was enough to create a "test suite" of doctest-based examples that shows several scan use cases [2]. This discussion started in a Brazilian list when someone asked how she could solve a simple uppercase/lowercase problem [3]. The goal was to alternate the upper/lower case of a string while neglecting the chars that doesn't apply (i.e., to "keep the state" when the char isn't a letter). After the discussion, I wrote the PyScanPrev package, and recently I've added this historical "alternate" function as the "conditional toggling" example [4]. Then I ask, can Python include that "scan" access to the last output in its list/set/dict comprehension and generator expression syntax? There are several possible applications for the scan itself as well as for the fold/reduce (signal processing, control theory, physics, economics, etc.), some of them I included as PyScanPrev examples. Some friends (people who like control engineering and/or signal processing) liked the "State-space model" example, where I included a "leaking bucket-spring-damper" simulation using the scan-enabled generator expressions [5]. About the syntax, there are several ideas on how that can be written. Given a "prev" identifier, a "target" identifier, an input "iterable" and an optional "start" value (and perhaps an optional "echo_start", which I assume True by default), some of them are: [func(prev, target) for target in iterable from prev = start] [func(prev, target) for target in iterable] -> prev = start [func(prev, target) for target in iterable] -> prev as start [func(prev, target) for target in iterable] from prev = start [func(prev, target) for target in iterable] from prev as start [func(prev, target) for target in iterable] with prev as start prev = start -> [func(prev, target) for target in iterable] prev(start) -> [func(prev, target) for target in iterable] [func(prev, target) for prev -> target in start -> iterable] [prev = start -> func(prev, target) for target in iterable] # With ``start`` being the first value of the iterable, i.e., # iterable = prepend(start, data) [func(prev, target) for target in iterable from prev] [func(prev, target) for target in iterable] -> prev [func(prev, target) for target in iterable] from prev prev -> [func(prev, target) for target in iterable] Before writing PyScanPrev, in [6] (Brazilian Portuguese) I used stackfull [7] to implement that idea, an accumulator example using that library is: >>> from stackfull import push, pop, stack >>> [push(pop() + el if stack() else el) for el in range(5)] [0, 1, 3, 6, 10] >>> list(itertools.accumulate(range(5))) [0, 1, 3, 6, 10] There are more I can say (e.g. the pyscanprev.scan function has a "start" value and an "echo_start" keyword argument, resources I missed in itertools.accumulate) but the links below already have a lot of information. [1] https://en.wikipedia.org/wiki/Prefix_sum#Scan_higher_order_function [2] https://pypi.python.org/pypi/pyscanprev [3] https://groups.google.com/forum/#!topic/grupy-sp/wTIj6G5_5S0 [4] https://github.com/danilobellini/pyscanprev/blob/v0.1.0/examples/conditiona… [5] https://github.com/danilobellini/pyscanprev/blob/v0.1.0/examples/state-spac… [6] https://groups.google.com/forum/#!topic/grupy-sp/UZp-lVSWK1s [7] https://pypi.python.org/pypi/stackfull -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)

11 41

Python Documentation
by adil gourinda 12 Nov '16

12 Nov '16

I) Thank you for responding to my message because it makes me happy to know that my message was read by your team. II) The “.okular” file is an archive made by “Okular software” and contains the “PDF” file with its annotations. “https://okular.kde.org/” this is the link to the software, I work with “openSUSE” “https://www.opensuse.org/”. III) For someone who wants to learn programming language and chose “Python” to start with it, Having a good tutorial is a crucial moment. Me as an example, I don’t follow any study in informatics field so I build my knowledge thanks to my efforts, In consequence I don’t have a strong base in programming language. Even that I found some parts in “Python Language Reference” and “Python library reference” easy to learn for someone like me. I suggest to aggregate those informations and add them to the “Python tutorial” to build a strong introduction to your great language “Python”.

1 0

Alternative to PEP 532: delayed evaluation of expressions
by Eric V. Smith 12 Nov '16

12 Nov '16

Creating a new thread, instead of hijacking the PEP 532 discussion. From PEP 532: > Abstract > ======== > > Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this PEP > proposes the addition of a new protocol-driven circuit breaking operator to > Python that allows the left operand to decide whether or not the expression > should short circuit and return a result immediately, or else continue > on with evaluation of the right operand:: > > exists(foo) else bar > missing(foo) else foo.bar() Instead of new syntax that only works in this one specific case, I'd prefer a more general solution. I accept being "more general" probably seals the deal in killing any proposal! I realize the following proposal has at least been hinted at before, but I couldn't find a specific discussion about it. Since it applies to the short-circuiting issues addressed by PEP 532 and its predecessors, I thought I'd bring it up here. It could also be used to solve some of the problems addressed by the rejected PEP 463 (Exception-catching expressions). See also PEP 312 (Simple Implicit Lambda). It might also be usable for some of the use cases presented in PEP 501 (General purpose string interpolation, aka i-strings). I'd rather see the ability to have unevaluated expressions, that can later be evaluated. I'll use backticks here to mean: "parse, but do not execute the enclosed code". This produces an object that can later be evaluated with a new builtin I'll call "evaluate_now". Obviously these are strawmen, and partly chosen to be ugly and unacceptable names and symbols in the form I'll discuss here. Then you could write a function: eval_else(`foo.bar`, `some_func()`) whose value is foo.bar, unless foo.bar cannot be evaluated, in which case the value is some_func(). def eval_else(expr, fallback, exlist=(AttributeError,)): try: return evaluate_now(expr) except exlist: return evaluate_now(fallback) Exactly which exceptions you catch is up to you. Of course there's the chance that someone would pass in something for which the caught exception is too broad, and it's raised deep inside evaluating the first expression, but that's no different than catching exceptions now. Except I grant that hiding the try/except inside a called function increases the risk. Like f-strings, the expressions are entirely created at the site they're specified inside ``. So they'd have access to locals and globals, etc., at the definition site. def x(foo, i): return eval_else(`foo.bar`, `some_func(i, __name__)`) And like the expressions in f-strings, they have to be valid expressions. But unlike f-strings, they aren't evaluated right when they're encountered. The fact that they may never be evaluated is one of their features. For example the if/else expression: if_else(`y`, x is None, `x.a`) could be defined as being exactly like: y if x is None else x.a including only evaluating x.a if x is not None. def if_else(a, test, b): if test: return evaluate_now(a) return evaluate_now(b) You could do fancier things that require more than 2 expressions. Whether `` returns an AST that could later be manipulated, or it's something else that's opaque is another discussion. Let's assume it's opaque for now. You could go further and say that any argument to a function that's specially marked would get an unevaluated expression. Suppose that you can mark arguments as & to mean "takes an unevaluated expression". Then you could write: def if_else(&a, test, &b): if test: return evaluate_now(a) return evaluate_now(b) And call it as: if_else(y, x is None, x.a) But now you've made it non-obvious at the caller site exactly what's happening. There are other downsides, such as only being able to create an unevaluated expression when calling a function. Or maybe that's a good thing! In any event, having unevaluated expressions would open up more possibilities than just the short-circuit evaluation model. And it doesn't involve a new protocol. Eric.

11 14

Improve error message when missing 'self' in method definition
by Nathan Goldbaum 12 Nov '16

12 Nov '16

Hi all, Recently pypy received a patch that improves the error message one gets when 'self' is missing in a method's signature: https://mail.python.org/pipermail/pypy-dev/2016-September/014678.html Here are the commits that implement the change in pypy: https://bitbucket.org/pypy/pypy/commits/all?search=branch(better-error-miss… I'm curious whether a similar improvement would also be received well in CPython. In particular, this guides one to the correct solution for a common programming mistake made by newcomers (and even not-so-newcomers). Here is a case study that found this was a common source of errors for newcomers: http://dl.acm.org/citation.cfm?id=2960327 -Nathan

17 25

Re: [Python-ideas] PEP 532: A circuit breaking operator and protocol
by Adam Bartoš 11 Nov '16

11 Nov '16

Just a small suggestion: wouldn't it be better to use adjectives instead of verbs for proposed builtin names and their variants? existing(foo) else bar non_None(foo) else bar instead of exists(foo) else bar is_not_None(foo) else bar Regards, Adam Bartoš

1 0

"Proposing a change" document? (was: Civility on this mailing list)
by Franklin? Lee 11 Nov '16

11 Nov '16

(Is that the right way to change subject? Anyway, context here: https://mail.python.org/pipermail/python-ideas/2016-October/043208.html) On Fri, Oct 21, 2016 at 11:07 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > > I wonder. Would there be value in adding a sign-up email to the list > (supported by a posting of that email to the list, to catch existing > contributors) that set out some of the basic principles of how changes > are judged for inclusion in Python? Turn it around. Have a document outlining how one should _argue_ for their idea. Instead of, "We get a lot of ideas. Are you SURE your idea is good?" say, "We WANT to hear your idea, and here is the best way to get us to listen." A list of reasons why ideas get rejected will make them defensive, while a list of points they need to prove will put them on the offensive, and also get them thinking deeply about the potential costs/issues of their idea. Encourage arguments with sample code (before and after their change). Encourage having real-world code (or a stripped-down version of actual code) and showing how their change makes the code better. Emphasize _how_ this helps convince others. I think this attitude will reduce, not increase, the number of posts (from the people who read the document). It's easier to think, "My idea is amazing," than to think, "I've demonstrated all that I needed to." You might also get better-quality posts: instead of _mentally_ going down a checklist, they'd be writing out how their idea fulfills that checklist. I'm worried about how the community would self-police with such a document. Maybe they'll bite harder when someone doesn't follow it. > * The fact that the default answer is typically "no", along with an > overview of the reasons *why* the status quo wins by default. Instead: "Any new feature adds maintenance burden in the code, in the docs, and in the minds of Python developers. You should show how the benefits of the feature will outweigh these costs." (P.S.: I don't believe a lack of implementor is a point against an idea (except when there's a competing idea). It doesn't further the discussion for _whether_ the idea is good. It's not unheard of for ideas to reach PEP status, then stop there because no one will do the work. Someone can always find the discussion later, see that it's popular, and decide to work on it. Anyone wanna convince me otherwise?) > * Some of the simple "rules of thumb" like "not every 2-line function > should be a builtin. That, in particular, is a bad response type. It implies the poster's supporting arguments are strongly based on having _universal_ consistency, which is rarely the case. In my experience, "not every" (anti-universal?) counterarguments are usually used against non-universal arguments, and then you have "I didn't say they did" responses. Instead, contrast a few levels of inclusion, like built-ins, batteries, itertools recipes, PyPI, personal library. Talk about different scales to measure features, like "broadly useful" and "easy to specify/understand, but hard to implement". Have them pick a function or two at the same level of inclusion as their function, and show that their function measures favorably against the included function on the given scales. Have them think about (putting an implementation on) PyPI as a way to prove that their idea will be useful to people. It's not a backseat to stdlib, it's the way toward stdlib. > * Basic reminders that Python is used by a very diverse set of users, > and proposals that are only beneficial for a limited group need to be > weighed against the disruption to the majority who get no benefit. Instead, have them discuss who will benefit from their idea, and what it will cost everyone else who doesn't need it. In particular, how would a Python user, especially a novice, learn to read code written that uses their idea? Do other implementations need to implement this feature to be compatible with CPython? > * The above comment, that we welcome ideas because it's important that > we don't stagnate and having assumptions challenged is valuable, even > if the bar for getting such ideas accepted is (necessarily) high. I don't like that last part. It's sort of prideful, and also unnecessary.

5 4

Re: [Python-ideas] Method signature syntactic sugar (especially for dunder methods)
by Nathan Dunn 08 Nov '16

08 Nov '16

> * __add__ is only part of the addition protocol, there is also > __radd__ and __iadd__ __iadd__ could be represented as def self += value:. Reflected binary operators do pose a problem. A possible solution would be to give self special meaning in this context, so def self + other: would correspond to __add__ and def other + self: to __radd__. But of course, this would be a more fundamental change to the language than just syntactic sugar. > * likewise, there is not a one-to-one correspondence between the > bool() builtin and the __bool__() special method (there are other ways > to support bool(), like defining __len__() on a container) That is a fair point, but I think this would be only a slight point of confusion. I believe for the most part other builtins such as __abs__, __len__, etc. do have a one-to-one correspondence so this syntax would make sense in almost all situations. But while were on the topic, I will take this opportunity to note that all the methods to support builtins suffer from the same ambiguity I alluded to earlier with bool(self). One solution could be to use a @builtin decorator or something to that effect, but that would more or less defeat the purpose of cleaner, more intuitive syntax. It may be that there is no viable alternative for the builtin function dunder methods. > * the mapping protocol covers more than just __getitem__ __setitem__(self, key, value) could be def self[key] = value likewise __delitem__(self, key) def del self[key]: __iter__(self) iter(self) __len__(self) len(self) > (and you also > need to decide if you're implementing a mapping, sequence, or > multi-dimensional array) Could you speak more about this? It doesn't seem to me that this would affect the method signatures since it is up to the implementer to interpret the semantic meaning of the keys. > it would break the current symmetry between between name binding > in def statements and target binding in assignment statements > With the proposed change, we'd face the > problem that the following would both be legal, but meant very > different things: > > cls.mymethod = lambda self: print(self) > def cls.mymethod(self): print(self) > > The former is already legal and assigns the given lambda function as a > method on the existing class, `cls` > > The latter currently throws SyntaxError. > > With the proposed change, rather than throwing SyntaxError as it does > now, the latter would instead be equivalent to: > > def mymethod(cls, self): print(self) > > which would be a very surprising difference in behaviour. I will admit that there is potential for confusion here as this class Formatter(object): def Formatter.displayDefault(obj): print('[{}]'.format(obj)) is rather suggestive of a static method, but I don't think the confusion arises because the syntax breaks the symmetry you reference. The self-dot syntax *is* meant to elicit target binding in assignment statements. Just as self.mymethod = lambda p1, p2: p1+p2 indicates that an instance of the class has a callable with two parameters bound to its mymethod attribute, so too does the statement def self.mymethod(p1, p2):. So I don't think this inherently breaks any symmetry. It is specifically in the case that the identifier is the same as the class name that this leads to surprising behavior, but there is already a similar risk of confusion any time the name of a class is used as an identifier within the class body. Perhaps my proposed syntax does increase this risk since the identifier doesn't look like a parameter or a variable, but hopefully the self form of this syntax would be so commonplace that seeing anything else before the dot would throw up a red flag. Thanks, -Nate

5 4

Civility on this mailing list
by Brett Cannon 07 Nov '16

07 Nov '16

Based on some emails I read in the " unpacking generalisations for list comprehension", I feel like I need to address this entire list about its general behaviour. If you don't follow me on Twitter you may not be aware that I am taking the entire month of October off from volunteering any personal time on Python for my personal well-being (this reply is being done on work time for instance). This stems from my wife pointing out that I had been rather stressed in July and August outside of work in relation to my Python volunteering (having your weekends ruined is never fun). That stress stemmed primarily from two rather bad interactions I had to contend with on the issue track in July and August ... and this mailing list. When I have talked to people about this mailing list it's often referred to by others as the "wild west" of Python development discussions (if you're not familiar with US culture, that turn of phrase basically means "anything goes"). To me that is not a compliment. When I created this list with Titus the goal was to provide a safe place where people could bring up ideas for Python where people could quickly provide basic feedback so people could know whether there was any chance that python-dev would consider the proposal. This was meant to be a win for proposers by not feeling like they were wasting python-dev's time and a win for python-dev by keeping that list focused on the development of Python and not fielding every idea that people want to propose. And while this list has definitely helped with the cognitive load on python-dev, it has not always provided a safe place for people to express ideas. I have seen people completely dismiss people's expertise and opinion. There has been name calling and yelling at people (which is always unnecessary). There have been threads that have completely derailed itself and gone entirely off-topic. IOW I would not hold this mailing list up as an example of the general discourse that I experience elsewhere within the community. Now I realize that we are all human beings coming from different cultural backgrounds and lives. We all have bad days and may not take the time to stop and think about what we are typing before sending it, leading to emails that are worded in a way that can be hurtful to others. It's also easy to forget that various cultures views things differently and so that can lead to people "reading between the lines" a lot and picking up things that were never intended. There are 1,031 people on this mailing list from around the world and it's easy to forget that e.g. Canadian humour may not translate well to Ukrainian culture (or something). What this means is it's okay to *nicely* say that something bothered you, but also try to give people the benefit of the doubt as you don't know what their day had been like before they wrote that email (I personally don't like the "just mute the thread" approach to dealing with bad actors when the muting is silent as that doesn't help new people who join this mailing list and the first email they see is someone being rude that everyone else didn't see because they muted the thread days ago). As for the off-topic threads, please remember there are 1,031 people on this mailing list (this doesn't count people reading through gmane or Google Groups). Being extremely generous and assuming every person on this list only spends 10 seconds deciding if they care about your email, that's still nearly 3 hours of cumulative time spent on your email. So please be cognisant when you reply, and if you want to have an off-topic conversation, please take it off-list. And finally, as one of the list administrators I am in a position of power when it comes to the rules of this list and the CoC. While I'm one of the judges on when someone has violated the CoC, I purposefully try not to play the role of police to avoid bias and abuse of power. What that means is that I never personally lodge a CoC complaint against anyone. That means that if you feel someone is being abusive here you cannot rely on list admins noticing and doing something about it. If you feel someone has continuously been abusive on this list and violating the CoC then you must email the list admins about it if you wish to see action taken (all communications are kept private among the admins). Now I'm not asking people to email us on every small infraction (as I said above, try to give everyone a break knowing we all have bad days), but if you notice a pattern then you need to speak up if you would like to see something change. When I started my month off I thought that maybe if I only read this mailing list once a week that the frequency would be low enough that I could handle the stress of being both a participant and admin who is ultimately responsible for the behaviour here, but I'm afraid that isn't going to cut it. What I don't think people realize is that I don't take my responsibility as admin lightly; any time anyone acts rudely I take it personally like I somehow failed by letting the atmosphere and discourse on this list become what it is. Because of this I'm afraid I need to mute this mailing list for the rest of my vacation from volunteering in the Python community after I send this email. I personally hope people do take the time to read this email and reflect upon how they conduct themselves on this mailing list -- and maybe on other lists as well -- so that when I attempt to come back in November I don't have to permanent stop being a participant on this list and simply become an admin for this list to prevent complete burn-out for me in the Python community (and I know this last sentence sounds dramatic, but I'm being serious; the irony of receiving the Frank Willison award the same year I'm having to contemplate fundamentally shifting how I engage with the community to not burn out is not lost on me). -Brett

7 9

Python Documentation
by adil gourinda 07 Nov '16

07 Nov '16

Hi I am an amateur of computer programming and my ambition push me to choose a good programming language to begin with and I chose "Python" because it is an easy language and the documentation is free and complete. For my learning I was based on your "Python Documentation" and one "Python Shell" to exercise what i learned. I want to send you my notes ( and the way by which are noted ) and I wish you to use them to enhance the documentation. Because I am an amateur I don't have a strong base in programming so what is noted I found it easy to learn (I think everyone can do the same ) and what is not noted I had difficulty in learning it ( with ideas from other peoples we can make it easy to learn ) Thank you for accepting my offer and I wish I was helpful

3 2

PythonOS
by victor rajewski 07 Nov '16

07 Nov '16

Firstly, apologies if this is the wrong forum for this idea; it's not so much about the language itself, but what surrounds it. If you have any ideas for better forums, please let me know. Also, if there is any work started on anything like this, let me know Now to the idea: PythonOS - a (likely linux-based) OS built specifically for python. I've been working in the education space as well as with embedded systems for quite a few years, and am really excited about tiny affordable standalone computing devices as a computing and science educational tool. At the moment however, there are two options for python in this space: - micropython, which, despite its awesomeness, lacks many library possibilities, doesn't have a debugger, and is restricted to a handful of devices. The learning curve for getting this up and running can be quite steep - A full linux setup on an SBC - let's take the raspberry pi. This has access to (just about) everything, but beyond just programming the python code, you need to have some (basic) system administration training to get it running. For someone new to programming, this can be quite intimidating (recent developments with drag-and-drop Pi configuration notwithstanding). In the latter category, there seems to be a lot of newer, cheaper hardware appearing - for example the Pi Zero, C.H.I.P and VoCore2, and these are already at very affordable price point; I think that before long the rationale for micropython will be lost, and $2 hardware will be running a fully capable linux setup. However, micropython has the advantage of simplicity. Upload main.py, and press reset. That's it. My proposal is a Linux distribution that has the simplicity of micropython, but the power of full python. Drop a file (let's call it main.py) on an SD card from your laptop, plug it into the device, and it boots and runs that file straight away. Or drop an entire project. If the device has USB device capabilities, then just drag the file across, or even edit it live. Easy SSH setup could allow more advanced users remote development and debugging options. Maybe jupyter could be leveraged to provide other network-based dev options (pynq <http://www.pynq.io/> already has a working linux distro like this). Connecting to a console would give you a python prompt. Ideally, key OS functions should be able to be controlled from within python, so that the user never has to leave the python prompt. For a start, things like network config, and possibly cron. What do you think? I'm not in a state to do all of this myself, but could give it a start if there is interest in the concept. -- Victor Rajewski Sent from my mobile device. please. Please excuse brevity and any errors.

3 2