Mailman 3 June 2019 - Python-ideas

Re: `if-unless` expressions in Python
by Steven D'Aprano June 26, 2019

June 26, 2019

On Mon, Jun 24, 2019 at 12:37:48PM -0700, Andrew Barnert wrote: > Since you bring up “feeble mail clients”: > > Good mail clients can be configured to collapse and expand quotes, and > to automatically start long nested quotes collapsed. Indeed you are correct, and my own mail client supports that as an optional add-on, which I've tried and discarded as more annoying than helpful. In my experience the problem is that trying to solve the problem of quoting via technology is too crude: all the ones I've see are all or nothing, hide all context or show all context. Its really a *human* problem, not a technology problem: as the reader, I want to see *relevant* context, but not the entire history of the discussion. I cannot see any automated tool being able to guess what is relevant any time soon. > In fact, most > feeble clients (like Gmail and Yahoo, and the builtin iOS and Android > apps) do this whether you want it or not. I can't speak for the others, but my recollection of Gmail is that replies default to top posting with the quoted text below your response. [...] > But you’re not going to convince everyone to do it traditionally all > the time I'm not trying to do that. But surely it's not too hard to ask that every few posts somebody in the thread trims quoting to keep it under control? I'm not bitching because somebody quoted seven lines rather than three, I'm pointing out that when you have something like five pages of quoted text to a paragraph or three of new text, quoting is out of control. If we can't keep the quoting short in every post, we can hopely cull some of it periodically. We're programmers. We've learned by hard experience to delete dead code, not to leave it sitting in the program commented out, and not just because it makes reading the source code harder. It makes it harder to search, it makes files and diffs larger, it increases the ratio of noise to signal. Excessive quoting in email is the same, whether your mail client hides it by default or not, the noise is still there. > And the one-time hassle of figuring out how to configure your MUA, or > even switching to a better one "Better" is subjective, and just because a client is arguably better in one regard doesn't make it better in others. -- Steven

2 1

`if-unless` expressions in Python
by James Lu June 25, 2019

June 25, 2019

`if-unless` expressions in Python if condition1 expr unless condition2 is an expression that roughly reduces to expr if condition1 and not condition2 else EMPTY This definition means that expr is only evaluated if `condition1 and not condition2` evaluates to true. It also means `not condition2` is only evaluated if `condition1` is true. # EMPTY EMPTY is not actually a real Python value-- it's a value that collapses into nothing when used inside a statement expression: print([ if False never_called() unless False, if False never_called() unless False, ]) # => [] print([ 3, if False never_called() unless False, if False never_called() unless False, 2, if True 5 unless False, 4 ]) # => [3, 2, 5, 4] EMPTY is neither a constant exposed to the Python runtime nor a symbol. It's a compiler-internal value. # Use cases Assertions. assert if condition1 predicate(object) unless condition2 (This would be more readable with assert expressions.) Macros. # Equivalent syntax in existing Python As a statement: if condition1 and not condition2: predicate(object) predicate(object) if condition1 and not condition2 # Backward compatibility The `unless` word is only recognized as special inside `if-unless` statements. The continued use of the word as a variable name is discouraged.

12 26

Make $ a valid identifier and a singleton
by James Lu June 24, 2019

June 24, 2019

Make $ a valid identifier and a singleton. $ is a useful placeholder in []. Possible function partial syntax: def foo(x, y): print(x, y) partialized = foo[$, 10] partialized(5) # => 5 10

4 3

Proposal: Allowing any variable to be used in a 'with... as...' expression
by Yonatan Zunger June 24, 2019

June 24, 2019

Hi everyone, I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK. *Summary: *The construction with expr1 as var1, expr2 as var2, ...: body fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False. *Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and *not* doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be. For example, this is a good pattern: with functionReturningFile(...) as input: doSomething(input) There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is *not* valid Python: with functionReturningOptionalFile(...) as input: doSomething(input) To handle this case, one has a few options. One may only use the 'with' in the known safe cases: inputFile = functionReturningOptionalFile(...) if inputFile: with inputFile as input: doSomething(input) else: doSomething(None) (NB that this requires factoring the with statement body into its own function, which may separately reduce readability and/or introduce overhead); one may dispense with the 'with' clause and do it in the pre-PEP343 way: try: input = functionReturningOptionalFile(...) doSomething(input) finally: if input: input.close() (This sacrifices all the benefits of the with statement, and requires the caller to explicitly call the cleanup methods, increasing error-proneness); or one may construct an explicit 'dev-null' class and return it instead of the file: class DevNullFile(object): .... implement the entire File API, including a context manager ... (This can only be described as god-awful, especially for complex API's like files) One obvious option would be to allow None to act as a context manager as well. We might contrast this with PEP 336 <https://www.python.org/dev/peps/pep-0336/>, "Make None Callable." This was rejected (rightly, I think) because "it is considered a feature that None raises an error if called." For example, it means that if a function variable has been nulled, attempting to call it later raises an error, as this usually indicates a code mistake. In the case where that is not correct, it is easy to assign a noop lambda to the function variable instead of None, thus allowing the error-checking and the function-deactivating behaviors to both persist, and in a clear and easily understandable way. In this case, OTOH, the AttributeError raised if None is passed to a with statement has significantly lower value. As the example above illustrates, there are many cases where None is an entirely legitimate value to want to pass, and unlike in the other situation, there is no equally easy way to pass it. Furthermore, if the passing of None *is* an error in some case, it is more useful to see that error at the site where the variable is actually used in the with statement body -- the thing for which it does not make sense to use None -- rather than at a structural declaration which essentially defines a variable scope. This is also the reason why such a change would impact relatively little existing code: code already has to be structured to prevent this from happening. If the assigned expression in the with statement could only return None as a result of a code bug, and a piece of existing code is relying on the with statement to catch it, it would instead fall through and be caught by their own body code, presumably giving a more coherent error anyway. This is a nonzero change in behavior, but it's well within the scope of behavior changes which normally occur from version to version. One alternative to this proposal would be to have only None allowed to act as a context manager. However, None is not particularly special in this regard; the logic above applies to any function which might return a Union type. Furthermore, allowing it for any type would permit the following construction as well: with var1 as expr1, var2 as expr2, ... .... body ... where the common factor between the variables is no longer their need for a guaranteed cleanup operation, but simply that they are semantically all tied to a single scope of the code. This improves code clarity, as it allows the syntax to follow the intent more closely, and also eliminates one other ugliness. In present Python, the required syntax for the above would be var1 = expr1 var3 = expr3 with var2 as expr2, var4 as expr4: ... body ... where the variables in the 'with' statement are those which satisfy the context manager protocol, and the ones above it are those which do not satisfy the protocol. The split between the two is entirely tied to a nonlocal fact about the code, namely the implementation of the return values of each of the expressions, making it nonobvious which is which. Worse, if the expressions depend on each other in sequence, this may have to be broken up into var1 = expr1 with var2 as expr2: var3 = expr3(var1, var2) with var4 as expr4(var3, ...): .... body ... This seems to lose on every measure of clarity and maintainability relative to the single compound 'with' statement above. Finally, one may ask if an (effective) default implementation of a protocol is ever a good idea. "Hidden defaults" are a great way to trigger surprising behavior, after all. However, in this case I would argue that the proposed default behavior is sufficiently obvious that there is no risk. Someone seeing a compound 'with' statement of the above form would naturally assume that its consequence is (a) to set each varN to the corresponding exprN, and (b) to execute any scope-initializers tied to exprN. Likewise, someone would naturally assume that nothing at all happens at scope exit, which is exactly the behavior of __exit__ being 'return False'. In fact, this *increases* local code clarity, since the counter-case -- where the implementation of each defaults (effectively) to raising an AttributeError -- is nonobvious and so requires that "nonlocal knowledge" of the code to assemble with statements. *Specific implementation proposal: *Actually defining __enter__ and __exit__ methods for each object would be a lot of overhead for no good value. Instead, we can easily implement this as a change to the specified behavior of the 'with' statement, simply by changing the error-handling behavior in the SETUP_WITH <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f62…> and WITH_CLEANUP_START <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f62…> cases in ceval.c. If this does proceed to the PEP stage, I'll put together a changelist, but it's very straightforward. Null values for enter and exit are no longer errors; if enter is null, then instead of decrementing the refcount of mgr and calling enter, we leave the mgr refcount alone and push it onto the stack in place of the result. If exit is null, we simply push it onto the stack just like we would normally, and ignore it in WITH_CLEANUP_START.

10 20

Lists create
by 21ispinks＠parklandsacademy.co.uk June 21, 2019

June 21, 2019

How to list create.[09]

2 1

Comparison operator support (>= and <=) for type
by eminbugrasaral＠me.com June 20, 2019

June 20, 2019

Let's assume you have this model: ``` class Base: pass class A(Base): pass class B(Base): pass class C(A): pass ``` While we can do `A == B`, or `B == C` or `B == B`, I would expect to be able to compare like this as well: `A >= B (if B is subclass or itself of A)`, or `B <= C (if B is a subclass or itself of C)` Because, since == means equality check. With the same logic, a class wraps another class is actually greater than this class from its type.

23 37

Re: [Python-ideas] Operator as first class citizens -- like in scala -- or yet another new operator?
by Yanghao Hua June 19, 2019

June 19, 2019

On Mon, Jun 3, 2019 at 8:01 PM Andrew Barnert <abarnert(a)yahoo.com> wrote: > > I think it was a mistake to even mention assignment here (much less arbitrary operators). Most of the resistance you’re facing is because of that, and I think you’re missing a fundamental reason behind that resistance. > > Overloading variable assignment makes no sense in Python. In a language with lvalue variable semantics like Scala or C++, a variable is a thing with type, address, and identity, and values live in variables. So assignment affects a variable, and variables have types that can overload how that works. In a language with name-binding variable semantics like Python, it’s the value that has type, address, and identity, and it lives somewhere unspecified on the heap; a variable is nothing more than a name that can be bound to a value in a namespace. So assignment doesn’t affect variables, and, even if it did, they don’t have types that could overload how that works. Overloading assignment isn’t impossible just because, it’s impossible because it makes no sense. > > But the fact that in Scala, and Verilog, you send values to signals with an assignment-like operator on an lvalue for the signal doesn’t mean it actually is assignment. What you’re doing is an operation on the signal object itself, not on the signal variable. So neither = nor := would have been a good fit in the first place, and the fact that the variable is just a name rather than an lvalue isn’t a limitation at all, and there’s really no good reason not to have a “send a value to a signal” operator other than the (already very high) bar to new operators. Acknowledged. I do love the dynamic nature of Python, and as many of you could have experienced that you might be developing a type system sooner or later. And newer python with type hint makes it even easier to do that. That being said, I fully agree that we should not mess up with the = or := assignment operators, that was taken as an example. And I fully agree with you I probably should not even use the term "assignment operators", what I needed is an operator that does not collide with all existing number/matrix operators. > If you just explained the operation without reference to assignment, then the questions would be about why << isn’t a good enough fit, how widely the new operator would be used, whether there might be uses in other domains (Erlang-style channels?), etc. At that point, PEP 465 is a much better parallel. Because << is really one of the operator I need for signals ... it is even very straight forward to be translated into actual hardware implementation. I looked at every single operator I could have re-used ... the most promising one being @= operation, but then I realized I really want my signal matrix to use that operator for the sake of readability, it is really far more beautiful to read. > As a side note, if you haven’t already looked at other expression tree libraries like SymPy and SQLAnywhere and their parallels in languages like Scala and C++, it’s worth doing so. For many things (like most desirable uses of SymPy) the lack of variable assignment is no problem; for a few things (like implicit futures, or replacing namespaces with dataflow), it’s an insurmountable problem. I think your problem is more like the former. I do have experience with ORMs (Object-Relational-Mapper) like in Django and others, and just looked at SymPy and it is a great project. And indeed I agree with you that this problem is actually easily fixable ... I can instead use: signal.next = something, or use signal.assign(something), which is doable and many people are doing it, and you have many different ways of doing so. And then the choice of "signal.next/assign" becomes arbitrary ... e.g. you can use signal._next, signal._assign so on and so forth. But all those for me are actually implementation details and it is killing me that there is no way to hide it. Every single other major HDLs in the world uses an operator to do so, (vhdl/verilog <=, chisel :=). And this is not actually forcing a HDL way of doing things on Python, it actually represented a common category of problems in Python: there is no operator support to mean: drop a value on me (where "me" is already defined somewhere else, it has a type and identity). In HDL implementation alone I can see already two significant use cases: * signal assignment: signal <== signal << 2 (vs. signal.next = signal << 2 or signal.assign(signal << 2)), and it allows too something like: signal_a <== signal_b <== signal_c (vs signal_a.next = signal_b, signal_b.next = signal_c). * module integration/connection and pipe line construction: module_a <== module_b <== module_c (vs. module_a.connect(module_b); module_b.connect(module_c)) And yet again let me acknowledge: I understood all these problem can be solved by using a function call instead, but it is not as pretty as an operator. This is the whole point of PEP 465 ... make it looks good, as readability matters. So my question: does this looking good to python community? (You don't care much on hardware design is probably another thing), I think it looks much better, and around <== a well defined concept can be established in a particular domain/project. > Also, I can imagine a reason why << isn’t acceptable: it’s probably not that uncommon to need actual bitshift operators. But if you could build something that uses << anyway, and works perfectly in today’s Python for examples that don’t need shift, and then show that off—and show an example where not having bitshift is a problem—you’d be in an even better position. (See the PEP 465 section on numerical libraries that used * for matmul. They clearly work, and are useful for many things, but not having * for elementwise multiplication significantly limits a wide range of useful programs, which is why @ was needed for numpy.) Yep, that is my plan too. But instead of using <<, I have already modified CPython anyway to support <== and ==> operators and release a fully open sourced, powered-by-python HDL simulator with all major features you'd expect in hardware design (including dumping waveforms to VCD files, automatically building the hierarchy of module instance and signal instance names etc.). Code is already working but I want to make sure a few corner cases are well covered and it will be released alone with a PEP. I understand that after the assignment expression (:=) PEP572 people are already not happy with anything that could be slightly related to assignment overloading, I actually don't understand why people object PEP572 so hard, its practical use is clear and it makes code shorter and easier to read. Sometimes it seems the obsession of people in one idea makes them to even deny the existence of problems ... problems that having been solved well in other programming languages, and is well understood by majority of the programmers.

19 60

venv.EnvBuilder environmental variable hooks
by Caleb Donovick June 18, 2019

June 18, 2019

Currently EnvBuilder allows for a number of customizations of virtual environments, changing which python is used, automatically installing libraries, etc... However, it does not allow for any modifications of the activate script itself (unless one wants to rewrite it completely). However, a fairly common requirement for python packages is to edit some environmental variable e.g., DJANGO_SETTINGS_MODULE=<settings>. Which leaves developers with two options set the variables globally or manually edit the activate script. Granted the activate script has been fairly static for the past few years, so a patch file can be used after manually editing once but its not the most elegant solution. Further, there is no guarantee that activate script will be static going forward. In my mind there is a very simple solution, allow EnvBuilder to extend the set of variables which are set and restored on activate / deactivate (or more generally have activate / deactivate hooks). In the cpython implementation of venv there is a number of template parameters in the skeleton activate scripts which are filled by the EnvBuilder (__VENV_DIR__, __VENV_NAME__, ...). A simple solution would be to extend these with __VENV_ACTIVATE_EXTRAS__ and __VENV_DEACTIVATE_EXTRAS__ where by default ``` #Let env_vars: Dict[str, str] be the custom environmental variables __VENV_ACTIVATE_EXTRAS__ = ''.join(f''' _OLD_VIRTUAL_{key}="${key}" {key}="{value}" export {key} ''' for key,value in env_vars.items()) __VENV_DEACTIVATE_EXTRAS__ = ''.join(f''' if [ -n "${{_OLD_VIRTUAL_{key}:-}}" ] ; then {key}="${{_OLD_VIRTUAL_{key}:-}}" export {key} unset _OLD_VIRTUAL_{key} fi ''' for key in env_vars) ``` With __VENV_ACTIVATE_EXTRAS__ at https://github.com/python/cpython/blob/54cf2e0780ca137dd9abea5d3d974578ce0c… and __VENV_DEACTIVATE_EXTRAS__ at https://github.com/python/cpython/blob/54cf2e0780ca137dd9abea5d3d974578ce0c… Full activate / deactivate hooks could be achieved by setting __VENV_*_EXTRAS__ to be arbitrary shell commands. Caleb Donovick

2 1

unittest block
by James Lu June 18, 2019

June 18, 2019

# regular code unittest: # test code goes here This code would be run when the runtime -unittest flag is passed. It would function as test usages and example client code. The code within the block would remain in .pyc, unless the -O switch is passed. Write """""" or ## above a unittest to include it in generated documentation.

2 1

Fwd: unittest block
by Jonathan Fine June 18, 2019

June 18, 2019

Hi James Something similar can already be done, say via from somewhere import unittest @unittest def f(): # test code goes here As it's your idea, I suggest you create and put up an implementation on PyPi, and post again to this list when you have some real-world examples of the use of this feature. -- Jonathan

1 0