Mailman 3 June 2020 - Python-ideas

json library: non-standards-compliant by default, and what to do about it.
by Serge Bazanski 27 Jun '20

27 Jun '20

Hi list, as you might be aware, the json library is non standards-compliant [1] by default: when fed {NaN, Inf, -Inf} floating point values in the serialization input, it will output {Nan, Inf, -Inf} literals in serialized form, unless the keyward argument allow_nan is explicitly set to False - and by default it's set to True. Therefore, the current state of affairs is that a simple `import json; json.dumps(float("NaN"))` is non-standards compliant. There is a symmetrical issue with deserialization - json.load and friends will by default happily treat NaN/Inf/-Inf values as valid and convert them to Python float values. However, I'd like to focus on the emitting/encoding/serializing part, as I think that's the one that's more problematic in practice. A parser that is by default more lenient than the standard dictates is generally less of an issue than when a serializer is. A quick Google search for 'allow_nan python github' brings up many, many examples of people being bitten by either the fact that their code was not emitting standards-compliant JSON, or by the fact that they have to deal with an external Python system that emits NaN/Inf/-Inf [2]. From my experience it's not uncommon to find existing, mature Python codebases that exhibit this issue, and I'd like future Python users to notice early that their code is likely emitting non-compliant JSON values, and take appropriate actions. Is there a general consensus on this state of affairs, or some discussion about this that I've missed? As far as I can tell, this behavior has existed in Python (and simplejson) since at least 2005. What does the list think of the following two ideas: 1) Document this lack of standards compliance better - eg., introduce a big emphasized box on top of the Python manual for the json library that mentions the importance of the allow_nan flag. Or, 2) Fix the current behavior of JSON encoding in Python with regards to NaN/Inf/-Inf values - keeping in mind that a simple flip of allow_nan to False by default would unfortunately cause obvious breakage to existing Python codebases (as attempts to emit out-of- range float values result in a ValueError being thrown). A discussion naturally arises on whether increased standards compliance is wroth the breakage of backwards compatibility, or whether there is a way to implement this change in a less drastic way (transition period with warning? defaulting to converting invalid values to null? something else?). I've been sufficiently annoyed by this behavior that I'm willing to drive either of these proposals to further discussion and possibly implementation, but I wanted to first gauge the consensus on this, and make sure there wasn't a previous discussion on this that I've missed. Kind regards, Serge Bazanski [1] - By standards-compliant, I refer to compliance with RFC8259 - but I couldn't find _any_ JSON specification that would allow for NaN/Inf/-Inf. [2] - I've personally had a similar experience not so long ago, which forced me to implement https://github.com/q3k/cursedjson - so I'm generally somewhat biased with regards to this.

10 22

Re: giving set.add a return value
by Chris Angelico 27 Jun '20

27 Jun '20

On Sat, Jun 27, 2020 at 10:06 PM Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > > > > On 27/06/2020 05:33, Chris Angelico wrote: > > On Sat, Jun 27, 2020 at 2:29 PM Steven D'Aprano <steve(a)pearwood.info> wrote: > >> Seriously, I genuinely thought that the existing behaviour was the > >> opposite and that `add` unconditionally added the element. "Last seen > >> wins". If I was designing sets, that's probably how I would design it. > >> After all, it's called *add*, not *add if not already there*. I was so > >> sure that this was the current behaviour that I didn't bother to check > >> it before posting, which is rare for me. > >> > >> So I think this counts as the principle of maximal surprise :-) > > Well, that's not how Python's dicts OR sets work, so that would be > > inconsistent. But feel free to do it your own way in your own > > language. > > > > ChrisA > > _______________________________________________ > > > I don't know what you mean Chris, ISTM that *is* the way dicts work: > > >>> d = { 'one' : 1 } > >>> d['one'] = 1.0 > >>> d > {'one': 1.0} > >>> > >>> d = { 'one' : 1 } > >>> d.update( {'one' : 1.0} ) > >>> d > {'one': 1.0} > > Have I misunderstood you? Sets behave like dict keys, not their values. Obviously the last *value* will win (because that's how assignment works), but with the keys, the one already in the dictionary will be retained. Switching your example around: >>> d = {1: 'one'} >>> d[1.0] = 'one' >>> d {1: 'one'} >>> d = {1: 'one'} >>> d.update({1.0: 'one'}) >>> d {1: 'one'} >>> dict(((1, 'one'), (1.0, 'one'))) {1: 'one'} >>> {1: 'one', 1.0: 'one'} {1: 'one'} When sets were introduced, I believe this behaviour of dicts was deliberately mimicked for sets, although those who've been Pythonning for longer than I have would be better able to say. ChrisA

1 0

Allow signal suppression
by Yonatan Zunger 27 Jun '20

27 Jun '20

Hey everyone, I've been developing code which (alas) needs to operate in a runtime environment which is quite *enthusiastic* about sending SIGTERMs and the like, and where there are critical short sections of code that, if interrupted, are very hard to resume without some user-visible anomaly happening. This means getting to know the signal handling logic far too well. In particular, it means that preventing signals in a "dangerous window" is very difficult in the current language: while you can change the signal handlers to "suppressing" handlers and then restore them from the main thread, if you have potentially critical regions running in any non-main thread, there's no good way for them to tell the main thread to change the handlers... except by sending the main thread a signal. That requires the "suppressing" handler to have nontrivial logic in it, *but* Python signal handlers are reentrant: signals are not suppressed *during a signal handler*, and so hilarity ensues. Digging through all of this, though, there seems to be one interesting thing that could be done in CPython in particular, and so I have a possibly-crazy proposal for a runtime-specific extension. *Proposal: Add (as an optional part of the spec, runtimes may choose to implement if they wish) sys.suppress_signals(bool).* Calling this function with a value of True would defer all signal handling until it was called again with a value of False. The reason this is potentially insane is that suppressing signals does all sorts of things: e.g., it prevents keyboard interrupts, blocks ENOPIPE errors if you try to write to a pipe with a terminated peer, stops child processes from reporting their exit to the parent, screws with profiling timers, and so on. This would be an a proper footgun if misused, the sort of thing you should only activate if you actually understand POSIX signals in depth. The implementation in CPython would be surprisingly simple: simply store the boolean value in an atomic int, and add a second check at the start of _PyErr_CheckSignalsTstate <https://github.com/python/cpython/blob/master/Modules/signalmodule.c#L1693>, prior to clearing is_tripped. How insane does this idea sound to people? Yonatan -- Yonatan Zunger Distinguished Engineer and Chief Ethics Officer He / Him zunger(a)humu.com 100 View St, Suite 101 Mountain View, CA 94041 Humu.com <https://www.humu.com> · LinkedIn <https://www.linkedin.com/company/humuhq> · Twitter <https://twitter.com/humuinc>

4 6

approximate equality operator ("PEP 485 follow-up")
by Sebastian M. Ernst 23 Jun '20

23 Jun '20

Hi all, after just having typed tons of `math.isclose` (see PEP 485 [1]) and `numpy.isclose` calls (while basically using their default tolerances most of the time), I was wondering whether it makes sense to add a matching operator. "Strict" equality check as usual: `a == b` "Approximate" equality check: -> `a ?= b` (similar to `!=`) -> `a ~= b` (my preference) -> `a === b` (just to confuse the JS crowd) A corresponding method could for instance be named `__ic__` (Is Close), `__ae__` (Approximate Equal) or `__ce__` (Close Equal). It's such a common problem when dealing with floating point numbers or similar data types (arrays of floats, float-based geometries etc). An operator of this kind would make many people's lives much easier, I bet. I have overloaded the modulo operator in some test code and it actually helps a lot to make some logic more readable. Best regards, Sebastian 1: https://www.python.org/dev/peps/pep-0485/

16 29

Please consider adding numbers.Boolean
by Neil Girdhar 23 Jun '20

23 Jun '20

I'm just curious if there was a reason why Boolean was omitted from the numeric tower in the numbers library? It seems that builtins.bool and numpy.bool_ would both be elements of Boolean, and Boolean itself would be Integral? Best, Neil

7 7

asyncio: return from multiple coroutines
by Pablo Alcain 23 Jun '20

23 Jun '20

Hey everyone. I have been looking into asyncio lately, and even though I have had my fair share of work, I still have some of it very shaky, so first of all forgive me if what I am saying here is already implemented and I totally missed it (so far, it looks *really* likely). Basically this is the situation: I have an application that listens on two websockets through the async library https://websockets.readthedocs.io/ and I have to perform the same function on the result, no matter where the message came from. I understand that I can create two coroutines that call the same function, but it would be much cleaner (because of implementation issues) if I can simply create a coroutine that yields the result of whichever connection arrives first. I have implemented a rather cumbersome solution with async Queues (as I would do in threading), in which each connection puts its message in a queue and an adapter class awaits the first element of the queue on "receive". Here I attach a pastebin with the minimal working example: https://pastebin.com/BzaxRbtF However, it looks like a more async-friendly solution should exist, something like ``` async def _start(): msg1 = recv("Messager 1", sleep_time=1) msg2 = recv("Messager 2", sleep_time=2) while True: result = await asyncio.on_first_return(msg1, msg2) print(result) ``` (I understand that this implementation would not work because the event loop doesn't know that it is "the same task repeated", but it's just to tell you the general idea) Again, it's quite likely I am not seeing something obvious, but I didn't know where else to ask. Thank you very much, Pablo

2 1

Proposal to introduce pattern matching syntax
by nate lust 22 Jun '20

22 Jun '20

Hello, I have been working on an idea that would introduce pattern matching syntax to python. I now have this syntax implemented in cpython, and feel this is the right time to gather further input. The repository and branch can be found at https://github.com/natelust/cpython/tree/match_syntax. The new syntax would clean up readability, ease things like visitor pattern style programming, localize matching behavior to a class, and support better signaling amongst other things. This is the tl;dr, I will get into a longer discussion below, but first I want to introduce the syntax and how it works with the following simple example. result = some_function_call() try match result: # try match some_function_call(): is also supported as Dog: print("is a dog") as Cat(lives): print(f"is a cat with {lives} lives") as tuple(result1, result2): print(f"got two results {result1} and {result2}") else: print("unknown result") The statement begins with a new compound keyword "try match" . This is treated as one logical block, the word match is not being turned into a keyword. There are no backwards compatibility issues as previously no symbols were allowed between try and :. The try match compound keyword was chosen to make it clearer to users that this is distinct from a try block, provide a hint on what the block is doing, and follow the Python tradition of being sensible when spoken out loud to an English speaker. This keyword is followed by an expression that is to be matched, called the match target. What follows is one or more match blocks. A match block is started with the keyword ‘as’ followed by a type, and optionally parameters. Matching begins by calling a __match__ (class)method on the type, with the match target as a parameter. The match method must return an object that can be evaluated as a bool. If the return value is True, the code block in this match branch is executed, and execution is passed to whatever comes after the match syntax. If __match__ returns False, execution is passed to the next match branch for testing. If a match branch contains a group of parameters, they are used in the matching process as well. If __match__ returns True, then the match target will be tested for the presence of a __unpack__ method. If there is no such method, the match target is tried as a sequence. If both of these fail, execution moves on. If there is a __unpack__ method, it is called and is expected to return a sequence. The length of the sequence (either the result of __unpack__, or the match target itself) is compared to the number of supplied arguments. If they match the sequence is unpacked into variables defined by the arguments and the match is considered a success and the body is executed. If the length of the sequence does not match the number of arguments, the match branch fails and execution continues. This is useful for differentiating tuples of different lengths, or objects that unpack differently depending on state. If all the match blocks are tested and fail, the try match statement will check for the presence of an else clause. If this is present the body is executed. This serves as a default execution block. What is the __match__ method and how does it determine if a match target is a match? This change introduces __match__ as a new default method on ‘object’. The default implementation first checks the match target with the ‘is’ operator against the object containing the __match__ method. If that is false, then it checks the match target using isinstnace. Objects are free to implement whatever __match__ method they want provided it matches the interface. This proposal also introduces __unpack__ as a new interface, but does not define any default implementation. This method should return a sequence. There is no specific form outside this definition, a class author is free to implement whatever representation they would like to use in a match statement. One use case for this method could be a class that stores the parameters passed to __init__ (or some other parameters) so someone could construct a new object such as Animal(*animal_instance.__unpack__()) [possibly with a builtin for calling unpack]. Another use, in keeping with the match syntax, is something like Stateful Enums. The behavior covered by try match can be emulated with some combination of compound and or nested if statements alongside type checking and parameter unpacking, so why introduce the new syntax? The benefits I see are: * Much easier to read and follow compared complicated if branching logic * the matching logic is now defined alongside the class in the __match__ method. This is in contrast to if statements where the logic is duplicated in each place. If it factored out into a function, the function may be unknown or unused in a cross package setting. Refactoring distributed logic to live next to an object is similar to the introduction of the format method on strings. * It introduces a well defined interface for programmers to depend on in contrast to new interfaces being built on a package by package base. For instance __unpack__ and __match__ behavior may be implemented on objects today with any variety of names, making discoverability difficult, and making if blocks more difficult for a user to parse. * A pattern often found in python is to use Exceptions for signaling conditions inside of execution which muddles the distinction between handling exceptional behavior and normal code execution. The try match syntax, along with object unpacking, provides a standardized way to signal information back to callers and standardized syntax for handling those signals. The try match syntax fits in well with the ethos of including more ways to use typing information within python. For instance something like typing.Union could have a corresponding __match__ method such that all of those types would be covariant under a single match block. In the opposite sense the try match syntax is also useful for taking a parameter defined as a union and dispatching to individual functions that transform the variable into a standardized type. Points I am unsure of: I am not sure about using “(parameters)” as part of the patch branch. In some ways it is very familiar to look at, but it also makes it look like these must be parameters used to construct the object, not parameters returned from __unpack__ (though they may be the same, they may not be). I have toyed with the the idea of something like “-> (a,b)” or uisng braces “{a,b}” and the serve the purpose of being different, but then that also makes them look different, so I dont really have a strong opinion formed on this part of the syntax. The implementation on my branch does work, but I am by no means an expert on all of python, there may be much better ways to do what I did. In particular I implemented some of the logic in the compiler, but it may be better served as an op code. The exact boundary of where best to put some logic is unclear. This implementation currently stops at the first matching block it comes to, not the best match out of all blocks. This is meant to make it easier to understand the “flow” of the statement, but it might be preferable to execute the block associated with the best match, though this would complicate the implementation a good deal. I am sure there is more that I have not considered with this proposal and I appreciate any feedback you choose to provide. Thank you for your time in reading this. -- Nate Lust, PhD. Astrophysics Dept. Princeton University

5 5

Permanent code objects (less memory, quicker load, less Unix Copy On Write)
by Jonathan Fine 22 Jun '20

22 Jun '20

Hi All Summary: Shared objects in Unix are a major influence. This proposal can be seen as a first step towards packaging pure Python modules as Unix shared objects. First, there's a high level overview. Then some technical stuff in Appendices. An object is transient if it can be garbage collected. An object is permanent if it will never be garbage collected. Every interpreted Python function has a code object (that contains instructions for the interpreter). Many of these code objects persist to the end of the program, and are used for little else than providing interpreter instructions. We show that extending Python, to provide and take advantage of permanent code objects, will bring some benefits. The cost is expected to be quite small. When a Python function is called, the interpreter increases the refcount of its code object. At the end of the function's execution, the interpreter decreases the refcount. (An example below shows this.) If Python were extended to take advantage of permanent code objects, then for example popular code objects could be loaded into memory in this way. This can reduce memory usage (by sharing immutable resources) and reduce startup time. In addition, a Unix forked process would have less need to do copy-on-write (see below). This is related to packaging pure Python modules as Unix shared objects. The core of implementing this change would be to provide if ... else ... branching, around the interpreter source code that changes the refcount of a code object. The interpreter itself will of course want direct access to the permanent code object. There is no harm in that. The cost is that unprivileged access to fn.__code__ will be slower, due to an additional indirection. However, as such commands are rarely executed in ordinary programs, the cost is expected to be small. It might be helpful, after checking the analysis and before coding, to do some simple timing tests and calculations to estimate the performance benefits and costs of making such a change. These would of course depend on the use case. I hope this helps. Jonathan APPENDICES =========== SOME IMPLEMENTATION DETAILS AND COMMENTS Because fn.__code__ must not return a permanent object, some sort of opaque proxy would be required. Because Python programs rarely inspect fn.__code__, in practice the cost of this additional indirection is likely to be small. As things are, the time spent changing the refcount of fn.__code__ is probably insignificant. The benefit is that permanent code objects are made immutable, and so can be stored safely in read-only memory (that can be shared across all processes and users). Code objects are special, in that they are only rarely looked at directly. Their main purpose is to be used by the interpreter. Python allows the user to replace fn.__code__ by a different code object. This is a rarely done dirty trick. The transient / permanent nature of fn.__code__ could be stored as a hidden field on the fn object. This would reduce the cost of the if ... else ... branching, as it amounts to caching the transient / permanent nature of fn.__code__. FORK AND COPY ON WRITE On Unix, the fork system call causes a process to make a child of itself. The parent and child share memory. To avoid confusion and errors, when either asks the system to write to shared memory, the system ensures that both parent and child have their own copy (of the page of memory that is being written to). This is an expensive operation. See: https://en.wikipedia.org/wiki/Copy-on-write INTERPRETER SESSION >>> from sys import getrefcount as grc # Identical functions with different code objects. >>> def f1(obj): return grc(obj) >>> def f2(obj): return grc(obj) >>> f1.__code__ is f2.__code__ False # Initial values. >>> grc(f1.__code__), grc(f2.__code__) (2, 2) # Calling f1 increases the refcount of f1.__code__. >>> f1(f1), f1(f2), f2(f1), f2(f2) (6, 4, 4, 6) # If fn is a generator function, then x = fn() will increase the # refcount of fn.__code__. >>> def f1(): yield True >>> grc(f1.__code__) 2 # Let's create and store 10 generators. >>> iterables = [f1() for i in range(10)] >>> grc(f1.__code__) 22 # Let's get one item from each. >>> [next(i) for i in iterables] [True, True, True, True, True, True, True, True, True, True] >>> grc(f1.__code__) 22 # Let's exhaust all the iterables. This reduces the refcount. >>> [next(i, False) for i in iterables] [False, False, False, False, False, False, False, False, False, False] >>> grc(f1.__code__) 12 # Nearly done. Now let go of the iterables. >>> del iterables >>> grc(f1.__code__) 2

13 38

"return if <expression>"
by artem6191 20 Jun '20

20 Jun '20

So yeah, we can "if <expression>: return", but why not? Examples: def some_function(a, b): return if a == b # That will return None return a * b print(some_function(2,2)) # None print(some_function(5,2)) # 10 def some_function_2(a, b): return if a > b or a == b return if a == 1 and b == 1 return a*b print(some_function_2(2,2)) # None print(some_function_2(1,1)) # None print(some_function_2(1,2)) # 2 print(some_function_2(5, 10)) # 50

11 13

Extend the range functionality to allow infinite ranges
by kempr＠edinburghacademy.org.uk 20 Jun '20

20 Jun '20

Proposal: range(start, ..., step) should function like itertools.count(start, step) Reason: It's pretty common to see people do things where they increment a count within a while True loop, and it would be nice to have something easily available for people to use to replace this. Usefulness: There is definitely a use for this, as the type of code mentioned above is common, and itertools.count is commonly used as well.

7 9