Mailman 3 July 2020 - Python-Dev

How to customize CPython to a minimal set
by Huang, Yang July 24, 2020

July 24, 2020

Hi, all There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support. So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs... Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that? Thank you.

8 10

Add mimetypes.mimesniff
by Dong-hee Na July 23, 2020

July 23, 2020

Hi, A few weeks ago, I suggested adding mimetypes.mimesniff on stdlib. (https://bugs.python.org/issue40841, https://github.com/python/cpython/pull/20720) Detecting MIME types well is an important feature and we already have mimetypes detection library but AFAIK it is not good enough. Note that some of our stdlib module already use the sniffing algorithm (e.g imghdr) The question is how exactly the mime sniffing should be done in terms of file formats and algorithm. Luckily, WHATWG published … [View More]

1 0

datetime module refactoring: folder vs parallel private modules
by Paul Ganssle July 21, 2020

July 21, 2020

Hi all, I was hoping to get some feedback on a proposed refactoring of the datetime module that should dramatically improve import performance. The datetime module is implemented more or less in full both in pure Python and in C; the way that this is currently achieved <https://github.com/python/cpython/blob/5a2bac7fe0e7a2b67fd57c7a9176a50feed0…> is that the pure Python implementation is defined in datetime.py, and the C implementation is in _datetime, and /after/ the full Python … [View More]version is defined, the C version is star-imported and thus any symbols defined in both versions are taken from the C version; if the C version is used, any private symbols used only in the pure Python implementation are manually deleted (see the end of the file <https://github.com/python/cpython/blob/5a2bac7fe0e7a2b67fd57c7a9176a50feed0…>). This adds a lot of unnecessary overhead, both to define a bunch of unused classes and functions and to import modules that are required for the pure Python implementation but not for the C implementation. In the issue he created about this <https://bugs.python.org/issue40799>, Victor Stinner demonstrated that moving the pure Python implementation to its own module would speed up the import of datetime by a factor of 4. I think that we should indeed move the pure Python implementation into its own module, despite the fact that this is almost guaranteed to break some people either relying on implementation details or doing something funky with the import system — I don't think it should break anyone relying on the guaranteed public interface. The issue at hand is that we have two options available for the refactoring: either move the pure Python implementation to its own private top-level module (single file) such as `_pydatetime`, or make `datetime` a folder with an `__init__.py` and move the pure Python implementation to `datetime._pydatetime` or something of that nature. The decimal and zoneinfo modules both have this same issue; the decimal module uses the first strategy with _pydecimal and decimal, the zoneinfo module uses a folder with a zoneinfo._zoneinfo submodule. Assuming we go forward with this, we need to decide which strategy to adopt for datetime. In favor of using a datetime/ folder, I'd say it's cleaner to put the pure Python implementation of datetime under the datetime namespace, and also it gives us more freedom to play with the module's structure in the future, since we could have lazily-imported sub-components, or we could implement some logic common to both implementations in Python and import it from a `datetime._common` module without requiring the C version to import the entire Python version, similar to the way zoneinfo has the zoneinfo._common <https://github.com/python/cpython/blob/master/Lib/zoneinfo/_common.py> module. The downside of the folder method is that it complicates the way datetime is imported — /especially/ if we add additional structure to the module, or add any logic into the __init__.py. Two single-file modules side-by-side, one imported by the other doesn't change anything about the nature of how the datetime module is imported, and is much less likely to break anything. Anyone have thoughts or strong preferences here? Anyone have use cases where one or the other approaches is likely to cause a bunch of undue hardship? I'd like to avoid moving this more than once. Best, Paul P.S. Victor's PR moving this code to _pydatetime <https://github.com/python/cpython/pull/20472> is currently done in such a way that the ability to backport changes from post-refactoring to pre-refactoring branches is preserved; I have not checked but I /think/ we should be able to do the same thing with the other strategy as well. [View Less]

4 3

Official joint communication from the Python Steering Council, PSF Board of Directors, and PSF Conduct WG
by Python Steering Council July 21, 2020

July 21, 2020

Recently, a series of discussions on this mailing list resulted in behavior that did not live up to the standards of the Python Community. The PSF Board of Directors, Python Steering Council, and the PSF Conduct Working Group would like to remind this community that our shared goal is to advance the Python language while simultaneously building a diverse, inclusive, and welcoming Python community. While the community has made progress in recent years, this discussion made it clear to many of us … [View More]

1 0

PEP 387: backwards compatibility policy
by Brett Cannon July 20, 2020

July 20, 2020

Started at a discussion at https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421 to try to finally resolve this PEP and our backwards compatibility policy.

2 2

[RELEASE] Python 3.8.5 released as a security hotfix. 3.9.0b5, the last beta before 3.9.0, also available
by Łukasz Langa July 20, 2020

July 20, 2020

This is a combined release of Python 3.8.5 and 3.9.0b5. Both are significant but for different reasons. Let’s dig in! Security content in 3.8.5 We decided to release 3.8.5 ahead of schedule due to a number of security-related fixes. All details can be found in the change log <https://docs.python.org/release/3.8.5/whatsnew/changelog.html#changelog> but the gist is: CVE-2019-20907 <https://bugs.python.org/issue39017>: infinite loop in a maliciously created .tar file BPO-41288 <… [View More]

1 0

In case you're wondering about 3.5.10rc1
by Larry Hastings July 20, 2020

July 20, 2020

It's held up on SSL. Ubuntu 20.04 changed some security parameter tunings, which breaks some uses of the SSL module, and approximately eight modules in the test suite. I assume this wasn't caught on the buildbots because they don't use Ubuntu--or at least not a build that fresh. SSL and the test suite are all completely happy on older Ubuntu releases. One could argue "it's fine, most people still using 3.5 are also using old OSes anyway". But I don't want to … [View More]

4 5

PEP 622 constant value syntax idea
by Mohammad Foroughi July 20, 2020

July 20, 2020

Hi, I had an idea regarding the pattern matching issue of comparing with a previous constant variable instead of assigning to a new local variable. I'm not sure if this has been brought up before but instead of using a symbol with the case statement what if we used a keyword. So for example, x = 2value = 2 match value: case global x: # value == x print("value matches") case x: # x = value print(f"value … [View More]

5 4

PEP 622 aspects
by Koos Zevenhoven July 19, 2020

July 19, 2020

PEP 622 authors, Overall, the PEP describes the proposal quite nicely. However, I do indeed have concerns and questions, some of which I describe in this email. (1) Class pattern that does isinstance and nothing else. If I understand the proposed semantics correctly, `Class()` is equivalent to checking `isinstance(obj, Class)`, also when `__match_args__` is not present. However, if a future match protocol is allowed to override this behavior to mean something else, for example `Class() == … [View More]obj`, then the plain isinstance checks won't work anymore! I do find `Class() == obj` to be a more intuitive and consistent meaning for `Class()` than plain `isinstance` is. Instead, the plain isinstance check would seem to be well described by a pattern like `Class(...)`. This would allow isinstance checks for any class, and there is even a workaround if you really want to refer to the Ellipsis object. This is also related to the following point. (2) The meaning of e.g. `Class(x=1, y=_)` versus `Class(x=1)` In the proposed semantics, cases like this are equivalent. I can see why that is desirable in many cases, although Class(x=1, ...)` would make it more clear. A possible improvement might be to add an optional element to `__match_args__` that separates optional arguments from required ones (although "optional" is not the same as "don't care"). (3) Check for exhaustiveness at runtime The PEP states: Check exhaustiveness at runtime > The question is what to do if no case clause has a matching pattern, and > there is no default case. An earlier version of the proposal specified that > the behavior in this case would be to throw an exception rather than > silently falling through. > The arguments back and forth were many, but in the end the EIBTI (Explicit > Is Better Than Implicit) argument won out: it's better to have the > programmer explicitly throw an exception if that is the behavior they want. > For cases such as sealed classes and enums, where the patterns are all > known to be members of a discrete set, static checkers can warn about > missing patterns. I don't understand this argument. Would it not be more explicit to have an `else` or `case _` branch to say what should happen in that case? (4) Check for exhaustiveness by static checkers About this, the PEP states: From a reliability perspective, experience shows that missing a case when > dealing with a set of possible data values leads to hard to debug issues, > thus forcing people to add safety asserts like this: > def get_first(data: Union[int, list[int]]) -> int: > if isinstance(data, list) and data: > return data[0] > elif isinstance(data, int): > return data > else: > assert False, "should never get here" > PEP 484 specifies that static type checkers should support exhaustiveness > in conditional checks with respect to enum values. PEP 586 later > generalized this requirement to literal types. > This PEP further generalizes this requirement to arbitrary patterns. This seems reasonable. However, why is the standard for static and runtime different? The corresponding runtime check is extremely easy and efficient to do, so if this is an error according to static analysis, why not make it an error at runtime too? —Koos [View Less]

2 2

Another take on PEP 622
by Tobias Kohn July 19, 2020

July 19, 2020

Hi Everyone, I feel there are still quite a few misconceptions around concerning PEP 622 and the new pattern matching feature it proposes. Please allow me therefore to take another attempt at explaining the ideas behind PEP 622 with a different approach. Bear in mind that I naturally cannot cover everything, though, and that some parts presented here are still slightly simplified. Let's start with perhaps the most crucial part: PEP 622 does **NOT** propose to introduce … [View More]a `switch`-statement! Indeed, pattern matching is much more closely related to concepts like regular expressions, sequence unpacking, visitor patterns, or function overloading. In particular, the patterns themselves share more similarities with formal parameters than with expressions. So, to start with I would like to invite you to think of PEP 622 not so much as proposing a new control structure in the sense of `if`-`elif`-`else`, `try`-`except`-`finally`, or even `switch`, but much rather as addressing the question: what would function overloading look like in Python? Of course, this does not fully capture pattern matching or do it justice, either, but it might offer a better basis to start with. 1. Function Overloading ----------------------- In a statically typed language, you might define a function `product` that either accepts two numbers or an iterable something like the following (I am sticking to Python syntax for simplicity): ``` def product(a: float, b: float) -> float: return a * b def product(it: Iterable[float]) -> float: result = 1.0 for x in it: result *= x return result ``` In Python, however, this needs to be done differently and the dispatch logic has to be put inside the function itself: ``` def product(*args): if len(args) == 2 and isinstance(args[0], float) and isinstance(args[1], float): return args[0] * args[1] elif len(args) == 1 and isinstance(args[0], Iterable): result = 1.0 for x in args[0]: result *= x return result ``` It is this use case to begin with that pattern matching is addressing by introducing a more declarative way. Each `case` represents one possibility of what the parameter list might look like: ``` def product(*args): match args: case (float(a), float(b)): return a * b case (Iterable(it),): result = 1.0 for x in it: result *= x return result ``` And if you squint a little, you might even see that these parameter lists could almost be written in C: `(float a, float b)`. In the context of more functional languages, you might also have seen some wilder stuff, where function overloading allows you to include literals. One of the most prevalent examples is the factorial function, usually defined a bit like this: ``` def fact(0): return 1 def fact(n): return n * fact(n-1) ``` Again, when doing the same thing in Python, we put the dispatch logic inside the function: ``` def fact(n): if n == 0: return 1 else: return n * fact(n - 1) ``` It is only natural to also allow pattern matching to express this use case: ``` def fact(arg): match arg: case 0: return 1 case n: return n * fact(n - 1) ``` And this here is where you probably start to see good old `switch` coming in. Indeed, pattern matching is powerful and versatile enough to act as a `switch` replacement in many cases. But given how we arrived here, you might start to understand that this is a happy accident and not by design of the structure itself. There is one big elephant in the room, of course: why do we need the `match`-line in the first place? If all we want is to somehow express function overloading, couldn't we just put the cases directly into the function body? In principle, yes, we could do that. But! As it turns out, there is something to be gained by a clear separation. When actually using pattern matching, you might discover that you do not always want to define a function. Sometimes, it is quite convenient being able to have a pattern like this, for instance: ``` def foo(*args): for arg in args: match arg: case ... ``` In all of these examples, it looks as if pattern matching is replacing an `if`-`elif`-`else` chain. However, this is not entirely accurate. What we really want to express is the function overloading in the first place. The `if`s are only needed to express the same idea in Python for the time being. In other words: the individual cases here express independent implementations and we leave it up to the Python interpreter to choose the right one. 2. Visitor Pattern and Dispatch ------------------------------- If you wanted to implement something like function overloading based on the type of the argument, you might use the _visitor pattern_ for that purpose. Thanks to Python's reflection capabilities, this is quite easy and simple to do. Take, e.g., this example of a function `draw()` that accepts a range of different geometric shapes. ``` def draw(geoShape): class Matcher: def case_Point(self, x, y): canvas.moveTo(x, y) canvas.dot(1) def case_Line(self, x1, y1, x2, y2): canvas.moveTo(x1, y1) canvas.lineTo(x2, y2) def case_Circle(self, cX, cY, radius): canvas.ellipse(cX-radius, cY-radius, cX+radius, cY+radius) def match(self, subject): n = 'case_' + subject.__class__.__name__ method = getattr(self, n) args = get_attribute_values(subject) method(*args) m = Matcher() m.match(geoShape) ``` Now let us look at the same thing written as pattern matching according to PEP 622: ``` def draw(geoShape): match geoShape: case Point(x, y): canvas.moveTo(x, y) canvas.dot(1) case Line(x1, y1, x2, y2): canvas.moveTo(x1, y1) canvas.lineTo(x2, y2) case Circle(cX, cY, radius): canvas.ellipse(cX-radius, cY-radius, cX+radius, cY+radius) ``` You might see that the design of pattern matching is not something entirely new and alien in Python, but well founded in ideas and principles already present. Moreover, observe, once again, how the variables in the patterns literally correspond to parameters in the visitor pattern above. In this instance, the `case` statements are quite close to function definitions in their entire syntactic structure (is it really that alien that `Point(x, y)` here does _not_ represent a function call?). This example also demonstrates something else I have brought up before: pattern matching does not strictly have the same semantics as an `if`-`elif`-`else` chain. It is thus not a simple control flow construct, as the emphasis is less on the control flow itself and more on the shape of the objects and data. There are, however, some clear limitations of the visitor pattern as implement in this way. If we go and define a new subclass of one of these geometric shapes---perhaps something like `HorizontalLine` as a specialisation of `Line`---, the visitor pattern will break down because there is no method `case_HorizontalLine`. Pattern matching, on the other hand, will still work as expected, because we are actually working in the space of classes with their full inheritance structure, and not only on the names of types. Mind, pattern matching is not a replacement for the visitor pattern in general, though. Even with pattern matching in place, there are valid use cases where you would certainly prefer the visitor pattern over pattern matching. This is only one example, after all, meant to illustrate the idea behind pattern matching and why it might be a good idea. 3. Shape and Structure ---------------------- Data is organised and comes in different shapes, structures, and representations. It is not always as easy as in the cases above, where the class of an object already contains all the information you need. As a first example, the visitor pattern as illustrated above could not easily differentiate between tuples of different lengths, say: you would have to encode the length of the tuple somehow in the method names, and then consider what happens if some objects does not have a length, say, and thus requires a different encoding scheme. In fact, the classic example of where pattern matching really starts to shine is a tree structure that we want to simplify (or otherwise process) according to specific rules. This might be an abstract syntax tree (AST) or a mathematical expression, but the principle is always the same: the decision which rule to apply depends on a number of different attributes, which are often scattered among nodes of different depths in your tree. The following illustrates this with a tree that represents mathematical expressions: ``` def simplify(node): match node: case BinOp(Num(left), '+', Num(right)): return Num(left + right) case BinOp(left, ('+'|'-'), Num(0)): return left case UnaryOp('-', UnaryOp('-', x)): return x case UnaryOp('-', Num(x)): return Num(-x) case anythingElse: return anythingElse ``` If you draw an actual diagram of the structure that each case imposes on the node, you will quickly find that pattern matching allows you to express a larger variety of structure than you could encode in method names. Not to mention if you had to write everything with `if`s and calls to `isinstance`, etc. It is possible, of course, but pattern matching introduces a very succinct way of expressing the structure declaratively and then let the compiler and interpreter figure out how to best compare the subject `node` against the different possibilities and extract the data you are actually interested in. By the way: in this case, the compiler might end up creating a structure of nested `if`s, perhaps something like: ``` def simplify(node): TOS = node try: if isinstance(TOS, BinOp) and isinstance(TOS.right, Num): if TOS.right.n == 0 and TOS.op in ('+','-'): return TOS.left elif TOS.op == '+' and isinstance(TOS.left, Num): return Num(TOS.left.n + TOS.right.n) elif isinstance(TOS, UnaryOp) and TOS.op == '-': value = TOS.operand if isinstance(value, UnaryOp) and value.op == '-': return value.operand elif isinstance(value, Num): return value.n return TOS: finally: del TOS ``` This illustrates, once again, that pattern matching is not so much specifying the control flow itself, but rather offering a set of alternatives. But there is neither the strict sequential structure of `if`-`elif`-`else`, not the jump tables of `switch`. And in case you wondered: the entire `match` block is actually a mini-scope, in which the subject `node` is defined. Kind regards, Tobias [View Less]

2 2