Python-ideas September 2018

python-ideas@python.org

108 participants
93 discussions

by Hans Polak

I have recently updated my code to use the more pythonic f-string instead of '{}'.format() Now, I want to start on the road to multilingual internationalization, and I run into two problems. The first problem is that f-strings do not combine with i18n. I have to revert to the '{}'.format() style. The second problem is that I need to translate strings on the fly. I propose to add a f''.language() method to the f-string format. Example: user = 'Pedro' f'Hi {user}' would be translated to 'Hola Pedro' if the locale were set to Spanish. f'Hi {user}'.language('es_ES') would be translated in the same way. To extract translatable strings from the source code, the source code could contain a 'HAS_LOCALES' flag (or something similar) at the top of the code. This way, the pygettext.py program would know that translatable f-strings are within the code. Rationale: More pythonic. At this moment, _('').format() is the way to go, so I would need to wrap another call around that: T(_(''), args, 'es_ES') <===This is an ugly hack. # Set the _() function to return the same string _ = lambda s: s es = gettext.translation('myapplication', languages=['es_ES']) def T(translatable_string, args_dictionary = None, language = None) if 'es_ES' == language: # Return translated, formatted string return es.gettext(translatable_string).format(args) # Default, return formatted string return translatable_string.format(args)

1 year, 8 months

Pattern Matching Syntax (reprise)

by Tobias Kohn

Hello Everyone, Please excuse my being late for properly responding to the last thread on "Pattern Matching Syntax" [1]. As Robert Roskam has already pointed out at the beginning of that thread, there has been much previous discussion about adding pattern matching to Python, and several proposals exist. It is therefore not my intention to propose yet another syntax choice for pattern matching, but more to share my experience with implementing it, in the hope of making a worthwhile contribution to the overall discussion. This summer, I basically ended up requiring pattern matching in Python for a research project I am working on. Some initial hacks have then grown into a library for pattern matching in Python [2]. On the one hand, my design is certainly heavily influence by Scala, with which I also work on a regular basis. On the other hand, I ran into various difficulties, challanges, and it has been particularly important to me to find a design that blends well with Python, and harnesses what Python already offers. I have written down my experience in the form of a discussion on several options concerning syntax [3], and implementation [4], respectively. As the articles have turned out longer than I originally intended, it might take up too much time for those who have little interest in this matter in the first place. However, considering that the subject of pattern matching has been coming up rather regularly, my experience might help to contribute something to the discussion. Let me provide a brief summary here: 1. PATTERN MATCHING IS NOT SWITCH/CASE -------------------------------------- When I am talking about pattern matching, my goal is to do a deep structural comparison, and extract information from an object. Consider, for instance, the problem of optimising the AST of a Python program, and eliminate patterns of the form `x + 0`, and `x - 0`. What pattern matching should be offering here is a kind of comparison along the lines of: `if node == BinOp(?, (Add() or Sub()), Num(0)): ...` Ideally, it should also return the value of what is marked by the question mark `?` here, and assign it to a variable `left`, say. The above comparison is often written as something like, e. g.: `case BinOp(left, Add()|Sub(), Num(0)): ...` This use of `case`, however, is not the same as a switch-statement. 2. ORTHOGONALITY ---------------- Getting the syntax and semantics of nested blocks right is hard. Every block/suite in Python allows any kind of statements to occur, which allows for things like nested functions definitions, or having more than just methods in a class. If we use a two-levelled block-structure, we run into the problem of finding good semantics for what the following means (note the variable `x` here): ``` match node: x = 0 case BinOp(left, Add(), Num(0)): ... x += 1 case BinOp(left, Mul(), Num(1)): ... ``` In the case of a "switch block", such additional statements like the `x=0`, and `x+=1` can become quite problematic. On the other hand, requiring all statements inside the block to be case-statements violates the orthogonality found otherwise in Python. I feel that this dilemma is one of the core issues why the syntax of switch statements, or pattern matching seems so exceptionally hard. In the end, it might therefore, indeed, make more sense to find a structure that is more in line with if/elif/else-chains. This would lead to a form of pattern matching with little support for switch-statement, though. 3. IMPLEMENTATION ----------------- For the implementation of pattern matching, my package compiles the patterns into context-manager-classes, adds these classes in the background to the code, and then uses `with`-statements to express the `case`-statement. If have found a neat way to make the execution of a `with`-statement's body conditional. Two things have been curcially important in the overall design: first, the "recompilation" cannot change the structure of the original code, or add any line. This way, all error messages, and tooling should work as expected; the invasive procedure should be as minimal as possible. Second, it is paramount that the semantics of how Python works is being preserved. Even though the actual matching of patterns is done in external classes, all names must be resolved "locally" where the original `case`-statement lives. Similarly, the variables defined by the pattern are to local variables, which are assigned if, and only if, the pattern actually matches. Should pattern matching ever be added to Python properly, then there will be no need to use `with`-statements and context managers, of course. But the implementation must make sure, nonetheless, that the usual semantics with name resolving of Python is being respected. 4. SYNTAX OF PATTERNS --------------------- The syntax of patterns themselves has been guided by two principles (again). First, if there is a way to already express the same thing in Python, use that. This applies, in particular, to sequence unpacking. Second, patterns specify a possible way how the object in question could have been created, or constructed in the first place. Hence, it is no accident that the pattern `BinOp(left, Add(), Num(0))` above looks almost like a constructor. Other implementation are happy to redefine the meanings of operators like `in`, or `is` (as in, e. g., `case x is str:`). While this might look very convenient at first, it will most probably lead to many subsequent bugs where people write `if x is str:` to mean the same thing (namely test the type of `x`). Even though expressing patterns requires reusing some operators in a new way, we have to extremely careful to choose widely, and minimise confusion, and surprises. Pattern matching could certainly make a great addition to Python, and various current implementations act as proof of concepts. However, choosing an appropriate syntax for pattern matching is hard, and we should work hard to make sure that any such addition feels natural in Python, even at the expense of having to write more, and being not as terse as other languages. I hope my thoughts in this matter can make a worthwhile constribution to the discussion. And I would like to emphasise once more, that my goal is not to propose a new syntax for pattern matching, but to report on my experience while implementing it. Kind regards, Tobias Kohn [1] https://groups.google.com/d/topic/python-ideas/nqW2_-kKrNg/discussion [2] https://github.com/Tobias-Kohn/pyPMatch [3] https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-pytho… [4] https://tobiaskohn.ch/index.php/2018/09/12/implementing-pattern-matching/

1 year, 8 months

Re: [Python-ideas] Moving to another forum system where

by James Lu

It would be nice if there was a guide on using Python-ideas and writing PEPs. It would make it less obscure.

1 year, 8 months

An experiment migrating bug tracker to GitLab

by Karthikeyan

PEP 581 proposes the migration of bug tracking to GitHub issues. I have done a project to collect all issues in https://bugs.python.org. I have parsed the HTML data and migrated the issues to GitLab along with labels for issues and comments which is pretty much similar to GitHub issues. I have just added a comment from my account preceded by the Author name. I have migrated around 140 issues out of the 30000 issues for a demonstration. I can see some immediate benefits as follows which also apply to GitHub : * GitHub and GitLab support markdown thus code snippets can have highlighting. * Labels can be filtered from the UI and are helpful for triaging. * GitLab allows subscription for a label so that developers can subscribe for new issues in a label. * There are categories like milestones and priority that can help in release management. * They provide API and thus we can build integrations around issues. Some notes : I haven't parsed code in comments to enable syntax highlighting since it's hard to parse. Repo and feedback welcome : https://gitlab.com/tirkarthi/python-bugs/issues Thanks

1 year, 8 months

Retire or reword the namesake of the Language

by Widom PsychoPath

Guten Tag, I am Jack and I am grateful to see the efficiency of scientific computing in Python. However, What deeply saddens me is that the namesake "Python" has unfortunately been derived from the title of the uncivilised British jester troupe "Monty Python". This is something that deeply infuriates me and is against the morals of my culture. Although humor is an integral aspect of the life of an Untermensch, I believe that Python, A language used as an interface to majority of Scientific computing software should be renamed to something more suitable. I hereby propose that the Language should be renamed to Cobra, after the brilliant military strategist Cobra Commander , who has been history's most efficient and brilliant strategist. I hope that my message is not mistaken for an attempt to humor. I am physically unable to experience the same. Please revert back to the mail with your thoughts and constructive criticism. Yours forever, Jack Daniels

1 year, 8 months

Deprecation utilities for the warnings module

by Ilya Kulakov

(Apologies if it's a duplicate. I originally posted to python-ideas(a)googlegroups.com) I've been recently working on an internal library written entirely in Python. The library while being under development was as actively used by other teams. The product was under pressure and hasty design decisions were made. Certain interfaces were exposed before scope of the problem was entirely clear. Didn't take long before the code started to beg for refactoring. But compatibility needed to be preserved. As a result a set of utilities, which I'd like to share, came to life. --- The warnings module is a robust foundation. Authors can warn what should be avoided and users can choose how to act. However in its current form it still requires utilities to be useful. E.g. it's non-trivial to properly replace a deprecated class. I'd like to propose an extension for the warnings module to address this problem. The extensions consists of 4 decorators: - @deprecated - @obsolete - @deprecated_arg - @obsolete_arg The @deprecated decorator marks an object to issue a warning if it's used: - Callable objects issue a warning upon a call - Property attributes issue a warning upon an access - Classes issue a warning upon instantiation and subclassing (directly or via a metaclass) The @obsolete decorator marks an object in the same way as @deprecated does but forwards usage to the replacement: - Callable object redirect the call - Property attribute redirect the access (get / set / del) - Class is replaced in a way that during both instantiation and subclassing replacement is used In case of classes extra care is taken to preserve validity of existing isinstance and issubclass checks. The @deprecated_arg and @obsolete_arg work with signatures of callable objects. Upon a call either a warning is issued or an argument mapping is performed. Please take a look at the implementation and a few examples: https://gist.github.com/Kentzo/53df97c7a54609d3febf5f8eb6b67118 Why I think it should be a part of stdlib: - Library authors are reluctant to add dependencies especially when it's for internal usage - Ease of use will reduce compatibility problems and ease migration burden since the soltion will be readily available - IDEs and static analyzers are more likely to support it --- Joshua Harlow shared a link to OpenStack's debtcollector: https://docs.openstack.org/debtcollector/latest/reference/index.html Certain aspects of my implementation are inspired by it. Please let me know what you think about the idea in general and implementation in particular. If that's something the community is interested in, I'm willing to work on it.

1 year, 8 months

Revert "RuntimeError: generator raised StopIteration"

by Danilo J. S. Bellini

Hi, The idea is simple: restore the "next" built-in and the "StopIteration" propagation behavior from Python 3.6. I'm using Python 3.7 for a while (as it's the default in Arch Linux), and there's one single backwards incompatible change from Python 3.6 that is breaking the code of some packages from PyPI. The reason is always the same: a "next" called inside a generator function was expected to propagate the StopIteration, but that no longer happens. As an example of something that had been made public (in PyPI), I've tried to run: from articlemeta.client import RestfulClient journals = list(RestfulClient().journals(collection="ecu")) This breaks with "RuntimeError: generator raised StopIteration". I already warned the maintainers of that project, it probably will be fixed. Another example (disclaimer: this time it's a package I've created): a simple "pip install dose" only works on Python<3.7, since some reStructuredText processing functions rely on the StopIteration propagation from calls to "next". I needed this package last saturday for a Coding Dojo, but I had to download the package from the repository and change some stuff before starting the Dojo (during the time reserved for it). I can fix this in the packages I'm maintaining by creating a new "fix_python37_next_runtime_error" decorator to restore the old behavior on every generator function that uses "next" in its body. But I can't do that to all packages from other people, and having to change/monkeypatch imported stuff in order to keep it working in this new Python version is getting annoying already. Perhaps adding a new kwarg to the "next" built-in to choose between a "propagate" default or a "error" alternative would avoid this. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)

1 year, 8 months

Re: [Python-ideas] Retire or reword the "Beautiful is better than ugly" Zen clause

by James Lu

It’s been almost a week since this “discussion” first started. Can we please stop this in the name of productive work on python-ideas? Frankly, you don’t need to reply just because you can point out something wrong with someone else’s argument. Post because it’s worthwhile to hear, not because you have something to say. This mindless and combative attitude is a big reason why Guido was motivated to suspend himself.

1 year, 8 months

Retire or reword the "Beautiful is better than ugly" Zen clause

by Samantha Quan

1 year, 8 months

Re: [Python-ideas] SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register

by Wes Turner

Are all current Python builds and C extensions vulnerable to Spectre variants {1, 2, *}? There are now multiple threads: "SEC: Spectre variant 2: GCC: -mindirect-branch=thunk -mindirect-branch-register" - https://mail.python.org/mm3/archives/list/distutils-sig@python.org/thread/4… - https://mail.python.org/pipermail/python-ideas/2018-September/053473.html - https://mail.python.org/pipermail/python-dev/2018-September/155199.html Original thread (that I forwarded to security@): "[Python-ideas] Executable space protection: NX bit," https://mail.python.org/pipermail/python-ideas/2018-September/053175.html > ~ Do trampolines / nested functions in C extensions switch off the NX bit? On Sunday, September 16, 2018, Nathaniel Smith <njs(a)pobox.com> wrote: > On Wed, Sep 12, 2018, 12:29 Joni Orponen <j.orponen(a)4teamwork.ch> wrote: > >> On Wed, Sep 12, 2018 at 8:48 PM Wes Turner <wes.turner(a)gmail.com> wrote: >> >>> Should C extensions that compile all add >>> `-mindirect-branch=thunk -mindirect-branch-register` [1] to mitigate the >>> risk of Spectre variant 2 (which does indeed affect user space applications >>> as well as kernels)? >>> >> >> Are those available on GCC <= 4.2.0 as per PEP 513? >> > > Pretty sure no manylinux1 compiler is ever going to get these mitigations. > > For manylinux2010 on x86-64, we can easily use a much newer compiler: RH > maintains a recent compiler, currently gcc 7.3, or if that doesn't work for > some reason then the conda folks have be apparently figured out how to > build the equivalent from gcc upstream releases. > Are there different CFLAGS and/or gcc compatibility flags in conda builds of Python and C extensions? Where are those set in conda builds? What's the best way to set CFLAGS in Python builds and C extensions? export CFLAGS="-mindirect-branch=thunk -mindirect-branch-register" ./configure make ? Why are we supposed to use an old version of GCC that doesn't have the retpoline patches that only mitigate Spectre variant 2? > > Unfortunately, the manylinux2010 infrastructure is not quite ready... I'm > pretty sure it needs some volunteers to push it to the finish line, though > unfortunately I haven't had enough time to keep track. > "PEP 571 -- The manylinux2010 Platform Tag" https://www.python.org/dev/peps/pep-0571/ "Tracking issue for manylinux2010 rollout" https://github.com/pypa/manylinux/issues/179 Are all current Python builds and C extensions vulnerable to Spectre variants {1, 2, *}? >

1 year, 8 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2018