Python-ideas May 2021

python-ideas@python.org

77 participants
41 discussions

dict.get_deep()
by Marco Sulla 24 May '21

24 May '21

I propose to add a get_deep(*args, default=_sentinel) method to dict. It can accept a single argument, that must be an iterable, or multiple arguments. The first element must be a key of the dict. If there's not a second element, the value is returned. If it's present, it tries to use it as an argument for the eventual __getitem__() of the value object, and so on. In this process, if a KeyError, an IndexError or a TypeError is raised, if default is set its value is returned, otherwise the exception will be raised. Example: d = {1: [42]} d.get_deep(1, 0) # 42 d.get_deep(range(3), default=1981) # 1981 d.get_deep((1, 1)) # IndexError: list index out of range

5 7

Re: New Idea: A safe keyword to execute the following statement in a failsafe mode.
by Shivam Saini 23 May '21

23 May '21

That's what I think it should be for. I know safe open(...) isn't a really good example for this, but I had just used that for demonstration purposes. Instead what I am saying is that sometimes we just don't care even if an statement raises exception. Like the first example in which I am sending an log, which isn't important. And in any case, the original try except keywords will still be there. Also I read the similar idea you mentioned, and I guess that is also a good way to implement it. We can replace *safe telegram_log("...")* with one liner: *telegram_log("...") except: None* After all, python is known for one liners and this would be an another great one liner if implemented.

5 4

New Idea: A safe keyword to execute the following statement in a failsafe mode.
by Shivam Saini 23 May '21

23 May '21

Sometimes, we need to execute a statement, which might throw an exception and we want to return None if it causes an exception. In short, we want to run that command as failsafe mode, so that it doesn't cause abnormal termination in case of any exception. *Approach till now:* def log_to_telegram(text): ''' Suppose you want to send some log to some telegram channel using telegram bot api. It might raise Network Error, or Invalid Bot token error. ''' send_message(CHANNEL, BOT_TOKEN, TEXT) # Suppose send_message is a function wrapper for telegram bot api. def some_func(): try: log_to_telegram('This is a not important message. It's okay even if it isn\'t delivered.') except: pass # Next work *Suggested approach:* # consider the same log_to_telegram function as defined in previous example def some_func(): safe log_to_telegram('This is a not important message. It's okay even if it isn\'t delivered.') # safe keyword ensures that it raises no exception and the process isn't terminated even if log_to_telegram function raises an exception. *Other examples:* *1. By default, it will return None.* * Example: * try: file = open('some_file') except: file = None * Can be rewritten as:* file = safe open('some_file') *2. We can provide a return value explicitly which should be returned in case of some exception.* * Example: * try: division = a/b except: division = 0 * Can be rewritten as:* division = safe(0) a/b *3. We can set what exceptions we want to be 'safed'.* * Example: * try: element = some_list[index]/divisor except ZeroDivisionError: element = 0 # It will make element zero if divisor is zero, but it will not except IndexError in case index is out of range for some_list * Can be rewritten as:* element = safe(0, ZeroDivisionError) some_list[index]/divisor *I think this keyword should be there in python, it will make code short and readable. *

6 8

Fractions vs. floats - let's have the cake and eat it
by Martin Teichmann 22 May '21

22 May '21

Hi List, some days ago I posted about my idea let integer division result in fractions, not floats. The upshot of the discussion was that it is a pity that we do not have literals for fractions, but that there is nothing to do about it, as all proposed syntaxes were a bit ugly. But why do we need to have different syntax for both, isn't it possible to simply do both at the same time? The interesting answer is: yes. So my proposal is: number literals (which are not integers) are both fractions and floats a the same time - only when we start calculating with them, the receiving function will pick whatever it prefers. For backwards compatiblity the default is float - but you may write code that looks at the fraction as well. I prototyped that here: https://github.com/tecki/cpython/tree/ratiofloat The idea is the following: when the parser (technically, the AST optimizer) creates the objects that represents the literals, let it add some bread crumbs as to where those data are coming from. Currently, 1/2 just means the float 0.5. Instead, let it be the object of a new class, I dubbed it ratiofloat, which inherits from float, but has the exact value added to it as well. ratiofloat just adds two C ints to the float class, making it a bit bigger. But as I said, only for literals, calculated floats still have the same size as before. To give an example (this is not fake, but from the prototype): >>> 2/5 0.4 >>> (2/5).denominator 5 >>> isinstance(2/5, float) True >>> type(2/5) <class 'ratiofloat'> Note that this is only done at compile time, no such behavior is done at run time, everything just behaves like normal floats: >>> two=2 >>> five=5 >>> (two/five) 0.4 >>> (two/five).numerator Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'float' object has no attribute 'numerator' I have tested this, and all tests pass except those where it is explicitly tested whether a value is a float, not one of its subclasses. I ran that also through numpy and sympy, and both behave mostly fine, except again where the tests are testing whether something is actually a float. All of this does not only work for integers, but also for float literals: >>> a=1/3 + 0.1 >>> a 0.43333333333333335 >>> a.numerator 13 >>> a.denominator 30 All this is only interesting once you teach some classes about it. To give an example here: >>> from decimal import Decimal >>> Decimal(1/3) Decimal('0.3333333333333333333333333333') >>> Decimal(0.1) Decimal('0.1') >>> from fractions import Fraction >>> Fraction(1/3) Fraction(1, 3) >>> Fraction(0.1) Fraction(1, 10) I also tries to teach sympy about this. While I succeeded in general, there were many tests that failed, and that for an interesting reason: the sympy tests seem to assume that if you use a float, you want to tell sympy to calculate numerically. So for the sympy tests, 0.1*x and x/10 are something completely different. IMHO this is actually an abuse of features: 0.1 simply is the same as one-tenth, and code should at least try to treat it the same, even if it fails at that. Other than that, sympy works fine (after I taught it): >>> from sympy import symbols >>> x = symbols("x") >>> 1.5*x 3*x/2 >>> x**(0.5) sqrt(x) I think this is now all good enough to be wrapped in a PEP, Chris, can you guide me through the bureaucracy? How would we go forward about this? The good news is that all that I described happens at compile time, so we can use a good ole' "from __future__ import" approach. So my suggestion is: implement it for 3.11, but activate it only with a future import or an interpreter option for the command line. This gives libraries time to adopt to this new style. For 3.12, make this option the default for the command line. So we can tell people to just switch that option off if it doesn't work. And in some far, glorious future, when everybody has a from __future__ line in all of their files, we can make it a default everywhere. Cheers Martin

15 27

Bring back PEP 501
by Nick Humrich 21 May '21

21 May '21

PEP 501 was deferred because more learning and time was wanted after introducing f-strings. Now that it has been 5 years, I wonder what the possibilities of revisiting PEP 501 are. I recently had the experience of using javascript "tagged template literals" and was able to build a SQL string parser that is impossible to have SQL injection with. This is done by having the database connection object only accept a certain type of object, and all sql tagged template literals become that object. Because variables are lazy evaluated, the template function can turn all dynamic inputs into parameters in a SQL query. It is impossible for a dev to accidentally add a user imputed string as a literal. PEP 501 already mentions how templates (i-strings?) can solve injection. This is a very incredible goal. Injection has been the #1 vulnerability on OWASP for over 10 years, and has been in the top 5 the entire time OWASP has existed (almost 20 years now). We have an opportunity to completely remove injection attacks. I won't go through and mention other possibilities of i-strings because the PEP already does an amazing job of doing that. All recent (within the last two years) discussions of PEP 501 have proposed PEP 501 as a solution to various idea suggested, but then no further discussion on 501 happened. At least, not that I am aware of. If any further discussion of 501 has happened, I would be happy to read up and try to address any concerns. Some recent discussions were 501 is mentioned: https://mail.python.org/archives/list/python-ideas@python.org/thread/T3B56I… https://mail.python.org/archives/list/python-ideas@python.org/thread/DX2ILP… https://mail.python.org/archives/list/python-ideas@python.org/thread/3Z2YTI… https://mail.python.org/archives/list/python-ideas@python.org/thread/DKW6Z6… https://mail.python.org/archives/list/python-ideas@python.org/thread/ASPNKH… I would be willing to do any work required to get this PEP improved, but am very new to the PEP process and is what is needed. What is needed to revisit PEP 501, and what can I do to help?

8 12

Python 3.10: ranges in pattern matching
by Valentin Dymchishin 18 May '21

18 May '21

Hi everyone! I've just read about pattern matching in Python 3.10, and it sounds really nice. I've also found out that Rust supports ranges in pattern matching. https://doc.rust-lang.org/book/ch18-03-pattern-syntax.html#matching-ranges-… fn main() { let x = 5; match x { 1..=5 => println!("one through five"), _ => println!("something else"), } } Can we have something similar please? I think this would be a nice addition to Python's pattern matching.

5 9

Add a mechanism so that multiple exceptions can be caught using `except E1, E2, E3:`
by Thomas Grainger 18 May '21

18 May '21

now that python2.7 is EOL, it might be worth resurrecting this syntax as discussed in https://www.python.org/dev/peps/pep-3100/#id13 eg, python 3.11 could support ``` try: ... except (E1, E2, E3) as e: ... ``` as equivalent to ``` try: ... except E1, E2, E3 as e: ... ``` see also https://mail.python.org/archives/list/python-dev@python.org/thread/HSN6ESRB…

10 15

Improved multi-tasking performance through deterministic GIL hand-off
by Sophist 11 May '21

11 May '21

I don't know how many people will remember some work that David Beazley did about a decade ago on how the GIL impacts multithreading performance - essentially he instrumented the Python interpreter to log how multiple threads competed for the GIL and gave several presentations over the space of 2-3 years. A couple of years ago I reached out to him with an idea on how to significantly improve the way that Python handles multi-threading hand-off of the GIL, but unfortunately he was not interested in pursuing this further. I am raising it here in the hope that someone else would be interested in implementing this. In essence my idea is to stop Python handing off the GIL through a competition between threads that are ready to run, and instead for Python to implement a scheduler for the GIL which decides which thread should get the GIL next and directly hands it over. Background ======== Here are the links to David Beazley's presentations: 2009: Inside the Python GIL - https://www.youtube.com/watch?v=ph374fJqFPE 2010: Understanding the Python GIL - https://speakerdeck.com/dabeaz/understanding-the-python-gil https://www.youtube.com/watch?v=Obt-vMVdM8s 2011: Embracing the Global Interpreter Lock - https://speakerdeck.com/dabeaz/embracing-the-global-interpreter-lock https://www.youtube.com/watch?v=fwzPF2JLoeU 2011: In Search of the Perfect Global Interpreter Lock - https://speakerdeck.com/dabeaz/in-search-of-the-perfect-global-interpreter-… https://www.youtube.com/watch?v=5jbG7UKT1l4 The Problem ======== A currently executing thread can give up the GIL (essentially) for two reasons: A) It has started a blocking operation (like a reading data from disk) and is no longer runnable; or B) because it is using a lot of CPU and has reached a Python CPU timeout (nothing to do with the O/S scheduler time-slice) and is still runnable. When the currently executing thread wants to give up the GIL, the current competitive approach essentially sends a signal to all runnable threads that the GIL is now available and the first thread to grab it gets to execute with the GIL. The problem with this (at least as I understand it) is that all threads except the thread that has just given up the GIL are not actually running on a CPU and need the O/S Scheduler to dispatch them before they can run, but the current thread is already running - so if the currently executing thread is is still runnable i.e. because it has reached the Python CPU timeout, then it is highly likely to be the first to grab the GIL back again. This leads to CPU-heavy threads dominating the CPU usage, and the I/O threads being starved out - and this is the opposite of what O/S Scheduling Theory says is optimum - which is to run the I/O threads first to keep the data flowing and everything responsive, typically using low levels of CPU, and let the CPU heavy threads mop up the remaining CPU. But even if the currently executing CPU heavy thread doesn't grab the GIL again, there is no guarantee that the most appropriate of the other threads will grab it instead. It should also be noted that this competitive approach has a lot of overhead - because every thread wakes up to try to compete for the GIL. Proposed Solution ============ My proposed solution is for Python to implement its own (light-weight) scheduling system for the GIL - maintaining a list of threads that are runnable and whether they are I/O bound (gave up the GIL because they became non-runnable due to blocking I/O or waiting for a signal) or CPU heavy (gave up the GIL because of reaching the Python timeout). Ideally we would also add functionality to Python to allow you to specify a thread priority (ideally adjusting the O/S thread priority to match) and we would also retain this priority as part of the GIL scheduling. As far as I can see this proposal is complementary to the current work on sub-interpreters - because there is a big base of existing code which would immediately benefit from these changes without any coding changes. The solution I have in mind is this: a. A scheduling queue is created with its own mutex. The reason for having a separate mutex is that threads can add themselves to the queue without needing to obtain the GIL. b. The queue consists of a tree with the following branches: i) A list of thread priorities, decreasing order ii) Two subqueues: I/O bound, CPU bound iii) Each has a FIFO list of runnable threads When a thread is created, it is assumed to be I/O bound and is added to the back of the appropriate queue. When the GIL is given up, if the currently executing thread is still runnable (reached the Python timeout) it adds itself to the back of the appropriate queue. It then takes the first runnable thread in the queue (highest priority, I/O bound if non empty, otherwise CPU bound) and signals that thread to take the GIL and start executing. As a thread becomes runnable (because it gets a signal it is waiting for or because a blocking I/O completes), it adds itself to the back of the appropriate queue. This is a deterministic solution with low overhead that should ensure that the appropriate thread gets the GIL. And I believe that it would be compatible with the intended sub-interpreter implementation. A couple of caveats: A. I think that there may be some other rare cases where a thread that has the GIL is forced to give it up - and the above algorithm might needs some tweaks to accommodate that. B. Although this should be deterministic, I can imagine a possibility that the signalled thread does not wake up and take the GIL (due to bugs or other reasons not imagined) - and it might be worthwhile having some watchdog timeout mechanism to be started immediately before sending the GIL transfer signal and terminated immediately after receiving the GIL transfer signal, which will be triggered if for any reason the signalled thread does not wake up and take the GIL within a short period. Because the code for handing off the GIL is quite localised, I believe that the implementation of this might be reasonably straight forward.

3 3

Barrier Object in asyncio lib
by Yves Duprat 10 May '21

10 May '21

Hi,the list, I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help. Yves

6 13

OT: Accessibility: Jana Schroeder's Holman Prize Application
by Jonathan Fine 10 May '21

10 May '21

Perhaps Off Topic, but for a good cause. This year I met Jana Scroeder, a blind person forced to change jobs as part of the social cost of Covid. Her outsider experience of computer coding training became a wish to make things better. She has applied for a Holman Prize ($25,000 over a year) to fund this. She's also set up a survey to reach and know better those with similar wishes. One simple way to help open to many is to volunteer to be a sighted helper for a code and math variant of BeMyEyes.org. I encourage you to listen to Jana's pitch for a Holman prize, and if you want to help complete the survey (whether you're blind or sighted, code or math, young or old). I've learnt a lot about accessibility from Jana. *Jana Schroeder's Holman pitch* (90 seconds): https://www.youtube.com/watch?v=3ywl5d162vU *Jana Schroeder's survey* (15 minutes): https://tinyurl.com/blindcodersurvey Finally, *The Holman Prize*: https://holman.lighthouse-sf.org/ best regards Jonathan

1 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas May 2021