Mailman 3 May 2021 - Python-ideas

dict.get_deep()
by Marco Sulla May 24, 2021

May 24, 2021

I propose to add a get_deep(*args, default=_sentinel) method to dict. It can accept a single argument, that must be an iterable, or multiple arguments. The first element must be a key of the dict. If there's not a second element, the value is returned. If it's present, it tries to use it as an argument for the eventual __getitem__() of the value object, and so on. In this process, if a KeyError, an IndexError or a TypeError is raised, if default is set its value is returned, otherwise the … [View More]

5 7

Re: New Idea: A safe keyword to execute the following statement in a failsafe mode.
by Shivam Saini May 23, 2021

May 23, 2021

That's what I think it should be for. I know safe open(...) isn't a really good example for this, but I had just used that for demonstration purposes. Instead what I am saying is that sometimes we just don't care even if an statement raises exception. Like the first example in which I am sending an log, which isn't important. And in any case, the original try except keywords will still be there. Also I read the similar idea you mentioned, and I guess that is also a good way to implement it. We … [View More]

5 4

New Idea: A safe keyword to execute the following statement in a failsafe mode.
by Shivam Saini May 23, 2021

May 23, 2021

Sometimes, we need to execute a statement, which might throw an exception and we want to return None if it causes an exception. In short, we want to run that command as failsafe mode, so that it doesn't cause abnormal termination in case of any exception. *Approach till now:* def log_to_telegram(text): ''' Suppose you want to send some log to some telegram channel using telegram bot api. It might raise Network Error, or Invalid Bot token error. ''' send_message(CHANNEL, BOT_TOKEN, … [View More]

6 8

Fractions vs. floats - let's have the cake and eat it
by Martin Teichmann May 22, 2021

May 22, 2021

Hi List, some days ago I posted about my idea let integer division result in fractions, not floats. The upshot of the discussion was that it is a pity that we do not have literals for fractions, but that there is nothing to do about it, as all proposed syntaxes were a bit ugly. But why do we need to have different syntax for both, isn't it possible to simply do both at the same time? The interesting answer is: yes. So my proposal is: number literals (which are not integers) are both fractions … [View More]and floats a the same time - only when we start calculating with them, the receiving function will pick whatever it prefers. For backwards compatiblity the default is float - but you may write code that looks at the fraction as well. I prototyped that here: https://github.com/tecki/cpython/tree/ratiofloat The idea is the following: when the parser (technically, the AST optimizer) creates the objects that represents the literals, let it add some bread crumbs as to where those data are coming from. Currently, 1/2 just means the float 0.5. Instead, let it be the object of a new class, I dubbed it ratiofloat, which inherits from float, but has the exact value added to it as well. ratiofloat just adds two C ints to the float class, making it a bit bigger. But as I said, only for literals, calculated floats still have the same size as before. To give an example (this is not fake, but from the prototype): >>> 2/5 0.4 >>> (2/5).denominator 5 >>> isinstance(2/5, float) True >>> type(2/5) <class 'ratiofloat'> Note that this is only done at compile time, no such behavior is done at run time, everything just behaves like normal floats: >>> two=2 >>> five=5 >>> (two/five) 0.4 >>> (two/five).numerator Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'float' object has no attribute 'numerator' I have tested this, and all tests pass except those where it is explicitly tested whether a value is a float, not one of its subclasses. I ran that also through numpy and sympy, and both behave mostly fine, except again where the tests are testing whether something is actually a float. All of this does not only work for integers, but also for float literals: >>> a=1/3 + 0.1 >>> a 0.43333333333333335 >>> a.numerator 13 >>> a.denominator 30 All this is only interesting once you teach some classes about it. To give an example here: >>> from decimal import Decimal >>> Decimal(1/3) Decimal('0.3333333333333333333333333333') >>> Decimal(0.1) Decimal('0.1') >>> from fractions import Fraction >>> Fraction(1/3) Fraction(1, 3) >>> Fraction(0.1) Fraction(1, 10) I also tries to teach sympy about this. While I succeeded in general, there were many tests that failed, and that for an interesting reason: the sympy tests seem to assume that if you use a float, you want to tell sympy to calculate numerically. So for the sympy tests, 0.1*x and x/10 are something completely different. IMHO this is actually an abuse of features: 0.1 simply is the same as one-tenth, and code should at least try to treat it the same, even if it fails at that. Other than that, sympy works fine (after I taught it): >>> from sympy import symbols >>> x = symbols("x") >>> 1.5*x 3*x/2 >>> x**(0.5) sqrt(x) I think this is now all good enough to be wrapped in a PEP, Chris, can you guide me through the bureaucracy? How would we go forward about this? The good news is that all that I described happens at compile time, so we can use a good ole' "from __future__ import" approach. So my suggestion is: implement it for 3.11, but activate it only with a future import or an interpreter option for the command line. This gives libraries time to adopt to this new style. For 3.12, make this option the default for the command line. So we can tell people to just switch that option off if it doesn't work. And in some far, glorious future, when everybody has a from __future__ line in all of their files, we can make it a default everywhere. Cheers Martin [View Less]

15 27

Bring back PEP 501
by Nick Humrich May 21, 2021

May 21, 2021

PEP 501 was deferred because more learning and time was wanted after introducing f-strings. Now that it has been 5 years, I wonder what the possibilities of revisiting PEP 501 are. I recently had the experience of using javascript "tagged template literals" and was able to build a SQL string parser that is impossible to have SQL injection with. This is done by having the database connection object only accept a certain type of object, and all sql tagged template literals become that object. … [View More]

8 12

Python 3.10: ranges in pattern matching
by Valentin Dymchishin May 18, 2021

May 18, 2021

Hi everyone! I've just read about pattern matching in Python 3.10, and it sounds really nice. I've also found out that Rust supports ranges in pattern matching. https://doc.rust-lang.org/book/ch18-03-pattern-syntax.html#matching-ranges-… fn main() { let x = 5; match x { 1..=5 => println!("one through five"), _ => println!("something else"), } } Can we have something similar please? I think this would be a nice addition to Python's pattern matching.

5 9

Add a mechanism so that multiple exceptions can be caught using `except E1, E2, E3:`
by Thomas Grainger May 18, 2021

May 18, 2021

now that python2.7 is EOL, it might be worth resurrecting this syntax as discussed in https://www.python.org/dev/peps/pep-3100/#id13 eg, python 3.11 could support ``` try: ... except (E1, E2, E3) as e: ... ``` as equivalent to ``` try: ... except E1, E2, E3 as e: ... ``` see also https://mail.python.org/archives/list/python-dev@python.org/thread/HSN6ESRB…

10 15

Improved multi-tasking performance through deterministic GIL hand-off
by Sophist May 11, 2021

May 11, 2021

I don't know how many people will remember some work that David Beazley did about a decade ago on how the GIL impacts multithreading performance - essentially he instrumented the Python interpreter to log how multiple threads competed for the GIL and gave several presentations over the space of 2-3 years. A couple of years ago I reached out to him with an idea on how to significantly improve the way that Python handles multi-threading hand-off of the GIL, but unfortunately he was not … [View More]interested in pursuing this further. I am raising it here in the hope that someone else would be interested in implementing this. In essence my idea is to stop Python handing off the GIL through a competition between threads that are ready to run, and instead for Python to implement a scheduler for the GIL which decides which thread should get the GIL next and directly hands it over. Background ======== Here are the links to David Beazley's presentations: 2009: Inside the Python GIL - https://www.youtube.com/watch?v=ph374fJqFPE 2010: Understanding the Python GIL - https://speakerdeck.com/dabeaz/understanding-the-python-gil https://www.youtube.com/watch?v=Obt-vMVdM8s 2011: Embracing the Global Interpreter Lock - https://speakerdeck.com/dabeaz/embracing-the-global-interpreter-lock https://www.youtube.com/watch?v=fwzPF2JLoeU 2011: In Search of the Perfect Global Interpreter Lock - https://speakerdeck.com/dabeaz/in-search-of-the-perfect-global-interpreter-… https://www.youtube.com/watch?v=5jbG7UKT1l4 The Problem ======== A currently executing thread can give up the GIL (essentially) for two reasons: A) It has started a blocking operation (like a reading data from disk) and is no longer runnable; or B) because it is using a lot of CPU and has reached a Python CPU timeout (nothing to do with the O/S scheduler time-slice) and is still runnable. When the currently executing thread wants to give up the GIL, the current competitive approach essentially sends a signal to all runnable threads that the GIL is now available and the first thread to grab it gets to execute with the GIL. The problem with this (at least as I understand it) is that all threads except the thread that has just given up the GIL are not actually running on a CPU and need the O/S Scheduler to dispatch them before they can run, but the current thread is already running - so if the currently executing thread is is still runnable i.e. because it has reached the Python CPU timeout, then it is highly likely to be the first to grab the GIL back again. This leads to CPU-heavy threads dominating the CPU usage, and the I/O threads being starved out - and this is the opposite of what O/S Scheduling Theory says is optimum - which is to run the I/O threads first to keep the data flowing and everything responsive, typically using low levels of CPU, and let the CPU heavy threads mop up the remaining CPU. But even if the currently executing CPU heavy thread doesn't grab the GIL again, there is no guarantee that the most appropriate of the other threads will grab it instead. It should also be noted that this competitive approach has a lot of overhead - because every thread wakes up to try to compete for the GIL. Proposed Solution ============ My proposed solution is for Python to implement its own (light-weight) scheduling system for the GIL - maintaining a list of threads that are runnable and whether they are I/O bound (gave up the GIL because they became non-runnable due to blocking I/O or waiting for a signal) or CPU heavy (gave up the GIL because of reaching the Python timeout). Ideally we would also add functionality to Python to allow you to specify a thread priority (ideally adjusting the O/S thread priority to match) and we would also retain this priority as part of the GIL scheduling. As far as I can see this proposal is complementary to the current work on sub-interpreters - because there is a big base of existing code which would immediately benefit from these changes without any coding changes. The solution I have in mind is this: a. A scheduling queue is created with its own mutex. The reason for having a separate mutex is that threads can add themselves to the queue without needing to obtain the GIL. b. The queue consists of a tree with the following branches: i) A list of thread priorities, decreasing order ii) Two subqueues: I/O bound, CPU bound iii) Each has a FIFO list of runnable threads When a thread is created, it is assumed to be I/O bound and is added to the back of the appropriate queue. When the GIL is given up, if the currently executing thread is still runnable (reached the Python timeout) it adds itself to the back of the appropriate queue. It then takes the first runnable thread in the queue (highest priority, I/O bound if non empty, otherwise CPU bound) and signals that thread to take the GIL and start executing. As a thread becomes runnable (because it gets a signal it is waiting for or because a blocking I/O completes), it adds itself to the back of the appropriate queue. This is a deterministic solution with low overhead that should ensure that the appropriate thread gets the GIL. And I believe that it would be compatible with the intended sub-interpreter implementation. A couple of caveats: A. I think that there may be some other rare cases where a thread that has the GIL is forced to give it up - and the above algorithm might needs some tweaks to accommodate that. B. Although this should be deterministic, I can imagine a possibility that the signalled thread does not wake up and take the GIL (due to bugs or other reasons not imagined) - and it might be worthwhile having some watchdog timeout mechanism to be started immediately before sending the GIL transfer signal and terminated immediately after receiving the GIL transfer signal, which will be triggered if for any reason the signalled thread does not wake up and take the GIL within a short period. Because the code for handing off the GIL is quite localised, I believe that the implementation of this might be reasonably straight forward. [View Less]

3 3

Barrier Object in asyncio lib
by Yves Duprat May 11, 2021

May 11, 2021

Hi,the list, I'm wondering why Barrier object does not exist in the synchronization primitives of the asyncio lib while it is present in threading and multiprocessing libs ? This may not be the right place to ask this question, but I never found an answer on the web. Thanks for your help. Yves

6 13

OT: Accessibility: Jana Schroeder's Holman Prize Application
by Jonathan Fine May 10, 2021

May 10, 2021

Perhaps Off Topic, but for a good cause. This year I met Jana Scroeder, a blind person forced to change jobs as part of the social cost of Covid. Her outsider experience of computer coding training became a wish to make things better. She has applied for a Holman Prize ($25,000 over a year) to fund this. She's also set up a survey to reach and know better those with similar wishes. One simple way to help open to many is to volunteer to be a sighted helper for a code and math variant of … [View More]

1 0