Python-ideas February 2020

python-ideas@python.org

95 participants
62 discussions

SerialExecutor for concurrent.futures + Convenience constructor

by Jonathan Crall

I'd like to propose an improvement to `concurrent.futures`. The library's ThreadPoolExecutor and ProcessPoolExecutor are excellent tools, but there is currently no mechanism for configuring which type of executor you want. Also, there is no duck-typed class that behaves like an executor, but does its processing in serial. Often times a develop will want to run a task in parallel, but depending on the environment they may want to disable threading or process execution. To address this I use a utility called a `SerialExecutor` which shares an API with ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially in the same python thread: ```python import concurrent.futures class SerialFuture( concurrent.futures.Future): """ Non-threading / multiprocessing version of future for drop in compatibility with concurrent.futures. """ def __init__(self, func, *args, **kw): super(SerialFuture, self).__init__() self.func = func self.args = args self.kw = kw # self._condition = FakeCondition() self._run_count = 0 # fake being finished to cause __get_result to be called self._state = concurrent.futures._base.FINISHED def _run(self): result = self.func(*self.args, **self.kw) self.set_result(result) self._run_count += 1 def set_result(self, result): """ Overrides the implementation to revert to pre python3.8 behavior """ with self._condition: self._result = result self._state = concurrent.futures._base.FINISHED for waiter in self._waiters: waiter.add_result(self) self._condition.notify_all() self._invoke_callbacks() def _Future__get_result(self): # overrides private __getresult method if not self._run_count: self._run() return self._result class SerialExecutor(object): """ Implements the concurrent.futures API around a single-threaded backend Example: >>> with SerialExecutor() as executor: >>> futures = [] >>> for i in range(100): >>> f = executor.submit(lambda x: x + 1, i) >>> futures.append(f) >>> for f in concurrent.futures.as_completed(futures): >>> assert f.result() > 0 >>> for i, f in enumerate(futures): >>> assert i + 1 == f.result() """ def __enter__(self): return self def __exit__(self, ex_type, ex_value, tb): pass def submit(self, func, *args, **kw): return SerialFuture(func, *args, **kw) def shutdown(self): pass ``` In order to make it easy to choose the type of parallel (or serial) backend with minimal code changes I use the following "Executor" wrapper class (although if this was integrated into concurrent.futures the name would need to change to something better): ```python class Executor(object): """ Wrapper around a specific executor. Abstracts Serial, Thread, and Process Executor via arguments. Args: mode (str, default='thread'): either thread, serial, or process max_workers (int, default=0): number of workers. If 0, serial is forced. """ def __init__(self, mode='thread', max_workers=0): from concurrent import futures if mode == 'serial' or max_workers == 0: backend = SerialExecutor() elif mode == 'thread': backend = futures.ThreadPoolExecutor(max_workers=max_workers) elif mode == 'process': backend = futures.ProcessPoolExecutor(max_workers=max_workers) else: raise KeyError(mode) self.backend = backend def __enter__(self): return self.backend.__enter__() def __exit__(self, ex_type, ex_value, tb): return self.backend.__exit__(ex_type, ex_value, tb) def submit(self, func, *args, **kw): return self.backend.submit(func, *args, **kw) def shutdown(self): return self.backend.shutdown() ``` So in summary, I'm proposing to add a SerialExecutor and SerialFuture class as an alternative to the ThreadPool / ProcessPool executors, and I'm also advocating for some sort of "ParamatrizedExecutor", where the user can construct it in "thread", "process", or "serial" model. -- -Jon

1 month, 2 weeks

Add the imath module

by Serhiy Storchaka

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math module. * as_integer_ration(x) Equivalents to: def as_integer_ration(x): if hasattr(x, 'as_integer_ration'): return x.as_integer_ration() else: return (x.numerator, x.denominator) * binom(n, k) Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more efficient algorithm. * sqrt(n) Returns the largest integer r such that r**2 <= n and (r+1)**2 > n. * isprime(n) Tests if n is a prime number. * primes() Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,... Are there more ideas?

2 months, 1 week

New explicit methods to trim strings

by Alex Grigoryev

Following the discussion here (https://link.getmailspring.com/link/7D84D131-65B6-4EF7-9C43-51957F9DFAA9@ge…) I propose to add 3 new string methods: str.trim, str.ltrim, str.rtrim Another option would be to change API for str.split method to work correctly with sequences. In [1]: def ltrim(s, seq): ...: return s[len(seq):] if s.startswith(seq) else s ...: In [2]: def rtrim(s, seq): ...: return s[:-len(seq)] if s.endswith(seq) else s ...: In [3]: def trim(s, seq): ...: return ltrim(rtrim(s, seq), seq) ...: In [4]: s = 'mailto:maria@gmail.com' In [5]: ltrim(s, 'mailto:') Out[5]: 'maria(a)gmail.com' In [6]: rtrim(s, 'com') Out[6]: 'mailto:maria@gmail.' In [7]: trim(s, 'm') Out[7]: 'ailto:maria@gmail.co'

2 months, 1 week

SQL string prefix idea

by minecraft2048＠gmail.com

The idea is to add a new string prefix 's' for SQL string. This string doesn't do anything in Python, unlike b"" or f"" strings, but interactive Python shells like IPython or Jupyter can parse the following characters as SQL syntax instead of Python syntax and give SQL syntax highlighting and autocompletion, and if they are configured correctly, they can do column name autocompletion. Unfortunately when I try to type s"select * from table" it gave me syntax error instead, so I think this need to be implemented in Python language itself instead of module

2 months, 2 weeks

Re: [Python-ideas] Allow star unpacking within an slice expression

by Neil Girdhar

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to allow > > item = object[1, *some_tuple, 2] > item = object[1, *some_tuple, :] > > Was this overlooked in the original proposal, or deliberately rejected? > > Eric >

2 months, 3 weeks

CSV Dictwriter - Handling escape characters

by allu.matikainen＠hotmail.com

Hi, I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters. At the moment, DictWriter writes \n:s as new lines (shown in notepad or Excel) and I would like it to just write "\n" characters in my csv file. I asked a question about this on Stack Overflow but have had no good answers yet. My question also has an example of the data and its form. https://stackoverflow.com/questions/60073809/python-how-to-write-literal-n-…

2 months, 3 weeks

Incremental step on road to improving situation around iterable strings

by Steve Jorgensen

From reading many of the suggestions in this list (including some of my own, of course) there seem to be a lot of us in the Python coder community who find the current behavior of strings being iterable collections of single-character strings to be somewhat problematic. At the same time, trying to change that would obviously have vast consequences, so not something that can simply be changed, even with at a major version boundary. I propose that we start by making a small, safe step in the right direction so we can maybe reach the bigger goal in a far future release (e.g. 5.x or 6.x). The step I propose is to add a `chars` method that returns a sequence (read-only view) of single-character strings that behaves exactly the same as `str` currently does when treated as a collection. Encourage developers to use that for iterating over characters in a string instead of iterating over the string directly. In maybe Python 4 or 5, directly using an `str` instance as a collection could become a warning. BTW, while adding `chars`, it might also be nice to have `ords` which would be a view of the string's character sequence as `ord` integer values.

2 months, 3 weeks

Re: Incremental step on road to improving situation around iterable strings

by Bruce Leban

--- Bruce On Mon, Feb 24, 2020 at 1:19 PM Alex Hall <alex.mojaki(a)gmail.com> wrote: > I think maybe you forgot to reply to the list as well? > > I did mean to reply to the list, so adding it back in. > How many scripts do you have where your own code directly iterated over a > string? How hard do you think it would be to update that code? > Actually quite a bit. I write a lot of code that manipulates words to create puzzles. In every one of those I use strings as iterables. For what it's worth, in one of these programs, the problem I encountered was that strings were not sufficiently performant to solve a very complex problem. I modeled the strings as integers, and built the subset of needed string-like operations on them -- including iteration. If the warning comes from a dependency that hasn't updated, you can simply > add the appropriate code to filter that category of warnings. I think you > have the option of two lines of Python code or one environment variable. > That's not a good answer IMHO. Warnings should matter. Telling me I can filter them out means it shouldn't be there in the first place. And I have to filter them out because warnings that don't matter obscure the ones that do. > > As for your sample, this has been requested twice by others, here was my > response: > > That represents a misunderstanding of my position. I think I'm an outlier >> among the advocates in this thread, but I do not believe that implementing >> any of the ideas in this proposal would significantly affect code that >> lives in the long term. Some code would become slightly better, some >> slightly worse. My concern surrounds the user experience when debugging >> code that accidentally iterates over a string. So it's impossible for me to >> show you code that becomes significantly better because that's not what I'm >> arguing about, and it's unfair to say that quoting people who have >> struggled with these bugs is not evidence for the problem. >> > The claim was made by someone that certain code is "problematic" because the strings are iterable. Asking to see before and after code is certainly relevant, so we can see exactly what the "problem" is and how it gets better. Maybe the code gets *worse* and debugging gets easier. That tradeoff is relevant. And more to the point, is there anything that can't be done by a better linter instead of mucking up the language? > Similarly for jdveiga, I cannot give you "real workable surprising code" >> because I'm talking about code that isn't workable as a result of being >> surprising. I have given examples of real non-working surprising code, and >> I can give more, and if that's not indicative of a real problem then I'm >> very confused and would appreciate more explanation. > > > Now my question is, what is the worst damage that this change could cause? > Because I haven't heard an argument that goes deeper than aesthetics. > The worst damage? The change generates spurious warnings from millions of lines of correct code. It annoys people to the extent they no longer think Python is a stable platform to use to build code. Python dies. Sure that worst case isn't likely but you asked. The bar for changing such a core feature of Python is much higher than "the worst case isn't going to kill the language." In my opinion, declaring code deprecated when there is zero likelihood that it will ever be removed is counter-productive at best. --- Bruce

2 months, 4 weeks

Means of comparing slices for intersection or containment containment and computing intersections or unions

by Steve Jorgensen

I am purposefully proposing this for slices as opposed to ranges because it is about the bounds of the slices, not the items in synthetic sequences. Also, slices can refer to any type of value, not just integers. Presumably, these operations would raise exceptions when used with slices that have `step` values other than `None`. Alternatively, those could hypothetically be valid in restricted cases such as when all properties are either int or Fraction types. Probably better to have them be simply unsupported though. ``` a in b # True if <a> is fully contained within <b> a.intersects(b) # True if any value could be within both <a> and <b> a & b # Intersection of <a> and <b> or None if no intersection a | b # Union of <a> and <b> or Exception if neither contiguous nor overlapping. ``` Also, it might be nice to be able to test whether a non-slice value falls within the slice's bounds. This would be using `x in s` as a shorthand for `s.start <= x < s.end`. Again, this is different than asking whether a value is "in" a range because a rage is a sequence of discrete integers whereas a slice represents a possibly continuous range of any kind of value to which magnitude is applicable. ``` slice(1, 2) in (0, 3) # => True because 1 >= 0 and 2 <= 3 slice(0.5, 1.5) in slice(0, 2) # => True because 0.5 >= 1.5 and 0.5 < 2 1 in (0, 3) # => True because 0 <= 1 < 3 'Joe' in slice('Alice', 'Riley') # => True because 'Alice' <= 'Joe' < 'Riley' slice(1.1, 5.9).intersects(slice(2, 10.5)) # => True because either... # 1.1 <= 2 < 5.9 or # 1.1 < 10.5 < 5.5 or # 2 <= 1.1 < 10.5 or # 2 < 5.9 < 10.5 slice(5.5, 15.5) & slice(10.25, 20.25) # => slice(10.25, 15.5) slice(5.5, 15.5) | slice(10.25, 20.25) # => slice(5.5, 20.25) slice('abc', 'fff') & slice('eee', 'xyz') # => slice('fff', 'eee') ```

2 months, 4 weeks

Add a __valid_getitem_requests__ protocol

by Soni L.

It'd be nice to have a __valid_getitem_requests__ protocol that, if present, yields pairs such that: for key, value in items(obj): assert obj[key] == value for any obj. Note that this definition explicitly allows subsets, which makes it useful for sequences and defaultdicts, as they can have all sorts of keys (or even infinitely many in the case of defaultdict): >>> [1, 2, 3, 4, 5][1] 2 >>> [1, 2, 3, 4, 5][1:3] [2, 3] >>> [1, 2, 3, 4, 5][-1] 5 and as such we should consider limiting the results for at least these 2 cases. I currently use a similar idea in my code but I feel like having this would make my code more extensible without ppl having to monkeypatch it.

3 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas February 2020