I'd like to propose an improvement to `concurrent.futures`. The library's
ThreadPoolExecutor and ProcessPoolExecutor are excellent tools, but there
is currently no mechanism for configuring which type of executor you want.
Also, there is no duck-typed class that behaves like an executor, but does
its processing in serial. Often times a develop will want to run a task in
parallel, but depending on the environment they may want to disable
threading or process execution. To address this I use a utility called a
`SerialExecutor` which shares an API with
ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially
in the same python thread:
class SerialFuture( concurrent.futures.Future):
Non-threading / multiprocessing version of future for drop in
def __init__(self, func, *args, **kw):
self.func = func
self.args = args
self.kw = kw
# self._condition = FakeCondition()
self._run_count = 0
# fake being finished to cause __get_result to be called
self._state = concurrent.futures._base.FINISHED
result = self.func(*self.args, **self.kw)
self._run_count += 1
def set_result(self, result):
Overrides the implementation to revert to pre python3.8 behavior
self._result = result
self._state = concurrent.futures._base.FINISHED
for waiter in self._waiters:
# overrides private __getresult method
if not self._run_count:
Implements the concurrent.futures API around a single-threaded backend
>>> with SerialExecutor() as executor:
>>> futures = 
>>> for i in range(100):
>>> f = executor.submit(lambda x: x + 1, i)
>>> for f in concurrent.futures.as_completed(futures):
>>> assert f.result() > 0
>>> for i, f in enumerate(futures):
>>> assert i + 1 == f.result()
def __exit__(self, ex_type, ex_value, tb):
def submit(self, func, *args, **kw):
return SerialFuture(func, *args, **kw)
In order to make it easy to choose the type of parallel (or serial) backend
with minimal code changes I use the following "Executor" wrapper class
(although if this was integrated into concurrent.futures the name would
need to change to something better):
Wrapper around a specific executor.
Abstracts Serial, Thread, and Process Executor via arguments.
mode (str, default='thread'): either thread, serial, or process
max_workers (int, default=0): number of workers. If 0, serial is
def __init__(self, mode='thread', max_workers=0):
from concurrent import futures
if mode == 'serial' or max_workers == 0:
backend = SerialExecutor()
elif mode == 'thread':
backend = futures.ThreadPoolExecutor(max_workers=max_workers)
elif mode == 'process':
backend = futures.ProcessPoolExecutor(max_workers=max_workers)
self.backend = backend
def __exit__(self, ex_type, ex_value, tb):
return self.backend.__exit__(ex_type, ex_value, tb)
def submit(self, func, *args, **kw):
return self.backend.submit(func, *args, **kw)
So in summary, I'm proposing to add a SerialExecutor and SerialFuture class
as an alternative to the ThreadPool / ProcessPool executors, and I'm also
advocating for some sort of "ParamatrizedExecutor", where the user can
construct it in "thread", "process", or "serial" model.
What are your thoughts about adding a new imath module for integer
mathematics? It could contain the following functions:
Is just moved from the math module, but non-integer types are rejected.
Currently math.factorial() accepts also integer floats like 3.0. It
looks to me, the rationale was that at the time when math.factorial()
was added, all function in the math module worked with floats. But now
we can revise this decision.
* gcd(n, m)
Is just moved from the math module.
if hasattr(x, 'as_integer_ration'):
return (x.numerator, x.denominator)
* binom(n, k)
Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more
Returns the largest integer r such that r**2 <= n and (r+1)**2 > n.
Tests if n is a prime number.
Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,...
Are there more ideas?
Following the discussion here (https://link.getmailspring.com/link/7D84D131-65B6-4EF7-9C43-51957F9DFAA9@ge…) I propose to add 3 new string methods: str.trim, str.ltrim, str.rtrim
Another option would be to change API for str.split method to work correctly with sequences.
In : def ltrim(s, seq):
...: return s[len(seq):] if s.startswith(seq) else s
In : def rtrim(s, seq):
...: return s[:-len(seq)] if s.endswith(seq) else s
In : def trim(s, seq):
...: return ltrim(rtrim(s, seq), seq)
In : s = 'mailto:email@example.com'
In : ltrim(s, 'mailto:')
In : rtrim(s, 'com')
In : trim(s, 'm')
The idea is to add a new string prefix 's' for SQL string. This string doesn't do anything in Python, unlike b"" or f"" strings, but interactive Python shells like IPython or Jupyter can parse the following characters as SQL syntax instead of Python syntax and give SQL syntax highlighting and autocompletion, and if they are configured correctly, they can do column name autocompletion. Unfortunately when I try to type s"select * from table" it gave me syntax error instead, so I think this need to be implemented in Python language itself instead of module
I didn't think of this when we were discussing 448. I ran into this today,
so I agree with you that it would be nice to have this.
On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote:
> I've been thinking about the * unpacking operator while writing some numpy
> code. PEP 448 allows the following:
> values = 1, *some_tuple, 2
> object[(1, *some_tuple, 2)]
> It seems reasonable to me that it should be extended to allow
> item = object[1, *some_tuple, 2]
> item = object[1, *some_tuple, :]
> Was this overlooked in the original proposal, or deliberately rejected?
I would like if the DictWriter would be able to write escape characters (\n and \r and their combinations such as \n\n\n and \r\n) as literal characters.
At the moment, DictWriter writes \n:s as new lines (shown in notepad or Excel) and I would like it to just write "\n" characters in my csv file.
I asked a question about this on Stack Overflow but have had no good answers yet. My question also has an example of the data and its form.
From reading many of the suggestions in this list (including some of my own, of course) there seem to be a lot of us in the Python coder community who find the current behavior of strings being iterable collections of single-character strings to be somewhat problematic. At the same time, trying to change that would obviously have vast consequences, so not something that can simply be changed, even with at a major version boundary.
I propose that we start by making a small, safe step in the right direction so we can maybe reach the bigger goal in a far future release (e.g. 5.x or 6.x).
The step I propose is to add a `chars` method that returns a sequence (read-only view) of single-character strings that behaves exactly the same as `str` currently does when treated as a collection. Encourage developers to use that for iterating over characters in a string instead of iterating over the string directly. In maybe Python 4 or 5, directly using an `str` instance as a collection could become a warning.
BTW, while adding `chars`, it might also be nice to have `ords` which would be a view of the string's character sequence as `ord` integer values.
On Mon, Feb 24, 2020 at 1:19 PM Alex Hall <alex.mojaki(a)gmail.com> wrote:
> I think maybe you forgot to reply to the list as well?
> I did mean to reply to the list, so adding it back in.
> How many scripts do you have where your own code directly iterated over a
> string? How hard do you think it would be to update that code?
Actually quite a bit. I write a lot of code that manipulates words to
create puzzles. In every one of those I use strings as iterables. For what
it's worth, in one of these programs, the problem I encountered was that
strings were not sufficiently performant to solve a very complex problem. I
modeled the strings as integers, and built the subset of needed string-like
operations on them -- including iteration.
If the warning comes from a dependency that hasn't updated, you can simply
> add the appropriate code to filter that category of warnings. I think you
> have the option of two lines of Python code or one environment variable.
That's not a good answer IMHO. Warnings should matter. Telling me I can
filter them out means it shouldn't be there in the first place. And I have
to filter them out because warnings that don't matter obscure the ones that
> As for your sample, this has been requested twice by others, here was my
> That represents a misunderstanding of my position. I think I'm an outlier
>> among the advocates in this thread, but I do not believe that implementing
>> any of the ideas in this proposal would significantly affect code that
>> lives in the long term. Some code would become slightly better, some
>> slightly worse. My concern surrounds the user experience when debugging
>> code that accidentally iterates over a string. So it's impossible for me to
>> show you code that becomes significantly better because that's not what I'm
>> arguing about, and it's unfair to say that quoting people who have
>> struggled with these bugs is not evidence for the problem.
The claim was made by someone that certain code is "problematic" because
the strings are iterable. Asking to see before and after code is certainly
relevant, so we can see exactly what the "problem" is and how it gets
better. Maybe the code gets *worse* and debugging gets easier. That
tradeoff is relevant. And more to the point, is there anything that can't
be done by a better linter instead of mucking up the language?
> Similarly for jdveiga, I cannot give you "real workable surprising code"
>> because I'm talking about code that isn't workable as a result of being
>> surprising. I have given examples of real non-working surprising code, and
>> I can give more, and if that's not indicative of a real problem then I'm
>> very confused and would appreciate more explanation.
> Now my question is, what is the worst damage that this change could cause?
> Because I haven't heard an argument that goes deeper than aesthetics.
The worst damage? The change generates spurious warnings from millions of
lines of correct code. It annoys people to the extent they no longer think
Python is a stable platform to use to build code. Python dies.
Sure that worst case isn't likely but you asked. The bar for changing such
a core feature of Python is much higher than "the worst case isn't going to
kill the language." In my opinion, declaring code deprecated when there is
zero likelihood that it will ever be removed is counter-productive at best.
I am purposefully proposing this for slices as opposed to ranges because it is about the bounds of the slices, not the items in synthetic sequences. Also, slices can refer to any type of value, not just integers.
Presumably, these operations would raise exceptions when used with slices that have `step` values other than `None`. Alternatively, those could hypothetically be valid in restricted cases such as when all properties are either int or Fraction types. Probably better to have them be simply unsupported though.
a in b # True if <a> is fully contained within <b>
a.intersects(b) # True if any value could be within both <a> and <b>
a & b # Intersection of <a> and <b> or None if no intersection
a | b # Union of <a> and <b> or Exception if neither contiguous nor overlapping.
Also, it might be nice to be able to test whether a non-slice value falls within the slice's bounds. This would be using `x in s` as a shorthand for `s.start <= x < s.end`. Again, this is different than asking whether a value is "in" a range because a rage is a sequence of discrete integers whereas a slice represents a possibly continuous range of any kind of value to which magnitude is applicable.
slice(1, 2) in (0, 3)
# => True because 1 >= 0 and 2 <= 3
slice(0.5, 1.5) in slice(0, 2)
# => True because 0.5 >= 1.5 and 0.5 < 2
1 in (0, 3)
# => True because 0 <= 1 < 3
'Joe' in slice('Alice', 'Riley')
# => True because 'Alice' <= 'Joe' < 'Riley'
slice(1.1, 5.9).intersects(slice(2, 10.5))
# => True because either...
# 1.1 <= 2 < 5.9 or
# 1.1 < 10.5 < 5.5 or
# 2 <= 1.1 < 10.5 or
# 2 < 5.9 < 10.5
slice(5.5, 15.5) & slice(10.25, 20.25)
# => slice(10.25, 15.5)
slice(5.5, 15.5) | slice(10.25, 20.25)
# => slice(5.5, 20.25)
slice('abc', 'fff') & slice('eee', 'xyz')
# => slice('fff', 'eee')
It'd be nice to have a __valid_getitem_requests__ protocol that, if
present, yields pairs such that:
for key, value in items(obj):
assert obj[key] == value
for any obj. Note that this definition explicitly allows subsets, which
makes it useful for sequences and defaultdicts, as they can have all
sorts of keys (or even infinitely many in the case of defaultdict):
>>> [1, 2, 3, 4, 5]
>>> [1, 2, 3, 4, 5][1:3]
>>> [1, 2, 3, 4, 5][-1]
and as such we should consider limiting the results for at least these 2
I currently use a similar idea in my code but I feel like having this
would make my code more extensible without ppl having to monkeypatch it.