Mailman 3 April 2020 - Python-ideas

Explicitly defining a string buffer object (aka StringIO += operator)
by Paul Sokolovsky April 12, 2020

April 12, 2020

Hello, 1. Intro -------- It is a well-known anti-pattern to use a string as a string buffer, to construct a long (perhaps very long) string piece-wise. A running example is: buf = "" for i in range(50000): buf += "foo" print(buf) An alternative is to use a buffer-like object explicitly designed for incremental updates, which for Python is io.StringIO: buf = io.StringIO() for i in range(50000): buf.write("foo") print(buf.getvalue()) As can be seen, this requires changing the way buffer is constructed (usually in one place), the way buffer value is taken (usually in one place), but more importantly, it requires changing each line which adds content to a buffer, and there can be many of those for more complex algorithms, leading to a code less clear than the original code, requiring noise-like changes, and complicating updates to 3rd-party code which needs optimization. To address this, this RFC proposes to add an __iadd__ method (i.e. implementing "+=" operator) to io.StringIO and io.BytesIO objects, making it the exact alias of .write() method. This will allow for the code very parallel to the original str-using code: buf = io.StringIO() for i in range(50000): buf += "foo" print(buf.getvalue()) This will still require updates for buffer construction/getting value, but that's usually 2 lines. But it will leave the rest of code intact, and not obfuscate the original content construction algorithm. 2. Performance Discussion ------------------------- The motivation for this change (of promoting usage of io.StringIO, by making it look&feel more like str) is performance. But is it really a problem? Turns out, it is such a pervasive anti-pattern, that recent versions on CPython3 have a special optimization for it. Let's use following script for testing: --------- import timeit import io def string(): sb = u"" for i in range(50000): sb += u"a" def strio(): sb = io.StringIO() for i in range(50000): sb.write(u"a") print(timeit.timeit(string, number=10)) print(timeit.timeit(strio, number=10)) --------- With CPython3.6 the result is: $ python3.6 str_iadd-vs-StringIO_write.py 0.03350826998939738 0.033480543992482126 In other words, there's no difference between usage of str vs StringIO. But it wasn't always like that, with CPython2.7.17: $ python2.7 str_iadd-vs-StringIO_write.py 2.10510993004 0.0399420261383 But Python2 is dead, right? Ok, let's see how Jython3 and IronPython3 fair. To my surprise, there're no (public releases of) such. Both projects sit firmly in the Python2 territory. So, let's try them: $ java -jar jython-standalone-2.7.2.jar str_iadd-vs-StringIO_write.py 10.8869998455 1.74700021744 Sadly, I wasn't able to get to run IronPython.2.7.9.zip on my Linux system, so I used the online version at https://tio.run/#python2-iron (after discovering that https://ironpython.net/try/ is dead). 2.7.9 (IronPython 2.7.9 (2.7.9.0) on Mono 4.0.30319.42000 (64-bit)) 26.2704391479 1.55628967285 So, it seems that rumors of Python2 being dead are somewhat exaggerated. Let's try a project which tries to provide "missing migration path" between Python2 and Python3 - https://github.com/naftaliharris/tauthon Tauthon 2.8.1+ (heads/master:7da5b76f5b, Mar 29 2020, 18:05:05) $ tauthon str_iadd-vs-StringIO_write.py 0.792158126831 0.0467159748077 Whoa, tauthon seems to be faithful to its promise of being half-way between CPython2 and CPython2. Anyway, let's get back to Python3. Fortunately, there's PyPy3, so let's try that: $ ./pypy3.6-v7.3.0-linux64/bin/pypy3 str_iadd-vs-StringIO_write.py 0.5423258490045555 0.01754526497097686 Let's not forget little Python brothers and continue with MicroPython 1.12 (https://github.com/micropython/micropython): $ micropython str_iadd-vs-StringIO_write.py 41.63419413566589 0.08073711395263672 Pycopy 3.0.6 (https://github.com/pfalcon/pycopy): $ pycopy str_iadd-vs-StringIO_write.py 25.03198313713074 0.0713810920715332 I also wanted to include TinyPy (http://tinypy.org/) and Snek (https://github.com/keith-packard/snek) in the shootout, but both (seem to) lack StringIO object. These results can be summarized as follows: of more than half-dozen Python implementations, CPython3 is the only implementation which optimizes for the dubious usage of an immutable string type as an accumulating character buffer. For all other implementations, unintended usage of str incurs overhead of about one order of magnitude, 2 order of magnitude for implementations optimized for particular usecases (this includes PyPy optimized for speed vs MicroPython/Pycopy optimized for small code size and memory usage). Consequently, other implementations have 2 choices: 1. Succumb to applying the same mis-optimization for string type as CPython3. (With the understanding that for speed-optimized projects, implementing mis-optimizations will eat into performance budget, and for memory-optimized projects, it likely will lead to noticeable memory bloat.) 2. Struggle against inefficient-by-concept usage, and promote usage of the correct object types for incremental construction of string content. This would require improving ergonomics of existing string buffer object, to make its usage less painful for both writing new code and refactoring existing. As you may imagine, the purpose of this RFC is to raise awareness and try to make headway with the choice 2. 3. Scope Creep, aka "Possible Future Work" ------------------------------------------ The purpose of this RFC is specifically to propose to apply *single* simple, obvious change. .__iadd__ is just an alias for .write, period. However, for completeness, it makes sense to consider both alternatives and where the path of adding "str-like functionality" may lead us. 1. One alternative to patching StringIO would be to introduce a completely different type, e.g. StringBuf. But that largely would be "creating more entities without necessity", given that StringIO already offers needed buffering functionality, and just needs a little touch of polish with interface. If 2 classes like StringIO and StringBuf existed, it would be extra quiz to explain difference between them and why they both exist. 2. On the other hand, this RFC fixates on the output buffering. But just image how much fun can be done re: input buffers! E.g., we can define "buf[0]" to have semantics of "tmp = buf.tell(); res = buf.read(1); buf.seek(tmp); return res". Ditto for .startswith(), etc., etc. So, well... This RFC is about making .__iadd__ to be an alias for .write, and cover with this output buffering usecase. Whoever may have interest in dealing with input buffer shortcuts would need to provide a separate RFC, with separate usecases and argumentation. 4. (Self-)Criticism and Risks. ------------------------------ 1. The biggest "criticism" I see is a response a-la "there's no problem with CPython3, so there's nothing to fix". This is related to a bigger questions "whether a life outside CPython exists", or put more formally, where's the border between Python-the-language and CPython-the-implementation. To address this point, I tried to collect performance stats for a pretty wide array of Python implementations. 2. Another potential criticism is that this may open a scope creep to add more str-like functionality to classes which classically expose a stream interface. Paragraph 3.2 is dedicated specifically to address this point, by invoking hopefully the best-practice approach: request to focus on the currently proposed feature, which requires very little changes for arguably noticeable improvements. (At the same time, for an abstract possibility of this change to be found positive, this may influence further proposals from interested parties). 3. Implementing the required functionality is pretty easy with a user subclass: class MyStringBuf(io.StringIO): def __iadd__(self, s): self.write(s) return self Voila. The problem is performance. Calling such .__iadd__() method implementing in Python is 3 times slower than calling .write() directly (with CPython3.6). But paradigmatic problem is even bigger: this RFC seeks to establish the best practice of using explicitly designed for the purpose type with ergonomic interface. Saying that "we lack such clearly designated type out of the box, but if you figure out that it's a problem (if you do figure that out), you can easily resolve that on your side, albeit with a performance hit when compared with it being provided out of the box" - that's not really welcoming or ergonomic. 5. Prior Art ------------ As many things related to Python, the idea is not new. I found thread from 2006 dedicated to it: https://mail.python.org/pipermail/python-list/2006-January/403480.html (strangely, there's some mixup in archives, and "thread view" shows another message as thread starter, though it's not: https://mail.python.org/pipermail/python-list/2006-January/357453.html) . The discussion there seemed to be without clear resolution and has been swamped into discussion of unicode handling complexities in Python2 and implementation details of "StringIO" vs "cStringIO" modules (both of which were deprecated in favor of "io"). -- Best regards, Paul mailto:pmiscml@gmail.com

19 76

Improve the Windows python installer to reduce new user confusion
by Barry Scott April 11, 2020

April 11, 2020

The Python Users list most often asked question seems to be from Windows users who run the installed multiple times because they assume that the install is the python program. Followed by I installed Python, where is it? My suggestion is that the installer wizard "Setup was successful" panel has a link to a getting started on Windows with Python page. I'd also suggest to layout the information that is key before the thank you to Mark Hammond. Suggested order: Getting started with Python on Windows Python Tutorial Python documentation What's new Special Thanks I would also suggest adding all of these as links in the Start Menu. Currently the menu is: IDLE Python Python manuals Python Module Docs Becomes: IDLE Python Getting started with Python on Windows Python documentation I'd also suggest being clear about the use of py.exe and python.exe in the getting started on windows docs. I have assumed that py.exe is the preferred command line tool to start python if that is right document it as such. With py.exe no need to mess with the PATH. Barry

4 5

RFC: For Loop Invariants
by Elliott Dehnbostel April 11, 2020

April 11, 2020

Hello Everyone, If I've done this incorrectly, please let me know so that I can improve/revise. I'm new to the Python community and quite enjoy the more functional features of Python 3, but have I have a peeve about it. I'd like to propose and discuss the following enhancement to Python 3: *Consider the following trivial for-loop:* chars = "abcaaabkjzhbjacvb" seek = {'a','b','c'} count = 0for a in chars: if a in seek: count += 1 Gross. Twice nested for a simple count. *We could refactor the block like so:* chars = "abcaaabkjzhbjacvb" seek = {'a','b','c'} count = 0for a in filter(lambda c: c in seek, chars): count += 1 Which is all well and good, but doesn't quite read like English. It's verbose, too. It also uses advanced concepts new programmers may not understand. *We could do this:* chars = "abcaaabkjzhbjacvb" seek = {'a','b','c'} count = sum([1 for a in chars if a in seek]) However, this changes important semantics by creating an entire new list before summing. Also, adding just one more expression to the most nested block thwarts that refactor. I propose the following enhancement: chars = "abcaaabkjzhbjacvb" seek = {'a','b','c'} count = 0for a in chars if a in seek: count += 1 *What happened there?* I've allowed a condition to follow the "for" construct without a colon or newline between. *To be clear, this remains incorrect:* chars = "abcaaabkjzhbjacvb" seek = {'a','b','c'} count = 0for a in chars # No colon prior to the newline if a in seek: count += 1 *Value proposal:* I assert that the inlined 'if' condition pattern is superior to the alternative refactors. Right now, the way to acquire an invariant in a more nasty loop would be: for a in iterable: if condition: continue But this is messy and not particularly Pythonic. The filter approach uses concepts that should not be necessary for this task. The comprehension approach has different, undesirable semantics. *Conclusion:* I wanted to submit my thoughts here before getting too deep into this. Any input would be appreciated! Thanks everyone.

13 24

SerialExecutor for concurrent.futures + Convenience constructor
by Jonathan Crall April 10, 2020

April 10, 2020

I'd like to propose an improvement to `concurrent.futures`. The library's ThreadPoolExecutor and ProcessPoolExecutor are excellent tools, but there is currently no mechanism for configuring which type of executor you want. Also, there is no duck-typed class that behaves like an executor, but does its processing in serial. Often times a develop will want to run a task in parallel, but depending on the environment they may want to disable threading or process execution. To address this I use a utility called a `SerialExecutor` which shares an API with ThreadPoolExecutor/ProcessPoolExecutor but executes processes sequentially in the same python thread: ```python import concurrent.futures class SerialFuture( concurrent.futures.Future): """ Non-threading / multiprocessing version of future for drop in compatibility with concurrent.futures. """ def __init__(self, func, *args, **kw): super(SerialFuture, self).__init__() self.func = func self.args = args self.kw = kw # self._condition = FakeCondition() self._run_count = 0 # fake being finished to cause __get_result to be called self._state = concurrent.futures._base.FINISHED def _run(self): result = self.func(*self.args, **self.kw) self.set_result(result) self._run_count += 1 def set_result(self, result): """ Overrides the implementation to revert to pre python3.8 behavior """ with self._condition: self._result = result self._state = concurrent.futures._base.FINISHED for waiter in self._waiters: waiter.add_result(self) self._condition.notify_all() self._invoke_callbacks() def _Future__get_result(self): # overrides private __getresult method if not self._run_count: self._run() return self._result class SerialExecutor(object): """ Implements the concurrent.futures API around a single-threaded backend Example: >>> with SerialExecutor() as executor: >>> futures = [] >>> for i in range(100): >>> f = executor.submit(lambda x: x + 1, i) >>> futures.append(f) >>> for f in concurrent.futures.as_completed(futures): >>> assert f.result() > 0 >>> for i, f in enumerate(futures): >>> assert i + 1 == f.result() """ def __enter__(self): return self def __exit__(self, ex_type, ex_value, tb): pass def submit(self, func, *args, **kw): return SerialFuture(func, *args, **kw) def shutdown(self): pass ``` In order to make it easy to choose the type of parallel (or serial) backend with minimal code changes I use the following "Executor" wrapper class (although if this was integrated into concurrent.futures the name would need to change to something better): ```python class Executor(object): """ Wrapper around a specific executor. Abstracts Serial, Thread, and Process Executor via arguments. Args: mode (str, default='thread'): either thread, serial, or process max_workers (int, default=0): number of workers. If 0, serial is forced. """ def __init__(self, mode='thread', max_workers=0): from concurrent import futures if mode == 'serial' or max_workers == 0: backend = SerialExecutor() elif mode == 'thread': backend = futures.ThreadPoolExecutor(max_workers=max_workers) elif mode == 'process': backend = futures.ProcessPoolExecutor(max_workers=max_workers) else: raise KeyError(mode) self.backend = backend def __enter__(self): return self.backend.__enter__() def __exit__(self, ex_type, ex_value, tb): return self.backend.__exit__(ex_type, ex_value, tb) def submit(self, func, *args, **kw): return self.backend.submit(func, *args, **kw) def shutdown(self): return self.backend.shutdown() ``` So in summary, I'm proposing to add a SerialExecutor and SerialFuture class as an alternative to the ThreadPool / ProcessPool executors, and I'm also advocating for some sort of "ParamatrizedExecutor", where the user can construct it in "thread", "process", or "serial" model. -- -Jon

6 26

Live variable analysis -> earlier release?
by Guido van Rossum April 10, 2020

April 10, 2020

Look at the following code. def foo(a, b): x = a + b if not x: return None sleep(1) # A calculation that does not use x return a*b This code DECREFs x when the frame is exited (at the return statement). But (assuming) we can clearly see that x is not needed during the sleep (representing a big calculation), we could insert a "del x" statement before the sleep. I think our compiler is smart enough to find out *some* cases where it could safely insert such del instructions. And this would potentially save some memory. (For example, in the above example, if a and b are large lists, x would be an even larger list, and its memory could be freed earlier.) For sure, we could manually insert del statements in code where it matters to us. But if the compiler could do it in all code, regardless of whether it matters to us or not, it would probably catch some useful places where we wouldn't even have thought of this idea, and we might see a modest memory saving for most programs. Can anyone tear this idea apart? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

12 19

Alternative to iterator unpacking that wraps iterator-produced ValueError
by Soni L. April 9, 2020

April 9, 2020

To put it simple, unpacking raises ValueError: >>> x, = () Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: not enough values to unpack (expected 1, got 0) >>> x, = (1, 2) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: too many values to unpack (expected 1) But if the iterator raises ValueError, there's no way to tell it apart from the unpacking: >>> def foo(): ... yield None ... raise ValueError ... >>> foo() <generator object foo at 0x7fa0e70e6430> >>> x = foo() >>> x, = foo() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in foo ValueError And the workaround for this is a bit ugly. We already convert e.g. StopIteration into RuntimeError in many cases, why can't we do so here too? For backwards compatibility, this should probably be an itertools utility tho.

4 4

Optimize out unused variables
by Serhiy Storchaka April 9, 2020

April 9, 2020

I doubted whether or not to write, but since Guido has already touched a similar topic (see "Live variable analysis -> earlier release"), so will I write. It is common to assign some values which are never used to variables because the syntax demands this. For example: head, _, rest = path.partition('/') first, second, *_ = line.split() for _ in range(10): ... [k for k, _ in pairs] What if make such assignments no-op? It will only work with some limitations: 1. The variable is local. Not global, not nonlocal, not cell. 2. The variable name must start with '_'. Otherwise there would be larger risk of breaking a legal code which uses locals() with str.format() or like. 3. "del" is considered a use of the variable. So explicitly deleted variables will not be optimized out. 4. Star-assignment still consumes the iterator. It just might not to keep all values in memory. STORE_FAST will be replaced with POP_TOP in these cases. I wrote some code few weeks ago, and it is not too complex. My doubts are only that the benefit of the optimization with the above limitations is very restricted.

3 4

Re: Improvement: __json__
by Tin Tvrtković April 8, 2020

April 8, 2020

As the author of one of these third-party libraries, I feel like I can contribute to this discussion. It can indeed be done very elegantly with type annotations, and it should for sure be left to the ecosystem. The only things we need from core Python are good tools for dealing with run-time type information. For example, singledispatch doesn't really work with types (i.e. Optionals, Unions, Sequences as opposed to actual classes), but all of that can be worked around. In my experience, runtime type information is extremely useful exactly in cases of deserialization, and for big projects starts being useful much before the type information starts being useful in a static analysis context. > From: Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Subject: [Python-ideas] Re: Improvement: __json__ > To: python-ideas(a)python.org > Message-ID: <e70016ba-5419-e252-d7bd-9705dbebf24c(a)canterbury.ac.nz> > Content-Type: text/plain; charset=utf-8; format=flowed > > On 7/04/20 4:57 am, Wes Turner wrote: > > Python object > JSON > object requires type information to be > serialized. > > Not necessarily -- Java's json library uses reflection on compile > time type information to deserialise json into an object hierarchy. > You tell it the class corresponding to the top level and it figures > out the rest. > > Something similar could be done in Python using type annotations. > > -- > Greg > >

4 12

"except BaseException with f as result"
by Soni L. April 8, 2020

April 8, 2020

the fact that exception handlers aren't reusable and composable is really bothering me so I would like to propose something to make them reusable and composable. for example, if I have this mess: def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError as exc: raise PropertyError from exc iterator = factory(self._obj) try: first = next(iterator) except StopIteration: return (x for x in ()) except abdl.exceptions.ValidationError as exc: raise LookupError from exc return itertools.chain([first], iterator) I get to refactor it into: def keyerror_handler(exc): raise PropertyError from exc def other_handler(exc): if isinstance(exc, StopIteration): return (x for x in ()) raise LookupError from exc def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError with keyerror_handler: pass iterator = factory(self._obj) try: first = next(iterator) except (StopIteration, abdl.exceptions.ValidationError) with other_handler as result: return result return itertools.chain([first], iterator) and the handlers become reusable! another idea would be to use semicolons: def get_property_values(self, prop): try: factory = self.get_supported_properties()[prop] except KeyError with keyerror_handler; iterator = factory(self._obj) try: first = next(iterator) except abdl.exceptions.ValidationError with validation_handler; except StopIteration: return (x for x in ()) return itertools.chain([first], iterator) but nobody likes semicolons

5 12

Re: Improvement: __json__
by Dan Cojocaru April 7, 2020

April 7, 2020

Hello, I'm not really sure how suggesting improvements to Python goes and I've found this mailing list, so I thought I'd email it. My suggestion is related to the json built in package, and more exactly about serialising custom classes. As of now, the process involves using a JSONEncoder class which, by default, supports the following types/values: dict, list, tuple, str, int, float, True, False, None and int- & float-derived Enums. In order to dump a custom class to JSON, the process involves creating a new class inheriting JSONEncoder and defining the default function in order to handle custom classes. I say that this approach is very complex and perhaps unpythonic, definitely very weird compared to other approaches. An alternative that comes to mind and would feel more natural is adding support for recognising a __json__ function. If a class would contain such a function, the default JSONEncoder should take that function's return value instead for encoding, assuming that such returned value will be able to be then serialised. If such a functionality would be added, many custom functions will most likely return a dict from their __json__ function, skipping the need for creating a class inheriting JSONEncoder altogether. I hope this idea is a good one and could perhaps even get implemented in the next version of Python. Respectfully, Dan Cojocaru.

6 14