Mailman 3 June 2022 - Python-ideas

Addition to fnmatch.py
by John Carter 07 Jun '22

07 Jun '22

I’d like to propose a simple addition to the 'fnmatch' module. Specificity a function that returns a list of names that match a pattern AND a list of those that don't. In a recent project I found that I wished to split a list of files and move those that matched a pattern to one folder and those that didn't to another folder. I was using fnmatch.filter to select the first set of files and then a list comprehension to generate the second set. For a small number of files (~ 10) this was perfectly adequate. However as I needed to process many files (>>10000) the execution time was very significant. Profiling the code showed that the time was spent in generating the second set. I tried a number of solutions including designing a negative filter, walking the file system to find those files that had not been moved and using more and more convoluted ways to improve the second selection. Eventually I gave in and hacked a local copy of fnmatch.py as below: def split(names, pat): """Return the subset of the list NAMES that match PAT.""" """Also returns those names not in NAMES""" result = [] notresult = [] pat = os.path.normcase(pat) pattern_match = _compile_pattern(pat) if os.path is posixpath: # normcase on posix is NOP. Optimize it away from the loop. for name in names: if not pattern_match(name): result.append(name) else: notresult.append(name) else: for name in names: if not pattern_match(os.path.normcase(name)): result.append(name) else: notresult.append(name) return result, notresult The change is the addition of else clauses to the if not pattermath statements. This solved the problem and benchmarking showed that it only took a very small additional time (20ms for a million strings) to generate both lists Number of tests cases 1000000 Example data ['Ba1txmKkiC', 'KlJx.f_AGj', 'Umwbw._Wa9', '4YlgA5LVpI’] Search for '*A*' Test Time(sec) Positive Negative WCmatch.filter 1.953125 26211 0 filter 0.328125 14259 0 split 0.343750 14259 85741 List Comp. 270.468751 14259 85741 The list comprehension [x for x in a if x not in b]*, was nearly 900 times slower. ‘fnmatch’ was an appropriate solution to this problem as typing ‘glob’ style search patterns was easier than having to enter regular expressions when prompted by my code. I would like to propose that split, even though it is very simple, be included in the 'fnmatch' module. John *a is the original and b is those that match.

14 26

Add .except() and .only(), and .values_at(). instance methods to dict
by Steve Jorgensen 06 Jun '22

06 Jun '22

I think these are an extremely common needs that are worth having standard methods for. If adding instance methods seems like a bad idea, then maybe add functions to the standard library that perform the same operations. m = {'a': 123, 'b': 456, 'c': 789} m.except(('a', 'c')) # {'b': 456} m.only(('b', 'c')) # {'b': 456, 'c': 789} m.values_at(('a', 'b')) # [123, 456] …or… from mappings import except, only, values_at m = {'a': 123, 'b': 456, 'c': 789} except(m, ('a', 'c')) # {'b': 456} only(m, ('b', 'c')) # {'b': 456, 'c': 789} values_at(m, ('a', 'b')) # [123, 456]

6 5

Slice 'size' parameter (in the style of pairwise)
by Kevin 04 Jun '22

04 Jun '22

I might be posting this prematurely, but I had an idea and wanted to float it. Also, I'm new here so hopefully this is appropriate. How about augmenting slicing with an additional parameter 'size' (name chosen to achieve alliteration; 'start', 'stop', 'step', and 'size'), as such: >>> a_list = [0, 1, 2, 3, 4, 5, 6, 7] >>> a_list[1:5::4] # start=1, stop=5, step=1 (default), size=4 [(1, 2, 3, 4), (2, 3, 4, 5), (3, 4, 5, 6), (4, 5, 6, 7)] >>> a_list[::2:3] [(0, 1, 2), (2, 3, 4), (4, 5, 6)] # all tuples have length 3, even though data is left out >>> pairwise(a_list) == a_list[:::2] True >>> a_list[2:4] # step and size default to 1; size=1 returns the individual list entries themselves not tuples [2, 3, 4] >>> a_list = urandom(1024) >>> a_list[::16:16] # 64 chunks of 16 bytes, no overlapping chunks In the case where the list is not evenly divisible by the provided parameters, I think for [] notation a guarantee that each item in the returned iterator will have len(item) == size is appropriate, rather than returning all the data even if len(item) < size; slice() could have enhanced parameters over and above [] notation that could allow for returning "deficient" items. A negative value size parameter could also be possible: >>> a_list = [0, 1, 2, 3, 4, 5, 6, 7] >>> a_list[:::-2] [(1, 0), (2, 1), (3, 2), (4, 3), (5, 4), (6, 5), (7, 6)] Thoughts? < Flash forward to 200 years from now. > >>> a_list[::::7:::::::::::::-1::] (42, 'of course!')

2 1

Expand the try-expect syntax to support conditional expect block
by Nadav Misgav 04 Jun '22

04 Jun '22

Hello, I was thinking of a way to expand the current try-expect block to support condinitial expect block. I was imagining the new syntax as try: expression block expect Exception if condition expression block where the expect expression block would only run if the condition is met else it would raise the exception, for example the following code try: config = get_config() except Exception: if use_default: config = default else: raise would be converted to try: config = get_config() except Exception if use_default: config = default I find it very useful and also I found the same pattern in the following packages that were installed on my system. more_itertools/more.py:227 cloudinit/net/__init__.py:364 matplotlib/__init__.py:330 google/protobuf/internal/containers.py:126 networkx/algorithms/connectivity/edge_augmentation.py:296 twisted/internet/pollreactor.py:160 pyqtgraph/opengl/shaders.py:257 setuptools/_vendor/pyparsing.py:2191 setuptools/sandbox.py:253 pip/_internal/utils/misc.py:115 urllib3/poolmanager.py:362 mako/util.py:65 serial/serialposix.py:294 PIL/PngImagePlugin.py:335 click/core.py:1472 numpy/core/memmap.py:216 pyparsing.py:2067 Looking forward to hear some feedback, Regards

8 11

Means of avoiding accidental import of/from an Implicit Namespace Package
by Steve Jorgensen 02 Jun '22

02 Jun '22

More than once, I've had bugs that were hard to track down because I was accidentally using an implicit namespace without realizing it. The last time this happened, it was a typo, and my init file was named `_init__.py` instead of `__init__.py`. The init file imported from sub-modules, including 1 with a class that was supposed be be registered via an `__init_subclass__` callback that was not happening. I'm sure that implicit namespace packages are here to stay, and I imagine I will actually want to use them on purpose at some point, but it would be nice if we could come up with a straightforward way to avoid the accidental usages. One idea that comes to mind is to add a new built-in context manager within which the importing of a purely implicit namespace raises an exception.

1 0

Opt-in “iter def” and/or “gen def” syntax for generator functions
by armload_lakes0c＠icloud.com 01 Jun '22

01 Jun '22

After getting used to writing async functions, I’ve been wanting use a similar syntax to declare generator functions. Something along the lines of `iter def my_iterator() -> T` and/or `gen def my_generator() -> (T, U, V)` Obviously, for backwards compatibility, this would need to be optional or have an opt-in mechanism. Would a feature like this be at all within the realm of possibility? I’d be happy to write up a much longer discussion if so. (I found a short discussion of such a feature in the archives about 8 years ago[1]. But, since it predates both `async def` and the current type checker regime, I thought it might be worth discussing. Apologies if I missed any more recent discussions.) Thanks, Aaron [1] https://mail.python.org/archives/list/python-ideas@python.org/thread/OVIHVR…

6 9

Python array multiplication for negative values should throw an error
by fjwillemsen＠icloud.com 01 Jun '22

01 Jun '22

Hopefully this is not a duplicate of an existing thread or in the wrong section, first time posting here. In Python, array multiplication is quite useful to repeat an existing array, e.g. [1,2,3] * 2 becomes [1,2,3,4,5,6]. Multiplication by 0 yields an empty array: [1,2,3] * 0 becomes []. However, operations such as [numpy array] * -1 are very common to get the inverse of an array of numbers. The confusion here stems from the lack of type checking: while the programmer should check whether the array is a NumPy array or a Python array, this is not always done, giving rise to difficult to trace cases where [1,2,3] * -1 yields [] instead of [-1,-2,-3]. I can not think of good reasons why Python array multiplication should not throw an error for negative multipliers, because it is meaningless to have array multiplication by negative value in the way it is intended in Python. Instead I would propose that array multiplication by negative value throws an error. I would like to hear your opinions on this matter.

7 8