Mailman 3 July 2019 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts?

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman

12 35

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Add the imath module
by Serhiy Storchaka March 22, 2020

March 22, 2020

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math module. * as_integer_ration(x) Equivalents to: def as_integer_ration(x): if hasattr(x, 'as_integer_ration'): return x.as_integer_ration() else: return (x.numerator, x.denominator) * binom(n, k) Returns factorial(n) // (factorial(k) * factorial(n-k)), but uses more efficient algorithm. * sqrt(n) Returns the largest integer r such that r**2 <= n and (r+1)**2 > n. * isprime(n) Tests if n is a prime number. * primes() Returns an iterator of prime numbers: 2, 3, 5, 7, 11, 13,... Are there more ideas?

14 38

New explicit methods to trim strings
by Alex Grigoryev March 21, 2020

March 21, 2020

Following the discussion here (https://link.getmailspring.com/link/7D84D131-65B6-4EF7-9C43-51957F9DFAA9@ge…) I propose to add 3 new string methods: str.trim, str.ltrim, str.rtrim Another option would be to change API for str.split method to work correctly with sequences. In [1]: def ltrim(s, seq): ...: return s[len(seq):] if s.startswith(seq) else s ...: In [2]: def rtrim(s, seq): ...: return s[:-len(seq)] if s.endswith(seq) else s ...: In [3]: def trim(s, seq): ...: return ltrim(rtrim(s, seq), seq) ...: In [4]: s = 'mailto:maria@gmail.com' In [5]: ltrim(s, 'mailto:') Out[5]: 'maria(a)gmail.com' In [6]: rtrim(s, 'com') Out[6]: 'mailto:maria@gmail.' In [7]: trim(s, 'm') Out[7]: 'ailto:maria@gmail.co'

28 162

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar March 7, 2020

March 7, 2020

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to allow > > item = object[1, *some_tuple, 2] > item = object[1, *some_tuple, :] > > Was this overlooked in the original proposal, or deliberately rejected? > > Eric >

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Fixed point format for numbers with locale based separators
by Łukasz Stelmach Dec. 20, 2019

Dec. 20, 2019

Hi, I would like to present two pull requests[1][2] implementing fixed point presentation of numbers and ask for comments. The first is mine. I learnt about the second after publishing mine. The only format using decimal separator from locale data for float/complex/decimal numbers at the moment is "n" which behaves like "g". The drawback of these formats, I would like to overcome, is the inability to print numbers ranging more than one order of magnitude with the same number of decimal digits without "manually" (with some additional custom code) adjusting precission. The other option is to "manually" replace "." as printed by "f" with a local decimal separator. Neither of these option is appealing to my. Formatting 1.23456789 * n (LC_ALL=3Dpl_PL.UTF-8) | n | ".2f" | ".3n" | |---+----------+----------| | 1 | 1.23 | 1,23 | | 2 | 12.35 | 12,3 | | 3 | 123.46 | 123 | | 4 | 1234.57 | 1,23e+03 | In the application I want to create I am going to present users numbers ranging up to 3 orders of magnitude and I (my users) want them to be presented consistently with regards to number of decimal digits AND I want to conform to rules of languages of my users. And I would like to avoid the exponent notation by all means. I can't say much about James Emerton's implementation or his intentions, but please take a look at our patches and give your comments so either of us or together we can implement this feature. PS. In theory both implementations could be merged because James chose to use "l" to use LC_MONETARY category and I chose "m" to use LC_NUMERIC. [1] https://github.com/python/cpython/pull/11405 [2] https://github.com/python/cpython/pull/8612 -- Miłego dnia, Łukasz Stelmach

5 9

PEP's shouldn't require a sponsor
by Batuhan Taskaya Dec. 3, 2019

Dec. 3, 2019

Why do i need to convince a core developer for my PEP? AFAIK the steering council can include non core developers (i know it isn't that current case but for the future this is important). And if the last authority who will approve my PEP is the steering council i just need to convince them not core developers. Sponsors can stay (and they should because guidance is important) but thy shouldn't be mandatory. Let everyone to send their peps.

22 29

Please consider adding of functions file system operations to pathlib
by George Fischhof Nov. 16, 2019

Nov. 16, 2019

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger Oct. 23, 2019

Oct. 23, 2019

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel

17 49

Why not accept lists or arbitrary iterables in str.endswith?
by Soni L. Oct. 14, 2019

Oct. 14, 2019

I'm parsing configs for domain filtering rules, and they come as a list. However, str.endswith requires a tuple. So I need to use str.endswith(tuple(list)). I don't know the reasoning for this, but why not just accept a list as well?

3 6

Skip modules by default in star-import
by Serhiy Storchaka Sept. 19, 2019

Sept. 19, 2019

From the description of the import statement in the language reference [1]: > The *public names* defined by a module are determined by checking the module's > namespace for a variable named ``__all__``; if defined, it must be a sequence > of strings which are names defined or imported by that module. The names > given in ``__all__`` are all considered public and are required to exist. If > ``__all__`` is not defined, the set of public names includes all names found > in the module's namespace which do not begin with an underscore character > (``'_'``). ``__all__`` should contain the entire public API. It is intended > to avoid accidentally exporting items that are not part of the API (such as > library modules which were imported and used within the module). As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident. For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1] There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example: >>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree <module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'> >>> ElementPath <module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'> Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good. I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules. In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`. [1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446

9 16

Namespace context managers
by Batuhan Taskaya Aug. 10, 2019

Aug. 10, 2019

I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects. It would make closures possible in python with a pythonic syntax. a = 4 namespace = {} with namespace: a = 3 assert a == 4 assert namespace["a"] == 3

15 32

Universal parsing library in the stdlib to alleviate security issues
by Nam Nguyen Aug. 9, 2019

Aug. 9, 2019

Hello list, I sent an email to this list two or three months ago about the same idea. In that discussion, there were both skepticism and support. Since I had some time during the previous long weekend, I have made my idea more concrete and I thought I would try with the list again, after having run it through some of you privately. GOAL: To have some parsing primitives in the stdlib so that other modules in the stdlib itself can make use of. This would alleviate various security issues we have seen throughout the years. With that goal in mind, I opine that any parsing library for this purpose should have the following characteristics: #. Can be expressed in code. My opinion is that it is hard to review generated code. Code review is even more crucial in security contexts. #. Small and verifiable. This helps build trust in the code that is meant to plug security holes. #. Less evolving. Being in the stdlib has its drawback that is development velocity. The library should be theoretically sound and stable from the beginning. #. Universal. Most of the times we'll parse left-factored context-free grammars, but sometimes we'll also want to parse context-sensitive grammars such as short XML fragments in which end tags must match start tags. I have implemented a tiny (~200 SLOCs) package at https://gitlab.com/nam-nguyen/parser_compynator that demonstrates something like this is possible. There are several examples for you to have a feel of it, as well as some early benchmark numbers to consider. This is far smaller than any of the Python parsing libraries I have looked at, yet more universal than many of them. I hope that it would convert the skeptics ;). Finally, my request to the list is: Please debate on: 1) whether we want a small (even private, underscore prefixed) parsing library in the stdlib to help with tasks that are a little too complex for regexes, and 2) if yes, how should it look like? I also welcome comments (naming, uses of operator overloading, features, bikeshedding, etc.) on the above package ;). Thanks! Nam

12 31

For-expression/throwaway comprehension
by Eli Berkowitz Aug. 7, 2019

Aug. 7, 2019

(This is my first time posting on any Python list; I've tried to search for this idea and didn't find it but if I looked in the wrong places/this has already been discussed I apologize and feel free to tell me!) Say you have a list and you want to perform some operation on each item in the list - but you don't need to store the result in a list. There are three simple ways of doing this, at least as far as I know: ([print(item)] could be any expression, just using it as an example) ``` lst = [1, 2, 3, 4] #1 for item in lst: print(item) # 2 [print(item) for item in lst] # 3 for item in lst: print(item) ``` #1 - In my opinion, this should be a one line operation so #1 is not ideal. #2 - It also shouldn't require storing results in array, to save time/memory, so #2 is out. #3 - I think #3 is just not good syntax, it seems very unpythonic to me - it breaks the norm that blocks go on their own lines. It does seem the best of the three though and I know my assessment is kind of subjective. I'm wondering if a possible alternative syntax could be a for-expression, like there's if-expressions, which always evaluates to None: ``` print(item) for item in lst ``` A more practical example of when this would be useful is extending list-2 with a modified version of list-1 - this syntax would avoid creating an intermediate list (not sure if the way lists are implemented in python removes this advantage by the way it resizes lists though). ``` lst1 = [1, 2, 3] lst2 = [4, 5, 6] lst1.append(item * 2) for item in lst1 ```

23 76

Non-standard evaluation for Python
by Nima Hamidi Aug. 5, 2019

Aug. 5, 2019

Hello all, I would like to briefly share my thoughts on non-standard evaluation (NSE), why this is useful, and how, potentially, it can be added to Python. In most languages, functions have access only to the *value* of their arguments, and not to the expressions that yielded those values. However, sometimes it is useful to have access to the expressions as well. For example, let plot(x, y, ...) be a function that draws the scatter plot of y against x and adds labels for the x- and y-axes. Currently, there is no way to distinguish plot(x, *z*, ...) from plot(x, *y*, ...) from within the function. As such, one needs to pass the names of the variables to the plot function to be used as the axes' labels. Using NSE, though, the function can look at the expressions in the method call and use those as the default value for labels. In R, this idea is used widely to make the syntax simpler. In the following, I sketch how I think this can be implemented: 1. Let BoundExpression be a class containing an ast.Expression as well as locals and globals dictionaries. BoundExpression can also have an eval method that evaluates its expression using its locals and globals dictionaries. 2. Let `x` be the short form for BoundExpression("x", locals(), globals()). Then, the plot function can be implemented in a way that plot(`x`, `y`) draws scatter plot of x and y *and also,* labels the axes correctly. No need to provide labels explicitly anymore. 3. A more challenging idea is to let developers decide whether their functions need NSE or not. For example, when a function is defined as def f(`x`), for any method call like f(x), the first argument should be wrapped with a BoundExpression instance. I would appreciate any feedback on these ideas. Best, Nima

5 18

asyncio: futures and tasks with synchronous callbacks
by aurelien.lambert.89＠gmail.com Aug. 1, 2019

Aug. 1, 2019

In asyncio, when a task awaits for another task (or future), it can be cancelled right after the awaited task finished (before the callback have been processed). Thus, if the awaited task has consumed data, the data is lost. For instance, with the following code: import asyncio available_data = [] data_ready = asyncio.Future() def feed_data(data): global data_ready available_data.append(data) data_ready.set_result(None) data_ready = asyncio.Future() async def consume_data(): while not available_data: await asyncio.shield(data_ready) return available_data.pop() async def wrapped_consumer(): task = asyncio.ensure_future(consume_data()) return await task If I perform those exact steps: async def test(): task = asyncio.ensure_future(wrapped_consumer()) await asyncio.sleep(0) feed_data('data') await asyncio.sleep(0) task.cancel() await asyncio.sleep(0) print ('task', task) print ('available_data', available_data) loop = asyncio.get_event_loop() loop.run_until_complete(test()) Then I can see that the task has been cancelled despite the data being consumed. Since the result of `wrapped_consumer` cannot be retrieved, the data is forever lost. task <Task cancelled coro=<wrapped_consumer() done, defined at <ipython-input-1-de4ad193b1d0>:17>> available_data [] This side effect does not happen when awaiting a coroutine, but coroutine are not as flexible as tasks. It happens when awaiting a `Future`, a `Task`, or any function like `asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit from or use `Future`). There is then no way to do anything equivalent to: stop_future = asyncio.Future() async def wrapped_consumer2(): task = asyncio.ensure_future(consume_data()) try: await asyncio.wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') This is due to the Future calling the callback asynchronously: https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214 for callback in callbacks: self._loop.call_soon(callback, self) I propose to create synchronous versions of those, or a `synchronous_callback` parameter, that turns the callbacks of `Future` synchronous. I've experimented a simple librairy `syncio` with CPython 3.6 to do this (it is harder to patch later versions due to the massive use of private methods). Basically, needs to: 1) replace the `Future._schedule_callbacks` method by a synchronous version 2) fix `Task._step` to not fail when cleaning `_current_tasks` (https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245) 3) rewrite all the functions to use synchronous futures instead of normal ones With that librairy, the previous functions are possible and intuitive import syncio async def wrapped_consumer(): task = syncio.ensure_sync_future(consume_data()) return await task stop_future = asyncio.Future() async def wrapped_consumer2(): task = syncio.ensure_sync_future(consume_data()) try: await syncio.sync_wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') No need to use `syncio` anywhere else in the code, which makes it totally transparent for the end user. This "library" can be found here: https://github.com/aure-olli/aiokafka/blob/216b4ad3b4115bc9fa463e44fe23636b… It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`, `sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6 only. To conclude: - asynchronous callbacks are preferrable in most case, but do not provide a coherent cancelled status in specific cases - implementing a version with synchronous callback (or a `synchronous_callback` parameter) is rather easy (however step 2 need to be clarified, probably a cleaner way to fix this) - it is totally transparent for the end user, as synchronous callback are totally compatible with asynchronous ones

4 6

Re: Entrypoint function for modules (AKA if __name__ == '__main__' ) with built-in argument parsing
by Agustín Scaramuzza July 31, 2019

July 31, 2019

Ouch! That's what happens when you don't proof read your emails :S Replace --verbose with --print_operation, the idea of the snippet was to show an example for keyword parameters.

2 1

for ... except, with ... except
by Serhiy Storchaka July 31, 2019

July 31, 2019

Python allows you to write code in tight and readable form. Consider the following example. with connect() as stream: # connect() or __enter__() can fail. for data in stream: # __next__() can fail write(data) # write() can fail The problem is that different lines can raise an exception of the same type (for example OSError). We want to catch and handle exceptions raised when open a connection, when read a data and when write a data in different ways. Currently you need to expand so convenient Python statements "with" and "for" (see PEP 343) _mgr = connect() _enter = type(_mgr).__enter__ _exit = type(_mgr).__exit__ _value = _enter(_mgr) _exc = True try: stream = _value _it = iter(stream) while True: try: data = next(_it) except StopIteration: break write(data) except: _exc = False if not _exit(_mgr, *sys.exc_info()): raise finally: if _exc: _exit(_mgr, None, None, None) and then add "try ... except" around corresponding explicit calls of `__enter__()` and `next()`. try: _mgr = connect() _enter = type(_mgr).__enter__ _exit = type(_mgr).__exit__ _value = _enter(_mgr) _exc = True except OSError: handle_connection_error() else: try: stream = _value try: _it = iter(stream) except OSError: handle_read_error() else: while True: try: data = next(_it) except StopIteration: break except OSError: handle_read_error() break try: write(data) except OSError: handle_write_error() except: _exc = False if not _exit(_mgr, *sys.exc_info()): raise finally: if _exc: _exit(_mgr, None, None, None) Does not it look ugly? I propose to add "except" clause to "for" and "with" statement to catch exceptions in the code that can't be wrapped with "try ... except". for VAR in EXPR: BLOCK except EXC: HANDLER should be equivalent to try: _it = iter(EXPR) except EXC: HANDLER else: while True: try: VAR = next(_it) except StopIteration: break except EXC: HANDLER break BLOCK and with EXPR as VAR: BLOCK except EXC: HANDLER try: _mgr = EXPR _enter = type(_mgr).__enter__ _exit = type(_mgr).__exit__ _value = _enter(_mgr) _exc = True except EXC: HANDLER else: try: VAR = _value BLOCK except: _exc = False if not _exit(_mgr, *sys.exc_info()): raise finally: if _exc: _exit(_mgr, None, None, None) And correspondingly for asynchronous versions "async for" and "async with". So you will be able to add errors handling like in: with connect() as stream: for data in stream: try: write(data) except OSError: handle_write_error() except OSError: handle_read_error() except OSError: handle_connection_error()

16 31

Entrypoint function for modules (AKA if __name__ == '__main__' ) with built-in argument parsing
by agustinscaramuzza＠gmail.com July 31, 2019

July 31, 2019

Maybe the def __main__() argument is already a dead horse, given the number of discussions it has created that have ended nowhere, but I think one argument in favour of its implementation would be including argument parsing in it, for example: # main.py def __run__(first_num, second_num, print_operation=False): """ Adds two numbers. positional arguments: - first_num: the first number. - second_num: the second number. optional arguments: - print_operation: prints the sum operation to stdout. """ result = int(first_num) + int(second_num) if print_operation: print(f'{first_num} + {second_num} = {result}') else: print(result) $ python main.py -h Adds two numbers. positional parameters: - first_num: the first number. - second_num: the second number. optional arguments: - print_operation: prints the sum operation to stdout. $ python main.py 1 2 --verbose 1 + 2 = 3 $ python main.py 1 2 3 The -h flag would print the function docstring. We could also add type hints for casting (I'm not entirely sure how feasible this is): def __run__(first_num: int, second_num: int, print_operation=False): ... result = first_num + second_num # no need for casting here ... Since __main__ is already a built-in function, I abstained from using it as the designated function name (I picked __run__, but any other suggestions would be welcome, also thought of __entrypoint__, __exec__). Thoughts?

4 5

asyncio futures and tasks with synchronous callbacks
by Aurélien Lambert July 31, 2019

July 31, 2019

In asyncio, when a task awaits for another task (or future), it can be cancelled right after the awaited task finished. Thus, if the awaited task has consumed data, the data is lost. For instance, with the following code: import asyncio available_data = [] data_ready = asyncio.Future() def feed_data(data): global data_ready available_data.append(data) data_ready.set_result(None) data_ready = asyncio.Future() async def consume_data(): while not available_data: await asyncio.shield(data_ready) return available_data.pop() async def wrapped_consumer(): task = asyncio.ensure_future(consume_data()) return await task If I perform those exact steps: async def test(): task = asyncio.ensure_future(wrapped_consumer()) await asyncio.sleep(0) feed_data('data') await asyncio.sleep(0) task.cancel() await asyncio.sleep(0) print ('task', task) print ('available_data', available_data) loop = asyncio.get_event_loop() loop.run_until_complete(test()) Then I can see that the task has been cancelled despite the data being consumed. Since the result of `wrapped_consumer` cannot be retrieved, the data is forever lost. task <Task cancelled coro=<wrapped_consumer() done, defined at <ipython-input-1-de4ad193b1d0>:17>> available_data [] This side effect does not happen when awaiting a coroutine, but coroutines are not as flexible as tasks (unless manipulated as a generator). It happens when awaiting a `Future`, a `Task`, or any function like `asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit from or use `Future`). There is then no way to do anything equivalent to: stop_future = asyncio.Future() async def wrapped_consumer2(): task = asyncio.ensure_future(consume_data()) try: await asyncio.wait([task, stop_future]) finally: task.cancel() if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') This is due to the Future calling the callback asynchronously: https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214 for callback in callbacks: self._loop.call_soon(callback, self) I propose to create synchronous versions of those, or a `synchronous_callback` parameter, that turns the callbacks of `Future` synchronous. I've experimented a simple library `syncio` with CPython 3.6 to do this (it is harder to patch later versions due to the massive use of private methods). Basically, needs to: 1) replace the `Future._schedule_callbacks` method by a synchronous version 2) fix `Task._step` to not fail when cleaning `_current_tasks` ( https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245) 3) rewrite all the functions to use synchronous futures instead of normal ones With that library, the previous functions are possible and intuitive import syncio async def wrapped_consumer(): task = syncio.ensure_sync_future(consume_data()) return await task stop_future = asyncio.Future() async def wrapped_consumer2(): task = syncio.ensure_sync_future(consume_data()) try: await syncio.sync_wait([task, stop_future]) finally: task.cancel() if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') No need to use `syncio` anywhere else in the code, which makes it totally transparent for the end user. `wrapped_consumer` and `wrapped_consumer2` are now cancelled if and only if the data hasn't been consumed, whatever is the order of the steps (and the presence of `asyncio.sleep`). This "library" can be found here: https://github.com/aure-olli/aiokafka/blob/3acb88d6ece4502a78e230b234f47b90… It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`, `sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6. To conclude: - asynchronous callbacks are preferable in most cases, but do not provide a coherent cancelled status in specific cases - implementing a version with synchronous callback (or a `synchronous_callback` parameter) is rather easy (however step 2 need to be clarified, probably a cleaner way to fix this) - it is totally transparent for the end user, as synchronous callbacks are totally compatible with asynchronous ones

1 0

Cartesian Product on `__mul__`
by Batuhan Taskaya July 30, 2019

July 30, 2019

I think it looks very fine when you type {1, 2, 3} * {"a", "b", "c"} and get set(itertools.product({1, 2, 3}, {"a", "b", "c"})). So i am proposing set multiplication implementation as cartesian product. >>>

7 11

Fwd: for ... except, with ... except
by Bruce Leban July 29, 2019

July 29, 2019

And another message that was rejected (I sent from an unregistered email address) On Sat, Jul 27, 2019 at 1:49 AM Serhiy Storchaka <storchaka(a)gmail.com> wrote: > 26.07.19 21:52, Bruce Leban пише: > > > To put this in a simpler way: the proposal is to add an except clause that > applies ONLY to the direct operation of the with or for statement and not > to the block. That's an interesting idea. > > The one thing I find confusing about your proposal is that the proposed > syntax does not imply the behavior. In a try statement, the except appears > at the end and after all possible statements that it could cover. The > proposal mimics that syntax but with different semantics. Something like > this would be much more clear what is going on: > > for VARIABLE in EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK > > with EXPRESSION as VARIABLE: > except EXCEPTION: > BLOCK > BLOCK > > while EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK > > > Besides an unusual for Python layout (a clause has different indentation > than the initial clause of the statement to which it belongs) there is > other problem. The exception block is not the part of the "for" or "with" > block. After handling an exception in the "for" clause you do not continue > to execute the "for" block, but leave the loop. After handling an exception > in the "with" clause you do not continue to execute the "with" block and do > not call `__exit__` when leave it. To me, this syntax is much more > confusing than my initial proposition. > And I find it less confusing. And neither of those is the standard to use. The goal is for syntax to imply semantics (which my proposal does and I do not think yours does, given several people commenting that they thought it applied to the entire loop) and to choose syntax that is more clear to more people (which requires more than two peoples' opinions). Consider how you would write this if everything was an expression in Python and we had braces: for VAR in ( EXPR except EXCEPTION: { BLOCK; break; } ): BLOCK I do agree that it is not obvious that the exception block breaks out of the loop. I think in actual code it will be fairly obvious what's happening as continuing into the loop when the loop expression through an expression doesn't make sense. I'm open to alternatives. On the other hand, an except clause at the bottom of the loop that does not apply to the loop body is going to catch me every time I see it. --- Bruce

1 0

Fwd: for ... except, with ... except
by Bruce Leban July 29, 2019

July 29, 2019

I sent this message earlier but it was rejected by the mailer. On Fri, Jul 26, 2019 at 11:27 AM Serhiy Storchaka <storchaka(a)gmail.com> wrote: > > So you will be able to add errors handling like in: > > with connect() as stream: > for data in stream: > try: > write(data) > except OSError: > handle_write_error() > except OSError: > handle_read_error() > except OSError: > handle_connection_error() > To put this in a simpler way: the proposal is to add an except clause that applies ONLY to the direct operation of the with or for statement and not to the block. That's an interesting idea. The one thing I find confusing about your proposal is that the proposed syntax does not imply the behavior. In a try statement, the except appears at the end and after all possible statements that it could cover. The proposal mimics that syntax but with different semantics. Something like this would be much more clear what is going on: for VARIABLE in EXPRESSION: except EXCEPTION: BLOCK BLOCK with EXPRESSION as VARIABLE: except EXCEPTION: BLOCK BLOCK while EXPRESSION: except EXCEPTION: BLOCK BLOCK or rewriting your example: with connect() as stream: except OSError: handle_connection_error() for data in stream: except OSError: handle_read_error() try: write(data) except OSError: handle_write_error()

1 0

support toml for pyproject support
by Jimmy Girardet July 29, 2019

July 29, 2019

Hi, I don't know if this was already debated but I don't know how to search in the whole archive of the list. For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module. I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent. Maybe we could thought to a readonly lib to limit the added code. If it's conceivable, I'd be happy to help in it. Nice Day guys and girls. Jimmy

13 15

Fwd: Re: Universal parsing library in the stdlib to alleviate security issues
by Nam Nguyen July 29, 2019

July 29, 2019

Forward to the list because Abusix had blocked google.com initially. Nam ---------- Forwarded message --------- From: Nam Nguyen <bitsink(a)gmail.com> Date: Sun, Jul 28, 2019 at 10:18 AM Subject: Re: [Python-ideas] Re: Universal parsing library in the stdlib to alleviate security issues To: Sebastian Kreft <skreft(a)gmail.com> Cc: Paul Moore <p.f.moore(a)gmail.com>, python-ideas <python-ideas(a)python.org> Let's circle back to the beginning one last time ;). On Thu, Jul 25, 2019 at 8:15 AM Sebastian Kreft <skreft(a)gmail.com> wrote: > Nam, I think it'd be better to frame the proposal as a security > enhancement. Stating some of the common bugs/gotchas found when manually > implementing parsers, and the impact this has had on python over the years. > Seeing a full list of security issues (CVEs) by module would give us a > sense of how widespread the problem is. > Since my final exam was done this weekend, I gathered some more info into this spreadsheet. https://docs.google.com/spreadsheets/d/1TlWSf8iM7eIzEPXanJAP8Ztyzt4ZD28xFvU… I think a strict parser can help with the majority of those problems. They are in HTTP headers, emails, cookies, URLs, and even low level socket code (inet_atoi). > Then survey the stdlib for what kind of grammars are currently being > parsed, what ad-hoc parsing strategy are implemented and provide examples > of whether having a general purpose parser would have prevented the > security issues you have previously cited. > Most grammars I have seen here come straight from RFCs, which are in ABNF and thus context-free. Current implementations are based on regexes or string splitting. My previous example showed that at least 30500, 36216, 36742 were non-issues if we started out with a strict parser. > > Right now, it is not clear what the impact of such refactor would be, nor > the worth of such attempt. > Exactly the kind of response I'm looking for. It is okay to suggest that the benefits aren't clear or that there are requirements X and Y that a general parser won't be able to meet, but it's not convincing to brush aside this because there is "existing, working code." Many of the bugs in that sheet are still open. It's not comfortable to say the code is working with a straight face as I have experienced with my own fix for 30500. I just couldn't tell if it was doing the right thing. > > What others have said earlier is that you are the one that needs to > provide some of the requirements for the proposed private parsing library. > And from what I read from your emails you do have some ideas. For example, > you want it to be easy to write and review (I guess here you would > eventually like it to be a close translation from whatever is specified in > the RFC or grammar specification). > Yes, that's the most important point because "readability counts." It's hard to reason about correctness when there are many transformations between the authoritative spec and the implementation. I definitely don't want to touch the regexes, string splits, and custom logic that I don't understand "why" they are that way in the beginning. How do I, for example, know what this regex is about ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? (It's from RFC 3986.) But you also need to take into consideration some of the list's concerns, > the parser library has to be performant, as a performance regression is > likely not to be tolerable. > Absolutely. That's where I need inputs from the list. I have provided my own set of requirements for such a parser library. I'm sure most of us have different needs too. So if a parser library can help you, let's hear what you want from it. If you think it can't, please let me understand why. Thanks, Nam > > > On Thu, Jul 25, 2019 at 10:54 AM Nam Nguyen <bitsink(a)gmail.com> wrote: > >> On Thu, Jul 25, 2019 at 2:32 AM Paul Moore <p.f.moore(a)gmail.com> wrote: >> >>> On Thu, 25 Jul 2019 at 02:16, Nam Nguyen <bitsink(a)gmail.com> wrote: >>> > Back to my original requests to the list: 1) Whether we want to have a >>> (possibly private) parsing library in the stdlib >>> >>> In the abstract, no. Propose a specific library, and that answer would >>> change to "maybe". >>> >> >> I have no specific library to propose. I'm looking for a list of features >> such a library should have. >> >> >>> >>> > and 2) What features it should have. >>> >>> That question only makes sense if you get agreement to the abstract >>> proposal that "we should add a parsing library. And as I said, I don't >>> agree to that so I can't answer the second question. >>> >> >> As Chris summarized it correctly, I am advocating for a general solution >> to individual problems (which have the same nature). We can certainly solve >> the problems when they are reported, or we can take a proactive approach to >> make them less likely to occur. I am talking about a class of input >> validation issues here and I thought parsing would be a very natural >> solution to that. This is quite similar to a context-sensitive templating >> library that prevents cross-site-scripting on the output side. So I don't >> know why (or what it takes) to convince people that it's a good thing(tm). >> >> >>> >>> Generally, things go into the stdlib when they have been developed >>> externally and proved their value. The bar for designing a whole >>> library from scratch, "specifically" targeted at stdlib inclusion, is >>> very high, and you're nowhere near reaching it IMO. >>> >> >> This is a misunderstanding. I have not proposed any from-scratch, or >> existing library to be used. And on this note, please allow me to make it >> clear once more time that I am not asking for a publicly-facing library >> either. >> >> >>> >>> > These are good points to set as targets! What does it take for me to >>> get the list to agree on one such set of criteria? >>> >>> You need to start by getting agreement on the premise that adding a >>> newly-written parser to the stdlib is a good idea. And so far your >>> *only* argument seems to be that "it will avoid a class of security >>> bugs" which I find extremely unconvincing (and I get the impression >>> others do, too). >> >> >> Why? What is unconvincing about a parsing library being able... parse >> (and therefore, validate) inputs? >> >> >>> But even if "using a real parser" was useful in that >>> context, there's *still* no argument for writing one from scratch, >>> rather than using an existing, proven library. >> >> >> Never a goal. >> >> >>> At the most basic >>> level, what if there's a bug in your new parsing library? If we're >>> using it in security-critical code, such a bug would be a >>> vulnerability just like the ones you're suggesting your parser would >>> avoid. Are you asking us to believe that your code will be robust >>> enough to trust over code that's been used in production systems for >>> years? >>> >>> I think you need to stop getting distracted by details, and focus on >>> your stated initial request "Whether we want to have a (possibly >>> private) parsing library in the stdlib". You don't seem to me to have >>> persuaded anyone of this basic suggestion yet, >> >> >> Good observation. How do I convince you that complex input validation >> tasks should be left to a parser? >> >> Thanks! >> Nam >> >> _______________________________________________ >> Python-ideas mailing list -- python-ideas(a)python.org >> To unsubscribe send an email to python-ideas-leave(a)python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/FCPU4… >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > Sebastian Kreft >

2 1

Utilities for easier debugging
by James Lu July 29, 2019

July 29, 2019

Minimal strawman proposal. New keyword debug. debug EXPRESSION Executes EXPRESSION when in debug mode. debug context Prints all the variables of the enclosing closure and all the variable names accessed within that block. For example, if in foo you access the global variable spam, spam would be printed. The format would be: variableName: value variableTwo: value where "value" is the repr() of the variable. Separated by new lines. The exact output format would not be part of the spec. ?identifier would print "identifier: value." Repr as before. Using this in non-debug mode emits a warning. ?identifier.property.property is also valid. A new property descriptor on the global variable, “debugger.” This is an alias for importing PDB and causing the debugger to pause there. The behavior of this descriptor in non-debug mode is TBD. Debug mode may be specified per-module at interpreter launch.

5 5

Fwd: Utilities for easier debugging
by James Lu July 28, 2019

July 28, 2019

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:22:11 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> This would break iPython’s improved interactive console, which already uses this syntax to provide a similar feature. > If it’s so similar, I doubt it would break anything. This is intended to make it easier to log variables for a complex application (multiple files).

1 0

Fwd: Utilities for easier debugging
by James Lu July 28, 2019

July 28, 2019

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:21:04 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> Why not just allow anything that’s valid as a target, like identifier[expr]? Or even just any expression at all? Is there an advantage to defining and using a similar but more limited syntax here? > Hmm, it should support any expression. I limited it at first because it was a minimal strawman.

1 0

Fwd: Utilities for easier debugging
by James Lu July 28, 2019

July 28, 2019

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:21:04 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> Why not just allow anything that’s valid as a target, like identifier[expr]? Or even just any expression at all? Is there an advantage to defining and using a similar but more limited syntax here? > Hmm, it should support any expression. I limited it at first because it was a minimal strawman.

1 0

Fwd: Utilities for easier debugging
by James Lu July 28, 2019

July 28, 2019

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:22:11 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> This would break iPython’s improved interactive console, which already uses this syntax to provide a similar feature. > If it’s so similar, I doubt it would break anything. This is intended to make it easier to log variables for a complex application (multiple files).

1 0

Fwd: Utilities for easier debugging
by James Lu July 28, 2019

July 28, 2019

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:23:06 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> You want this added to every module’s globals? Why not put it in builtins instead? > It’s not a function call, it’s an expression that when evaluated triggers a breakpoint, so you have to add it as a property descriptor.

1 0

Re: for ... except, with ... except
by Ethan Furman July 28, 2019

July 28, 2019

[redirecting back to the list] On 07/27/2019 09:38 PM, James Lu wrote: >> On Jul 27, 2019, at 12:44 PM, Ethan Furman wrote: >> Sure, folks /know/ what it means, but it's a common bug because >> it doesn't read as "if some_var is assigned 7" but as "if some_var >> is equal to 7". > > That’s a straw man- a modern linter or compiler would catch that. > Many editors have linters integrated inside. The way to disable it > is to put parentheses around the assignment expesssion.

1 0

Add a "partial with placeholders" function to the functools module
by Dominik Vilsmeier July 28, 2019

July 28, 2019

Hello, currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods. I propose adding a function to `functools` that works with placeholders and thus offers even greater flexibility. The Ellipsis literal `...` seems a intuitive choice for that task. When eventually calling such a "partial placeholder" object, it would fill in placeholders from the left and add remaining `args` to the right. In terms of implementation this can be realized as a subclass of `partial` itself. ## Implementation from functools import partial from reprlib import recursive_repr class partial_placehold(partial): placeholder = Ellipsis def __call__(self, /, *args, **keywords): args = iter(args) try: old_args = [x if x is not self.placeholder else next(args) for x in self.args] except StopIteration: raise TypeError('too few arguments were supplied') from None keywords = {**self.keywords, **keywords} return self.func(*old_args, *args, **keywords) @recursive_repr() def __repr__(self): qualname = type(self).__qualname__ args = [repr(self.func)] args.extend(repr(x) if x is not self.placeholder else '...' for x in self.args) # Only this line deviates from `partial.__repr__`; could also factor that out into a separate method. args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items()) if type(self).__module__ == "functools": return f"functools.{qualname}({', '.join(args)})" return f"{qualname}({', '.join(args)})" # Would need to add something for compatibility with `partial`, i.e. for partializing a placeholder function. ## Example This allows for example the following usage: replace_dots_with_underscore = partial_placehold(str.replace, ..., '.', '_') replace_dots_with_underscore('foo.bar.baz') ## Relevance Sure we could also use a `lambda` instead ([as discussed here](https://mail.python.org/archives/list/python-ideas@python.org/message…) but there was a reason `partial` was introduced and I think the same arguments apply here too. Though most functions allow partializing via keyword arguments and this is undoubtedly a cleaner way, some might not and for example built-ins' methods won't allow it. Especially Python 3.8's introduction of positional-only parameters (PEP 570) might give rise to cases where `partial` is not sufficient. In case inspection is desired a `lambda` does not provide much information (sure you could always dig deeper with `inspect` for example but that's not the point). Consider the following example of a pre-defined sequence of default postprocessing steps and the user might add their own or remove existing ones, as appropriate: postprocessing_steps = [ lambda s: s.replace('foo', 'bar'), ] print(postprocessing_steps[0]) # <function <lambda> at 0x7f94a850dd30> This doesn't give a lot of information about what the lambda actually does (and thus whether the user should remove it or not). Using the `partial_placehold` instead, it's clear what is happening: postprocessing_steps = [ partial_placehold(str.replace, ..., 'foo', 'bar'), ] print(postprocessing_steps[0]) # partial_placehold(<method 'replace' of 'str' objects>, ..., 'foo', 'bar') ## Compatibility The proposed solution works with the current syntax and the usage of Ellipsis as a placeholder object is likely not to collide with actually used values (in any case the user might still reassign the `.placeholder` attribute). Because the direction of partializing is unchanged (still left to right) this doesn't introduce ambiguities which might come with a "right partial" function. Creating a placeholder function from a `partial` object is possible without any changes, the opposite way requires an additional check to result in a placeholder object again. ## Possible confusion Regarding the usage of Ellipsis right now, in `numpy` or `typing` for example, it always represents a placeholder for multiple "things", not a single one: array[..., None] # All the dimensions of `array` plus a new one. typing.Tuple[str, ...] # Any number of str objects. So the expectations might be biased in that sense. For example: def foo(a, b, c, d): pass p_foo = partial_placehold(foo, ..., 1, 2) p_foo(3, 4) Someone else reviewing the code might now assume that the `...` means to act as a placeholder for all arguments except the last two (and hence `p_foo(3, 4)` would be equivalent to `foo(3, 4, 1, 2)` while it actually is equivalent to `foo(3, 1, 2, 4)`). But this would be again some kind of "right partial" function and also the function name implies something else; documentation might clarify as well, of course. ## Conclusion Adding a "partial with placeholders" function to `functools` allows for covering use cases where the standard `partial` is not sufficient. No new syntax is required and the implementation is fairly straightforward given the inheritance from `partial`. Ellipsis `...` seems an intuitive choice for acting as a placeholder (concerning both, conflicts with actual partial values and code readability). There are uses cases where such a function would provide a clean solution and there is an interest in the community (https://stackoverflow.com/q/7811247/3767239, https://stackoverflow.com/q/19701775/3767239 for example). Especially with the introduction of positional-only parameters new use cases are likely to arise. ----- **Related threads:** * https://mail.python.org/archives/list/python-ideas@python.org/message/TVNCM… - Mentions essentially a similar idea. The original [PEP 309 -- Partial Function Application](https://www.python.org/dev/peps/pep-0309/) also mentions: > Partially applying arguments from the right, or inserting arguments at arbitrary positions creates its own problems, but pending discovery of a good implementation and non-confusing semantics, I don't think it should be ruled out.

7 10

Re: for ... except, with ... except
by Serhiy Storchaka July 27, 2019

July 27, 2019

26.07.19 21:52, Bruce Leban пише: > > On Fri, Jul 26, 2019 at 11:27 AM Serhiy Storchaka <storchaka(a)gmail.com > <mailto:storchaka@gmail.com>> wrote: > > > So you will be able to add errors handling like in: > > with connect() as stream: > for data in stream: > try: > write(data) > except OSError: > handle_write_error() > except OSError: > handle_read_error() > except OSError: > handle_connection_error() > > > To put this in a simpler way: the proposal is to add an except clause > that applies ONLY to the direct operation of the with or for statement > and not to the block. That's an interesting idea. > > The one thing I find confusing about your proposal is that the > proposed syntax does not imply the behavior. In a try statement, the > except appears at the end and after all possible statements that it > could cover. The proposal mimics that syntax but with different > semantics. Something like this would be much more clear what is going on: > > for VARIABLE in EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK > > with EXPRESSION as VARIABLE: > except EXCEPTION: > BLOCK > BLOCK > > while EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK Besides an unusual for Python layout (a clause has different indentation than the initial clause of the statement to which it belongs) there is other problem. The exception block is not the part of the "for" or "with" block. After handling an exception in the "for" clause you do not continue to execute the "for" block, but leave the loop. After handling an exception in the "with" clause you do not continue to execute the "with" block and do not call `__exit__` when leave it. To me, this syntax is much more confusing than my initial proposition.

1 0

Scope painting
by Yonatan Zunger July 19, 2019

July 19, 2019

Hi everyone, I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope. *Conceptual TL;DR: *Add a function which attaches a string label to the current interpreter frame. (This would go away when you either explicitly cleared it, or the frame ended, i.e. the current function returned) You can then use it for: - *Debugging: *Print this value out, if set, in the traceback formatters. This can be used e.g. in a server, by putting a unique request identifier as the label for the top-level handler method; then if something in the particular query triggers a crash, it's *much* easier to see what caused it. (When I was at Google, we did something very similar to this for the servers in search. It completely transformed the way we detected "queries of death," and made hunting down data-dependent bugs an order of magnitude easier.) - *CPU and heap profiling: *If a profiler or tracer stores this information when it's grabbing the stack trace, this can be used for two different kinds of analysis, depending on user needs: - *Splitting: *Consider two stack traces different if they differ in any of the labels. This lets you separate flows which seem to be going through the same part of the logic, but are actually doing it on very distinct paths. - *Joining: *Group stack traces by the label; this lets you identify "call with this label value and its descendants," which lets you very easily establish a user-defined aggregation for total CPU or heap usage. - *Network request tracing: *If you were feeling really fancy, you could attach this to a cross-network-request trace ID and propagate the same value of this across multiple servers in a complicated request chain. Then you could perform all of the above joinings, plus similar joinings from cross-server profiling systems, as well. *Implementation: *Pretty simple: add a new field PyObject *f_label to PyFrameObject which contains an optional Unicode string; the frame owns a reference. Add get and set methods, exposing those in the Python API, and probably also a simple context manager to set and restore the value of this field because that's going to be the 90% way it's used. (Based on experience with doing similar things in C++) Modify the traceback, profile, cProfile, and tracemalloc libraries (and maybe others?) to use this value as well. What do people think? Yonatan

3 6

Stdlib Module
by Abdur-Rahmaan Janhangeer July 17, 2019

July 17, 2019

Greetings, Just like builtins module, would it be a good idea to have a stdlib module so that we can know all modules in the std lib (just like builtins let us know builtins)? Abdur-Rahmaan Janhangeer Mauritius

6 18

Should tempfile.NamedTemporaryFile() not raise FileNotFoundError if the file does not exist at the moment of closing?
by Andrew Pashkin July 17, 2019

July 17, 2019

What does everyone think about making "tempfile.NamedTemporaryFile()" not raising "FileNotFoundError" in case if when it closes and the file is already deleted?

2 1

Adding where keyword inside list coprehensions
by Fabrizio Messina July 17, 2019

July 17, 2019

Hello, It is likely not the first time such a proposal is being made but let's see. I would like to explicitly set variable names in list comprehensions using a where keyword, eventually set after the if keyword: [price for item in basket if price is not None where price := item.get( 'price')] For this example one could use the walrus operator which is indeed smaller: [price for item in basket if price := item.get('price') is not None] But I fell that this approach is a bit opportunistic as one is doing two things at a time, if you allow me more lines for a somewhat more complex example: [ price * (1 + vat) for item in basket if price is not None where price := item.get('price') vat := get_vat(item, user) ] Now this example may look pretty stupid and probably one may simply use for loop, but I feel that this kind of Haskell-like where inside of list comprehension will let the programmer have a space where they can explicitly state the variables. Best regards, Fabrizio

3 2

Fwd: Adding where keyword inside list coprehensions
by David Mertz July 15, 2019

July 15, 2019

This comes up pretty often. Every example can be replaced by a loop over a single item list. It's a bit idiomatic, but not difficult. Many, as you note, can use the new walrus operator instead. [price for price in [item.get('price')] for item in basket if price is not None] On Mon, Jul 15, 2019 at 10:21 AM Fabrizio Messina <zauddelig(a)gmail.com> wrote: > Hello, > > It is likely not the first time such a proposal is being made but let's > see. > I would like to explicitly set variable names in list comprehensions using > a where keyword, eventually set after the if keyword: > > [price for item in basket if price is not None where price := item.get( > 'price')] > > For this example one could use the walrus operator which is indeed > smaller: > [price for item in basket if price := item.get('price') is not None] > > But I fell that this approach is a bit opportunistic as one is doing two > things at a time, if you allow me more lines for a somewhat more complex > example: > > [ > price * (1 + vat) > for > item in basket > if > price is not None > where > price := item.get('price') > vat := get_vat(item, user) > ] > > Now this example may look pretty stupid and probably one may simply use > for loop, but I feel that this kind of Haskell-like where inside of list > comprehension will let the programmer have a space where they can > explicitly state the variables. > > Best regards, > Fabrizio > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/AHI6N… > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

1 0

Coding using Unicode
by Adrien Ricocotam July 15, 2019

July 15, 2019

Hi all, What would you think if we could write our code using unicode ? It would be especially useful for scientific programming (we could use the greek letters), it could also be nice to use emojis for some variables. I don't see any bad consequences (apart from people that would use misleading characters but that's already possible). I don't know how hard it is to do that switch but it feels like it could be nice; Cheers

10 12

Re: Non-standard evaluation for Python
by Andrew Barnert July 15, 2019

July 15, 2019

(Re-sending, because this was originally a reply to an off-list message by Nima Hamidi) On Jul 13, 2019, at 14:12, Nima Hamidi <hamidi(a)stanford.edu> wrote: > > Sometimes it's necessary not to evaluate the expression. Two such applications of NSE in R are as follows: > > 1. Data-tables have cleaner syntax. For example, letting dt be a data-table with a column called price, one can retrieve items cheaper than $1 using the following: dt [price < 1]. Pandas syntax requires something like dt[dt.price < 1]. This is currently inevitable as the expression is evaluated *before* __getitem__ is invoked. Using NSE, dt.__getitem__ can, first, add its columns to locals() dictionary and then evaluate the expression in the new context. This one looks good. I can also imagine it being useful for SQLAlchemy, appscript, etc. just as it is for Pandas. But in your proposal, wouldn’t this have to be written as dt[`price < 1`]? I think the cost of putting the expression in ticks is at least as bad as the cost of naming the dt. Also: dt.price < 1 is a perfectly valid expression, with a useful value. You can store it in a temporary variable to avoid repeating it, or stash it for later, or print it out to see what’s happening. But price < 1 on its own is a NameError, and I’m not sure what `price < 1` is worth on its own. Would this invite code that’s hard to refactor and even harder to debug? > 2. Pipe-lining in R is also much cleaner. Dplyr provided an operator %>% which passes the return value of its LHS as the first argument of its RHS. In other words, f() %>% g() is equivalent to g(f()). This is pretty useful for long pipelines. The way that it works is that the operator %>% changes AST and then evaluates the modified expression. In this example, evaluating g() is undesirable. This doesn’t seem necessary in a language with first-class functions. Why can’t you just write the pipeline as something f %>% g, much as you would in, say, Haskell, which would just define a function (presumably equivalent to either lambda: g(f()) or lambda *a, **kw: g(f(*a, **kw))) that represents the pipeline that you then just call normally? I don’t see the benefit in being able to write g() instead of g here, and in fact it seems actively misleading, because it implies calling g on no arguments instead of one. Also, given that Python doesn’t have such an operator, or a way to define custom operators, and that proposals for even simpler operators on functions like @ for compose have been rejected every time they’ve been suggested, I wouldn’t expect much traction from this example. Is there something similar that could plausibly be done in Python, and feel Pythonic? —- A couple more things I thought of since the initial reply… I’m pretty sure Python’s AST objects don’t contain the original source text. So, what is your plot function going to actually do with its arguments to get the axes? What if it’s called with plot(`x[..., 3:]`)? Will plot—and every other function that wants to do something similar—need to come up with a way to generate the nicest source text that could produce the given AST? Or do we need to add a decompile to the stdlib for them? I suppose you could solve this by just adding more fields to BoundExpression, but I’m not sure that wouldn’t make it a lot harder to implement the backtick feature. Backticks are supposed to be banned for the life of Python since 3.0 eliminated them as shorthand for repr. That could be revisited, but it might be a tough sell. Maybe the original “grit on Tim’s screen” reason is no longer as compelling because of higher-res screens and more uniform console fonts, but the rise to ubiquity of markdown seems like an even better reason not to use them. Today, you can paste Python code between backticks to mark it as code in markdown; if Python code can contain backticks, that’s no longer true. People who use languages that rely on backticks have been complaining about that for years; so we want to join them? Finally, I think you need a fully worked-through example, not just a description of one. Show what the implementation of plot would look like if it could be handed BoundExpression objects. (Although pd.DataFrame.__getitem__ seems like the killer use case here, so maybe show that one instead, even though it’s probably more complicated.)

4 9

Make tuple a context manager
by haael July 14, 2019

July 14, 2019

Could we add __enter__ and __exit__ to the tuple type? Look at the following code: a = open('a.tmp', 'w') b = open('b.tmp', 'w') with (a, b) as (af, bf): af.write("1") bf.write("2") Even better example: with tuple(open(str(_n) + '.tmp', 'w') for _n in range(1000)) as f: for n, fn in enumerate(f): f.write(str(n)) Tuple as context manager would invoke __enter__ for each of its elements and return a tuple of the results. On exit, the __exit__ method would be invoked for every element. We could even generalize it to every kind of iterable. This is somewhat consistent with treatment of exception types in 'except' clause. try: something() except Exception1 as error: handlerA(error) except (Exception2, Exception3) as error: handlerB(error) Tuple of exception types is accepted in 'except' clause, as well as a single exception type. We could apply that rule to the 'with' clause.

6 8

Administrivia update: moderating subscriptions, and (some) new subscribers
by C. Titus Brown July 12, 2019

July 12, 2019

Hi all, in response to some of the recent spam, Brett and I have decided to put in place moderation for subscriptions, as well as moderating some of the newly subscribed accounts. We feel that moderation of all messages is too much, given the (occasional) high volume of python-ideas, and its utility as a free-form discussion forum. best, —titus

1 0

Suggestion: Windows launcher default to not using pre-releases by default
by Steve Barnes July 11, 2019

July 11, 2019

Currently the py[w] command will launch the latest python by default however I feel that this discourages the testing of pre-releases & release candidates as once they are installed they will become the default. What I would like is for the default to be the highest version number of a full release but the user to be able to specify a specific version even if it is a pre-release. The currently py or py -3 would give python 3.7 (if installed) but py -3.8 would give the pre-release/release candidate if installed. Any thoughts on whether this would be a good idea - I am quite willing to undertake the changes. Steve Barnes

10 17

Shouldn't this: ' myVar: "value" ' be a syntax error ?
by Shay Cohen July 10, 2019

July 10, 2019

>>> a: 3 >>> type(a: 3) File "<stdin>", line 1 type(a: 3) ^ SyntaxError: invalid syntax >>> a: 3 >>> type(a) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'a' is not defined >>> a Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'a' is not defined

4 3

Sildenafil is the best medicine to cure ED issues
by John Brown July 8, 2019

July 8, 2019

At the sensual mixing state, nitric oxide starts getting free from the man's body that makes and spare cGMP in his hot organ. Finally, penile muscles get broaden and speed of blood is augmented due to unwinding and extension of male penile channels. Therefore, a man finally gets a hard and suffering erection. Sildenafil citrate is the generic of Sildenafil 100mg, which works by hampering the movement of PDE-5 proteins that further deters the breakdown of cGMP existing in the male private district. A presentation tablet of Sildenafil should be taken orally with one glass of water. https://www.247pharmacymart.com/shop/buy-cenforce-150-online-paypal/ https://www.kamagra24uk.com/product/cenforce-100-sildenafil-citrate-100mg/

4 3

Python including packages
by Siddharth Prajosh July 8, 2019

July 8, 2019

Hey all, after this talk <http://pyfound.blogspot.com/2019/05/amber-brown-batteries-included-but.html> on how useful standard libraries are this has been in talks in multiple channels. I just wanted to present my idea on the same. Why not keep the essentials (ensurepip) and strip off everything else. When someone imports a package like datetime, we can catch the error (ImportError) and install it. Or something similar. Please do let me know your feedback on why this might or might not be a good option so that we can come up with better solutions for issues. Thank you!

4 4

Re: Getting the version number for a package or module, reliably
by Sylvain MARIE July 6, 2019

July 6, 2019

Thanks for the feedback Xavier Indeed, this “last resort” strategy only works if your project source is a git repository. I rely on the excellent setuptools_scm to handle that special case. By the way if someone knows an equivalent for other popular scm systems, please let me know (or better suggest it here: https://github.com/smarie/python-getversion/issues ) I forgot to add something in my previous email: the original reason for developing this library was to provide a version-aware persistency layer based on json. Basically when you deserialize an object, your code would receive the associated version, which allows for legacy-aware deserialization (typically storing a machine learning model for 6 months and needing to upgrade your code while still being able to deserialize it). I guess that this use case could also apply to pickle – from what I remember, pickle does not like it too much when the object to deserialize does not correspond to the same versions of the classes used. Best Sylvain De : Xavier Combelle <xavier.combelle(a)gmail.com> Envoyé : samedi 6 juillet 2019 11:34 À : Sylvain MARIE <sylvain.marie(a)se.com> Objet : Re: [Python-ideas] Getting the version number for a package or module, reliably [External email: Use caution with links and attachments] ________________________________ I'm certainly wrong, but version of a development version of a typical library is probably not reliable, as typically the version number is bumped when a new version is shipped so the code can be very different of the version given. Le sam. 6 juil. 2019 11:13, Sylvain MARIE via Python-ideas <python-ideas(a)python.org<mailto:python-ideas@python.org>> a écrit : Dear python enthusiasts, For some industrial project a few years ago we needed a reliable way to get the version number for a package or module. I found out that there was actually no method working in all edge cases, including: * Built-in modules * Unzipped wheels and eggs added to the python patch * Non-installed project under development with version control information available * Packages both installed and added to the python path (typically a developer working on a new version) So I created one, and finally found the time to publish it. No rocket science here, but you may find this new package useful: https://smarie.github.io/python-getversion/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsmarie.gi…> It works with any imported module for now, including submodules. Along with the version you get details about why a given version number is returned (is it because of the __version__ attribute that was found, or because of the Version metadata in the installed package, etc.) Also, if one edge case is missing, it is fairly easy to add it. If I missed something in the stdlib (I acme across the importlib.metadata initiative but as of now it does not seem to cover all of the above cases), please let me know so that I can cite it in the documentation and even redirect to it if it happens to already cover all the cases. Happy summer to all ! -- Sylvain _______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org<mailto:python-ideas@python.org> To unsubscribe send an email to python-ideas-leave(a)python.org<mailto:python-ideas-leave@python.org> https://mail.python.org/mailman3/lists/python-ideas.python.org/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OEXCP…<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…> Code of Conduct: http://python.org/psf/codeofconduct/<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org…> ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. ______________________________________________________________________

1 0

Getting the version number for a package or module, reliably
by Sylvain MARIE July 6, 2019

July 6, 2019

Dear python enthusiasts, For some industrial project a few years ago we needed a reliable way to get the version number for a package or module. I found out that there was actually no method working in all edge cases, including: * Built-in modules * Unzipped wheels and eggs added to the python patch * Non-installed project under development with version control information available * Packages both installed and added to the python path (typically a developer working on a new version) So I created one, and finally found the time to publish it. No rocket science here, but you may find this new package useful: https://smarie.github.io/python-getversion/ It works with any imported module for now, including submodules. Along with the version you get details about why a given version number is returned (is it because of the __version__ attribute that was found, or because of the Version metadata in the installed package, etc.) Also, if one edge case is missing, it is fairly easy to add it. If I missed something in the stdlib (I acme across the importlib.metadata initiative but as of now it does not seem to cover all of the above cases), please let me know so that I can cite it in the documentation and even redirect to it if it happens to already cover all the cases. Happy summer to all ! -- Sylvain

1 0

Proposal: `python-config` should be available in venv virtual environment.
by Stefan Droege July 3, 2019

July 3, 2019

Currently, when creating a virtualenv with the PEP-405 `venv` module, the python-config executable will not be copied/symlinked to the virtualenv. That means for projects that link against the python interpreter, *and* which are built in a venv virtual environment, some custom "magic" has to be done to find the correct `python-config` executable in order to get the full name of libpython. A concrete case I have described in this issue of the cocotb project https://github.com/potentialventures/cocotb/issues/978 Note that finding the correct `python-config` is not trivial because of version and/or ABI version tags that are appended. While it seems we have found a workaround for now, I think it makes sense to make `python-config` itself. My "wish" is to include this feature in the standard library venv module PEP-405. But for reference I'll also link this issue, of the `virtualenv` tool, which discusses a similar wish. Unfortunatly this was closed without progress: https://github.com/pypa/virtualenv/issues/169

3 2

Integer infinity and extended integers
by Neil Girdhar July 2, 2019

July 2, 2019

Now that Python is beginning to embrace type annotations, is it worth revisiting the idea of having extended integers and an integer infinity? I found myself trying to annotate this line: events_to_do: Union[int, float] = math.inf where I am only including float in the union to accommodate math.inf. I'm interested in exploring this concrete proposal: Add a class to the numeric hierarchy (https://www.python.org/dev/peps/pep-3141/) ExtendedIntegral whereby Real :> ExtendedIntegral :> Integral. Add a sentinel math.int_inf that obeys all of the same kinds of rules as math.inf does. Then, I could annotate more simply: events_to_do: ExtendedIntegral = math.int_inf With respect to Python, this is discussed somewhat here https://stackoverflow.com/questions/24587994/infinite-integer-in-python/357…. The name "extended integer" is discussed somewhat here https://math.stackexchange.com/questions/1442961/extended-integers. A quick search of papers shows that it is sometimes used in this sense: https://scholar.google.com/scholar?q=%22extended+integer Best, Neil

2 1

Re: [Python-Dev] strip behavior provides inconsistent results with certain strings
by Kirill Balunov July 1, 2019

July 1, 2019

> > This was quite extensively discussed on python-ideas recently: > > > https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS… > > (I'm finding it hard to find a good thread view in the new interface -- > but that will get you started) > > My memory of that thread is that there was a lot of bike shedding, and > quite a lot of resistance to adding a couple new methods, which I > personally never understood (Not why we don't want to add methods > willy-nilly, but why there was this much resistance to what seems like an > low-disruption, low maintenance, and helpful addition) > > I think it just kind of petered out, rather than being rejected, so if > someone wants to take up the mantle, that would be great -- and some > support from a core dev or two would probably help. > > -CHB > > > > > On Fri, Jun 28, 2019 at 10:44 AM Brett Cannon <brett(a)python.org> wrote: > >> Glenn Linderman wrote: >> > On 6/27/2019 3:09 PM, Brett Cannon wrote: >> > > My guess is that without Guido to just ask this will >> > > have to go to a PEP as it changes a built-in. >> > > How does adding two new methods change a built-in? >> > > Now if an extra parameter were added to modify lstrip, rstrip, and >> strip >> > to make them do something different, yes. >> > But adding new methods doesn't change anything, unless someone is >> > checking for their existence. >> >> Sure, but the built-ins are so widely used that we don't want to blindly >> add every method idea that someone comes up with either. We all very much >> share ownership of the built-ins, so we should all agree to changes to >> them, and getting agreement means either clear consensus and/or a PEP. >> >> -Brett >> >> > My preferred color is pstrip and sstrip (prefix and suffix strip) >> since >> > lstrip and rstrip mean left and right. >> > And maybe there should be a psstrip, that takes two parameters, the >> > prefix and the suffix to strip. >> > Such functions would certainly reduce code in a lot of places where I do >> > if string.startswith('foo'): >> > string = string[ 3: ]; >> > as well as making it more robust, because the string and its length >> have >> > to stay synchronized when changes are made. >> > I think this type of discussion better fits here. Quickly looking through the aforementioned branch - I want to say that that branch died due to bike-shedding :) I'm sure this has already been proposed, but it seems to me that the most natural would be to expand the functionality of existing `.lstrip`, `.rstrip` and maybe `.strip` methods. Probably many do not like what _names_ were given to these methods. But nothing can be done about it, they are what they are and will not go anywhere. Adding two more methods with similar functionality will only confuse. Moreover since we already have a precedent with other string methods like `.startswith` and `.endswith` that accepts as an argument both a string and a tuple of strings. On the other hand, this suggests that this functionality makes sense only for `.lstrip` and `.rstrip` variants. Is such a discrepancy acceptable? If it is, I will suggest to allow `.lstrip` and `.rstrip` to accept tuple of strings. The _stripping_ will be carried out in accordance with the linear probing and only one stripping at a time. >>> s = 'paparazzi' >>> s.lstrip(('pa', 'ra')) 'parazzi' Which also deviates from the existing semantics of strinst.split(strinst). Is this also permissible? Perhaps all these discrepancies lead to the idea of having new names for these methods. But for me personally, all these deviations are an inevitable price for practicality and, of course, echoes of past decisions. I would like to have such a functionality and find it usedul. Just today, I needed to remove the “model_” substring from beginnings of the set of file filenames, but this does not work because some of the files have names, like "model_ma...", "model_me...", "model_or..." and so on. with kind regards, -gdg

2 1

Overloading assignment concrete proposal (Re: Re: Operator as first class citizens -- like in scala -- or yet another new operator?)
by Andrew Barnert July 1, 2019

July 1, 2019

The thread on operators as first-class citizens keeps getting vague ideas about assignment overloading that wouldn't actually work, or don't even make sense. I think it's worth writing down the simplest design that would actually work, so people can see why it's not a good idea (or explain why they think it would be anyway). in pseudocode, just as x += y means this: xval = globals()['x'] try: result = xval.__iadd__(y) except AttributeError: result = xval.__add__(y) globals()['x'] = result … x = y would mean this: try: xval = globals()['x'] result = xval.__iassign__(y) except (LookupErrorr, AttributeError): result = y globals()['x'] = result If you don't understand why this would work or why it wouldn't be a great idea (or want to nitpick details), read on; otherwise, you can skip the rest of this message. --- First, why is there even a problem? Because Python doesn't even have "variables" in the same sense that languages like C++ that allow assignment overloading do. In C++, a variable is an "lvalue", a location with identity and type, and an object is just a value that lives in a location. So assignment is an operation on variables: x = 2 is the same as XClass::operator=(&x, y). In Python, an object is a value that lives wherever it wants, with identity and type, and a variable is just a name that can be bound to a value in a namespace. So assignment is an operation on namespaces, not on variables: x = 2 is the same as dict.__settem__(globals(), 'x', 2). The same thing is true for more complicated assignments. For example, a.x = 2 is just an operation on a's namespace instead of the global namespace: type(a).__setattr__(a, 'x', 2). Likewise, a.b['x'] = 2 is type(a.b).__setitem__(a.b, 'x', 2), And so on, --- But Python allows overloading augmented assignment. How does that work? There's a perfectly normal namespace lookup at the start and namespace store at the end—but in between, the existing value of the target gets to specify the value being assigned. Immutable types like int don't define __iadd__, and __add__ creates and returns a new object. So, x += y ends up the same as x = x + y. But mutable types like list define an __iadd__ that mutates self in-place and then returns self, so x gets harmlessly rebound to the same object it was already bound to. So x += y ends up the same as x.extend(y); x = x. The exact same technique would work for overloading normal assignment. The only difference is that x += y is illegal if x is unbound, while x = y obviously has to be legal (and mean there is no value to intercept the assignment). So, the fallback happens when xval doesn't define __iassign__, but also when x isn't bound at all. So, for immutable types like eint, and almost all mutable types like list—and when x is unbound—x = y does the same thing it always did. But special types that want to act like transparent mutable handles define an __iassign__ that mutates self in place and returns self, so x gets harmlessly rebound to the same object. So x = y ends up the same as, say, x.set_target(y); x = x. This all works the same if the variables are local rather than global, or for more complicated targets like attribution or subscription, and even for target lists; the intercept still happens the same way, between the (more complicated) lookup and storage steps. --- Now, why is this a bad idea? First, the benefit of __iassign__ is a lot smaller than __iadd__. A sizable fraction of "x += y" statements are for mutable "x" values, but only a rare handful of "x = y" statements would be for special handle "x" values. Even the same cost for a much smaller benefit would be a much harder sell. But the runtime performance cost difference is huge. If augmented assignment weren't overloadable, it would still have to lookup the value, lookup and call a special method on it, and store the value. The only cost overloading adds is trying two special methods instead of one, which is tiny. But regular assignment doesn't have to do a value lookup or a special method call at all, only a store; adding those steps would roughly double the cost of every new variable assignment, and even more for every reassignment. And assignments are very common in Python, even within inner loops, so we're talking about a huge slowdown to almost every program out there. Also, the fact that assignment always means assignment makes Python code easier both for humans to skim, and for automated programs to process. Consider, for example, a static type checker like mypy. Today, x = 2 means that x must now be an int, always. But if x could be a Signal object with an overloaded __iassign__, then, x = 2 might mean that x must now be an int, or it might mean that x must now be whatever type(x).__iassign__ returns. Finally, the complexity of __iassign__ is at least a little higher than __iadd__. Notice that in my pseudocode above, I cheated—obviously the xval = and result = lines are not supposed to recursively call the same pseudocode, but to directly store a value in new temporary local variable. In the real implementation, there wouldn't even be such a temporary variable (in CPython, the values would just be pushed on the stack), but for documenting the behavior, teaching it to students, etc., that doesn't matter. Being precise here wouldn't be hugely difficult, but it is a little more difficult than with __iadd__, where there's no similar potential confusion even possible. On Wednesday, June 19, 2019, 10:54:04 AM PDT, Andrew Barnert via Python-ideas <python-ideas(a)python.org> wrote: On Jun 18, 2019, at 12:43, nate lust <natelust(a)linux.com> wrote: I have been following this discussion for a long time, and coincidentally I recently started working on a project that could make use of assignment overloading. (As an aside it is a configuration system for a astronomical data analysis pipeline that makes heavy use of descriptors to work around historical decisions and backward compatibility). Our system makes use of nested chains of objects and descriptors and proxy object to manage where state is actually stored. The whole system could collapse down nicely if there were assignment overloading. However, this works OK most of the time, but sometimes at the end of the chain things can become quite complicated. I was new to this code base and tasked with making some additions to it, and wished for an assignment operator, but knew the data binding model of python was incompatible from p. This got me thinking. I didnt actually need to overload assignment per-say, data binding could stay just how it was, but if there was a magic method that worked similar to how __get__ works for descriptors but would be called on any variable lookup (if the method was defined) it would allow for something akin to assignment. What counts as “variable lookup”? In particular: For example: class Foo: def __init__(self): self.value = 6 self.myself = weakref.ref(self) def important_work(self): print(self.value) … why doesn’t every one of those “self” lookups call self.__get_self__()? It’s a local variable being looked up by name, just like your “foo” below, and it finds the same value, which has the same __get_self__ method on its type. The only viable answer seems to that it does. So, to avoid infinite circularity, your class needs to use the same kind of workaround used for attribute lookup in classes that define __getattribute__ and/or __setattr__: def important_work(self): print(object.__get_self__(self).value) def __get_self__(self): return object.__get_self__(self).myself But even that won’t work here, because you still have to look up self to call the superclass method on it. I think it would require some new syntax, or at least something horrible involving locals(), to allow you to write the appropriate methods. def __get_self__(self): return self.myself Besides recursively calling itself for that “self” lookup, why doesn’t this also call weakref.ref.__get_self__ for that “myself” lookup? It’s an attribute lookup rather than a local namespace lookup, but surely you need that to work too, or as soon as you store a Foo instance in another object it stops overloading. For this case there’s at least an obvious answer: because weakref.ref doesn’t override that method, the variable lookup doesn’t get intercepted. But notice that this means every single value access in Python now has to do an extra special-method lookup that almost always does nothing, which is going to be very expensive. def __setattr__(self, name, value): self.value = value You can’t write __setattr__ methods this way. That assignment statement just calls self.__setattr__(‘value’, value), which will endlessly recurse. That’s why you need something like the object method call to break the circularity. Also, this will take over the attribute assignments in your __init__ method. And, because it ignores the name and always sets the value attribute, it means that self.myself = is just going to override value rather than setting myself. To solve both of these problems, you want a standard __setattr__ body here: def __setattr__(self, name, value): object.__setattr__(self, name, value) But that immediately makes it obvious that your __setattr__ isn’t actually doing anything, and could just be left out entirely. foo = Foo() # Create an instancefoo # The interpreter would return foo.myselffoo.value # The interpreter would return foo.myself.value foo = 19 # The interpreter would run foo.myself = 6 which would invoke foo.__setattr__('myself', 19) For this last one, why would it do that? There’s no lookup here at all, only an assignment. The only way to make this work would be for the interpreter to lookup the current value of the target on every assignment before assigning to it, so that lookup could be overloaded. If that were doable, then assignment would already be overloadable, and this whole discussion wouldn’t exist. But, even if you did add that, __get_self__ is just returning the value self.myself, not some kind of reference to it. How can the interpreter figure out that the weakref.ref value it got came from looking up the name “myself” on the Foo instance? (This is the same reason __getattr__ can’t help you override attribute setting, and a separate method __setattr__ is needed.) To make this work, you’d need a __set_self__ to go along with __get_self__. Otherwise, your changes not only don’t provide a way to do assignment overloading, they’d break assignment overloading if it existed. Also, all of the extra stuff you’re trying to add on top of assignment overloading can already be done today. You just want a transparent proxy: a class whose instances act like a reference to some other object, and delegate all methods (and maybe attribute lookups and assignments) to it. This is already pretty easy; you can define __getattr__ (and __setattr__) to do it dynamically, or you can do some clever stuff to create static delegating methods (and properties) explicitly at object-creation or class-creation time. Then foo.value returns foo.myself.value, foo.important_work() calls the Foo method but foo.__str__() calls foo.myself.__str__(), you can even make it pass isinstance checks if you want. The only thing it can’t do is overload assignment. I think the real problem here is that you’re thinking about references to variables rather than values, and overloading operators on variables rather than values, and neither of those makes sense in Python. Looking up, or assigning to, a local variable named “foo” is not an operation on “the foo variable”, because there is no such thing; it’s an operation on the locals namespace._______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4JMNZ… Code of Conduct: http://python.org/psf/codeofconduct/

14 80