Mailman 3 June 2019 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 17, 2022

Feb. 17, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 11, 2021

March 11, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Add the imath module
by Serhiy Storchaka March 22, 2020

March 22, 2020

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math … [View More]

14 38

New explicit methods to trim strings
by Alex Grigoryev March 21, 2020

March 21, 2020

Following the discussion here (https://link.getmailspring.com/link/7D84D131-65B6-4EF7-9C43-51957F9DFAA9@ge…) I propose to add 3 new string methods: str.trim, str.ltrim, str.rtrim Another option would be to change API for str.split method to work correctly with sequences. In [1]: def ltrim(s, seq): ...: return s[len(seq):] if s.startswith(seq) else s ...: In [2]: def rtrim(s, seq): ...: return s[:-len(seq)] if s.endswith(seq) else s ...: In [3]: def trim(s, seq): ...: return ltrim(rtrim(s, … [View More]

28 162

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar March 6, 2020

March 6, 2020

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to … [View More]

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 21, 2020

Feb. 21, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Fixed point format for numbers with locale based separators
by Łukasz Stelmach Dec. 20, 2019

Dec. 20, 2019

Hi, I would like to present two pull requests[1][2] implementing fixed point presentation of numbers and ask for comments. The first is mine. I learnt about the second after publishing mine. The only format using decimal separator from locale data for float/complex/decimal numbers at the moment is "n" which behaves like "g". The drawback of these formats, I would like to overcome, is the inability to print numbers ranging more than one order of magnitude with the same number of decimal digits … [View More]

5 9

Please consider adding of functions file system operations to pathlib
by George Fischhof Nov. 16, 2019

Nov. 16, 2019

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger Oct. 23, 2019

Oct. 23, 2019

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed … [View More]for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel [View Less]

17 49

Why not accept lists or arbitrary iterables in str.endswith?
by Soni L. Oct. 14, 2019

Oct. 14, 2019

I'm parsing configs for domain filtering rules, and they come as a list. However, str.endswith requires a tuple. So I need to use str.endswith(tuple(list)). I don't know the reasoning for this, but why not just accept a list as well?

3 6

support toml for pyproject support
by Jimmy Girardet July 29, 2019

July 29, 2019

Hi, I don't know if this was already debated but I don't know how to search in the whole archive of the list. For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to … [View More]

13 15

Integer infinity and extended integers
by Neil Girdhar July 2, 2019

July 2, 2019

Now that Python is beginning to embrace type annotations, is it worth revisiting the idea of having extended integers and an integer infinity? I found myself trying to annotate this line: events_to_do: Union[int, float] = math.inf where I am only including float in the union to accommodate math.inf. I'm interested in exploring this concrete proposal: Add a class to the numeric hierarchy (https://www.python.org/dev/peps/pep-3141/) ExtendedIntegral whereby Real :> ExtendedIntegral :… [View More]

2 1

Overloading assignment concrete proposal (Re: Re: Operator as first class citizens -- like in scala -- or yet another new operator?)
by Andrew Barnert July 1, 2019

July 1, 2019

The thread on operators as first-class citizens keeps getting vague ideas about assignment overloading that wouldn't actually work, or don't even make sense. I think it's worth writing down the simplest design that would actually work, so people can see why it's not a good idea (or explain why they think it would be anyway). in pseudocode, just as x += y means this: xval = globals()['x'] try: result = xval.__iadd__(y) except … [View More]AttributeError: result = xval.__add__(y) globals()['x'] = result … x = y would mean this: try: xval = globals()['x'] result = xval.__iassign__(y) except (LookupErrorr, AttributeError): result = y globals()['x'] = result If you don't understand why this would work or why it wouldn't be a great idea (or want to nitpick details), read on; otherwise, you can skip the rest of this message. --- First, why is there even a problem? Because Python doesn't even have "variables" in the same sense that languages like C++ that allow assignment overloading do. In C++, a variable is an "lvalue", a location with identity and type, and an object is just a value that lives in a location. So assignment is an operation on variables: x = 2 is the same as XClass::operator=(&x, y). In Python, an object is a value that lives wherever it wants, with identity and type, and a variable is just a name that can be bound to a value in a namespace. So assignment is an operation on namespaces, not on variables: x = 2 is the same as dict.__settem__(globals(), 'x', 2). The same thing is true for more complicated assignments. For example, a.x = 2 is just an operation on a's namespace instead of the global namespace: type(a).__setattr__(a, 'x', 2). Likewise, a.b['x'] = 2 is type(a.b).__setitem__(a.b, 'x', 2), And so on, --- But Python allows overloading augmented assignment. How does that work? There's a perfectly normal namespace lookup at the start and namespace store at the end—but in between, the existing value of the target gets to specify the value being assigned. Immutable types like int don't define __iadd__, and __add__ creates and returns a new object. So, x += y ends up the same as x = x + y. But mutable types like list define an __iadd__ that mutates self in-place and then returns self, so x gets harmlessly rebound to the same object it was already bound to. So x += y ends up the same as x.extend(y); x = x. The exact same technique would work for overloading normal assignment. The only difference is that x += y is illegal if x is unbound, while x = y obviously has to be legal (and mean there is no value to intercept the assignment). So, the fallback happens when xval doesn't define __iassign__, but also when x isn't bound at all. So, for immutable types like eint, and almost all mutable types like list—and when x is unbound—x = y does the same thing it always did. But special types that want to act like transparent mutable handles define an __iassign__ that mutates self in place and returns self, so x gets harmlessly rebound to the same object. So x = y ends up the same as, say, x.set_target(y); x = x. This all works the same if the variables are local rather than global, or for more complicated targets like attribution or subscription, and even for target lists; the intercept still happens the same way, between the (more complicated) lookup and storage steps. --- Now, why is this a bad idea? First, the benefit of __iassign__ is a lot smaller than __iadd__. A sizable fraction of "x += y" statements are for mutable "x" values, but only a rare handful of "x = y" statements would be for special handle "x" values. Even the same cost for a much smaller benefit would be a much harder sell. But the runtime performance cost difference is huge. If augmented assignment weren't overloadable, it would still have to lookup the value, lookup and call a special method on it, and store the value. The only cost overloading adds is trying two special methods instead of one, which is tiny. But regular assignment doesn't have to do a value lookup or a special method call at all, only a store; adding those steps would roughly double the cost of every new variable assignment, and even more for every reassignment. And assignments are very common in Python, even within inner loops, so we're talking about a huge slowdown to almost every program out there. Also, the fact that assignment always means assignment makes Python code easier both for humans to skim, and for automated programs to process. Consider, for example, a static type checker like mypy. Today, x = 2 means that x must now be an int, always. But if x could be a Signal object with an overloaded __iassign__, then, x = 2 might mean that x must now be an int, or it might mean that x must now be whatever type(x).__iassign__ returns. Finally, the complexity of __iassign__ is at least a little higher than __iadd__. Notice that in my pseudocode above, I cheated—obviously the xval = and result = lines are not supposed to recursively call the same pseudocode, but to directly store a value in new temporary local variable. In the real implementation, there wouldn't even be such a temporary variable (in CPython, the values would just be pushed on the stack), but for documenting the behavior, teaching it to students, etc., that doesn't matter. Being precise here wouldn't be hugely difficult, but it is a little more difficult than with __iadd__, where there's no similar potential confusion even possible. On Wednesday, June 19, 2019, 10:54:04 AM PDT, Andrew Barnert via Python-ideas <python-ideas(a)python.org> wrote: On Jun 18, 2019, at 12:43, nate lust <natelust(a)linux.com> wrote: I have been following this discussion for a long time, and coincidentally I recently started working on a project that could make use of assignment overloading. (As an aside it is a configuration system for a astronomical data analysis pipeline that makes heavy use of descriptors to work around historical decisions and backward compatibility). Our system makes use of nested chains of objects and descriptors and proxy object to manage where state is actually stored. The whole system could collapse down nicely if there were assignment overloading. However, this works OK most of the time, but sometimes at the end of the chain things can become quite complicated. I was new to this code base and tasked with making some additions to it, and wished for an assignment operator, but knew the data binding model of python was incompatible from p. This got me thinking. I didnt actually need to overload assignment per-say, data binding could stay just how it was, but if there was a magic method that worked similar to how __get__ works for descriptors but would be called on any variable lookup (if the method was defined) it would allow for something akin to assignment. What counts as “variable lookup”? In particular: For example: class Foo: def __init__(self): self.value = 6 self.myself = weakref.ref(self) def important_work(self): print(self.value) … why doesn’t every one of those “self” lookups call self.__get_self__()? It’s a local variable being looked up by name, just like your “foo” below, and it finds the same value, which has the same __get_self__ method on its type. The only viable answer seems to that it does. So, to avoid infinite circularity, your class needs to use the same kind of workaround used for attribute lookup in classes that define __getattribute__ and/or __setattr__: def important_work(self): print(object.__get_self__(self).value) def __get_self__(self): return object.__get_self__(self).myself But even that won’t work here, because you still have to look up self to call the superclass method on it. I think it would require some new syntax, or at least something horrible involving locals(), to allow you to write the appropriate methods. def __get_self__(self): return self.myself Besides recursively calling itself for that “self” lookup, why doesn’t this also call weakref.ref.__get_self__ for that “myself” lookup? It’s an attribute lookup rather than a local namespace lookup, but surely you need that to work too, or as soon as you store a Foo instance in another object it stops overloading. For this case there’s at least an obvious answer: because weakref.ref doesn’t override that method, the variable lookup doesn’t get intercepted. But notice that this means every single value access in Python now has to do an extra special-method lookup that almost always does nothing, which is going to be very expensive. def __setattr__(self, name, value): self.value = value You can’t write __setattr__ methods this way. That assignment statement just calls self.__setattr__(‘value’, value), which will endlessly recurse. That’s why you need something like the object method call to break the circularity. Also, this will take over the attribute assignments in your __init__ method. And, because it ignores the name and always sets the value attribute, it means that self.myself = is just going to override value rather than setting myself. To solve both of these problems, you want a standard __setattr__ body here: def __setattr__(self, name, value): object.__setattr__(self, name, value) But that immediately makes it obvious that your __setattr__ isn’t actually doing anything, and could just be left out entirely. foo = Foo() # Create an instancefoo # The interpreter would return foo.myselffoo.value # The interpreter would return foo.myself.value foo = 19 # The interpreter would run foo.myself = 6 which would invoke foo.__setattr__('myself', 19) For this last one, why would it do that? There’s no lookup here at all, only an assignment. The only way to make this work would be for the interpreter to lookup the current value of the target on every assignment before assigning to it, so that lookup could be overloaded. If that were doable, then assignment would already be overloadable, and this whole discussion wouldn’t exist. But, even if you did add that, __get_self__ is just returning the value self.myself, not some kind of reference to it. How can the interpreter figure out that the weakref.ref value it got came from looking up the name “myself” on the Foo instance? (This is the same reason __getattr__ can’t help you override attribute setting, and a separate method __setattr__ is needed.) To make this work, you’d need a __set_self__ to go along with __get_self__. Otherwise, your changes not only don’t provide a way to do assignment overloading, they’d break assignment overloading if it existed. Also, all of the extra stuff you’re trying to add on top of assignment overloading can already be done today. You just want a transparent proxy: a class whose instances act like a reference to some other object, and delegate all methods (and maybe attribute lookups and assignments) to it. This is already pretty easy; you can define __getattr__ (and __setattr__) to do it dynamically, or you can do some clever stuff to create static delegating methods (and properties) explicitly at object-creation or class-creation time. Then foo.value returns foo.myself.value, foo.important_work() calls the Foo method but foo.__str__() calls foo.myself.__str__(), you can even make it pass isinstance checks if you want. The only thing it can’t do is overload assignment. I think the real problem here is that you’re thinking about references to variables rather than values, and overloading operators on variables rather than values, and neither of those makes sense in Python. Looking up, or assigning to, a local variable named “foo” is not an operation on “the foo variable”, because there is no such thing; it’s an operation on the locals namespace._______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4JMNZ… Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

14 80

shutil.symlink to allow non-race replacement of existing link targets
by Tom Hale June 30, 2019

June 30, 2019

As suggested by Toshio Kuratomi at https://bugs.python.org/issue36656, I am raising this here for inclusion in the shutil module. Mimicking POSIX, os.symlink() will raise FileExistsError if the link name to be created already exists. A common use case is overwriting an existing file (often a symlink) with a symlink. Naively, one would delete the file named link_name file if it exists, then call symlink(). This "solution" is already 3 lines of code, and without exception handling it … [View More]introduces the race condition of a file named link_name being created between unlink and symlink. Depending on the functionality required, I suggest: * os.symlink() - the new link name is expected to NOT exist * shutil.symlink() - the new symlink replaces an existing file Handling all possible race conditions (some detailed in issue36656) is non-trivial, however this is the best that I have come up with so far: ========================================================================== import os, tempfile def symlink(target, link_name): '''Create a symbolic link link_name pointing to target. Overwrites link_name if it exists. ''' # os.replace() may fail if files are on different filesystems link_dir = os.path.dirname(link_name) # Link to a temporary filename that doesn't exist while True: temp_link_name = tempfile.mktemp(dir=link_dir) # os.* functions mimic as closely as possible system functions # The POSIX symlink() returns EEXIST if link_name already exists # https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html try: os.symlink(target, temp_link_name) break except FileExistsError: pass # Replace link_name with temp_link_name try: # Pre-empt os.replace on a directory with a nicer message if os.path.isdir(link_name): raise IsADirectoryError(f"Cannot symlink over existing directory: '{link_name}'") os.replace(temp_link_name, link_name) except: if os.path.islink(temp_link_name): os.remove(temp_link_name) raise ========================================================================== The documentation (https://docs.python.org/3/library/shutil.html) I suggest for this is: shutil.symlink(target, link_name) Create a symbolic link named link_name pointing to target, overwriting target if it exists. If link_name is a directory, IsADirectoryError is raised. To not overwrite target, use os.symlink() ========================================================================== It would be tempting to do: while True: try: os.symlink(target, link_name) break except FileExistsError: os.remove(link_name) But this has a race condition when replacing a symlink should should *always* exist, eg: /lib/critical.so -> /lib/critical.so.1.2 When upgrading by: symlink('/lib/critical.so.2.0', '/lib/critical.so') There is a point in time when /lib/critical.so doesn't exist. ========================================================================== One issue I see with my suggested code is that the file at temp_link_name could be changed before target is replaced with it. This is mitigated by the randomness introduced by mktemp(). While it is far less likely that a file is accessed with a random and unknown name than with an existing known name, I seek input on a solution if this is an unacceptable risk. Prior art: * https://bugs.python.org/issue36656 (already mentioned above) * https://stackoverflow.com/a/55742015/5353461 * https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/ln.c -- Tom Hale [View Less]

14 49

something like sscanf for Python
by James Lu June 29, 2019

June 29, 2019

> On Jun 26, 2019, at 7:13 PM, Chris Angelico <rosuav(a)gmail.com> wrote: > > The main advantage of sscanf over a regular expression is that it > performs a single left-to-right pass over the format string and the > target string simultaneously, with no backtracking. (This is also its > main DISadvantage compared to a regular expression.) A tiny amount of > look-ahead in the format string is the sole exception (for instance, > format string "%s$%d" would collect a … [View More]

8 10

Canceling thread in python
by Matúš Valo June 27, 2019

June 27, 2019

Hi All, Currently it is not possible to "kill" thread which is blocked. The rationale for this is handling situations when thread is blocked - e.g. when thread is quering DB when lock occurred on Database. In this case, the main thread has no way how to stop the blocked thread. Killing a thread is also popular question - see [1][2]. pthread library and Windows API contains mechanisms for forced termination of threads - see [3] and [4]. It is also simple to use them using ctypes library but … [View More]

9 18

Re: A proposal (and implementation) to add assignment and LOAD overloading
by Steven D'Aprano June 27, 2019

June 27, 2019

On Fri, Jun 28, 2019 at 02:44:28AM +1000, Chris Angelico wrote: > If it's ALWAYS called, then it's almost useless. The wrapper object > will vanish the moment you attempt to do anything with it, devolving > instantly to the result of getself. I don't understand why it is useless. If the wrapper object is no longer needed, then getself will return the object which is needed, and the wrapper is superfluous and should be garbage collected. If the wrapper object is needed, then … [View More]

3 6

Proposal: Using % sign for percentage
by Ronie Martinez June 26, 2019

June 26, 2019

Good day! As Python focuses on readability, why not use % sign for actual percentages? For example: ``` rate = 0.1058 # float rate = 10.58% # percent, similar to above ``` It does not interfere with modulo operator as modulo follows a different format: ``` a = x % y ``` This looks like a small feature but it will surely set Python a level higher in terms of readability. Thanks a lot!

7 6

Re: `if-unless` expressions in Python
by Steven D'Aprano June 25, 2019

June 25, 2019

On Mon, Jun 24, 2019 at 12:37:48PM -0700, Andrew Barnert wrote: > Since you bring up “feeble mail clients”: > > Good mail clients can be configured to collapse and expand quotes, and > to automatically start long nested quotes collapsed. Indeed you are correct, and my own mail client supports that as an optional add-on, which I've tried and discarded as more annoying than helpful. In my experience the problem is that trying to solve the problem of quoting via technology is … [View More]

2 1

`if-unless` expressions in Python
by James Lu June 25, 2019

June 25, 2019

`if-unless` expressions in Python if condition1 expr unless condition2 is an expression that roughly reduces to expr if condition1 and not condition2 else EMPTY This definition means that expr is only evaluated if `condition1 and not condition2` evaluates to true. It also means `not condition2` is only evaluated if `condition1` is true. # EMPTY EMPTY is not actually a real Python value-- it's a value that collapses into nothing when used inside a statement expression: print([… [View More]

12 26

Make $ a valid identifier and a singleton
by James Lu June 23, 2019

June 23, 2019

Make $ a valid identifier and a singleton. $ is a useful placeholder in []. Possible function partial syntax: def foo(x, y): print(x, y) partialized = foo[$, 10] partialized(5) # => 5 10

4 3

Proposal: Allowing any variable to be used in a 'with... as...' expression
by Yonatan Zunger June 23, 2019

June 23, 2019

Hi everyone, I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK. *Summary: *The construction with expr1 as var1, expr2 as var2, ...: body fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not … [View More]expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False. *Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and *not* doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be. For example, this is a good pattern: with functionReturningFile(...) as input: doSomething(input) There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is *not* valid Python: with functionReturningOptionalFile(...) as input: doSomething(input) To handle this case, one has a few options. One may only use the 'with' in the known safe cases: inputFile = functionReturningOptionalFile(...) if inputFile: with inputFile as input: doSomething(input) else: doSomething(None) (NB that this requires factoring the with statement body into its own function, which may separately reduce readability and/or introduce overhead); one may dispense with the 'with' clause and do it in the pre-PEP343 way: try: input = functionReturningOptionalFile(...) doSomething(input) finally: if input: input.close() (This sacrifices all the benefits of the with statement, and requires the caller to explicitly call the cleanup methods, increasing error-proneness); or one may construct an explicit 'dev-null' class and return it instead of the file: class DevNullFile(object): .... implement the entire File API, including a context manager ... (This can only be described as god-awful, especially for complex API's like files) One obvious option would be to allow None to act as a context manager as well. We might contrast this with PEP 336 <https://www.python.org/dev/peps/pep-0336/>, "Make None Callable." This was rejected (rightly, I think) because "it is considered a feature that None raises an error if called." For example, it means that if a function variable has been nulled, attempting to call it later raises an error, as this usually indicates a code mistake. In the case where that is not correct, it is easy to assign a noop lambda to the function variable instead of None, thus allowing the error-checking and the function-deactivating behaviors to both persist, and in a clear and easily understandable way. In this case, OTOH, the AttributeError raised if None is passed to a with statement has significantly lower value. As the example above illustrates, there are many cases where None is an entirely legitimate value to want to pass, and unlike in the other situation, there is no equally easy way to pass it. Furthermore, if the passing of None *is* an error in some case, it is more useful to see that error at the site where the variable is actually used in the with statement body -- the thing for which it does not make sense to use None -- rather than at a structural declaration which essentially defines a variable scope. This is also the reason why such a change would impact relatively little existing code: code already has to be structured to prevent this from happening. If the assigned expression in the with statement could only return None as a result of a code bug, and a piece of existing code is relying on the with statement to catch it, it would instead fall through and be caught by their own body code, presumably giving a more coherent error anyway. This is a nonzero change in behavior, but it's well within the scope of behavior changes which normally occur from version to version. One alternative to this proposal would be to have only None allowed to act as a context manager. However, None is not particularly special in this regard; the logic above applies to any function which might return a Union type. Furthermore, allowing it for any type would permit the following construction as well: with var1 as expr1, var2 as expr2, ... .... body ... where the common factor between the variables is no longer their need for a guaranteed cleanup operation, but simply that they are semantically all tied to a single scope of the code. This improves code clarity, as it allows the syntax to follow the intent more closely, and also eliminates one other ugliness. In present Python, the required syntax for the above would be var1 = expr1 var3 = expr3 with var2 as expr2, var4 as expr4: ... body ... where the variables in the 'with' statement are those which satisfy the context manager protocol, and the ones above it are those which do not satisfy the protocol. The split between the two is entirely tied to a nonlocal fact about the code, namely the implementation of the return values of each of the expressions, making it nonobvious which is which. Worse, if the expressions depend on each other in sequence, this may have to be broken up into var1 = expr1 with var2 as expr2: var3 = expr3(var1, var2) with var4 as expr4(var3, ...): .... body ... This seems to lose on every measure of clarity and maintainability relative to the single compound 'with' statement above. Finally, one may ask if an (effective) default implementation of a protocol is ever a good idea. "Hidden defaults" are a great way to trigger surprising behavior, after all. However, in this case I would argue that the proposed default behavior is sufficiently obvious that there is no risk. Someone seeing a compound 'with' statement of the above form would naturally assume that its consequence is (a) to set each varN to the corresponding exprN, and (b) to execute any scope-initializers tied to exprN. Likewise, someone would naturally assume that nothing at all happens at scope exit, which is exactly the behavior of __exit__ being 'return False'. In fact, this *increases* local code clarity, since the counter-case -- where the implementation of each defaults (effectively) to raising an AttributeError -- is nonobvious and so requires that "nonlocal knowledge" of the code to assemble with statements. *Specific implementation proposal: *Actually defining __enter__ and __exit__ methods for each object would be a lot of overhead for no good value. Instead, we can easily implement this as a change to the specified behavior of the 'with' statement, simply by changing the error-handling behavior in the SETUP_WITH <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f62…> and WITH_CLEANUP_START <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f62…> cases in ceval.c. If this does proceed to the PEP stage, I'll put together a changelist, but it's very straightforward. Null values for enter and exit are no longer errors; if enter is null, then instead of decrementing the refcount of mgr and calling enter, we leave the mgr refcount alone and push it onto the stack in place of the result. If exit is null, we simply push it onto the stack just like we would normally, and ignore it in WITH_CLEANUP_START. [View Less]

10 20

Lists create
by 21ispinks＠parklandsacademy.co.uk June 21, 2019

June 21, 2019

How to list create.[09]

2 1