Python-ideas July 2019

python-ideas@python.org

69 participants
55 discussions

by James Lu

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:21:04 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> Why not just allow anything that’s valid as a target, like identifier[expr]? Or even just any expression at all? Is there an advantage to defining and using a similar but more limited syntax here? > Hmm, it should support any expression. I limited it at first because it was a minimal strawman.

2 years, 2 months

Fwd: Utilities for easier debugging

by James Lu

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:22:11 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> This would break iPython’s improved interactive console, which already uses this syntax to provide a similar feature. > If it’s so similar, I doubt it would break anything. This is intended to make it easier to log variables for a complex application (multiple files).

2 years, 2 months

Fwd: Utilities for easier debugging

by James Lu

Sent from my iPhone Begin forwarded message: > From: James Lu <jamtlu(a)gmail.com> > Date: July 28, 2019 at 6:23:06 PM EDT > To: Andrew Barnert <abarnert(a)yahoo.com> > Subject: Re: [Python-ideas] Utilities for easier debugging > > >> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> >> You want this added to every module’s globals? Why not put it in builtins instead? > It’s not a function call, it’s an expression that when evaluated triggers a breakpoint, so you have to add it as a property descriptor.

2 years, 2 months

Re: for ... except, with ... except

by Ethan Furman

[redirecting back to the list] On 07/27/2019 09:38 PM, James Lu wrote: >> On Jul 27, 2019, at 12:44 PM, Ethan Furman wrote: >> Sure, folks /know/ what it means, but it's a common bug because >> it doesn't read as "if some_var is assigned 7" but as "if some_var >> is equal to 7". > > That’s a straw man- a modern linter or compiler would catch that. > Many editors have linters integrated inside. The way to disable it > is to put parentheses around the assignment expesssion.

2 years, 2 months

Add a "partial with placeholders" function to the functools module

by Dominik Vilsmeier

Hello, currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods. I propose adding a function to `functools` that works with placeholders and thus offers even greater flexibility. The Ellipsis literal `...` seems a intuitive choice for that task. When eventually calling such a "partial placeholder" object, it would fill in placeholders from the left and add remaining `args` to the right. In terms of implementation this can be realized as a subclass of `partial` itself. ## Implementation from functools import partial from reprlib import recursive_repr class partial_placehold(partial): placeholder = Ellipsis def __call__(self, /, *args, **keywords): args = iter(args) try: old_args = [x if x is not self.placeholder else next(args) for x in self.args] except StopIteration: raise TypeError('too few arguments were supplied') from None keywords = {**self.keywords, **keywords} return self.func(*old_args, *args, **keywords) @recursive_repr() def __repr__(self): qualname = type(self).__qualname__ args = [repr(self.func)] args.extend(repr(x) if x is not self.placeholder else '...' for x in self.args) # Only this line deviates from `partial.__repr__`; could also factor that out into a separate method. args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items()) if type(self).__module__ == "functools": return f"functools.{qualname}({', '.join(args)})" return f"{qualname}({', '.join(args)})" # Would need to add something for compatibility with `partial`, i.e. for partializing a placeholder function. ## Example This allows for example the following usage: replace_dots_with_underscore = partial_placehold(str.replace, ..., '.', '_') replace_dots_with_underscore('foo.bar.baz') ## Relevance Sure we could also use a `lambda` instead ([as discussed here](https://mail.python.org/archives/list/python-ideas@python.org/message…) but there was a reason `partial` was introduced and I think the same arguments apply here too. Though most functions allow partializing via keyword arguments and this is undoubtedly a cleaner way, some might not and for example built-ins' methods won't allow it. Especially Python 3.8's introduction of positional-only parameters (PEP 570) might give rise to cases where `partial` is not sufficient. In case inspection is desired a `lambda` does not provide much information (sure you could always dig deeper with `inspect` for example but that's not the point). Consider the following example of a pre-defined sequence of default postprocessing steps and the user might add their own or remove existing ones, as appropriate: postprocessing_steps = [ lambda s: s.replace('foo', 'bar'), ] print(postprocessing_steps[0]) # <function <lambda> at 0x7f94a850dd30> This doesn't give a lot of information about what the lambda actually does (and thus whether the user should remove it or not). Using the `partial_placehold` instead, it's clear what is happening: postprocessing_steps = [ partial_placehold(str.replace, ..., 'foo', 'bar'), ] print(postprocessing_steps[0]) # partial_placehold(<method 'replace' of 'str' objects>, ..., 'foo', 'bar') ## Compatibility The proposed solution works with the current syntax and the usage of Ellipsis as a placeholder object is likely not to collide with actually used values (in any case the user might still reassign the `.placeholder` attribute). Because the direction of partializing is unchanged (still left to right) this doesn't introduce ambiguities which might come with a "right partial" function. Creating a placeholder function from a `partial` object is possible without any changes, the opposite way requires an additional check to result in a placeholder object again. ## Possible confusion Regarding the usage of Ellipsis right now, in `numpy` or `typing` for example, it always represents a placeholder for multiple "things", not a single one: array[..., None] # All the dimensions of `array` plus a new one. typing.Tuple[str, ...] # Any number of str objects. So the expectations might be biased in that sense. For example: def foo(a, b, c, d): pass p_foo = partial_placehold(foo, ..., 1, 2) p_foo(3, 4) Someone else reviewing the code might now assume that the `...` means to act as a placeholder for all arguments except the last two (and hence `p_foo(3, 4)` would be equivalent to `foo(3, 4, 1, 2)` while it actually is equivalent to `foo(3, 1, 2, 4)`). But this would be again some kind of "right partial" function and also the function name implies something else; documentation might clarify as well, of course. ## Conclusion Adding a "partial with placeholders" function to `functools` allows for covering use cases where the standard `partial` is not sufficient. No new syntax is required and the implementation is fairly straightforward given the inheritance from `partial`. Ellipsis `...` seems an intuitive choice for acting as a placeholder (concerning both, conflicts with actual partial values and code readability). There are uses cases where such a function would provide a clean solution and there is an interest in the community (https://stackoverflow.com/q/7811247/3767239, https://stackoverflow.com/q/19701775/3767239 for example). Especially with the introduction of positional-only parameters new use cases are likely to arise. ----- **Related threads:** * https://mail.python.org/archives/list/python-ideas@python.org/message/TVNCM… - Mentions essentially a similar idea. The original [PEP 309 -- Partial Function Application](https://www.python.org/dev/peps/pep-0309/) also mentions: > Partially applying arguments from the right, or inserting arguments at arbitrary positions creates its own problems, but pending discovery of a good implementation and non-confusing semantics, I don't think it should be ruled out.

2 years, 2 months

Re: for ... except, with ... except

by Serhiy Storchaka

26.07.19 21:52, Bruce Leban пише: > > On Fri, Jul 26, 2019 at 11:27 AM Serhiy Storchaka <storchaka(a)gmail.com > <mailto:storchaka@gmail.com>> wrote: > > > So you will be able to add errors handling like in: > > with connect() as stream: > for data in stream: > try: > write(data) > except OSError: > handle_write_error() > except OSError: > handle_read_error() > except OSError: > handle_connection_error() > > > To put this in a simpler way: the proposal is to add an except clause > that applies ONLY to the direct operation of the with or for statement > and not to the block. That's an interesting idea. > > The one thing I find confusing about your proposal is that the > proposed syntax does not imply the behavior. In a try statement, the > except appears at the end and after all possible statements that it > could cover. The proposal mimics that syntax but with different > semantics. Something like this would be much more clear what is going on: > > for VARIABLE in EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK > > with EXPRESSION as VARIABLE: > except EXCEPTION: > BLOCK > BLOCK > > while EXPRESSION: > except EXCEPTION: > BLOCK > BLOCK Besides an unusual for Python layout (a clause has different indentation than the initial clause of the statement to which it belongs) there is other problem. The exception block is not the part of the "for" or "with" block. After handling an exception in the "for" clause you do not continue to execute the "for" block, but leave the loop. After handling an exception in the "with" clause you do not continue to execute the "with" block and do not call `__exit__` when leave it. To me, this syntax is much more confusing than my initial proposition.

2 years, 2 months

Scope painting

by Yonatan Zunger

Hi everyone, I'm in the middle of developing a fancy heap profiler for Python (for those times when tracemalloc isn't enough), and in the process, thought of a small change in the interpreter which could have a lot of uses. I call it "scope painting" because it lets you paint a label on an interpreter scope. *Conceptual TL;DR: *Add a function which attaches a string label to the current interpreter frame. (This would go away when you either explicitly cleared it, or the frame ended, i.e. the current function returned) You can then use it for: - *Debugging: *Print this value out, if set, in the traceback formatters. This can be used e.g. in a server, by putting a unique request identifier as the label for the top-level handler method; then if something in the particular query triggers a crash, it's *much* easier to see what caused it. (When I was at Google, we did something very similar to this for the servers in search. It completely transformed the way we detected "queries of death," and made hunting down data-dependent bugs an order of magnitude easier.) - *CPU and heap profiling: *If a profiler or tracer stores this information when it's grabbing the stack trace, this can be used for two different kinds of analysis, depending on user needs: - *Splitting: *Consider two stack traces different if they differ in any of the labels. This lets you separate flows which seem to be going through the same part of the logic, but are actually doing it on very distinct paths. - *Joining: *Group stack traces by the label; this lets you identify "call with this label value and its descendants," which lets you very easily establish a user-defined aggregation for total CPU or heap usage. - *Network request tracing: *If you were feeling really fancy, you could attach this to a cross-network-request trace ID and propagate the same value of this across multiple servers in a complicated request chain. Then you could perform all of the above joinings, plus similar joinings from cross-server profiling systems, as well. *Implementation: *Pretty simple: add a new field PyObject *f_label to PyFrameObject which contains an optional Unicode string; the frame owns a reference. Add get and set methods, exposing those in the Python API, and probably also a simple context manager to set and restore the value of this field because that's going to be the 90% way it's used. (Based on experience with doing similar things in C++) Modify the traceback, profile, cProfile, and tracemalloc libraries (and maybe others?) to use this value as well. What do people think? Yonatan

2 years, 2 months

Stdlib Module

by Abdur-Rahmaan Janhangeer

Greetings, Just like builtins module, would it be a good idea to have a stdlib module so that we can know all modules in the std lib (just like builtins let us know builtins)? Abdur-Rahmaan Janhangeer Mauritius

2 years, 2 months

Should tempfile.NamedTemporaryFile() not raise FileNotFoundError if the file does not exist at the moment of closing?

by Andrew Pashkin

What does everyone think about making "tempfile.NamedTemporaryFile()" not raising "FileNotFoundError" in case if when it closes and the file is already deleted?

2 years, 3 months

Adding where keyword inside list coprehensions

by Fabrizio Messina

Hello, It is likely not the first time such a proposal is being made but let's see. I would like to explicitly set variable names in list comprehensions using a where keyword, eventually set after the if keyword: [price for item in basket if price is not None where price := item.get( 'price')] For this example one could use the walrus operator which is indeed smaller: [price for item in basket if price := item.get('price') is not None] But I fell that this approach is a bit opportunistic as one is doing two things at a time, if you allow me more lines for a somewhat more complex example: [ price * (1 + vat) for item in basket if price is not None where price := item.get('price') vat := get_vat(item, user) ] Now this example may look pretty stupid and probably one may simply use for loop, but I feel that this kind of Haskell-like where inside of list comprehension will let the programmer have a space where they can explicitly state the variables. Best regards, Fabrizio

2 years, 3 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas July 2019