Sent from my iPhone
Begin forwarded message:
> From: James Lu <jamtlu(a)gmail.com>
> Date: July 28, 2019 at 6:21:04 PM EDT
> To: Andrew Barnert <abarnert(a)yahoo.com>
> Subject: Re: [Python-ideas] Utilities for easier debugging
>
>
>> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>>
>> Why not just allow anything that’s valid as a target, like identifier[expr]? Or even just any expression at all? Is there an advantage to defining and using a similar but more limited syntax here?
> Hmm, it should support any expression. I limited it at first because it was a minimal strawman.
Sent from my iPhone
Begin forwarded message:
> From: James Lu <jamtlu(a)gmail.com>
> Date: July 28, 2019 at 6:21:04 PM EDT
> To: Andrew Barnert <abarnert(a)yahoo.com>
> Subject: Re: [Python-ideas] Utilities for easier debugging
>
>
>> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>>
>> Why not just allow anything that’s valid as a target, like identifier[expr]? Or even just any expression at all? Is there an advantage to defining and using a similar but more limited syntax here?
> Hmm, it should support any expression. I limited it at first because it was a minimal strawman.
Sent from my iPhone
Begin forwarded message:
> From: James Lu <jamtlu(a)gmail.com>
> Date: July 28, 2019 at 6:22:11 PM EDT
> To: Andrew Barnert <abarnert(a)yahoo.com>
> Subject: Re: [Python-ideas] Utilities for easier debugging
>
>
>> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>>
>> This would break iPython’s improved interactive console, which already uses this syntax to provide a similar feature.
> If it’s so similar, I doubt it would break anything. This is intended to make it easier to log variables for a complex application (multiple files).
Sent from my iPhone
Begin forwarded message:
> From: James Lu <jamtlu(a)gmail.com>
> Date: July 28, 2019 at 6:23:06 PM EDT
> To: Andrew Barnert <abarnert(a)yahoo.com>
> Subject: Re: [Python-ideas] Utilities for easier debugging
>
>
>> On Jul 28, 2019, at 4:26 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote:
>>
>> You want this added to every module’s globals? Why not put it in builtins instead?
> It’s not a function call, it’s an expression that when evaluated triggers a breakpoint, so you have to add it as a property descriptor.
[redirecting back to the list]
On 07/27/2019 09:38 PM, James Lu wrote:
>> On Jul 27, 2019, at 12:44 PM, Ethan Furman wrote:
>> Sure, folks /know/ what it means, but it's a common bug because
>> it doesn't read as "if some_var is assigned 7" but as "if some_var
>> is equal to 7".
>
> That’s a straw man- a modern linter or compiler would catch that.
> Many editors have linters integrated inside. The way to disable it
> is to put parentheses around the assignment expesssion.
Hello,
currently, regarding positional arguments, `partial` gives us the option to partialize functions from the left. There's been some interest about partializing functions from the right instead (e.g. [SO post, 9k views, 39 upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the various `str` methods.
I propose adding a function to `functools` that works with placeholders and thus offers even greater flexibility. The Ellipsis literal `...` seems a intuitive choice for that task. When eventually calling such a "partial placeholder" object, it would fill in placeholders from the left and add remaining `args` to the right. In terms of implementation this can be realized as a subclass of `partial` itself.
## Implementation
from functools import partial
from reprlib import recursive_repr
class partial_placehold(partial):
placeholder = Ellipsis
def __call__(self, /, *args, **keywords):
args = iter(args)
try:
old_args = [x if x is not self.placeholder else next(args) for x in self.args]
except StopIteration:
raise TypeError('too few arguments were supplied') from None
keywords = {**self.keywords, **keywords}
return self.func(*old_args, *args, **keywords)
@recursive_repr()
def __repr__(self):
qualname = type(self).__qualname__
args = [repr(self.func)]
args.extend(repr(x) if x is not self.placeholder else '...' for x in self.args) # Only this line deviates from `partial.__repr__`; could also factor that out into a separate method.
args.extend(f"{k}={v!r}" for (k, v) in self.keywords.items())
if type(self).__module__ == "functools":
return f"functools.{qualname}({', '.join(args)})"
return f"{qualname}({', '.join(args)})"
# Would need to add something for compatibility with `partial`, i.e. for partializing a placeholder function.
## Example
This allows for example the following usage:
replace_dots_with_underscore = partial_placehold(str.replace, ..., '.', '_')
replace_dots_with_underscore('foo.bar.baz')
## Relevance
Sure we could also use a `lambda` instead ([as discussed here](https://mail.python.org/archives/list/python-ideas@python.org/messa...) but there was a reason `partial` was introduced and I think the same arguments apply here too. Though most functions allow partializing via keyword arguments and this is undoubtedly a cleaner way, some might not and for example built-ins' methods won't allow it. Especially Python 3.8's introduction of positional-only parameters (PEP 570) might give rise to cases where `partial` is not sufficient.
In case inspection is desired a `lambda` does not provide much information (sure you could always dig deeper with `inspect` for example but that's not the point). Consider the following example of a pre-defined sequence of default postprocessing steps and the user might add their own or remove existing ones, as appropriate:
postprocessing_steps = [
lambda s: s.replace('foo', 'bar'),
]
print(postprocessing_steps[0]) # <function <lambda> at 0x7f94a850dd30>
This doesn't give a lot of information about what the lambda actually does (and thus whether the user should remove it or not). Using the `partial_placehold` instead, it's clear what is happening:
postprocessing_steps = [
partial_placehold(str.replace, ..., 'foo', 'bar'),
]
print(postprocessing_steps[0]) # partial_placehold(<method 'replace' of 'str' objects>, ..., 'foo', 'bar')
## Compatibility
The proposed solution works with the current syntax and the usage of Ellipsis as a placeholder object is likely not to collide with actually used values (in any case the user might still reassign the `.placeholder` attribute).
Because the direction of partializing is unchanged (still left to right) this doesn't introduce ambiguities which might come with a "right partial" function. Creating a placeholder function from a `partial` object is possible without any changes, the opposite way requires an additional check to result in a placeholder object again.
## Possible confusion
Regarding the usage of Ellipsis right now, in `numpy` or `typing` for example, it always represents a placeholder for multiple "things", not a single one:
array[..., None] # All the dimensions of `array` plus a new one.
typing.Tuple[str, ...] # Any number of str objects.
So the expectations might be biased in that sense. For example:
def foo(a, b, c, d):
pass
p_foo = partial_placehold(foo, ..., 1, 2)
p_foo(3, 4)
Someone else reviewing the code might now assume that the `...` means to act as a placeholder for all arguments except the last two (and hence `p_foo(3, 4)` would be equivalent to `foo(3, 4, 1, 2)` while it actually is equivalent to `foo(3, 1, 2, 4)`). But this would be again some kind of "right partial" function and also the function name implies something else; documentation might clarify as well, of course.
## Conclusion
Adding a "partial with placeholders" function to `functools` allows for covering use cases where the standard `partial` is not sufficient. No new syntax is required and the implementation is fairly straightforward given the inheritance from `partial`. Ellipsis `...` seems an intuitive choice for acting as a placeholder (concerning both, conflicts with actual partial values and code readability). There are uses cases where such a function would provide a clean solution and there is an interest in the community (https://stackoverflow.com/q/7811247/3767239, https://stackoverflow.com/q/19701775/3767239 for example). Especially with the introduction of positional-only parameters new use cases are likely to arise.
-----
**Related threads:**
* https://mail.python.org/archives/list/python-ideas@python.org/message/TVN... - Mentions essentially a similar idea.
The original [PEP 309 -- Partial Function Application](https://www.python.org/dev/peps/pep-0309/) also mentions:
> Partially applying arguments from the right, or inserting arguments at arbitrary positions creates its own problems, but pending discovery of a good implementation and non-confusing semantics, I don't think it should be ruled out.
26.07.19 21:52, Bruce Leban пише:
>
> On Fri, Jul 26, 2019 at 11:27 AM Serhiy Storchaka <storchaka(a)gmail.com
> <mailto:storchaka@gmail.com>> wrote:
>
>
> So you will be able to add errors handling like in:
>
> with connect() as stream:
> for data in stream:
> try:
> write(data)
> except OSError:
> handle_write_error()
> except OSError:
> handle_read_error()
> except OSError:
> handle_connection_error()
>
>
> To put this in a simpler way: the proposal is to add an except clause
> that applies ONLY to the direct operation of the with or for statement
> and not to the block. That's an interesting idea.
>
> The one thing I find confusing about your proposal is that the
> proposed syntax does not imply the behavior. In a try statement, the
> except appears at the end and after all possible statements that it
> could cover. The proposal mimics that syntax but with different
> semantics. Something like this would be much more clear what is going on:
>
> for VARIABLE in EXPRESSION:
> except EXCEPTION:
> BLOCK
> BLOCK
>
> with EXPRESSION as VARIABLE:
> except EXCEPTION:
> BLOCK
> BLOCK
>
> while EXPRESSION:
> except EXCEPTION:
> BLOCK
> BLOCK
Besides an unusual for Python layout (a clause has different indentation
than the initial clause of the statement to which it belongs) there is
other problem. The exception block is not the part of the "for" or
"with" block. After handling an exception in the "for" clause you do not
continue to execute the "for" block, but leave the loop. After handling
an exception in the "with" clause you do not continue to execute the
"with" block and do not call `__exit__` when leave it. To me, this
syntax is much more confusing than my initial proposition.
Hi everyone,
I'm in the middle of developing a fancy heap profiler for Python (for those
times when tracemalloc isn't enough), and in the process, thought of a
small change in the interpreter which could have a lot of uses. I call it
"scope painting" because it lets you paint a label on an interpreter scope.
*Conceptual TL;DR: *Add a function which attaches a string label to the
current interpreter frame. (This would go away when you either explicitly
cleared it, or the frame ended, i.e. the current function returned)
You can then use it for:
- *Debugging: *Print this value out, if set, in the traceback
formatters. This can be used e.g. in a server, by putting a unique request
identifier as the label for the top-level handler method; then if something
in the particular query triggers a crash, it's *much* easier to see what
caused it. (When I was at Google, we did something very similar to this for
the servers in search. It completely transformed the way we detected
"queries of death," and made hunting down data-dependent bugs an order of
magnitude easier.)
- *CPU and heap profiling: *If a profiler or tracer stores this
information when it's grabbing the stack trace, this can be used for two
different kinds of analysis, depending on user needs:
- *Splitting: *Consider two stack traces different if they differ in
any of the labels. This lets you separate flows which seem to be going
through the same part of the logic, but are actually doing it on very
distinct paths.
- *Joining: *Group stack traces by the label; this lets you identify
"call with this label value and its descendants," which lets you very
easily establish a user-defined aggregation for total CPU or heap usage.
- *Network request tracing: *If you were feeling really fancy, you could
attach this to a cross-network-request trace ID and propagate the same
value of this across multiple servers in a complicated request chain. Then
you could perform all of the above joinings, plus similar joinings from
cross-server profiling systems, as well.
*Implementation: *Pretty simple: add a new field PyObject *f_label to
PyFrameObject which contains an optional Unicode string; the frame owns a
reference. Add get and set methods, exposing those in the Python API, and
probably also a simple context manager to set and restore the value of this
field because that's going to be the 90% way it's used. (Based on
experience with doing similar things in C++)
Modify the traceback, profile, cProfile, and tracemalloc libraries (and
maybe others?) to use this value as well.
What do people think?
Yonatan
Greetings,
Just like builtins module, would it be a good idea to have a stdlib module
so that we can know all modules in the std lib (just like builtins let us
know builtins)?
Abdur-Rahmaan Janhangeer
Mauritius
What does everyone think about making "tempfile.NamedTemporaryFile()" not raising "FileNotFoundError" in case if when it closes and the file is already deleted?
