I hope that in this thread we will share and develop our understanding
of how Python handles the interface between defining a function and
calling a function.
In this message, I describe a tension between the caller and
implementer of a function. I intend in further messages to cover
Elias Tarhini's post on __iter__(), keys(), and the mapping protocol
[This affects the behaviour of fn(**kwargs), perhaps to our disadvantage.]
https://mail.python.org/pipermail/python-ideas/2018-September/053320.html
Marko Ristin-Kaufmann post on Pre-conditions and post-conditions
[This is, in part, about wrapping a function, so it can be monitored.]
https://mail.python.org/pipermail/python-ideas/2018-August/052781.html
and probably some other topics. My main concern is that we know what
choices are already available, and the human forces that good design
decisions will balance.
The signature of a function has at least three purposes. First, to
provide the CALLER of the function with guidance as to how the
function should be used. Second, to provide the IMPLEMENTER of the
function with already initialised variables, local to the function
body. Third, to provide both caller and implementer with visible
default values.
When there are many arguments, these two purposes can be opposed.
Here's an example. The IMPLEMENTER might want to write (not tested)
def method(self, **kwargs):
# Do something with kwargs.
# Possibly mutate kwargs.
# Change values, remove items, add items.
# And now pass the method up.
super().method(**kwargs)
In some case, the implementer might prefer
def method(self, aaa, bbb, **kwargs):
# Do something with aaa and bbb.
super().method(**kwargs)
However, the CALLER might wish that the implementer has a signature
def method(self, aaa, bbb, ccc, ddd, eee, fff, ggg, hhh):
and this encourages the implementer to write super().method(aaa=aaa, ...).
However, there is an alternative:
def method(self, aaa, bbb, ccc, ddd, eee, fff, ggg, hhh):
lcls = dict(locals())
lcls.pop('aaa')
lcls.pop('bbb')
# Do something with aaa and bbb.
super().method(**lcls)
This implementation bring benefits to both the user and the
implementer. But it's decidedly odd that the local variables ccc
through to hhh are initialised, but not explicitly used. I think it
would help to dig deeper into this, via some well-chosen examples,
taken for real code.
By the way, Steve Barnes has suggested Python be extended to provide
"access to a method, or dictionary, called __params__ which would give
access, as a dictionary, to the parameters as supplied in the call,
(or filled in by the defaults)."
[See: https://mail.python.org/pipermail/python-ideas/2018-September/053322.html]
Such a thing already exists, and I've just showed how it might be used.
https://docs.python.org/3/library/functions.html#locals
>>> def fn(a, b, c, d=4, e=5): return locals()
>>> fn(1, 2, 3, e=6)
{'e': 6, 'd': 4, 'c': 3, 'a': 1, 'b': 2}
I think, once we understand Python function and code objects better,
we can make progress without extending Python, and know better what
extensions are needed. Much of what we need to know is contained in
the inspect module:
https://docs.python.org/3/library/inspect.html
--
Jonathan
Many features on this list propose different syntax to python, producing different python "dialects" that can statically be transformed to python :
- a,b += f(x) → _t = f(x); a += _t; b += _t; (augmented assignement unpacking)
- a = 2x + 1 → a = 2*x + 1 (juxtaposition is product)
- f(*, x, y) → f(x=x, y=y) (simplekwargs)
- DSL specific language
- all def become @partially def
- etc...
Using a modified version of ast, it is relatively easy to modifiy the syntax tree of a program to produce another program. So one could compile the "python dialect" into regular python. The last example with partially for example doesn't even need new syntax.
Those solutions that are too specific would then be provided as a module on pip that has a common interface for "compiling" :
$ cat test.dialect.py<http://test.dialect.py>
#! dialect: juxtaposition
a = 2x + 1
$ python -m compile test.dialect.py<http://test.dialect.py>
$ cat test.py
#! compiled with dialect juxtaposition
a = 2x + 1
The generated file should also be read only if the filesystem provides the option.
In the web world, it's very common to compile into html, css or js. One of the reason was that the web must be veeeery generic and will not try to fit everyone needs.
- less compiles scss into css
- coffeescript into js
- source map provides a standard way to map each line of the new file into lines of the old files (useful for exceptions !)
One useful feature of those compilers is the --watch Option that allows to avoid to launch the compilation manually.
Of course, in the js world, the syntax was improved in the end after a long maturation of the compiling and not compiling libraries.
In the java world, languages such as Scala compile into bytecode, that's another idea.
If a standard module like "compile" is written, users can write their own module that will automatically be read by "compile" (for example, pip install compile_juxtaposition would allow the juxtaposition dialect). Compile doesn't even have to be on the standard python, it can be a lib.
One could write a module using multiple dialect like dialect: juxtaposition, simplekwargs
The order would be technically important but functionally non important.
Actually, I might start to write this lib, that looks fun.
Did you mean to take this off-list? I hope not, as I'm bringing it back on.
On Tue, Sep 11, 2018 at 8:09 AM, Anders Hovmöller <boxed(a)killingar.net>
wrote:
> I've spent this whole thread thinking: "who in the world is writing code
> with a lot of spam=spam arguments? If you are transferring that much state
> in a function call, maybe you should have a class that holds that state? Or
> pass in a **kwargs dict?
>
>
> Kwargs isn’t good because it breaks static analysis which we really want.
>
well, Python isn't a static language, and I personally have my doubts about
trying to make it more so -- but that makes it sound like we need some
solution for type annotations, rather than executable code. But see my
other note -- I do agree that a well specified function signature is a good
thing.
> for .format() -- that's why we now have f-strings -- done.
>
>
> Not really no. F-strings can’t be used for many strings if you need to
> localize your app. If you don’t have that need then yes f-strings are great
> and I’m fortunate to work on an app where we don’t need to but we shouldn’t
> dismiss people who have this need.
>
Darn -- I hope this was brought up in the original f-string conversation.
> for templates -- are you really passing all that data in from a bunch of
> variables?? as opposed to, say, a dict? That strikes me as getting code and
> data confused (which is sometimes hard not to do...)
>
> Yes. For example we have decorators on our views that fetch up objects
> from our DB based on url fragments or query parameters and then pass to the
> function. This means we have all access control in our decorators and you
> can’t forget to do this because a view without access control decorators
> are stopped by a middleware. So several of variables are there before we
> even start the actual view function.
>
hmm -- this is a bit of what I mean by mixing data and code -- in my mind a
database record is more data than code, so better to use a dict than a
class with attributes matching teh fields. However, I also see the
advantage of mixing the code -- providing additional logic, and pre and
post processing, like it sounds like you are doing.
But python is a nifty dynamic language -- could your views have an
"as_dict" property that provided just the fields in the existing instance?
If you were in Stockholm I’d offer you to come by our office and I could
> show some code behind closed doors :)
>
As it happens, I am much closer than usual -- in the Netherlands, but still
not Stockholm :-)
- CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker(a)noaa.gov
Hi,
For technical reasons, many functions of the Python standard libraries
implemented in C have positional-only parameters. Example:
-------
$ ./python
Python 3.7.0a0 (default, Feb 25 2017, 04:30:32)
>>> help(str.replace)
replace(self, old, new, count=-1, /) # <== notice "/" at the end
...
>>> "a".replace("x", "y") # ok
'a'
>>> "a".replace(old="x", new="y") # ERR!
TypeError: replace() takes at least 2 arguments (0 given)
-------
When converting the methods of the builtin str type to the internal
"Argument Clinic" tool (tool to generate the function signature,
function docstring and the code to parse arguments in C), I asked if
we should add support for keyword arguments in str.replace(). The
answer was quick: no! It's a deliberate design choice.
Quote of Yury Selivanov's message:
"""
I think Guido explicitly stated that he doesn't like the idea to
always allow keyword arguments for all methods. I.e. `str.find('aaa')`
just reads better than `str.find(needle='aaa')`. Essentially, the idea
is that for most of the builtins that accept one or two arguments,
positional-only parameters are better.
"""
http://bugs.python.org/issue29286#msg285578
I just noticed a module on PyPI to implement this behaviour on Python functions:
https://pypi.python.org/pypi/positional
My question is: would it make sense to implement this feature in
Python directly? If yes, what should be the syntax? Use "/" marker?
Use the @positional() decorator?
Do you see concrete cases where it's a deliberate choice to deny
passing arguments as keywords?
Don't you like writing int(x="123") instead of int("123")? :-) (I know
that Serhiy Storshake hates the name of the "x" parameter of the int
constructor ;-))
By the way, I read that "/" marker is unknown by almost all Python
developers, and [...] syntax should be preferred, but
inspect.signature() doesn't support this syntax. Maybe we should fix
signature() and use [...] format instead?
Replace "replace(self, old, new, count=-1, /)" with "replace(self,
old, new[, count=-1])" (or maybe even not document the default
value?).
Python 3.5 help (docstring) uses "S.replace(old, new[, count])".
Victor
On Sat, Sep 08, 2018 at 09:54:59PM +0200, Anders Hovmöller wrote:
[...]
[Steven (me)]
> > If we believe that this consistency is desirable then maybe this would
> > be a good thing. Linters could warn when you use "name=spam" instead of
> > "*, name"; style guides can demand that code always uses this idiom
> > whenever practical, tutorials and blog posts will encourage it, and the
> > peer pressure to rename variables to match the called function's
> > parameters would be a good thing too.
>
> That’s the same straw man argument as before as far as I can tell.
[...]
Enough with the false accusations of straw-manning. I am stating my
prediction of the consequence of your proposal. Just because you
dislike the conclusion I draw doesn't make it a straw-man, and your
repeated false accusations (whether intentional or not) amount to
Poisoning The Well.
"Steven's arguments are straw-men, so we don't need to address
them or pay attention to him."
Except it isn't a straw-man. You might disagree, which is your right,
you might even just dismiss them as "well that's your opinion" (it *is*
my opinion, but I hope a reasoned, logical one), you might even think
that the consequences I state are a good thing. But stop trying to
poison my reputation by labelling me a straw-manner.
> > But if consistency for consistency's sake is not generally a good thing,
> > then we ought not to add such syntax just for conciseness.
>
> But conciseness for conciseness sake is just as not-good and we do
> have special syntax for that: positional arguments. I’m proposing to
> level the playing field.
For short argument lists, there's little or nothing wrong with
positional arguments. In fact, I would say that positional arguments
are, in some circumstances, far better:
len(obj) versus len(target=obj)
mylist.append(item) versus mylist.append(obj=item)
My argument isn't about making absolute judgements of what is good or
bad in all cases. It is about weighing up the benefit in some cases
(which I have acknowledged) against the disadvantage in other cases.
[...]
> > Of course I understand that with this proposal, there's nothing
> > *forcing* people to use it. But it shifts the *preferred* idiom from
> > explicit "name=spam" to implicit "*, name" and puts the onus on people
> > to justify why they aren't naming their local variables the same as the
> > function parameter, instead of treating "the same name" as just another
> > name.
>
> Did you just argue that my proposed syntax is so great it’s going to
> become the preferred idiom? :)
No. I argued that your proposed syntax could become the perferred idiom
because it is short and concise and people mistake that for "great".
I'm not blind to the advantage. If I had to make many function calls
that looked like this:
result = function(spam=spam, eggs=eggs, cheese=cheese,
foo=foo, bar=bar, baz=baz, fe=fe,
fi=fi, fo=fo, fum=fum)
I'd be sick of it too and want something better. But we have to balance
many competing and often contradictory interests, and I don't think you
have made your case that the pain of the above is worse than the
(alleged) benefit of being able to write it in a more terse, implicit
way:
result = function(*, spam, eggs, cheese, foo, bar, baz,
fe, fi, fo, fum)
is shorter, but is it really better? I don't think you've made that
case. But many people do seem to think shorter is better as a general
rule.
> > but that active preference for long parameter lists seems to be a very
> > rare, more common is the view that *at best* long parameter lists is a
> > necessary evil that needs mitigation.
>
> I think that’s overly doom and gloomy. Just look at all the arguments
> for open().
Yes, open() is a good example of how long parameter lists are not
necessarily a bad thing. With sensible defaults, probably 95% of
calls to open need no more than one or two arguments.
But can you remember the Bad Old Days before programming languages
supported default values? I do. Now imagine making a call to open() with
eight required arguments.
> In any case I’m proposing a way to mitigate the pain in a subset of
> cases (probably not the case of open()!).
Yes, I acknowledge that. Just because I sympathise with your pain
doesn't mean I want to see your syntax added to the language.
> > certainly wouldn't want to put a hard limit on the number of
> > parameters allowed. But in general, I think it is unquestionable that
> > long parameter lists are a code-smell.
>
> Or a domain smell. Some domains are smelly.
Indeed.
> > It is also relevant in this sense. Large, complex function calls are
> > undoubtably painful. We have mitigated that pain somewhat by various
> > means, probably the best of which are named keyword arguments, and
> > sensible default values. The unintended consequence of this is that it
> > has reduced the pressure on developers to redesign their code to avoid
> > long function signatures, leading to more technical debt in the long
> > run.
>
> So... you’re arguing Python would have been better off without keyword
> arguments and default values?
Not at all. As I already stated, keyword arguments are great, and I'm
not arguing for hair-shirt programming where we intentionally make
things as painful as possible.
I'll make an analogy here. Pain, real physical pain, is bad, and it is
both kind and medically advantagous to reduce it when necessary. But
*necessary* is the key word, because sometimes pain is an important
biological signal that says Don't Do That. People with no pain receptors
tend to die young because they repeatedly injure themselves and don't
even know it. Good doctors make careful judgement about the minimum
amount of painkillers needed, and don't hand out morphine for stubbed
toes because the consequences will be worse than the benefit gained.
The analogy in programming follows: reducing immediate pain can,
sometimes, cause long-term pain in the form of technical debt,
which is worse. It takes a careful balancing act to decide when it is
appropriate to take on technical debt in order to avoid short-term
annoyance. We cannot hope to make such a sensible decision if we focus
only on the immediate relief and not on the long term consequences.
> > Your suggestion would also reduce the pain of functions that require
> > many arguments. That is certainly good news if the long argument list is
> > *truly necessary* but it does nothing to reduce the amount of complexity
> > or technical debt.
> > The unintended consequence is likewise that it
> > reduces the pressure on developers to avoid designing such functions in
> > the first place.
> >
> > This might sound like I am a proponent of hair-shirt programming where
> > everything is made as painful as possible so as to force people to
> > program the One True Way. That's not my intention at all. I love my
> > syntactic sugar as much as the next guy. But I'd rather deal with the
> > trap of technical debt and excessive complexity by avoiding it in the
> > first place, not by making it easier to fall into.
>
> But you’ve only proposed or implied ways to avoid it by force and
> pain.
I don't think the pain is all that great. Its a minor annoyance to write
keyword arguments like parameter=parameter, and a good IDE or editor can
help there. But I acknowledge there is some pain, and the more you need
to do this, the more you feel it.
But in any case, you made a proposal. I don't have to come up with a
better proposal before I am permitted to argue against yours.
"We must do something. This is something, therefore we must do it."
is never a valid argument.
> And you’re arguing my suggestion is bad because it lowers pain.
> So I think you’re contradicting yourself here. That’s fine but you
> should admit it and call it a trade off at least.
I have done nothing but call it a trade off! I've repeatedly argued that
in my opinion the benefit (which I have repeatedly acknowledged!)
doesn't outweigh the (predicted) costs as I see them.
[...]
> >> There are actual APIs that have lots of arguments. GUI toolkits are a great
> >> example. Another great example is to send a context dict to a template
> >> engine.
> >
> > Indeed. And I'm sympathetic that some tasks are inherently complex and
> > require many arguments. Its a matter of finding a balance between being
> > able to use them, without encouraging them.
>
> What does that mean? We shouldn’t encourage GUI toolkits for Python?
> Or we shouldn’t encourage working with complex domains? You probably
> meant to say something else but it came across weirdly.
Sorry for being unclear. I meant that we shouldn't encourage complex,
multi-argument functions, while still accepting that sometimes they are
unavoidable (and even occasionally a good thing).
> > You claimed the benefit of "conciseness", but that doesn't actually
> > exist unless your arguments are already local variables named the same
> > as the parameters of the function you are calling.
>
> Which they often already are. Again: I have numbers. You do not.
I don't dispute the numbers you get from your code base. You should be
careful about generalising from your code to other people's, especially
if you haven't seen their code.
> > Pointing out that weakness in your argument is not a straw man.
>
> You did notice that not just me called it a straw man on this list right?
Yes I did. Many people call the world flat too. What's your point?
--
Steve
I wrote a blog post
<http://paddy3118.blogspot.com/2009/07/case-of-disappearing-over-bar.html>nearly
a decade ago on extending a Rosetta Code task example
<http://rosettacode.org/wiki/Reverse_a_string>to handle the correct
reversal of strings with combining characters.
On checking my blog statistics today I found that it still had a readership
and revisited the code
<http://rosettacode.org/wiki/Reverse_a_string#Python:_Unicode_reversal>
(and updated it to Python3.6)..
I found that amongst the nearly 200 languages that complete the RC
task,there were a smattering of languages that correctly handled reversing
strings having Unicode combining characters,
including Perl 6 <http://rosettacode.org/wiki/Reverse_a_string#Perl_6>
which uses flip.
I would like to propose that Python add a Unicode-aware *str.reverse *method.
The problem is, I'm a Brit, who only speaks English and only very rarely
dips into Unicode.* I don't know how useful this would be!*
Cheers, Paddy.
>
> It’s obvious but there is one easy way to shorten the code: using
> **kwargs. It’s way shorter but the down sides are:
- the “real” function signature gets hidden so IDEs for example won’t pick
> it up
> - the error when you make a mistake when calling is not in your code
> anymore but one level down.
This is confusing. One could imagine solving this specific case by having a
> type annotation of “this function has the types of that function”. Maybe:
> def _open(*args: args_of_(sync_open), **kwargs: kwargs_of(sync_open) ->
> return_of(sync_open): But of course this only solves the case where there
> is a 1:1 mapping. / Anders
These problems could be solved by a decorator that accepts string
representation of the signature. The decorator would then have to parse the
signature at importing time and set it to the __signature__ attribute on
the resultant function. This decorator would also need to bind the
arguments e.g. sig.bind(*args, **kwargs), to handle out of order positional
arguments. Therefore this would raise an error in the decorator,
essentially solving your second point.
This would make the example look like this, a lot clearer in my opionion:
@signature('''(file, mode='r', buffering=-1, encoding=None, errors=None,
newline=None,
closefd=True, opener=None, *, loop=None,
executor=None)''')
def open(*args, **kwargs):
return AiofilesContextManager(_open(*args, **kwargs))
@asyncio.coroutine
def _open(*args, loop=None, executor=None, **kwargs):
"""Open an asyncio file."""
if loop is None:
loop = asyncio.get_event_loop()
cb = partial(sync_open, *args, **kwargs)
f = yield from loop.run_in_executor(executor, cb)
return wrap(f, loop=loop, executor=executor)
Ben Lewis
>
> def foo():
> a, b, c = 1, 2, 3
> function(a=77, **use('b d'))
>
> foo()
> You get the output “77 None 33 None”. So basically it doesn’t work at all.
> For the reason I wrote clearly in my last mail. You have to dig through the
> stack frames to make use() work.
>
OK, you are right. Improved implementation. Still not very hard.
In any case, I'm concerned with the API to *use* the `use()` function, not
how it's implemented. The point is really just that we can accomplish the
same thing you want without syntax added.
>>> import inspect
>>> def reach(name):
... for f in inspect.stack():
... if name in f[0].f_locals:
... return f[0].f_locals[name]
... return None
...
>>> def use(names):
... kws = {}
... for name in names.split():
... kws[name] = reach(name)
... return kws
...
>>> def function(a=11, b=22, c=33, d=44):
... print(a, b, c, d)
...
>>> function(a=77, **use('b d'))
77 None 33 None
>>> def foo():
... a, b, c = 1, 2, 3
... function(a=77, **use('b d'))
...
>>> foo()
77 2 33 None
Op za 8 sep. 2018 13:33 schreef Paddy3118 <paddy3118(a)gmail.com>:
>
> I would like to propose that Python add a Unicode-aware *str.reverse *method.
> The problem is, I'm a Brit, who only speaks English and only very rarely
> dips into Unicode.* I don't know how useful this would be!*
>
To be honest, quite apart from the Unicode issue, I never had a need to
reverse a string in real code.
.ytilibigel edepmi ot sdnet yllareneg tI
Stephan
> Cheers, Paddy.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas(a)python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
