BACKGROUND (skippable if you're a know-it-all) Argument parsing for Python functions follows some very strict rules. Unless the function implements its own parsing like so: def black_box(*args, **kwargs): there are some semantics that are always true. For example: * Any parameter that has a default value is optional, and vice-versa. * It doesn't matter whether you pass in a parameter by name or by position, it behaves the same. * You can see the default values by examining its inspect.Signature. * Calling a function and passing in the default value for a parameter is identical to calling the function without that parameter. e.g. (assuming foo is a pure function): def foo(a=value): ... foo() == foo(value) == foo(a=value) With that signature, foo() literally can't tell the difference between those three calls. And it doesn't matter what the type of value is or where you got it. Python builtins are a little less regular. They effectively do their own parsing. So they *could* do any crazy thing they want. 99.9% of the time they do one of four standard things: * They parse their arguments with a single call to PyArg_ParseTuple(). * They parse their arguments with a single call to PyArg_ParseTupleAndKeywords(). * They take a single argument of type "object" (METH_O). * They take no arguments (METH_NOARGS). PyArg_ParseTupleAndKeywords() behaves almost exactly like a Python function. PyArg_ParseTuple() is a little less like a Python function, because it doesn't support keyword arguments. (Surely this behavior is familiar to you!) But then there's that funny 0.1%, the builtins that came up with their own unique approach for parsing arguments--given them funny semantics. Argument Clinic tries to accomodate these as best it can. (That's why it supports "optional groups" for example.) But it can only do so much. THE PROBLEM Argument Clinic's original goal was to provide an introspection signature for every builtin in Python. But a small percentage of builtins have funny semantics that aren't expressable in a valid Python signature. This makes them hard to convert to Argument Clinic, and makes their signature inaccurate. If we want these functions to have an accurate Python introspection signature, their argument parsing will have to change. THE QUESTION What should someone converting functions to Argument Clinic do when faced with one of these functions? Of course, the simplest answer is "nothing"--don't convert the function to Argument Clinic. We're in beta, and any change that isn't a bugfix is impermissible. We can try again for 3.5. But if "any change" is impermissible, then we wouldn't have the community support to convert to Argument Clinic right now. The community wants proper signatures for builtins badly enough that we're doing it now, even though we're already in beta for Python 3.4. Converting to Argument Clinic is, in the vast majority of cases, a straightforward and low-risk change--but it is *a* change. Therefore perhaps the answer isn't an automatic "no". Perhaps additional straightforward, low-risk changes are permissible. The trick is, what constitutes a straightforward, low-risk change? Where should we draw the line? Let's discuss it. Perhaps a consensus will form around an answer besides a flat "no". THE SPECIFICS I'm sorting the problems we see into four rough categories. a) Functions where there's a static Python value that behaves identically to not passing in that parameter (aka "the NULL problem") Example: _sha1.sha1(). Its optional parameter has a default value in C of NULL. We can't express NULL in a Python signature. However, it just so happens that _sha1.sha1(b'') is exactly equivalent to _sha1.sha1(). b'' makes for a fine replacement default value. Same holds for list.__init__(). its optional "sequence" parameter has a default value in C of NULL. But this signature: list.__init__(sequence=()) works fine. The way Clinic works, we can actually still use the NULL as the default value in C. Clinic will let you use completely different values as the published default value in Python and the real default value in C. (Consenting adults rule and all that.) So we could lie to Python and everything works just the way we want it to. Possible Solutions: 0) Do nothing, don't convert the function. 1) Use that clever static value as the default. b) Functions where there's no static Python value that behaves identically to not passing in that parameter (aka "the dynamic default problem") There are functions with parameters whose defaults are mildly dynamic, responding to other parameters. Example: I forget its name, but someone recently showed me a builtin that took a list as its first parameter, and its optional second parameter defaulted to the length of the list. As I recall this function didn't allow negative numbers, so -1 wasn't a good fit. Possible solutions: 0) Do nothing, don't convert the function. 1) Use a magic value as None. Preferably of the same type as the function accepts, but failing that use None. If they pass in the magic value use the previous default value. Guido himself suggested this in 2) Use an Argument Clinic "optional group". This only works for functions that don't support keyword arguments. Also, I hate this, because "optional groups" are not expressable in Python syntax, so these functions automatically have invalid signatures. c) Functions that accept an 'int' when they mean 'boolean' (aka the "ints instead of bools" problem) This is specific but surprisingly common. Before Python 3.3 there was no PyArg_ParseTuple format unit that meant "boolean value". Functions generally used "i" (int). Even older functions accepted an object and called PyLong_AsLong() on it. Passing in True or False for "i" (or PyLong_AsLong()) works, because boolean inherits from long. But anything other than ints and bools throws an exception. In Python 3.3 I added the "p" format unit for boolean arguments. This calls PyObject_IsTrue() which accepts nearly any Python value. I assert that Python has a crystal clear definition of what constitutes "true" and "false". These parameters are clearly intended as booleans but they don't conform to the boolean protocol. So I suggest every instance of this is a (very mild!) bug. But changing these parameters to use "p" is a change: they'll accept many more values than before. Right now people convert these using 'int' because that's an exact match. But sometimes they are optional, and the person doing the conversion wants to use True or False as a default value, and it doesn't work: Argument Clinic's type enforcement complains and they have to work around it. (Argument Clinic has to enforce some type-safety here because the values are used as defaults for C variables.) I've been asked to allow True and False as defaults for "int" parameters specifically because of this. Example: str.splitlines(keepends) Solution: 1) Use "bool". 2) Use "int", and I'll go relax Argument Clinic so they can use bool values as defaults for int parameters. d) Functions with behavior that deliberately defy being expressed as a Python signature (aka the "untranslatable signature" problem) Example: itertools.repeat(), which behaves differently depending on whether "times" is supplied as a positional or keyword argument. (If "times" is <0, and was supplied via position, the function yields 0 times. If "times" is <0, and was supplied via keyword, the function yields infinitely-many times.) Solution: 0) Do nothing, don't convert the function. 1) Change the signature until it is Python compatible. This new signature *must* accept a superset of the arguments accepted by the existing signature. (This is being discussed right now in issue #19145.) //arry/