[Python-Dev] AC Derby and accepting None for optional positional arguments

[Larry Hastings]

On 01/15/2014 08:35 PM, Ryan Smith-Roberts wrote:
> On Wed, Jan 15, 2014 at 7:57 PM, Ryan Smith-Roberts <rmsr at lab.net 
> <mailto:rmsr at lab.net>> wrote:
>
>     socket.getservbyname(servicename[, protocolname])
>     ->
>     socket.getservbyname(servicename, protocolname=None)
>
>
> Here is a more complicated example, since the above does technically 
> have an alternative fix:
>
> sockobj.sendmsg(buffers[, ancdata[, flags[, address]]])
> ->
> sockobj.sendmsg(buffers, ancdata=None, flags=0, address=None)

I feel like Ryan didn't sufficiently explain the problem.  I'll elaborate.


For functions implemented in Python, it's always true that:

  * a parameter that is optional always has a default value, and
  * this default value is visible to Python and is a Python value.

The default value of every parameter is part of the function's 
signature--you can see them with inspect.signature() or 
inspect.getfullargspec().

A corollary of this: for every function implemented in Python, you can 
construct a call to it where you fill in every optional value with its 
published default value, and this is exactly equivalent to calling it 
without specifying those arguments:

    def foo(a=any_python_value): ...

    sig = inspect.signature(foo)
    defaults = [p.default for p in sig.parameters.values()]
    foo(*defaults) == foo()

Assuming neither foo nor "any_python_value" have side effects, those two 
calls to foo() will be exactly the same in every way.


But!  Builtin functions implemented in C commonly have optional 
parameters whose default value is not only opaque to Python, it's not 
renderable as a Python value.  That default value is NULL. Consider the 
first two paragraphs of SHA1_new() in Modules/sha1module.c:

    static PyObject *
    SHA1_new(PyObject *self, PyObject *args, PyObject *kwdict)
    {
         static char *kwlist[] = {"string", NULL};
         SHA1object *new;
         PyObject *data_obj = NULL;
         Py_buffer buf;

         if (!PyArg_ParseTupleAndKeywords(args, kwdict, "|O:new", kwlist,
                                          &data_obj)) {
             return NULL;
         }
    ...

The "string" parameter is optional.  Its value, if specified, is written 
to "data_obj".  "data_obj" has a default value of NULL. There is no 
Python value you could pass in to this function that would result in 
"data_obj" being (redundantly) set to NULL.  In Python SHA1_new is known 
as "_sha1.sha1".  And:

    sig = inspect.signature(_sha1.sha1)
    defaults = [p.default for p in sig.parameters.values()]
    _sha1.sha1(*defaults) == _sha1.sha1()

There's no value we could put in the signature for _sha1.sha1 that would 
behave exactly the same as that NULL.

The ultimate goal of Argument Clinic is to provide introspection 
information for builtins.  But we can't provide a default value to 
Python for the "string" parameter to _sha1.sha1() without changing the 
semantics of the function.  We're stuck.

Ryan's question, then, is "can we change a function like this so it 
accepts None?"  My initial reaction is "no".  That might be okay for 
_sha1.sha1(), but it'd be best to avoid.

In the specific case of SHA1_new's "string" parameter, we could lie and 
claim that the default value is b''.  Internally we could still use NULL 
as a default and get away with it.  But this is only a happy 
coincidence.  Many (most?) functions like this won't have a clever 
Python value we can trick you with.

What else could we do?  We could add a special value, let's call it 
sys.NULL, whose specific semantics are "turns into NULL when passed into 
builtins".  This would solve the problem but it's really, really awful.

The only other option I can see: don't convert SHA1_new() to use 
Argument Clinic, and don't provide introspection information for it either.

Can you, gentle reader, suggest a better option?


//arry/

p.s. Ryan's function signatures above suggest that he's converting code 
from using PyArg_ParseTuple into using PyArg_ParseTupleAndKeywords.  I 
don't think he's *actually* doing that, and if I saw that in patches 
submitted to me I would ask that it be fixed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140115/45413b69/attachment-0001.html>