Mailman 3 January 2014 - Python-Dev

lambda (x, y):
by Ram Rachum 26 Jan '14

26 Jan '14

I don't like how in Python 3.x, you can't do this: lambda (x, y): whatever It's quite useful in Python 2 if I understand correctly, it's a side effect of such packed arguments not being allowed in function definitions. (i.e. def instead of lambda) Can you please refer me to the original discussion in which it was decided to remove this grammar in Python 3? I'd like to understand the arguments for it. Thanks, Ram.

8 9

Deprecation policy
by Ezio Melotti 26 Jan '14

26 Jan '14

Hi, a couple of years ago I suggested to define and document our deprecation policy in this thread: https://mail.python.org/pipermail/python-dev/2011-October/114199.html I didn't receive many replies and eventually nothing was done. Lately the same issue came up on #python-dev and Larry and Nick suggested me to bring this up again. Nick also suggested to document our deprecation policy in PEP 5 (Guidelines for Language Evolution: http://www.python.org/dev/peps/pep-0005/ ). I'm including below the full text of the original email. Best Regards, Ezio Melotti ------------------------------- Hi, our current deprecation policy is not so well defined (see e.g. [0]), and it seems to me that it's something like: 1) deprecate something and add a DeprecationWarning; 2) forget about it after a while; 3) wait a few versions until someone notices it; 4) actually remove it; I suggest to follow the following process: 1) deprecate something and add a DeprecationWarning; 2) decide how long the deprecation should last; 3) use the deprecated-remove[1] directive to document it; 4) add a test that fails after the update so that we remember to remove it[2]; Other related issues: PendingDeprecationWarnings: * AFAIK the difference between PDW and DW is that PDW are silenced by default; * now DW are silence by default too, so there are no differences; * I therefore suggest we stop using it, but we can leave it around[3] (other projects might be using it for something different); Deprecation Progression: Before, we more or less used to deprecated in release X and remove in X+1, or add a PDW in X, DW in X+1, and remove it in X+2. I suggest we drop this scheme and just use DW until X+N, where N is >=1 and depends on what is being removed. We can decide to leave the DW for 2-3 versions before removing something widely used, or just deprecate in X and remove in X+1 for things that are less used. Porting from 2.x to 3.x: Some people will update directly from 2.7 to 3.2 or even later versions (3.3, 3.4, ...), without going through earlier 3.x versions. If something is deprecated on 3.2 but not in 2.7 and then is removed in 3.3, people updating from 2.7 to 3.3 won't see any warning, and this will make the porting even more difficult. I suggest that: * nothing that is available and not deprecated in 2.7, will be removed until 3.x (x needs to be defined); * possibly we start backporting warnings to 2.7 so that they are visible while running with -3; Documenting the deprecations: In order to advertise the deprecations, they should be documented: * in their doc, using the deprecated-removed directive (and possibly not the 'deprecated' one); * in the what's new, possibly listing everything that is currently deprecated, and when it will be removed; Django seems to do something similar[4]. (Another thing I would like is a different rending for deprecated functions. Some part of the docs have a deprecation warning on the top of the section and the single functions look normal if you miss that. Also while linking to a deprecated function it would be nice to have it rendered with a different color or something similar.) Testing the deprecations: Tests that fail when a new release is made and the version number is bumped should be added to make sure we don't forget to remove it. The test should have a related issue with a patch to remove the deprecated function and the test. Setting the priority of the issue to release blocker or deferred blocker can be done in addition/instead, but that works well only when N == 1 (the priority could be updated for every release though). The tests could be marked with an expected failure to give some time after the release to remove them. All the deprecation-related tests might be added to the same file, or left in the test file of their module. Where to add this: Once we agree about the process we should write it down somewhere. Possible candidates are: * PEP387: Backwards Compatibility Policy[5] (it has a few lines about this); * a new PEP; * the devguide; I think having it in a PEP would be good, the devguide can then link to it. Best Regards, Ezio Melotti [0]: http://bugs.python.org/issue13248 [1]: deprecated-removed doesn't seem to be documented in the documenting doc, but it was added here: http://hg.python.org/cpython/rev/03296316a892 [2]: see e.g. http://hg.python.org/cpython/file/default/Lib/unittest/test/test_case.py#l1… [3]: we could also introduce a MetaDeprecationWarning and make PendingDeprecationWarning inherit from it so that it can be used to pending-deprecate itself. Once PendingDeprecationWarning is gone, the MetaDeprecationWarning will become useless and can then be used to meta-deprecate itself. [4]: https://docs.djangoproject.com/en/dev/internals/deprecation/ [5]: http://www.python.org/dev/peps/pep-0387/

6 5

Re: [Python-Dev] cpython: Issue 19944: Fix importlib.find_spec() so it imports parents as needed.
by Antoine Pitrou 26 Jan '14

26 Jan '14

On Sat, 25 Jan 2014 23:37:49 +0100 (CET) eric.snow <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/665f1ba77b57 > changeset: 88710:665f1ba77b57 > user: Eric Snow <ericsnowcurrently(a)gmail.com> > date: Sat Jan 25 15:32:46 2014 -0700 > summary: > Issue 19944: Fix importlib.find_spec() so it imports parents as needed. > > The function is also moved to importlib.util. Is there a reason to have separate "importlib" (toplevel) and "importlib.util" namespaces? (the doc says "This module contains the various objects that help in the construction of an importer", which doesn't sound related to find_spec()) Regards Antoine.

3 2

Re: [Python-Dev] cpython: Issue #20133: The audioop module now uses Argument Clinic.
by Christian Heimes 26 Jan '14

26 Jan '14

On 25.01.2014 10:58, serhiy.storchaka wrote: > http://hg.python.org/cpython/rev/d4099b8a7d0f > changeset: 88687:d4099b8a7d0f > user: Serhiy Storchaka <storchaka(a)gmail.com> > date: Sat Jan 25 11:57:59 2014 +0200 > summary: > Issue #20133: The audioop module now uses Argument Clinic. > > files: > Modules/audioop.c | 1077 ++++++++++++++----------- > Modules/audioop.clinic.c | 836 ++++++++++++++++++++ > 2 files changed, 1427 insertions(+), 486 deletions(-) > Coverity has detected an issue in this commit: ** CID 1164423: Division or modulo by zero (DIVIDE_BY_ZERO) /Modules/audioop.c: 1375 in audioop_ratecv_impl() ________________________________________________________________________________________________________ *** CID 1164423: Division or modulo by zero (DIVIDE_BY_ZERO) /Modules/audioop.c: 1375 in audioop_ratecv_impl() 1369 without spurious overflow is the challenge; we can 1370 settle for a reasonable upper bound, though, in this 1371 case ceiling(len/inrate) * outrate. */ 1372 1373 /* compute ceiling(len/inrate) without overflow */ 1374 Py_ssize_t q = len > 0 ? 1 + (len - 1) / inrate : 0; >>> CID 1164423: Division or modulo by zero (DIVIDE_BY_ZERO) >>> In expression "9223372036854775807L / q", division by expression "q" which may be zero has undefined behavior. 1375 if (outrate > PY_SSIZE_T_MAX / q / bytes_per_frame) 1376 str = NULL; 1377 else 1378 str = PyBytes_FromStringAndSize(NULL, 1379 q * outrate * bytes_per_frame); 1380 }

2 1

New policies for the Derby -- please read!
by Larry Hastings 26 Jan '14

26 Jan '14

1) New policy for what can and cannot be converted during the Derby First, let me apologize for only figuring this out now. The Derby has been a learning process, discovering things that Argument Clinic didn't handle. And there were a lot of funny edge cases that we weren't going to discover until we tried doing the conversion. And to be honest I was pushing harder than I should have. *ahem* The new policy for conversion work for Python 3.4: We may only convert functions that have signatures that we can represent 100% accurately in an inspect.Signature object. IF the function has a default value that can't be represented in Python (e.g. NULL), but we can find a value in Python that behaves identically to not passing in that parameter (e.g. _sha1.sha1(b'') == _sha1.sha1()), then we may convert that function using that clever default value (e.g. "string: object(c_default='NULL') = b'' "). IF the function has parameters with default values that are dynamic or cannot be represented accurately in Python, it cannot be converted without changing its semantics. So it cannot be converted for 3.4. IF the function *requires* "optional groups" (as in, the original function used switch(PyTuple_GET_SIZE(args)) and multiple calls to PyArg_ParseTuple()), then it's permissible to convert it for 3.4, but they are low priority. Such functions have semantics that are so weird, we will have to modify inspect.Parameter to support them, and that will only happen in 3.5. However, I will ensure that they otherwise convert correctly for 3.4. (They won't actually generate signatures. They will however generate the first line of the docstring.) For 3.5, I expect we'll have more leeway in doing things like "this should accept an int or None". But we should talk about that then. If you have patches outstanding that convert functions that shouldn't be converted for 3.4, please put them aside. We can almost certainly use them for 3.5. If any conversions have been committed that change the semantics of the function, someone will have to back them out. I'd appreciate it if the person who checked it in could do it. I expect to make a pass before rc1 to check all the conversions myself. (Which I hope will be quite time-consuming, as hopefully there will be lots of converted functions by then!) _______________________________________________________________________________ 2) New recommendation for marking to-do functions If you examine a function and determine that it can't be converted to Argument Clinic right now, please add a comment to that effect. The comment should be one line, and contain a special marker so we can find it easily with searches. I nominate two different markers: "AC 3.4" means "It's okay to convert this function to Argument Clinic in 3.4, but it can't be converted right now because of a bug or missing feature in Argument Clinic." "AC 3.5" means "This function can't be converted to Argument Clinic in 3.4, it must wait for 3.5." I encourage you to add a little text saying why, like: /* AC 3.4: waiting for *args support */ /* AC 3.5: value parameter has default value of NULL */ _______________________________________________________________________________ Finally, I've realized that right now there's no good way to stay abreast of what's new and changing in clinic.py. I check in a patch just about every day for clinic.py, and sometimes I don't remember to update the "howto". The best way unfortunately is to read the output of "hg log". If you have questions, you can email me directly at "larry at hastings dot org", or you can find me in the #python-dev IRC channel. Thank you, everybody who's participating in the Derby, and again I'm sorry I didn't realize sooner this *had* to be the policy for 3.4. Hope to see your issues in the tracker, /arry

3 3

Quick poll: should help() show bound arguments?
by Larry Hastings 25 Jan '14

25 Jan '14

(Quick, because apparently nobody reads the long ones!) In Python 3.3: >>> class C: ... def foo(self, a): pass ... >>> c = C() >>> help(c.foo) shows you the signature "foo(self, a)". As in, it claims it accepts two parameters. The function actually only accepts one parameter, because "self" has already been bound. inspect.signature gets this right: >>> import inspect >>> str(inspect.signature(c.foo)) '(a)' but inspect.getfullargspec does not: >>> inspect.getfullargspec(c.foo) FullArgSpec(args=['self', 'a'], varargs=None, varkw=None, defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={}) help() gets its text from pydoc. pydoc uses inspect.getfullargspec to produce the signature. When I added support for introspection on builtins, I wanted help() to show their signature too. But inspect.getfullargspec doesn't support introspection on builtins. So I had to use inspect.signature. Which means the behavior is inconsistent: help() on a method of an instance of a builtin class *doesn't* show "self". FYI, the relevant issues: help(instance_of_builtin_class.method) does not display self http://bugs.python.org/issue20379 inspect.getfullargspec should use __siganture__ http://bugs.python.org/issue17481 What should it be? A) pydoc and help() should not show bound parameters in the signature, like inspect.signature. B) pydoc and help() should show bound parameters in the signature, like inspect.getfullargspec. I'll tally the results if there's interest. I'd assume a "vote for A" = +1 on A and -1 on B. You can express your vote numerically if you like. I'm voting for A. And yes, that was short for me, //arry/

9 11

Re: [Python-Dev] cpython: Issue #20311: asyncio: Add a granularity attribute to BaseEventLoop: maximum
by Antoine Pitrou 25 Jan '14

25 Jan '14

On Sat, 25 Jan 2014 15:02:56 +0100 (CET) victor.stinner <python-checkins(a)python.org> wrote: > + > + @tasks.coroutine > + def wait(): > + loop = self.loop > + calls.append(loop._run_once_counter) > + yield from tasks.sleep(loop.granularity * 10, loop=loop) > + calls.append(loop._run_once_counter) > + yield from tasks.sleep(loop.granularity / 10, loop=loop) > + calls.append(loop._run_once_counter) > + > + self.loop.run_until_complete(wait()) > + calls.append(self.loop._run_once_counter) > + self.assertEqual(calls, [1, 3, 5, 6]) Could you add a comment explaining the number of calls to run_once()? For example, why does it jump from 1 to 3 and then 3 to 5, rather than 1 to 2 and then 2 to 3? Regards Antoine.

1 0

Argument Clinic: what to do with builtins with non-standard signatures?
by Larry Hastings 25 Jan '14

25 Jan '14

BACKGROUND (skippable if you're a know-it-all) Argument parsing for Python functions follows some very strict rules. Unless the function implements its own parsing like so: def black_box(*args, **kwargs): there are some semantics that are always true. For example: * Any parameter that has a default value is optional, and vice-versa. * It doesn't matter whether you pass in a parameter by name or by position, it behaves the same. * You can see the default values by examining its inspect.Signature. * Calling a function and passing in the default value for a parameter is identical to calling the function without that parameter. e.g. (assuming foo is a pure function): def foo(a=value): ... foo() == foo(value) == foo(a=value) With that signature, foo() literally can't tell the difference between those three calls. And it doesn't matter what the type of value is or where you got it. Python builtins are a little less regular. They effectively do their own parsing. So they *could* do any crazy thing they want. 99.9% of the time they do one of four standard things: * They parse their arguments with a single call to PyArg_ParseTuple(). * They parse their arguments with a single call to PyArg_ParseTupleAndKeywords(). * They take a single argument of type "object" (METH_O). * They take no arguments (METH_NOARGS). PyArg_ParseTupleAndKeywords() behaves almost exactly like a Python function. PyArg_ParseTuple() is a little less like a Python function, because it doesn't support keyword arguments. (Surely this behavior is familiar to you!) But then there's that funny 0.1%, the builtins that came up with their own unique approach for parsing arguments--given them funny semantics. Argument Clinic tries to accomodate these as best it can. (That's why it supports "optional groups" for example.) But it can only do so much. THE PROBLEM Argument Clinic's original goal was to provide an introspection signature for every builtin in Python. But a small percentage of builtins have funny semantics that aren't expressable in a valid Python signature. This makes them hard to convert to Argument Clinic, and makes their signature inaccurate. If we want these functions to have an accurate Python introspection signature, their argument parsing will have to change. THE QUESTION What should someone converting functions to Argument Clinic do when faced with one of these functions? Of course, the simplest answer is "nothing"--don't convert the function to Argument Clinic. We're in beta, and any change that isn't a bugfix is impermissible. We can try again for 3.5. But if "any change" is impermissible, then we wouldn't have the community support to convert to Argument Clinic right now. The community wants proper signatures for builtins badly enough that we're doing it now, even though we're already in beta for Python 3.4. Converting to Argument Clinic is, in the vast majority of cases, a straightforward and low-risk change--but it is *a* change. Therefore perhaps the answer isn't an automatic "no". Perhaps additional straightforward, low-risk changes are permissible. The trick is, what constitutes a straightforward, low-risk change? Where should we draw the line? Let's discuss it. Perhaps a consensus will form around an answer besides a flat "no". THE SPECIFICS I'm sorting the problems we see into four rough categories. a) Functions where there's a static Python value that behaves identically to not passing in that parameter (aka "the NULL problem") Example: _sha1.sha1(). Its optional parameter has a default value in C of NULL. We can't express NULL in a Python signature. However, it just so happens that _sha1.sha1(b'') is exactly equivalent to _sha1.sha1(). b'' makes for a fine replacement default value. Same holds for list.__init__(). its optional "sequence" parameter has a default value in C of NULL. But this signature: list.__init__(sequence=()) works fine. The way Clinic works, we can actually still use the NULL as the default value in C. Clinic will let you use completely different values as the published default value in Python and the real default value in C. (Consenting adults rule and all that.) So we could lie to Python and everything works just the way we want it to. Possible Solutions: 0) Do nothing, don't convert the function. 1) Use that clever static value as the default. b) Functions where there's no static Python value that behaves identically to not passing in that parameter (aka "the dynamic default problem") There are functions with parameters whose defaults are mildly dynamic, responding to other parameters. Example: I forget its name, but someone recently showed me a builtin that took a list as its first parameter, and its optional second parameter defaulted to the length of the list. As I recall this function didn't allow negative numbers, so -1 wasn't a good fit. Possible solutions: 0) Do nothing, don't convert the function. 1) Use a magic value as None. Preferably of the same type as the function accepts, but failing that use None. If they pass in the magic value use the previous default value. Guido himself suggested this in 2) Use an Argument Clinic "optional group". This only works for functions that don't support keyword arguments. Also, I hate this, because "optional groups" are not expressable in Python syntax, so these functions automatically have invalid signatures. c) Functions that accept an 'int' when they mean 'boolean' (aka the "ints instead of bools" problem) This is specific but surprisingly common. Before Python 3.3 there was no PyArg_ParseTuple format unit that meant "boolean value". Functions generally used "i" (int). Even older functions accepted an object and called PyLong_AsLong() on it. Passing in True or False for "i" (or PyLong_AsLong()) works, because boolean inherits from long. But anything other than ints and bools throws an exception. In Python 3.3 I added the "p" format unit for boolean arguments. This calls PyObject_IsTrue() which accepts nearly any Python value. I assert that Python has a crystal clear definition of what constitutes "true" and "false". These parameters are clearly intended as booleans but they don't conform to the boolean protocol. So I suggest every instance of this is a (very mild!) bug. But changing these parameters to use "p" is a change: they'll accept many more values than before. Right now people convert these using 'int' because that's an exact match. But sometimes they are optional, and the person doing the conversion wants to use True or False as a default value, and it doesn't work: Argument Clinic's type enforcement complains and they have to work around it. (Argument Clinic has to enforce some type-safety here because the values are used as defaults for C variables.) I've been asked to allow True and False as defaults for "int" parameters specifically because of this. Example: str.splitlines(keepends) Solution: 1) Use "bool". 2) Use "int", and I'll go relax Argument Clinic so they can use bool values as defaults for int parameters. d) Functions with behavior that deliberately defy being expressed as a Python signature (aka the "untranslatable signature" problem) Example: itertools.repeat(), which behaves differently depending on whether "times" is supplied as a positional or keyword argument. (If "times" is <0, and was supplied via position, the function yields 0 times. If "times" is <0, and was supplied via keyword, the function yields infinitely-many times.) Solution: 0) Do nothing, don't convert the function. 1) Change the signature until it is Python compatible. This new signature *must* accept a superset of the arguments accepted by the existing signature. (This is being discussed right now in issue #19145.) //arry/

5 9

str.rreplace
by Ram Rachum 25 Jan '14

25 Jan '14

Question: Why is there no str.rreplace in Python?

12 27

Re: [Python-Dev] [Python-checkins] Cron <docs@dinsdale> /home/docs/build-devguide
by Benjamin Peterson 25 Jan '14

25 Jan '14

On Fri, Jan 24, 2014, at 08:45 PM, Cron Daemon wrote: > Could not find platform independent libraries <prefix> > Could not find platform dependent libraries <exec_prefix> > Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] > 'import site' failed; use -v for traceback > Traceback (most recent call last): > File "/data/hg/sphinx-env/bin/sphinx-build", line 5, in <module> > from pkg_resources import load_entry_point > ImportError: No module named pkg_resources I recreated the sphinx virtualenv on dinsdale, so hopefully this will be fixed now.

2 1