[Python-Dev] Rationale behind removing kwdict from keyword.py?

[Tim Peters]

[Fernando Perez]
> I recently got a report from a user of a project of mine breaking under 2.4a1.
> The reason is that keyword stopped having a kwdict variable in it.  Is there
> any rationale behind this change?

Looking at the code, because keyword.py no longer uses a dictionary
(Raymond changed it to use a set), and kwdict wasn't part of keyword's
public API so there was no reason to avoid changing this.

>  I know I can always work around it, but this honestly feels, to me, like
> gratuitous breakage.
>
> It would be nice if things like public members (it wasn't called _kwdict, so I
> had no reason to assume it was off-limits) ...

You actually had two reasons, and strong ones.  As you noted later, it
wasn't part of keyword's documented interface, and you *always* act at
your own risk when you go beyond the docs.

The second reason is that 'kwdict' didn't appear in the
keyword.__all__ list.  As the docs for Python's import statement say:

    The public names defined by a module are determined by checking the
    module's namespace for a variable named __all__; if defined, it must be a
    sequence of strings which are names defined or imported by that module.
    The names given in __all__ are all considered public and are
required to exist.
    If __all__ is not defined, the set of public names includes all
names found in
    the module's namespace which do not begin with an underscore
character ("_").
    __all__ should contain the entire public API. It is intended to
avoid accidentally
    exporting items that are not part of the API (such as library modules which
    were imported and used within the module).

Note that it's very common in older modules (which keyword.py is) to
define the newer __all__ rather than risk breaking code by massive
renaming of variables to stick an underscore in front of them.

> ...
> While the html docs for keyword don't mention kwdict, I often simply use
> help(module) to look for what I need, and I saw kwdict happily
> listed there, without any leading _ to indicate any sense of it being private.

You should stop doing that, then -- this is the kind of trouble it
leads to, and you'll only get more of it.  help(keyword) did advertise
the keyword.__all__ value in 2.3.4:

...
DATA
    __all__ = ['iskeyword', 'kwlist']
    keyword = 'yield'
    kwdict = {'and': 1, 'assert': 1, 'break': 1, 'class': 1, 'continue': 1...
    kwlist = ['and', 'assert', 'break', 'class', 'continue', 'def', 'del',...

Since 'keyword' and 'kwdict' weren't in __all__, you couldn't rely on
either of them; and, indeed, they're both gone in 2.4, while
keyword.__all__ hasn't changed.

> The changes were made, according to viewCVS, in revision 1.13.  I know I can
> use iskeyword() for my purposes, and patch accordingly.  But I'd like to
> remind the core developers that when things go into the stdlib, people start
> using them.  Removing stuff will _always_ break code.

The developers were careful to keep the public API in this case
exactly the same.

> I'll gladly file a SF bug report against this, but I'd like not to waste
> anyone's time if it won't be considered.

Sorry, no, I would not consider this one.