[Python-Dev] Breaking undocumented API

[Brett Cannon]

On Mon, Nov 8, 2010 at 09:20, Alexander Belopolsky
<belopolsky at users.sourceforge.net> wrote:
> Was: [issue2001] Pydoc interactive browsing enhancement
>
> On Sun, Nov 7, 2010 at 9:17 AM, Nick Coghlan <report at bugs.python.org> wrote:
> ..
>>
>> I'd actually started typing out the command to commit this before it finally clicked that the patch changes public
>> APIs of the pydoc module in incompatible ways. Sure, they aren't documented, but the fact they aren't protected
>> by an underscore means I'm not comfortable with the idea of removing them or radically change their functionality
>> without going through a deprecation period first.
>>
>
> I have a similar issue with the trace module and would appreciate some
> guidance on this as well.  The trace module documented API includes
> just the Trace class, but the module defines several helper functions
> and classes  that do not start with a leading underscore and are not
> excluded from * imports by __all__.  (There is no trace.__all__.)

I think we need to, as a group, decide how to handle undocumented APIs
that don't have a leading underscore: they get treated just the same
as the documented APIs, or are they private regardless and thus we can
change them at our whim?

>
> I don't think a strict don't remove without deprecation policy is
> workable.  For example, is trace.rx_blank constant part of the trace
> module API that needs to be preserved indefinitely?  I don't even know
> if it is possible to add a deprecation warning to it, but
> CoverageResults._blank_re would certainly be a better place for it.

The deprecation policy obviously cannot apply to module-level attributes.

>
> The functions I have specific need to modify (See
> http://bugs.python.org/issue10342) are trace.find_strings(), and
> find_executable_linenos().  The functions take module's file name, but
> I need to make them to take the module object in order to be able to
> deal with modules that have custom loaders.
>
> The trace.find_strings() function is clearly internal.  It's name does
> not even reflect what it does (finding docstring locations), so it was
> never intended for use outside of the trace module.  However, google
> code search reveals that people do use it and other functions in their
> code.
>
> This suggests that trace.find_strings() should probably be preserved
> or properly deprecated.  If this is the case, should we fix bugs in
> it?  Note that it currently has a bug because it ignores the coding
> cookie when opening python source file.  Should this be fixed?
>
> I freely admit that I have more questions than answers, so I would
> like to hear from a wider audience.

The main reason I have said that non-underscore names should be
properly deprecated (assuming they are not contained in an
underscored-named module) is that dir() and help() do not distinguish.
If you are perusing a module from the interpreter prompt you have no
way to know whether something is public or private if it lacks an
underscore. Is it reasonable to assume that any API found through
dir() or help() must be checked with the official docs before you can
consider using it, even if you have no explicit need to read the
official docs?

I (unfortunately) say no, which is why I have argued that
non-underscored names need to be properly deprecated. This obviously
places a nasty burden on us, though, so I don't like taking this
position. Unless we can make it clearly known through help() or
something that the official docs must be checked to know what can and
cannot be reliably used I don't think it is reasonable to force users
to not be able to rely on help() (we should probably change help() to
print a big disclaimer for anything with a leading underscore,
though).

But that doesn't mean we can't go through, fix up our names, and
deprecate the old public names; that's fair game in my book.