[Distutils] Adding the "Description-Content-Type" metadata field

[Nick Coghlan]

Hi folks,

Marc Abramowitz has prepared a PR for the Core Metadata section of the
specifications page [1] that adds a new "Description-Content-Type"
field: https://github.com/pypa/python-packaging-user-guide/pull/258

The draft text has now reached the point where I'm prepared to accept
it, so this thread offers folks one last chance to provide feedback
before we make it official.

Full text of the new subsection
=========================================

Description-Content-Type
~~~~~~~~~~~~~~~~~~~~~~~~

A string containing the format of the distribution's description, so that
tools can intelligently render the description.

Historically, PyPI supported descriptions in plain text and `reStructuredText
(reST) <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html>`_,
and could render reST into HTML. However, it is common for distribution
authors to write the description in `Markdown
<https://daringfireball.net/projects/markdown/>`_ (`RFC 7763
<https://tools.ietf.org/html/rfc7763>`_) as many code hosting sites render
Markdown READMEs, and authors would reuse the file for the description. PyPI
didn't recognize the format and so could not render the description correctly.
This resulted in many packages on PyPI with poorly-rendered descriptions when
Markdown is left as plain text, or worse, was attempted to be rendered as reST.
This field allows the distribution author to specify the format of their
description, opening up the possibility for PyPI and other tools to be able to
render Markdown and other formats.

The format of this field is the same as the ``Content-Type`` header in HTTP
(e.g.:
`RFC 1341 <https://www.w3.org/Protocols/rfc1341/4_Content-Type.html>`_).
Briefly, this means that it has a ``type/subtype`` part and then it can
optionally have a number of parameters:

Format::

    Description-Content-Type: <type>/<subtype>; charset=<charset>[;
<param_name>=<param value> ...]

The ``type/subtype`` part has only a few legal values:

- ``text/plain``
- ``text/x-rst``
- ``text/markdown``

The ``charset`` parameter can be used to specify whether the character set in
use is UTF-8, ASCII, etc. If ``charset`` is not provided, then it is
recommended that the implementation (e.g.: PyPI) treat the content as
UTF-8.

Other parameters might be specific to the chosen subtype. For example, for the
``markdown`` subtype, there is a ``variant`` parameter that allows specifying
the variant of Markdown in use, such as:

- ``CommonMark`` for `CommonMark`
  <https://tools.ietf.org/html/rfc7764#section-3.5>`_

- ``GFM`` for `GitHub Flavored Markdown (GFM)
  <https://tools.ietf.org/html/rfc7764#section-3.2>`_

- ``Original`` for `Gruber's original Markdown syntax
  <https://tools.ietf.org/html/rfc7763#section-6.1.4>`_

Example::

    Description-Content-Type: text/plain; charset=UTF-8

Example::

    Description-Content-Type: text/x-rst; charset=UTF-8

Example::

    Description-Content-Type: text/markdown; charset=UTF-8; variant=CommonMark

Example::

    Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

Example::

    Description-Content-Type: text/markdown; charset=UTF-8; variant=Original

If a ``Description-Content-Type`` is not specified or it's set to an
unrecognized value, then the assumed content type is ``text/x-rst;
charset=UTF-8``.

If the ``charset`` is not specified or it's set to an unrecognized value, then
the assumed ``charset`` is ``UTF-8``.

If the subtype is ``markdown`` and ``variant`` is not specified or it's set to
an unrecognized value, then the assumed ``variant`` is ``CommonMark``.
=========================================

[1] https://packaging.python.org/specifications/#core-metadata

Regards,
Nick.

P.S. I know I still need to update
https://www.pypa.io/en/latest/specifications/ to reflect the ability
to make small backwards compatible adjustments to the specifications
without a PEP, so I'll get that sorted today, since I've been talking
about it for approximately forever.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia