[Distutils] PyPI and Uploading Documentation

[Erik Bray]

On Fri, May 15, 2015 at 9:48 AM, Donald Stufft <donald at stufft.io> wrote:
> Hey!
>
> First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so
> that instead of storing files and documentation locally (really in a glusterfs
> cluster) it will store them inside of S3. This will reduce maintenance overhead
> of running PyPI by two servers since we'll no longer need to run our own
> glusterfs cluster as well as improve the reliaiblity and scalability of the
> PyPI service as a whole since we've had nothing but problems from glusterfs in
> this regard.
>
> One of the things that this brought to light was that the documentation
> upload ability in PyPI is something that is not often used* however it
> represents something which is one of our slowest routes. It's not a well
> supported feature and I feel that it's going outside of the core competancy for
> PyPI itself and instead PyPI should be focused on the files themselves. In
> addition since the time this was added to PyPI a number of free services or
> cheap services have came about that allow people to sanely upload raw document
> without a reliance on any particular documentation system and we've also had
> the rise of ReadTheDocs for when someone is using Sphinx as their documentation
> system.
>
> I think that it's time to retire this aspect of PyPI which has never been well
> supported and instead focus on just the things that are core to PyPI. I don't
> have a fully concrete proposal for doing this, but I wanted to reach out here
> and figure out if anyone had any ideas. The rough idea I have currently is to
> simply disable new documentation uploads and add two new small features. One
> will allow users to delete their existing documentation from PyPI and the other
> would allow them to register a redirect which would take them from the current
> location to wherever they move their documentation too. In order to prevent
> breaking documentation for projects which are defunct or not actively
> maintained we would maintain the archived documentation (sans what anyone has
> deleted) indefinetely.
>
> Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think
> that ReadTheDocs is a great service with heavy ties to the Python community.
> They will do a better job at hosting documentation than PyPI ever could since
> that is their core goal. In addition there is a dialog between ReadTheDocs and
> PyPI where there is an opportunity to add integration between the two sites as
> well as features to ReadTheDocs that it currently lacks that people feel are a
> requirement before we move PyPI's documentation to read-only.
>
> Thoughts?
>
> * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not
>   easy to tell if all of them are still using it as their primary source of
>   documentation though or if it's old documentation that they just can't
>   delete.

+1 for all the stated reasons.

I have a few docs hosted on pythonhosted.org, but it's become a
nuisance to maintain since it does not support multiple doc versions
like ReadTheDocs, so now I've wound up with documentation for the same
projects on both sites.  The nuisance comes not so much in the process
(like Barry wrote, I've enjoyed the simplicity of `setup.py
upload_docs`), but because more often than not I've had to redirect
users to the Readthedocs docs to make sure they're using the correct
version of the docs.  So I wish I were not locked into updating the
pythonhosted.org docs and would be happy to retire them altogether
(much as I appreciated the service).

One question is how this would be handled at the tooling end.
setup.py upload_docs would have to be retired somehow.  Though it
might also be nice if some simple tools were added to make it just as
easy to add docs to ReadTheDocs.  I know something like upload_docs
doesn't really make sense, since RTD handles the checkout and build of
the docs.  But there's still a manual step of enabling new versions of
the docs that it would be nice to make as effortless as `setup.py
upload_docs`.  I gues that's off-topic for the PyPI end of things
though.

Erik