[Distutils] PyPI and Uploading Documentation

[Donald Stufft]

> On May 15, 2015, at 2:23 PM, Erik Bray <erik.m.bray at gmail.com> wrote:
> 
> On Fri, May 15, 2015 at 9:48 AM, Donald Stufft <donald at stufft.io> wrote:
>> Hey!
>> 
>> First, for anyone who isn't aware we recently migrated PyPI and TestPyPI so
>> that instead of storing files and documentation locally (really in a glusterfs
>> cluster) it will store them inside of S3. This will reduce maintenance overhead
>> of running PyPI by two servers since we'll no longer need to run our own
>> glusterfs cluster as well as improve the reliaiblity and scalability of the
>> PyPI service as a whole since we've had nothing but problems from glusterfs in
>> this regard.
>> 
>> One of the things that this brought to light was that the documentation
>> upload ability in PyPI is something that is not often used* however it
>> represents something which is one of our slowest routes. It's not a well
>> supported feature and I feel that it's going outside of the core competancy for
>> PyPI itself and instead PyPI should be focused on the files themselves. In
>> addition since the time this was added to PyPI a number of free services or
>> cheap services have came about that allow people to sanely upload raw document
>> without a reliance on any particular documentation system and we've also had
>> the rise of ReadTheDocs for when someone is using Sphinx as their documentation
>> system.
>> 
>> I think that it's time to retire this aspect of PyPI which has never been well
>> supported and instead focus on just the things that are core to PyPI. I don't
>> have a fully concrete proposal for doing this, but I wanted to reach out here
>> and figure out if anyone had any ideas. The rough idea I have currently is to
>> simply disable new documentation uploads and add two new small features. One
>> will allow users to delete their existing documentation from PyPI and the other
>> would allow them to register a redirect which would take them from the current
>> location to wherever they move their documentation too. In order to prevent
>> breaking documentation for projects which are defunct or not actively
>> maintained we would maintain the archived documentation (sans what anyone has
>> deleted) indefinetely.
>> 
>> Ideally I hope people start to use ReadTheDocs instead of PyPI itself. I think
>> that ReadTheDocs is a great service with heavy ties to the Python community.
>> They will do a better job at hosting documentation than PyPI ever could since
>> that is their core goal. In addition there is a dialog between ReadTheDocs and
>> PyPI where there is an opportunity to add integration between the two sites as
>> well as features to ReadTheDocs that it currently lacks that people feel are a
>> requirement before we move PyPI's documentation to read-only.
>> 
>> Thoughts?
>> 
>> * Out of ~60k projects only ~2.8k have ever uploaded documentation. It's not
>>  easy to tell if all of them are still using it as their primary source of
>>  documentation though or if it's old documentation that they just can't
>>  delete.
> 
> +1 for all the stated reasons.
> 
> I have a few docs hosted on pythonhosted.org, but it's become a
> nuisance to maintain since it does not support multiple doc versions
> like ReadTheDocs, so now I've wound up with documentation for the same
> projects on both sites.  The nuisance comes not so much in the process
> (like Barry wrote, I've enjoyed the simplicity of `setup.py
> upload_docs`), but because more often than not I've had to redirect
> users to the Readthedocs docs to make sure they're using the correct
> version of the docs.  So I wish I were not locked into updating the
> pythonhosted.org docs and would be happy to retire them altogether
> (much as I appreciated the service).
> 
> One question is how this would be handled at the tooling end.
> setup.py upload_docs would have to be retired somehow.  Though it
> might also be nice if some simple tools were added to make it just as
> easy to add docs to ReadTheDocs.  I know something like upload_docs
> doesn't really make sense, since RTD handles the checkout and build of
> the docs.  But there's still a manual step of enabling new versions of
> the docs that it would be nice to make as effortless as `setup.py
> upload_docs`.  I gues that's off-topic for the PyPI end of things
> though.
> 
> Erik


So I can’t speak for ReadTheDocs, but I believe that they are considering
and/or are planning on offering arbitrary HTML uploads similarly to how
you can upload documentation to PyPI. I don’t know if this will actually
happen and what it would look like but I know they are thinking about it.

As far as retiring upload_docs, the sanest thing to do I think would be
to just have PyPI return an error code that upload_docs would display
to the end user. The command itself is in use by a few other systems I think
so we might not want to remove it wholesale from Python itself (or maybe
we do? It’s a hard question since it’s tied to an external service unlike
most of the stdlib).

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150515/4ecf3111/attachment.sig>