[Distutils] on integrated docs in Warehouse and PyPI

[Ralf Gommers]

On Sun, Jun 5, 2016 at 6:33 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 4 Jun 2016 6:54 am, "Donald Stufft" <donald at stufft.io> wrote:
> >
> >
> >> On Jun 4, 2016, at 9:33 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> I think everyone would agree that having some nice doc hosting service
> available as an option would be, well, nice. Everyone likes options. But
> the current doc hosting is unpopular and feature poor, falls outside of the
> PyPI core mission, and is redundant with other more popular services, at a
> time when the PyPI developers are struggling to maintain core services.
> >
> >
> >
> > To add to what Nathaniel said here, there are a few problems with the
> current situation:
> >
> > Documentation hosting largely worked “OK” when it was just writing files
> out to disk, however we’ve since removed all use of the local disk (so that
> we can scale past 1 machine) and we’re now storing things in S3. This makes
> documentation hosting particularly expensive in terms of API calls because
> we need to do expensive list key operations to discover which files exist
> (versus package files where we have a database full of files).
>
> Amazon do offer higher level alternatives like https://aws.amazon.com/efs/
> for use cases like PyPI's docs hosting that assume they have access to a
> normal filesystem.
>
> Given the credential management benefits of integrated docs,
>
>From the RTD blog post linked by Nathaniel:
""
Our proposed grant, for $48,000, is to build a separate instance that
integrates with the Python Package Index’s upcoming website, Warehouse
<https://warehouse.python.org/>. This integration will provide automatic
API reference documentation upon package release, with authentication tied
to PyPI and simple configuration inside the distribution.
""

> it does seem worthwhile to me for the PSF to invest in a lowest common
> denominator static file hosting capability,
>
Seems like a very poor way to spend money and developer time imho. The
original post by Jason brings up a few shortcomings of RTD, but I'm amazed
that that leads multiple people here to conclude that starting a new doc
hosting effort is the right answer to that. The much better alternative is:
read the RTD contributing guide [1] and their plans for PyPI integration
[2], then start helping out with adding those features to RTD.

There is very little chance that a new effort as discussed here can come
close to RTD, which is a quite active project with by now over 200
contributors. Starting a new project should be done for the right reasons:
existing projects don't have and don't want to implement features you need,
you have a better technical design, you want to reimplement to learn from
it, etc. There are no such reasons here as far as I can tell.

If there's money left for packaging related work, I'm sure we can think of
better ways to spend it. First thoughts:
- Accelerate PyPI integration plans for RTD
- Accelerate work on Warehouse
- Pay someone to review and merge distutils patches in the Python bug
tracker


Final thought: there's nothing wrong with distributed infrastructure for
projects. A typical project today may have code hosting on GitHub or
Bitbucket, use multiple CI providers in parallel, use a separate code
coverage service, upload releases to PyPI, conda-forge and GitHub Releases,
and host docs on RTD. Integrating doc hosting with PyPI doesn't change that
picture really.

Ralf

[1] https://github.com/rtfd/readthedocs.org/blob/master/docs/contribute.rst
[2] https://github.com/rtfd/readthedocs.org/issues/1957
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160605/d60f2d3e/attachment.html>