Basic Markdown Readme Support
Markdown READMEs are becoming increasingly ubiquitous for many projects. GitHub, GitLab, Bitbucket, among others, happily detect .md readme files and render them in their web interfaces. rST is nice, but is generally overkill for single-page documents (as opposed to more intricate documentation). To get something done sooner, rather than later, I'd prefer to come up with a two-phase solution, one narrow and "opt-in" (status-quo for all existing packages unless the maintainer does something) for quick implementation with hopefully minimal pushback. The other, later, not-proposed-here, could be more feature-rich/heuristic. So, to get Markdown supported in some form, here's some talking points to debate: * Add a "long_description_filename" to setup (suggested by @msabramo/GH [1]), which does the usual boilerplate "[codecs.]open(x, 'r', encoding=y).read()". To determine the format look at an additional "long_description_content_type" field (if provided), otherwise look at the file extension and assume/require UTF-8. * As an alternative, if there is no long_description, and the fall-back to README.rst fails, look for README.md and grab that. Such a strategy wouldn't be fully opt-in, however. * Markdown (just like reStructuredText) allows arbitrary HTML to be added. The renderer must then be upstream of the (existing) clean (with bleach) step. * [Optional]: Use common extensions provided by the PyPI/Markdown library to support GFM/SO stuff: fenced_code, smart_strong, nl2br Nick Timkovich Amaral Lab, Northwestern University [1]: https://github.com/pypa/readme_renderer/pull/3#issuecomment-72302732 [2]: https://github.com/pypa/readme_renderer/pull/3#issuecomment-66569248
I am -1 on this on the basis that the services mentioned also happily support restructured text READMEs On 5/2/2016 12:40, Nick Timkovich wrote:
Markdown READMEs are becoming increasingly ubiquitous for many projects. GitHub, GitLab, Bitbucket, among others, happily detect .md readme files and render them in their web interfaces. rST is nice, but is generally overkill for single-page documents (as opposed to more intricate documentation). To get something done sooner, rather than later, I'd prefer to come up with a two-phase solution, one narrow and "opt-in" (status-quo for all existing packages unless the maintainer does something) for quick implementation with hopefully minimal pushback. The other, later, not-proposed-here, could be more feature-rich/heuristic.
So, to get Markdown supported in some form, here's some talking points to debate:
* Add a "long_description_filename" to setup (suggested by @msabramo/GH [1]), which does the usual boilerplate "[codecs.]open(x, 'r', encoding=y).read()". To determine the format look at an additional "long_description_content_type" field (if provided), otherwise look at the file extension and assume/require UTF-8.
* As an alternative, if there is no long_description, and the fall-back to README.rst fails, look for README.md and grab that. Such a strategy wouldn't be fully opt-in, however.
* Markdown (just like reStructuredText) allows arbitrary HTML to be added. The renderer must then be upstream of the (existing) clean (with bleach) step.
* [Optional]: Use common extensions provided by the PyPI/Markdown library to support GFM/SO stuff: fenced_code, smart_strong, nl2br
Nick Timkovich Amaral Lab, Northwestern University
[1]: https://github.com/pypa/readme_renderer/pull/3#issuecomment-72302732 [2]: https://github.com/pypa/readme_renderer/pull/3#issuecomment-66569248
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
The justification was "Because Github et. al. support markdown, pypi should too", presumably for the purpose of allowing one to write their README once, and have it work in both places. This is already possible, and only adds unneeded complexity to an already complex system. If you want to make your README a write-once document, use the format already supported on both platforms. On 5/3/2016 00:27, Robert Collins wrote:
On 3 May 2016 4:19 PM, "Alexander Walters" <tritium-list@sdamon.com <mailto:tritium-list@sdamon.com>> wrote:
I am -1 on this on the basis that the services mentioned also
happily support restructured text READMEs
I don't understand why that makes you say no to the ability to support markdown.
Rob
On 3 May 2016 at 14:33, Alexander Walters <tritium-list@sdamon.com> wrote:
The justification was "Because Github et. al. support markdown, pypi should too", presumably for the purpose of allowing one to write their README once, and have it work in both places. This is already possible, and only adds unneeded complexity to an already complex system. If you want to make your README a write-once document, use the format already supported on both platforms.
As I understand it, it's more a matter of folks finding the context switch between Markdown and non-Sphinx reStructuredText a pain (with the main differences being double-backticks for inline code and `link text <link target>`_ instead of [link text](link target) for hyperlinks) and not being aware of (or caring about) the dire lack of commercial investment in tooling support for upstream Python software distribution (since all the redistributors have their own software distribution platforms that they recommend their users use instead). Encountering volunteer maintained community services can be quite a shock to the system for folks otherwise used to software tooling developed by venture capital backed companies making a landgrab for developer mindshare in the hopes of creating the next Oracle or Microsoft :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-----Original Message----- From: Nick Coghlan [mailto:ncoghlan@gmail.com] As I understand it, it's more a matter of folks finding the context switch between Markdown and non-Sphinx reStructuredText a pain (with the main differences being double-backticks for inline code and `link text <link target>`_ instead of [link text](link target) for hyperlinks) and not being aware of (or caring about) the dire lack of commercial investment in tooling support for upstream Python software distribution (since all the redistributors have their own software distribution platforms that they recommend their users use instead).
As someone who writes quite a bit of non-sphinx Restructured Text, and a lot of Markdown (because I have to), I have zero sympathy for the argument of context switching. I can come up with an overblown analogy, but I will just reduce it to "it's not complicated to keep the two syntaxes straight".
My perspective, for what it's worth, is that while I find Markdown a horrible pain, there are a lot of people who pick it up before picking up Python, and tools like GitHub and BitBucket encourage (and make it easier to add) README.md to a project. For someone who isn't familiar with reStructuredText, it's an easier on-ramp. So while I'm all for encouraging developers to prefer reStructuredText, I'm in favor of supporting Markdown as a long_description format. The format for a README file just doesn't seem such a big deal that alienating potential community members is worth it. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Tue, May 3, 2016 at 9:18 AM, Fred Drake <fred@fdrake.net> wrote:
My perspective, for what it's worth, is that while I find Markdown a horrible pain,
But wait, it's worse. Unlike ReStructuredText, there's no Markdown standard. In my last job, I had to use a suite of tools (from a single company that I won't name but is easy to guess :) ) for which no 2 tools used the same dialect of Markdown. :( ...
So while I'm all for encouraging developers to prefer reStructuredText, I'm in favor of supporting Markdown as a long_description format. The format for a README file just doesn't seem such a big deal that alienating potential community members is worth it.
Which begs the question, which dialect of Markdown are you suggesting we support. :) Jim -- Jim Fulton http://jimfulton.info
On Tue, May 3, 2016 at 9:29 AM, Jim Fulton <jim@jimfulton.info> wrote:
But wait, it's worse. Unlike ReStructuredText, there's no Markdown standard.
We agree that this is a problem, and it's why I don't use Markdown when tools don't force it.
In my last job, I had to use a suite of tools (from a single company that I won't name but is easy to guess :) ) for which no 2 tools used the same dialect of Markdown. :(
Truly a joy. I'm glad things are better now. :-)
Which begs the question, which dialect of Markdown are you suggesting we support. :)
If we go with Nick's idea of a client-side solution, we really don't have to pick just one flavor, though I'd expect GFM would win out for practical purposes. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Tue, 3 May 2016, Jim Fulton wrote:
In my last job, I had to use a suite of tools (from a single company that I won't name but is easy to guess :) ) for which no 2 tools used the same dialect of Markdown. :(
Which begs the question, which dialect of Markdown are you suggesting we support. :)
My personal preference is CommonMark - mainly because it's actually well-defined. There are only a couple of cases that I've encountered where CommonMark didn't render exactly what I expected. But at least it's consistent, so it doesn't take much to adjust to. For a simple README, I actually *prefer* CommonMark/Markdown - I find it has all the features I need. Of course for more complicated documentation, reStructuredText has a lot more power, so I'm down with the extra complexity. Plus, I have actually come across more than one project on pypi right now where the readme is in markdown format, so looks fine on Github, but pretty funky on pypi. -Wayne
On 3 May 2016 at 14:18, Fred Drake <fred@fdrake.net> wrote:
My perspective, for what it's worth, is that while I find Markdown a horrible pain, there are a lot of people who pick it up before picking up Python, and tools like GitHub and BitBucket encourage (and make it easier to add) README.md to a project. For someone who isn't familiar with reStructuredText, it's an easier on-ramp.
So while I'm all for encouraging developers to prefer reStructuredText, I'm in favor of supporting Markdown as a long_description format. The format for a README file just doesn't seem such a big deal that alienating potential community members is worth it.
Agreed. And like it or not, people get familiar with Markdown from things like github comment boxes, so it's very easy to just "go with the familiar option". Having said that, someone still needs to do the work to add Markdown support - and that's not trivial, as it will involve changes to how the metadata is held to include a content type for the long_description. And that means a discussion on how to include that. So while I don't mind if people want this support, it's not going to happen unless someone actually goes through the work of managing the proposal, co-ordinating the discussions, writing the code, etc. And I suspect that this is unlikely to be a priority of any of the PyPA team in the near future - so it'll need someone who's interested in seeing this feature added to take on that work. Paul
On 3 May 2016 at 23:18, Fred Drake <fred@fdrake.net> wrote:
My perspective, for what it's worth, is that while I find Markdown a horrible pain, there are a lot of people who pick it up before picking up Python, and tools like GitHub and BitBucket encourage (and make it easier to add) README.md to a project. For someone who isn't familiar with reStructuredText, it's an easier on-ramp.
So while I'm all for encouraging developers to prefer reStructuredText, I'm in favor of supporting Markdown as a long_description format. The format for a README file just doesn't seem such a big deal that alienating potential community members is worth it.
Exactly. The lack of support for Markdown README files is mainly a matter of a historical quirk of the way the PyPI metadata upload API works making this way more work to implement than it seems at first glance, the current PyPI codebase being sufficiently fragile that we actively avoid changing it, and getting Warehouse to the point of being sufficiently feature complete for it to take over primary service responsibilities being a long hard slog for the folks working on it (it's a lot harder to find volunteers interested in working on paying down technical debt than it is to find folks that want to work on new user facing features). However, this SO answer provides some ideas on ways to convert from Markdown to reStructured Text when producing the sdist metadata, or to derive a checked in .rst file from a README.md file: http://stackoverflow.com/questions/10718767/have-the-same-readme-both-in-mar... It is also seems plausible to me that a client-side solution could be designed that allowed the description metadata stored in the sdist to be overridden when uploading to PyPI (i.e. the description in PKG-INFO could be Markdown, but the upload tool could use pypandoc to convert that to reStructuredText in the uploaded metadata). I'm not sure if Donald would be open to that in twine (presumably via an extra to avoid having pypandoc as a standard dependency), but client-only changes are generally an easier pitch than changes to the interfaces between client tools and PyPI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On May 3, 2016, at 9:47 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 3 May 2016 at 23:18, Fred Drake <fred@fdrake.net> wrote:
My perspective, for what it's worth, is that while I find Markdown a horrible pain, there are a lot of people who pick it up before picking up Python, and tools like GitHub and BitBucket encourage (and make it easier to add) README.md to a project. For someone who isn't familiar with reStructuredText, it's an easier on-ramp.
So while I'm all for encouraging developers to prefer reStructuredText, I'm in favor of supporting Markdown as a long_description format. The format for a README file just doesn't seem such a big deal that alienating potential community members is worth it.
Exactly. The lack of support for Markdown README files is mainly a matter of a historical quirk of the way the PyPI metadata upload API works making this way more work to implement than it seems at first glance, the current PyPI codebase being sufficiently fragile that we actively avoid changing it, and getting Warehouse to the point of being sufficiently feature complete for it to take over primary service responsibilities being a long hard slog for the folks working on it (it's a lot harder to find volunteers interested in working on paying down technical debt than it is to find folks that want to work on new user facing features).
This is basically the answer here. It looks like the original post by Nick Timkovich tried to start a discussion about what this might look like, but really I think it focused too much on the setup.py API which isn't really the issue, we can do whatever there, what's more the issue is how is that represented in the metadata. Right now we have a singular metadata field which just contains all of the text of the long_description without any other information about it. The simplest thing to do is probably to just add a new field, something like:: Description-Markup: <value> We'd need to define some values for for it like "txt", "md", "rst" or something along those lines. I'd suggest extensions so that in the future we can move the long description into it's own file in the metadata and just move that value to the file extension (like `DESCRIPTION.[ext]`). We'd also want to declare what behavior should be expected when that value doesn't exist (codifying the current behavior of, attempt to register as rst, fall back to txt). We'd probably also want some recommendations on what the different types of tools should do when encountering invalid markup for the declared markup language and also what they should do when encountering a markup language they don't know. For the server side (e.g. PyPI) I'd suggest erroring the upload whenever an invalid or unknown markup is attempted to be uploaded (where a undeclined markup is never invalid, it just does the fallback) and for anything clientside it jsut falls back to plaintext. This has a few benefits: * No more ugly when plaintext (or markdown!) accidently get rendered wrongly as restructeredtext. * We can hard fail uploads when their rendering is broken, leading people to be able to fix the problems instead of ending up with a bunch of broken markup all over PyPI. * We can allow markup languages other than txt and rst. However, I'm not going to have time or motivation to really work this into a valid spec or even a fully fleshed out idea. Nor will I be able to handle implementing this in PyPI or Warehouse right now since I'm primarily focused on trying to get Warehouse itself deployed for real. I am happy to review PRs and actual specs though, whether they take this idea or they use a different one.
However, this SO answer provides some ideas on ways to convert from Markdown to reStructured Text when producing the sdist metadata, or to derive a checked in .rst file from a README.md file: http://stackoverflow.com/questions/10718767/have-the-same-readme-both-in-mar...
It is also seems plausible to me that a client-side solution could be designed that allowed the description metadata stored in the sdist to be overridden when uploading to PyPI (i.e. the description in PKG-INFO could be Markdown, but the upload tool could use pypandoc to convert that to reStructuredText in the uploaded metadata). I'm not sure if Donald would be open to that in twine (presumably via an extra to avoid having pypandoc as a standard dependency), but client-only changes are generally an easier pitch than changes to the interfaces between client tools and PyPI.
I actually mostly don’t do much with twine anymore, Ian Cordasco has more or less taken over maintenance of it so it'd be up to him ultimately. That being said I think doing it in twine is the wrong layer. Ideally the metadata in PyPI matches the metadata in the file and people can "recreate" the PyPI database using nothing but the files [1]. I think if you want to shim over this on the client side, your best hope would be in setuptools, but I think if someone is motivated enough to actually do the spec and implementation work we can get proper support landed too. [1] Ok, this doesn't exactly work because of the dynamic nature of setup.py, but in practice you can get close, and we're moving closer to that day. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (10)
-
Alexander Walters -
Donald Stufft -
Fred Drake -
Jim Fulton -
Nick Coghlan -
Nick Timkovich -
Paul Moore -
Robert Collins -
tritium-list＠sdamon.com
-
Wayne Werner