On Sat, May 19, 2007 at 10:48:29AM -0700, Josiah Carlson wrote:

I'm generally a curmudgeon when it comes to 'the docs could be done better'. But this? I like it. A lot. Especially if you can get these other features in:

...

- a "quick-dispatch" function: e.g., docs.python.org/q?os.path.split would redirect you to the matching location.

Seconded! -- even if it's just for modules, this would be great. I can't count the times I've wished I could type e.g., 'docs.python.org/httplib' the way I can type 'php.net/array_search' to try to find out whether the needle comes before or after the haystack. Dustin

On Sat, May 19, 2007 at 03:31:59PM -0500, Ron Adam wrote:

- The html syntax highlighters. (Pydoc can use those)

I have a patch on the docutils patch tracker that does this. Code is probably of a rather bad quality, but it outputs LaTeX and HTML. If we can work together to improve this patch and get it in docutils it will avoid having different syntaxes and behavior depending on the front-end to docutils being used (I am thinking of rest2web, trac, and I am probably forgetting some others). The patch has been sitting there for almost 6 months without review, but I have that if people other than me work on it and ask for review it will both improve, and get reviewed, and eventually get in ! Sorry for the shameless plug, but I really do think we need a unifying approach to this. Gaël

Martin Blais wrote:

e.g. are you still marking classes as classes and functions as functions in the ReST source

It seems so (modulo XXX's and TODO's in Georg's implementation, probably ^_^) -- all of the pages have "show source" links, so you can see for yourself. I'm not an expert with the documentation system, but the markup on <http://pydoc.gbrandl.de/modules/codecs.txt> looks pretty complete to me.

(Somewhat related, but another idea from back then, which was never implemented IMO, was to find a way to automatically pull and convert the docstrings from the source code into the documentation, thus unifying all the information in one place.)

While it's probably not possible to simply generate the documentation from the docstrings, it would certainly seem interesting to get have some means (like a directive) to pull docstrings into the documentation. I think however that while migrating the docs do reStructuredText is comparatively straightforward [1]_, pulling documentation from the docstrings will require quite a bit of design and discussion work. So I'd suggest we postpone this idea until we have a working documentation system in reStructuredText, so we don't clutter the discussion. .. [1] I'm sure there will still be quite a few issues to sort out that I'm simply not seeing right now. Best wishes, Lea

Vinay Sajip schrieb:

Georg Brandl <g.brandl <at> gmx.net> writes:

...

For the impatient: the result can be seen at <http://pydoc.gbrandl.de>.

- the toolchain is pure Python, therefore can easily be shipped

Very nice! As well as looking very attractive and professional, [...]

BTW, I have to give lots of credit for the looks to Armin Ronacher. I'm not so much of a designer ;) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

On 5/20/07, Georg Brandl <g.brandl@gmx.net> wrote:

...

Very nice! As well as looking very attractive and professional, the all-Python toolset should make it easier to build the documentation - I've not been able to get a trouble-free setup of the docs toolchain on Windows.

Yep. As it is now, you need three packages from the Cheese Shop: Docutils, Pygments (the highlighter) and Jinja (the templating engine). This shouldn't be problematic, though they could also be stripped down and included.

This is great. IMHO if this is to compete to become the official Python docs, I would argue for even less dependencies, even at the cost of more generic/bland output, for portability reasons and to stimulate greater adoption. If we can make some of those dependencies optional and only rely on docutils, that could make it ubiquitous. Another thing to keep in mind: I don't know if the directives you defined are very generic, but if they are, it would be interesting to consider migrating them up into docutils (if it makes sense), and see if they could support documenting other programming languages. Could this be a language-independent documenting toolkit? Could we document LISP or Ruby code with it? Georg, thanks again!

Scott Dial schrieb:

Neal Becker wrote:

...

Sounds very interesting. I just have one concern/question. I hope that while moving away from latex, we are not precluding the ability to write math as part of the documentation. What would be my choices for add math to the documentation? Hopefully using latex, since there really isn't AFAIK any other competitor for this.

Where in the current documentation is there any math notation /at all/? In all my reading of it, I have not run across anything that appeared like it was being used. Besides that question, is the full power of LaTeX math notation really necessary here? I somehow doubt anything more than simple expressions of runtime performance and container behaviors are appropriate for any documentation we have.

There is exactly one instance of LaTeX math in the whole docs, it's in the description of audioop, AFAIR, an contains a sum over square roots... So, that's not really a concern of mine ;) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Neal Becker wrote:

There is an effort as part of numpy to come up with a new system using docstrings. It seems to me it would be unfortunate if these two efforts were not coordinated.

I don't think so. The issue with numpy is getting our act together and making parseable docstrings for auto-generated API documentation using existing tools or slightly modified versions thereof. No one is actually contemplating building a new tool. AFAICT, Georg's (excellent) work doesn't address that use. I don't think there is anything to coordinate, here. Provided that Georg's system doesn't place too many restrictions on the reST it handles, we could use the available reST math options if we wanted to use Georg's system. I'd much rather see Georg spend his time working on the docs for the Python language and the feature set it requires. If the numpy project needs to extend that feature set, we'll provide the manpower ourselves. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

>>> What would be my choices for add math to the documentation? >> Where in the current documentation is there any math notation /at >> all/? Georg> There is exactly one instance of LaTeX math in the whole docs, Georg> it's in the description of audioop, AFAIR, an contains a sum over Georg> square roots... Georg> So, that's not really a concern of mine ;) You must realize that people will use the core tools to create documentation for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not. Skip

Georg> So, that's not really a concern of mine ;)

You must realize that people will use the core tools to create documentation for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not.

I disagree. The documentation infrastructure of Python should only consider the needs of Python itself. If other people can use that infrastructure for other purposes, fine - if they find that it does not meet their needs, they have to look elsewhere. We are developing a programming language here, not a typesetting system. Regards, Martin

I would agree with the point that python core should be considered first, but I would also only see beneficial to leave the door open to the need of other packages. I have (briefly but intensely) worked on a revamp of pydoc earlier on this year, and while collecting requirements from a number of places having maths expressions or else appeared important for a number of cases (and a very reasonable request in a case) . That particular point leads to something that I see important for what a new/better documentation system should provide: a good and modular interface to access the documentation, process it, and navigate it. When looking at the particular example discussed here, it could be implemented by allowing a "pluggable" processing components for docstrings (and let a given package developer the possibility to use as much as the default documentation machinery as possible and implement the processing mathml, latex, whatever, as wanted). One can consider the possibility to have the "custom" processing of the docstring embedded in the package itself. Laurent 2007/5/21, Brett Cannon <brett@python.org>:

On 5/20/07, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

...

Georg> So, that's not really a concern of mine ;)

You must realize that people will use the core tools to create

documentation

...

for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not.

I disagree. The documentation infrastructure of Python should only consider the needs of Python itself. If other people can use that infrastructure for other purposes, fine - if they find that it does not meet their needs, they have to look elsewhere.

Martin beat me to my comment. =) Python's needs should come first, period. If Georg wants to add math support, fine. But honestly I would rather he spend his time on Python-specific stuff then get bogged down to support possible third parties.

-Brett

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com

skip@pobox.com schrieb:

Brett> Martin beat me to my comment. =) Python's needs should come Brett> first, period. If Georg wants to add math support, fine. But Brett> honestly I would rather he spend his time on Python-specific Brett> stuff then get bogged down to support possible third parties.

I think the people who have responded to my comment read too much into it. Nowhere do I think I asked Georg to write an equation typesetter to include in the Python documentation toolchain. I asked that math capability be considered.

And that is reasonable, of course.

I have no idea what tools he used to build his new documentation set. I only briefly glanced at a couple of the output pages. I think what he has done is marvelous. However, I don't think the door should be shut on equation display. Is there a route to it based on the tools Georg is using?

In the end, it all depends on what kind of support basic reST can deliver. IMO, you still get the best math output from LaTeX, but I don't really know many other things. That is also something I want to convey: I'm very fond of LaTeX, and use it regularly for all my University work. For the Python docs, however, I can see many advantages of the docutils approach.

If not, then I think some accommodation should be made. I'm being vague here on purpose because I'm unfamiliar with the available tools. The one thing I do know is that LaTeX provides that today and by removing it from the toolchain you have removed a significant piece of functionality.

That's the point I see differently: for the Python core docs, it's not significant, and my efforts are primarily limited to that area. cheers, Georg

On 5/21/07, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

I think the people who have responded to my comment read too much into it. Nowhere do I think I asked Georg to write an equation typesetter to include in the Python documentation toolchain. I asked that math capability be considered. I have no idea what tools he used to build his new documentation set. I only briefly glanced at a couple of the output pages. I think what he has done is marvelous. However, I don't think the door should be shut on equation display. Is there a route to it based on the tools Georg is using?

I don't think anything in the world can replace TeX for math typesetting. So if math typesetting was a requirement (which it should not be, for that very reason), then we could not consider anything but TeX.

You can use docutils to generate LaTeX output from reST, and you can put raw LaTeX into the output with ".. raw:: latex". I would imagine this is sufficient for now. -bob

On Monday 21 May 2007, skip@pobox.com wrote:

Then I submit that you are probably removing some significant piece of functionality from the provided documentation toolchain which some people probably rely on. After all, that's what LaTeX excels at. They will be able to continue to use the old tools, but where will they get them if they are no longer part of Python?

I'll be happy to pull the existing tools out into a separate distribution if we move to something else for Python. There are too many users of the existing tools to abandon. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On Mon, May 21, 2007 at 09:23:47AM -0400, Fred L. Drake, Jr. wrote:

I'll be happy to pull the existing tools out into a separate distribution if we move to something else for Python. There are too many users of the existing tools to abandon.

That seems like a straightforward task. The big stumbling block in switching away from LaTeX has always been the effort of making a good conversion; if Georg's work does 80% of the job, we should definitely take advantage of the opportunity and try to switch. Advantages of reST: * The required tool chain shrinks (at least if you're not making printed output, which will probably still go through LaTeX). * Tool chain is now more easily scriptable, and it'll be easier to make use of the docs from Python code. * We can produce XML output through the rst2xml script. Disadvantages: * reST markup isn't much simpler than LaTeX. --amk

Hoi, Fred L. Drake, Jr. <fdrake <at> acm.org> writes:

On Monday 21 May 2007, A.M. Kuchling wrote:

...

Disadvantages:

* reST markup isn't much simpler than LaTeX.

* reST doesn't support nested markup, which is used in the current documentation.

For a lightweight markup language that is human readable (which rst certainly is) the syntax is surprisingly powerful. You can nest any block tag and I'm not sure how often you have to nest roles and stuff like that. The goal of the new docs is a less complex syntax and currently nothing beats reStructuredText in terms of readability and possibilities. rst is simpler than latex: LaTeX: \item[\code{*?}, \code{+?}, \code{??}] The \character{*}, \character{+}, and \character{?} qualifiers are all \dfn{greedy}; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE \regexp{<.*>} is matched against \code{'<H1>title</H1>'}, it will match the entire string, and not just \code{'<H1>'}. Adding \character{?} after the qualifier makes it perform the match in \dfn{non-greedy} or \dfn{minimal} fashion; as \emph{few} characters as possible will be matched. Using \regexp{.*?} in the previous expression will match only \code{'<H1>'}. Here the same in rst: ``*?``, ``+?``, ``??`` The ``'\*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE :regexp:`<.\*>` is matched against ``'<H1>title</H1>'``, it will match the entire string, and not just ``'<H1>'``. Adding ``'?'`` after the qualifier makes it perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few* characters as possible will be matched. Using :regexp:`.\*?` in the previous expression will match only ``'<H1>'``. Regards, Armin

Armin Ronacher writes:

rst is simpler than latex:

LaTeX:

\item[\code{*?}, \code{+?}, \code{??}] The \character{*}, \character{+}, and \character{?} qualifiers are all \dfn{greedy}; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE \regexp{<.*>} is matched against \code{'<H1>title</H1>'}, it will match the entire string, and not just \code{'<H1>'}. Adding \character{?} after the qualifier makes it perform the match in \dfn{non-greedy} or \dfn{minimal} fashion; as \emph{few} characters as possible will be matched. Using \regexp{.*?} in the previous expression will match only \code{'<H1>'}.

Here the same in rst:

``*?``, ``+?``, ``??`` The ``'\*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE :regexp:`<.\*>` is matched against ``'<H1>title</H1>'``, it will match the entire string, and not just ``'<H1>'``. Adding ``'?'`` after the qualifier makes it perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few* characters as possible will be matched. Using :regexp:`.\*?` in the previous expression will match only ``'<H1>'``.

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST. I don't think that's an argument against switching to reST, though. Georg's site speaks for itself. Kudos!

...

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST.

Well. That was a bad example. But if you look at the converted sources and open the source file you can see that rst is a lot cleaner that latex for this type of documentation.

In your examples, I think the ReST version can be cleaned up quite a bit. First by using the .. default-role:: literal directive so that you can type `foo()` instead of using double back quotes and then you can remove the redundant semantic markup. Like this: `\*?`, `+?`, `??` The "`*`", "`+`" and "`?`" qualifiers are all *greedy*; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE `<.*>` is matched against `'<H1>title</H1>'`, it will match the entire string, and not just `'<H1>'`. Adding "`?`" after the qualifier makes it perform the match in *non-greedy* or *minimal* fashion; as *few* characters as possible will be matched. Using `.*?` in the previous expression will match only `'<H1>'`. The above is the most readable version. For example, semantic markup like :regexp:`<.\*>` doesn't serve any useful purpose. The end result is that the text is typesetted with a fixed-width font, no matter if you prepend :regexp: to it or not. -- mvh Björn

...

In your examples, I think the ReST version can be cleaned up quite a bit. First by using the .. default-role:: literal directive so that you can type `foo()` instead of using double back quotes and then you can remove the redundant semantic markup. Like this:

I've already assigned the default role to `var` since it's used most frequently.

Having two ways of spelling literal code is wasting markup, for me.

Reassign it then. :) `var` makes italic text but you can use *var* for that instead. Minor point I know, but it would make reading ReST source just a tiny bit easier. -- mvh Björn

skip@pobox.com wrote:

It would appear that while we slept Jens Mortensen was busy at work on his rst2{latex,latexmath,mathml}.py scripts:

http://docutils.sourceforge.net/sandbox/jensj/latex_math/

Note the date on the files. It seems to work pretty well, and as others have pointed out, LaTeX notation is probably more familiar to people interested in math display than anything else.

I know almost nothing about docutils internals. How do I 'install' the above?

Stephen J. Turnbull wrote:

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST.

It's interesting how perceptions can differ - I find heavily marked up latex tends to blur into a huge wall of text when I try to read it because of all of the {} and \ characters everywhere. With reST, on the other hand, I find the use of the relatively 'light' backquote and colon characters to delineate the markup breaks things up sufficiently that I can easily ignore the markup in order to read what has actually been written. So in Armin's example, I found the reST version *much* easier to read. Whether that difference in perception is due simply to my relative lack of experience in using LaTeX, or to something else, I have no idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

- If you make a mistake in LaTeX, you will get a cryptic error which is usually a little difficult to figure out (if you're not used to it). You can an error though.

FWIW, the pure Python program in Tools/scripts/texchecker.py does a pretty good job of catching typical LaTeX mistakes and giving high-quality error reporting. Raymond

- If you make a mistake in LaTeX, you will get a cryptic error which is usually a little difficult to figure out (if you're not used to it). You can an error though.

FWIW, the pure Python program in Tools/scripts/texchecker.py does a pretty good job of catching typical LaTeX mistakes and giving high-quality error reporting. With that tool, I've been making doc contributions for years and not needed my own LaTeX build. Also, I did not need to learn LaTeX itself. It was sufficient to read a little of Documenting Python and then model the markup from existing docs. In contrast, whenever I've tried to build a complex ReST document, it was *always* a struggle. Copying from existing docs doesn't help much there because the cues are more subtle. As Martin pointed out, most errors slide-by because the mis-markup is typically read as valid, unmarked-up text. I find myself having to continously build and view and html file as I write. I like ResT for light-weight work but think it is not ready for prime-time with respect to more complex requirements. Fred is also correct in that we don't seem to have people rushing to contribute docs (more than a line or two). For a long-time, we've always said that it is okay to submit plain text doc contributions and that another person downstream would do the mark-up. We've had few takers. Raymond

On Tue, May 22, 2007 at 06:13:36PM +0200, Georg Brandl wrote:

Finding the location that tells you how to suggest changes, and opening a new bug in the infamous SF tracker is not really something people do happily. A "click here to suggest a change" link that leads to a pseudo- edit-form, complete with preview facility, might prove more effective.

Indeed. I know my instinctive reaction to the Python docs is "oh, this is not something which the public can contribute to". Something more like the PHP-style "public annotations" might be good - with an appropriate moderation / voting system on the annotations it could possibly be very good.

- If you make a mistake in LaTeX, you will get a cryptic error which is usually a little difficult to figure out (if you're not used to it). You can an error though.

FWIW, the pure Python program in Tools/scripts/texchecker.py does a pretty good job of catching typical LaTeX mistakes and giving high-quality error reporting. With that tool, I've been making doc contributions for years and not needed my own LaTeX build. Also, I did not need to learn LaTeX itself. It was sufficient to read a little of Documenting Python and then model the markup from existing docs. In contrast, whenever I've tried to build a complex ReST document, it was *always* a struggle. Copying from existing docs doesn't help much there because the cues are more subtle. As Martin pointed out, most errors slide-by because the mis-markup is typically read as valid, unmarked-up text. I find myself having to continously build and view and html file as I write. I like ResT for light-weight work but think it is not ready for prime-time with respect to more complex requirements. Fred is also correct in that we don't seem to have people rushing to contribute docs (more than a line or two). For a long-time, we've always said that it is okay to submit plain text doc contributions and that another person downstream would do the mark-up. We've had few takers. Raymond

On 5/22/07, Martin Blais <blais@furius.ca> wrote:

ReST works well only when there is little markup. Writing code documentation generally requires a lot of markup, you want to make variables, classes, functions, parameters, constants, etc.. (A better avenue IMHO would be to augment docutils with some code to automatically figure out the syntax of functions, parameters, classes, etc., i.e., less markup, and if we do this in Python we may be able to use introspection. This is a challenge, however, I don't know if it can be done at all.)

Just to follow-up on that idea: I don't think it would be very difficult to write a very small modification to docutils that interprets the default role with more "smarts", for example, you can all guess what the types of these are about: `class Foo` (this is a class Foo) `bar(a, b, c) -> str` (this is a function "bar" which returns a string) `name (attribute)` (this is an attribute) ...so why couldn't the computer solve that problem for you? I'm sure we could make it happen. Essentially, what is missing from ReST is "less markup for documenting programs". By restricting the problem-set to Python programs, we can go a long way towards making much of this automatic, even without resorting to introspecting the source code that is being documented.

Talin wrote:

As in the above example, the use of backticks can be signal to the document processor that the enclosed text should be examined for identifiers and other Python syntax.

Does this mean it's time for "pyST" -- Python-structured text?-) -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

On 5/24/07, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

Talin wrote:

...

As in the above example, the use of backticks can be signal to the document processor that the enclosed text should be examined for identifiers and other Python syntax.

Does this mean it's time for "pyST" -- Python-structured text?-)

Not before someone writes it. Georg Brandl's awesome ReST based system has the nice property that it actually exists and works. For a great number of reasons it is superior to the existing LaTeX based system, and I hope and think that they are strong enough to replace LaTeX with it. -- mvh Björn

On 5/22/07, Armin Ronacher <armin.ronacher@active-4.com> wrote:

...

...

* reST markup isn't much simpler than LaTeX.

* reST doesn't support nested markup, which is used in the current documentation.

For a lightweight markup language that is human readable (which rst certainly is) the syntax is surprisingly powerful. You can nest any block tag and I'm not sure how often you have to nest roles and stuff like that. The goal of the new docs is a less complex syntax and currently nothing beats reStructuredText in terms of readability and possibilities.

About possibilities: I'm sorry but that is simply not true. LaTeX provides the full power of macro expansion, while ReST is a fixed (almost, roles are an exception) syntax which has its own set of problems and ambiguities. It is far from perfect, and definitely does not have the flexibility that LaTeX provides. Some of the syntax cannot be nested (try to combine ** with literals).

rst is simpler than latex:

That, and the ability to already parse it from Python and more easily convert to other formats (one of LaTeX's weaknesses), are the only benefits that I can see to switching away from LaTeX. I have to admit I'm afraid we would be moving to a lesser technology, and the driver for that seems to be people's reluctance to work with the more powerful, more complex tool. Not saying it is invalid (it's about people, in the end), but I still don't see what's the big problem with LaTeX.

On Tuesday 22 May 2007, Barry Warsaw wrote:

considerably. Even with a nice distro packaging system it can be a PITA to get all the tools you need to build the documentation properly installed. A pure-Python solution, even a lesser one, would be a win if we can still produce top quality online and written documentation from one source.

The biggest potential wins I see for a new system are: - more contributions - platform-independent processing I remain sceptical on being able to achieve the first, but there some hope for it. The later should make things easier for people who are willing to put the work into contribution, which is valuable in its own right. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

skip@pobox.com schrieb:

Fred> The biggest potential wins I see for a new system are:

Fred> - more contributions

Fred> - platform-independent processing

Fred> I remain sceptical on being able to achieve the first, but there Fred> some hope for it.

You at least take away a common excuse for lack of contributions. True whiners will just come up with new ones (e.g., "the documentation isn't available in Sanskrit yet" or "the dog ate my changes before I could type them into the computer"). ;-)

But that's at least funnier than before :) Georg

skip@pobox.com wrote:

Fred> The biggest potential wins I see for a new system are:

Fred> - more contributions

Fred> - platform-independent processing

Fred> I remain sceptical on being able to achieve the first, but there Fred> some hope for it.

You at least take away a common excuse for lack of contributions. True whiners will just come up with new ones (e.g., "the documentation isn't available in Sanskrit yet" or "the dog ate my changes before I could type them into the computer"). ;-)

But doesn't *everyone* now know that documentation contributions don't have to be marked up? It's certainly been said enough. Maybe that fact should be more prominent in the documentation? Then we'll just have to worry about getting people to read it ... regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden ------------------ Asciimercial --------------------- Get on the web: Blog, lens and tag your way to fame!! holdenweb.blogspot.com squidoo.com/pythonology tagged items: del.icio.us/steve.holden/python All these services currently offer free registration! -------------- Thank You for Reading ----------------

On Tue, May 22, 2007 at 11:45:04AM -0500, skip@pobox.com wrote: -> -> >> You at least take away a common excuse for lack of contributions. -> >> True whiners will just come up with new ones (e.g., "the -> >> documentation isn't available in Sanskrit yet" or "the dog ate my -> >> changes before I could type them into the computer"). ;-) -> -> Steve> But doesn't *everyone* now know that documentation contributions -> Steve> don't have to be marked up? It's certainly been said -> Steve> enough. -> -> Sure, but that doesn't stop the true whiners. ;-) Nothing stops the true whiners ;). I think new and exciting ways of viewing, searching, annotating, linking to/from, and indexing the docs are more important than new formats for (not) writing the docs. For example, this rocks! :: http://pydoc.gbrandl.de/search.html?q=os.path&area=default There have been (good) efforts to wikify the docs in the past. IMO what would make them really work would be to have docs.python.org, the "official" Python docs location, start hosting these efforts. As long as that location is static, I think the majority of users will ignore other efforts. cheers, --titus p.s. Are there good directions for installing the toolchain for current docs building anywhere? I've tried once or twice, but despite a lot of LaTeX experience I could never get everything hooked up right.

On Tue, May 22, 2007 at 01:19:36PM -0400, Steve Holden wrote:

...

For example, this rocks! ::

http://pydoc.gbrandl.de/search.html?q=os.path&area=default

It would be more impressive if the search string returned hits ...

Also if it was not completely reliant on JavaScript... (Maybe it's not finished yet?)

On Tue, 22 May 2007, Steve Holden wrote:

But doesn't *everyone* now know that documentation contributions don't have to be marked up? It's certainly been said enough. Maybe that fact should be more prominent in the documentation? Then we'll just have to worry about getting people to read it ...

I think the issue is instant gratification. You don't get the satisfaction of seeing your changes unless you're willing to write them in LaTeX, and that's a pretty big barrier -- a lot of what motivates open source volunteers is the sense of fulfillment. (Hence, by the same nature, Wiki-like editing with instant changes visible online will probably greatly increase contributions.) -- ?!ng

Armin Ronacher schrieb:

...

That, and the ability to already parse it from Python and more easily convert to other formats (one of LaTeX's weaknesses), are the only benefits that I can see to switching away from LaTeX. I have to admit I'm afraid we would be moving to a lesser technology, and the driver for that seems to be people's reluctance to work with the more powerful, more complex tool. Not saying it is invalid (it's about people, in the end), but I still don't see what's the big problem with LaTeX. The problem with latex is that it requires more knowledge thus the amount of people that can contribute is a lot lower. It's a lot harder to generate a searchindex, do interlinking, generate a dynamic documentation with comments etc.

This could of course be done with LaTeX as well, but the current toolchain uses latex2html which is written and customized in Perl -- I'd never try to do anything with it.

Don't get me wrong, LaTeX is a powerful tool and I use it for every bigger document i type. I just think it's not the best choice for documenting scripting languages.

Who's documenting a scripting language? Georg

On 5/22/07, Georg Brandl <g.brandl@gmx.net> wrote:

...

Don't get me wrong, LaTeX is a powerful tool and I use it for every bigger document i type. I just think it's not the best choice for documenting scripting languages.

Who's documenting a scripting language?

Hehe I can't believe I just wrote that...

We are developing a programming language here, not a typesetting system.

Good point, Martin. Are you implying that the documentation should be kept in LaTeX, a widely-accepted widely-disseminated stable documentation language, which someone else maintains, rather than ReST, which elements of the the Python community maintain? Bill

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On May 21, 2007, at 12:28 AM, Martin v. Löwis wrote:

I disagree. The documentation infrastructure of Python should only consider the needs of Python itself. If other people can use that infrastructure for other purposes, fine - if they find that it does not meet their needs, they have to look elsewhere.

I would take a fairly broad view of the term "for Python" though. Specifically, third party modules and applications written for and in Python should be explicitly supported. I think we'd like to see one (preferred) documentation tool chain for core Python, plus the vast number of third party modules and apps. I don't see any reason why Georg's reST-based system couldn't provide that (80/20 rule perhaps?). I'd point to for example the howto templates, which can be easily used for third party applications. Mailman uses howtos for example. BTW, Georg excellent work. I'm a big latex fan and long-time user, but I do think that using reST will open the door to a lot more contributions. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRlHzJXEjvBPtnXfVAQLNXQP/QUuZ2gc/DpoidI9jYt7mr66Z+JYHsslv fe4CvFSjd9OxwA3eOynd9dSOSkO6QHQPDVomW8axEkJJSHWNosnr9gmDZcC75nAD JTt4rGImqwkcVIAzE91pZ3fmce/ltp9p1Ru3B1dDRmXbNgHxZ9njaz60MPFszC/H 19jSBo5sqSU= =hlhi -----END PGP SIGNATURE-----

Neal> It happens that for math markup, there isn't really anything Neal> better (or more familiar) than latex. True enough. There is MathML and its offspring, ASCIIMathML, which are probably worth looking at. http://www.w3.org/Math/ http://www1.chapman.edu/~jipsen/asciimath.html I have no idea if either one has backend support for presentation outside the web, but if people are interested in this (probably within the docutils scope) they probably should be considered. ASCIIMathML in particular is probably worth using now within even if you can't convert it to any other format. It's about as readable as LaTeX source. Skip

Neal Becker writes:

Perhaps my comment was misunderstood. I have no objection to a new system, and it does not have to be based on latex. I just hope there will be some escape mechanism that allows math.

Docutils already provides the "raw" directive. I don't know if the latex backend supports it, but adding support shouldn't be hard. I don't think even thinking about that is Georg's responsibility, of course.

Georg Brandl <g.brandl@gmx.net> wrote:

Nick Craig-Wood schrieb:

...

Being a seasoned unix user, I tend to reach for pydoc as my first stab at finding some documentation rather than (after excavating the mouse from under a pile of paper) use a web browser.

If you've ever used pydoc you'll know it reads docstrings and for some modules they are great and for others they are sorely lacking.

If pydoc could show all this documentation as well I'd be very happy!

Maybe your quick dispatch feature could be added to pydoc too?

It is my intention to work together with Ron Adam to make the pydoc <-> documentation integration as seamless as possible.

So I'll be able to read the main docs for a module in a terminal without reaching for the web browser (or info)? That would be great! How would pydoc decide which bit of docs it is going to show? If I type "pydoc re" is it going to give me the rather sparse __doc__ from the re module or the nice reST docs? Or maybe both, one after the other? Or will I have to use a flag to dis-ambiguate? If you type "pydoc re" at the moment then it says in it MODULE DOCS http://www.python.org/doc/current/lib/module-re.html which is pretty much useless to me when ssh-ed in to a linux box half way around the world...

...

It is missing conversion of ``comment'' at the moment as I'm sure you know...

Sorry, what did you mean?

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'. For an example see the first paragraph here: http://pydoc.gbrandl.de/reference/index.html -- Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

On Wednesday, May 23, 2007, at 12:40PM, <skip@pobox.com> wrote:

Nick> If you type "pydoc re" at the moment then it says in it

Nick> MODULE DOCS Nick> http://www.python.org/doc/current/lib/module-re.html

Nick> which is pretty much useless to me when ssh-ed in to a linux box Nick> half way around the world...

I get quite a bit of information about re (I've never known /F to be a documentation slouch). Only one bit of that information is a reference to the page in the library reference manual. And if I happen to be ssh'd into a machine halfway round the world through a Gnome terminal I can right mouse over that URL and pop the page up in my default local browser. If you set the PYTHONDOCS environment variable you can point it to a local (or at least different) copy of the libref manual. A flag could be added to pydoc to show that content instead, however being html it probably would be difficult to read unless pumped through lynx -dump or something similar.

pydoc can already do this for the language reference (try 'pydoc import' on a system with a local install of the python documentation). Ronald

Skip _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

2007/5/23, Nick Craig-Wood <nick@craig-wood.com>:

Georg Brandl <g.brandl@gmx.net> wrote:

...

Nick Craig-Wood schrieb:

...

Being a seasoned unix user, I tend to reach for pydoc as my first stab at finding some documentation rather than (after excavating the mouse from under a pile of paper) use a web browser.

If you've ever used pydoc you'll know it reads docstrings and for some modules they are great and for others they are sorely lacking.

If pydoc could show all this documentation as well I'd be very happy!

Maybe your quick dispatch feature could be added to pydoc too?

It is my intention to work together with Ron Adam to make the pydoc <-> documentation integration as seamless as possible.

So I'll be able to read the main docs for a module in a terminal without reaching for the web browser (or info)? That would be great!

One option is to use a text-mode browser (lynx, links, or the likes). The other is to develop a terminal mode application (currently in pydoc, I believe)

How would pydoc decide which bit of docs it is going to show?

If I type "pydoc re" is it going to give me the rather sparse __doc__ from the re module or the nice reST docs? Or maybe both, one after the other? Or will I have to use a flag to dis-ambiguate?

I really think that making pydoc a solid library to extract/search/navigate the documentation offers a lot of interesting perspectives. When one think beyond the application discussed here, there are a lot of tools (ipython, or any IDE for example) that could make great use of the facility. [note: Ron and I seemed to disagree on what (and how) pydoc should be, and that in particular, but I keep a keen interest in having such a library.]

If you type "pydoc re" at the moment then it says in it

MODULE DOCS http://www.python.org/doc/current/lib/module-re.html

which is pretty much useless to me when ssh-ed in to a linux box half way around the world...

...

...

It is missing conversion of ``comment'' at the moment as I'm sure you know...

Sorry, what did you mean?

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'.

For an example see the first paragraph here:

http://pydoc.gbrandl.de/reference/index.html

-- Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com

Nick Craig-Wood schrieb:

...

...

It is missing conversion of ``comment'' at the moment as I'm sure you know...

Sorry, what did you mean?

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'.

Ahh, now the dime has fallen ;) (sorry, German phrase) Yes, it should probably use Unicode equivalents of these quotes, as it does with en- and em-dashes. There are also nifty "post-processor" filters which operate on complete HTML pages and replace normal quotes by "smart" ones, perhaps that is the way to go. Georg

Nick Craig-Wood wrote:

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'.

For an example see the first paragraph here:

http://pydoc.gbrandl.de/reference/index.html

In fairness to Georg, latex2html also misses the smart quotes. See the same paragraph here: http://docs.python.org/ref/front.html -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu

Nick Craig-Wood wrote:

Georg Brandl <g.brandl@gmx.net> wrote:

...

Nick Craig-Wood schrieb:

...

Being a seasoned unix user, I tend to reach for pydoc as my first stab at finding some documentation rather than (after excavating the mouse from under a pile of paper) use a web browser.

If you've ever used pydoc you'll know it reads docstrings and for some modules they are great and for others they are sorely lacking.

If pydoc could show all this documentation as well I'd be very happy!

Maybe your quick dispatch feature could be added to pydoc too? It is my intention to work together with Ron Adam to make the pydoc <-> documentation integration as seamless as possible.

So I'll be able to read the main docs for a module in a terminal without reaching for the web browser (or info)? That would be great!

How would pydoc decide which bit of docs it is going to show?

Pydoc currently gets topic info for some items by scraping the text from document 'local' web pages. This is kind of messy for a couple of reasons. - The documents may not be installed locally. - It can be problematic locating the docs even if they are installed. - They are not formatted well after they are retrieved. I think this is an area for improvement. This feature is also limited to a small list where the word being searched for is a keyword, or a very common topic reference, *and* they are not likely to clash with other module, class, or function names. The introspection help parts of pydoc are completely separate from topic help parts. So replacing this part can be done without much trouble. What the best behavior is and how it should work would need to be discussed. Keep in mind doc strings are meant to be more of a quick reference to an item, and Pydoc is the interface for that.

If I type "pydoc re" is it going to give me the rather sparse __doc__ from the re module or the nice reST docs? Or maybe both, one after the other? Or will I have to use a flag to dis-ambiguate?

If retrieval from the full docs is desired, then it will probably need to be disambiguated in some way or be a separate interface. help('re') # Quick reference on 're'. helpdocs('re') # Full documentation for 're'.

If you type "pydoc re" at the moment then it says in it

MODULE DOCS http://www.python.org/doc/current/lib/module-re.html

which is pretty much useless to me when ssh-ed in to a linux box half way around the world...

...

...

It is missing conversion of ``comment'' at the moment as I'm sure you know... Sorry, what did you mean?

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'.

For an example see the first paragraph here:

http://pydoc.gbrandl.de/reference/index.html

Nick Craig-Wood wrote:
 > On Wed, May 23, 2007 at 12:46:50PM -0500, Ron Adam wrote:
 >> Nick Craig-Wood wrote:
 >>> So I'll be able to read the main docs for a module in a terminal
 >>> without reaching for the web browser (or info)?  That would be great!
 >>>
 >>> How would pydoc decide which bit of docs it is going to show?
 >> Pydoc currently gets topic info for some items by scraping the text from
 >> document 'local' web pages.  This is kind of messy for a couple of reasons.
 >>    - The documents may not be installed locally.
 >>    - It can be problematic locating the docs even if they are installed.
 >>    - They are not formatted well after they are retrieved.
 >>
 >> I think this is an area for improvement.
 >
 > And it would be improved by converting the docs to reST I imagine.

Yes, this will need a reST to html converter for displaying in the html 
browser.  DocUtils provides that, but it's not part of the library. (?)

And for console text output, is the unmodified reST suitable, or would it 
be desired to modify it in some way?

Should a subset of the main documents be included with pydoc to avoid the 
documents not available messages if they are not installed?

Or should the topics retrieval code be moved from pydoc to the main 
document tools so it's installed with the documents.  Then that can be 
maintianed with the documents instead of being maintained with pydoc.  Then 
pydoc will just looks for it, instead of looking for the html pages.


 >> This feature is also limited to a small list where the word being searched
 >> for is a keyword, or a very common topic reference, *and* they are not
 >> likely to clash with other module, class, or function names.
 >>
 >> The introspection help parts of pydoc are completely separate from topic
 >> help parts.  So replacing this part can be done without much trouble. 
What
 >> the best behavior is and how it should work would need to be discussed.
 >>
 >> Keep in mind doc strings are meant to be more of a quick reference to an
 >> item, and Pydoc is the interface for that.
 >
 > I think that if reST was an acceptable form for the documentation, and
 > it could be auto included in the main docs from docstrings then you
 > would find more modules completely documented in __doc__.

That would be fine for third party modules if they want to do that or if 
there is not much difference between the two.

 >>> If I type "pydoc re" is it going to give me the rather sparse __doc__
 >> >from the re module or the nice reST docs?  Or maybe both, one after
 >>> the other?  Or will I have to use a flag to dis-ambiguate?
 >> If retrieval from the full docs is desired, then it will probably need to
 >> be disambiguated in some way or be a separate interface.
 >>
 >>    help('re')        # Quick reference on 're'.
 >>    helpdocs('re')    # Full documentation for 're'.
 >
 > Actually if it gave both sets of docs quick, then long, one after the
 > other that would suit me fine.

That may work well for the full documentation, but the quick reference 
wouldn't be a short quick reference any more.


I'm attempting to have a pydoc api call that gets a single item or sub-item 
and format it to a desired output so it can be included in other content. 
That's makes it possible for the full docs (not necessarily pythons) to 
embed pydoc output in it if it's desirable.  This will need pydoc 
formatters for the target document type.  I hope to include a reST output 
formatter for pydoc.

The help() function is imported from pydoc by site.py when you start 
python.  It may not be difficult to have it as a function that first tries 
pydoc to get a request, and if the original request is returned unchanged, 
tries to get information from the full documentation.  There could be a way 
to select one or the other, (or both).

But this feature doesn't need to be built into pydoc, or the full 
documentation.  They just need to be able to work together so things like 
this are possible in an easy to write 4 or 5 line function. (give or take a 
few lines)

So it looks like most of these issues are more a matter of how to organize 
the interfaces.  It turns out that what I've done with pydoc, and what 
Georg is doing with the main documentation should work together quite nicely.

Cheers,
    Ron

This subject is generating a lot of discussion and [almost entirely] 
positive feedback. It would be a great shame to run out of steam.

Does it need a PEP to see a chance of it getting accepted as the formal 
documentation system? (or a pronouncement that it will never happen...)

Michael Foord


Georg Brandl wrote:
> Ron Adam schrieb:
>   
>> Nick Craig-Wood wrote:
>>  > On Wed, May 23, 2007 at 12:46:50PM -0500, Ron Adam wrote:
>>  >> Nick Craig-Wood wrote:
>>  >>> So I'll be able to read the main docs for a module in a terminal
>>  >>> without reaching for the web browser (or info)?  That would be great!
>>  >>>
>>  >>> How would pydoc decide which bit of docs it is going to show?
>>  >> Pydoc currently gets topic info for some items by scraping the text from
>>  >> document 'local' web pages.  This is kind of messy for a couple of reasons.
>>  >>    - The documents may not be installed locally.
>>  >>    - It can be problematic locating the docs even if they are installed.
>>  >>    - They are not formatted well after they are retrieved.
>>  >>
>>  >> I think this is an area for improvement.
>>  >
>>  > And it would be improved by converting the docs to reST I imagine.
>>
>> Yes, this will need a reST to html converter for displaying in the html 
>> browser.  DocUtils provides that, but it's not part of the library. (?)
>>
>> And for console text output, is the unmodified reST suitable, or would it 
>> be desired to modify it in some way?
>>     
>
> A text writer for docutils should not be hard to write. You'd get something that
> looks like the reST, but stripped of markup that makes no sense when viewed on
> a terminal, such as :class:`xyz`.
>
> Georg
>
>   

Martin v. Löwis schrieb:

Michael Foord schrieb:

...

This subject is generating a lot of discussion and [almost entirely] positive feedback. It would be a great shame to run out of steam.

Does it need a PEP to see a chance of it getting accepted as the formal documentation system? (or a pronouncement that it will never happen...)

No. First of all, it needs a dedicated developer (preferably, but not necessarily a committer) who indicates willingness to maintain that for the coming years and releases.

It might be that Fred Drake's offer to maintain the documentation would be still valid after such a switch, but we should not assume so without explicit confirmation. It might be that this would be the time to pass one documentation maintenance to somebody else (and I seriously do not have any one particular in mind here).

Assuming that Fred goes into well-earned retirement from the doc maintainer position (private mail exchange hinted that way), and nobody more qualified steps up, I'd be available to take that post. (If someone else wants to take maintainership of the content, very good, I'd have to be maintainer of the build tools anyway.) I'd then try to form a doc maintaining team, just as the PEP editor team that was created recently, to deal with the (hopefully relatively large ;) ) amount of comments and edit requests.

Then, I think a should be made where the documentation is converted. Again, a volunteer would be needed to create the branch, and then eventually merge it back to the trunk. It might be helpful, but isn't strictly necessary, to close all documentation patches before doing so, as they all break with the conversion. For that activity, multiple volunteers would be useful.

I agree. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Nick Craig-Wood schrieb:

...

...

It is missing conversion of ``comment'' at the moment as I'm sure you know...

Sorry, what did you mean?

``comment'' produces smart quotes in latex if I remember correctly. You probably want to convert it somehow because it looks a bit odd on the web page as it stands. I'm not sure what the reST replacement might be, but converting it just to "comment" would probably be OK. Likewise with `comment' to 'comment'.

For an example see the first paragraph here:

http://pydoc.gbrandl.de/reference/index.html

Okay, there's now support for SmartyPants in Subversion -- it converts these quotes as well as triple dashes to their pretty equivalents. cheers, Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Jon Ribbens schrieb:

On Sat, May 19, 2007 at 07:14:09PM +0200, Georg Brandl wrote:

...

For the impatient: the result can be seen at <http://pydoc.gbrandl.de>.

I think that looks great.

One comment I have, I don't know if it's relevant - it perhaps depends on whether the "Global Module Index" is auto-generated or not. This is the page I visit the most out of all the Python documentation, and it's far too large and unwieldy. IMHO it would be much better if only the top-level modules were shown here - having the single package 'distutils', for example, take up nearly 50 entries in the list is almost certainly hindering a lot more people than it helps. It would perhaps be better if such packages show up as one entry, which shows the sub-modules when clicked on.

Sure, that is certainly possible. Georg

On Monday 21 May 2007, skip@pobox.com wrote:

Take a look at <http://www.webfast.com/modindex/>. It records request counts for the various pages and presents the most frequently requested pages in a section at the top of the page. I can make the script available if anyone wants it (it uses Myghty - Mason in Python.)

This is very cool. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On 5/21/07, skip@pobox.com <skip@pobox.com> wrote:

>> One comment I have, I don't know if it's relevant - it perhaps >> depends on whether the "Global Module Index" is auto-generated or >> not. This is the page I visit the most out of all the Python >> documentation, and it's far too large and unwieldy. IMHO it would be >> much better if only the top-level modules were shown here - having >> the single package 'distutils', for example, take up nearly 50 >> entries in the list is almost certainly hindering a lot more people >> than it helps. It would perhaps be better if such packages show up as >> one entry, which shows the sub-modules when clicked on.

Georg> Sure, that is certainly possible.

Take a look at <http://www.webfast.com/modindex/>. It records request counts for the various pages and presents the most frequently requested pages in a section at the top of the page. I can make the script available if anyone wants it (it uses Myghty - Mason in Python.)

+1 for integrating this with the official docs. I loved this the last time you posted it too. STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

Georg Brandl schrieb:

Armin Ronacher schrieb:

...

Hoi,

Additionally to the offline docs that Georg published some days ago there is also a web version which currently looks and works pretty much like the offline version. There are however some differences that are worth knowing:

- Cleaner URLs. You can actually guess them because we took the idea the PHP people had and check for similar pages if a page does not exist. We do however redirect if there was a match so that the URL stays unique.

- The search doesn't require JavaScript (but is currently disabled due to a buggy stemmer and indexer)

Also, try

http://pydoc.gbrandl.de:3000/os.path.exists

and

http://pydoc.gbrandl.de:3000/os.paht.exists Georg

Am Wed, 23 May 2007 08:30:17 +0200 schrieb Georg Brandl <g.brandl@gmx.net>:

[...] Also, try

http://pydoc.gbrandl.de:3000/os.path.exists [...]

Looks good. But should the source pages really use syntax highlighting? I think if somebody is interested in the source then they should get the real source without any highlighting. If you decide to keep the syntax highlighting then the highlighting of multiline ReST strings should be fixed. For example see the source for splitext(). Thanks for the work, Dennis Benzinger

Hoi, Due to some server issues I had to take the web version down. But expect an updated version in a few days. Regards, Armin

On Sat, May 19, 2007 at 10:48:29AM -0700, Josiah Carlson wrote:

I'm generally a curmudgeon when it comes to 'the docs could be done better'. But this? I like it. A lot. Especially if you can get these other features in:

...

- a "quick-dispatch" function: e.g., docs.python.org/q?os.path.split would redirect you to the matching location.

Seconded! -- even if it's just for modules, this would be great. I can't count the times I've wished I could type e.g., 'docs.python.org/httplib' the way I can type 'php.net/array_search' to try to find out whether the needle comes before or after the haystack. Dustin

On Sat, May 19, 2007 at 03:31:59PM -0500, Ron Adam wrote:

- The html syntax highlighters. (Pydoc can use those)

I have a patch on the docutils patch tracker that does this. Code is probably of a rather bad quality, but it outputs LaTeX and HTML. If we can work together to improve this patch and get it in docutils it will avoid having different syntaxes and behavior depending on the front-end to docutils being used (I am thinking of rest2web, trac, and I am probably forgetting some others). The patch has been sitting there for almost 6 months without review, but I have that if people other than me work on it and ask for review it will both improve, and get reviewed, and eventually get in ! Sorry for the shameless plug, but I really do think we need a unifying approach to this. Gaël

Martin Blais wrote:

e.g. are you still marking classes as classes and functions as functions in the ReST source

It seems so (modulo XXX's and TODO's in Georg's implementation, probably ^_^) -- all of the pages have "show source" links, so you can see for yourself. I'm not an expert with the documentation system, but the markup on <http://pydoc.gbrandl.de/modules/codecs.txt> looks pretty complete to me.

(Somewhat related, but another idea from back then, which was never implemented IMO, was to find a way to automatically pull and convert the docstrings from the source code into the documentation, thus unifying all the information in one place.)

While it's probably not possible to simply generate the documentation from the docstrings, it would certainly seem interesting to get have some means (like a directive) to pull docstrings into the documentation. I think however that while migrating the docs do reStructuredText is comparatively straightforward [1]_, pulling documentation from the docstrings will require quite a bit of design and discussion work. So I'd suggest we postpone this idea until we have a working documentation system in reStructuredText, so we don't clutter the discussion. .. [1] I'm sure there will still be quite a few issues to sort out that I'm simply not seeing right now. Best wishes, Lea

Vinay Sajip schrieb:

Georg Brandl <g.brandl <at> gmx.net> writes:

...

For the impatient: the result can be seen at <http://pydoc.gbrandl.de>.

- the toolchain is pure Python, therefore can easily be shipped

Very nice! As well as looking very attractive and professional, [...]

BTW, I have to give lots of credit for the looks to Armin Ronacher. I'm not so much of a designer ;) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

On 5/20/07, Georg Brandl <g.brandl@gmx.net> wrote:

...

Very nice! As well as looking very attractive and professional, the all-Python toolset should make it easier to build the documentation - I've not been able to get a trouble-free setup of the docs toolchain on Windows.

Yep. As it is now, you need three packages from the Cheese Shop: Docutils, Pygments (the highlighter) and Jinja (the templating engine). This shouldn't be problematic, though they could also be stripped down and included.

This is great. IMHO if this is to compete to become the official Python docs, I would argue for even less dependencies, even at the cost of more generic/bland output, for portability reasons and to stimulate greater adoption. If we can make some of those dependencies optional and only rely on docutils, that could make it ubiquitous. Another thing to keep in mind: I don't know if the directives you defined are very generic, but if they are, it would be interesting to consider migrating them up into docutils (if it makes sense), and see if they could support documenting other programming languages. Could this be a language-independent documenting toolkit? Could we document LISP or Ruby code with it? Georg, thanks again!

Scott Dial schrieb:

Neal Becker wrote:

...

Sounds very interesting. I just have one concern/question. I hope that while moving away from latex, we are not precluding the ability to write math as part of the documentation. What would be my choices for add math to the documentation? Hopefully using latex, since there really isn't AFAIK any other competitor for this.

Where in the current documentation is there any math notation /at all/? In all my reading of it, I have not run across anything that appeared like it was being used. Besides that question, is the full power of LaTeX math notation really necessary here? I somehow doubt anything more than simple expressions of runtime performance and container behaviors are appropriate for any documentation we have.

There is exactly one instance of LaTeX math in the whole docs, it's in the description of audioop, AFAIR, an contains a sum over square roots... So, that's not really a concern of mine ;) Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

Neal Becker wrote:

There is an effort as part of numpy to come up with a new system using docstrings. It seems to me it would be unfortunate if these two efforts were not coordinated.

I don't think so. The issue with numpy is getting our act together and making parseable docstrings for auto-generated API documentation using existing tools or slightly modified versions thereof. No one is actually contemplating building a new tool. AFAICT, Georg's (excellent) work doesn't address that use. I don't think there is anything to coordinate, here. Provided that Georg's system doesn't place too many restrictions on the reST it handles, we could use the available reST math options if we wanted to use Georg's system. I'd much rather see Georg spend his time working on the docs for the Python language and the feature set it requires. If the numpy project needs to extend that feature set, we'll provide the manpower ourselves. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

>>> What would be my choices for add math to the documentation? >> Where in the current documentation is there any math notation /at >> all/? Georg> There is exactly one instance of LaTeX math in the whole docs, Georg> it's in the description of audioop, AFAIR, an contains a sum over Georg> square roots... Georg> So, that's not really a concern of mine ;) You must realize that people will use the core tools to create documentation for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not. Skip

Georg> So, that's not really a concern of mine ;)

You must realize that people will use the core tools to create documentation for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not.

I disagree. The documentation infrastructure of Python should only consider the needs of Python itself. If other people can use that infrastructure for other purposes, fine - if they find that it does not meet their needs, they have to look elsewhere. We are developing a programming language here, not a typesetting system. Regards, Martin

I would agree with the point that python core should be considered first, but I would also only see beneficial to leave the door open to the need of other packages. I have (briefly but intensely) worked on a revamp of pydoc earlier on this year, and while collecting requirements from a number of places having maths expressions or else appeared important for a number of cases (and a very reasonable request in a case) . That particular point leads to something that I see important for what a new/better documentation system should provide: a good and modular interface to access the documentation, process it, and navigate it. When looking at the particular example discussed here, it could be implemented by allowing a "pluggable" processing components for docstrings (and let a given package developer the possibility to use as much as the default documentation machinery as possible and implement the processing mathml, latex, whatever, as wanted). One can consider the possibility to have the "custom" processing of the docstring embedded in the package itself. Laurent 2007/5/21, Brett Cannon <brett@python.org>:

On 5/20/07, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

...

Georg> So, that's not really a concern of mine ;)

You must realize that people will use the core tools to create

documentation

...

for third party packages which aren't in the core. If you replace LaTeX with something else I think you need to keep math in mind whether it's used in the core documentation or not.

I disagree. The documentation infrastructure of Python should only consider the needs of Python itself. If other people can use that infrastructure for other purposes, fine - if they find that it does not meet their needs, they have to look elsewhere.

Martin beat me to my comment. =) Python's needs should come first, period. If Georg wants to add math support, fine. But honestly I would rather he spend his time on Python-specific stuff then get bogged down to support possible third parties.

-Brett

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com

skip@pobox.com schrieb:

Brett> Martin beat me to my comment. =) Python's needs should come Brett> first, period. If Georg wants to add math support, fine. But Brett> honestly I would rather he spend his time on Python-specific Brett> stuff then get bogged down to support possible third parties.

I think the people who have responded to my comment read too much into it. Nowhere do I think I asked Georg to write an equation typesetter to include in the Python documentation toolchain. I asked that math capability be considered.

And that is reasonable, of course.

I have no idea what tools he used to build his new documentation set. I only briefly glanced at a couple of the output pages. I think what he has done is marvelous. However, I don't think the door should be shut on equation display. Is there a route to it based on the tools Georg is using?

In the end, it all depends on what kind of support basic reST can deliver. IMO, you still get the best math output from LaTeX, but I don't really know many other things. That is also something I want to convey: I'm very fond of LaTeX, and use it regularly for all my University work. For the Python docs, however, I can see many advantages of the docutils approach.

If not, then I think some accommodation should be made. I'm being vague here on purpose because I'm unfamiliar with the available tools. The one thing I do know is that LaTeX provides that today and by removing it from the toolchain you have removed a significant piece of functionality.

That's the point I see differently: for the Python core docs, it's not significant, and my efforts are primarily limited to that area. cheers, Georg

On 5/21/07, "Martin v. Löwis" <martin@v.loewis.de> wrote:

...

I think the people who have responded to my comment read too much into it. Nowhere do I think I asked Georg to write an equation typesetter to include in the Python documentation toolchain. I asked that math capability be considered. I have no idea what tools he used to build his new documentation set. I only briefly glanced at a couple of the output pages. I think what he has done is marvelous. However, I don't think the door should be shut on equation display. Is there a route to it based on the tools Georg is using?

I don't think anything in the world can replace TeX for math typesetting. So if math typesetting was a requirement (which it should not be, for that very reason), then we could not consider anything but TeX.

You can use docutils to generate LaTeX output from reST, and you can put raw LaTeX into the output with ".. raw:: latex". I would imagine this is sufficient for now. -bob

On Monday 21 May 2007, skip@pobox.com wrote:

Then I submit that you are probably removing some significant piece of functionality from the provided documentation toolchain which some people probably rely on. After all, that's what LaTeX excels at. They will be able to continue to use the old tools, but where will they get them if they are no longer part of Python?

I'll be happy to pull the existing tools out into a separate distribution if we move to something else for Python. There are too many users of the existing tools to abandon. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

On Mon, May 21, 2007 at 09:23:47AM -0400, Fred L. Drake, Jr. wrote:

I'll be happy to pull the existing tools out into a separate distribution if we move to something else for Python. There are too many users of the existing tools to abandon.

That seems like a straightforward task. The big stumbling block in switching away from LaTeX has always been the effort of making a good conversion; if Georg's work does 80% of the job, we should definitely take advantage of the opportunity and try to switch. Advantages of reST: * The required tool chain shrinks (at least if you're not making printed output, which will probably still go through LaTeX). * Tool chain is now more easily scriptable, and it'll be easier to make use of the docs from Python code. * We can produce XML output through the rst2xml script. Disadvantages: * reST markup isn't much simpler than LaTeX. --amk

Hoi, Fred L. Drake, Jr. <fdrake <at> acm.org> writes:

On Monday 21 May 2007, A.M. Kuchling wrote:

...

Disadvantages:

* reST markup isn't much simpler than LaTeX.

* reST doesn't support nested markup, which is used in the current documentation.

For a lightweight markup language that is human readable (which rst certainly is) the syntax is surprisingly powerful. You can nest any block tag and I'm not sure how often you have to nest roles and stuff like that. The goal of the new docs is a less complex syntax and currently nothing beats reStructuredText in terms of readability and possibilities. rst is simpler than latex: LaTeX: \item[\code{*?}, \code{+?}, \code{??}] The \character{*}, \character{+}, and \character{?} qualifiers are all \dfn{greedy}; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE \regexp{<.*>} is matched against \code{'<H1>title</H1>'}, it will match the entire string, and not just \code{'<H1>'}. Adding \character{?} after the qualifier makes it perform the match in \dfn{non-greedy} or \dfn{minimal} fashion; as \emph{few} characters as possible will be matched. Using \regexp{.*?} in the previous expression will match only \code{'<H1>'}. Here the same in rst: ``*?``, ``+?``, ``??`` The ``'\*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE :regexp:`<.\*>` is matched against ``'<H1>title</H1>'``, it will match the entire string, and not just ``'<H1>'``. Adding ``'?'`` after the qualifier makes it perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few* characters as possible will be matched. Using :regexp:`.\*?` in the previous expression will match only ``'<H1>'``. Regards, Armin

Armin Ronacher writes:

rst is simpler than latex:

LaTeX:

\item[\code{*?}, \code{+?}, \code{??}] The \character{*}, \character{+}, and \character{?} qualifiers are all \dfn{greedy}; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE \regexp{<.*>} is matched against \code{'<H1>title</H1>'}, it will match the entire string, and not just \code{'<H1>'}. Adding \character{?} after the qualifier makes it perform the match in \dfn{non-greedy} or \dfn{minimal} fashion; as \emph{few} characters as possible will be matched. Using \regexp{.*?} in the previous expression will match only \code{'<H1>'}.

Here the same in rst:

``*?``, ``+?``, ``??`` The ``'\*'``, ``'+'``, and ``'?'`` qualifiers are all :dfn:`greedy`; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE :regexp:`<.\*>` is matched against ``'<H1>title</H1>'``, it will match the entire string, and not just ``'<H1>'``. Adding ``'?'`` after the qualifier makes it perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few* characters as possible will be matched. Using :regexp:`.\*?` in the previous expression will match only ``'<H1>'``.

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST. I don't think that's an argument against switching to reST, though. Georg's site speaks for itself. Kudos!

...

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST.

Well. That was a bad example. But if you look at the converted sources and open the source file you can see that rst is a lot cleaner that latex for this type of documentation.

In your examples, I think the ReST version can be cleaned up quite a bit. First by using the .. default-role:: literal directive so that you can type `foo()` instead of using double back quotes and then you can remove the redundant semantic markup. Like this: `\*?`, `+?`, `??` The "`*`", "`+`" and "`?`" qualifiers are all *greedy*; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE `<.*>` is matched against `'<H1>title</H1>'`, it will match the entire string, and not just `'<H1>'`. Adding "`?`" after the qualifier makes it perform the match in *non-greedy* or *minimal* fashion; as *few* characters as possible will be matched. Using `.*?` in the previous expression will match only `'<H1>'`. The above is the most readable version. For example, semantic markup like :regexp:`<.\*>` doesn't serve any useful purpose. The end result is that the text is typesetted with a fixed-width font, no matter if you prepend :regexp: to it or not. -- mvh Björn

...

In your examples, I think the ReST version can be cleaned up quite a bit. First by using the .. default-role:: literal directive so that you can type `foo()` instead of using double back quotes and then you can remove the redundant semantic markup. Like this:

I've already assigned the default role to `var` since it's used most frequently.

Having two ways of spelling literal code is wasting markup, for me.

Reassign it then. :) `var` makes italic text but you can use *var* for that instead. Minor point I know, but it would make reading ReST source just a tiny bit easier. -- mvh Björn

skip@pobox.com wrote:

It would appear that while we slept Jens Mortensen was busy at work on his rst2{latex,latexmath,mathml}.py scripts:

http://docutils.sourceforge.net/sandbox/jensj/latex_math/

Note the date on the files. It seems to work pretty well, and as others have pointed out, LaTeX notation is probably more familiar to people interested in math display than anything else.

I know almost nothing about docutils internals. How do I 'install' the above?

Stephen J. Turnbull wrote:

IMO that pair of examples shows clearly that, in this application, reST is not an improvement over LaTeX in terms of readability/ writability of source. It's probably not worse, although I can't help muttering "EIBTI". In particular I find the "``'...'``" construct horribly unreadable because it makes it hard to find the Python syntax in all the reST.

It's interesting how perceptions can differ - I find heavily marked up latex tends to blur into a huge wall of text when I try to read it because of all of the {} and \ characters everywhere. With reST, on the other hand, I find the use of the relatively 'light' backquote and colon characters to delineate the markup breaks things up sufficiently that I can easily ignore the markup in order to read what has actually been written. So in Armin's example, I found the reST version *much* easier to read. Whether that difference in perception is due simply to my relative lack of experience in using LaTeX, or to something else, I have no idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

- If you make a mistake in LaTeX, you will get a cryptic error which is usually a little difficult to figure out (if you're not used to it). You can an error though.

FWIW, the pure Python program in Tools/scripts/texchecker.py does a pretty good job of catching typical LaTeX mistakes and giving high-quality error reporting. Raymond