Proposal: go/doc: headings, lists, and links in Go doc comments

Russ Cox
January 2022

Earlier discussion at https://go.dev/issue/48305 and https://go.dev/issue/45533.
Proposal at https://go.dev/issue/51082.

§ Abstract

This proposal improves support for headings, lists, and links in Go doc comments, while remaining backwards compatible with existing comments. It includes a new package, go/doc/comment, exposing a parsed syntax tree for doc comments, and it includes changes to go/printer and therefore gofmt to format doc comments in a standard way.

<style> th, td { vertical-align: top; } </style>

For example, existing lists reformat from the display on the left to the one on the right:

<table> <tr><td><img src="51082/list-before.png" width="308" height="271"> <td><img src="51082/list-after.png" width="307" height="289"> </table>

URL links can be rewritten to change the display on the left to the one on the right:

<table> <tr><td><img src="51082/link-before.png" width="307" height="94"> <td><img src="51082/link-after.png" width="308" height="103"> </table>

And package doc comments (and others) can be rewritten to link to specific symbols:

<table> <tr><td><img src="51082/doclink.png" width="366" height="383"> </table>

(Gerrit's Markdown viewer does not render the images. See the GitHub rendering instead.)

§ Background

Go's doc comments today support plain text and preformatted (HTML <pre>) blocks, along with a subtle rule for turning certain lines into headings. The specific rules are partially documented in the doc comment for go/doc.ToHTML:

Each span of unindented non-blank lines is converted into a single paragraph. There is one exception to the rule: a span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation other than parentheses and commas is formatted as a heading.

A span of indented lines is converted into a <pre> block, with the common indent prefix removed.

URLs in the comment text are converted into links; if the URL also appears in the words map, the link is taken from the map (if the corresponding map value is the empty string, the URL is not converted into a link).

A pair of (consecutive) backticks (`) is converted to a unicode left quote (“), and a pair of (consecutive) single quotes (') is converted to a unicode right quote (”).

Go identifiers that appear in the words map are italicized; if the corresponding map value is not the empty string, it is considered a URL and the word is converted into a link.

The current Go doc comment format has served us well since its introduction in 2009. There has only been one significant change, which was the addition of headings in 2011, now clearly a bad design (see below). But there are also a few long-open issues and proposals about doc comments, including:

• #7349 points out that the headings rule does not work well with non-Roman scripts.
• #31739 points out that lines ending with double quotes cannot be headings.
• #34377 points out that lines ending with parens cannot be headings.
• #7873 asks for list support.
• #45533 proposes linking of symbols, written as [io.EOF], to make it easier to write good top-level doc comments and cross-reference with other packages.

It makes sense, as we approach a decade of experience, to take what we've learned and make one coherent revision, setting the syntax for the next 10 or so years.

Goals and non-goals

The primary design criteria for Go doc comments was to make them readable as ordinary comments when viewing the source code directly, in contrast to systems like C#'s Xmldoc, Java's Javadoc, and Perl's Perlpod. The goal was to prioritize readability, avoiding syntactic ceremony and complexity. This remains as a primary goal.

Another concern, new since 2009, is backwards compatibility. Whatever changes we make, existing doc comments must generally continue to render well. Less important, but still something to keep in mind, is forward compatibility: keeping new doc comments rendering well in older Go versions, for a smoother transition.

Another goal for the revamp is that it include writing a separate, standalone web page explaining how to write Go doc comments. Today that information is squirreled away in the doc.ToHTML comment and is not easily found or widely known.

Within those constraints, the focus I have set for this revamp is to address the issues listed above. Specifically:

1. Make the header syntax more predictable. The headings rule is clearly difficult to remember and has too many false negatives. But further adjustments of the current rule run the risk of false positives.

2. Add support for lists. There are many times in documentation when a bullet or numbered list is called for. Those appear in many doc comments today, as indented <pre> blocks.

3. Add support for links to URLs. Today the only way to link to something is by writing the URL directly, but those can sometimes be quite unreadable and interrupt the text.

4. Add support for links to Go API documentation, in the current package and in other packages. This would have multiple benefits, but one is the ability in large packages to write top-level doc comments that give a good overview and link directly to the functions and types being described.

I believe it also makes sense to add another goal:

5. Add formatting of doc comments to gofmt, to promote consistent appearance and create more room for future changes.

It is not a goal to support every possible kind of documentation or markup. For example:

• Plain text has served us very well so far, and while some might prefer that comments allow font changes, the syntactic ceremony and complexity involved seems not worth the benefit, no matter how it is done.

• People have asked for support for embedding images in documentation (see #39513), but that adds significant complexity as well: image size hints, different resolutions, image sets, images suitable for both light and dark mode presentation, and so on. It is also difficult (but not impossible) to render them on the command line. Although images clearly have important uses, all this complexity is in direct conflict with the primary goal. For these reasons, images are out of scope. I also note that C#'s Xmldoc and Perl's Perlpod seem not to have image support, although Java's Javadoc does.

Based on a preliminary GitHub discussion, which in turn built on an earlier discussion of doc links, this proposal aims to address headings, lists, and links. The following subsections elaborate the background for each feature.

Headings

As noted above, headings were added in 2011, and the current documentation says:

a span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation other than parentheses and commas is formatted as a heading.

This is not quite accurate. The code has been updated over time without maintaining the comment. Today it includes a special case to allow “apostrophe s” so “Go's doc comments” is a heading, but “Go's heading rule doesn't work” is not a heading. On the other hand, Unicode single right quotes are not rejected, so “Go’s heading rule doesn’t work” is a heading. On the other other hand, certain Unicode punctuation is rejected, so that “The § symbol” is not a heading, even though “The ¶ symbol” is a heading. The rule also includes a special case for periods, to permit “The go.dev site” but not “Best. Heading. Ever”. The rule started out simple but insufficient, and now the accumulated patches have made it a mess.

Lists

There is no support for lists today. Documentation needing lists uses indented preformatted text instead.

For example, here are the docs for cookiejar.PublicSuffixList:

// PublicSuffixList provides the public suffix of a domain. For example:
//      - the public suffix of "example.com" is "com",
//      - the public suffix of "foo1.foo2.foo3.co.uk" is "co.uk", and
//      - the public suffix of "bar.pvt.k12.ma.us" is "pvt.k12.ma.us".
//
// Implementations of PublicSuffixList must be safe for concurrent use by
// multiple goroutines.

And here are the docs for url.URL.String:

// In the second form, the following rules apply:
//      - if u.Scheme is empty, scheme: is omitted.
//      - if u.User is nil, userinfo@ is omitted.
//      - if u.Host is empty, host/ is omitted.
//      - if u.Scheme and u.Host are empty and u.User is nil,
//         the entire scheme://userinfo@host/ is omitted.
//      - if u.Host is non-empty and u.Path begins with a /,
//         the form host/path does not add its own /.
//      - if u.RawQuery is empty, ?query is omitted.
//      - if u.Fragment is empty, #fragment is omitted.

Ideally, we'd like to adopt a rule that makes these into bullet lists without any edits at all.

Links

Documentation is more useful with clear links to other web pages. For example, the encoding/json package doc today says:

// Package json implements encoding and decoding of JSON as defined in
// RFC 7159. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// See "JSON and Go" for an introduction to this package:
// https://golang.org/doc/articles/json_and_go.html

There is no link to the actual RFC 7159, leaving the reader to Google it. And the link to the “JSON and Go” article must be copied and pasted.

Documentation is also more useful with clear links to other documentation, whether it's one function linking to another, preferred version or a top-level doc comment summarizing the overall API of the package, with links to the key types and functions. Today there is no way to do this. Names can be mentioned, of course, but users must find the docs on their own.

§ Proposal

This proposal has nine parts:

1. New syntax for headings.
2. New syntax for lists.
3. New syntax for URL links
4. New syntax for documentation links.
5. Reformatting of doc comments by go/printer and gofmt.
6. New documentation for doc comments.
7. Rendering the new syntax on go.dev and pkg.go.dev.
8. A new go/doc/comment package.
9. Changes in the go/doc package.

Note that the new syntax is inspired by and aims to be a subset of Markdown, but it is not full Markdown. This is discussed in the Rationale section below.

New syntax for headings

If a span of non-blank lines is a single line beginning with # followed by a space or tab and then additional text, then that line is a heading.

# This is a heading

Here are some examples of variations that do not satisfy the rule and are therefore not headings:

#This is not a heading, because there is no space.

# This is not a heading,
# because it is multiple lines.

# This is not a heading,
because it is also multiple lines.

The next span is not a heading, because there is no additional text:

#

In the middle of a span of non-blank lines,
# this is not a heading either.

 # This is not a heading, because it is indented.

The old heading rule will remain valid, which is acceptable since it mainly has false negatives, not false positives. This will keep existing doc comments rendering as they do today.

New syntax for lists

In a span of lines all blank or indented by one or more spaces or tabs (which would otherwise be a <pre> block), if the first indented line begins with a bullet list marker or a numbered list marker, then that span of indented lines is a bullet list or numbered list. A bullet list marker is a dash, star, or plus followed by a space or tab and then text. In a bullet list, each line beginning with a bullet list marker starts a new list item. A numbered list marker is a decimal number followed by a period or right parenthesis, then a space or tab, and then text. In a numbered list, each line beginning with a number list marker starts a new list item. Item numbers are left as is, never renumbered (unlike Markdown).

Using this rule, most doc comments with <pre> bullet lists today will instead be rendered as proper bullet lists.

Note that the rule means that a list item followed by a blank line followed by additional indented text continues the list item (regardless of comparative indentation level):

// Text.
//
//  - A bullet.
//
//     Another paragraph of that first bullet.
//
// - A second bullet.

Note also that there are no code blocks inside list items—any indented paragraph following a list item continues the list item, and the list ends at the next unindented line—nor are there nested lists. This avoids any the space-counting subtlety like in Markdown.

To re-emphasize, a critical property of this definition of lists is that it makes existing doc comments written with pseudo-lists turn into doc comments with real lists.

Markdown recognizes three different bullets: -, *, and +. In the main Go repo, the dash is dominant: in comments of the form `//[ \t]+[-+*] ` (grepping, so some of these may not be in doc comments), 84% use -, 14% use *, and 2% use +. In a now slightly dated corpus of external Go code, the star is dominant: 37.6% -, 61.8% *, 0.7% +.

Markdown also recognizes two different numeric list item suffixes: “1.” and “1)”. In the main Go repo, 66% of comments use “1.” (versus 34% for “1)”). In the external corpus, “1.” is again the dominant choice, 81% to 19%.

We have two conflicting goals: handle existing comments well, and avoid needless variation. To satisfy both, all three bullets and both forms of numbers will be recognized, but gofmt (see below) will rewrite them to a single canonical form: dash for bullets, and “N.” for numbers. (Why dashes and not asterisks? Proper typesetting of bullet lists sometimes does use dashes, but never uses asterisks, so using dashes keeps the comments looking as typographically clean as possible.)

New syntax for URL links

A span of unindented non-blank lines defines link targets when each line is of the form “[Text]: URL”. In other text, “[Text]” represents a link to URL using the given text—in HTML, <a href="URL">Text</a>.

For example:

// Package json implements encoding and decoding of JSON as defined in
// [RFC 7159]. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// For an introduction to this package, see the article
// “[JSON and Go].”
//
// [RFC 7159]: https://tools.ietf.org/html/rfc7159
// [JSON and Go]: https://golang.org/doc/articles/json_and_go.html

Note that the link definitions can only be given in their own “paragraph” (span of non-blank unindented lines), which can contain more than one such definition, one per line. If there is no corresponding URL declaration, then (except for doc links, described in the next section) the text is not a hyperlink, and the square brackets are preserved.

This format only minimally interrupts the flow of the actual text, since the URLs are moved to a separate section. It also roughly matches the Markdown shortcut reference link format, without the optional title text.

New syntax for documentation links

Doc links are links of the form “[Name1]” or “[Name1.Name2]” to refer to exported identifiers in the current package, or “[pkg]”, “[pkg.Name1]”, or “[pkg.Name1.Name2]” to refer to identifiers in other packages. In the second form, “pkg” can be either a full import path or the assumed package name of an existing import. The assumed package name is either the identifier in a renamed import or else the name assumed by goimports. (Goimports inserts renamings when that assumption is not correct, so this rule should work for essentially all Go code.) A “pkg” is only assumed to be a full import path if it starts with a domain name (a path element with a dot) or is one of the packages from the standard library (“[os]”, “[encoding/json]”, and so on). To avoid problems with maps, generics, and array types, doc links must be both preceded and followed by punctuation, spaces, tabs, or the start or end of a line.

For example, if the current package imports encoding/json, then “[json.Decoder]” can be written in place of “[encoding/json.Decoder]” to link to the docs for encoding/json's Decoder.

The implications and potential false positives of this implied URL link are presented by Joe Tsai here. In particular, the false positive rate appears to be low enough not to worry about.

To illustrate the need for the punctuation restriction, consider:

// Constant folding computes the exact constant value ([constant.Value])
// for every expression ([ast.Expr]) that is a compile-time constant.

versus

// The Types field, a map[ast.Expr]TypeAndValue,
// holds type-checking results for all AST expressions.

and

// A SHA1 hash is a [Size]byte.

Reformatting doc comments

We propose that gofmt and go/printer reformat doc comments to a conventional presentation, updating old syntax to new syntax and standardizing details such as the indentation used for preformatted blocks, the exact spacing in lists, and so on.

The reformatting would canonicalize a doc comment so that it renders exactly as before but uses standard layout. Specifically:

• All paragraphs are separated by single blank lines.
• Legacy headings are converted to “#” headings.
• All preformatted blocks are indented by a single tab.
• All preformatted blocks have a blank line before and after.
• All list markers are written as “␠␠-␠“ (space space dash space) or “␠N.␠“ (space number dot space).
• All list item continuation text, including additional paragraphs, is indented by four spaces.
• Lists that themselves contain any blank lines are separated from the preceding paragraph or heading by a blank line.
• All lists are followed by a blank line (except at the end of the doc comment).
• If there is a blank line anywhere in a list, there are blank lines between all list elements.
• All bullet list items use - as the bullet (* and + are converted).
• All numbered list items use the “N.” form (“N)” is converted).
• The ASCII double-single-quote forms that have always been defined to render as “ and ” are replaced with those.
• Link URL definitions are moved to the bottom of the doc comment, in two different blank-line-separated groups: definitions used by the doc comment and definitions not used. Separating the second group makes it easy first to recognize that there are unused definitions and second to delete them.
• Tool directive comments, such as //go:build linux, are moved to the end of the doc comment (after link URLs).

The exact details have been chosen to make as