\documentclass{article} \usepackage{a4wide} \title{Parsing Rd files} \author{Duncan Murdoch} \usepackage{Sweave} \SweaveOpts{concordance=true} <>= options(width=65) @ \newcommand{\parseRd}{\texttt{parse\_Rd()} } \newcommand{\Rdtag}{\texttt{Rd\_tag} } \newcommand{\Rdoption}{\texttt{Rd\_option} } \sloppy \begin{document} \maketitle \abstract{ R 2.9.x introduced a parser for Rd format help files. When integrated into the build/install process in R 2.10.x, it will allow easier processing, easier syntax checking, and eventually, easier conversion to other formats. To write this parser, it was necessary to make some small changes to the specification of the format as described in Writing R Extensions, and to make some choices when that description was ambiguous. Some existing Rd files do not meet the stricter format requirements (and some were incorrectly formatted under the old requirements, but the errors were missed by the older checks). The new stricter format is being informally called Rdoc version 2. R 2.10.x will include some changes to the format, including some Sweave-like ways to include executable code in R documentation files. This document describes the new format, necessary changes to existing Rd files, and the structure returned from the \parseRd function. It also includes some documentation of the new \verb!\Sexpr! markup macro. } \section{Introduction} Prior to this document, the R documentation (Rd) file format did not have a formal description. It grew over time, with pieces added, and R and Perl code written to process it. This document describes a formal parser (written in Bison) for the format, and the R structure that it outputs. The intention is to make it easier for the format to evolve, or to be replaced with a completely new one. The next section describes the syntax; section \ref{sec:changes} describes changes from what is described in Writing R Extensions for R 2.11.0. The following section describes the \parseRd function and its output. Some additions to the specification have been made in R-devel (to become R 2.12.x); these are noted earlier and discussed in more detail in the final section. Except for the new \verb!\Sexpr! macro, this document does not describe the interpretation of the markup in Rd files; see Writing R Extensions for that. \section{Rd Syntax Specification} Rd files are text files with an associated encoding, readable as text connections in R. The syntax only depends on a few ASCII characters, so at the level of this specification the encoding is not important, however higher level interpretation will depend on the text connection having read the proper encoding of the file. There are three distinct types of text within an Rd file: LaTeX-like, R-like, and verbatim. The first two contain markup, text and comments; verbatim text contains only text and comments. \subsection{Encodings} The parser works on connections, which have an associated encoding. It is the caller's responsibility to declare the connection encoding properly when it is opened. Not all encodings may be supported, because the libraries R uses cannot always perform conversions. In particular, Asian double byte encodings are likely to cause trouble when that is not the native encoding. Plain ASCII, UTF-8 and Latin1 should be supported on all systems when the system is complete. \subsection{Lexical structure} The characters \verb!\!, \verb!%!, \verb!{!, and \verb!}! have special meaning in almost all parts of an Rd file; details are given below. The exceptions are listed in section \ref{sec:nonescapes}. End of line (newline) characters mark the end of pieces of text. The end of a file does the same. The newline markers are returned as part of the text. The brackets \verb![! and \verb!]! have special meaning in certain contexts; see section \ref{sec:options}. Whitespace (blanks, tabs, and formfeeds) is included in text. \subsubsection{The backslash} The backslash \verb!\! is used as an escape character: \verb!\\!, \verb!\%!, \verb!\{! and \verb!\}! remove the special meaning of the second character. The parser will drop the initial backslash, and return the other character as part of the text. The backslash is also used as an initial character for markup macros. In an R-like or LaTeX-like context, a backslash followed by an alphabetic character starts a macro; the macro name continues until the first non-alphanumeric character. If the name is not recognized the parser drops all digits from the end, and tries again. If it is still not recognized it is returned as an UNKNOWN token. All other uses of backslashes are allowed, and are passed through by the parser as text. \subsubsection{The percent symbol} An unescaped percent symbol \verb!%! marks the beginning of an Rd comment, which runs to the end of the current line. The parser returns these marked as COMMENT tokens. The newline or EOF at the end of the comment is not included as part of it. \subsubsection{Braces} The braces \verb!{! and \verb!}! are used to mark the arguments to markup macros, and may also appear within text, provided they balance. In R-like and verbatim text, the parser keeps a count of brace depth, and the return to zero depth signals the end of the token. In Latex-like mode, braces group tokens; groups may be empty. Opening and closing braces on arguments and when used for grouping in Latex-like text are not returned by the parser, but balanced braces within R-like or verbatim text are. \subsection{Markup} Markup includes macros starting with a backslash \verb!\! with the second character alphabetic. Some macros (e.g. \verb!\R!) take no arguments, some take one or more arguments surrounded by braces (e.g. \verb!\code{}!, \verb!\section{}{}!). There are also three special classes of macros. The \verb!\link{}! macro \label{sec:options} (and \verb!\Sexpr! in R 2.10.x) may take one argument or two, with the first marked as an option in square brackets, e.g. \verb!\link[option]{arg}!. The \verb!\eqn{}! and \verb!deqn{}! macros may take one or two arguments in braces, e.g. \verb!\eqn{}! or \verb!\eqn{}{}!. The last special class of macros consists of \verb!#ifdef ... #endif! and \verb!#ifndef ... #endif!. The \verb!#! symbols must appear as the first character on a line starting with \verb!#ifdef!, \verb!#ifndef!, or \verb!#endif!, or it is not considered special. These look like C preprocessor directives, but are processed by the parser as two argument macros, with the first argument being the token on the line of the first directive, and the second one being the text between the first directive and the \verb!#endif!. Any text on the line following the \verb!#endif! will be discarded. For example, \begin{verbatim} #ifdef unix Some text #endif (text to be discarded) \end{verbatim} is processed with first argument \verb!unix! (surrounded by whitespace) and second argument \verb!Some text!. Markup macros are subdivided into those that start sections of the Rd file, and those that may be used within a section. The sectioning macros may only be used at the top level, while the others must be nested within the arguments of other macros. (The \verb!\Sexpr! macro, to be introduced in R 2.10.0, is normally used within a section, but can be used as a sectioning macro. See section \ref{sec:Sexpr}.) Markup macros are also subdivided into those that contain list-like content, and those that don't. The \verb!\item! macro may only be used within the argument of a list-like macro. Within \verb!\enumerate{}! or \verb!\itemize{}! it takes no arguments, within \verb!\arguments{}!, \verb!\value{}! and \verb!\describe{}! it takes two arguments. Finally, the text within the argument of each macro is classified into one of the three types of text mentioned above. For example, the text within \verb!\code{}! macros is R-like, and that within \verb!\samp{}! or \verb!\verb{}! macros is verbatim. The complete list of Rd file macros is shown in Tables \ref{tab:sectioning} to \ref{tab:other}. \begin{table} \caption{Table of sectioning macros. \label{tab:sectioning}} \begin{center}\begin{tabular}{lcccc} \hline Macro & Arguments & Section? & List? & Text type \\ \hline \verb!\arguments! & 1 & yes & \verb!\item{}{}! & Latex-like \\ \verb!\author! & 1 & yes & no & Latex-like \\ \verb!\concept! & 1 & yes & no & Latex-like \\ \verb!\description!& 1 & yes & no & Latex-like \\ \verb!\details! & 1 & yes & no & Latex-like \\ \verb!\docType! & 1 & yes & no & Latex-like \\ \verb!\encoding!& 1 & yes & no & Latex-like \\ \verb!\format! & 1 & yes & no & Latex-like \\ \verb!\keyword! & 1 & yes & no & Latex-like \\ \verb!\name! & 1 & yes & no & Latex-like \\ \verb!\note! & 1 & yes & no & Latex-like \\ \verb!\references! & 1 & yes & no & Latex-like \\ \verb!\section! & 2 & yes & no & Latex-like \\ \verb!\seealso! & 1 & yes & no & Latex-like \\ \verb!\source! & 1 & yes & no & Latex-like \\ \verb!\title! & 1 & yes & no & Latex-like \\ \verb!\value! & 1 & yes & \verb!\item{}{}! & Latex-like \\ & & & & \\ \verb!\examples!& 1 & yes & no & R-like \\ \verb!\usage! & 1 & yes & no & R-like \\ & & & & \\ \verb!\alias! & 1 & yes & no & Verbatim \\ \verb!\Rdversion!& 1 & yes & no & Verbatim \\ \verb!\synopsis! & 1 & yes & no & Verbatim \\ & & & & \\ \verb!\Sexpr! & option plus 1 & sometimes & no & R-like \\ \verb!\RdOpts! & 1 & yes & no & Verbatim \\ \hline \end{tabular}\end{center} \end{table} \begin{table} \caption{Table of markup macros within sections taking LaTeX-like text. \label{tab:latex}} \begin{center}\begin{tabular}{lcccc} \hline Macro & Arguments & Section? & List? & Text type \\ \hline \verb!\acronym! & 1 & no & no & Latex-like \\ \verb!\bold! & 1 & no & no & Latex-like \\ \verb!\cite! & 1 & no & no & Latex-like \\ \verb!\command! & 1 & no & no & Latex-like \\ \verb!\dfn! & 1 & no & no & Latex-like \\ \verb!\dQuote! & 1 & no & no & Latex-like \\ \verb!\email! & 1 & no & no & Latex-like \\ \verb!\emph! & 1 & no & no & Latex-like \\ \verb!\file! & 1 & no & no & Latex-like \\ \verb!\item! & special & no & no & Latex-like \\ \verb!\linkS4class! & 1 & no & no & Latex-like \\ \verb!\pkg! & 1 & no & no & Latex-like \\ \verb!\sQuote! & 1 & no & no & Latex-like \\ \verb!\strong! & 1 & no & no & Latex-like \\ \verb!\var! & 1 & no & no & Latex-like \\ & & & & \\ \verb!\describe!& 1 & no & \verb!\item{}{}! & Latex-like \\ \verb!\enumerate!& 1 & no & \verb!\item! & Latex-like \\ \verb!\itemize! & 1 & no & \verb!\item! & Latex-like \\ & & & & \\ \verb!\enc! & 2 & no & no & Latex-like \\ \verb!\if! & 2 & no & no & Latex-like \\ \verb!\ifelse! & 3 & no & no & Latex-like \\ \verb!\method! & 2 & no & no & Latex-like \\ \verb!\S3method!& 2 & no & no & Latex-like \\ \verb!\S4method!& 2 & no & no & Latex-like \\ \verb!\tabular! & 2 & no & no & Latex-like \\ \verb!\subsection!& 2 & no & no & Latex-like \\ & & & & \\ \verb!\link! & option plus 1 & no & no & Latex-like \\ & & & & \\ \verb!\href! & 1+1 & no & no & Verbatim then Latex-like \\ \hline \end{tabular}\end{center} \end{table} \begin{table} \caption{Table of markup macros within sections taking no text, R-like text, or verbatim text. \label{tab:other}} \begin{center}\begin{tabular}{lcccc} \hline Macro & Arguments & Section? & List? & Text type \\ \hline \verb!\cr! & 0 & no & no & \\ \verb!\dots! & 0 & no & no & \\ \verb!\ldots! & 0 & no & no & \\ \verb!\R! & 0 & no & no & \\ \verb!\tab! & 0 & no & no & \\ & & & & \\ \verb!\code! & 1 & no & no & R-like \\ \verb!\dontshow!& 1 & no & no & R-like \\ \verb!\donttest!& 1 & no & no & R-like \\ \verb!\testonly!& 1 & no & no & R-like \\ & & & & \\ \verb!\dontrun! & 1 & no & no & Verbatim \\ \verb!\env! & 1 & no & no & Verbatim \\ \verb!\kbd! & 1 & no & no & Verbatim \\ \verb!\option! & 1 & no & no & Verbatim \\ \verb!\out! & 1 & no & no & Verbatim \\ \verb!\preformatted! & 1 & no & no & Verbatim \\ \verb!\samp! & 1 & no & no & Verbatim \\ \verb!\special! & 1 & no & no & Verbatim \\ \verb!\url! & 1 & no & no & Verbatim \\ \verb!\verb! & 1 & no & no & Verbatim \\ \verb!\deqn! & 1 or 2 & no & no & Verbatim \\ \verb!\eqn! & 1 or 2 & no & no & Verbatim \\ \verb!\newcommand! & 2 & both & no & Verbatim \\ \verb!\renewcommand! & 2 & both & no & Verbatim \\ \hline \end{tabular}\end{center} \end{table} \subsection{LaTeX-like text} LaTeX-like text in an Rd file is a stream of markup, text, and comments. Text in LaTeX-like mode consists of all characters other than markup and comments. The white space within text is kept as part of it. This is subject to change. Braces within LaTeX-like text must be escaped as \verb!\{! and \verb!\}! to be considered as part of the text. If not escaped, they group tokens in a LIST group. Brackets \verb![! and \verb!]! within LaTeX-like text only have syntactic relevance after the \verb!\link! or \verb!\Sexpr! macros, where they are used to delimit the optional argument. Quotes \verb!"!, \verb!'! and \verb!`! have no syntactic relevance within LaTeX-like text. \subsection{R-like text} \label{sec:Rlike} R-like text in an Rd file is a stream of markup, R code, and comments. The underlying mental model is that the markup could be replaced by suitable text and the R code would be parseable by \verb!parse()!, but \parseRd does not enforce this. There are two types of comments in R-like mode. As elsewhere in Rd files, Rd comments start with \verb!%!, and run to the end of the line. R-like comments start with \verb!#! and run to either the end of the line, or a brace that closes the block containing the R-like text. Unlike Rd comments, the Rd parser will return R comments as part of the text of the code; the Rd comment will be returned marked as a COMMENT token. Quoted strings (using \verb!"!, \verb!'! or \verb!`!) within R-like text follow R rules: the string delimiters must match and markup and comments within them are taken to be part of the strings and are not interpreted by the Rd parser. This includes braces and R-like comments, but there are two exceptions: \begin{enumerate} \item \verb!%! characters must be escaped even within strings, or they will be taken as Rd comments. \item The sequences \verb!\l! or \verb!\v! in a string will be taken to start a markup macro. This is intended to allow \verb!\link! or \verb!\var! to be used in a string (e.g. the common idiom \verb!\code{"\link{someclass}"}!). While \verb!\l! is not legal in an R string, \verb!\v! is a way to encode the rarely used ``vertical tab''. To insert a vertical tab into a string within an Rd file it is necessary to use \verb!\\v!. \label{link} \end{enumerate} Braces within R-like text outside of quoted strings must balance, or be escaped. Outside of a quoted string, in R-like text the escape character \verb!\! indicates the start of a markup macro. No markup macros are recognized within quoted strings except as noted in \ref{link} above. \subsection{Verbatim text} Verbatim text within an Rd file is a pure stream of text, uninterpreted by the parser, with the exceptions that braces must balance or be escaped, and \verb!%! comments are recognized, and backslashes that could be interpreted as escapes must themselves be escaped. No markup macros are recognized within verbatim text. \subsection{Exceptions to special character handling} \label{sec:nonescapes} The escape handling for \verb!\!, \verb!{!, \verb!}! and \verb!%! described in the preceding sections has the following exceptions: \begin{enumerate} \item In quoted strings in R-like code, braces are not counted and should not be escaped. For example, \verb!\code{ "{" }! is legal. \item In the first argument to \verb!\eqn! or \verb!\deqn! (whether in one- or two-argument form), no escapes or Rd comments are processed. Braces are counted unless escaped, so must be paired or escaped. This argument is passed verbatim to \LaTeX\ for processing, including all escapes. Thus \begin{verbatim} \deqn{ f(x) = \left\{ \begin{array}{ll} 0 & x < 0 \\ 1 & x \ge 0 \end{array} \right. }{ (non-Latex version) } \end{verbatim} can be used to display the equation $$ f(x) = \left\{ \begin{array}{ll} 0 & x < 0 \\ 1 & x \ge 0 \end{array} \right. $$ when rendered in \LaTeX, and a non-\LaTeX\ version in other formats. \end{enumerate} \section{Changes from R 2.8.x Rd format} \label{sec:changes} The following list describes syntax that was accepted in R 2.8.x but which is not accepted by the \parseRd parser. \begin{enumerate} \item The \verb!\code{}! macro was used for general verbatim text, similar to \verb!\samp{}!; now \verb!\verb{}! (or \verb!\kbd!, or \verb!\preformatted!) must be used when the text is not R-like. This mainly affects text containing the quote characters \verb!"!, \verb!'! or \verb!`!, as these will be taken to start quoted strings in R code. Escape sequences (e.g. \verb!\code{\a}!) should now be written as \verb!\verb{\a}!, as otherwise \verb!\a! would be taken to be a markup macro. Descriptions of languages other than R (e.g. examples of regular expressions) are often not syntactically valid R and may not be parsed properly in \verb!\code{}!. Note that currently \verb!\verb{}! is only supported in Rdoc version 2. \item Treating \verb!#ifdef ... #endif! and \verb!#ifndef ... #endif! as markup macros means that they must be wholly nested within other macros. For example, the construction \begin{verbatim} \title{ #ifdef unix Unix title} #endif #ifdef windows Windows title} #endif \end{verbatim} needs to be rewritten as \begin{verbatim} \title{ #ifdef unix Unix title #endif #ifdef windows Windows title #endif } \end{verbatim} \item R strings must be completely nested within markup macros. For example, \verb!\code{"my string}"! will now be taken to be an unterminated \verb!\code! macro, since the closing brace is within the string. \item Macros should be followed by a non-alphanumeric character, not just a numeric one. For example, \verb!1\dots10! now should be coded as \verb!1\dots{}10!. (In this case, it will be parsed properly even though \verb!\dots10! is not a legal macro, because the parser will attempt to drop the digits and reinterpret it as \verb!\dots! followed by the text \verb!10!. However, \verb!1a\dots10b! will be parsed as text \verb!1a! followed by the unknown macro \verb!\dots10b!.) There is an internal limit (currently about 25) on the number of digits that can be removed before the pushback buffer in the parser overflows. \item In processing in earlier versions, braces in R strings could be escaped, or left unescaped if they balanced within the section. Now if they are escaped within a string the escape character will be treated as part of the string, and since \verb!"\{"! is not legal R code, this can lead to problems. In order to create files compatible with both the new parser and the older system, braces should not be quoted within strings, and they should be balanced, perhaps by adding a matching brace in a comment. For example, \begin{center} \verb!\code{"\{"}! \end{center} could now be entered as \begin{center} \verb!\code{"{" # not "}"}! \end{center} The matching brace is not needed if the new parser is the only target. \end{enumerate} \section{The parsing function} The \parseRd function takes a text connection and produces a parsed version of it. The arguments to the \parseRd function are described in its man page. The general structure of the result is a list of section markup, with one entry per Rd section. Each section consists of a list of text and markup macros, with the text stored as one-element character vectors and the markup macros as lists. Single argument macros store the elements of the argument directly, with each element tagged as described below. Double argument macros are stored as two element lists; the first element is the first argument, the second element is the second argument. Neither one is tagged. The macros with either one or two arguments (currently \verb!\eqn! and \verb!\deqn!) store the two element form like other double argument macros, and store the one element form in an analogous one element list. The attributes of each element give information about the type of element. The following attributes are used: \begin{description} \item[class] The object returned by \parseRd is of class ``Rd". \item[\Rdtag] Each object in the list generated from markup macros has a tag corresponding to its type. If the item is a markup macro, the tag is the macro (e.g. \verb!\code! or \verb!#ifdef!). Non-macro tags include \verb!TEXT!, \verb!RCODE!, \verb!VERB!, \verb!COMMENT!, \verb!UNKNOWN! (an unrecognized macro), and \verb!LIST! (in Latex-like mode, a group of tokens in braces). \item[\Rdoption] Markup lists which had an optional parameter will have it stored in the \Rdoption attribute. \item[srcref and srcfile] Objects include source references. \end{description} \subsection{Example} The following example looks at the \parseRd man page. The \verb!tools:::RdTags! function extracts the tags from an ``Rd'' object. See the \verb!Rd2HTML! function in the \verb!tools! package for an extended example of working with the object. <<>>= library(tools) infile <- file.path(tools:::.R_top_srcdir_from_Rd(), "src/library/tools/man/parse_Rd.Rd") Rd <- parse_Rd(infile) print(tags <- tools:::RdTags(Rd)) Rd[[1]] Rd[[which(tags == "\\title")]] tools:::RdTags(Rd[[which(tags == "\\value")]]) @ <<>>= # Do a verbose parse Rd <- parse_Rd(infile, verbose=TRUE) @ \section{The \texttt{Sexpr} macro in R 2.10.0} \label{sec:Sexpr} R 2.10.0 will contain the new macros \verb!\Sexpr! and \verb!\RdOpts!. These are modelled after Sweave, and are intended to control R expressions in the Rd file. To the parser, the \verb!\Sexpr! macro is simply a macro taking an optional verbatim argument in square brackets, and a required R-like argument in the braces. For example, \verb!\!\verb!Sexpr{ x <- 1 }! or \verb!\Sexpr[stage=build]{ loadDatabase() }!. The \verb!\RdOpts! macro takes a single verbatim argument, intended to set defaults for the options in \verb!\Sexpr!. These two macros are modelled after Sweave, but the syntax and options are not identical. (We will expand on the differences below.) The R-like argument to \verb!\Sexpr! must be valid R code that can be executed; it cannot contain any expandable macros other than \verb!#ifdef/#ifndef/#endif!. Depending on the options, the code may be executed at package build time, package install time, or man page rendering time. Since package tarballs are built with the conditionals included, \verb!#ifdef/#ifndef/#endif! blocks cannot be included in code designed to be executed at build time. Rd comments using \verb!%! are ignored during execution. The options follow the same format as in Sweave, but different options are supported. Currently the allowed options and their defauls are: \begin{description} \item[eval=TRUE] Whether the R code should be evaluated. \item[echo=FALSE] Whether the R code should be echoed. If TRUE, a display will be given in a preformatted block. For example, \verb!\Sexpr[echo=TRUE]{ x <- 1 }! will be displayed as \begin{verbatim} > x <- 1 \end{verbatim} \item[keep.source=TRUE] Whether to keep the author's formatting when displaying the code, or throw it away and use a deparsed version. \item[results=text] How should the results be displayed? The possibilities are \begin{description} \item[text] Apply \verb!as.character()! to the result of the code, and insert it as a text element. \item[verbatim] Print the results of the code just as if it was executed at the console, and include the printed results verbatim. (Invisible results will not print.) \item[rd] The result is assumed to be a character vector containing markup to be passed to \verb!parse_Rd(fragment=TRUE)!, with the result inserted in place. This could be used to insert computed aliases, for instance. \item[hide] Insert no output. \end{description} \item[strip.white=TRUE] Remove leading and trailing white space from each line of output if \verb!strip.white=TRUE!. With \verb!strip.white=all!, also remove blank lines. \item[stage=install] Control when this macro is run. Possible values are \begin{description} \item[build] The macro is run when building a source tarball. \item[install] The macro is run when installing from source. \item[render] The macro is run when displaying the help page. \end{description} The \verb!#ifdef! conditionals are applied after the \texttt{build} macros but before the \texttt{install} macros. In some situations (e.g. installing directly from a source directory without a tarball, or building a binary package) the above descriptions may not be accurate, but authors should be able to rely on the sequence being \texttt{build}, \verb!#ifdef!, \texttt{install}, \texttt{render}, with all stages executed. Code is only run once in each stage, so a \verb!\Sexpr[results=rd]! macro can output an \verb!\Sexpr! macro designed for a later stage, but not for the current one or any earlier stage. \item[width, height, fig] These options are currently allowed but ignored. Eventually the intention is that they will allow insertion of graphics into the man page. \end{description} \subsection{Differences from Sweave} As of the current writing, these are the important differences from Sweave: \begin{enumerate} \item Our \verb!\Sexpr! macro takes options, and can give full displayed output. In Sweave the Noweb syntax or other macros are needed to process options. \item The Sweave \verb!\Sexpr! macro is roughly equivalent to our default \verb!\Sexpr[eval=TRUE,echo=FALSE,results=text]! but we will insert whatever \verb!as.character()! gives for the whole result, while Sweave only inserts the first element of a vector result. \item We use \verb!\RdOpts! rather than \verb!\SweaveOpts! here, to emphasize that we are not running Sweave, just something modelled on it. \item We run \verb!\Sexpr! macros in the three stage system, whereas Sweave does calculations in one pass. \end{enumerate} \section{New macros in R 2.12.0} The \verb!\href! macro takes one verbatim argument (the URL) and one LaTeX-like argument (the text to display to the user). The \verb!\newcommand! and \verb!\renewcommand! macros each take two verbatim arguments, allowing users to define their own macros. User defined macros all take verbatim arguments. See Writing R Extensions for a discussion of their use. \end{document}