diff options
Diffstat (limited to 'Doc/lib/libtokenize.tex')
| -rw-r--r-- | Doc/lib/libtokenize.tex | 5 |
1 files changed, 4 insertions, 1 deletions
diff --git a/Doc/lib/libtokenize.tex b/Doc/lib/libtokenize.tex index 82d2e68..3c5e8ef 100644 --- a/Doc/lib/libtokenize.tex +++ b/Doc/lib/libtokenize.tex @@ -36,9 +36,12 @@ The scanner is exposed by a single function: All constants from the \refmodule{token} module are also exported from -\module{tokenize}, as is one additional token type value that might be +\module{tokenize}, as are two additional token type values that might be passed to the \var{tokeneater} function by \function{tokenize()}: \begin{datadesc}{COMMENT} Token value used to indicate a comment. \end{datadesc} +\begin{datadesc}{NL} + Token value used to indicate a newline. +\end{datadesc} |