bpo-25324: Move the description of tokenize tokens to token.rst. (#1911)

1 files changed, 30 insertions, 11 deletions

diff --git a/Doc/library/token.rst b/Doc/library/token.rst
index 4bf15d5..b7ca9db 100644
--- a/Doc/library/token.rst
+++ b/Doc/library/token.rst
@@ -101,18 +101,37 @@ The token constants are:
           AWAIT
           ASYNC
           ERRORTOKEN
-          COMMENT
-          NL
-          ENCODING
           N_TOKENS
           NT_OFFSET
 
-   .. versionchanged:: 3.5
-      Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with
-      Python 3.7, "async" and "await" will be tokenized as :data:`NAME`
-      tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed.
 
-   .. versionchanged:: 3.7
-      Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` to bring
-      the tokens in the C code in line with the tokens needed in
-      :mod:`tokenize` module. These tokens aren't used by the C tokenizer.
\ No newline at end of file
+The following token type values aren't used by the C tokenizer but are needed for
+the :mod:`tokenize` module.
+
+.. data:: COMMENT
+
+   Token value used to indicate a comment.
+
+
+.. data:: NL
+
+   Token value used to indicate a non-terminating newline.  The
+   :data:`NEWLINE` token indicates the end of a logical line of Python code;
+   ``NL`` tokens are generated when a logical line of code is continued over
+   multiple physical lines.
+
+
+.. data:: ENCODING
+
+   Token value that indicates the encoding used to decode the source bytes
+   into text. The first token returned by :func:`tokenize.tokenize` will
+   always be an ``ENCODING`` token.
+
+
+.. versionchanged:: 3.5
+   Added :data:`AWAIT` and :data:`ASYNC` tokens. Starting with
+   Python 3.7, "async" and "await" will be tokenized as :data:`NAME`
+   tokens, and :data:`AWAIT` and :data:`ASYNC` will be removed.
+
+.. versionchanged:: 3.7
+   Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens.