diff options
Diffstat (limited to 'Doc/reference/lexical_analysis.rst')
| -rw-r--r-- | Doc/reference/lexical_analysis.rst | 758 |
1 files changed, 758 insertions, 0 deletions
diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst new file mode 100644 index 0000000..35e92cf --- /dev/null +++ b/Doc/reference/lexical_analysis.rst @@ -0,0 +1,758 @@ + +.. _lexical: + +**************** +Lexical analysis +**************** + +.. index:: + single: lexical analysis + single: parser + single: token + +A Python program is read by a *parser*. Input to the parser is a stream of +*tokens*, generated by the *lexical analyzer*. This chapter describes how the +lexical analyzer breaks a file into tokens. + +Python uses the 7-bit ASCII character set for program text. + +.. versionadded:: 2.3 + An encoding declaration can be used to indicate that string literals and + comments use an encoding different from ASCII. + +For compatibility with older versions, Python only warns if it finds 8-bit +characters; those warnings should be corrected by either declaring an explicit +encoding, or using escape sequences if those bytes are binary data, instead of +characters. + +The run-time character set depends on the I/O devices connected to the program +but is generally a superset of ASCII. + +**Future compatibility note:** It may be tempting to assume that the character +set for 8-bit characters is ISO Latin-1 (an ASCII superset that covers most +western languages that use the Latin alphabet), but it is possible that in the +future Unicode text editors will become common. These generally use the UTF-8 +encoding, which is also an ASCII superset, but with very different use for the +characters with ordinals 128-255. While there is no consensus on this subject +yet, it is unwise to assume either Latin-1 or UTF-8, even though the current +implementation appears to favor Latin-1. This applies both to the source +character set and the run-time character set. + + +.. _line-structure: + +Line structure +============== + +.. index:: single: line structure + +A Python program is divided into a number of *logical lines*. + + +.. _logical: + +Logical lines +------------- + +.. index:: + single: logical line + single: physical line + single: line joining + single: NEWLINE token + +The end of a logical line is represented by the token NEWLINE. Statements +cannot cross logical line boundaries except where NEWLINE is allowed by the +syntax (e.g., between statements in compound statements). A logical line is +constructed from one or more *physical lines* by following the explicit or +implicit *line joining* rules. + + +.. _physical: + +Physical lines +-------------- + +A physical line is a sequence of characters terminated by an end-of-line +sequence. In source files, any of the standard platform line termination +sequences can be used - the Unix form using ASCII LF (linefeed), the Windows +form using the ASCII sequence CR LF (return followed by linefeed), or the +Macintosh form using the ASCII CR (return) character. All of these forms can be +used equally, regardless of platform. + +When embedding Python, source code strings should be passed to Python APIs using +the standard C conventions for newline characters (the ``\n`` character, +representing ASCII LF, is the line terminator). + + +.. _comments: + +Comments +-------- + +.. index:: + single: comment + single: hash character + +A comment starts with a hash character (``#``) that is not part of a string +literal, and ends at the end of the physical line. A comment signifies the end +of the logical line unless the implicit line joining rules are invoked. Comments +are ignored by the syntax; they are not tokens. + + +.. _encodings: + +Encoding declarations +--------------------- + +.. index:: + single: source character set + single: encodings + +If a comment in the first or second line of the Python script matches the +regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an +encoding declaration; the first group of this expression names the encoding of +the source code file. The recommended forms of this expression are :: + + # -*- coding: <encoding-name> -*- + +which is recognized also by GNU Emacs, and :: + + # vim:fileencoding=<encoding-name> + +which is recognized by Bram Moolenaar's VIM. In addition, if the first bytes of +the file are the UTF-8 byte-order mark (``'\xef\xbb\xbf'``), the declared file +encoding is UTF-8 (this is supported, among others, by Microsoft's +:program:`notepad`). + +If an encoding is declared, the encoding name must be recognized by Python. The +encoding is used for all lexical analysis, in particular to find the end of a +string, and to interpret the contents of Unicode literals. String literals are +converted to Unicode for syntactical analysis, then converted back to their +original encoding before interpretation starts. The encoding declaration must +appear on a line of its own. + +.. % XXX there should be a list of supported encodings. + + +.. _explicit-joining: + +Explicit line joining +--------------------- + +.. index:: + single: physical line + single: line joining + single: line continuation + single: backslash character + +Two or more physical lines may be joined into logical lines using backslash +characters (``\``), as follows: when a physical line ends in a backslash that is +not part of a string literal or comment, it is joined with the following forming +a single logical line, deleting the backslash and the following end-of-line +character. For example: + +.. % + +:: + + if 1900 < year < 2100 and 1 <= month <= 12 \ + and 1 <= day <= 31 and 0 <= hour < 24 \ + and 0 <= minute < 60 and 0 <= second < 60: # Looks like a valid date + return 1 + +A line ending in a backslash cannot carry a comment. A backslash does not +continue a comment. A backslash does not continue a token except for string +literals (i.e., tokens other than string literals cannot be split across +physical lines using a backslash). A backslash is illegal elsewhere on a line +outside a string literal. + + +.. _implicit-joining: + +Implicit line joining +--------------------- + +Expressions in parentheses, square brackets or curly braces can be split over +more than one physical line without using backslashes. For example:: + + month_names = ['Januari', 'Februari', 'Maart', # These are the + 'April', 'Mei', 'Juni', # Dutch names + 'Juli', 'Augustus', 'September', # for the months + 'Oktober', 'November', 'December'] # of the year + +Implicitly continued lines can carry comments. The indentation of the +continuation lines is not important. Blank continuation lines are allowed. +There is no NEWLINE token between implicit continuation lines. Implicitly +continued lines can also occur within triple-quoted strings (see below); in that +case they cannot carry comments. + + +.. _blank-lines: + +Blank lines +----------- + +.. index:: single: blank line + +A logical line that contains only spaces, tabs, formfeeds and possibly a +comment, is ignored (i.e., no NEWLINE token is generated). During interactive +input of statements, handling of a blank line may differ depending on the +implementation of the read-eval-print loop. In the standard implementation, an +entirely blank logical line (i.e. one containing not even whitespace or a +comment) terminates a multi-line statement. + + +.. _indentation: + +Indentation +----------- + +.. index:: + single: indentation + single: whitespace + single: leading whitespace + single: space + single: tab + single: grouping + single: statement grouping + +Leading whitespace (spaces and tabs) at the beginning of a logical line is used +to compute the indentation level of the line, which in turn is used to determine +the grouping of statements. + +First, tabs are replaced (from left to right) by one to eight spaces such that +the total number of characters up to and including the replacement is a multiple +of eight (this is intended to be the same rule as used by Unix). The total +number of spaces preceding the first non-blank character then determines the +line's indentation. Indentation cannot be split over multiple physical lines +using backslashes; the whitespace up to the first backslash determines the +indentation. + +**Cross-platform compatibility note:** because of the nature of text editors on +non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the +indentation in a single source file. It should also be noted that different +platforms may explicitly limit the maximum indentation level. + +A formfeed character may be present at the start of the line; it will be ignored +for the indentation calculations above. Formfeed characters occurring elsewhere +in the leading whitespace have an undefined effect (for instance, they may reset +the space count to zero). + +.. index:: + single: INDENT token + single: DEDENT token + +The indentation levels of consecutive lines are used to generate INDENT and +DEDENT tokens, using a stack, as follows. + +Before the first line of the file is read, a single zero is pushed on the stack; +this will never be popped off again. The numbers pushed on the stack will +always be strictly increasing from bottom to top. At the beginning of each +logical line, the line's indentation level is compared to the top of the stack. +If it is equal, nothing happens. If it is larger, it is pushed on the stack, and +one INDENT token is generated. If it is smaller, it *must* be one of the +numbers occurring on the stack; all numbers on the stack that are larger are +popped off, and for each number popped off a DEDENT token is generated. At the +end of the file, a DEDENT token is generated for each number remaining on the +stack that is larger than zero. + +Here is an example of a correctly (though confusingly) indented piece of Python +code:: + + def perm(l): + # Compute the list of all permutations of l + if len(l) <= 1: + return [l] + r = [] + for i in range(len(l)): + s = l[:i] + l[i+1:] + p = perm(s) + for x in p: + r.append(l[i:i+1] + x) + return r + +The following example shows various indentation errors:: + + def perm(l): # error: first line indented + for i in range(len(l)): # error: not indented + s = l[:i] + l[i+1:] + p = perm(l[:i] + l[i+1:]) # error: unexpected indent + for x in p: + r.append(l[i:i+1] + x) + return r # error: inconsistent dedent + +(Actually, the first three errors are detected by the parser; only the last +error is found by the lexical analyzer --- the indentation of ``return r`` does +not match a level popped off the stack.) + + +.. _whitespace: + +Whitespace between tokens +------------------------- + +Except at the beginning of a logical line or in string literals, the whitespace +characters space, tab and formfeed can be used interchangeably to separate +tokens. Whitespace is needed between two tokens only if their concatenation +could otherwise be interpreted as a different token (e.g., ab is one token, but +a b is two tokens). + + +.. _other-tokens: + +Other tokens +============ + +Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist: +*identifiers*, *keywords*, *literals*, *operators*, and *delimiters*. Whitespace +characters (other than line terminators, discussed earlier) are not tokens, but +serve to delimit tokens. Where ambiguity exists, a token comprises the longest +possible string that forms a legal token, when read from left to right. + + +.. _identifiers: + +Identifiers and keywords +======================== + +.. index:: + single: identifier + single: name + +Identifiers (also referred to as *names*) are described by the following lexical +definitions: + +.. productionlist:: + identifier: (`letter`|"_") (`letter` | `digit` | "_")* + letter: `lowercase` | `uppercase` + lowercase: "a"..."z" + uppercase: "A"..."Z" + digit: "0"..."9" + +Identifiers are unlimited in length. Case is significant. + + +.. _keywords: + +Keywords +-------- + +.. index:: + single: keyword + single: reserved word + +The following identifiers are used as reserved words, or *keywords* of the +language, and cannot be used as ordinary identifiers. They must be spelled +exactly as written here:: + + and def for is raise + as del from lambda return + assert elif global not try + break else if or while + class except import pass with + continue finally in print yield + +.. versionchanged:: 2.4 + :const:`None` became a constant and is now recognized by the compiler as a name + for the built-in object :const:`None`. Although it is not a keyword, you cannot + assign a different object to it. + +.. versionchanged:: 2.5 + Both :keyword:`as` and :keyword:`with` are only recognized when the + ``with_statement`` future feature has been enabled. It will always be enabled in + Python 2.6. See section :ref:`with` for details. Note that using :keyword:`as` + and :keyword:`with` as identifiers will always issue a warning, even when the + ``with_statement`` future directive is not in effect. + + +.. _id-classes: + +Reserved classes of identifiers +------------------------------- + +Certain classes of identifiers (besides keywords) have special meanings. These +classes are identified by the patterns of leading and trailing underscore +characters: + +``_*`` + Not imported by ``from module import *``. The special identifier ``_`` is used + in the interactive interpreter to store the result of the last evaluation; it is + stored in the :mod:`__builtin__` module. When not in interactive mode, ``_`` + has no special meaning and is not defined. See section :ref:`import`. + + .. note:: + + The name ``_`` is often used in conjunction with internationalization; + refer to the documentation for the :mod:`gettext` module for more + information on this convention. + +``__*__`` + System-defined names. These names are defined by the interpreter and its + implementation (including the standard library); applications should not expect + to define additional names using this convention. The set of names of this + class defined by Python may be extended in future versions. See section + :ref:`specialnames`. + +``__*`` + Class-private names. Names in this category, when used within the context of a + class definition, are re-written to use a mangled form to help avoid name + clashes between "private" attributes of base and derived classes. See section + :ref:`atom-identifiers`. + + +.. _literals: + +Literals +======== + +.. index:: + single: literal + single: constant + +Literals are notations for constant values of some built-in types. + + +.. _strings: + +String literals +--------------- + +.. index:: single: string literal + +String literals are described by the following lexical definitions: + +.. index:: single: ASCII@ASCII + +.. productionlist:: + stringliteral: [`stringprefix`](`shortstring` | `longstring`) + stringprefix: "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR" + shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' + longstring: ""'" `longstringitem`* ""'" + : | '"""' `longstringitem`* '"""' + shortstringitem: `shortstringchar` | `escapeseq` + longstringitem: `longstringchar` | `escapeseq` + shortstringchar: <any source character except "\" or newline or the quote> + longstringchar: <any source character except "\"> + escapeseq: "\" <any ASCII character> + +One syntactic restriction not indicated by these productions is that whitespace +is not allowed between the :token:`stringprefix` and the rest of the string +literal. The source character set is defined by the encoding declaration; it is +ASCII if no encoding declaration is given in the source file; see section +:ref:`encodings`. + +.. index:: + single: triple-quoted string + single: Unicode Consortium + single: string; Unicode + single: raw string + +In plain English: String literals can be enclosed in matching single quotes +(``'``) or double quotes (``"``). They can also be enclosed in matching groups +of three single or double quotes (these are generally referred to as +*triple-quoted strings*). The backslash (``\``) character is used to escape +characters that otherwise have a special meaning, such as newline, backslash +itself, or the quote character. String literals may optionally be prefixed with +a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and use +different rules for interpreting backslash escape sequences. A prefix of +``'u'`` or ``'U'`` makes the string a Unicode string. Unicode strings use the +Unicode character set as defined by the Unicode Consortium and ISO 10646. Some +additional escape sequences, described below, are available in Unicode strings. +The two prefix characters may be combined; in this case, ``'u'`` must appear +before ``'r'``. + +In triple-quoted strings, unescaped newlines and quotes are allowed (and are +retained), except that three unescaped quotes in a row terminate the string. (A +"quote" is the character used to open the string, i.e. either ``'`` or ``"``.) + +.. index:: + single: physical line + single: escape sequence + single: Standard C + single: C + +Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in strings are +interpreted according to rules similar to those used by Standard C. The +recognized escape sequences are: + ++-----------------+---------------------------------+-------+ +| Escape Sequence | Meaning | Notes | ++=================+=================================+=======+ +| ``\newline`` | Ignored | | ++-----------------+---------------------------------+-------+ +| ``\\`` | Backslash (``\``) | | ++-----------------+---------------------------------+-------+ +| ``\'`` | Single quote (``'``) | | ++-----------------+---------------------------------+-------+ +| ``\"`` | Double quote (``"``) | | ++-----------------+---------------------------------+-------+ +| ``\a`` | ASCII Bell (BEL) | | ++-----------------+---------------------------------+-------+ +| ``\b`` | ASCII Backspace (BS) | | ++-----------------+---------------------------------+-------+ +| ``\f`` | ASCII Formfeed (FF) | | ++-----------------+---------------------------------+-------+ +| ``\n`` | ASCII Linefeed (LF) | | ++-----------------+---------------------------------+-------+ +| ``\N{name}`` | Character named *name* in the | | +| | Unicode database (Unicode only) | | ++-----------------+---------------------------------+-------+ +| ``\r`` | ASCII Carriage Return (CR) | | ++-----------------+---------------------------------+-------+ +| ``\t`` | ASCII Horizontal Tab (TAB) | | ++-----------------+---------------------------------+-------+ +| ``\uxxxx`` | Character with 16-bit hex value | \(1) | +| | *xxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ +| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(2) | +| | *xxxxxxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ +| ``\v`` | ASCII Vertical Tab (VT) | | ++-----------------+---------------------------------+-------+ +| ``\ooo`` | Character with octal value | (3,5) | +| | *ooo* | | ++-----------------+---------------------------------+-------+ +| ``\xhh`` | Character with hex value *hh* | (4,5) | ++-----------------+---------------------------------+-------+ + +.. index:: single: ASCII@ASCII + +Notes: + +(1) + Individual code units which form parts of a surrogate pair can be encoded using + this escape sequence. + +(2) + Any Unicode character can be encoded this way, but characters outside the Basic + Multilingual Plane (BMP) will be encoded using a surrogate pair if Python is + compiled to use 16-bit code units (the default). Individual code units which + form parts of a surrogate pair can be encoded using this escape sequence. + +(3) + As in Standard C, up to three octal digits are accepted. + +(4) + Unlike in Standard C, at most two hex digits are accepted. + +(5) + In a string literal, hexadecimal and octal escapes denote the byte with the + given value; it is not necessary that the byte encodes a character in the source + character set. In a Unicode literal, these escapes denote a Unicode character + with the given value. + +.. index:: single: unrecognized escape sequence + +Unlike Standard C, all unrecognized escape sequences are left in the string +unchanged, i.e., *the backslash is left in the string*. (This behavior is +useful when debugging: if an escape sequence is mistyped, the resulting output +is more easily recognized as broken.) It is also important to note that the +escape sequences marked as "(Unicode only)" in the table above fall into the +category of unrecognized escapes for non-Unicode string literals. + +When an ``'r'`` or ``'R'`` prefix is present, a character following a backslash +is included in the string without change, and *all backslashes are left in the +string*. For example, the string literal ``r"\n"`` consists of two characters: +a backslash and a lowercase ``'n'``. String quotes can be escaped with a +backslash, but the backslash remains in the string; for example, ``r"\""`` is a +valid string literal consisting of two characters: a backslash and a double +quote; ``r"\"`` is not a valid string literal (even a raw string cannot end in +an odd number of backslashes). Specifically, *a raw string cannot end in a +single backslash* (since the backslash would escape the following quote +character). Note also that a single backslash followed by a newline is +interpreted as those two characters as part of the string, *not* as a line +continuation. + +When an ``'r'`` or ``'R'`` prefix is used in conjunction with a ``'u'`` or +``'U'`` prefix, then the ``\uXXXX`` and ``\UXXXXXXXX`` escape sequences are +processed while *all other backslashes are left in the string*. For example, +the string literal ``ur"\u0062\n"`` consists of three Unicode characters: 'LATIN +SMALL LETTER B', 'REVERSE SOLIDUS', and 'LATIN SMALL LETTER N'. Backslashes can +be escaped with a preceding backslash; however, both remain in the string. As a +result, ``\uXXXX`` escape sequences are only recognized when there are an odd +number of backslashes. + + +.. _string-catenation: + +String literal concatenation +---------------------------- + +Multiple adjacent string literals (delimited by whitespace), possibly using +different quoting conventions, are allowed, and their meaning is the same as +their concatenation. Thus, ``"hello" 'world'`` is equivalent to +``"helloworld"``. This feature can be used to reduce the number of backslashes +needed, to split long strings conveniently across long lines, or even to add +comments to parts of strings, for example:: + + re.compile("[A-Za-z_]" # letter or underscore + "[A-Za-z0-9_]*" # letter, digit or underscore + ) + +Note that this feature is defined at the syntactical level, but implemented at +compile time. The '+' operator must be used to concatenate string expressions +at run time. Also note that literal concatenation can use different quoting +styles for each component (even mixing raw strings and triple quoted strings). + + +.. _numbers: + +Numeric literals +---------------- + +.. index:: + single: number + single: numeric literal + single: integer literal + single: plain integer literal + single: long integer literal + single: floating point literal + single: hexadecimal literal + single: octal literal + single: binary literal + single: decimal literal + single: imaginary literal + single: complex; literal + +There are four types of numeric literals: plain integers, long integers, +floating point numbers, and imaginary numbers. There are no complex literals +(complex numbers can be formed by adding a real number and an imaginary number). + +Note that numeric literals do not include a sign; a phrase like ``-1`` is +actually an expression composed of the unary operator '``-``' and the literal +``1``. + + +.. _integers: + +Integer literals +---------------- + +Integer literals are described by the following lexical definitions: + +.. productionlist:: + integer: `decimalinteger` | `octinteger` | `hexinteger` + decimalinteger: `nonzerodigit` `digit`* | "0"+ + octinteger: "0" ("o" | "O") `octdigit`+ + hexinteger: "0" ("x" | "X") `hexdigit`+ + bininteger: "0" ("b" | "B") `bindigit`+ + nonzerodigit: "1"..."9" + octdigit: "0"..."7" + hexdigit: `digit` | "a"..."f" | "A"..."F" + bindigit: "0"..."1" + +Plain integer literals that are above the largest representable plain integer +(e.g., 2147483647 when using 32-bit arithmetic) are accepted as if they were +long integers instead. [#]_ There is no limit for long integer literals apart +from what can be stored in available memory. + +Note that leading zeros in a non-zero decimal number are not allowed. This is +for disambiguation with C-style octal literals, which Python used before version +3.0. + +Some examples of integer literals:: + + 7 2147483647 0o177 0b100110111 + 3 79228162514264337593543950336 0o377 0x100000000 + 79228162514264337593543950336 0xdeadbeef + + +.. _floating: + +Floating point literals +----------------------- + +Floating point literals are described by the following lexical definitions: + +.. productionlist:: + floatnumber: `pointfloat` | `exponentfloat` + pointfloat: [`intpart`] `fraction` | `intpart` "." + exponentfloat: (`intpart` | `pointfloat`) `exponent` + intpart: `digit`+ + fraction: "." `digit`+ + exponent: ("e" | "E") ["+" | "-"] `digit`+ + +Note that the integer and exponent parts are always interpreted using radix 10. +For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The +allowed range of floating point literals is implementation-dependent. Some +examples of floating point literals:: + + 3.14 10. .001 1e100 3.14e-10 0e0 + +Note that numeric literals do not include a sign; a phrase like ``-1`` is +actually an expression composed of the unary operator ``-`` and the literal +``1``. + + +.. _imaginary: + +Imaginary literals +------------------ + +Imaginary literals are described by the following lexical definitions: + +.. productionlist:: + imagnumber: (`floatnumber` | `intpart`) ("j" | "J") + +An imaginary literal yields a complex number with a real part of 0.0. Complex +numbers are represented as a pair of floating point numbers and have the same +restrictions on their range. To create a complex number with a nonzero real +part, add a floating point number to it, e.g., ``(3+4j)``. Some examples of +imaginary literals:: + + 3.14j 10.j 10j .001j 1e100j 3.14e-10j + + +.. _operators: + +Operators +========= + +.. index:: single: operators + +The following tokens are operators:: + + + - * ** / // % + << >> & | ^ ~ + < > <= >= == != + + +.. _delimiters: + +Delimiters +========== + +.. index:: single: delimiters + +The following tokens serve as delimiters in the grammar:: + + ( ) [ ] { } @ + , : . ` = ; + += -= *= /= //= %= + &= |= ^= >>= <<= **= + +The period can also occur in floating-point and imaginary literals. A sequence +of three periods has a special meaning as an ellipsis in slices. The second half +of the list, the augmented assignment operators, serve lexically as delimiters, +but also perform an operation. + +The following printing ASCII characters have special meaning as part of other +tokens or are otherwise significant to the lexical analyzer:: + + ' " # \ + +.. index:: single: ASCII@ASCII + +The following printing ASCII characters are not used in Python. Their +occurrence outside string literals and comments is an unconditional error:: + + $ ? + +.. rubric:: Footnotes + +.. [#] In versions of Python prior to 2.4, octal and hexadecimal literals in the range + just above the largest representable plain integer but below the largest + unsigned 32-bit number (on a machine using 32-bit arithmetic), 4294967296, were + taken as the negative plain integer obtained by subtracting 4294967296 from + their unsigned value. + |