diff options
Diffstat (limited to 'Doc/reference/lexical_analysis.rst')
| -rw-r--r-- | Doc/reference/lexical_analysis.rst | 720 |
1 files changed, 282 insertions, 438 deletions
diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index c0e13b5..435dfc5 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -5,16 +5,38 @@ Lexical analysis **************** -.. index:: lexical analysis, parser, token +.. index:: + single: lexical analysis + single: parser + single: token A Python program is read by a *parser*. Input to the parser is a stream of *tokens*, generated by the *lexical analyzer*. This chapter describes how the lexical analyzer breaks a file into tokens. -Python reads program text as Unicode code points; the encoding of a source file -can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120` -for details. If the source file cannot be decoded, a :exc:`SyntaxError` is -raised. +Python uses the 7-bit ASCII character set for program text. + +.. versionadded:: 2.3 + An encoding declaration can be used to indicate that string literals and + comments use an encoding different from ASCII. + +For compatibility with older versions, Python only warns if it finds 8-bit +characters; those warnings should be corrected by either declaring an explicit +encoding, or using escape sequences if those bytes are binary data, instead of +characters. + +The run-time character set depends on the I/O devices connected to the program +but is generally a superset of ASCII. + +**Future compatibility note:** It may be tempting to assume that the character +set for 8-bit characters is ISO Latin-1 (an ASCII superset that covers most +western languages that use the Latin alphabet), but it is possible that in the +future Unicode text editors will become common. These generally use the UTF-8 +encoding, which is also an ASCII superset, but with very different use for the +characters with ordinals 128-255. While there is no consensus on this subject +yet, it is unwise to assume either Latin-1 or UTF-8, even though the current +implementation appears to favor Latin-1. This applies both to the source +character set and the run-time character set. .. _line-structure: @@ -22,17 +44,21 @@ raised. Line structure ============== -.. index:: line structure +.. index:: single: line structure A Python program is divided into a number of *logical lines*. -.. _logical-lines: +.. _logical: Logical lines ------------- -.. index:: logical line, physical line, line joining, NEWLINE token +.. index:: + single: logical line + single: physical line + single: line joining + single: NEWLINE token The end of a logical line is represented by the token NEWLINE. Statements cannot cross logical line boundaries except where NEWLINE is allowed by the @@ -41,7 +67,7 @@ constructed from one or more *physical lines* by following the explicit or implicit *line joining* rules. -.. _physical-lines: +.. _physical: Physical lines -------------- @@ -64,13 +90,14 @@ representing ASCII LF, is the line terminator). Comments -------- -.. index:: comment, hash character - single: # (hash); comment +.. index:: + single: comment + single: hash character A comment starts with a hash character (``#``) that is not part of a string literal, and ends at the end of the physical line. A comment signifies the end of the logical line unless the implicit line joining rules are invoked. Comments -are ignored by the syntax. +are ignored by the syntax; they are not tokens. .. _encodings: @@ -79,7 +106,6 @@ Encoding declarations --------------------- .. index:: source character set, encoding declarations (source file) - single: # (hash); source encoding declaration If a comment in the first or second line of the Python script matches the regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an @@ -94,16 +120,16 @@ which is recognized also by GNU Emacs, and :: # vim:fileencoding=<encoding-name> -which is recognized by Bram Moolenaar's VIM. - -If no encoding declaration is found, the default encoding is UTF-8. In -addition, if the first bytes of the file are the UTF-8 byte-order mark -(``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, -among others, by Microsoft's :program:`notepad`). +which is recognized by Bram Moolenaar's VIM. In addition, if the first bytes of +the file are the UTF-8 byte-order mark (``'\xef\xbb\xbf'``), the declared file +encoding is UTF-8 (this is supported, among others, by Microsoft's +:program:`notepad`). If an encoding is declared, the encoding name must be recognized by Python. The -encoding is used for all lexical analysis, including string literals, comments -and identifiers. +encoding is used for all lexical analysis, in particular to find the end of a +string, and to interpret the contents of Unicode literals. String literals are +converted to Unicode for syntactical analysis, then converted back to their +original encoding before interpretation starts. .. XXX there should be a list of supported encodings. @@ -113,7 +139,11 @@ and identifiers. Explicit line joining --------------------- -.. index:: physical line, line joining, line continuation, backslash character +.. index:: + single: physical line + single: line joining + single: line continuation + single: backslash character Two or more physical lines may be joined into logical lines using backslash characters (``\``), as follows: when a physical line ends in a backslash that is @@ -163,9 +193,9 @@ Blank lines A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored (i.e., no NEWLINE token is generated). During interactive input of statements, handling of a blank line may differ depending on the -implementation of the read-eval-print loop. In the standard interactive -interpreter, an entirely blank logical line (i.e. one containing not even -whitespace or a comment) terminates a multi-line statement. +implementation of the read-eval-print loop. In the standard implementation, an +entirely blank logical line (i.e. one containing not even whitespace or a +comment) terminates a multi-line statement. .. _indentation: @@ -173,24 +203,27 @@ whitespace or a comment) terminates a multi-line statement. Indentation ----------- -.. index:: indentation, leading whitespace, space, tab, grouping, statement grouping +.. index:: + single: indentation + single: whitespace + single: leading whitespace + single: space + single: tab + single: grouping + single: statement grouping Leading whitespace (spaces and tabs) at the beginning of a logical line is used to compute the indentation level of the line, which in turn is used to determine the grouping of statements. -Tabs are replaced (from left to right) by one to eight spaces such that the -total number of characters up to and including the replacement is a multiple of -eight (this is intended to be the same rule as used by Unix). The total number -of spaces preceding the first non-blank character then determines the line's -indentation. Indentation cannot be split over multiple physical lines using -backslashes; the whitespace up to the first backslash determines the +First, tabs are replaced (from left to right) by one to eight spaces such that +the total number of characters up to and including the replacement is a multiple +of eight (this is intended to be the same rule as used by Unix). The total +number of spaces preceding the first non-blank character then determines the +line's indentation. Indentation cannot be split over multiple physical lines +using backslashes; the whitespace up to the first backslash determines the indentation. -Indentation is rejected as inconsistent if a source file mixes tabs and spaces -in a way that makes the meaning dependent on the worth of a tab in spaces; a -:exc:`TabError` is raised in that case. - **Cross-platform compatibility note:** because of the nature of text editors on non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the indentation in a single source file. It should also be noted that different @@ -201,7 +234,9 @@ for the indentation calculations above. Formfeed characters occurring elsewhere in the leading whitespace have an undefined effect (for instance, they may reset the space count to zero). -.. index:: INDENT token, DEDENT token +.. index:: + single: INDENT token + single: DEDENT token The indentation levels of consecutive lines are used to generate INDENT and DEDENT tokens, using a stack, as follows. @@ -276,57 +311,22 @@ possible string that forms a legal token, when read from left to right. Identifiers and keywords ======================== -.. index:: identifier, name +.. index:: + single: identifier + single: name Identifiers (also referred to as *names*) are described by the following lexical -definitions. - -The syntax of identifiers in Python is based on the Unicode standard annex -UAX-31, with elaboration and changes as defined below; see also :pep:`3131` for -further details. +definitions: -Within the ASCII range (U+0001..U+007F), the valid characters for identifiers -are the same as in Python 2.x: the uppercase and lowercase letters ``A`` through -``Z``, the underscore ``_`` and, except for the first character, the digits -``0`` through ``9``. - -Python 3.0 introduces additional characters from outside the ASCII range (see -:pep:`3131`). For these characters, the classification uses the version of the -Unicode Character Database as included in the :mod:`unicodedata` module. +.. productionlist:: + identifier: (`letter`|"_") (`letter` | `digit` | "_")* + letter: `lowercase` | `uppercase` + lowercase: "a"..."z" + uppercase: "A"..."Z" + digit: "0"..."9" Identifiers are unlimited in length. Case is significant. -.. productionlist:: - identifier: `xid_start` `xid_continue`* - id_start: <all characters in general categories Lu, Ll, Lt, Lm, Lo, Nl, the underscore, and characters with the Other_ID_Start property> - id_continue: <all characters in `id_start`, plus characters in the categories Mn, Mc, Nd, Pc and others with the Other_ID_Continue property> - xid_start: <all characters in `id_start` whose NFKC normalization is in "id_start xid_continue*"> - xid_continue: <all characters in `id_continue` whose NFKC normalization is in "id_continue*"> - -The Unicode category codes mentioned above stand for: - -* *Lu* - uppercase letters -* *Ll* - lowercase letters -* *Lt* - titlecase letters -* *Lm* - modifier letters -* *Lo* - other letters -* *Nl* - letter numbers -* *Mn* - nonspacing marks -* *Mc* - spacing combining marks -* *Nd* - decimal numbers -* *Pc* - connector punctuations -* *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/12.1.0/ucd/PropList.txt>`_ to support backwards - compatibility -* *Other_ID_Continue* - likewise - -All identifiers are converted into the normal form NFKC while parsing; comparison -of identifiers is based on NFKC. - -A non-normative HTML file listing all valid identifier characters for Unicode -4.1 can be found at -https://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html. - .. _keywords: @@ -343,17 +343,27 @@ exactly as written here: .. sourcecode:: text - False await else import pass - None break except in raise - True class finally is return - and continue for lambda try - as def from nonlocal while - assert del global not with - async elif if or yield + and del from not while + as elif global or with + assert else if pass yield + break except import print + class exec in raise + continue finally is return + def for lambda try + +.. versionchanged:: 2.4 + :const:`None` became a constant and is now recognized by the compiler as a name + for the built-in object :const:`None`. Although it is not a keyword, you cannot + assign a different object to it. + +.. versionchanged:: 2.5 + Using :keyword:`as` and :keyword:`with` as identifiers triggers a warning. To + use them as keywords, enable the ``with_statement`` future feature . + +.. versionchanged:: 2.6 + :keyword:`as` and :keyword:`with` are full keywords. + -.. index:: - single: _, identifiers - single: __, identifiers .. _id-classes: Reserved classes of identifiers @@ -366,7 +376,7 @@ characters: ``_*`` Not imported by ``from module import *``. The special identifier ``_`` is used in the interactive interpreter to store the result of the last evaluation; it is - stored in the :mod:`builtins` module. When not in interactive mode, ``_`` + stored in the :mod:`__builtin__` module. When not in interactive mode, ``_`` has no special meaning and is not defined. See section :ref:`import`. .. note:: @@ -395,128 +405,83 @@ characters: Literals ======== -.. index:: literal, constant +.. index:: + single: literal + single: constant Literals are notations for constant values of some built-in types. -.. index:: string literal, bytes literal, ASCII - single: ' (single quote); string literal - single: " (double quote); string literal - single: u'; string literal - single: u"; string literal .. _strings: -String and Bytes literals -------------------------- +String literals +--------------- + +.. index:: single: string literal String literals are described by the following lexical definitions: +.. index:: single: ASCII@ASCII + .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) - stringprefix: "r" | "u" | "R" | "U" | "f" | "F" - : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" + stringprefix: "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR" + : | "b" | "B" | "br" | "Br" | "bR" | "BR" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' - longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' - shortstringitem: `shortstringchar` | `stringescapeseq` - longstringitem: `longstringchar` | `stringescapeseq` + longstring: "'''" `longstringitem`* "'''" + : | '"""' `longstringitem`* '"""' + shortstringitem: `shortstringchar` | `escapeseq` + longstringitem: `longstringchar` | `escapeseq` shortstringchar: <any source character except "\" or newline or the quote> longstringchar: <any source character except "\"> - stringescapeseq: "\" <any source character> - -.. productionlist:: - bytesliteral: `bytesprefix`(`shortbytes` | `longbytes`) - bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB" - shortbytes: "'" `shortbytesitem`* "'" | '"' `shortbytesitem`* '"' - longbytes: "'''" `longbytesitem`* "'''" | '"""' `longbytesitem`* '"""' - shortbytesitem: `shortbyteschar` | `bytesescapeseq` - longbytesitem: `longbyteschar` | `bytesescapeseq` - shortbyteschar: <any ASCII character except "\" or newline or the quote> - longbyteschar: <any ASCII character except "\"> - bytesescapeseq: "\" <any ASCII character> + escapeseq: "\" <any ASCII character> One syntactic restriction not indicated by these productions is that whitespace -is not allowed between the :token:`stringprefix` or :token:`bytesprefix` and the -rest of the literal. The source character set is defined by the encoding -declaration; it is UTF-8 if no encoding declaration is given in the source file; -see section :ref:`encodings`. +is not allowed between the :token:`stringprefix` and the rest of the string +literal. The source character set is defined by the encoding declaration; it is +ASCII if no encoding declaration is given in the source file; see section +:ref:`encodings`. -.. index:: triple-quoted string, Unicode Consortium, raw string - single: """; string literal - single: '''; string literal +.. index:: + single: triple-quoted string + single: Unicode Consortium + single: string; Unicode + single: raw string -In plain English: Both types of literals can be enclosed in matching single quotes +In plain English: String literals can be enclosed in matching single quotes (``'``) or double quotes (``"``). They can also be enclosed in matching groups of three single or double quotes (these are generally referred to as *triple-quoted strings*). The backslash (``\``) character is used to escape characters that otherwise have a special meaning, such as newline, backslash -itself, or the quote character. - -.. index:: - single: b'; bytes literal - single: b"; bytes literal - -Bytes literals are always prefixed with ``'b'`` or ``'B'``; they produce an -instance of the :class:`bytes` type instead of the :class:`str` type. They -may only contain ASCII characters; bytes with a numeric value of 128 or greater -must be expressed with escapes. +itself, or the quote character. String literals may optionally be prefixed with +a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and use +different rules for interpreting backslash escape sequences. A prefix of +``'u'`` or ``'U'`` makes the string a Unicode string. Unicode strings use the +Unicode character set as defined by the Unicode Consortium and ISO 10646. Some +additional escape sequences, described below, are available in Unicode strings. +A prefix of ``'b'`` or ``'B'`` is ignored in Python 2; it indicates that the +literal should become a bytes literal in Python 3 (e.g. when code is +automatically converted with 2to3). A ``'u'`` or ``'b'`` prefix may be followed +by an ``'r'`` prefix. + +In triple-quoted strings, unescaped newlines and quotes are allowed (and are +retained), except that three unescaped quotes in a row terminate the string. (A +"quote" is the character used to open the string, i.e. either ``'`` or ``"``.) .. index:: - single: r'; raw string literal - single: r"; raw string literal - -Both string and bytes literals may optionally be prefixed with a letter ``'r'`` -or ``'R'``; such strings are called :dfn:`raw strings` and treat backslashes as -literal characters. As a result, in string literals, ``'\U'`` and ``'\u'`` -escapes in raw strings are not treated specially. Given that Python 2.x's raw -unicode literals behave differently than Python 3.x's the ``'ur'`` syntax -is not supported. - -.. versionadded:: 3.3 - The ``'rb'`` prefix of raw bytes literals has been added as a synonym - of ``'br'``. + single: physical line + single: escape sequence + single: Standard C + single: C -.. versionadded:: 3.3 - Support for the unicode legacy literal (``u'value'``) was reintroduced - to simplify the maintenance of dual Python 2.x and 3.x codebases. - See :pep:`414` for more information. - -.. index:: - single: f'; formatted string literal - single: f"; formatted string literal - -A string literal with ``'f'`` or ``'F'`` in its prefix is a -:dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be -combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw -formatted strings are possible, but formatted bytes literals are not. - -In triple-quoted literals, unescaped newlines and quotes are allowed (and are -retained), except that three unescaped quotes in a row terminate the literal. (A -"quote" is the character used to open the literal, i.e. either ``'`` or ``"``.) - -.. index:: physical line, escape sequence, Standard C, C - single: \ (backslash); escape sequence - single: \\; escape sequence - single: \a; escape sequence - single: \b; escape sequence - single: \f; escape sequence - single: \n; escape sequence - single: \r; escape sequence - single: \t; escape sequence - single: \v; escape sequence - single: \x; escape sequence - single: \N; escape sequence - single: \u; escape sequence - single: \U; escape sequence - -Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in string and -bytes literals are interpreted according to rules similar to those used by -Standard C. The recognized escape sequences are: +Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in strings are +interpreted according to rules similar to those used by Standard C. The +recognized escape sequences are: +-----------------+---------------------------------+-------+ | Escape Sequence | Meaning | Notes | +=================+=================================+=======+ -| ``\newline`` | Backslash and newline ignored | | +| ``\newline`` | Ignored | | +-----------------+---------------------------------+-------+ | ``\\`` | Backslash (``\``) | | +-----------------+---------------------------------+-------+ @@ -532,90 +497,92 @@ Standard C. The recognized escape sequences are: +-----------------+---------------------------------+-------+ | ``\n`` | ASCII Linefeed (LF) | | +-----------------+---------------------------------+-------+ +| ``\N{name}`` | Character named *name* in the | | +| | Unicode database (Unicode only) | | ++-----------------+---------------------------------+-------+ | ``\r`` | ASCII Carriage Return (CR) | | +-----------------+---------------------------------+-------+ | ``\t`` | ASCII Horizontal Tab (TAB) | | +-----------------+---------------------------------+-------+ +| ``\uxxxx`` | Character with 16-bit hex value | \(1) | +| | *xxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ +| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(2) | +| | *xxxxxxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ | ``\v`` | ASCII Vertical Tab (VT) | | +-----------------+---------------------------------+-------+ -| ``\ooo`` | Character with octal value | (1,3) | +| ``\ooo`` | Character with octal value | (3,5) | | | *ooo* | | +-----------------+---------------------------------+-------+ -| ``\xhh`` | Character with hex value *hh* | (2,3) | +| ``\xhh`` | Character with hex value *hh* | (4,5) | +-----------------+---------------------------------+-------+ -Escape sequences only recognized in string literals are: - -+-----------------+---------------------------------+-------+ -| Escape Sequence | Meaning | Notes | -+=================+=================================+=======+ -| ``\N{name}`` | Character named *name* in the | \(4) | -| | Unicode database | | -+-----------------+---------------------------------+-------+ -| ``\uxxxx`` | Character with 16-bit hex value | \(5) | -| | *xxxx* | | -+-----------------+---------------------------------+-------+ -| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(6) | -| | *xxxxxxxx* | | -+-----------------+---------------------------------+-------+ +.. index:: single: ASCII@ASCII Notes: (1) - As in Standard C, up to three octal digits are accepted. + Individual code units which form parts of a surrogate pair can be encoded using + this escape sequence. (2) - Unlike in Standard C, exactly two hex digits are required. + Any Unicode character can be encoded this way, but characters outside the Basic + Multilingual Plane (BMP) will be encoded using a surrogate pair if Python is + compiled to use 16-bit code units (the default). (3) - In a bytes literal, hexadecimal and octal escapes denote the byte with the - given value. In a string literal, these escapes denote a Unicode character - with the given value. + As in Standard C, up to three octal digits are accepted. (4) - .. versionchanged:: 3.3 - Support for name aliases [#]_ has been added. + Unlike in Standard C, exactly two hex digits are required. (5) - Exactly four hex digits are required. - -(6) - Any Unicode character can be encoded this way. Exactly eight hex digits - are required. - + In a string literal, hexadecimal and octal escapes denote the byte with the + given value; it is not necessary that the byte encodes a character in the source + character set. In a Unicode literal, these escapes denote a Unicode character + with the given value. -.. index:: unrecognized escape sequence +.. index:: single: unrecognized escape sequence Unlike Standard C, all unrecognized escape sequences are left in the string -unchanged, i.e., *the backslash is left in the result*. (This behavior is +unchanged, i.e., *the backslash is left in the string*. (This behavior is useful when debugging: if an escape sequence is mistyped, the resulting output is more easily recognized as broken.) It is also important to note that the -escape sequences only recognized in string literals fall into the category of -unrecognized escapes for bytes literals. - - .. versionchanged:: 3.6 - Unrecognized escape sequences produce a :exc:`DeprecationWarning`. In - a future Python version they will be a :exc:`SyntaxWarning` and - eventually a :exc:`SyntaxError`. - -Even in a raw literal, quotes can be escaped with a backslash, but the -backslash remains in the result; for example, ``r"\""`` is a valid string -literal consisting of two characters: a backslash and a double quote; ``r"\"`` -is not a valid string literal (even a raw string cannot end in an odd number of -backslashes). Specifically, *a raw literal cannot end in a single backslash* -(since the backslash would escape the following quote character). Note also -that a single backslash followed by a newline is interpreted as those two -characters as part of the literal, *not* as a line continuation. - - -.. _string-concatenation: +escape sequences marked as "(Unicode only)" in the table above fall into the +category of unrecognized escapes for non-Unicode string literals. + +When an ``'r'`` or ``'R'`` prefix is present, a character following a backslash +is included in the string without change, and *all backslashes are left in the +string*. For example, the string literal ``r"\n"`` consists of two characters: +a backslash and a lowercase ``'n'``. String quotes can be escaped with a +backslash, but the backslash remains in the string; for example, ``r"\""`` is a +valid string literal consisting of two characters: a backslash and a double +quote; ``r"\"`` is not a valid string literal (even a raw string cannot end in +an odd number of backslashes). Specifically, *a raw string cannot end in a +single backslash* (since the backslash would escape the following quote +character). Note also that a single backslash followed by a newline is +interpreted as those two characters as part of the string, *not* as a line +continuation. + +When an ``'r'`` or ``'R'`` prefix is used in conjunction with a ``'u'`` or +``'U'`` prefix, then the ``\uXXXX`` and ``\UXXXXXXXX`` escape sequences are +processed while *all other backslashes are left in the string*. For example, +the string literal ``ur"\u0062\n"`` consists of three Unicode characters: 'LATIN +SMALL LETTER B', 'REVERSE SOLIDUS', and 'LATIN SMALL LETTER N'. Backslashes can +be escaped with a preceding backslash; however, both remain in the string. As a +result, ``\uXXXX`` escape sequences are only recognized when there are an odd +number of backslashes. + + +.. _string-catenation: String literal concatenation ---------------------------- -Multiple adjacent string or bytes literals (delimited by whitespace), possibly -using different quoting conventions, are allowed, and their meaning is the same -as their concatenation. Thus, ``"hello" 'world'`` is equivalent to +Multiple adjacent string literals (delimited by whitespace), possibly using +different quoting conventions, are allowed, and their meaning is the same as +their concatenation. Thus, ``"hello" 'world'`` is equivalent to ``"helloworld"``. This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings, for example:: @@ -627,134 +594,7 @@ comments to parts of strings, for example:: Note that this feature is defined at the syntactical level, but implemented at compile time. The '+' operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting -styles for each component (even mixing raw strings and triple quoted strings), -and formatted string literals may be concatenated with plain string literals. - - -.. index:: - single: formatted string literal - single: interpolated string literal - single: string; formatted literal - single: string; interpolated literal - single: f-string - single: {} (curly brackets); in formatted string literal - single: ! (exclamation); in formatted string literal - single: : (colon); in formatted string literal -.. _f-strings: - -Formatted string literals -------------------------- - -.. versionadded:: 3.6 - -A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal -that is prefixed with ``'f'`` or ``'F'``. These strings may contain -replacement fields, which are expressions delimited by curly braces ``{}``. -While other string literals always have a constant value, formatted strings -are really expressions evaluated at run time. - -Escape sequences are decoded like in ordinary string literals (except when -a literal is also marked as a raw string). After decoding, the grammar -for the contents of the string is: - -.. productionlist:: - f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)* - replacement_field: "{" `f_expression` ["!" `conversion`] [":" `format_spec`] "}" - f_expression: (`conditional_expression` | "*" `or_expr`) - : ("," `conditional_expression` | "," "*" `or_expr`)* [","] - : | `yield_expression` - conversion: "s" | "r" | "a" - format_spec: (`literal_char` | NULL | `replacement_field`)* - literal_char: <any code point except "{", "}" or NULL> - -The parts of the string outside curly braces are treated literally, -except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced -with the corresponding single curly brace. A single opening curly -bracket ``'{'`` marks a replacement field, which starts with a -Python expression. After the expression, there may be a conversion field, -introduced by an exclamation point ``'!'``. A format specifier may also -be appended, introduced by a colon ``':'``. A replacement field ends -with a closing curly bracket ``'}'``. - -Expressions in formatted string literals are treated like regular -Python expressions surrounded by parentheses, with a few exceptions. -An empty expression is not allowed, and both :keyword:`lambda` and -assignment expressions ``:=`` must be surrounded by explicit parentheses. -Replacement expressions can contain line breaks (e.g. in triple-quoted -strings), but they cannot contain comments. Each expression is evaluated -in the context where the formatted string literal appears, in order from -left to right. - -If a conversion is specified, the result of evaluating the expression -is converted before formatting. Conversion ``'!s'`` calls :func:`str` on -the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. - -The result is then formatted using the :func:`format` protocol. The -format specifier is passed to the :meth:`__format__` method of the -expression or conversion result. An empty string is passed when the -format specifier is omitted. The formatted result is then included in -the final value of the whole string. - -Top-level format specifiers may include nested replacement fields. These nested -fields may include their own conversion fields and :ref:`format specifiers -<formatspec>`, but may not include more deeply-nested replacement fields. The -:ref:`format specifier mini-language <formatspec>` is the same as that used by -the string .format() method. - -Formatted string literals may be concatenated, but replacement fields -cannot be split across literals. - -Some examples of formatted string literals:: - - >>> name = "Fred" - >>> f"He said his name is {name!r}." - "He said his name is 'Fred'." - >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r - "He said his name is 'Fred'." - >>> width = 10 - >>> precision = 4 - >>> value = decimal.Decimal("12.34567") - >>> f"result: {value:{width}.{precision}}" # nested fields - 'result: 12.35' - >>> today = datetime(year=2017, month=1, day=27) - >>> f"{today:%B %d, %Y}" # using date format specifier - 'January 27, 2017' - >>> number = 1024 - >>> f"{number:#0x}" # using integer format specifier - '0x400' - -A consequence of sharing the same syntax as regular string literals is -that characters in the replacement fields must not conflict with the -quoting used in the outer formatted string literal:: - - f"abc {a["x"]} def" # error: outer string literal ended prematurely - f"abc {a['x']} def" # workaround: use different quoting - -Backslashes are not allowed in format expressions and will raise -an error:: - - f"newline: {ord('\n')}" # raises SyntaxError - -To include a value in which a backslash escape is required, create -a temporary variable. - - >>> newline = ord('\n') - >>> f"newline: {newline}" - 'newline: 10' - -Formatted string literals cannot be used as docstrings, even if they do not -include expressions. - -:: - - >>> def foo(): - ... f"Not a docstring" - ... - >>> foo.__doc__ is None - True - -See also :pep:`498` for the proposal that added formatted string literals, -and :meth:`str.format`, which uses a related format string mechanism. +styles for each component (even mixing raw strings and triple quoted strings). .. _numbers: @@ -762,69 +602,66 @@ and :meth:`str.format`, which uses a related format string mechanism. Numeric literals ---------------- -.. index:: number, numeric literal, integer literal - floating point literal, hexadecimal literal - octal literal, binary literal, decimal literal, imaginary literal, complex literal - -There are three types of numeric literals: integers, floating point numbers, and -imaginary numbers. There are no complex literals (complex numbers can be formed -by adding a real number and an imaginary number). +.. index:: + single: number + single: numeric literal + single: integer literal + single: plain integer literal + single: long integer literal + single: floating point literal + single: hexadecimal literal + single: binary literal + single: octal literal + single: decimal literal + single: imaginary literal + single: complex; literal + +There are four types of numeric literals: plain integers, long integers, +floating point numbers, and imaginary numbers. There are no complex literals +(complex numbers can be formed by adding a real number and an imaginary number). Note that numeric literals do not include a sign; a phrase like ``-1`` is actually an expression composed of the unary operator '``-``' and the literal ``1``. -.. index:: - single: 0b; integer literal - single: 0o; integer literal - single: 0x; integer literal - single: _ (underscore); in numeric literal - .. _integers: -Integer literals ----------------- +Integer and long integer literals +--------------------------------- -Integer literals are described by the following lexical definitions: +Integer and long integer literals are described by the following lexical +definitions: .. productionlist:: - integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger` - decinteger: `nonzerodigit` (["_"] `digit`)* | "0"+ (["_"] "0")* - bininteger: "0" ("b" | "B") (["_"] `bindigit`)+ - octinteger: "0" ("o" | "O") (["_"] `octdigit`)+ - hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+ + longinteger: `integer` ("l" | "L") + integer: `decimalinteger` | `octinteger` | `hexinteger` | `bininteger` + decimalinteger: `nonzerodigit` `digit`* | "0" + octinteger: "0" ("o" | "O") `octdigit`+ | "0" `octdigit`+ + hexinteger: "0" ("x" | "X") `hexdigit`+ + bininteger: "0" ("b" | "B") `bindigit`+ nonzerodigit: "1"..."9" - digit: "0"..."9" - bindigit: "0" | "1" octdigit: "0"..."7" + bindigit: "0" | "1" hexdigit: `digit` | "a"..."f" | "A"..."F" -There is no limit for the length of integer literals apart from what can be -stored in available memory. +Although both lower case ``'l'`` and upper case ``'L'`` are allowed as suffix +for long integers, it is strongly recommended to always use ``'L'``, since the +letter ``'l'`` looks too much like the digit ``'1'``. -Underscores are ignored for determining the numeric value of the literal. They -can be used to group digits for enhanced readability. One underscore can occur -between digits, and after base specifiers like ``0x``. +Plain integer literals that are above the largest representable plain integer +(e.g., 2147483647 when using 32-bit arithmetic) are accepted as if they were +long integers instead. [#]_ There is no limit for long integer literals apart +from what can be stored in available memory. -Note that leading zeros in a non-zero decimal number are not allowed. This is -for disambiguation with C-style octal literals, which Python used before version -3.0. +Some examples of plain integer literals (first row) and long integer literals +(second and third rows):: -Some examples of integer literals:: + 7 2147483647 0177 + 3L 79228162514264337593543950336L 0377L 0x100000000L + 79228162514264337593543950336 0xdeadbeef - 7 2147483647 0o177 0b100110111 - 3 79228162514264337593543950336 0o377 0xdeadbeef - 100_000_000_000 0b_1110_0101 -.. versionchanged:: 3.6 - Underscores are now allowed for grouping purposes in literals. - - -.. index:: - single: . (dot); in numeric literal - single: e; in numeric literal - single: _ (underscore); in numeric literal .. _floating: Floating point literals @@ -834,27 +671,25 @@ Floating point literals are described by the following lexical definitions: .. productionlist:: floatnumber: `pointfloat` | `exponentfloat` - pointfloat: [`digitpart`] `fraction` | `digitpart` "." - exponentfloat: (`digitpart` | `pointfloat`) `exponent` - digitpart: `digit` (["_"] `digit`)* - fraction: "." `digitpart` - exponent: ("e" | "E") ["+" | "-"] `digitpart` + pointfloat: [`intpart`] `fraction` | `intpart` "." + exponentfloat: (`intpart` | `pointfloat`) `exponent` + intpart: `digit`+ + fraction: "." `digit`+ + exponent: ("e" | "E") ["+" | "-"] `digit`+ -Note that the integer and exponent parts are always interpreted using radix 10. -For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The -allowed range of floating point literals is implementation-dependent. As in -integer literals, underscores are supported for digit grouping. +Note that the integer and exponent parts of floating point numbers can look like +octal integers, but are interpreted using radix 10. For example, ``077e010`` is +legal, and denotes the same number as ``77e10``. The allowed range of floating +point literals is implementation-dependent. Some examples of floating point +literals:: -Some examples of floating point literals:: + 3.14 10. .001 1e100 3.14e-10 0e0 - 3.14 10. .001 1e100 3.14e-10 0e0 3.14_15_93 - -.. versionchanged:: 3.6 - Underscores are now allowed for grouping purposes in literals. +Note that numeric literals do not include a sign; a phrase like ``-1`` is +actually an expression composed of the unary operator ``-`` and the literal +``1``. -.. index:: - single: j; in numeric literal .. _imaginary: Imaginary literals @@ -863,7 +698,7 @@ Imaginary literals Imaginary literals are described by the following lexical definitions: .. productionlist:: - imagnumber: (`floatnumber` | `digitpart`) ("j" | "J") + imagnumber: (`floatnumber` | `intpart`) ("j" | "J") An imaginary literal yields a complex number with a real part of 0.0. Complex numbers are represented as a pair of floating point numbers and have the same @@ -871,7 +706,7 @@ restrictions on their range. To create a complex number with a nonzero real part, add a floating point number to it, e.g., ``(3+4j)``. Some examples of imaginary literals:: - 3.14j 10.j 10j .001j 1e100j 3.14e-10j 3.14_15_93j + 3.14j 10.j 10j .001j 1e100j 3.14e-10j .. _operators: @@ -886,9 +721,12 @@ The following tokens are operators: .. code-block:: none - + - * ** / // % @ - << >> & | ^ ~ := - < > <= >= == != + + - * ** / // % + << >> & | ^ ~ + < > <= >= == != <> + +The comparison operators ``<>`` and ``!=`` are alternate spellings of the same +operator. ``!=`` is the preferred spelling; ``<>`` is obsolescent. .. _delimiters: @@ -902,13 +740,13 @@ The following tokens serve as delimiters in the grammar: .. code-block:: none - ( ) [ ] { } - , : . ; @ = -> - += -= *= /= //= %= @= + ( ) [ ] { } @ + , : . ` = ; + += -= *= /= //= %= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence -of three periods has a special meaning as an ellipsis literal. The second half +of three periods has a special meaning as an ellipsis in slices. The second half of the list, the augmented assignment operators, serve lexically as delimiters, but also perform an operation. @@ -919,14 +757,20 @@ tokens or are otherwise significant to the lexical analyzer: ' " # \ +.. index:: single: ASCII@ASCII + The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error: .. code-block:: none - $ ? ` - + $ ? .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt +.. [#] In versions of Python prior to 2.4, octal and hexadecimal literals in the range + just above the largest representable plain integer but below the largest + unsigned 32-bit number (on a machine using 32-bit arithmetic), 4294967296, were + taken as the negative plain integer obtained by subtracting 4294967296 from + their unsigned value. + |