diff options
Diffstat (limited to 'Doc/reference')
| -rw-r--r-- | Doc/reference/compound_stmts.rst | 603 | ||||
| -rw-r--r-- | Doc/reference/datamodel.rst | 2414 | ||||
| -rw-r--r-- | Doc/reference/executionmodel.rst | 261 | ||||
| -rw-r--r-- | Doc/reference/expressions.rst | 1239 | ||||
| -rw-r--r-- | Doc/reference/import.rst | 1072 | ||||
| -rw-r--r-- | Doc/reference/index.rst | 1 | ||||
| -rw-r--r-- | Doc/reference/introduction.rst | 21 | ||||
| -rw-r--r-- | Doc/reference/lexical_analysis.rst | 720 | ||||
| -rw-r--r-- | Doc/reference/simple_stmts.rst | 897 | ||||
| -rw-r--r-- | Doc/reference/toplevel_components.rst | 30 |
10 files changed, 2637 insertions, 4621 deletions
diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 988eec6..523d9b5 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -13,34 +13,31 @@ statement may be contained in one line. The :keyword:`if`, :keyword:`while` and :keyword:`for` statements implement traditional control flow constructs. :keyword:`try` specifies exception -handlers and/or cleanup code for a group of statements, while the -:keyword:`with` statement allows the execution of initialization and -finalization code around a block of code. Function and class definitions are -also syntactically compound statements. +handlers and/or cleanup code for a group of statements. Function and class +definitions are also syntactically compound statements. .. index:: single: clause single: suite - single: ; (semicolon) -A compound statement consists of one or more 'clauses.' A clause consists of a +Compound statements consist of one or more 'clauses.' A clause consists of a header and a 'suite.' The clause headers of a particular compound statement are all at the same indentation level. Each clause header begins with a uniquely identifying keyword and ends with a colon. A suite is a group of statements controlled by a clause. A suite can be one or more semicolon-separated simple statements on the same line as the header, following the header's colon, or it can be one or more indented statements on subsequent lines. Only the latter -form of a suite can contain nested compound statements; the following is illegal, +form of suite can contain nested compound statements; the following is illegal, mostly because it wouldn't be clear to which :keyword:`if` clause a following -:keyword:`else` clause would belong:: +:keyword:`else` clause would belong: :: - if test1: if test2: print(x) + if test1: if test2: print x Also note that the semicolon binds tighter than the colon in this context, so -that in the following example, either all or none of the :func:`print` calls are -executed:: +that in the following example, either all or none of the :keyword:`print` +statements are executed:: - if x < y < z: print(x); print(y); print(z) + if x < y < z: print x; print y; print z Summarizing: @@ -52,9 +49,7 @@ Summarizing: : | `with_stmt` : | `funcdef` : | `classdef` - : | `async_with_stmt` - : | `async_for_stmt` - : | `async_funcdef` + : | `decorated` suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT statement: `stmt_list` NEWLINE | `compound_stmt` stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"] @@ -65,7 +60,7 @@ Summarizing: pair: dangling; else Note that statements always end in a ``NEWLINE`` possibly followed by a -``DEDENT``. Also note that optional continuation clauses always begin with a +``DEDENT``. Also note that optional continuation clauses always begin with a keyword that cannot start a statement, thus there are no ambiguities (the 'dangling :keyword:`else`' problem is solved in Python by requiring nested :keyword:`if` statements to be indented). @@ -78,20 +73,19 @@ on a separate line for clarity. .. _elif: .. _else: -The :keyword:`!if` statement -============================ +The :keyword:`if` statement +=========================== .. index:: - ! statement: if + statement: if keyword: elif keyword: else - single: : (colon); compound statement The :keyword:`if` statement is used for conditional execution: .. productionlist:: if_stmt: "if" `expression` ":" `suite` - : ("elif" `expression` ":" `suite`)* + : ( "elif" `expression` ":" `suite` )* : ["else" ":" `suite`] It selects exactly one of the suites by evaluating the expressions one by one @@ -103,14 +97,13 @@ false, the suite of the :keyword:`else` clause, if present, is executed. .. _while: -The :keyword:`!while` statement -=============================== +The :keyword:`while` statement +============================== .. index:: - ! statement: while - keyword: else + statement: while pair: loop; statement - single: : (colon); compound statement + keyword: else The :keyword:`while` statement is used for repeated execution as long as an expression is true: @@ -121,7 +114,7 @@ expression is true: This repeatedly tests the expression and, if it is true, executes the first suite; if the expression is false (which may be the first time it is tested) the -suite of the :keyword:`!else` clause, if present, is executed and the loop +suite of the :keyword:`else` clause, if present, is executed and the loop terminates. .. index:: @@ -129,24 +122,23 @@ terminates. statement: continue A :keyword:`break` statement executed in the first suite terminates the loop -without executing the :keyword:`!else` clause's suite. A :keyword:`continue` +without executing the :keyword:`else` clause's suite. A :keyword:`continue` statement executed in the first suite skips the rest of the suite and goes back to testing the expression. .. _for: -The :keyword:`!for` statement -============================= +The :keyword:`for` statement +============================ .. index:: - ! statement: for + statement: for + pair: loop; statement keyword: in keyword: else pair: target; list - pair: loop; statement object: sequence - single: : (colon); compound statement The :keyword:`for` statement is used to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object: @@ -157,42 +149,34 @@ The :keyword:`for` statement is used to iterate over the elements of a sequence The expression list is evaluated once; it should yield an iterable object. An iterator is created for the result of the ``expression_list``. The suite is -then executed once for each item provided by the iterator, in the order returned -by the iterator. Each item in turn is assigned to the target list using the -standard rules for assignments (see :ref:`assignment`), and then the suite is -executed. When the items are exhausted (which is immediately when the sequence -is empty or an iterator raises a :exc:`StopIteration` exception), the suite in -the :keyword:`!else` clause, if present, is executed, and the loop terminates. +then executed once for each item provided by the iterator, in the order of +ascending indices. Each item in turn is assigned to the target list using the +standard rules for assignments, and then the suite is executed. When the items +are exhausted (which is immediately when the sequence is empty), the suite in +the :keyword:`else` clause, if present, is executed, and the loop terminates. .. index:: statement: break statement: continue A :keyword:`break` statement executed in the first suite terminates the loop -without executing the :keyword:`!else` clause's suite. A :keyword:`continue` +without executing the :keyword:`else` clause's suite. A :keyword:`continue` statement executed in the first suite skips the rest of the suite and continues -with the next item, or with the :keyword:`!else` clause if there is no next +with the next item, or with the :keyword:`else` clause if there was no next item. -The for-loop makes assignments to the variables in the target list. -This overwrites all previous assignments to those variables including -those made in the suite of the for-loop:: - - for i in range(10): - print(i) - i = 5 # this will not affect the for-loop - # because i will be overwritten with the next - # index in the range - +The suite may assign to the variable(s) in the target list; this does not affect +the next item assigned to it. .. index:: builtin: range + pair: Pascal; language -Names in the target list are not deleted when the loop is finished, but if the -sequence is empty, they will not have been assigned to at all by the loop. Hint: -the built-in function :func:`range` returns an iterator of integers suitable to -emulate the effect of Pascal's ``for i := a to b do``; e.g., ``list(range(3))`` -returns the list ``[0, 1, 2]``. +The target list is not deleted when the loop is finished, but if the sequence is +empty, it will not have been assigned to at all by the loop. Hint: the built-in +function :func:`range` returns a sequence of integers suitable to emulate the +effect of Pascal's ``for i := a to b do``; e.g., ``range(3)`` returns the list +``[0, 1, 2]``. .. note:: @@ -201,16 +185,16 @@ returns the list ``[0, 1, 2]``. single: mutable sequence; loop over There is a subtlety when the sequence is being modified by the loop (this can - only occur for mutable sequences, e.g. lists). An internal counter is used - to keep track of which item is used next, and this is incremented on each + only occur for mutable sequences, e.g. lists). An internal counter is used to + keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) - item from the sequence, the next item will be skipped (since it gets the - index of the current item which has already been treated). Likewise, if the - suite inserts an item in the sequence before the current item, the current - item will be treated again the next time through the loop. This can lead to - nasty bugs that can be avoided by making a temporary copy using a slice of - the whole sequence, e.g., :: + item from the sequence, the next item will be skipped (since it gets the index + of the current item which has already been treated). Likewise, if the suite + inserts an item in the sequence before the current item, the current item will + be treated again the next time through the loop. This can lead to nasty bugs + that can be avoided by making a temporary copy using a slice of the whole + sequence, e.g., :: for x in a[:]: if x < 0: a.remove(x) @@ -220,40 +204,41 @@ returns the list ``[0, 1, 2]``. .. _except: .. _finally: -The :keyword:`!try` statement -============================= +The :keyword:`try` statement +============================ .. index:: - ! statement: try + statement: try keyword: except keyword: finally - keyword: else - keyword: as - single: : (colon); compound statement The :keyword:`try` statement specifies exception handlers and/or cleanup code for a group of statements: .. productionlist:: - try_stmt: `try1_stmt` | `try2_stmt` + try_stmt: try1_stmt | try2_stmt try1_stmt: "try" ":" `suite` - : ("except" [`expression` ["as" `identifier`]] ":" `suite`)+ + : ("except" [`expression` [("as" | ",") `identifier`]] ":" `suite`)+ : ["else" ":" `suite`] : ["finally" ":" `suite`] try2_stmt: "try" ":" `suite` : "finally" ":" `suite` +.. versionchanged:: 2.5 + In previous versions of Python, :keyword:`try`...\ :keyword:`except`...\ + :keyword:`finally` did not work. :keyword:`try`...\ :keyword:`except` had to be + nested in :keyword:`try`...\ :keyword:`finally`. The :keyword:`except` clause(s) specify one or more exception handlers. When no exception occurs in the :keyword:`try` clause, no exception handler is executed. -When an exception occurs in the :keyword:`!try` suite, a search for an exception +When an exception occurs in the :keyword:`try` suite, a search for an exception handler is started. This search inspects the except clauses in turn until one is found that matches the exception. An expression-less except clause, if present, must be last; it matches any exception. For an except clause with an expression, that expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception -object or a tuple containing an item compatible with the exception. +object, or a tuple containing an item compatible with the exception. If no except clause matches the exception, the search for an exception handler continues in the surrounding code and on the invocation stack. [#]_ @@ -263,46 +248,32 @@ exception, the original search for a handler is canceled and a search starts for the new exception in the surrounding code and on the call stack (it is treated as if the entire :keyword:`try` statement raised the exception). -.. index:: single: as; except clause - When a matching except clause is found, the exception is assigned to the target -specified after the :keyword:`!as` keyword in that except clause, if present, and -the except clause's suite is executed. All except clauses must have an -executable block. When the end of this block is reached, execution continues -normally after the entire try statement. (This means that if two nested -handlers exist for the same exception, and the exception occurs in the try -clause of the inner handler, the outer handler will not handle the exception.) - -When an exception has been assigned using ``as target``, it is cleared at the -end of the except clause. This is as if :: - - except E as N: - foo - -was translated to :: - - except E as N: - try: - foo - finally: - del N - -This means the exception must be assigned to a different name to be able to -refer to it after the except clause. Exceptions are cleared because with the -traceback attached to them, they form a reference cycle with the stack frame, -keeping all locals in that frame alive until the next garbage collection occurs. +specified in that except clause, if present, and the except clause's suite is +executed. All except clauses must have an executable block. When the end of +this block is reached, execution continues normally after the entire try +statement. (This means that if two nested handlers exist for the same +exception, and the exception occurs in the try clause of the inner handler, the +outer handler will not handle the exception.) .. index:: module: sys object: traceback + single: exc_type (in module sys) + single: exc_value (in module sys) + single: exc_traceback (in module sys) Before an except clause's suite is executed, details about the exception are -stored in the :mod:`sys` module and can be accessed via :func:`sys.exc_info`. -:func:`sys.exc_info` returns a 3-tuple consisting of the exception class, the -exception instance and a traceback object (see section :ref:`types`) identifying -the point in the program where the exception occurred. :func:`sys.exc_info` -values are restored to their previous values (before the call) when returning -from a function that handled an exception. +assigned to three variables in the :mod:`sys` module: ``sys.exc_type`` receives +the object identifying the exception; ``sys.exc_value`` receives the exception's +parameter; ``sys.exc_traceback`` receives a traceback object (see section +:ref:`types`) identifying the point in the program where the exception +occurred. These details are also available through the :func:`sys.exc_info` +function, which returns a tuple ``(exc_type, exc_value, exc_traceback)``. Use +of the corresponding variables is deprecated in favor of this function, since +their use is unsafe in a threaded program. As of Python 1.5, the variables are +restored to their previous values (before the call) when returning from a +function that handled an exception. .. index:: keyword: else @@ -310,23 +281,22 @@ from a function that handled an exception. statement: break statement: continue -The optional :keyword:`!else` clause is executed if the control flow leaves the +The optional :keyword:`else` clause is executed if the control flow leaves the :keyword:`try` suite, no exception was raised, and no :keyword:`return`, :keyword:`continue`, or :keyword:`break` statement was executed. Exceptions in -the :keyword:`!else` clause are not handled by the preceding :keyword:`except` +the :keyword:`else` clause are not handled by the preceding :keyword:`except` clauses. .. index:: keyword: finally If :keyword:`finally` is present, it specifies a 'cleanup' handler. The :keyword:`try` clause is executed, including any :keyword:`except` and -:keyword:`!else` clauses. If an exception occurs in any of the clauses and is -not handled, the exception is temporarily saved. The :keyword:`!finally` clause -is executed. If there is a saved exception it is re-raised at the end of the -:keyword:`!finally` clause. If the :keyword:`!finally` clause raises another -exception, the saved exception is set as the context of the new exception. -If the :keyword:`!finally` clause executes a :keyword:`return`, :keyword:`break` -or :keyword:`continue` statement, the saved exception is discarded:: +:keyword:`else` clauses. If an exception occurs in any of the clauses and is +not handled, the exception is temporarily saved. The :keyword:`finally` clause +is executed. If there is a saved exception, it is re-raised at the end of the +:keyword:`finally` clause. If the :keyword:`finally` clause raises another +exception or executes a :keyword:`return` or :keyword:`break` statement, the +saved exception is discarded:: >>> def f(): ... try: @@ -346,12 +316,15 @@ the :keyword:`finally` clause. statement: continue When a :keyword:`return`, :keyword:`break` or :keyword:`continue` statement is -executed in the :keyword:`try` suite of a :keyword:`!try`...\ :keyword:`!finally` -statement, the :keyword:`finally` clause is also executed 'on the way out.' +executed in the :keyword:`try` suite of a :keyword:`try`...\ :keyword:`finally` +statement, the :keyword:`finally` clause is also executed 'on the way out.' A +:keyword:`continue` statement is illegal in the :keyword:`finally` clause. (The +reason is a problem with the current implementation --- this restriction may be +lifted in the future). The return value of a function is determined by the last :keyword:`return` statement executed. Since the :keyword:`finally` clause always executes, a -:keyword:`!return` statement executed in the :keyword:`!finally` clause will +:keyword:`return` statement executed in the :keyword:`finally` clause will always be the last one executed:: >>> def foo(): @@ -367,31 +340,26 @@ Additional information on exceptions can be found in section :ref:`exceptions`, and information on using the :keyword:`raise` statement to generate exceptions may be found in section :ref:`raise`. -.. versionchanged:: 3.8 - Prior to Python 3.8, a :keyword:`continue` statement was illegal in the - :keyword:`finally` clause due to a problem with the implementation. - .. _with: .. _as: -The :keyword:`!with` statement -============================== +The :keyword:`with` statement +============================= .. index:: - ! statement: with - keyword: as - single: as; with statement - single: , (comma); with statement - single: : (colon); compound statement + statement: with + single: as; with statement + +.. versionadded:: 2.5 The :keyword:`with` statement is used to wrap the execution of a block with -methods defined by a context manager (see section :ref:`context-managers`). -This allows common :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` -usage patterns to be encapsulated for convenient reuse. +methods defined by a context manager (see section :ref:`context-managers`). This +allows common :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` usage +patterns to be encapsulated for convenient reuse. .. productionlist:: - with_stmt: "with" `with_item` ("," `with_item`)* ":" `suite` + with_stmt: "with" with_item ("," with_item)* ":" `suite` with_item: `expression` ["as" `target`] The execution of the :keyword:`with` statement with one "item" proceeds as follows: @@ -408,27 +376,26 @@ The execution of the :keyword:`with` statement with one "item" proceeds as follo .. note:: - The :keyword:`with` statement guarantees that if the :meth:`__enter__` - method returns without an error, then :meth:`__exit__` will always be - called. Thus, if an error occurs during the assignment to the target list, - it will be treated the same as an error occurring within the suite would - be. See step 6 below. + The :keyword:`with` statement guarantees that if the :meth:`__enter__` method + returns without an error, then :meth:`__exit__` will always be called. Thus, if + an error occurs during the assignment to the target list, it will be treated the + same as an error occurring within the suite would be. See step 6 below. #. The suite is executed. -#. The context manager's :meth:`__exit__` method is invoked. If an exception +#. The context manager's :meth:`__exit__` method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to :meth:`__exit__`. Otherwise, three :const:`None` arguments are supplied. If the suite was exited due to an exception, and the return value from the - :meth:`__exit__` method was false, the exception is reraised. If the return + :meth:`__exit__` method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the :keyword:`with` statement. - If the suite was exited for any reason other than an exception, the return - value from :meth:`__exit__` is ignored, and execution proceeds at the normal - location for the kind of exit that was taken. + If the suite was exited for any reason other than an exception, the return value + from :meth:`__exit__` is ignored, and execution proceeds at the normal location + for the kind of exit that was taken. With more than one item, the context managers are processed as if multiple :keyword:`with` statements were nested:: @@ -442,7 +409,13 @@ is equivalent to :: with B() as b: suite -.. versionchanged:: 3.1 +.. note:: + + In Python 2.5, the :keyword:`with` statement is only allowed when the + ``with_statement`` feature has been enabled. It is always enabled in + Python 2.6. + +.. versionchanged:: 2.7 Support for multiple context expressions. .. seealso:: @@ -468,32 +441,25 @@ Function definitions pair: name; binding object: user-defined function object: function - pair: function; name - pair: name; binding - single: () (parentheses); function definition - single: , (comma); parameter list - single: : (colon); compound statement A function definition defines a user-defined function object (see section :ref:`types`): .. productionlist:: - funcdef: [`decorators`] "def" `funcname` "(" [`parameter_list`] ")" - : ["->" `expression`] ":" `suite` + decorated: decorators (classdef | funcdef) decorators: `decorator`+ decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE + funcdef: "def" `funcname` "(" [`parameter_list`] ")" ":" `suite` dotted_name: `identifier` ("." `identifier`)* - parameter_list: `defparameter` ("," `defparameter`)* "," "/" ["," [`parameter_list_no_posonly`]] - : | `parameter_list_no_posonly` - parameter_list_no_posonly: `defparameter` ("," `defparameter`)* ["," [`parameter_list_starargs`]] - : | `parameter_list_starargs` - parameter_list_starargs: "*" [`parameter`] ("," `defparameter`)* ["," ["**" `parameter` [","]]] - : | "**" `parameter` [","] - parameter: `identifier` [":" `expression`] + parameter_list: (`defparameter` ",")* + : ( "*" `identifier` ["," "**" `identifier`] + : | "**" `identifier` + : | `defparameter` [","] ) defparameter: `parameter` ["=" `expression`] + sublist: `parameter` ("," `parameter`)* [","] + parameter: `identifier` | "(" `sublist` ")" funcname: `identifier` - A function definition is an executable statement. Its execution binds the function name in the current local namespace to a function object (a wrapper around the executable code for the function). This function object contains a @@ -504,48 +470,44 @@ The function definition does not execute the function body; this gets executed only when the function is called. [#]_ .. index:: - single: @ (at); function definition + statement: @ A function definition may be wrapped by one or more :term:`decorator` expressions. Decorator expressions are evaluated when the function is defined, in the scope that contains the function definition. The result must be a callable, which is invoked with the function object as the only argument. The returned value is bound to the function name instead of the function object. Multiple decorators -are applied in nested fashion. For example, the following code :: +are applied in nested fashion. For example, the following code:: @f1(arg) @f2 def func(): pass -is roughly equivalent to :: +is equivalent to:: def func(): pass func = f1(arg)(f2(func)) -except that the original function is not temporarily bound to the name ``func``. - .. index:: triple: default; parameter; value single: argument; function definition - single: = (equals); function definition - -When one or more :term:`parameters <parameter>` have the form *parameter* ``=`` -*expression*, the function is said to have "default parameter values." For a -parameter with a default value, the corresponding :term:`argument` may be -omitted from a call, in which -case the parameter's default value is substituted. If a parameter has a default -value, all following parameters up until the "``*``" must also have a default + +When one or more top-level :term:`parameters <parameter>` have the form +*parameter* ``=`` *expression*, the function is said to have "default parameter +values." For a parameter with a default value, the corresponding +:term:`argument` may be omitted from a call, in which +case the parameter's default value is substituted. If a +parameter has a default value, all following parameters must also have a default value --- this is a syntactic restriction that is not expressed by the grammar. -**Default parameter values are evaluated from left to right when the function -definition is executed.** This means that the expression is evaluated once, when -the function is defined, and that the same "pre-computed" value is used for each -call. This is especially important to understand when a default parameter is a -mutable object, such as a list or a dictionary: if the function modifies the -object (e.g. by appending an item to a list), the default value is in effect -modified. This is generally not what was intended. A way around this is to use -``None`` as the default, and explicitly test for it in the body of the function, -e.g.:: +**Default parameter values are evaluated when the function definition is +executed.** This means that the expression is evaluated once, when the function +is defined, and that the same "pre-computed" value is used for each call. This +is especially important to understand when a default parameter is a mutable +object, such as a list or a dictionary: if the function modifies the object +(e.g. by appending an item to a list), the default value is in effect modified. +This is generally not what was intended. A way around this is to use ``None`` +as the default, and explicitly test for it in the body of the function, e.g.:: def whats_on_the_telly(penguin=None): if penguin is None: @@ -554,37 +516,16 @@ e.g.:: return penguin .. index:: - single: * (asterisk); function definition - single: **; function definition + statement: * + statement: ** Function call semantics are described in more detail in section :ref:`calls`. A function call always assigns values to all parameters mentioned in the parameter list, either from position arguments, from keyword arguments, or from default values. If the form "``*identifier``" is present, it is initialized to a tuple -receiving any excess positional parameters, defaulting to the empty tuple. -If the form "``**identifier``" is present, it is initialized to a new -ordered mapping receiving any excess keyword arguments, defaulting to a -new empty mapping of the same type. Parameters after "``*``" or -"``*identifier``" are keyword-only parameters and may only be passed -used keyword arguments. - -.. index:: - pair: function; annotations - single: ->; function annotations - single: : (colon); function annotations - -Parameters may have an :term:`annotation <function annotation>` of the form "``: expression``" -following the parameter name. Any parameter may have an annotation, even those of the form -``*identifier`` or ``**identifier``. Functions may have "return" annotation of -the form "``-> expression``" after the parameter list. These annotations can be -any valid Python expression. The presence of annotations does not change the -semantics of a function. The annotation values are available as values of -a dictionary keyed by the parameters' names in the :attr:`__annotations__` -attribute of the function object. If the ``annotations`` import from -:mod:`__future__` is used, annotations are preserved as strings at runtime which -enables postponed evaluation. Otherwise, they are evaluated when the function -definition is executed. In this case annotations may be evaluated in -a different order than they appear in the source code. +receiving any excess positional parameters, defaulting to the empty tuple. If +the form "``**identifier``" is present, it is initialized to a new dictionary +receiving any excess keyword arguments, defaulting to a new empty dictionary. .. index:: pair: lambda; expression @@ -593,31 +534,15 @@ name), for immediate use in expressions. This uses lambda expressions, describe section :ref:`lambda`. Note that the lambda expression is merely a shorthand for a simplified function definition; a function defined in a ":keyword:`def`" statement can be passed around or assigned to another name just like a function -defined by a lambda expression. The ":keyword:`!def`" form is actually more powerful -since it allows the execution of multiple statements and annotations. +defined by a lambda expression. The ":keyword:`def`" form is actually more powerful +since it allows the execution of multiple statements. -**Programmer's note:** Functions are first-class objects. A "``def``" statement +**Programmer's note:** Functions are first-class objects. A "``def``" form executed inside a function definition defines a local function that can be returned or passed around. Free variables used in the nested function can access the local variables of the function containing the def. See section :ref:`naming` for details. -.. seealso:: - - :pep:`3107` - Function Annotations - The original specification for function annotations. - - :pep:`484` - Type Hints - Definition of a standard meaning for annotations: type hints. - - :pep:`526` - Syntax for Variable Annotations - Ability to type hint variable declarations, including class - variables and instance variables - - :pep:`563` - Postponed Evaluation of Annotations - Support for forward references within annotations by preserving - annotations in a string form at runtime instead of eager evaluation. - .. _class: @@ -633,213 +558,39 @@ Class definitions pair: execution; frame single: inheritance single: docstring - single: () (parentheses); class definition - single: , (comma); expression list - single: : (colon); compound statement A class definition defines a class object (see section :ref:`types`): .. productionlist:: - classdef: [`decorators`] "class" `classname` [`inheritance`] ":" `suite` - inheritance: "(" [`argument_list`] ")" + classdef: "class" `classname` [`inheritance`] ":" `suite` + inheritance: "(" [`expression_list`] ")" classname: `identifier` -A class definition is an executable statement. The inheritance list usually -gives a list of base classes (see :ref:`metaclasses` for more advanced uses), so -each item in the list should evaluate to a class object which allows -subclassing. Classes without an inheritance list inherit, by default, from the -base class :class:`object`; hence, :: - - class Foo: - pass - -is equivalent to :: - - class Foo(object): - pass - -The class's suite is then executed in a new execution frame (see :ref:`naming`), -using a newly created local namespace and the original global namespace. -(Usually, the suite contains mostly function definitions.) When the class's -suite finishes execution, its execution frame is discarded but its local -namespace is saved. [#]_ A class object is then created using the inheritance -list for the base classes and the saved local namespace for the attribute -dictionary. The class name is bound to this class object in the original local -namespace. - -The order in which attributes are defined in the class body is preserved -in the new class's ``__dict__``. Note that this is reliable only right -after the class is created and only for classes that were defined using -the definition syntax. - -Class creation can be customized heavily using :ref:`metaclasses <metaclasses>`. - -.. index:: - single: @ (at); class definition - -Classes can also be decorated: just like when decorating functions, :: - - @f1(arg) - @f2 - class Foo: pass - -is roughly equivalent to :: - - class Foo: pass - Foo = f1(arg)(f2(Foo)) - -The evaluation rules for the decorator expressions are the same as for function -decorators. The result is then bound to the class name. +A class definition is an executable statement. It first evaluates the +inheritance list, if present. Each item in the inheritance list should evaluate +to a class object or class type which allows subclassing. The class's suite is +then executed in a new execution frame (see section :ref:`naming`), using a +newly created local namespace and the original global namespace. (Usually, the +suite contains only function definitions.) When the class's suite finishes +execution, its execution frame is discarded but its local namespace is +saved. [#]_ A class object is then created using the inheritance list for the +base classes and the saved local namespace for the attribute dictionary. The +class name is bound to this class object in the original local namespace. **Programmer's note:** Variables defined in the class definition are class -attributes; they are shared by instances. Instance attributes can be set in a -method with ``self.name = value``. Both class and instance attributes are -accessible through the notation "``self.name``", and an instance attribute hides -a class attribute with the same name when accessed in this way. Class -attributes can be used as defaults for instance attributes, but using mutable -values there can lead to unexpected results. :ref:`Descriptors <descriptors>` -can be used to create instance variables with different implementation details. - - -.. seealso:: - - :pep:`3115` - Metaclasses in Python 3000 - The proposal that changed the declaration of metaclasses to the current - syntax, and the semantics for how classes with metaclasses are - constructed. - - :pep:`3129` - Class Decorators - The proposal that added class decorators. Function and method decorators - were introduced in :pep:`318`. - - -.. _async: - -Coroutines -========== - -.. versionadded:: 3.5 - -.. index:: statement: async def -.. _`async def`: - -Coroutine function definition ------------------------------ - -.. productionlist:: - async_funcdef: [`decorators`] "async" "def" `funcname` "(" [`parameter_list`] ")" - : ["->" `expression`] ":" `suite` - -.. index:: - keyword: async - keyword: await - -Execution of Python coroutines can be suspended and resumed at many points -(see :term:`coroutine`). Inside the body of a coroutine function, ``await`` and -``async`` identifiers become reserved keywords; :keyword:`await` expressions, -:keyword:`async for` and :keyword:`async with` can only be used in -coroutine function bodies. - -Functions defined with ``async def`` syntax are always coroutine functions, -even if they do not contain ``await`` or ``async`` keywords. - -It is a :exc:`SyntaxError` to use a ``yield from`` expression inside the body -of a coroutine function. - -An example of a coroutine function:: - - async def func(param1, param2): - do_stuff() - await some_coroutine() - - -.. index:: statement: async for -.. _`async for`: - -The :keyword:`!async for` statement ------------------------------------ - -.. productionlist:: - async_for_stmt: "async" `for_stmt` - -An :term:`asynchronous iterable` is able to call asynchronous code in its -*iter* implementation, and :term:`asynchronous iterator` can call asynchronous -code in its *next* method. - -The ``async for`` statement allows convenient iteration over asynchronous -iterators. - -The following code:: - - async for TARGET in ITER: - BLOCK - else: - BLOCK2 - -Is semantically equivalent to:: - - iter = (ITER) - iter = type(iter).__aiter__(iter) - running = True - while running: - try: - TARGET = await type(iter).__anext__(iter) - except StopAsyncIteration: - running = False - else: - BLOCK - else: - BLOCK2 - -See also :meth:`__aiter__` and :meth:`__anext__` for details. - -It is a :exc:`SyntaxError` to use an ``async for`` statement outside the -body of a coroutine function. - - -.. index:: statement: async with -.. _`async with`: - -The :keyword:`!async with` statement ------------------------------------- - -.. productionlist:: - async_with_stmt: "async" `with_stmt` - -An :term:`asynchronous context manager` is a :term:`context manager` that is -able to suspend execution in its *enter* and *exit* methods. - -The following code:: - - async with EXPR as VAR: - BLOCK - -Is semantically equivalent to:: - - mgr = (EXPR) - aexit = type(mgr).__aexit__ - aenter = type(mgr).__aenter__(mgr) - - VAR = await aenter - try: - BLOCK - except: - if not await aexit(mgr, *sys.exc_info()): - raise - else: - await aexit(mgr, None, None, None) - -See also :meth:`__aenter__` and :meth:`__aexit__` for details. - -It is a :exc:`SyntaxError` to use an ``async with`` statement outside the -body of a coroutine function. - -.. seealso:: - - :pep:`492` - Coroutines with async and await syntax - The proposal that made coroutines a proper standalone concept in Python, - and added supporting syntax. - +variables; they are shared by all instances. To create instance variables, they +can be set in a method with ``self.name = value``. Both class and instance +variables are accessible through the notation "``self.name``", and an instance +variable hides a class variable with the same name when accessed in this way. +Class variables can be used as defaults for instance variables, but using +mutable values there can lead to unexpected results. For :term:`new-style +class`\es, descriptors can be used to create instance variables with different +implementation details. + +Class definitions, like function definitions, may be wrapped by one or more +:term:`decorator` expressions. The evaluation rules for the decorator +expressions are the same as for functions. The result must be a class object, +which is then bound to the class name. .. rubric:: Footnotes diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index b22ed92..92f95c2 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -29,25 +29,15 @@ represented by objects.) single: mutable object single: immutable object -.. XXX it *is* now possible in some cases to change an object's - type, under certain controlled conditions - Every object has an identity, a type and a value. An object's *identity* never changes once it has been created; you may think of it as the object's address in memory. The ':keyword:`is`' operator compares the identity of two objects; the -:func:`id` function returns an integer representing its identity. - -.. impl-detail:: - - For CPython, ``id(x)`` is the memory address where ``x`` is stored. - +:func:`id` function returns an integer representing its identity (currently +implemented as its address). An object's :dfn:`type` is also unchangeable. [#]_ An object's type determines the operations that the object supports (e.g., "does it have a length?") and also defines the possible values for objects of that type. The :func:`type` function returns an object's type (which is an object -itself). Like its identity, an object's :dfn:`type` is also unchangeable. -[#]_ - -The *value* of some objects can change. Objects whose value can +itself). The *value* of some objects can change. Objects whose value can change are said to be *mutable*; objects whose value is unchangeable once they are created are called *immutable*. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value @@ -77,7 +67,7 @@ are still reachable. module for information on controlling the collection of cyclic garbage. Other implementations act differently and CPython may change. Do not depend on immediate finalization of objects when they become - unreachable (so you should always close files explicitly). + unreachable (ex: always close files). Note that the use of the implementation's tracing or debugging facilities may keep objects alive that would normally be collectable. Also note that catching @@ -90,7 +80,7 @@ garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a :meth:`close` method. Programs are strongly recommended to explicitly close such objects. The ':keyword:`try`...\ :keyword:`finally`' statement -and the ':keyword:`with`' statement provide convenient ways to do this. +provides a convenient way to do this. .. index:: single: container @@ -129,8 +119,7 @@ The standard type hierarchy Below is a list of the types that are built into Python. Extension modules (written in C, Java, or other languages, depending on the implementation) can define additional types. Future versions of Python may add types to the type -hierarchy (e.g., rational numbers, efficiently stored arrays of integers, etc.), -although such additions will often be provided via the standard library instead. +hierarchy (e.g., rational numbers, efficiently stored arrays of integers, etc.). .. index:: single: attribute @@ -154,24 +143,18 @@ NotImplemented This type has a single value. There is a single object with this value. This object is accessed through the built-in name ``NotImplemented``. Numeric methods - and rich comparison methods should return this value if they do not implement the + and rich comparison methods may return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.) Its truth value is true. - See - :ref:`implementing-the-arithmetic-operations` - for more details. - - Ellipsis - .. index:: - object: Ellipsis - single: ...; ellipsis literal + .. index:: object: Ellipsis This type has a single value. There is a single object with this value. This - object is accessed through the literal ``...`` or the built-in name - ``Ellipsis``. Its truth value is true. + object is accessed through the built-in name ``Ellipsis``. It is used to + indicate the presence of the ``...`` syntax in a slice. Its truth value is + true. :class:`numbers.Number` .. index:: object: numeric @@ -191,32 +174,54 @@ Ellipsis These represent elements from the mathematical set of integers (positive and negative). - There are two types of integers: - - Integers (:class:`int`) + There are three types of integers: - These represent numbers in an unlimited range, subject to available (virtual) - memory only. For the purpose of shift and mask operations, a binary - representation is assumed, and negative numbers are represented in a variant of - 2's complement which gives the illusion of an infinite string of sign bits - extending to the left. - - Booleans (:class:`bool`) + Plain integers + .. index:: + object: plain integer + single: OverflowError (built-in exception) + + These represent numbers in the range -2147483648 through 2147483647. + (The range may be larger on machines with a larger natural word size, + but not smaller.) When the result of an operation would fall outside + this range, the result is normally returned as a long integer (in some + cases, the exception :exc:`OverflowError` is raised instead). For the + purpose of shift and mask operations, integers are assumed to have a + binary, 2's complement notation using 32 or more bits, and hiding no + bits from the user (i.e., all 4294967296 different bit patterns + correspond to different values). + + Long integers + .. index:: object: long integer + + These represent numbers in an unlimited range, subject to available + (virtual) memory only. For the purpose of shift and mask operations, a + binary representation is assumed, and negative numbers are represented + in a variant of 2's complement which gives the illusion of an infinite + string of sign bits extending to the left. + + Booleans .. index:: object: Boolean single: False single: True - These represent the truth values False and True. The two objects representing - the values ``False`` and ``True`` are the only Boolean objects. The Boolean type is a - subtype of the integer type, and Boolean values behave like the values 0 and 1, - respectively, in almost all contexts, the exception being that when converted to - a string, the strings ``"False"`` or ``"True"`` are returned, respectively. + These represent the truth values False and True. The two objects + representing the values ``False`` and ``True`` are the only Boolean objects. + The Boolean type is a subtype of plain integers, and Boolean values + behave like the values 0 and 1, respectively, in almost all contexts, + the exception being that when converted to a string, the strings + ``"False"`` or ``"True"`` are returned, respectively. .. index:: pair: integer; representation - The rules for integer representation are intended to give the most meaningful - interpretation of shift and mask operations involving negative integers. + The rules for integer representation are intended to give the most + meaningful interpretation of shift and mask operations involving negative + integers and the least surprises when switching between the plain and long + integer domains. Any operation, if it yields a result in the plain + integer domain, will yield the same result in the long integer domain or + when using mixed operands. The switch between domains is transparent to + the programmer. :class:`numbers.Real` (:class:`float`) .. index:: @@ -233,7 +238,7 @@ Ellipsis overhead of using objects in Python, so there is no reason to complicate the language with two kinds of floating point numbers. - :class:`numbers.Complex` (:class:`complex`) + :class:`numbers.Complex` .. index:: object: complex pair: complex; number @@ -263,6 +268,8 @@ Sequences sequence of the same type. This implies that the index set is renumbered so that it starts at 0. + .. index:: single: extended slicing + Some sequences also support "extended slicing" with a third "step" parameter: ``a[i:j:k]`` selects all items of *a* with index *x* where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<`` *j*. @@ -281,28 +288,56 @@ Sequences The following types are immutable sequences: - .. index:: - single: string; immutable sequences - Strings .. index:: builtin: chr builtin: ord + object: string + single: character + single: byte + single: ASCII@ASCII + + The items of a string are characters. There is no separate character type; a + character is represented by a string of one item. Characters represent (at + least) 8-bit bytes. The built-in functions :func:`chr` and :func:`ord` convert + between characters and nonnegative integers representing the byte values. Bytes + with the values 0--127 usually represent the corresponding ASCII values, but the + interpretation of values is up to the program. The string data type is also + used to represent arrays of bytes, e.g., to hold data read from a file. + + .. index:: + single: ASCII@ASCII + single: EBCDIC + single: character set + pair: string; comparison + builtin: chr + builtin: ord + + (On systems whose native character set is not ASCII, strings may use EBCDIC in + their internal representation, provided the functions :func:`chr` and + :func:`ord` implement a mapping between ASCII and EBCDIC, and string comparison + preserves the ASCII order. Or perhaps someone can propose a better rule?) + + Unicode + .. index:: + builtin: unichr + builtin: ord + builtin: unicode + object: unicode single: character single: integer single: Unicode - A string is a sequence of values that represent Unicode code points. - All the code points in the range ``U+0000 - U+10FFFF`` can be - represented in a string. Python doesn't have a :c:type:`char` type; - instead, every code point in the string is represented as a string - object with length ``1``. The built-in function :func:`ord` - converts a code point from its string form to an integer in the - range ``0 - 10FFFF``; :func:`chr` converts an integer in the range - ``0 - 10FFFF`` to the corresponding length ``1`` string object. - :meth:`str.encode` can be used to convert a :class:`str` to - :class:`bytes` using the given text encoding, and - :meth:`bytes.decode` can be used to achieve the opposite. + The items of a Unicode object are Unicode code units. A Unicode code unit is + represented by a Unicode object of one item and can hold either a 16-bit or + 32-bit value representing a Unicode ordinal (the maximum value for the ordinal + is given in ``sys.maxunicode``, and depends on how Python is configured at + compile time). Surrogate pairs may be present in the Unicode object, and will + be reported as two separate items. The built-in functions :func:`unichr` and + :func:`ord` convert between code units and nonnegative integers representing the + Unicode ordinals as defined in the Unicode Standard 3.0. Conversion from and to + other encodings are possible through the Unicode method :meth:`encode` and the + built-in function :func:`unicode`. Tuples .. index:: @@ -310,21 +345,11 @@ Sequences pair: singleton; tuple pair: empty; tuple - The items of a tuple are arbitrary Python objects. Tuples of two or - more items are formed by comma-separated lists of expressions. A tuple - of one item (a 'singleton') can be formed by affixing a comma to an - expression (an expression by itself does not create a tuple, since - parentheses must be usable for grouping of expressions). An empty - tuple can be formed by an empty pair of parentheses. - - Bytes - .. index:: bytes, byte - - A bytes object is an immutable array. The items are 8-bit bytes, - represented by integers in the range 0 <= x < 256. Bytes literals - (like ``b'abc'``) and the built-in :func:`bytes()` constructor - can be used to create bytes objects. Also, bytes objects can be - decoded to strings via the :meth:`~bytes.decode` method. + The items of a tuple are arbitrary Python objects. Tuples of two or more items + are formed by comma-separated lists of expressions. A tuple of one item (a + 'singleton') can be formed by affixing a comma to an expression (an expression + by itself does not create a tuple, since parentheses must be usable for grouping + of expressions). An empty tuple can be formed by an empty pair of parentheses. Mutable sequences .. index:: @@ -343,22 +368,22 @@ Sequences Lists .. index:: object: list - The items of a list are arbitrary Python objects. Lists are formed by - placing a comma-separated list of expressions in square brackets. (Note - that there are no special cases needed to form lists of length 0 or 1.) + The items of a list are arbitrary Python objects. Lists are formed by placing a + comma-separated list of expressions in square brackets. (Note that there are no + special cases needed to form lists of length 0 or 1.) Byte Arrays .. index:: bytearray A bytearray object is a mutable array. They are created by the built-in - :func:`bytearray` constructor. Aside from being mutable - (and hence unhashable), byte arrays otherwise provide the same interface - and functionality as immutable :class:`bytes` objects. + :func:`bytearray` constructor. Aside from being mutable (and hence + unhashable), byte arrays otherwise provide the same interface and + functionality as immutable bytes objects. .. index:: module: array - The extension module :mod:`array` provides an additional example of a - mutable sequence type, as does the :mod:`collections` module. + The extension module :mod:`array` provides an additional example of a mutable + sequence type. Set types .. index:: @@ -424,12 +449,12 @@ Mappings section :ref:`dict`). .. index:: - module: dbm.ndbm - module: dbm.gnu + module: dbm + module: gdbm + module: bsddb - The extension modules :mod:`dbm.ndbm` and :mod:`dbm.gnu` provide - additional examples of mapping types, as does the :mod:`collections` - module. + The extension modules :mod:`dbm`, :mod:`gdbm`, and :mod:`bsddb` provide + additional examples of mapping types. Callable types .. index:: @@ -462,149 +487,175 @@ Callable types single: __module__ (function attribute) single: __dict__ (function attribute) single: __defaults__ (function attribute) - single: __closure__ (function attribute) single: __code__ (function attribute) single: __globals__ (function attribute) - single: __annotations__ (function attribute) - single: __kwdefaults__ (function attribute) + single: __closure__ (function attribute) + single: func_doc (function attribute) + single: func_name (function attribute) + single: func_dict (function attribute) + single: func_defaults (function attribute) + single: func_code (function attribute) + single: func_globals (function attribute) + single: func_closure (function attribute) pair: global; namespace - +-------------------------+-------------------------------+-----------+ - | Attribute | Meaning | | - +=========================+===============================+===========+ - | :attr:`__doc__` | The function's documentation | Writable | - | | string, or ``None`` if | | - | | unavailable; not inherited by | | - | | subclasses. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`~definition.\ | The function's name. | Writable | - | __name__` | | | - +-------------------------+-------------------------------+-----------+ - | :attr:`~definition.\ | The function's | Writable | - | __qualname__` | :term:`qualified name`. | | - | | | | - | | .. versionadded:: 3.3 | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__module__` | The name of the module the | Writable | - | | function was defined in, or | | - | | ``None`` if unavailable. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__defaults__` | A tuple containing default | Writable | - | | argument values for those | | - | | arguments that have defaults, | | - | | or ``None`` if no arguments | | - | | have a default value. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__code__` | The code object representing | Writable | - | | the compiled function body. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__globals__` | A reference to the dictionary | Read-only | - | | that holds the function's | | - | | global variables --- the | | - | | global namespace of the | | - | | module in which the function | | - | | was defined. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`~object.__dict__`| The namespace supporting | Writable | - | | arbitrary function | | - | | attributes. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__closure__` | ``None`` or a tuple of cells | Read-only | - | | that contain bindings for the | | - | | function's free variables. | | - | | See below for information on | | - | | the ``cell_contents`` | | - | | attribute. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__annotations__` | A dict containing annotations | Writable | - | | of parameters. The keys of | | - | | the dict are the parameter | | - | | names, and ``'return'`` for | | - | | the return annotation, if | | - | | provided. | | - +-------------------------+-------------------------------+-----------+ - | :attr:`__kwdefaults__` | A dict containing defaults | Writable | - | | for keyword-only parameters. | | - +-------------------------+-------------------------------+-----------+ + +-----------------------+-------------------------------+-----------+ + | Attribute | Meaning | | + +=======================+===============================+===========+ + | :attr:`__doc__` | The function's documentation | Writable | + | :attr:`func_doc` | string, or ``None`` if | | + | | unavailable. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`~definition.\ | The function's name | Writable | + | __name__` | | | + | :attr:`func_name` | | | + +-----------------------+-------------------------------+-----------+ + | :attr:`__module__` | The name of the module the | Writable | + | | function was defined in, or | | + | | ``None`` if unavailable. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`__defaults__` | A tuple containing default | Writable | + | :attr:`func_defaults` | argument values for those | | + | | arguments that have defaults, | | + | | or ``None`` if no arguments | | + | | have a default value. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`__code__` | The code object representing | Writable | + | :attr:`func_code` | the compiled function body. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`__globals__` | A reference to the dictionary | Read-only | + | :attr:`func_globals` | that holds the function's | | + | | global variables --- the | | + | | global namespace of the | | + | | module in which the function | | + | | was defined. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`~object.\ | The namespace supporting | Writable | + | __dict__` | arbitrary function | | + | :attr:`func_dict` | attributes. | | + +-----------------------+-------------------------------+-----------+ + | :attr:`__closure__` | ``None`` or a tuple of cells | Read-only | + | :attr:`func_closure` | that contain bindings for the | | + | | function's free variables. | | + +-----------------------+-------------------------------+-----------+ Most of the attributes labelled "Writable" check the type of the assigned value. + .. versionchanged:: 2.4 + ``func_name`` is now writable. + + .. versionchanged:: 2.6 + The double-underscore attributes ``__closure__``, ``__code__``, + ``__defaults__``, and ``__globals__`` were introduced as aliases for + the corresponding ``func_*`` attributes for forwards compatibility + with Python 3. + Function objects also support getting and setting arbitrary attributes, which can be used, for example, to attach metadata to functions. Regular attribute dot-notation is used to get and set such attributes. *Note that the current implementation only supports function attributes on user-defined functions. Function attributes on built-in functions may be supported in the future.* - A cell object has the attribute ``cell_contents``. This can be used to get - the value of the cell, as well as set the value. - Additional information about a function's definition can be retrieved from its - code object; see the description of internal types below. The - :data:`cell <types.CellType>` type can be accessed in the :mod:`types` - module. + code object; see the description of internal types below. - Instance methods + User-defined methods .. index:: object: method object: user-defined method pair: user-defined; method - An instance method object combines a class, a class instance and any - callable object (normally a user-defined function). + A user-defined method object combines a class, a class instance (or ``None``) + and any callable object (normally a user-defined function). + + Special read-only attributes: :attr:`im_self` is the class instance object, + :attr:`im_func` is the function object; :attr:`im_class` is the class of + :attr:`im_self` for bound methods or the class that asked for the method for + unbound methods; :attr:`__doc__` is the method's documentation (same as + ``im_func.__doc__``); :attr:`~definition.__name__` is the method name (same as + ``im_func.__name__``); :attr:`__module__` is the name of the module the method + was defined in, or ``None`` if unavailable. + + .. versionchanged:: 2.2 + :attr:`im_self` used to refer to the class that defined the method. + + .. versionchanged:: 2.6 + For Python 3 forward-compatibility, :attr:`im_func` is also available as + :attr:`__func__`, and :attr:`im_self` as :attr:`__self__`. .. index:: - single: __func__ (method attribute) - single: __self__ (method attribute) single: __doc__ (method attribute) single: __name__ (method attribute) single: __module__ (method attribute) - - Special read-only attributes: :attr:`__self__` is the class instance object, - :attr:`__func__` is the function object; :attr:`__doc__` is the method's - documentation (same as ``__func__.__doc__``); :attr:`~definition.__name__` is the - method name (same as ``__func__.__name__``); :attr:`__module__` is the - name of the module the method was defined in, or ``None`` if unavailable. + single: im_func (method attribute) + single: im_self (method attribute) Methods also support accessing (but not setting) the arbitrary function attributes on the underlying function object. - User-defined method objects may be created when getting an attribute of a - class (perhaps via an instance of that class), if that attribute is a - user-defined function object or a class method object. - - When an instance method object is created by retrieving a user-defined - function object from a class via one of its instances, its - :attr:`__self__` attribute is the instance, and the method object is said - to be bound. The new method's :attr:`__func__` attribute is the original - function object. - - When an instance method object is created by retrieving a class method - object from a class or instance, its :attr:`__self__` attribute is the - class itself, and its :attr:`__func__` attribute is the function object - underlying the class method. - - When an instance method object is called, the underlying function - (:attr:`__func__`) is called, inserting the class instance - (:attr:`__self__`) in front of the argument list. For instance, when - :class:`C` is a class which contains a definition for a function - :meth:`f`, and ``x`` is an instance of :class:`C`, calling ``x.f(1)`` is - equivalent to calling ``C.f(x, 1)``. - - When an instance method object is derived from a class method object, the - "class instance" stored in :attr:`__self__` will actually be the class - itself, so that calling either ``x.f(1)`` or ``C.f(1)`` is equivalent to - calling ``f(C,1)`` where ``f`` is the underlying function. - - Note that the transformation from function object to instance method - object happens each time the attribute is retrieved from the instance. In - some cases, a fruitful optimization is to assign the attribute to a local - variable and call that local variable. Also notice that this - transformation only happens for user-defined functions; other callable - objects (and all non-callable objects) are retrieved without - transformation. It is also important to note that user-defined functions - which are attributes of a class instance are not converted to bound - methods; this *only* happens when the function is an attribute of the - class. + User-defined method objects may be created when getting an attribute of a class + (perhaps via an instance of that class), if that attribute is a user-defined + function object, an unbound user-defined method object, or a class method + object. When the attribute is a user-defined method object, a new method object + is only created if the class from which it is being retrieved is the same as, or + a derived class of, the class stored in the original method object; otherwise, + the original method object is used as it is. + + .. index:: + single: im_class (method attribute) + single: im_func (method attribute) + single: im_self (method attribute) + + When a user-defined method object is created by retrieving a user-defined + function object from a class, its :attr:`im_self` attribute is ``None`` + and the method object is said to be unbound. When one is created by + retrieving a user-defined function object from a class via one of its + instances, its :attr:`im_self` attribute is the instance, and the method + object is said to be bound. In either case, the new method's + :attr:`im_class` attribute is the class from which the retrieval takes + place, and its :attr:`im_func` attribute is the original function object. + + .. index:: single: im_func (method attribute) + + When a user-defined method object is created by retrieving another method object + from a class or instance, the behaviour is the same as for a function object, + except that the :attr:`im_func` attribute of the new instance is not the + original method object but its :attr:`im_func` attribute. + + .. index:: + single: im_class (method attribute) + single: im_func (method attribute) + single: im_self (method attribute) + + When a user-defined method object is created by retrieving a class method object + from a class or instance, its :attr:`im_self` attribute is the class itself, and + its :attr:`im_func` attribute is the function object underlying the class method. + + When an unbound user-defined method object is called, the underlying function + (:attr:`im_func`) is called, with the restriction that the first argument must + be an instance of the proper class (:attr:`im_class`) or of a derived class + thereof. + + When a bound user-defined method object is called, the underlying function + (:attr:`im_func`) is called, inserting the class instance (:attr:`im_self`) in + front of the argument list. For instance, when :class:`C` is a class which + contains a definition for a function :meth:`f`, and ``x`` is an instance of + :class:`C`, calling ``x.f(1)`` is equivalent to calling ``C.f(x, 1)``. + + When a user-defined method object is derived from a class method object, the + "class instance" stored in :attr:`im_self` will actually be the class itself, so + that calling either ``x.f(1)`` or ``C.f(1)`` is equivalent to calling ``f(C,1)`` + where ``f`` is the underlying function. + + Note that the transformation from function object to (unbound or bound) method + object happens each time the attribute is retrieved from the class or instance. + In some cases, a fruitful optimization is to assign the attribute to a local + variable and call that local variable. Also notice that this transformation only + happens for user-defined functions; other callable objects (and all non-callable + objects) are retrieved without transformation. It is also important to note + that user-defined functions which are attributes of a class instance are not + converted to bound methods; this *only* happens when the function is an + attribute of the class. Generator functions .. index:: @@ -612,44 +663,16 @@ Callable types single: generator; iterator A function or method which uses the :keyword:`yield` statement (see section - :ref:`yield`) is called a :dfn:`generator function`. Such a function, when - called, always returns an iterator object which can be used to execute the - body of the function: calling the iterator's :meth:`iterator.__next__` - method will cause the function to execute until it provides a value - using the :keyword:`!yield` statement. When the function executes a + :ref:`yield`) is called a :dfn:`generator + function`. Such a function, when called, always returns an iterator object + which can be used to execute the body of the function: calling the iterator's + :meth:`~iterator.next` method will cause the function to execute until + it provides a value + using the :keyword:`yield` statement. When the function executes a :keyword:`return` statement or falls off the end, a :exc:`StopIteration` exception is raised and the iterator will have reached the end of the set of values to be returned. - Coroutine functions - .. index:: - single: coroutine; function - - A function or method which is defined using :keyword:`async def` is called - a :dfn:`coroutine function`. Such a function, when called, returns a - :term:`coroutine` object. It may contain :keyword:`await` expressions, - as well as :keyword:`async with` and :keyword:`async for` statements. See - also the :ref:`coroutine-objects` section. - - Asynchronous generator functions - .. index:: - single: asynchronous generator; function - single: asynchronous generator; asynchronous iterator - - A function or method which is defined using :keyword:`async def` and - which uses the :keyword:`yield` statement is called a - :dfn:`asynchronous generator function`. Such a function, when called, - returns an asynchronous iterator object which can be used in an - :keyword:`async for` statement to execute the body of the function. - - Calling the asynchronous iterator's :meth:`aiterator.__anext__` method - will return an :term:`awaitable` which when awaited - will execute until it provides a value using the :keyword:`yield` - expression. When the function executes an empty :keyword:`return` - statement or falls off the end, a :exc:`StopAsyncIteration` exception - is raised and the asynchronous iterator will have reached the end of - the set of values to be yielded. - Built-in functions .. index:: object: built-in function @@ -677,60 +700,53 @@ Callable types this case, the special read-only attribute :attr:`__self__` is set to the object denoted by *alist*. - Classes - Classes are callable. These objects normally act as factories for new - instances of themselves, but variations are possible for class types that - override :meth:`__new__`. The arguments of the call are passed to - :meth:`__new__` and, in the typical case, to :meth:`__init__` to - initialize the new instance. - - Class Instances - Instances of arbitrary classes can be made callable by defining a - :meth:`__call__` method in their class. + Class Types + Class types, or "new-style classes," are callable. These objects normally act + as factories for new instances of themselves, but variations are possible for + class types that override :meth:`__new__`. The arguments of the call are passed + to :meth:`__new__` and, in the typical case, to :meth:`__init__` to initialize + the new instance. + Classic Classes + .. index:: + single: __init__() (object method) + object: class + object: class instance + object: instance + pair: class object; call + + Class objects are described below. When a class object is called, a new class + instance (also described below) is created and returned. This implies a call to + the class's :meth:`__init__` method if it has one. Any arguments are passed on + to the :meth:`__init__` method. If there is no :meth:`__init__` method, the + class must be called without arguments. + + Class instances + Class instances are described below. Class instances are callable only when the + class has a :meth:`__call__` method; ``x(arguments)`` is a shorthand for + ``x.__call__(arguments)``. Modules .. index:: statement: import object: module - Modules are a basic organizational unit of Python code, and are created by - the :ref:`import system <importsystem>` as invoked either by the - :keyword:`import` statement, or by calling - functions such as :func:`importlib.import_module` and built-in - :func:`__import__`. A module object has a namespace implemented by a - dictionary object (this is the dictionary referenced by the ``__globals__`` - attribute of functions defined in the module). Attribute references are - translated to lookups in this dictionary, e.g., ``m.x`` is equivalent to - ``m.__dict__["x"]``. A module object does not contain the code object used - to initialize the module (since it isn't needed once the initialization is - done). - - Attribute assignment updates the module's namespace dictionary, e.g., - ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``. + Modules are imported by the :keyword:`import` statement (see section + :ref:`import`). A module object has a + namespace implemented by a dictionary object (this is the dictionary referenced + by the func_globals attribute of functions defined in the module). Attribute + references are translated to lookups in this dictionary, e.g., ``m.x`` is + equivalent to ``m.__dict__["x"]``. A module object does not contain the code + object used to initialize the module (since it isn't needed once the + initialization is done). - .. index:: - single: __name__ (module attribute) - single: __doc__ (module attribute) - single: __file__ (module attribute) - single: __annotations__ (module attribute) - pair: module; namespace - - Predefined (writable) attributes: :attr:`__name__` is the module's name; - :attr:`__doc__` is the module's documentation string, or ``None`` if - unavailable; :attr:`__annotations__` (optional) is a dictionary containing - :term:`variable annotations <variable annotation>` collected during module - body execution; :attr:`__file__` is the pathname of the file from which the - module was loaded, if it was loaded from a file. The :attr:`__file__` - attribute may be missing for certain types of modules, such as C modules - that are statically linked into the interpreter; for extension modules - loaded dynamically from a shared library, it is the pathname of the shared - library file. + Attribute assignment updates the module's namespace dictionary, e.g., ``m.x = + 1`` is equivalent to ``m.__dict__["x"] = 1``. .. index:: single: __dict__ (module attribute) - Special read-only attribute: :attr:`~object.__dict__` is the module's - namespace as a dictionary object. + Special read-only attribute: :attr:`~object.__dict__` is the module's namespace as a + dictionary object. .. impl-detail:: @@ -739,19 +755,36 @@ Modules dictionary still has live references. To avoid this, copy the dictionary or keep the module around while using its dictionary directly. -Custom classes - Custom class types are typically created by class definitions (see section + .. index:: + single: __name__ (module attribute) + single: __doc__ (module attribute) + single: __file__ (module attribute) + pair: module; namespace + + Predefined (writable) attributes: :attr:`__name__` is the module's name; + :attr:`__doc__` is the module's documentation string, or ``None`` if + unavailable; :attr:`__file__` is the pathname of the file from which the module + was loaded, if it was loaded from a file. The :attr:`__file__` attribute is not + present for C modules that are statically linked into the interpreter; for + extension modules loaded dynamically from a shared library, it is the pathname + of the shared library file. + +Classes + Both class types (new-style classes) and class objects (old-style/classic + classes) are typically created by class definitions (see section :ref:`class`). A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., - ``C.x`` is translated to ``C.__dict__["x"]`` (although there are a number of - hooks which allow for other means of locating attributes). When the attribute - name is not found there, the attribute search continues in the base classes. - This search of the base classes uses the C3 method resolution order which - behaves correctly even in the presence of 'diamond' inheritance structures - where there are multiple inheritance paths leading back to a common ancestor. - Additional details on the C3 MRO used by Python can be found in the - documentation accompanying the 2.3 release at - https://www.python.org/download/releases/2.3/mro/. + ``C.x`` is translated to ``C.__dict__["x"]`` (although for new-style classes + in particular there are a number of hooks which allow for other means of + locating attributes). When the attribute name is not found there, the + attribute search continues in the base classes. For old-style classes, the + search is depth-first, left-to-right in the order of occurrence in the base + class list. New-style classes use the more complex C3 method resolution + order which behaves correctly even in the presence of 'diamond' + inheritance structures where there are multiple inheritance paths + leading back to a common ancestor. Additional details on the C3 MRO used by + new-style classes can be found in the documentation accompanying the + 2.3 release at https://www.python.org/download/releases/2.3/mro/. .. XXX: Could we add that MRO doc as an appendix to the language ref? @@ -765,12 +798,16 @@ Custom classes pair: class; attribute When a class attribute reference (for class :class:`C`, say) would yield a - class method object, it is transformed into an instance method object whose - :attr:`__self__` attribute is :class:`C`. When it would yield a static - method object, it is transformed into the object wrapped by the static method - object. See section :ref:`descriptors` for another way in which attributes - retrieved from a class may differ from those actually contained in its - :attr:`~object.__dict__`. + user-defined function object or an unbound user-defined method object whose + associated class is either :class:`C` or one of its base classes, it is + transformed into an unbound user-defined method object whose :attr:`im_class` + attribute is :class:`C`. When it would yield a class method object, it is + transformed into a bound user-defined method object whose + :attr:`im_self` attribute is :class:`C`. When it would yield a + static method object, it is transformed into the object wrapped by the static + method object. See section :ref:`descriptors` for another way in which + attributes retrieved from a class may differ from those actually contained in + its :attr:`~object.__dict__` (note that only new-style classes support descriptors). .. index:: triple: class; attribute; assignment @@ -787,16 +824,13 @@ Custom classes single: __dict__ (class attribute) single: __bases__ (class attribute) single: __doc__ (class attribute) - single: __annotations__ (class attribute) Special attributes: :attr:`~definition.__name__` is the class name; :attr:`__module__` is the module name in which the class was defined; :attr:`~object.__dict__` is the dictionary containing the class's namespace; :attr:`~class.__bases__` is a - tuple containing the base classes, in the order of their occurrence in the - base class list; :attr:`__doc__` is the class's documentation string, - or ``None`` if undefined; :attr:`__annotations__` (optional) is a dictionary - containing :term:`variable annotations <variable annotation>` collected during - class body execution. + tuple (possibly empty or a singleton) containing the base classes, in the + order of their occurrence in the base class list; :attr:`__doc__` is the + class's documentation string, or ``None`` if undefined. Class instances .. index:: @@ -805,19 +839,22 @@ Class instances pair: class; instance pair: class instance; attribute - A class instance is created by calling a class object (see above). A class - instance has a namespace implemented as a dictionary which is the first place - in which attribute references are searched. When an attribute is not found - there, and the instance's class has an attribute by that name, the search - continues with the class attributes. If a class attribute is found that is a - user-defined function object, it is transformed into an instance method - object whose :attr:`__self__` attribute is the instance. Static method and - class method objects are also transformed; see above under "Classes". See - section :ref:`descriptors` for another way in which attributes of a class - retrieved via its instances may differ from the objects actually stored in - the class's :attr:`~object.__dict__`. If no class attribute is found, and the - object's class has a :meth:`__getattr__` method, that is called to satisfy - the lookup. + A class instance is created by calling a class object (see above). A class + instance has a namespace implemented as a dictionary which is the first place in + which attribute references are searched. When an attribute is not found there, + and the instance's class has an attribute by that name, the search continues + with the class attributes. If a class attribute is found that is a user-defined + function object or an unbound user-defined method object whose associated class + is the class (call it :class:`C`) of the instance for which the attribute + reference was initiated or one of its bases, it is transformed into a bound + user-defined method object whose :attr:`im_class` attribute is :class:`C` and + whose :attr:`im_self` attribute is the instance. Static method and class method + objects are also transformed, as if they had been retrieved from class + :class:`C`; see above under "Classes". See section :ref:`descriptors` for + another way in which attributes of a class retrieved via its instances may + differ from the objects actually stored in the class's :attr:`~object.__dict__`. If no + class attribute is found, and the object's class has a :meth:`__getattr__` + method, that is called to satisfy the lookup. .. index:: triple: class instance; attribute; assignment @@ -841,10 +878,10 @@ Class instances Special attributes: :attr:`~object.__dict__` is the attribute dictionary; :attr:`~instance.__class__` is the instance's class. -I/O objects (also known as file objects) +Files .. index:: + object: file builtin: open - module: io single: popen() (in module os) single: makefile() (socket method) single: sys.stdin @@ -855,17 +892,14 @@ I/O objects (also known as file objects) single: stdout (in module sys) single: stderr (in module sys) - A :term:`file object` represents an open file. Various shortcuts are - available to create file objects: the :func:`open` built-in function, and - also :func:`os.popen`, :func:`os.fdopen`, and the - :meth:`~socket.socket.makefile` method of socket objects (and perhaps by - other functions or methods provided by extension modules). - - The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are - initialized to file objects corresponding to the interpreter's standard - input, output and error streams; they are all open in text mode and - therefore follow the interface defined by the :class:`io.TextIOBase` - abstract class. + A file object represents an open file. File objects are created by the + :func:`open` built-in function, and also by :func:`os.popen`, + :func:`os.fdopen`, and the :meth:`makefile` method of socket objects (and + perhaps by other functions or methods provided by extension modules). The + objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are initialized to + file objects corresponding to the interpreter's standard input, output and + error streams. See :ref:`bltin-file-objects` for complete documentation of + file objects. Internal types .. index:: @@ -890,8 +924,6 @@ Internal types .. index:: single: co_argcount (code object attribute) - single: co_posonlyargcount (code object attribute) - single: co_kwonlyargcount (code object attribute) single: co_code (code object attribute) single: co_consts (code object attribute) single: co_filename (code object attribute) @@ -907,26 +939,21 @@ Internal types single: co_freevars (code object attribute) Special read-only attributes: :attr:`co_name` gives the function name; - :attr:`co_argcount` is the total number of positional arguments - (including positional-only arguments and arguments with default values); - :attr:`co_posonlyargcount` is the number of positional-only arguments - (including arguments with default values); :attr:`co_kwonlyargcount` is - the number of keyword-only arguments (including arguments with default - values); :attr:`co_nlocals` is the number of local variables used by the - function (including arguments); :attr:`co_varnames` is a tuple containing + :attr:`co_argcount` is the number of positional arguments (including arguments + with default values); :attr:`co_nlocals` is the number of local variables used + by the function (including arguments); :attr:`co_varnames` is a tuple containing the names of the local variables (starting with the argument names); - :attr:`co_cellvars` is a tuple containing the names of local variables - that are referenced by nested functions; :attr:`co_freevars` is a tuple - containing the names of free variables; :attr:`co_code` is a string - representing the sequence of bytecode instructions; :attr:`co_consts` is - a tuple containing the literals used by the bytecode; :attr:`co_names` is - a tuple containing the names used by the bytecode; :attr:`co_filename` is - the filename from which the code was compiled; :attr:`co_firstlineno` is - the first line number of the function; :attr:`co_lnotab` is a string - encoding the mapping from bytecode offsets to line numbers (for details - see the source code of the interpreter); :attr:`co_stacksize` is the - required stack size (including local variables); :attr:`co_flags` is an - integer encoding a number of flags for the interpreter. + :attr:`co_cellvars` is a tuple containing the names of local variables that are + referenced by nested functions; :attr:`co_freevars` is a tuple containing the + names of free variables; :attr:`co_code` is a string representing the sequence + of bytecode instructions; :attr:`co_consts` is a tuple containing the literals + used by the bytecode; :attr:`co_names` is a tuple containing the names used by + the bytecode; :attr:`co_filename` is the filename from which the code was + compiled; :attr:`co_firstlineno` is the first line number of the function; + :attr:`co_lnotab` is a string encoding the mapping from bytecode offsets to + line numbers (for details see the source code of the interpreter); + :attr:`co_stacksize` is the required stack size (including local variables); + :attr:`co_flags` is an integer encoding a number of flags for the interpreter. .. index:: object: generator @@ -955,7 +982,7 @@ Internal types .. index:: object: frame Frame objects represent execution frames. They may occur in traceback objects - (see below), and are also passed to registered trace functions. + (see below). .. index:: single: f_back (frame attribute) @@ -964,51 +991,32 @@ Internal types single: f_locals (frame attribute) single: f_lasti (frame attribute) single: f_builtins (frame attribute) + single: f_restricted (frame attribute) Special read-only attributes: :attr:`f_back` is to the previous stack frame (towards the caller), or ``None`` if this is the bottom stack frame; :attr:`f_code` is the code object being executed in this frame; :attr:`f_locals` is the dictionary used to look up local variables; :attr:`f_globals` is used for global variables; :attr:`f_builtins` is used for built-in (intrinsic) names; - :attr:`f_lasti` gives the precise instruction (this is an index into the - bytecode string of the code object). + :attr:`f_restricted` is a flag indicating whether the function is executing in + restricted execution mode; :attr:`f_lasti` gives the precise instruction (this + is an index into the bytecode string of the code object). .. index:: single: f_trace (frame attribute) - single: f_trace_lines (frame attribute) - single: f_trace_opcodes (frame attribute) + single: f_exc_type (frame attribute) + single: f_exc_value (frame attribute) + single: f_exc_traceback (frame attribute) single: f_lineno (frame attribute) Special writable attributes: :attr:`f_trace`, if not ``None``, is a function - called for various events during code execution (this is used by the debugger). - Normally an event is triggered for each new source line - this can be - disabled by setting :attr:`f_trace_lines` to :const:`False`. - - Implementations *may* allow per-opcode events to be requested by setting - :attr:`f_trace_opcodes` to :const:`True`. Note that this may lead to - undefined interpreter behaviour if exceptions raised by the trace - function escape to the function being traced. - - :attr:`f_lineno` is the current line number of the frame --- writing to this - from within a trace function jumps to the given line (only for the bottom-most - frame). A debugger can implement a Jump command (aka Set Next Statement) - by writing to f_lineno. - - Frame objects support one method: - - .. method:: frame.clear() - - This method clears all references to local variables held by the - frame. Also, if the frame belonged to a generator, the generator - is finalized. This helps break reference cycles involving frame - objects (for example when catching an exception and storing its - traceback for later use). - - :exc:`RuntimeError` is raised if the frame is currently executing. - - .. versionadded:: 3.4 - - .. _traceback-objects: + called at the start of each source code line (this is used by the debugger); + :attr:`f_exc_type`, :attr:`f_exc_value`, :attr:`f_exc_traceback` represent the + last exception raised in the parent frame provided another exception was ever + raised in the current frame (in all other cases they are ``None``); :attr:`f_lineno` + is the current line number of the frame --- writing to this from within a trace + function jumps to the given line (only for the bottom-most frame). A debugger + can implement a Jump command (aka Set Next Statement) by writing to f_lineno. Traceback objects .. index:: @@ -1017,62 +1025,48 @@ Internal types pair: exception; handler pair: execution; stack single: exc_info (in module sys) + single: exc_traceback (in module sys) single: last_traceback (in module sys) single: sys.exc_info + single: sys.exc_traceback single: sys.last_traceback Traceback objects represent a stack trace of an exception. A traceback object - is implicitly created when an exception occurs, and may also be explicitly - created by calling :class:`types.TracebackType`. - - For implicitly created tracebacks, when the search for an exception handler + is created when an exception occurs. When the search for an exception handler unwinds the execution stack, at each unwound level a traceback object is inserted in front of the current traceback. When an exception handler is entered, the stack trace is made available to the program. (See section - :ref:`try`.) It is accessible as the third item of the - tuple returned by ``sys.exc_info()``, and as the ``__traceback__`` attribute - of the caught exception. - - When the program contains no suitable - handler, the stack trace is written (nicely formatted) to the standard error - stream; if the interpreter is interactive, it is also made available to the user - as ``sys.last_traceback``. - - For explicitly created tracebacks, it is up to the creator of the traceback - to determine how the ``tb_next`` attributes should be linked to form a - full stack trace. + :ref:`try`.) It is accessible as ``sys.exc_traceback``, + and also as the third item of the tuple returned by ``sys.exc_info()``. The + latter is the preferred interface, since it works correctly when the program is + using multiple threads. When the program contains no suitable handler, the stack + trace is written (nicely formatted) to the standard error stream; if the + interpreter is interactive, it is also made available to the user as + ``sys.last_traceback``. .. index:: + single: tb_next (traceback attribute) single: tb_frame (traceback attribute) single: tb_lineno (traceback attribute) single: tb_lasti (traceback attribute) statement: try - Special read-only attributes: - :attr:`tb_frame` points to the execution frame of the current level; - :attr:`tb_lineno` gives the line number where the exception occurred; - :attr:`tb_lasti` indicates the precise instruction. - The line number and last instruction in the traceback may differ from the - line number of its frame object if the exception occurred in a - :keyword:`try` statement with no matching except clause or with a - finally clause. - - .. index:: - single: tb_next (traceback attribute) - - Special writable attribute: :attr:`tb_next` is the next level in the stack - trace (towards the frame where the exception occurred), or ``None`` if - there is no next level. - - .. versionchanged:: 3.7 - Traceback objects can now be explicitly instantiated from Python code, - and the ``tb_next`` attribute of existing instances can be updated. + Special read-only attributes: :attr:`tb_next` is the next level in the stack + trace (towards the frame where the exception occurred), or ``None`` if there is + no next level; :attr:`tb_frame` points to the execution frame of the current + level; :attr:`tb_lineno` gives the line number where the exception occurred; + :attr:`tb_lasti` indicates the precise instruction. The line number and last + instruction in the traceback may differ from the line number of its frame object + if the exception occurred in a :keyword:`try` statement with no matching except + clause or with a finally clause. Slice objects .. index:: builtin: slice - Slice objects are used to represent slices for :meth:`__getitem__` - methods. They are also created by the built-in :func:`slice` function. + Slice objects are used to represent slices when *extended slice syntax* is used. + This is a slice using two colons, or multiple slices or ellipses separated by + commas, e.g., ``a[i:j:step]``, ``a[i:j, k:l]``, or ``a[..., i:j]``. They are + also created by the built-in :func:`slice` function. .. index:: single: start (slice object attribute) @@ -1085,14 +1079,17 @@ Internal types Slice objects support one method: + .. method:: slice.indices(self, length) - This method takes a single integer argument *length* and computes - information about the slice that the slice object would describe if - applied to a sequence of *length* items. It returns a tuple of three - integers; respectively these are the *start* and *stop* indices and the - *step* or stride length of the slice. Missing or out-of-bounds indices - are handled in a manner consistent with regular slices. + This method takes a single integer argument *length* and computes information + about the extended slice that the slice object would describe if applied to a + sequence of *length* items. It returns a tuple of three integers; respectively + these are the *start* and *stop* indices and the *step* or stride length of the + slice. Missing or out-of-bounds indices are handled in a manner consistent with + regular slices. + + .. versionadded:: 2.3 Static method objects Static method objects provide a way of defeating the transformation of function @@ -1112,6 +1109,55 @@ Internal types by the built-in :func:`classmethod` constructor. +.. _newstyle: + +New-style and classic classes +============================= + +Classes and instances come in two flavors: old-style (or classic) and new-style. + +Up to Python 2.1 the concept of ``class`` was unrelated to the concept of +``type``, and old-style classes were the only flavor available. For an +old-style class, the statement ``x.__class__`` provides the class of *x*, but +``type(x)`` is always ``<type 'instance'>``. This reflects the fact that all +old-style instances, independent of their class, are implemented with a single +built-in type, called ``instance``. + +New-style classes were introduced in Python 2.2 to unify the concepts of +``class`` and ``type``. A new-style class is simply a user-defined type, +no more, no less. If *x* is an instance of a new-style class, then ``type(x)`` +is typically the same as ``x.__class__`` (although this is not guaranteed -- a +new-style class instance is permitted to override the value returned for +``x.__class__``). + +The major motivation for introducing new-style classes is to provide a unified +object model with a full meta-model. It also has a number of practical +benefits, like the ability to subclass most built-in types, or the introduction +of "descriptors", which enable computed properties. + +For compatibility reasons, classes are still old-style by default. New-style +classes are created by specifying another new-style class (i.e. a type) as a +parent class, or the "top-level type" :class:`object` if no other parent is +needed. The behaviour of new-style classes differs from that of old-style +classes in a number of important details in addition to what :func:`type` +returns. Some of these changes are fundamental to the new object model, like +the way special methods are invoked. Others are "fixes" that could not be +implemented before for compatibility concerns, like the method resolution order +in case of multiple inheritance. + +While this manual aims to provide comprehensive coverage of Python's class +mechanics, it may still be lacking in some areas when it comes to its coverage +of new-style classes. Please see https://www.python.org/doc/newstyle/ for +sources of additional information. + +.. index:: + single: class; new-style + single: class; classic + single: class; old-style + +Old-style classes are removed in Python 3, leaving only new-style classes. + + .. _specialnames: Special method names @@ -1127,16 +1173,11 @@ with special names. This is Python's approach to :dfn:`operator overloading`, allowing classes to define their own behavior with respect to language operators. For instance, if a class defines a method named :meth:`__getitem__`, and ``x`` is an instance of this class, then ``x[i]`` is roughly equivalent -to ``type(x).__getitem__(x, i)``. Except where mentioned, attempts to execute an +to ``x.__getitem__(i)`` for old-style classes and ``type(x).__getitem__(x, i)`` +for new-style classes. Except where mentioned, attempts to execute an operation raise an exception when no appropriate method is defined (typically :exc:`AttributeError` or :exc:`TypeError`). -Setting a special method to ``None`` indicates that the corresponding -operation is not available. For example, if a class sets -:meth:`__iter__` to ``None``, the class is not iterable, so calling -:func:`iter` on its instances will raise a :exc:`TypeError` (without -falling back to :meth:`__getitem__`). [#]_ - When implementing a class that emulates any built-in type, it is important that the emulation only be implemented to the degree that it makes sense for the object being modelled. For example, some sequences may work well with retrieval @@ -1162,14 +1203,14 @@ Basic customization (usually an instance of *cls*). Typical implementations create a new instance of the class by invoking the - superclass's :meth:`__new__` method using ``super().__new__(cls[, ...])`` - with appropriate arguments and then modifying the newly-created instance - as necessary before returning it. + superclass's :meth:`__new__` method using ``super(currentclass, + cls).__new__(cls[, ...])`` with appropriate arguments and then modifying the + newly-created instance as necessary before returning it. - If :meth:`__new__` is invoked during object construction and it returns an - instance or subclass of *cls*, then the new instance’s :meth:`__init__` method - will be invoked like ``__init__(self[, ...])``, where *self* is the new instance - and the remaining arguments are the same as were passed to the object constructor. + If :meth:`__new__` returns an instance of *cls*, then the new instance's + :meth:`__init__` method will be invoked like ``__init__(self[, ...])``, where + *self* is the new instance and the remaining arguments are the same as were + passed to :meth:`__new__`. If :meth:`__new__` does not return an instance of *cls*, then the new instance's :meth:`__init__` method will not be invoked. @@ -1185,13 +1226,13 @@ Basic customization Called after the instance has been created (by :meth:`__new__`), but before it is returned to the caller. The arguments are those passed to the - class constructor expression. If a base class has an :meth:`__init__` - method, the derived class's :meth:`__init__` method, if any, must explicitly - call it to ensure proper initialization of the base class part of the - instance; for example: ``super().__init__([args...])``. + class constructor expression. If a base class has an :meth:`__init__` method, + the derived class's :meth:`__init__` method, if any, must explicitly call it to + ensure proper initialization of the base class part of the instance; for + example: ``BaseClass.__init__(self, [args...])``. Because :meth:`__new__` and :meth:`__init__` work together in constructing - objects (:meth:`__new__` to create it, and :meth:`__init__` to customize it), + objects (:meth:`__new__` to create it, and :meth:`__init__` to customise it), no non-``None`` value may be returned by :meth:`__init__`; doing so will cause a :exc:`TypeError` to be raised at runtime. @@ -1200,145 +1241,97 @@ Basic customization .. index:: single: destructor - single: finalizer statement: del Called when the instance is about to be destroyed. This is also called a - finalizer or (improperly) a destructor. If a base class has a - :meth:`__del__` method, the derived class's :meth:`__del__` method, - if any, must explicitly call it to ensure proper deletion of the base - class part of the instance. - - It is possible (though not recommended!) for the :meth:`__del__` method - to postpone destruction of the instance by creating a new reference to - it. This is called object *resurrection*. It is implementation-dependent - whether :meth:`__del__` is called a second time when a resurrected object - is about to be destroyed; the current :term:`CPython` implementation - only calls it once. - - It is not guaranteed that :meth:`__del__` methods are called for objects - that still exist when the interpreter exits. + destructor. If a base class has a :meth:`__del__` method, the derived class's + :meth:`__del__` method, if any, must explicitly call it to ensure proper + deletion of the base class part of the instance. Note that it is possible + (though not recommended!) for the :meth:`__del__` method to postpone destruction + of the instance by creating a new reference to it. It may then be called at a + later time when this new reference is deleted. It is not guaranteed that + :meth:`__del__` methods are called for objects that still exist when the + interpreter exits. .. note:: ``del x`` doesn't directly call ``x.__del__()`` --- the former decrements the reference count for ``x`` by one, and the latter is only called when - ``x``'s reference count reaches zero. - - .. impl-detail:: - It is possible for a reference cycle to prevent the reference count - of an object from going to zero. In this case, the cycle will be - later detected and deleted by the :term:`cyclic garbage collector - <garbage collection>`. A common cause of reference cycles is when - an exception has been caught in a local variable. The frame's - locals then reference the exception, which references its own - traceback, which references the locals of all frames caught in the - traceback. - - .. seealso:: - Documentation for the :mod:`gc` module. + ``x``'s reference count reaches zero. Some common situations that may + prevent the reference count of an object from going to zero include: + circular references between objects (e.g., a doubly-linked list or a tree + data structure with parent and child pointers); a reference to the object + on the stack frame of a function that caught an exception (the traceback + stored in ``sys.exc_traceback`` keeps the stack frame alive); or a + reference to the object on the stack frame that raised an unhandled + exception in interactive mode (the traceback stored in + ``sys.last_traceback`` keeps the stack frame alive). The first situation + can only be remedied by explicitly breaking the cycles; the latter two + situations can be resolved by storing ``None`` in ``sys.exc_traceback`` or + ``sys.last_traceback``. Circular references which are garbage are + detected when the option cycle detector is enabled (it's on by default), + but can only be cleaned up if there are no Python-level :meth:`__del__` + methods involved. Refer to the documentation for the :mod:`gc` module for + more information about how :meth:`__del__` methods are handled by the + cycle detector, particularly the description of the ``garbage`` value. .. warning:: Due to the precarious circumstances under which :meth:`__del__` methods are invoked, exceptions that occur during their execution are ignored, and a warning - is printed to ``sys.stderr`` instead. In particular: + is printed to ``sys.stderr`` instead. Also, when :meth:`__del__` is invoked in + response to a module being deleted (e.g., when execution of the program is + done), other globals referenced by the :meth:`__del__` method may already have + been deleted or in the process of being torn down (e.g. the import + machinery shutting down). For this reason, :meth:`__del__` methods + should do the absolute + minimum needed to maintain external invariants. Starting with version 1.5, + Python guarantees that globals whose name begins with a single underscore are + deleted from their module before other globals are deleted; if no other + references to such globals exist, this may help in assuring that imported + modules are still available at the time when the :meth:`__del__` method is + called. + + See also the :option:`-R` command-line option. - * :meth:`__del__` can be invoked when arbitrary code is being executed, - including from any arbitrary thread. If :meth:`__del__` needs to take - a lock or invoke any other blocking resource, it may deadlock as - the resource may already be taken by the code that gets interrupted - to execute :meth:`__del__`. - - * :meth:`__del__` can be executed during interpreter shutdown. As a - consequence, the global variables it needs to access (including other - modules) may already have been deleted or set to ``None``. Python - guarantees that globals whose name begins with a single underscore - are deleted from their module before other globals are deleted; if - no other references to such globals exist, this may help in assuring - that imported modules are still available at the time when the - :meth:`__del__` method is called. +.. method:: object.__repr__(self) - .. index:: - single: repr() (built-in function); __repr__() (object method) + .. index:: builtin: repr -.. method:: object.__repr__(self) + Called by the :func:`repr` built-in function and by string conversions (reverse + quotes) to compute the "official" string representation of an object. If at all + possible, this should look like a valid Python expression that could be used to + recreate an object with the same value (given an appropriate environment). If + this is not possible, a string of the form ``<...some useful description...>`` + should be returned. The return value must be a string object. If a class + defines :meth:`__repr__` but not :meth:`__str__`, then :meth:`__repr__` is also + used when an "informal" string representation of instances of that class is + required. - Called by the :func:`repr` built-in function to compute the "official" string - representation of an object. If at all possible, this should look like a - valid Python expression that could be used to recreate an object with the - same value (given an appropriate environment). If this is not possible, a - string of the form ``<...some useful description...>`` should be returned. - The return value must be a string object. If a class defines :meth:`__repr__` - but not :meth:`__str__`, then :meth:`__repr__` is also used when an - "informal" string representation of instances of that class is required. + .. index:: + pair: string; conversion + pair: reverse; quotes + pair: backward; quotes + single: back-quotes This is typically used for debugging, so it is important that the representation is information-rich and unambiguous. - .. index:: - single: string; __str__() (object method) - single: format() (built-in function); __str__() (object method) - single: print() (built-in function); __str__() (object method) - .. method:: object.__str__(self) - Called by :func:`str(object) <str>` and the built-in functions - :func:`format` and :func:`print` to compute the "informal" or nicely - printable string representation of an object. The return value must be a - :ref:`string <textseq>` object. - - This method differs from :meth:`object.__repr__` in that there is no - expectation that :meth:`__str__` return a valid Python expression: a more - convenient or concise representation can be used. - - The default implementation defined by the built-in type :class:`object` - calls :meth:`object.__repr__`. - - .. XXX what about subclasses of string? - - -.. method:: object.__bytes__(self) - - .. index:: builtin: bytes - - Called by :ref:`bytes <func-bytes>` to compute a byte-string representation - of an object. This should return a :class:`bytes` object. - .. index:: - single: string; __format__() (object method) - pair: string; conversion - builtin: print - - -.. method:: object.__format__(self, format_spec) - - Called by the :func:`format` built-in function, - and by extension, evaluation of :ref:`formatted string literals - <f-strings>` and the :meth:`str.format` method, to produce a "formatted" - string representation of an object. The *format_spec* argument is - a string that contains a description of the formatting options desired. - The interpretation of the *format_spec* argument is up to the type - implementing :meth:`__format__`, however most classes will either - delegate formatting to one of the built-in types, or use a similar - formatting option syntax. - - See :ref:`formatspec` for a description of the standard formatting syntax. + builtin: str + statement: print + Called by the :func:`str` built-in function and by the :keyword:`print` + statement to compute the "informal" string representation of an object. This + differs from :meth:`__repr__` in that it does not have to be a valid Python + expression: a more convenient or concise representation may be used instead. The return value must be a string object. - .. versionchanged:: 3.4 - The __format__ method of ``object`` itself raises a :exc:`TypeError` - if passed any non-empty string. - - .. versionchanged:: 3.7 - ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather - than ``format(str(self), '')``. - -.. _richcmpfuncs: .. method:: object.__lt__(self, other) object.__le__(self, other) object.__eq__(self, other) @@ -1346,13 +1339,16 @@ Basic customization object.__gt__(self, other) object.__ge__(self, other) + .. versionadded:: 2.1 + .. index:: single: comparisons - These are the so-called "rich comparison" methods. The correspondence between + These are the so-called "rich comparison" methods, and are called for comparison + operators in preference to :meth:`__cmp__` below. The correspondence between operator symbols and method names is as follows: ``x<y`` calls ``x.__lt__(y)``, - ``x<=y`` calls ``x.__le__(y)``, ``x==y`` calls ``x.__eq__(y)``, ``x!=y`` calls - ``x.__ne__(y)``, ``x>y`` calls ``x.__gt__(y)``, and ``x>=y`` calls + ``x<=y`` calls ``x.__le__(y)``, ``x==y`` calls ``x.__eq__(y)``, ``x!=y`` and + ``x<>y`` call ``x.__ne__(y)``, ``x>y`` calls ``x.__gt__(y)``, and ``x>=y`` calls ``x.__ge__(y)``. A rich comparison method may return the singleton ``NotImplemented`` if it does @@ -1362,14 +1358,10 @@ Basic customization context (e.g., in the condition of an ``if`` statement), Python will call :func:`bool` on the value to determine if the result is true or false. - By default, :meth:`__ne__` delegates to :meth:`__eq__` and - inverts the result unless it is ``NotImplemented``. There are no other - implied relationships among the comparison operators, for example, - the truth of ``(x<y or x==y)`` does not imply ``x<=y``. - To automatically generate ordering operations from a single root operation, - see :func:`functools.total_ordering`. - - See the paragraph on :meth:`__hash__` for + There are no implied relationships among the comparison operators. The truth + of ``x==y`` does not imply that ``x!=y`` is false. Accordingly, when + defining :meth:`__eq__`, one should also define :meth:`__ne__` so that the + operators will behave as expected. See the paragraph on :meth:`__hash__` for some important notes on creating :term:`hashable` objects which support custom comparison operations and are usable as dictionary keys. @@ -1378,11 +1370,34 @@ Basic customization rather, :meth:`__lt__` and :meth:`__gt__` are each other's reflection, :meth:`__le__` and :meth:`__ge__` are each other's reflection, and :meth:`__eq__` and :meth:`__ne__` are their own reflection. - If the operands are of different types, and right operand's type is - a direct or indirect subclass of the left operand's type, - the reflected method of the right operand has priority, otherwise - the left operand's method has priority. Virtual subclassing is - not considered. + + Arguments to rich comparison methods are never coerced. + + To automatically generate ordering operations from a single root operation, + see :func:`functools.total_ordering`. + +.. method:: object.__cmp__(self, other) + + .. index:: + builtin: cmp + single: comparisons + + Called by comparison operations if rich comparison (see above) is not + defined. Should return a negative integer if ``self < other``, zero if + ``self == other``, a positive integer if ``self > other``. If no + :meth:`__cmp__`, :meth:`__eq__` or :meth:`__ne__` operation is defined, class + instances are compared by object identity ("address"). See also the + description of :meth:`__hash__` for some important notes on creating + :term:`hashable` objects which support custom comparison operations and are + usable as dictionary keys. (Note: the restriction that exceptions are not + propagated by :meth:`__cmp__` has been removed since Python 1.5.) + + +.. method:: object.__rcmp__(self, other) + + .. versionchanged:: 2.1 + No longer supported. + .. method:: object.__hash__(self) @@ -1392,7 +1407,7 @@ Basic customization Called by built-in function :func:`hash` and for operations on members of hashed collections including :class:`set`, :class:`frozenset`, and - :class:`dict`. :meth:`__hash__` should return an integer. The only required + :class:`dict`. :meth:`__hash__` should return an integer. The only required property is that objects which compare equal have the same hash value; it is advised to mix together the hash values of the components of the object that also play a part in comparison of objects by packing them into a tuple and @@ -1401,80 +1416,59 @@ Basic customization def __hash__(self): return hash((self.name, self.nick, self.color)) - .. note:: + If a class does not define a :meth:`__cmp__` or :meth:`__eq__` method it + should not define a :meth:`__hash__` operation either; if it defines + :meth:`__cmp__` or :meth:`__eq__` but not :meth:`__hash__`, its instances + will not be usable in hashed collections. If a class defines mutable objects + and implements a :meth:`__cmp__` or :meth:`__eq__` method, it should not + implement :meth:`__hash__`, since hashable collection implementations require + that an object's hash value is immutable (if the object's hash value changes, + it will be in the wrong hash bucket). - :func:`hash` truncates the value returned from an object's custom - :meth:`__hash__` method to the size of a :c:type:`Py_ssize_t`. This is - typically 8 bytes on 64-bit builds and 4 bytes on 32-bit builds. If an - object's :meth:`__hash__` must interoperate on builds of different bit - sizes, be sure to check the width on all supported builds. An easy way - to do this is with - ``python -c "import sys; print(sys.hash_info.width)"``. - - If a class does not define an :meth:`__eq__` method it should not define a - :meth:`__hash__` operation either; if it defines :meth:`__eq__` but not - :meth:`__hash__`, its instances will not be usable as items in hashable - collections. If a class defines mutable objects and implements an - :meth:`__eq__` method, it should not implement :meth:`__hash__`, since the - implementation of hashable collections requires that a key's hash value is - immutable (if the object's hash value changes, it will be in the wrong hash - bucket). - - User-defined classes have :meth:`__eq__` and :meth:`__hash__` methods + User-defined classes have :meth:`__cmp__` and :meth:`__hash__` methods by default; with them, all objects compare unequal (except with themselves) - and ``x.__hash__()`` returns an appropriate value such that ``x == y`` - implies both that ``x is y`` and ``hash(x) == hash(y)``. - - A class that overrides :meth:`__eq__` and does not define :meth:`__hash__` - will have its :meth:`__hash__` implicitly set to ``None``. When the - :meth:`__hash__` method of a class is ``None``, instances of the class will - raise an appropriate :exc:`TypeError` when a program attempts to retrieve - their hash value, and will also be correctly identified as unhashable when - checking ``isinstance(obj, collections.abc.Hashable)``. - - If a class that overrides :meth:`__eq__` needs to retain the implementation - of :meth:`__hash__` from a parent class, the interpreter must be told this - explicitly by setting ``__hash__ = <ParentClass>.__hash__``. + and ``x.__hash__()`` returns a result derived from ``id(x)``. - If a class that does not override :meth:`__eq__` wishes to suppress hash - support, it should include ``__hash__ = None`` in the class definition. - A class which defines its own :meth:`__hash__` that explicitly raises - a :exc:`TypeError` would be incorrectly identified as hashable by - an ``isinstance(obj, collections.abc.Hashable)`` call. + Classes which inherit a :meth:`__hash__` method from a parent class but + change the meaning of :meth:`__cmp__` or :meth:`__eq__` such that the hash + value returned is no longer appropriate (e.g. by switching to a value-based + concept of equality instead of the default identity based equality) can + explicitly flag themselves as being unhashable by setting ``__hash__ = None`` + in the class definition. Doing so means that not only will instances of the + class raise an appropriate :exc:`TypeError` when a program attempts to + retrieve their hash value, but they will also be correctly identified as + unhashable when checking ``isinstance(obj, collections.Hashable)`` (unlike + classes which define their own :meth:`__hash__` to explicitly raise + :exc:`TypeError`). + .. versionchanged:: 2.5 + :meth:`__hash__` may now also return a long integer object; the 32-bit + integer is then derived from the hash of that object. - .. note:: - - By default, the :meth:`__hash__` values of str and bytes objects are - "salted" with an unpredictable random value. Although they - remain constant within an individual Python process, they are not - predictable between repeated invocations of Python. + .. versionchanged:: 2.6 + :attr:`__hash__` may now be set to :const:`None` to explicitly flag + instances of a class as unhashable. - This is intended to provide protection against a denial-of-service caused - by carefully-chosen inputs that exploit the worst case performance of a - dict insertion, O(n^2) complexity. See - http://www.ocert.org/advisories/ocert-2011-003.html for details. - Changing hash values affects the iteration order of sets. - Python has never made guarantees about this ordering - (and it typically varies between 32-bit and 64-bit builds). +.. method:: object.__nonzero__(self) - See also :envvar:`PYTHONHASHSEED`. + .. index:: single: __len__() (mapping object method) - .. versionchanged:: 3.3 - Hash randomization is enabled by default. + Called to implement truth value testing and the built-in operation ``bool()``; + should return ``False`` or ``True``, or their integer equivalents ``0`` or + ``1``. When this method is not defined, :meth:`__len__` is called, if it is + defined, and the object is considered true if its result is nonzero. + If a class defines neither :meth:`__len__` nor :meth:`__nonzero__`, all its + instances are considered true. -.. method:: object.__bool__(self) +.. method:: object.__unicode__(self) - .. index:: single: __len__() (mapping object method) + .. index:: builtin: unicode - Called to implement truth value testing and the built-in operation - ``bool()``; should return ``False`` or ``True``. When this method is not - defined, :meth:`__len__` is called, if it is defined, and the object is - considered true if its result is nonzero. If a class defines neither - :meth:`__len__` nor :meth:`__bool__`, all its instances are considered - true. + Called to implement :func:`unicode` built-in; should return a Unicode object. + When this method is not defined, string conversion is attempted, and the result + of string conversion is converted to Unicode using the system default encoding. .. _attribute-access: @@ -1485,17 +1479,15 @@ Customizing attribute access The following methods can be defined to customize the meaning of attribute access (use of, assignment to, or deletion of ``x.name``) for class instances. -.. XXX explain how descriptors interfere here! - .. method:: object.__getattr__(self, name) - Called when the default attribute access fails with an :exc:`AttributeError` - (either :meth:`__getattribute__` raises an :exc:`AttributeError` because - *name* is not an instance attribute or an attribute in the class tree - for ``self``; or :meth:`__get__` of a *name* property raises - :exc:`AttributeError`). This method should either return the (computed) - attribute value or raise an :exc:`AttributeError` exception. + Called when an attribute lookup has not found the attribute in the usual places + (i.e. it is not an instance attribute nor is it found in the class tree for + ``self``). ``name`` is the attribute name. This method should return the + (computed) attribute value or raise an :exc:`AttributeError` exception. + + .. index:: single: __setattr__() (object method) Note that if the attribute is found through the normal mechanism, :meth:`__getattr__` is not called. (This is an intentional asymmetry between @@ -1504,37 +1496,25 @@ access (use of, assignment to, or deletion of ``x.name``) for class instances. other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the - :meth:`__getattribute__` method below for a way to actually get total control - over attribute access. - - -.. method:: object.__getattribute__(self, name) - - Called unconditionally to implement attribute accesses for instances of the - class. If the class also defines :meth:`__getattr__`, the latter will not be - called unless :meth:`__getattribute__` either calls it explicitly or raises an - :exc:`AttributeError`. This method should return the (computed) attribute value - or raise an :exc:`AttributeError` exception. In order to avoid infinite - recursion in this method, its implementation should always call the base class - method with the same name to access any attributes it needs, for example, - ``object.__getattribute__(self, name)``. - - .. note:: - - This method may still be bypassed when looking up special methods as the - result of implicit invocation via language syntax or built-in functions. - See :ref:`special-lookup`. + :meth:`__getattribute__` method below for a way to actually get total control in + new-style classes. .. method:: object.__setattr__(self, name, value) - Called when an attribute assignment is attempted. This is called instead of - the normal mechanism (i.e. store the value in the instance dictionary). - *name* is the attribute name, *value* is the value to be assigned to it. + Called when an attribute assignment is attempted. This is called instead of the + normal mechanism (i.e. store the value in the instance dictionary). *name* is + the attribute name, *value* is the value to be assigned to it. + + .. index:: single: __dict__ (instance attribute) - If :meth:`__setattr__` wants to assign to an instance attribute, it should - call the base class method with the same name, for example, - ``object.__setattr__(self, name, value)``. + If :meth:`__setattr__` wants to assign to an instance attribute, it should not + simply execute ``self.name = value`` --- this would cause a recursive call to + itself. Instead, it should insert the value in the dictionary of instance + attributes, e.g., ``self.__dict__[name] = value``. For new-style classes, + rather than accessing the instance dictionary, it should call the base class + method with the same name, for example, ``object.__setattr__(self, name, + value)``. .. method:: object.__delattr__(self, name) @@ -1543,66 +1523,30 @@ access (use of, assignment to, or deletion of ``x.name``) for class instances. should only be implemented if ``del obj.name`` is meaningful for the object. -.. method:: object.__dir__(self) - - Called when :func:`dir` is called on the object. A sequence must be - returned. :func:`dir` converts the returned sequence to a list and sorts it. - - -Customizing module attribute access -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. index:: - single: __getattr__ (module attribute) - single: __dir__ (module attribute) - single: __class__ (module attribute) +.. _new-style-attribute-access: -Special names ``__getattr__`` and ``__dir__`` can be also used to customize -access to module attributes. The ``__getattr__`` function at the module level -should accept one argument which is the name of an attribute and return the -computed value or raise an :exc:`AttributeError`. If an attribute is -not found on a module object through the normal lookup, i.e. -:meth:`object.__getattribute__`, then ``__getattr__`` is searched in -the module ``__dict__`` before raising an :exc:`AttributeError`. If found, -it is called with the attribute name and the result is returned. +More attribute access for new-style classes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The ``__dir__`` function should accept no arguments, and return a sequence of -strings that represents the names accessible on module. If present, this -function overrides the standard :func:`dir` search on a module. +The following methods only apply to new-style classes. -For a more fine grained customization of the module behavior (setting -attributes, properties, etc.), one can set the ``__class__`` attribute of -a module object to a subclass of :class:`types.ModuleType`. For example:: - import sys - from types import ModuleType - - class VerboseModule(ModuleType): - def __repr__(self): - return f'Verbose {self.__name__}' - - def __setattr__(self, attr, value): - print(f'Setting {attr}...') - super().__setattr__(attr, value) - - sys.modules[__name__].__class__ = VerboseModule - -.. note:: - Defining module ``__getattr__`` and setting module ``__class__`` only - affect lookups made using the attribute access syntax -- directly accessing - the module globals (whether by code within the module, or via a reference - to the module's globals dictionary) is unaffected. - -.. versionchanged:: 3.5 - ``__class__`` module attribute is now writable. +.. method:: object.__getattribute__(self, name) -.. versionadded:: 3.7 - ``__getattr__`` and ``__dir__`` module attributes. + Called unconditionally to implement attribute accesses for instances of the + class. If the class also defines :meth:`__getattr__`, the latter will not be + called unless :meth:`__getattribute__` either calls it explicitly or raises an + :exc:`AttributeError`. This method should return the (computed) attribute value + or raise an :exc:`AttributeError` exception. In order to avoid infinite + recursion in this method, its implementation should always call the base class + method with the same name to access any attributes it needs, for example, + ``object.__getattribute__(self, name)``. -.. seealso:: + .. note:: - :pep:`562` - Module __getattr__ and __dir__ - Describes the ``__getattr__`` and ``__dir__`` functions on modules. + This method may still be bypassed when looking up special methods as the + result of implicit invocation via language syntax or built-in functions. + See :ref:`new-style-special-lookup`. .. _descriptors: @@ -1618,68 +1562,27 @@ refers to the attribute whose name is the key of the property in the owner class' :attr:`~object.__dict__`. -.. method:: object.__get__(self, instance, owner=None) - - Called to get the attribute of the owner class (class attribute access) or - of an instance of that class (instance attribute access). The optional - *owner* argument is the owner class, while *instance* is the instance that - the attribute was accessed through, or ``None`` when the attribute is - accessed through the *owner*. +.. method:: object.__get__(self, instance, owner) - This method should return the computed attribute value or raise an - :exc:`AttributeError` exception. + Called to get the attribute of the owner class (class attribute access) or of an + instance of that class (instance attribute access). *owner* is always the owner + class, while *instance* is the instance that the attribute was accessed through, + or ``None`` when the attribute is accessed through the *owner*. This method + should return the (computed) attribute value or raise an :exc:`AttributeError` + exception. - :PEP:`252` specifies that :meth:`__get__` is callable with one or two - arguments. Python's own built-in descriptors support this specification; - however, it is likely that some third-party tools have descriptors - that require both arguments. Python's own :meth:`__getattribute__` - implementation always passes in both arguments whether they are required - or not. .. method:: object.__set__(self, instance, value) Called to set the attribute on an instance *instance* of the owner class to a new value, *value*. - Note, adding :meth:`__set__` or :meth:`__delete__` changes the kind of - descriptor to a "data descriptor". See :ref:`descriptor-invocation` for - more details. .. method:: object.__delete__(self, instance) Called to delete the attribute on an instance *instance* of the owner class. -.. method:: object.__set_name__(self, owner, name) - - Called at the time the owning class *owner* is created. The - descriptor has been assigned to *name*. - - .. note:: - - :meth:`__set_name__` is only called implicitly as part of the - :class:`type` constructor, so it will need to be called explicitly with - the appropriate parameters when a descriptor is added to a class after - initial creation:: - - class A: - pass - descr = custom_descriptor() - A.attr = descr - descr.__set_name__(A, 'attr') - - See :ref:`class-object-creation` for more details. - - .. versionadded:: 3.6 - -The attribute :attr:`__objclass__` is interpreted by the :mod:`inspect` module -as specifying the class where this object was defined (setting this -appropriately can assist in runtime introspection of dynamic class attributes). -For callables, it may indicate that an instance of the given type (or a -subclass) is expected or required as the first positional argument (for example, -CPython sets this attribute for unbound methods that are implemented in C). - - .. _descriptor-invocation: Invoking Descriptors @@ -1698,7 +1601,9 @@ continuing through the base classes of ``type(a)`` excluding metaclasses. However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which -descriptor methods were defined and how they were called. +descriptor methods were defined and how they were called. Note that descriptors +are only invoked for new style objects or classes (ones that subclass +:class:`object()` or :class:`type()`). The starting point for descriptor invocation is a binding, ``a.x``. How the arguments are assembled depends on ``a``: @@ -1708,16 +1613,16 @@ Direct Call descriptor method: ``x.__get__(a)``. Instance Binding - If binding to an object instance, ``a.x`` is transformed into the call: + If binding to a new-style object instance, ``a.x`` is transformed into the call: ``type(a).__dict__['x'].__get__(a, type(a))``. Class Binding - If binding to a class, ``A.x`` is transformed into the call: + If binding to a new-style class, ``A.x`` is transformed into the call: ``A.__dict__['x'].__get__(None, A)``. Super Binding - If ``a`` is an instance of :class:`super`, then the binding ``super(B, obj).m()`` - searches ``obj.__class__.__mro__`` for the base class ``A`` + If ``a`` is an instance of :class:`super`, then the binding ``super(B, + obj).m()`` searches ``obj.__class__.__mro__`` for the base class ``A`` immediately preceding ``B`` and then invokes the descriptor with the call: ``A.__dict__['m'].__get__(obj, obj.__class__)``. @@ -1730,7 +1635,7 @@ the descriptor defines :meth:`__set__` and/or :meth:`__delete__`, it is a data descriptor; if it defines neither, it is a non-data descriptor. Normally, data descriptors define both :meth:`__get__` and :meth:`__set__`, while non-data descriptors have just the :meth:`__get__` method. Data descriptors with -:meth:`__get__` and :meth:`__set__` (and/or :meth:`__delete__`) defined always override a redefinition in an +:meth:`__set__` and :meth:`__get__` defined always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances. @@ -1748,49 +1653,61 @@ instances cannot override the behavior of a property. __slots__ ^^^^^^^^^ -*__slots__* allow us to explicitly declare data members (like -properties) and deny the creation of *__dict__* and *__weakref__* -(unless explicitly declared in *__slots__* or available in a parent.) +By default, instances of both old and new-style classes have a dictionary for +attribute storage. This wastes space for objects having very few instance +variables. The space consumption can become acute when creating large numbers +of instances. + +The default can be overridden by defining *__slots__* in a new-style class +definition. The *__slots__* declaration takes a sequence of instance variables +and reserves just enough space in each instance to hold a value for each +variable. Space is saved because *__dict__* is not created for each instance. -The space saved over using *__dict__* can be significant. -Attribute lookup speed can be significantly improved as well. -.. data:: object.__slots__ +.. data:: __slots__ - This class variable can be assigned a string, iterable, or sequence of - strings with variable names used by instances. *__slots__* reserves space - for the declared variables and prevents the automatic creation of *__dict__* - and *__weakref__* for each instance. + This class variable can be assigned a string, iterable, or sequence of strings + with variable names used by instances. If defined in a new-style class, + *__slots__* reserves space for the declared variables and prevents the automatic + creation of *__dict__* and *__weakref__* for each instance. + .. versionadded:: 2.2 Notes on using *__slots__* -"""""""""""""""""""""""""" -* When inheriting from a class without *__slots__*, the *__dict__* and - *__weakref__* attribute of the instances will always be accessible. +* When inheriting from a class without *__slots__*, the *__dict__* attribute of + that class will always be accessible, so a *__slots__* definition in the + subclass is meaningless. * Without a *__dict__* variable, instances cannot be assigned new variables not listed in the *__slots__* definition. Attempts to assign to an unlisted variable name raises :exc:`AttributeError`. If dynamic assignment of new - variables is desired, then add ``'__dict__'`` to the sequence of strings in - the *__slots__* declaration. + variables is desired, then add ``'__dict__'`` to the sequence of strings in the + *__slots__* declaration. + + .. versionchanged:: 2.3 + Previously, adding ``'__dict__'`` to the *__slots__* declaration would not + enable the assignment of new attributes not specifically listed in the sequence + of instance variable names. * Without a *__weakref__* variable for each instance, classes defining *__slots__* do not support weak references to its instances. If weak reference support is needed, then add ``'__weakref__'`` to the sequence of strings in the *__slots__* declaration. + .. versionchanged:: 2.3 + Previously, adding ``'__weakref__'`` to the *__slots__* declaration would not + enable support for weak references. + * *__slots__* are implemented at the class level by creating descriptors (:ref:`descriptors`) for each variable name. As a result, class attributes cannot be used to set default values for instance variables defined by *__slots__*; otherwise, the class attribute would overwrite the descriptor assignment. -* The action of a *__slots__* declaration is not limited to the class - where it is defined. *__slots__* declared in parents are available in - child classes. However, child subclasses will get a *__dict__* and - *__weakref__* unless they also define *__slots__* (which should only - contain names of any *additional* slots). +* The action of a *__slots__* declaration is limited to the class where it is + defined. As a result, subclasses will have a *__dict__* unless they also define + *__slots__* (which must only contain names of any *additional* slots). * If a class defines a slot also defined in a base class, the instance variable defined by the base class slot is inaccessible (except by retrieving its @@ -1798,7 +1715,7 @@ Notes on using *__slots__* program undefined. In the future, a check may be added to prevent this. * Nonempty *__slots__* does not work for classes derived from "variable-length" - built-in types such as :class:`int`, :class:`bytes` and :class:`tuple`. + built-in types such as :class:`long`, :class:`str` and :class:`tuple`. * Any non-string iterable may be assigned to *__slots__*. Mappings may also be used; however, in the future, special meaning may be assigned to the values @@ -1806,243 +1723,75 @@ Notes on using *__slots__* * *__class__* assignment works only if both classes have the same *__slots__*. -* Multiple inheritance with multiple slotted parent classes can be used, - but only one parent is allowed to have attributes created by slots - (the other bases must have empty slot layouts) - violations raise - :exc:`TypeError`. + .. versionchanged:: 2.6 + Previously, *__class__* assignment raised an error if either new or old class + had *__slots__*. -* If an iterator is used for *__slots__* then a descriptor is created for each - of the iterator's values. However, the *__slots__* attribute will be an empty - iterator. -.. _class-customization: +.. _metaclasses: Customizing class creation -------------------------- -Whenever a class inherits from another class, *__init_subclass__* is -called on that class. This way, it is possible to write classes which -change the behavior of subclasses. This is closely related to class -decorators, but where class decorators only affect the specific class they're -applied to, ``__init_subclass__`` solely applies to future subclasses of the -class defining the method. - -.. classmethod:: object.__init_subclass__(cls) +By default, new-style classes are constructed using :func:`type`. A class +definition is read into a separate namespace and the value of class name is +bound to the result of ``type(name, bases, dict)``. - This method is called whenever the containing class is subclassed. - *cls* is then the new subclass. If defined as a normal instance method, - this method is implicitly converted to a class method. +When the class definition is read, if *__metaclass__* is defined then the +callable assigned to it will be called instead of :func:`type`. This allows +classes or functions to be written which monitor or alter the class creation +process: - Keyword arguments which are given to a new class are passed to - the parent's class ``__init_subclass__``. For compatibility with - other classes using ``__init_subclass__``, one should take out the - needed keyword arguments and pass the others over to the base - class, as in:: +* Modifying the class dictionary prior to the class being created. - class Philosopher: - def __init_subclass__(cls, /, default_name, **kwargs): - super().__init_subclass__(**kwargs) - cls.default_name = default_name +* Returning an instance of another class -- essentially performing the role of a + factory function. - class AustralianPhilosopher(Philosopher, default_name="Bruce"): - pass +These steps will have to be performed in the metaclass's :meth:`__new__` method +-- :meth:`type.__new__` can then be called from this method to create a class +with different properties. This example adds a new element to the class +dictionary before creating the class:: - The default implementation ``object.__init_subclass__`` does - nothing, but raises an error if it is called with any arguments. + class metacls(type): + def __new__(mcs, name, bases, dict): + dict['foo'] = 'metacls was here' + return type.__new__(mcs, name, bases, dict) - .. note:: +You can of course also override other class methods (or add new methods); for +example defining a custom :meth:`__call__` method in the metaclass allows custom +behavior when the class is called, e.g. not always creating a new instance. - The metaclass hint ``metaclass`` is consumed by the rest of the type - machinery, and is never passed to ``__init_subclass__`` implementations. - The actual metaclass (rather than the explicit hint) can be accessed as - ``type(cls)``. - .. versionadded:: 3.6 +.. data:: __metaclass__ + This variable can be any callable accepting arguments for ``name``, ``bases``, + and ``dict``. Upon class creation, the callable is used instead of the built-in + :func:`type`. -.. _metaclasses: + .. versionadded:: 2.2 -Metaclasses -^^^^^^^^^^^ +The appropriate metaclass is determined by the following precedence rules: -.. index:: - single: metaclass - builtin: type - single: = (equals); class definition +* If ``dict['__metaclass__']`` exists, it is used. -By default, classes are constructed using :func:`type`. The class body is -executed in a new namespace and the class name is bound locally to the -result of ``type(name, bases, namespace)``. +* Otherwise, if there is at least one base class, its metaclass is used (this + looks for a *__class__* attribute first and if not found, uses its type). -The class creation process can be customized by passing the ``metaclass`` -keyword argument in the class definition line, or by inheriting from an -existing class that included such an argument. In the following example, -both ``MyClass`` and ``MySubclass`` are instances of ``Meta``:: +* Otherwise, if a global variable named __metaclass__ exists, it is used. - class Meta(type): - pass - - class MyClass(metaclass=Meta): - pass - - class MySubclass(MyClass): - pass - -Any other keyword arguments that are specified in the class definition are -passed through to all metaclass operations described below. - -When a class definition is executed, the following steps occur: - -* MRO entries are resolved; -* the appropriate metaclass is determined; -* the class namespace is prepared; -* the class body is executed; -* the class object is created. - - -Resolving MRO entries -^^^^^^^^^^^^^^^^^^^^^ - -If a base that appears in class definition is not an instance of :class:`type`, -then an ``__mro_entries__`` method is searched on it. If found, it is called -with the original bases tuple. This method must return a tuple of classes that -will be used instead of this base. The tuple may be empty, in such case -the original base is ignored. - -.. seealso:: - - :pep:`560` - Core support for typing module and generic types - - -Determining the appropriate metaclass -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. index:: - single: metaclass hint - -The appropriate metaclass for a class definition is determined as follows: - -* if no bases and no explicit metaclass are given, then :func:`type` is used; -* if an explicit metaclass is given and it is *not* an instance of - :func:`type`, then it is used directly as the metaclass; -* if an instance of :func:`type` is given as the explicit metaclass, or - bases are defined, then the most derived metaclass is used. - -The most derived metaclass is selected from the explicitly specified -metaclass (if any) and the metaclasses (i.e. ``type(cls)``) of all specified -base classes. The most derived metaclass is one which is a subtype of *all* -of these candidate metaclasses. If none of the candidate metaclasses meets -that criterion, then the class definition will fail with ``TypeError``. - - -.. _prepare: - -Preparing the class namespace -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. index:: - single: __prepare__ (metaclass method) - -Once the appropriate metaclass has been identified, then the class namespace -is prepared. If the metaclass has a ``__prepare__`` attribute, it is called -as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the -additional keyword arguments, if any, come from the class definition). - -If the metaclass has no ``__prepare__`` attribute, then the class namespace -is initialised as an empty ordered mapping. - -.. seealso:: - - :pep:`3115` - Metaclasses in Python 3000 - Introduced the ``__prepare__`` namespace hook - - -Executing the class body -^^^^^^^^^^^^^^^^^^^^^^^^ - -.. index:: - single: class; body - -The class body is executed (approximately) as -``exec(body, globals(), namespace)``. The key difference from a normal -call to :func:`exec` is that lexical scoping allows the class body (including -any methods) to reference names from the current and outer scopes when the -class definition occurs inside a function. - -However, even when the class definition occurs inside the function, methods -defined inside the class still cannot see names defined at the class scope. -Class variables must be accessed through the first parameter of instance or -class methods, or through the implicit lexically scoped ``__class__`` reference -described in the next section. - -.. _class-object-creation: - -Creating the class object -^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. index:: - single: __class__ (method cell) - single: __classcell__ (class namespace entry) - - -Once the class namespace has been populated by executing the class body, -the class object is created by calling -``metaclass(name, bases, namespace, **kwds)`` (the additional keywords -passed here are the same as those passed to ``__prepare__``). - -This class object is the one that will be referenced by the zero-argument -form of :func:`super`. ``__class__`` is an implicit closure reference -created by the compiler if any methods in a class body refer to either -``__class__`` or ``super``. This allows the zero argument form of -:func:`super` to correctly identify the class being defined based on -lexical scoping, while the class or instance that was used to make the -current call is identified based on the first argument passed to the method. - -.. impl-detail:: - - In CPython 3.6 and later, the ``__class__`` cell is passed to the metaclass - as a ``__classcell__`` entry in the class namespace. If present, this must - be propagated up to the ``type.__new__`` call in order for the class to be - initialised correctly. - Failing to do so will result in a :exc:`RuntimeError` in Python 3.8. - -When using the default metaclass :class:`type`, or any metaclass that ultimately -calls ``type.__new__``, the following additional customisation steps are -invoked after creating the class object: - -* first, ``type.__new__`` collects all of the descriptors in the class - namespace that define a :meth:`~object.__set_name__` method; -* second, all of these ``__set_name__`` methods are called with the class - being defined and the assigned name of that particular descriptor; -* finally, the :meth:`~object.__init_subclass__` hook is called on the - immediate parent of the new class in its method resolution order. - -After the class object is created, it is passed to the class decorators -included in the class definition (if any) and the resulting object is bound -in the local namespace as the defined class. - -When a new class is created by ``type.__new__``, the object provided as the -namespace parameter is copied to a new ordered mapping and the original -object is discarded. The new copy is wrapped in a read-only proxy, which -becomes the :attr:`~object.__dict__` attribute of the class object. - -.. seealso:: - - :pep:`3135` - New super - Describes the implicit ``__class__`` closure reference - - -Uses for metaclasses -^^^^^^^^^^^^^^^^^^^^ +* Otherwise, the old-style, classic metaclass (types.ClassType) is used. The potential uses for metaclasses are boundless. Some ideas that have been -explored include enum, logging, interface checking, automatic delegation, -automatic property creation, proxies, frameworks, and automatic resource +explored including logging, interface checking, automatic delegation, automatic +property creation, proxies, frameworks, and automatic resource locking/synchronization. Customizing instance and subclass checks ---------------------------------------- +.. versionadded:: 2.6 + The following methods are used to override the default behavior of the :func:`isinstance` and :func:`issubclass` built-in functions. @@ -2080,27 +1829,6 @@ case the instance is itself a class. module) to the language. -Emulating generic types ------------------------ - -One can implement the generic class syntax as specified by :pep:`484` -(for example ``List[int]``) by defining a special method: - -.. classmethod:: object.__class_getitem__(cls, key) - - Return an object representing the specialization of a generic class - by type arguments found in *key*. - -This method is looked up on the class object itself, and when defined in -the class body, this method is implicitly a class method. Note, this -mechanism is primarily reserved for use with static type hints, other usage -is discouraged. - -.. seealso:: - - :pep:`560` - Core support for typing module and generic types - - .. _callable-types: Emulating callable objects @@ -2126,38 +1854,41 @@ but can represent other containers as well. The first set of methods is used either to emulate a sequence or to emulate a mapping; the difference is that for a sequence, the allowable keys should be the integers *k* for which ``0 <= k < N`` where *N* is the length of the sequence, or slice objects, which define a -range of items. It is also recommended that mappings provide the methods -:meth:`keys`, :meth:`values`, :meth:`items`, :meth:`get`, :meth:`clear`, -:meth:`setdefault`, :meth:`pop`, :meth:`popitem`, :meth:`!copy`, and -:meth:`update` behaving similar to those for Python's standard dictionary -objects. The :mod:`collections.abc` module provides a -:class:`~collections.abc.MutableMapping` -abstract base class to help create those methods from a base set of -:meth:`__getitem__`, :meth:`__setitem__`, :meth:`__delitem__`, and :meth:`keys`. -Mutable sequences should provide methods :meth:`append`, :meth:`count`, -:meth:`index`, :meth:`extend`, :meth:`insert`, :meth:`pop`, :meth:`remove`, -:meth:`reverse` and :meth:`sort`, like Python standard list objects. Finally, -sequence types should implement addition (meaning concatenation) and -multiplication (meaning repetition) by defining the methods :meth:`__add__`, -:meth:`__radd__`, :meth:`__iadd__`, :meth:`__mul__`, :meth:`__rmul__` and -:meth:`__imul__` described below; they should not define other numerical -operators. It is recommended that both mappings and sequences implement the -:meth:`__contains__` method to allow efficient use of the ``in`` operator; for -mappings, ``in`` should search the mapping's keys; for sequences, it should -search through the values. It is further recommended that both mappings and -sequences implement the :meth:`__iter__` method to allow efficient iteration -through the container; for mappings, :meth:`__iter__` should iterate -through the object's keys; for sequences, it should iterate through the values. +range of items. (For backwards compatibility, the method :meth:`__getslice__` +(see below) can also be defined to handle simple, but not extended slices.) It +is also recommended that mappings provide the methods :meth:`keys`, +:meth:`values`, :meth:`items`, :meth:`has_key`, :meth:`get`, :meth:`clear`, +:meth:`setdefault`, :meth:`iterkeys`, :meth:`itervalues`, :meth:`iteritems`, +:meth:`pop`, :meth:`popitem`, :meth:`!copy`, and :meth:`update` behaving similar +to those for Python's standard dictionary objects. The :mod:`UserDict` module +provides a :class:`DictMixin` class to help create those methods from a base set +of :meth:`__getitem__`, :meth:`__setitem__`, :meth:`__delitem__`, and +:meth:`keys`. Mutable sequences should provide methods :meth:`append`, +:meth:`count`, :meth:`index`, :meth:`extend`, :meth:`insert`, :meth:`pop`, +:meth:`remove`, :meth:`reverse` and :meth:`sort`, like Python standard list +objects. Finally, sequence types should implement addition (meaning +concatenation) and multiplication (meaning repetition) by defining the methods +:meth:`__add__`, :meth:`__radd__`, :meth:`__iadd__`, :meth:`__mul__`, +:meth:`__rmul__` and :meth:`__imul__` described below; they should not define +:meth:`__coerce__` or other numerical operators. It is recommended that both +mappings and sequences implement the :meth:`__contains__` method to allow +efficient use of the ``in`` operator; for mappings, ``in`` should be equivalent +of :meth:`has_key`; for sequences, it should search through the values. It is +further recommended that both mappings and sequences implement the +:meth:`__iter__` method to allow efficient iteration through the container; for +mappings, :meth:`__iter__` should be the same as :meth:`iterkeys`; for +sequences, it should iterate through the values. + .. method:: object.__len__(self) .. index:: builtin: len - single: __bool__() (object method) + single: __nonzero__() (object method) Called to implement the built-in function :func:`len`. Should return the length of the object, an integer ``>=`` 0. Also, an object that doesn't define a - :meth:`__bool__` method and whose :meth:`__len__` method returns zero is + :meth:`__nonzero__` method and whose :meth:`__len__` method returns zero is considered to be false in a Boolean context. .. impl-detail:: @@ -2166,38 +1897,13 @@ through the object's keys; for sequences, it should iterate through the values. If the length is larger than :attr:`!sys.maxsize` some features (such as :func:`len`) may raise :exc:`OverflowError`. To prevent raising :exc:`!OverflowError` by truth value testing, an object must define a - :meth:`__bool__` method. - - -.. method:: object.__length_hint__(self) - - Called to implement :func:`operator.length_hint`. Should return an estimated - length for the object (which may be greater or less than the actual length). - The length must be an integer ``>=`` 0. The return value may also be - :const:`NotImplemented`, which is treated the same as if the - ``__length_hint__`` method didn't exist at all. This method is purely an - optimization and is never required for correctness. - - .. versionadded:: 3.4 - - -.. index:: object: slice - -.. note:: - - Slicing is done exclusively with the following three methods. A call like :: - - a[1:2] = b - - is translated to :: - - a[slice(1, 2, None)] = b - - and so forth. Missing slice items are always filled in with ``None``. + :meth:`__nonzero__` method. .. method:: object.__getitem__(self, key) + .. index:: object: slice + Called to implement evaluation of ``self[key]``. For sequence types, the accepted keys should be integers and slice objects. Note that the special interpretation of negative indexes (if the class wishes to emulate a sequence @@ -2241,7 +1947,8 @@ through the object's keys; for sequences, it should iterate through the values. This method is called when an iterator is required for a container. This method should return a new iterator object that can iterate over all the objects in the - container. For mappings, it should iterate over the keys of the container. + container. For mappings, it should iterate over the keys of the container, and + should also be made available as the method :meth:`iterkeys`. Iterator objects also need to implement this method; they are required to return themselves. For more information on iterator objects, see :ref:`typeiter`. @@ -2259,11 +1966,13 @@ through the object's keys; for sequences, it should iterate through the values. only provide :meth:`__reversed__` if they can provide an implementation that is more efficient than the one provided by :func:`reversed`. + .. versionadded:: 2.6 + The membership test operators (:keyword:`in` and :keyword:`not in`) are normally -implemented as an iteration through a container. However, container objects can +implemented as an iteration through a sequence. However, container objects can supply the following special method with a more efficient implementation, which -also does not require the object be iterable. +also does not require the object be a sequence. .. method:: object.__contains__(self, item) @@ -2277,6 +1986,98 @@ also does not require the object be iterable. reference <membership-test-details>`. +.. _sequence-methods: + +Additional methods for emulation of sequence types +-------------------------------------------------- + +The following optional methods can be defined to further emulate sequence +objects. Immutable sequences methods should at most only define +:meth:`__getslice__`; mutable sequences might define all three methods. + + +.. method:: object.__getslice__(self, i, j) + + .. deprecated:: 2.0 + Support slice objects as parameters to the :meth:`__getitem__` method. + (However, built-in types in CPython currently still implement + :meth:`__getslice__`. Therefore, you have to override it in derived + classes when implementing slicing.) + + Called to implement evaluation of ``self[i:j]``. The returned object should + be of the same type as *self*. Note that missing *i* or *j* in the slice + expression are replaced by zero or :attr:`sys.maxsize`, respectively. If + negative indexes are used in the slice, the length of the sequence is added + to that index. If the instance does not implement the :meth:`__len__` method, + an :exc:`AttributeError` is raised. No guarantee is made that indexes + adjusted this way are not still negative. Indexes which are greater than the + length of the sequence are not modified. If no :meth:`__getslice__` is found, + a slice object is created instead, and passed to :meth:`__getitem__` instead. + + +.. method:: object.__setslice__(self, i, j, sequence) + + Called to implement assignment to ``self[i:j]``. Same notes for *i* and *j* as + for :meth:`__getslice__`. + + This method is deprecated. If no :meth:`__setslice__` is found, or for extended + slicing of the form ``self[i:j:k]``, a slice object is created, and passed to + :meth:`__setitem__`, instead of :meth:`__setslice__` being called. + + +.. method:: object.__delslice__(self, i, j) + + Called to implement deletion of ``self[i:j]``. Same notes for *i* and *j* as for + :meth:`__getslice__`. This method is deprecated. If no :meth:`__delslice__` is + found, or for extended slicing of the form ``self[i:j:k]``, a slice object is + created, and passed to :meth:`__delitem__`, instead of :meth:`__delslice__` + being called. + +Notice that these methods are only invoked when a single slice with a single +colon is used, and the slice method is available. For slice operations +involving extended slice notation, or in absence of the slice methods, +:meth:`__getitem__`, :meth:`__setitem__` or :meth:`__delitem__` is called with a +slice object as argument. + +The following example demonstrate how to make your program or module compatible +with earlier versions of Python (assuming that methods :meth:`__getitem__`, +:meth:`__setitem__` and :meth:`__delitem__` support slice objects as +arguments):: + + class MyClass: + ... + def __getitem__(self, index): + ... + def __setitem__(self, index, value): + ... + def __delitem__(self, index): + ... + + if sys.version_info < (2, 0): + # They won't be defined if version is at least 2.0 final + + def __getslice__(self, i, j): + return self[max(0, i):max(0, j):] + def __setslice__(self, i, j, seq): + self[max(0, i):max(0, j):] = seq + def __delslice__(self, i, j): + del self[max(0, i):max(0, j):] + ... + +Note the calls to :func:`max`; these are necessary because of the handling of +negative indices before the :meth:`__\*slice__` methods are called. When +negative indexes are used, the :meth:`__\*item__` methods receive them as +provided, but the :meth:`__\*slice__` methods get a "cooked" form of the index +values. For each negative index value, the length of the sequence is added to +the index before calling the method (which may still result in a negative +index); this is the customary handling of negative indexes by the built-in +sequence types, and the :meth:`__\*item__` methods are expected to do this as +well. However, since they should already be doing that, negative indexes cannot +be passed in; they must be constrained to the bounds of the sequence before +being passed to the :meth:`__\*item__` methods. Calling ``max(0, i)`` +conveniently returns the proper value. + + .. _numeric-types: Emulating numeric types @@ -2291,8 +2092,6 @@ left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) - object.__matmul__(self, other) - object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) object.__divmod__(self, other) @@ -2308,25 +2107,34 @@ left undefined. builtin: pow builtin: pow - These methods are called to implement the binary arithmetic operations - (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, - :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to - evaluate the expression ``x + y``, where *x* is an instance of a class that - has an :meth:`__add__` method, ``x.__add__(y)`` is called. The - :meth:`__divmod__` method should be the equivalent to using - :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to - :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept - an optional third argument if the ternary version of the built-in :func:`pow` - function is to be supported. + These methods are called to implement the binary arithmetic operations (``+``, + ``-``, ``*``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, + ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression + ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` + method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the + equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be + related to :meth:`__truediv__` (described below). Note that :meth:`__pow__` + should be defined to accept an optional third argument if the ternary version of + the built-in :func:`pow` function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. +.. method:: object.__div__(self, other) + object.__truediv__(self, other) + + The division operator (``/``) is implemented by these methods. The + :meth:`__truediv__` method is used when ``__future__.division`` is in effect, + otherwise :meth:`__div__` is used. If only one of these two methods is defined, + the object will not support division in the alternate context; :exc:`TypeError` + will be raised instead. + + .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) - object.__rmatmul__(self, other) + object.__rdiv__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) @@ -2342,14 +2150,14 @@ left undefined. builtin: divmod builtin: pow - These methods are called to implement the binary arithmetic operations - (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, - :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected - (swapped) operands. These functions are only called if the left operand does - not support the corresponding operation [#]_ and the operands are of different - types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is - an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` - is called if ``x.__sub__(y)`` returns *NotImplemented*. + These methods are called to implement the binary arithmetic operations (``+``, + ``-``, ``*``, ``/``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, ``>>``, + ``&``, ``^``, ``|``) with reflected (swapped) operands. These functions are + only called if the left operand does not support the corresponding operation and + the operands are of different types. [#]_ For instance, to evaluate the + expression ``x - y``, where *y* is an instance of a class that has an + :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns + *NotImplemented*. .. index:: builtin: pow @@ -2367,7 +2175,7 @@ left undefined. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) - object.__imatmul__(self, other) + object.__idiv__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) @@ -2379,17 +2187,15 @@ left undefined. object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments - (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, - ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the - operation in-place (modifying *self*) and return the result (which could be, - but does not have to be, *self*). If a specific method is not defined, the - augmented assignment falls back to the normal methods. For instance, if *x* - is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is - equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and - ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In - certain situations, augmented assignment can result in unexpected errors (see - :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact - part of the data model. + (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, + ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation + in-place (modifying *self*) and return the result (which could be, but does + not have to be, *self*). If a specific method is not defined, the augmented + assignment falls back to the normal methods. For instance, to execute the + statement ``x += y``, where *x* is an instance of a class that has an + :meth:`__iadd__` method, ``x.__iadd__(y)`` is called. If *x* is an instance + of a class that does not define a :meth:`__iadd__` method, ``x.__add__(y)`` + and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. .. method:: object.__neg__(self) @@ -2405,46 +2211,154 @@ left undefined. .. method:: object.__complex__(self) object.__int__(self) + object.__long__(self) object.__float__(self) .. index:: builtin: complex builtin: int + builtin: long builtin: float - Called to implement the built-in functions :func:`complex`, - :func:`int` and :func:`float`. Should return a value - of the appropriate type. + Called to implement the built-in functions :func:`complex`, :func:`int`, + :func:`long`, and :func:`float`. Should return a value of the appropriate type. + + +.. method:: object.__oct__(self) + object.__hex__(self) + + .. index:: + builtin: oct + builtin: hex + + Called to implement the built-in functions :func:`oct` and :func:`hex`. Should + return a string value. .. method:: object.__index__(self) - Called to implement :func:`operator.index`, and whenever Python needs to - losslessly convert the numeric object to an integer object (such as in - slicing, or in the built-in :func:`bin`, :func:`hex` and :func:`oct` - functions). Presence of this method indicates that the numeric object is - an integer type. Must return an integer. + Called to implement :func:`operator.index`. Also called whenever Python needs + an integer object (such as in slicing). Must return an integer (int or long). + + .. versionadded:: 2.5 + + +.. method:: object.__coerce__(self, other) + + Called to implement "mixed-mode" numeric arithmetic. Should either return a + 2-tuple containing *self* and *other* converted to a common numeric type, or + ``None`` if conversion is impossible. When the common type would be the type of + ``other``, it is sufficient to return ``None``, since the interpreter will also + ask the other object to attempt a coercion (but sometimes, if the implementation + of the other type cannot be changed, it is useful to do the conversion to the + other type here). A return value of ``NotImplemented`` is equivalent to + returning ``None``. + + +.. _coercion-rules: + +Coercion rules +-------------- + +This section used to document the rules for coercion. As the language has +evolved, the coercion rules have become hard to document precisely; documenting +what one version of one particular implementation does is undesirable. Instead, +here are some informal guidelines regarding coercion. In Python 3, coercion +will not be supported. + +* + + If the left operand of a % operator is a string or Unicode object, no coercion + takes place and the string formatting operation is invoked instead. + +* + + It is no longer recommended to define a coercion operation. Mixed-mode + operations on types that don't define coercion pass the original arguments to + the operation. + +* + + New-style classes (those derived from :class:`object`) never invoke the + :meth:`__coerce__` method in response to a binary operator; the only time + :meth:`__coerce__` is invoked is when the built-in function :func:`coerce` is + called. + +* + + For most intents and purposes, an operator that returns ``NotImplemented`` is + treated the same as one that is not implemented at all. + +* + + Below, :meth:`__op__` and :meth:`__rop__` are used to signify the generic method + names corresponding to an operator; :meth:`__iop__` is used for the + corresponding in-place operator. For example, for the operator '``+``', + :meth:`__add__` and :meth:`__radd__` are used for the left and right variant of + the binary operator, and :meth:`__iadd__` for the in-place variant. + +* + + For objects *x* and *y*, first ``x.__op__(y)`` is tried. If this is not + implemented or returns ``NotImplemented``, ``y.__rop__(x)`` is tried. If this + is also not implemented or returns ``NotImplemented``, a :exc:`TypeError` + exception is raised. But see the following exception: + +* + + Exception to the previous item: if the left operand is an instance of a built-in + type or a new-style class, and the right operand is an instance of a proper + subclass of that type or class and overrides the base's :meth:`__rop__` method, + the right operand's :meth:`__rop__` method is tried *before* the left operand's + :meth:`__op__` method. - If :meth:`__int__`, :meth:`__float__` and :meth:`__complex__` are not - defined then corresponding built-in functions :func:`int`, :func:`float` - and :func:`complex` fall back to :meth:`__index__`. + This is done so that a subclass can completely override binary operators. + Otherwise, the left operand's :meth:`__op__` method would always accept the + right operand: when an instance of a given class is expected, an instance of a + subclass of that class is always acceptable. +* -.. method:: object.__round__(self, [,ndigits]) - object.__trunc__(self) - object.__floor__(self) - object.__ceil__(self) + When either operand type defines a coercion, this coercion is called before that + type's :meth:`__op__` or :meth:`__rop__` method is called, but no sooner. If + the coercion returns an object of a different type for the operand whose + coercion is invoked, part of the process is redone using the new object. - .. index:: builtin: round +* - Called to implement the built-in function :func:`round` and :mod:`math` - functions :func:`~math.trunc`, :func:`~math.floor` and :func:`~math.ceil`. - Unless *ndigits* is passed to :meth:`!__round__` all these methods should - return the value of the object truncated to an :class:`~numbers.Integral` - (typically an :class:`int`). + When an in-place operator (like '``+=``') is used, if the left operand + implements :meth:`__iop__`, it is invoked without any coercion. When the + operation falls back to :meth:`__op__` and/or :meth:`__rop__`, the normal + coercion rules apply. - If :meth:`__int__` is not defined then the built-in function :func:`int` - falls back to :meth:`__trunc__`. +* + + In ``x + y``, if *x* is a sequence that implements sequence concatenation, + sequence concatenation is invoked. + +* + + In ``x * y``, if one operand is a sequence that implements sequence + repetition, and the other is an integer (:class:`int` or :class:`long`), + sequence repetition is invoked. + +* + + Rich comparisons (implemented by methods :meth:`__eq__` and so on) never use + coercion. Three-way comparison (implemented by :meth:`__cmp__`) does use + coercion under the same conditions as other binary operations use it. + +* + + In the current implementation, the built-in numeric types :class:`int`, + :class:`long`, :class:`float`, and :class:`complex` do not use coercion. + All these types implement a :meth:`__coerce__` method, for use by the built-in + :func:`coerce` function. + + .. versionchanged:: 2.7 + + The complex type no longer makes implicit calls to the :meth:`__coerce__` + method for mixed-type binary arithmetic operations. .. _context-managers: @@ -2452,11 +2366,13 @@ left undefined. With Statement Context Managers ------------------------------- +.. versionadded:: 2.5 + A :dfn:`context manager` is an object that defines the runtime context to be established when executing a :keyword:`with` statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the -:keyword:`!with` statement (described in section :ref:`with`), but can also be +:keyword:`with` statement (described in section :ref:`with`), but can also be used by directly invoking their methods. .. index:: @@ -2473,7 +2389,7 @@ For more information on context managers, see :ref:`typecontextmanager`. Enter the runtime context related to this object. The :keyword:`with` statement will bind this method's return value to the target(s) specified in the - :keyword:`!as` clause of the statement, if any. + :keyword:`as` clause of the statement, if any. .. method:: object.__exit__(self, exc_type, exc_value, traceback) @@ -2497,17 +2413,44 @@ For more information on context managers, see :ref:`typecontextmanager`. statement. -.. _special-lookup: +.. _old-style-special-lookup: + +Special method lookup for old-style classes +------------------------------------------- + +For old-style classes, special methods are always looked up in exactly the +same way as any other method or attribute. This is the case regardless of +whether the method is being looked up explicitly as in ``x.__getitem__(i)`` +or implicitly as in ``x[i]``. + +This behaviour means that special methods may exhibit different behaviour +for different instances of a single old-style class if the appropriate +special attributes are set differently:: + + >>> class C: + ... pass + ... + >>> c1 = C() + >>> c2 = C() + >>> c1.__len__ = lambda: 5 + >>> c2.__len__ = lambda: 9 + >>> len(c1) + 5 + >>> len(c2) + 9 -Special method lookup ---------------------- -For custom classes, implicit invocations of special methods are only guaranteed +.. _new-style-special-lookup: + +Special method lookup for new-style classes +------------------------------------------- + +For new-style classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object's type, not in the object's instance dictionary. That behaviour is the reason why the following code raises an -exception:: +exception (unlike the equivalent example with old-style classes):: - >>> class C: + >>> class C(object): ... pass ... >>> c = C() @@ -2544,15 +2487,16 @@ correctness, implicit special method lookup generally also bypasses the :meth:`__getattribute__` method even of the object's metaclass:: >>> class Meta(type): - ... def __getattribute__(*args): - ... print("Metaclass getattribute invoked") - ... return type.__getattribute__(*args) + ... def __getattribute__(*args): + ... print "Metaclass getattribute invoked" + ... return type.__getattribute__(*args) ... - >>> class C(object, metaclass=Meta): + >>> class C(object): + ... __metaclass__ = Meta ... def __len__(self): ... return 10 ... def __getattribute__(*args): - ... print("Class getattribute invoked") + ... print "Class getattribute invoked" ... return object.__getattribute__(*args) ... >>> c = C() @@ -2572,187 +2516,13 @@ special methods (the special method *must* be set on the class object itself in order to be consistently invoked by the interpreter). -.. index:: - single: coroutine - -Coroutines -========== - - -Awaitable Objects ------------------ - -An :term:`awaitable` object generally implements an :meth:`__await__` method. -:term:`Coroutine` objects returned from :keyword:`async def` functions -are awaitable. - -.. note:: - - The :term:`generator iterator` objects returned from generators - decorated with :func:`types.coroutine` or :func:`asyncio.coroutine` - are also awaitable, but they do not implement :meth:`__await__`. - -.. method:: object.__await__(self) - - Must return an :term:`iterator`. Should be used to implement - :term:`awaitable` objects. For instance, :class:`asyncio.Future` implements - this method to be compatible with the :keyword:`await` expression. - -.. versionadded:: 3.5 - -.. seealso:: :pep:`492` for additional information about awaitable objects. - - -.. _coroutine-objects: - -Coroutine Objects ------------------ - -:term:`Coroutine` objects are :term:`awaitable` objects. -A coroutine's execution can be controlled by calling :meth:`__await__` and -iterating over the result. When the coroutine has finished executing and -returns, the iterator raises :exc:`StopIteration`, and the exception's -:attr:`~StopIteration.value` attribute holds the return value. If the -coroutine raises an exception, it is propagated by the iterator. Coroutines -should not directly raise unhandled :exc:`StopIteration` exceptions. - -Coroutines also have the methods listed below, which are analogous to -those of generators (see :ref:`generator-methods`). However, unlike -generators, coroutines do not directly support iteration. - -.. versionchanged:: 3.5.2 - It is a :exc:`RuntimeError` to await on a coroutine more than once. - - -.. method:: coroutine.send(value) - - Starts or resumes execution of the coroutine. If *value* is ``None``, - this is equivalent to advancing the iterator returned by - :meth:`__await__`. If *value* is not ``None``, this method delegates - to the :meth:`~generator.send` method of the iterator that caused - the coroutine to suspend. The result (return value, - :exc:`StopIteration`, or other exception) is the same as when - iterating over the :meth:`__await__` return value, described above. - -.. method:: coroutine.throw(type[, value[, traceback]]) - - Raises the specified exception in the coroutine. This method delegates - to the :meth:`~generator.throw` method of the iterator that caused - the coroutine to suspend, if it has such a method. Otherwise, - the exception is raised at the suspension point. The result - (return value, :exc:`StopIteration`, or other exception) is the same as - when iterating over the :meth:`__await__` return value, described - above. If the exception is not caught in the coroutine, it propagates - back to the caller. - -.. method:: coroutine.close() - - Causes the coroutine to clean itself up and exit. If the coroutine - is suspended, this method first delegates to the :meth:`~generator.close` - method of the iterator that caused the coroutine to suspend, if it - has such a method. Then it raises :exc:`GeneratorExit` at the - suspension point, causing the coroutine to immediately clean itself up. - Finally, the coroutine is marked as having finished executing, even if - it was never started. - - Coroutine objects are automatically closed using the above process when - they are about to be destroyed. - -.. _async-iterators: - -Asynchronous Iterators ----------------------- - -An *asynchronous iterator* can call asynchronous code in -its ``__anext__`` method. - -Asynchronous iterators can be used in an :keyword:`async for` statement. - -.. method:: object.__aiter__(self) - - Must return an *asynchronous iterator* object. - -.. method:: object.__anext__(self) - - Must return an *awaitable* resulting in a next value of the iterator. Should - raise a :exc:`StopAsyncIteration` error when the iteration is over. - -An example of an asynchronous iterable object:: - - class Reader: - async def readline(self): - ... - - def __aiter__(self): - return self - - async def __anext__(self): - val = await self.readline() - if val == b'': - raise StopAsyncIteration - return val - -.. versionadded:: 3.5 - -.. versionchanged:: 3.7 - Prior to Python 3.7, ``__aiter__`` could return an *awaitable* - that would resolve to an - :term:`asynchronous iterator <asynchronous iterator>`. - - Starting with Python 3.7, ``__aiter__`` must return an - asynchronous iterator object. Returning anything else - will result in a :exc:`TypeError` error. - - -.. _async-context-managers: - -Asynchronous Context Managers ------------------------------ - -An *asynchronous context manager* is a *context manager* that is able to -suspend execution in its ``__aenter__`` and ``__aexit__`` methods. - -Asynchronous context managers can be used in an :keyword:`async with` statement. - -.. method:: object.__aenter__(self) - - Semantically similar to :meth:`__enter__`, the only - difference being that it must return an *awaitable*. - -.. method:: object.__aexit__(self, exc_type, exc_value, traceback) - - Semantically similar to :meth:`__exit__`, the only - difference being that it must return an *awaitable*. - -An example of an asynchronous context manager class:: - - class AsyncContextManager: - async def __aenter__(self): - await log('entering context') - - async def __aexit__(self, exc_type, exc, tb): - await log('exiting context') - -.. versionadded:: 3.5 - - .. rubric:: Footnotes .. [#] It *is* possible in some cases to change an object's type, under certain controlled conditions. It generally isn't a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly. -.. [#] The :meth:`__hash__`, :meth:`__iter__`, :meth:`__reversed__`, and - :meth:`__contains__` methods have special handling for this; others - will still raise a :exc:`TypeError`, but may do so by relying on - the behavior that ``None`` is not callable. - -.. [#] "Does not support" here means that the class has no such method, or - the method returns ``NotImplemented``. Do not set the method to - ``None`` if you want to force fallback to the right operand's reflected - method—that will instead have the opposite effect of explicitly - *blocking* such fallback. - .. [#] For operands of the same type, it is assumed that if the non-reflected method (such as :meth:`__add__`) fails the operation is not supported, which is why the reflected method is not called. + diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 55ac01b..91ac72f 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -5,34 +5,8 @@ Execution model *************** -.. index:: - single: execution model - pair: code; block - -.. _prog_structure: - -Structure of a program -====================== - -.. index:: block - -A Python program is constructed from code blocks. -A :dfn:`block` is a piece of Python program text that is executed as a unit. -The following are blocks: a module, a function body, and a class definition. -Each command typed interactively is a block. A script file (a file given as -standard input to the interpreter or specified as a command line argument to the -interpreter) is a code block. A script command (a command specified on the -interpreter command line with the :option:`-c` option) is a code block. -A module run as a top level script (as module ``__main__``) from the command -line using a :option:`-m` argument is also a code block. The string -argument passed to the built-in functions :func:`eval` and :func:`exec` is a -code block. +.. index:: single: execution model -.. index:: pair: execution; frame - -A code block is executed in an :dfn:`execution frame`. A frame contains some -administrative information (used for debugging) and determines where and how -execution continues after the code block's execution has completed. .. _naming: @@ -40,61 +14,51 @@ Naming and binding ================== .. index:: + pair: code; block single: namespace single: scope -.. _bind_names: - -Binding of names ----------------- - .. index:: single: name pair: binding; name :dfn:`Names` refer to objects. Names are introduced by name binding operations. +Each occurrence of a name in the program text refers to the :dfn:`binding` of +that name established in the innermost function block containing the use. -.. index:: single: from; import statement - -The following constructs bind names: formal parameters to functions, -:keyword:`import` statements, class and function definitions (these bind the -class or function name in the defining block), and targets that are identifiers -if occurring in an assignment, :keyword:`for` loop header, or after -:keyword:`!as` in a :keyword:`with` statement or :keyword:`except` clause. -The :keyword:`!import` statement -of the form ``from ... import *`` binds all names defined in the imported -module, except those beginning with an underscore. This form may only be used -at the module level. - -A target occurring in a :keyword:`del` statement is also considered bound for -this purpose (though the actual semantics are to unbind the name). - -Each assignment or import statement occurs within a block defined by a class or -function definition or at the module level (the top-level code block). - -.. index:: pair: free; variable - -If a name is bound in a block, it is a local variable of that block, unless -declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at -the module level, it is a global variable. (The variables of the module code -block are local and global.) If a variable is used in a code block but not -defined there, it is a :dfn:`free variable`. +.. index:: single: block -Each occurrence of a name in the program text refers to the :dfn:`binding` of -that name established by the following name resolution rules. +A :dfn:`block` is a piece of Python program text that is executed as a unit. +The following are blocks: a module, a function body, and a class definition. +Each command typed interactively is a block. A script file (a file given as +standard input to the interpreter or specified on the interpreter command line +the first argument) is a code block. A script command (a command specified on +the interpreter command line with the '**-c**' option) is a code block. The +file read by the built-in function :func:`execfile` is a code block. The string +argument passed to the built-in function :func:`eval` and to the :keyword:`exec` +statement is a code block. The expression read and evaluated by the built-in +function :func:`input` is a code block. -.. _resolve_names: +.. index:: pair: execution; frame -Resolution of names -------------------- +A code block is executed in an :dfn:`execution frame`. A frame contains some +administrative information (used for debugging) and determines where and how +execution continues after the code block's execution has completed. -.. index:: scope +.. index:: single: scope A :dfn:`scope` defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding -for the name. +for the name. The scope of names defined in a class block is limited to the +class block; it does not extend to the code blocks of methods -- this includes +generator expressions since they are implemented using a function scope. This +means that the following will fail:: + + class A: + a = 42 + b = list(a + i for i in range(10)) .. index:: single: environment @@ -102,84 +66,88 @@ When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block's :dfn:`environment`. +.. index:: pair: free; variable + +If a name is bound in a block, it is a local variable of that block. If a name +is bound at the module level, it is a global variable. (The variables of the +module code block are local and global.) If a variable is used in a code block +but not defined there, it is a :dfn:`free variable`. + .. index:: single: NameError (built-in exception) single: UnboundLocalError -When a name is not found at all, a :exc:`NameError` exception is raised. -If the current scope is a function scope, and the name refers to a local -variable that has not yet been bound to a value at the point where the name is -used, an :exc:`UnboundLocalError` exception is raised. -:exc:`UnboundLocalError` is a subclass of :exc:`NameError`. +When a name is not found at all, a :exc:`NameError` exception is raised. If the +name refers to a local variable that has not been bound, a +:exc:`UnboundLocalError` exception is raised. :exc:`UnboundLocalError` is a +subclass of :exc:`NameError`. + +.. index:: statement: from + +The following constructs bind names: formal parameters to functions, +:keyword:`import` statements, class and function definitions (these bind the +class or function name in the defining block), and targets that are identifiers +if occurring in an assignment, :keyword:`for` loop header, in the second +position of an :keyword:`except` clause header or after :keyword:`as` in a +:keyword:`with` statement. The :keyword:`import` statement +of the form ``from ... import *`` binds all names defined in the imported +module, except those beginning with an underscore. This form may only be used +at the module level. + +A target occurring in a :keyword:`del` statement is also considered bound for +this purpose (though the actual semantics are to unbind the name). It is +illegal to unbind a name that is referenced by an enclosing scope; the compiler +will report a :exc:`SyntaxError`. + +Each assignment or import statement occurs within a block defined by a class or +function definition or at the module level (the top-level code block). If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can -lead to errors when a name is used within a block before it is bound. This rule +lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. -If the :keyword:`global` statement occurs within a block, all uses of the name -specified in the statement refer to the binding of that name in the top-level -namespace. Names are resolved in the top-level namespace by searching the -global namespace, i.e. the namespace of the module containing the code block, -and the builtins namespace, the namespace of the module :mod:`builtins`. The -global namespace is searched first. If the name is not found there, the -builtins namespace is searched. The :keyword:`!global` statement must precede -all uses of the name. +If the global statement occurs within a block, all uses of the name specified in +the statement refer to the binding of that name in the top-level namespace. +Names are resolved in the top-level namespace by searching the global namespace, +i.e. the namespace of the module containing the code block, and the builtins +namespace, the namespace of the module :mod:`__builtin__`. The global namespace +is searched first. If the name is not found there, the builtins namespace is +searched. The global statement must precede all uses of the name. -The :keyword:`global` statement has the same scope as a name binding operation -in the same block. If the nearest enclosing scope for a free variable contains -a global statement, the free variable is treated as a global. +.. index:: pair: restricted; execution -.. XXX say more about "nonlocal" semantics here +The builtins namespace associated with the execution of a code block is actually +found by looking up the name ``__builtins__`` in its global namespace; this +should be a dictionary or a module (in the latter case the module's dictionary +is used). By default, when in the :mod:`__main__` module, ``__builtins__`` is +the built-in module :mod:`__builtin__` (note: no 's'); when in any other module, +``__builtins__`` is an alias for the dictionary of the :mod:`__builtin__` module +itself. ``__builtins__`` can be set to a user-created dictionary to create a +weak form of restricted execution. -The :keyword:`nonlocal` statement causes corresponding names to refer -to previously bound variables in the nearest enclosing function scope. -:exc:`SyntaxError` is raised at compile time if the given name does not -exist in any enclosing function scope. +.. impl-detail:: + + Users should not touch ``__builtins__``; it is strictly an implementation + detail. Users wanting to override values in the builtins namespace should + :keyword:`import` the :mod:`__builtin__` (no 's') module and modify its + attributes appropriately. .. index:: module: __main__ The namespace for a module is automatically created the first time a module is imported. The main module for a script is always called :mod:`__main__`. -Class definition blocks and arguments to :func:`exec` and :func:`eval` are -special in the context of name resolution. -A class definition is an executable statement that may use and define names. -These references follow the normal rules for name resolution with an exception -that unbound local variables are looked up in the global namespace. -The namespace of the class definition becomes the attribute dictionary of -the class. The scope of names defined in a class block is limited to the -class block; it does not extend to the code blocks of methods -- this includes -comprehensions and generator expressions since they are implemented using a -function scope. This means that the following will fail:: - - class A: - a = 42 - b = list(a + i for i in range(10)) - -.. _restrict_exec: - -Builtins and restricted execution ---------------------------------- - -.. index:: pair: restricted; execution - -.. impl-detail:: - - Users should not touch ``__builtins__``; it is strictly an implementation - detail. Users wanting to override values in the builtins namespace should - :keyword:`import` the :mod:`builtins` module and modify its - attributes appropriately. +The :keyword:`global` statement has the same scope as a name binding operation +in the same block. If the nearest enclosing scope for a free variable contains +a global statement, the free variable is treated as a global. -The builtins namespace associated with the execution of a code block -is actually found by looking up the name ``__builtins__`` in its -global namespace; this should be a dictionary or a module (in the -latter case the module's dictionary is used). By default, when in the -:mod:`__main__` module, ``__builtins__`` is the built-in module -:mod:`builtins`; when in any other module, ``__builtins__`` is an -alias for the dictionary of the :mod:`builtins` module itself. +A class definition is an executable statement that may use and define names. +These references follow the normal rules for name resolution. The namespace of +the class definition becomes the attribute dictionary of the class. Names +defined at the class scope are not visible in methods. .. _dynamic-features: @@ -187,23 +155,30 @@ alias for the dictionary of the :mod:`builtins` module itself. Interaction with dynamic features --------------------------------- -Name resolution of free variables occurs at runtime, not at compile time. -This means that the following code will print 42:: +There are several cases where Python statements are illegal when used in +conjunction with nested scopes that contain free variables. + +If a variable is referenced in an enclosing scope, it is illegal to delete the +name. An error will be reported at compile time. - i = 10 - def f(): - print(i) - i = 42 - f() +If the wild card form of import --- ``import *`` --- is used in a function and +the function contains or is a nested block with free variables, the compiler +will raise a :exc:`SyntaxError`. -.. XXX from * also invalid with relative imports (at least currently) +If :keyword:`exec` is used in a function and the function contains or is a +nested block with free variables, the compiler will raise a :exc:`SyntaxError` +unless the exec explicitly specifies the local namespace for the +:keyword:`exec`. (In other words, ``exec obj`` would be illegal, but ``exec obj +in ns`` would be legal.) -The :func:`eval` and :func:`exec` functions do not have access to the full -environment for resolving names. Names may be resolved in the local and global -namespaces of the caller. Free variables are not resolved in the nearest -enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and -:func:`eval` functions have optional arguments to override the global and local -namespace. If only one namespace is specified, it is used for both. +The :func:`eval`, :func:`execfile`, and :func:`input` functions and the +:keyword:`exec` statement do not have access to the full environment for +resolving names. Names may be resolved in the local and global namespaces of +the caller. Free variables are not resolved in the nearest enclosing namespace, +but in the global namespace. [#]_ The :keyword:`exec` statement and the +:func:`eval` and :func:`execfile` functions have optional arguments to override +the global and local namespace. If only one namespace is specified, it is used +for both. .. _exceptions: @@ -245,24 +220,28 @@ re-entering the offending piece of code from the top). When an exception is not handled at all, the interpreter terminates execution of the program, or returns to its interactive main loop. In either case, it prints -a stack traceback, except when the exception is :exc:`SystemExit`. +a stack backtrace, except when the exception is :exc:`SystemExit`. Exceptions are identified by class instances. The :keyword:`except` clause is selected depending on the class of the instance: it must reference the class of the instance or a base class thereof. The instance can be received by the handler and can carry additional information about the exceptional condition. +Exceptions can also be identified by strings, in which case the +:keyword:`except` clause is selected by object identity. An arbitrary value can +be raised along with the identifying string which can be passed to the handler. + .. note:: - Exception messages are not part of the Python API. Their contents may change - from one version of Python to the next without warning and should not be + Messages to exceptions are not part of the Python API. Their contents may + change from one version of Python to the next without warning and should not be relied on by code which will run under multiple versions of the interpreter. See also the description of the :keyword:`try` statement in section :ref:`try` and :keyword:`raise` statement in section :ref:`raise`. - .. rubric:: Footnotes -.. [#] This limitation occurs because the code that is executed by these operations - is not available at the time the module is compiled. +.. [#] This limitation occurs because the code that is executed by these operations is + not available at the time the module is compiled. + diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index d9db33a..6afc867 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -5,10 +5,12 @@ Expressions *********** -.. index:: expression, BNF +.. index:: single: expression This chapter explains the meaning of the elements of expressions in Python. +.. index:: single: BNF + **Syntax Notes:** In this and the following chapters, extended BNF notation will be used to describe syntax, not lexical analysis. When (one alternative of) a syntax rule has the form @@ -16,6 +18,8 @@ syntax rule has the form .. productionlist:: * name: `othername` +.. index:: single: syntax + and no semantics are given, the semantics of this form of ``name`` are the same as for ``othername``. @@ -28,19 +32,22 @@ Arithmetic conversions .. index:: pair: arithmetic; conversion When a description of an arithmetic operator below uses the phrase "the numeric -arguments are converted to a common type," this means that the operator -implementation for built-in types works as follows: +arguments are converted to a common type," the arguments are coerced using the +coercion rules listed at :ref:`coercion-rules`. If both arguments are standard +numeric types, the following coercions are applied: * If either argument is a complex number, the other is converted to complex; * otherwise, if either argument is a floating point number, the other is converted to floating point; -* otherwise, both must be integers and no conversion is necessary. +* otherwise, if either argument is a long integer, the other is converted to + long integer; -Some additional rules apply for certain operators (e.g., a string as a left -argument to the '%' operator). Extensions must define their own conversion -behavior. +* otherwise, both must be plain integers and no conversion is necessary. + +Some additional rules apply for certain operators (e.g., a string left argument +to the '%' operator). Extensions can define their own coercions. .. _atoms: @@ -48,16 +55,18 @@ behavior. Atoms ===== -.. index:: atom +.. index:: single: atom Atoms are the most basic elements of expressions. The simplest atoms are -identifiers or literals. Forms enclosed in parentheses, brackets or braces are -also categorized syntactically as atoms. The syntax for atoms is: +identifiers or literals. Forms enclosed in reverse quotes or in parentheses, +brackets or braces are also categorized syntactically as atoms. The syntax for +atoms is: .. productionlist:: atom: `identifier` | `literal` | `enclosure` - enclosure: `parenth_form` | `list_display` | `dict_display` | `set_display` - : | `generator_expression` | `yield_atom` + enclosure: `parenth_form` | `list_display` + : | `generator_expression` | `dict_display` | `set_display` + : | `string_conversion` | `yield_atom` .. _atom-identifiers: @@ -65,7 +74,9 @@ also categorized syntactically as atoms. The syntax for atoms is: Identifiers (Names) ------------------- -.. index:: name, identifier +.. index:: + single: name + single: identifier An identifier occurring as an atom is a name. See section :ref:`identifiers` for lexical definition and section :ref:`naming` for documentation of naming and @@ -94,6 +105,7 @@ long (longer than 255 characters), implementation defined truncation may happen. If the class name consists only of underscores, no transformation is done. + .. _atom-literals: Literals @@ -101,15 +113,15 @@ Literals .. index:: single: literal -Python supports string and bytes literals and various numeric literals: +Python supports string literals and various numeric literals: .. productionlist:: - literal: `stringliteral` | `bytesliteral` - : | `integer` | `floatnumber` | `imagnumber` + literal: `stringliteral` | `integer` | `longinteger` + : | `floatnumber` | `imagnumber` -Evaluation of a literal yields an object of the given type (string, bytes, -integer, floating point number, complex number) with the given value. The value -may be approximated in the case of floating point and imaginary (complex) +Evaluation of a literal yields an object of the given type (string, integer, +long integer, floating point number, complex number) with the given value. The +value may be approximated in the case of floating point and imaginary (complex) literals. See section :ref:`literals` for details. .. index:: @@ -128,14 +140,12 @@ value. Parenthesized forms ------------------- -.. index:: - single: parenthesized form - single: () (parentheses); tuple display +.. index:: single: parenthesized form A parenthesized form is an optional expression list enclosed in parentheses: .. productionlist:: - parenth_form: "(" [`starred_expression`] ")" + parenth_form: "(" [`expression_list`] ")" A parenthesized expression list yields whatever that expression list yields: if the list contains at least one comma, it yields a tuple; otherwise, it yields @@ -144,12 +154,12 @@ the single expression that makes up the expression list. .. index:: pair: empty; tuple An empty pair of parentheses yields an empty tuple object. Since tuples are -immutable, the same rules as for literals apply (i.e., two occurrences of the empty +immutable, the rules for literals apply (i.e., two occurrences of the empty tuple may or may not yield the same object). .. index:: single: comma - single: , (comma) + pair: tuple; display Note that tuples are not formed by the parentheses, but rather by use of the comma operator. The exception is the empty tuple, for which parentheses *are* @@ -157,12 +167,50 @@ required --- allowing unparenthesized "nothing" in expressions would cause ambiguities and allow common typos to pass uncaught. +.. _lists: + +List displays +------------- + +.. index:: + pair: list; display + pair: list; comprehensions + +A list display is a possibly empty series of expressions enclosed in square +brackets: + +.. productionlist:: + list_display: "[" [`expression_list` | `list_comprehension`] "]" + list_comprehension: `expression` `list_for` + list_for: "for" `target_list` "in" `old_expression_list` [`list_iter`] + old_expression_list: `old_expression` [("," `old_expression`)+ [","]] + old_expression: `or_test` | `old_lambda_expr` + list_iter: `list_for` | `list_if` + list_if: "if" `old_expression` [`list_iter`] + +.. index:: + pair: list; comprehensions + object: list + pair: empty; list + +A list display yields a new list object. Its contents are specified by +providing either a list of expressions or a list comprehension. When a +comma-separated list of expressions is supplied, its elements are evaluated from +left to right and placed into the list object in that order. When a list +comprehension is supplied, it consists of a single expression followed by at +least one :keyword:`for` clause and zero or more :keyword:`for` or :keyword:`if` +clauses. In this case, the elements of the new list are those that would be +produced by considering each of the :keyword:`for` or :keyword:`if` clauses a +block, nesting from left to right, and evaluating the expression to produce a +list element each time the innermost block is reached [#]_. + + .. _comprehensions: -Displays for lists, sets and dictionaries ------------------------------------------ +Displays for sets and dictionaries +---------------------------------- -For constructing a list, a set or a dictionary Python provides special syntax +For constructing a set or a dictionary Python provides special syntax called "displays", each of them in two flavors: * either the container contents are listed explicitly, or @@ -170,128 +218,62 @@ called "displays", each of them in two flavors: * they are computed via a set of looping and filtering instructions, called a :dfn:`comprehension`. -.. index:: - single: for; in comprehensions - single: if; in comprehensions - single: async for; in comprehensions - Common syntax elements for comprehensions are: .. productionlist:: comprehension: `expression` `comp_for` - comp_for: ["async"] "for" `target_list` "in" `or_test` [`comp_iter`] + comp_for: "for" `target_list` "in" `or_test` [`comp_iter`] comp_iter: `comp_for` | `comp_if` comp_if: "if" `expression_nocond` [`comp_iter`] The comprehension consists of a single expression followed by at least one -:keyword:`!for` clause and zero or more :keyword:`!for` or :keyword:`!if` clauses. +:keyword:`for` clause and zero or more :keyword:`for` or :keyword:`if` clauses. In this case, the elements of the new container are those that would be produced -by considering each of the :keyword:`!for` or :keyword:`!if` clauses a block, +by considering each of the :keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. -However, aside from the iterable expression in the leftmost :keyword:`!for` clause, -the comprehension is executed in a separate implicitly nested scope. This ensures -that names assigned to in the target list don't "leak" into the enclosing scope. - -The iterable expression in the leftmost :keyword:`!for` clause is evaluated -directly in the enclosing scope and then passed as an argument to the implicitly -nested scope. Subsequent :keyword:`!for` clauses and any filter condition in the -leftmost :keyword:`!for` clause cannot be evaluated in the enclosing scope as -they may depend on the values obtained from the leftmost iterable. For example: -``[x*y for x in range(10) for y in range(x, x+10)]``. - -To ensure the comprehension always results in a container of the appropriate -type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly -nested scope. - -.. index:: - single: await; in comprehensions - -Since Python 3.6, in an :keyword:`async def` function, an :keyword:`!async for` -clause may be used to iterate over a :term:`asynchronous iterator`. -A comprehension in an :keyword:`!async def` function may consist of either a -:keyword:`!for` or :keyword:`!async for` clause following the leading -expression, may contain additional :keyword:`!for` or :keyword:`!async for` -clauses, and may also use :keyword:`await` expressions. -If a comprehension contains either :keyword:`!async for` clauses -or :keyword:`!await` expressions it is called an -:dfn:`asynchronous comprehension`. An asynchronous comprehension may -suspend the execution of the coroutine function in which it appears. -See also :pep:`530`. - -.. versionadded:: 3.6 - Asynchronous comprehensions were introduced. - -.. versionchanged:: 3.8 - ``yield`` and ``yield from`` prohibited in the implicitly nested scope. - - -.. _lists: - -List displays -------------- - -.. index:: - pair: list; display - pair: list; comprehensions - pair: empty; list - object: list - single: [] (square brackets); list expression - single: , (comma); expression list - -A list display is a possibly empty series of expressions enclosed in square -brackets: - -.. productionlist:: - list_display: "[" [`starred_list` | `comprehension`] "]" - -A list display yields a new list object, the contents being specified by either -a list of expressions or a comprehension. When a comma-separated list of -expressions is supplied, its elements are evaluated from left to right and -placed into the list object in that order. When a comprehension is supplied, -the list is constructed from the elements resulting from the comprehension. +Note that the comprehension is executed in a separate scope, so names assigned +to in the target list don't "leak" in the enclosing scope. -.. _set: +.. _genexpr: -Set displays ------------- +Generator expressions +--------------------- -.. index:: - pair: set; display - object: set - single: {} (curly brackets); set expression - single: , (comma); expression list +.. index:: pair: generator; expression + object: generator -A set display is denoted by curly braces and distinguishable from dictionary -displays by the lack of colons separating keys and values: +A generator expression is a compact generator notation in parentheses: .. productionlist:: - set_display: "{" (`starred_list` | `comprehension`) "}" + generator_expression: "(" `expression` `comp_for` ")" -A set display yields a new mutable set object, the contents being specified by -either a sequence of expressions or a comprehension. When a comma-separated -list of expressions is supplied, its elements are evaluated from left to right -and added to the set object. When a comprehension is supplied, the set is -constructed from the elements resulting from the comprehension. +A generator expression yields a new generator object. Its syntax is the same as +for comprehensions, except that it is enclosed in parentheses instead of +brackets or curly braces. -An empty set cannot be constructed with ``{}``; this literal constructs an empty -dictionary. +Variables used in the generator expression are evaluated lazily when the +:meth:`__next__` method is called for generator object (in the same fashion as +normal generators). However, the leftmost :keyword:`for` clause is immediately +evaluated, so that an error produced by it can be seen before any other possible +error in the code that handles the generator expression. Subsequent +:keyword:`for` clauses cannot be evaluated immediately since they may depend on +the previous :keyword:`for` loop. For example: ``(x*y for x in range(10) for y +in bar(x))``. +The parentheses can be omitted on calls with only one argument. See section +:ref:`calls` for the detail. .. _dict: Dictionary displays ------------------- -.. index:: - pair: dictionary; display - key, datum, key/datum pair - object: dictionary - single: {} (curly brackets); dictionary expression - single: : (colon); in dictionary expressions - single: , (comma); in dictionary displays +.. index:: pair: dictionary; display + key, datum, key/datum pair + object: dictionary A dictionary display is a possibly empty series of key/datum pairs enclosed in curly braces: @@ -299,7 +281,7 @@ curly braces: .. productionlist:: dict_display: "{" [`key_datum_list` | `dict_comprehension`] "}" key_datum_list: `key_datum` ("," `key_datum`)* [","] - key_datum: `expression` ":" `expression` | "**" `or_expr` + key_datum: `expression` ":" `expression` dict_comprehension: `expression` ":" `expression` `comp_for` A dictionary display yields a new dictionary object. @@ -310,18 +292,6 @@ used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary's value for that key will be the last one given. -.. index:: - unpacking; dictionary - single: **; in dictionary displays - -A double asterisk ``**`` denotes :dfn:`dictionary unpacking`. -Its operand must be a :term:`mapping`. Each mapping item is added -to the new dictionary. Later values replace values already set by -earlier key/datum pairs and earlier dictionary unpackings. - -.. versionadded:: 3.5 - Unpacking into dictionary displays, originally proposed by :pep:`448`. - A dict comprehension, in contrast to list and set comprehensions, needs two expressions separated with a colon followed by the usual "for" and "if" clauses. When the comprehension is run, the resulting key and value elements are inserted @@ -336,66 +306,74 @@ all mutable objects.) Clashes between duplicate keys are not detected; the last datum (textually rightmost in the display) stored for a given key value prevails. -.. versionchanged:: 3.8 - Prior to Python 3.8, in dict comprehensions, the evaluation order of key - and value was not well-defined. In CPython, the value was evaluated before - the key. Starting with 3.8, the key is evaluated before the value, as - proposed by :pep:`572`. +.. _set: -.. _genexpr: +Set displays +------------ -Generator expressions ---------------------- +.. index:: pair: set; display + object: set + +A set display is denoted by curly braces and distinguishable from dictionary +displays by the lack of colons separating keys and values: + +.. productionlist:: + set_display: "{" (`expression_list` | `comprehension`) "}" + +A set display yields a new mutable set object, the contents being specified by +either a sequence of expressions or a comprehension. When a comma-separated +list of expressions is supplied, its elements are evaluated from left to right +and added to the set object. When a comprehension is supplied, the set is +constructed from the elements resulting from the comprehension. + +An empty set cannot be constructed with ``{}``; this literal constructs an empty +dictionary. + + +.. _string-conversions: + +String conversions +------------------ .. index:: - pair: generator; expression - object: generator - single: () (parentheses); generator expression + pair: string; conversion + pair: reverse; quotes + pair: backward; quotes + single: back-quotes -A generator expression is a compact generator notation in parentheses: +A string conversion is an expression list enclosed in reverse (a.k.a. backward) +quotes: .. productionlist:: - generator_expression: "(" `expression` `comp_for` ")" + string_conversion: "`" `expression_list` "`" -A generator expression yields a new generator object. Its syntax is the same as -for comprehensions, except that it is enclosed in parentheses instead of -brackets or curly braces. - -Variables used in the generator expression are evaluated lazily when the -:meth:`~generator.__next__` method is called for the generator object (in the same -fashion as normal generators). However, the iterable expression in the -leftmost :keyword:`!for` clause is immediately evaluated, so that an error -produced by it will be emitted at the point where the generator expression -is defined, rather than at the point where the first value is retrieved. -Subsequent :keyword:`!for` clauses and any filter condition in the leftmost -:keyword:`!for` clause cannot be evaluated in the enclosing scope as they may -depend on the values obtained from the leftmost iterable. For example: -``(x*y for x in range(10) for y in range(x, x+10))``. +A string conversion evaluates the contained expression list and converts the +resulting object into a string according to rules specific to its type. -The parentheses can be omitted on calls with only one argument. See section -:ref:`calls` for details. +If the object is a string, a number, ``None``, or a tuple, list or dictionary +containing only objects whose type is one of these, the resulting string is a +valid Python expression which can be passed to the built-in function +:func:`eval` to yield an expression with the same value (or an approximation, if +floating point numbers are involved). -To avoid interfering with the expected operation of the generator expression -itself, ``yield`` and ``yield from`` expressions are prohibited in the -implicitly defined generator. +(In particular, converting a string adds quotes around it and converts "funny" +characters to escape sequences that are safe to print.) -If a generator expression contains either :keyword:`!async for` -clauses or :keyword:`await` expressions it is called an -:dfn:`asynchronous generator expression`. An asynchronous generator -expression returns a new asynchronous generator object, -which is an asynchronous iterator (see :ref:`async-iterators`). +.. index:: object: recursive -.. versionadded:: 3.6 - Asynchronous generator expressions were introduced. +Recursive objects (for example, lists or dictionaries that contain a reference +to themselves, directly or indirectly) use ``...`` to indicate a recursive +reference, and the result cannot be passed to :func:`eval` to get an equal value +(:exc:`SyntaxError` will be raised instead). -.. versionchanged:: 3.7 - Prior to Python 3.7, asynchronous generator expressions could - only appear in :keyword:`async def` coroutines. Starting - with 3.7, any function can use asynchronous generator expressions. +.. index:: + builtin: repr + builtin: str -.. versionchanged:: 3.8 - ``yield`` and ``yield from`` prohibited in the implicitly nested scope. +The built-in function :func:`repr` performs exactly the same conversion in its +argument as enclosing it in parentheses and reverse quotes does. The built-in +function :func:`str` performs a similar but more user-friendly conversion. .. _yieldexpr: @@ -405,113 +383,43 @@ Yield expressions .. index:: keyword: yield - keyword: from pair: yield; expression pair: generator; function .. productionlist:: yield_atom: "(" `yield_expression` ")" - yield_expression: "yield" [`expression_list` | "from" `expression`] + yield_expression: "yield" [`expression_list`] -The yield expression is used when defining a :term:`generator` function -or an :term:`asynchronous generator` function and -thus can only be used in the body of a function definition. Using a yield -expression in a function's body causes that function to be a generator, -and using it in an :keyword:`async def` function's body causes that -coroutine function to be an asynchronous generator. For example:: +.. versionadded:: 2.5 - def gen(): # defines a generator function - yield 123 - - async def agen(): # defines an asynchronous generator function - yield 123 - -Due to their side effects on the containing scope, ``yield`` expressions -are not permitted as part of the implicitly defined scopes used to -implement comprehensions and generator expressions. - -.. versionchanged:: 3.8 - Yield expressions prohibited in the implicitly nested scopes used to - implement comprehensions and generator expressions. - -Generator functions are described below, while asynchronous generator -functions are described separately in section -:ref:`asynchronous-generator-functions`. +The :keyword:`yield` expression is only used when defining a generator function, +and can only be used in the body of a function definition. Using a +:keyword:`yield` expression in a function definition is sufficient to cause that +definition to create a generator function instead of a normal function. When a generator function is called, it returns an iterator known as a -generator. That generator then controls the execution of the generator function. +generator. That generator then controls the execution of a generator function. The execution starts when one of the generator's methods is called. At that -time, the execution proceeds to the first yield expression, where it is -suspended again, returning the value of :token:`expression_list` to the generator's -caller. By suspended, we mean that all local state is retained, including the -current bindings of local variables, the instruction pointer, the internal -evaluation stack, and the state of any exception handling. When the execution -is resumed by calling one of the -generator's methods, the function can proceed exactly as if the yield expression -were just another external call. The value of the yield expression after -resuming depends on the method which resumed the execution. If -:meth:`~generator.__next__` is used (typically via either a :keyword:`for` or -the :func:`next` builtin) then the result is :const:`None`. Otherwise, if -:meth:`~generator.send` is used, then the result will be the value passed in to -that method. +time, the execution proceeds to the first :keyword:`yield` expression, where it +is suspended again, returning the value of :token:`expression_list` to +generator's caller. By suspended we mean that all local state is retained, +including the current bindings of local variables, the instruction pointer, and +the internal evaluation stack. When the execution is resumed by calling one of +the generator's methods, the function can proceed exactly as if the +:keyword:`yield` expression was just another external call. The value of the +:keyword:`yield` expression after resuming depends on the method which resumed +the execution. .. index:: single: coroutine All of this makes generator functions quite similar to coroutines; they yield multiple times, they have more than one entry point and their execution can be suspended. The only difference is that a generator function cannot control -where the execution should continue after it yields; the control is always +where should the execution continue after it yields; the control is always transferred to the generator's caller. -Yield expressions are allowed anywhere in a :keyword:`try` construct. If the -generator is not resumed before it is -finalized (by reaching a zero reference count or by being garbage collected), -the generator-iterator's :meth:`~generator.close` method will be called, -allowing any pending :keyword:`finally` clauses to execute. - -.. index:: - single: from; yield from expression - -When ``yield from <expr>`` is used, it treats the supplied expression as -a subiterator. All values produced by that subiterator are passed directly -to the caller of the current generator's methods. Any values passed in with -:meth:`~generator.send` and any exceptions passed in with -:meth:`~generator.throw` are passed to the underlying iterator if it has the -appropriate methods. If this is not the case, then :meth:`~generator.send` -will raise :exc:`AttributeError` or :exc:`TypeError`, while -:meth:`~generator.throw` will just raise the passed in exception immediately. - -When the underlying iterator is complete, the :attr:`~StopIteration.value` -attribute of the raised :exc:`StopIteration` instance becomes the value of -the yield expression. It can be either set explicitly when raising -:exc:`StopIteration`, or automatically when the subiterator is a generator -(by returning a value from the subgenerator). - - .. versionchanged:: 3.3 - Added ``yield from <expr>`` to delegate control flow to a subiterator. - -The parentheses may be omitted when the yield expression is the sole expression -on the right hand side of an assignment statement. - -.. seealso:: - - :pep:`255` - Simple Generators - The proposal for adding generators and the :keyword:`yield` statement to Python. - - :pep:`342` - Coroutines via Enhanced Generators - The proposal to enhance the API and syntax of generators, making them - usable as simple coroutines. - - :pep:`380` - Syntax for Delegating to a Subgenerator - The proposal to introduce the :token:`yield_from` syntax, making delegation - to subgenerators easy. - - :pep:`525` - Asynchronous Generators - The proposal that expanded on :pep:`492` by adding generator capabilities to - coroutine functions. - .. index:: object: generator -.. _generator-methods: + Generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -525,35 +433,33 @@ is already executing raises a :exc:`ValueError` exception. .. index:: exception: StopIteration -.. method:: generator.__next__() +.. method:: generator.next() - Starts the execution of a generator function or resumes it at the last - executed yield expression. When a generator function is resumed with a - :meth:`~generator.__next__` method, the current yield expression always - evaluates to :const:`None`. The execution then continues to the next yield + Starts the execution of a generator function or resumes it at the last executed + :keyword:`yield` expression. When a generator function is resumed with a + :meth:`~generator.next` method, the current :keyword:`yield` expression + always evaluates to + :const:`None`. The execution then continues to the next :keyword:`yield` expression, where the generator is suspended again, and the value of the - :token:`expression_list` is returned to :meth:`__next__`'s caller. If the - generator exits without yielding another value, a :exc:`StopIteration` - exception is raised. - - This method is normally called implicitly, e.g. by a :keyword:`for` loop, or - by the built-in :func:`next` function. - + :token:`expression_list` is returned to :meth:`~generator.next`'s caller. + If the generator + exits without yielding another value, a :exc:`StopIteration` exception is + raised. .. method:: generator.send(value) Resumes the execution and "sends" a value into the generator function. The - *value* argument becomes the result of the current yield expression. The - :meth:`send` method returns the next value yielded by the generator, or - raises :exc:`StopIteration` if the generator exits without yielding another - value. When :meth:`send` is called to start the generator, it must be called - with :const:`None` as the argument, because there is no yield expression that - could receive the value. + ``value`` argument becomes the result of the current :keyword:`yield` + expression. The :meth:`send` method returns the next value yielded by the + generator, or raises :exc:`StopIteration` if the generator exits without + yielding another value. When :meth:`send` is called to start the generator, it + must be called with :const:`None` as the argument, because there is no + :keyword:`yield` expression that could receive the value. .. method:: generator.throw(type[, value[, traceback]]) - Raises an exception of type ``type`` at the point where the generator was paused, + Raises an exception of type ``type`` at the point where generator was paused, and returns the next value yielded by the generator function. If the generator exits without yielding another value, a :exc:`StopIteration` exception is raised. If the generator function does not catch the passed-in exception, or @@ -565,176 +471,47 @@ is already executing raises a :exc:`ValueError` exception. .. method:: generator.close() Raises a :exc:`GeneratorExit` at the point where the generator function was - paused. If the generator function then exits gracefully, is already closed, - or raises :exc:`GeneratorExit` (by not catching the exception), close - returns to its caller. If the generator yields a value, a - :exc:`RuntimeError` is raised. If the generator raises any other exception, - it is propagated to the caller. :meth:`close` does nothing if the generator - has already exited due to an exception or normal exit. - -.. index:: single: yield; examples - -Examples -^^^^^^^^ + paused. If the generator function then raises :exc:`StopIteration` (by exiting + normally, or due to already being closed) or :exc:`GeneratorExit` (by not + catching the exception), close returns to its caller. If the generator yields a + value, a :exc:`RuntimeError` is raised. If the generator raises any other + exception, it is propagated to the caller. :meth:`close` does nothing if the + generator has already exited due to an exception or normal exit. Here is a simple example that demonstrates the behavior of generators and generator functions:: >>> def echo(value=None): - ... print("Execution starts when 'next()' is called for the first time.") + ... print "Execution starts when 'next()' is called for the first time." ... try: ... while True: ... try: ... value = (yield value) - ... except Exception as e: + ... except Exception, e: ... value = e ... finally: - ... print("Don't forget to clean up when 'close()' is called.") + ... print "Don't forget to clean up when 'close()' is called." ... >>> generator = echo(1) - >>> print(next(generator)) + >>> print generator.next() Execution starts when 'next()' is called for the first time. 1 - >>> print(next(generator)) + >>> print generator.next() None - >>> print(generator.send(2)) + >>> print generator.send(2) 2 >>> generator.throw(TypeError, "spam") TypeError('spam',) >>> generator.close() Don't forget to clean up when 'close()' is called. -For examples using ``yield from``, see :ref:`pep-380` in "What's New in -Python." - -.. _asynchronous-generator-functions: - -Asynchronous generator functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The presence of a yield expression in a function or method defined using -:keyword:`async def` further defines the function as an -:term:`asynchronous generator` function. - -When an asynchronous generator function is called, it returns an -asynchronous iterator known as an asynchronous generator object. -That object then controls the execution of the generator function. -An asynchronous generator object is typically used in an -:keyword:`async for` statement in a coroutine function analogously to -how a generator object would be used in a :keyword:`for` statement. - -Calling one of the asynchronous generator's methods returns an -:term:`awaitable` object, and the execution starts when this object -is awaited on. At that time, the execution proceeds to the first yield -expression, where it is suspended again, returning the value of -:token:`expression_list` to the awaiting coroutine. As with a generator, -suspension means that all local state is retained, including the -current bindings of local variables, the instruction pointer, the internal -evaluation stack, and the state of any exception handling. When the execution -is resumed by awaiting on the next object returned by the asynchronous -generator's methods, the function can proceed exactly as if the yield -expression were just another external call. The value of the yield expression -after resuming depends on the method which resumed the execution. If -:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if -:meth:`~agen.asend` is used, then the result will be the value passed in to -that method. - -In an asynchronous generator function, yield expressions are allowed anywhere -in a :keyword:`try` construct. However, if an asynchronous generator is not -resumed before it is finalized (by reaching a zero reference count or by -being garbage collected), then a yield expression within a :keyword:`!try` -construct could result in a failure to execute pending :keyword:`finally` -clauses. In this case, it is the responsibility of the event loop or -scheduler running the asynchronous generator to call the asynchronous -generator-iterator's :meth:`~agen.aclose` method and run the resulting -coroutine object, thus allowing any pending :keyword:`!finally` clauses -to execute. - -To take care of finalization, an event loop should define -a *finalizer* function which takes an asynchronous generator-iterator -and presumably calls :meth:`~agen.aclose` and executes the coroutine. -This *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`. -When first iterated over, an asynchronous generator-iterator will store the -registered *finalizer* to be called upon finalization. For a reference example -of a *finalizer* method see the implementation of -``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`. - -The expression ``yield from <expr>`` is a syntax error when used in an -asynchronous generator function. - -.. index:: object: asynchronous-generator -.. _asynchronous-generator-methods: - -Asynchronous generator-iterator methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This subsection describes the methods of an asynchronous generator iterator, -which are used to control the execution of a generator function. - - -.. index:: exception: StopAsyncIteration - -.. coroutinemethod:: agen.__anext__() - - Returns an awaitable which when run starts to execute the asynchronous - generator or resumes it at the last executed yield expression. When an - asynchronous generator function is resumed with an :meth:`~agen.__anext__` - method, the current yield expression always evaluates to :const:`None` in - the returned awaitable, which when run will continue to the next yield - expression. The value of the :token:`expression_list` of the yield - expression is the value of the :exc:`StopIteration` exception raised by - the completing coroutine. If the asynchronous generator exits without - yielding another value, the awaitable instead raises a - :exc:`StopAsyncIteration` exception, signalling that the asynchronous - iteration has completed. - - This method is normally called implicitly by a :keyword:`async for` loop. - - -.. coroutinemethod:: agen.asend(value) - - Returns an awaitable which when run resumes the execution of the - asynchronous generator. As with the :meth:`~generator.send()` method for a - generator, this "sends" a value into the asynchronous generator function, - and the *value* argument becomes the result of the current yield expression. - The awaitable returned by the :meth:`asend` method will return the next - value yielded by the generator as the value of the raised - :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the - asynchronous generator exits without yielding another value. When - :meth:`asend` is called to start the asynchronous - generator, it must be called with :const:`None` as the argument, - because there is no yield expression that could receive the value. - - -.. coroutinemethod:: agen.athrow(type[, value[, traceback]]) - - Returns an awaitable that raises an exception of type ``type`` at the point - where the asynchronous generator was paused, and returns the next value - yielded by the generator function as the value of the raised - :exc:`StopIteration` exception. If the asynchronous generator exits - without yielding another value, a :exc:`StopAsyncIteration` exception is - raised by the awaitable. - If the generator function does not catch the passed-in exception, or - raises a different exception, then when the awaitable is run that exception - propagates to the caller of the awaitable. - -.. index:: exception: GeneratorExit +.. seealso:: -.. coroutinemethod:: agen.aclose() + :pep:`342` - Coroutines via Enhanced Generators + The proposal to enhance the API and syntax of generators, making them usable as + simple coroutines. - Returns an awaitable that when run will throw a :exc:`GeneratorExit` into - the asynchronous generator function at the point where it was paused. - If the asynchronous generator function then exits gracefully, is already - closed, or raises :exc:`GeneratorExit` (by not catching the exception), - then the returned awaitable will raise a :exc:`StopIteration` exception. - Any further awaitables returned by subsequent calls to the asynchronous - generator will raise a :exc:`StopAsyncIteration` exception. If the - asynchronous generator yields a value, a :exc:`RuntimeError` is raised - by the awaitable. If the asynchronous generator raises any other exception, - it is propagated to the caller of the awaitable. If the asynchronous - generator has already exited due to an exception or normal exit, then - further calls to :meth:`aclose` will return an awaitable that does nothing. .. _primaries: @@ -755,9 +532,7 @@ syntax is: Attribute references -------------------- -.. index:: - pair: attribute; reference - single: . (dot); attribute reference +.. index:: pair: attribute; reference An attribute reference is a primary followed by a period and a name: @@ -770,12 +545,11 @@ An attribute reference is a primary followed by a period and a name: object: list The primary must evaluate to an object of a type that supports attribute -references, which most objects do. This object is then asked to produce the -attribute whose name is the identifier. This production can be customized by -overriding the :meth:`__getattr__` method. If this attribute is not available, -the exception :exc:`AttributeError` is raised. Otherwise, the type and value of -the object produced is determined by the object. Multiple evaluations of the -same attribute reference may yield different objects. +references, e.g., a module, list, or an instance. This object is then asked to +produce the attribute whose name is the identifier. If this attribute is not +available, the exception :exc:`AttributeError` is raised. Otherwise, the type +and value of the object produced is determined by the object. Multiple +evaluations of the same attribute reference may yield different objects. .. _subscriptions: @@ -783,9 +557,7 @@ same attribute reference may yield different objects. Subscriptions ------------- -.. index:: - single: subscription - single: [] (square brackets); subscription +.. index:: single: subscription .. index:: object: sequence @@ -802,29 +574,19 @@ A subscription selects an item of a sequence (string, tuple or list) or mapping .. productionlist:: subscription: `primary` "[" `expression_list` "]" -The primary must evaluate to an object that supports subscription (lists or -dictionaries for example). User-defined objects can support subscription by -defining a :meth:`__getitem__` method. - -For built-in objects, there are two types of objects that support subscription: +The primary must evaluate to an object of a sequence or mapping type. If the primary is a mapping, the expression list must evaluate to an object whose value is one of the keys of the mapping, and the subscription selects the value in the mapping that corresponds to that key. (The expression list is a tuple except if it has exactly one item.) -If the primary is a sequence, the expression list must evaluate to an integer -or a slice (as discussed in the following section). - -The formal syntax makes no special provision for negative indices in -sequences; however, built-in sequences all provide a :meth:`__getitem__` -method that interprets negative indices by adding the length of the sequence -to the index (so that ``x[-1]`` selects the last item of ``x``). The -resulting value must be a nonnegative integer less than the number of items in -the sequence, and the subscription selects the item whose index is that value -(counting from zero). Since the support for negative indices and slicing -occurs in the object's :meth:`__getitem__` method, subclasses overriding -this method will need to explicitly add that support. +If the primary is a sequence, the expression list must evaluate to a plain +integer. If this value is negative, the length of the sequence is added to it +(so that, e.g., ``x[-1]`` selects the last item of ``x``.) The resulting value +must be a nonnegative integer less than the number of items in the sequence, and +the subscription selects the item whose index is that value (counting from +zero). .. index:: single: character @@ -842,8 +604,6 @@ Slicings .. index:: single: slicing single: slice - single: : (colon); slicing - single: , (comma); slicing .. index:: object: sequence @@ -856,34 +616,52 @@ or list). Slicings may be used as expressions or as targets in assignment or :keyword:`del` statements. The syntax for a slicing: .. productionlist:: - slicing: `primary` "[" `slice_list` "]" + slicing: `simple_slicing` | `extended_slicing` + simple_slicing: `primary` "[" `short_slice` "]" + extended_slicing: `primary` "[" `slice_list` "]" slice_list: `slice_item` ("," `slice_item`)* [","] - slice_item: `expression` | `proper_slice` - proper_slice: [`lower_bound`] ":" [`upper_bound`] [ ":" [`stride`] ] + slice_item: `expression` | `proper_slice` | `ellipsis` + proper_slice: `short_slice` | `long_slice` + short_slice: [`lower_bound`] ":" [`upper_bound`] + long_slice: `short_slice` ":" [`stride`] lower_bound: `expression` upper_bound: `expression` stride: `expression` + ellipsis: "..." + +.. index:: pair: extended; slicing There is ambiguity in the formal syntax here: anything that looks like an expression list also looks like a slice list, so any subscription can be interpreted as a slicing. Rather than further complicating the syntax, this is disambiguated by defining that in this case the interpretation as a subscription takes priority over the interpretation as a slicing (this is the case if the -slice list contains no proper slice). +slice list contains no proper slice nor ellipses). Similarly, when the slice +list has exactly one short slice and no trailing comma, the interpretation as a +simple slicing takes priority over that as an extended slicing. + +The semantics for a simple slicing are as follows. The primary must evaluate to +a sequence object. The lower and upper bound expressions, if present, must +evaluate to plain integers; defaults are zero and the ``sys.maxint``, +respectively. If either bound is negative, the sequence's length is added to +it. The slicing now selects all items with index *k* such that ``i <= k < j`` +where *i* and *j* are the specified lower and upper bounds. This may be an +empty sequence. It is not an error if *i* or *j* lie outside the range of valid +indexes (such items don't exist so they aren't selected). .. index:: single: start (slice object attribute) single: stop (slice object attribute) single: step (slice object attribute) -The semantics for a slicing are as follows. The primary is indexed (using the -same :meth:`__getitem__` method as -normal subscription) with a key that is constructed from the slice list, as -follows. If the slice list contains at least one comma, the key is a tuple -containing the conversion of the slice items; otherwise, the conversion of the -lone slice item is the key. The conversion of a slice item that is an -expression is that expression. The conversion of a proper slice is a slice -object (see section :ref:`types`) whose :attr:`~slice.start`, +The semantics for an extended slicing are as follows. The primary must evaluate +to a mapping object, and it is indexed with a key that is constructed from the +slice list, as follows. If the slice list contains at least one comma, the key +is a tuple containing the conversion of the slice items; otherwise, the +conversion of the lone slice item is the key. The conversion of a slice item +that is an expression is that expression. The conversion of an ellipsis slice +item is the built-in ``Ellipsis`` object. The conversion of a proper slice is a +slice object (see section :ref:`types`) whose :attr:`~slice.start`, :attr:`~slice.stop` and :attr:`~slice.step` attributes are the values of the expressions given as lower bound, upper bound and stride, respectively, substituting ``None`` for missing expressions. @@ -893,9 +671,6 @@ substituting ``None`` for missing expressions. object: callable single: call single: argument; call semantics - single: () (parentheses); call - single: , (comma); argument list - single: = (equals); in function calls .. _calls: @@ -906,31 +681,31 @@ A call calls a callable object (e.g., a :term:`function`) with a possibly empty series of :term:`arguments <argument>`: .. productionlist:: - call: `primary` "(" [`argument_list` [","] | `comprehension`] ")" - argument_list: `positional_arguments` ["," `starred_and_keywords`] - : ["," `keywords_arguments`] - : | `starred_and_keywords` ["," `keywords_arguments`] - : | `keywords_arguments` - positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* - starred_and_keywords: ("*" `expression` | `keyword_item`) - : ("," "*" `expression` | "," `keyword_item`)* - keywords_arguments: (`keyword_item` | "**" `expression`) - : ("," `keyword_item` | "," "**" `expression`)* + call: `primary` "(" [`argument_list` [","] + : | `expression` `genexpr_for`] ")" + argument_list: `positional_arguments` ["," `keyword_arguments`] + : ["," "*" `expression`] ["," `keyword_arguments`] + : ["," "**" `expression`] + : | `keyword_arguments` ["," "*" `expression`] + : ["," "**" `expression`] + : | "*" `expression` ["," `keyword_arguments`] ["," "**" `expression`] + : | "**" `expression` + positional_arguments: `expression` ("," `expression`)* + keyword_arguments: `keyword_item` ("," `keyword_item`)* keyword_item: `identifier` "=" `expression` -An optional trailing comma may be present after the positional and keyword arguments -but does not affect the semantics. +A trailing comma may be present after the positional and keyword arguments but +does not affect the semantics. .. index:: single: parameter; call semantics The primary must evaluate to a callable object (user-defined functions, built-in functions, methods of built-in objects, class objects, methods of class -instances, and all objects having a :meth:`__call__` method are callable). All -argument expressions are evaluated before the call is attempted. Please refer -to section :ref:`function` for the syntax of formal :term:`parameter` lists. - -.. XXX update with kwonly args PEP +instances, and certain class instances themselves are callable; extensions may +define additional callable object types). All argument expressions are +evaluated before the call is attempted. Please refer to section :ref:`function` +for the syntax of formal :term:`parameter` lists. If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the @@ -972,22 +747,21 @@ and the argument values as corresponding values), or a (new) empty dictionary if there were no excess keyword arguments. .. index:: - single: * (asterisk); in function calls - single: unpacking; in function calls + single: *; in function calls If the syntax ``*expression`` appears in the function call, ``expression`` must -evaluate to an :term:`iterable`. Elements from these iterables are -treated as if they were additional positional arguments. For the call -``f(x1, x2, *y, x3, x4)``, if *y* evaluates to a sequence *y1*, ..., *yM*, -this is equivalent to a call with M+4 positional arguments *x1*, *x2*, -*y1*, ..., *yM*, *x3*, *x4*. +evaluate to an iterable. Elements from this iterable are treated as if they +were additional positional arguments; if there are positional arguments +*x1*, ..., *xN*, and ``expression`` evaluates to a sequence *y1*, ..., *yM*, this +is equivalent to a call with M+N positional arguments *x1*, ..., *xN*, *y1*, +..., *yM*. A consequence of this is that although the ``*expression`` syntax may appear -*after* explicit keyword arguments, it is processed *before* the -keyword arguments (and any ``**expression`` arguments -- see below). So:: +*after* some keyword arguments, it is processed *before* the keyword arguments +(and the ``**expression`` argument, if any -- see below). So:: >>> def f(a, b): - ... print(a, b) + ... print a, b ... >>> f(b=1, *(2,)) 2 1 @@ -1005,19 +779,16 @@ used in the same call, so in practice this confusion does not arise. single: **; in function calls If the syntax ``**expression`` appears in the function call, ``expression`` must -evaluate to a :term:`mapping`, the contents of which are treated as -additional keyword arguments. If a keyword is already present -(as an explicit keyword argument, or from another unpacking), -a :exc:`TypeError` exception is raised. +evaluate to a mapping, the contents of which are treated as additional keyword +arguments. In the case of a keyword appearing in both ``expression`` and as an +explicit keyword argument, a :exc:`TypeError` exception is raised. Formal parameters using the syntax ``*identifier`` or ``**identifier`` cannot be -used as positional argument slots or as keyword argument names. - -.. versionchanged:: 3.5 - Function calls accept any number of ``*`` and ``**`` unpackings, - positional arguments may follow iterable unpackings (``*``), - and keyword arguments may follow dictionary unpackings (``**``). - Originally proposed by :pep:`448`. +used as positional argument slots or as keyword argument names. Formal +parameters using the syntax ``(sublist)`` cannot be used as keyword argument +names; the outermost sublist corresponds to a single unnamed argument slot, and +the argument value is assigned to the sublist using the usual tuple assignment +rules after all other parameter processing is done. A call always returns some value, possibly ``None``, unless it raises an exception. How this value is computed depends on the type of the callable @@ -1078,35 +849,16 @@ a class instance: if that method was called. -.. index:: keyword: await -.. _await: - -Await expression -================ - -Suspend the execution of :term:`coroutine` on an :term:`awaitable` object. -Can only be used inside a :term:`coroutine function`. - -.. productionlist:: - await_expr: "await" `primary` - -.. versionadded:: 3.5 - - .. _power: The power operator ================== -.. index:: - pair: power; operation - operator: ** - The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is: .. productionlist:: - power: (`await_expr` | `primary`) ["**" `u_expr`] + power: `primary` ["**" `u_expr`] Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order @@ -1115,16 +867,19 @@ for the operands): ``-1**2`` results in ``-1``. The power operator has the same semantics as the built-in :func:`pow` function, when called with two arguments: it yields its left argument raised to the power of its right argument. The numeric arguments are first converted to a common -type, and the result is of that type. +type. The result type is that of the arguments after coercion. -For int operands, the result has the same type as the operands unless the second -argument is negative; in that case, all arguments are converted to float and a -float result is delivered. For example, ``10**2`` returns ``100``, but -``10**-2`` returns ``0.01``. +With mixed operand types, the coercion rules for binary arithmetic operators +apply. For int and long int operands, the result has the same type as the +operands (after coercion) unless the second argument is negative; in that case, +all arguments are converted to float and a float result is delivered. For +example, ``10**2`` returns ``100``, but ``10**-2`` returns ``0.01``. (This last +feature was added in Python 2.2. In Python 2.1 and before, if both arguments +were of integer types and the second argument was negative, an exception was +raised). Raising ``0.0`` to a negative power results in a :exc:`ZeroDivisionError`. -Raising a negative number to a fractional power results in a :class:`complex` -number. (In earlier versions it raised a :exc:`ValueError`.) +Raising a negative number to a fractional power results in a :exc:`ValueError`. .. _unary: @@ -1144,25 +899,18 @@ All unary arithmetic and bitwise operations have the same priority: .. index:: single: negation single: minus - single: operator; - (minus) - single: - (minus); unary operator The unary ``-`` (minus) operator yields the negation of its numeric argument. -.. index:: - single: plus - single: operator; + (plus) - single: + (plus); unary operator +.. index:: single: plus The unary ``+`` (plus) operator yields its numeric argument unchanged. -.. index:: - single: inversion - operator: ~ (tilde) +.. index:: single: inversion -The unary ``~`` (invert) operator yields the bitwise inversion of its integer -argument. The bitwise inversion of ``x`` is defined as ``-(x+1)``. It only -applies to integral numbers. +The unary ``~`` (invert) operator yields the bitwise inversion of its plain or +long integer argument. The bitwise inversion of ``x`` is defined as +``-(x+1)``. It only applies to integral numbers. .. index:: exception: TypeError @@ -1183,46 +931,30 @@ from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: - m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | - : `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` | - : `m_expr` "%" `u_expr` + m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` + : | `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` -.. index:: - single: multiplication - operator: * (asterisk) +.. index:: single: multiplication The ``*`` (multiplication) operator yields the product of its arguments. The -arguments must either both be numbers, or one argument must be an integer and -the other must be a sequence. In the former case, the numbers are converted to a -common type and then multiplied together. In the latter case, sequence -repetition is performed; a negative repetition factor yields an empty sequence. - -.. index:: - single: matrix multiplication - operator: @ (at) - -The ``@`` (at) operator is intended to be used for matrix multiplication. No -builtin Python types implement this operator. - -.. versionadded:: 3.5 +arguments must either both be numbers, or one argument must be an integer (plain +or long) and the other must be a sequence. In the former case, the numbers are +converted to a common type and then multiplied together. In the latter case, +sequence repetition is performed; a negative repetition factor yields an empty +sequence. .. index:: exception: ZeroDivisionError single: division - operator: / (slash) - operator: // The ``/`` (division) and ``//`` (floor division) operators yield the quotient of their arguments. The numeric arguments are first converted to a common type. -Division of integers yields a float, while floor division of integers results in an -integer; the result is that of mathematical division with the 'floor' function -applied to the result. Division by zero raises the :exc:`ZeroDivisionError` -exception. +Plain or long integer division yields an integer of the same type; the result is +that of mathematical division with the 'floor' function applied to the result. +Division by zero raises the :exc:`ZeroDivisionError` exception. -.. index:: - single: modulo - operator: % (percent) +.. index:: single: modulo The ``%`` (modulo) operator yields the remainder from the division of the first argument by the second. The numeric arguments are first converted to a common @@ -1233,34 +965,31 @@ result with the same sign as its second operand (or zero); the absolute value of the result is strictly smaller than the absolute value of the second operand [#]_. -The floor division and modulo operators are connected by the following -identity: ``x == (x//y)*y + (x%y)``. Floor division and modulo are also -connected with the built-in function :func:`divmod`: ``divmod(x, y) == (x//y, -x%y)``. [#]_. +The integer division and modulo operators are connected by the following +identity: ``x == (x/y)*y + (x%y)``. Integer division and modulo are also +connected with the built-in function :func:`divmod`: ``divmod(x, y) == (x/y, +x%y)``. These identities don't hold for floating point numbers; there similar +identities hold approximately where ``x/y`` is replaced by ``floor(x/y)`` or +``floor(x/y) - 1`` [#]_. In addition to performing the modulo operation on numbers, the ``%`` operator is -also overloaded by string objects to perform old-style string formatting (also -known as interpolation). The syntax for string formatting is described in the -Python Library Reference, section :ref:`old-string-formatting`. +also overloaded by string and unicode objects to perform string formatting (also +known as interpolation). The syntax for string formatting is described in the +Python Library Reference, section :ref:`string-formatting`. -The floor division operator, the modulo operator, and the :func:`divmod` -function are not defined for complex numbers. Instead, convert to a floating -point number using the :func:`abs` function if appropriate. +.. deprecated:: 2.3 + The floor division operator, the modulo operator, and the :func:`divmod` + function are no longer defined for complex numbers. Instead, convert to a + floating point number using the :func:`abs` function if appropriate. -.. index:: - single: addition - single: operator; + (plus) - single: + (plus); binary operator +.. index:: single: addition -The ``+`` (addition) operator yields the sum of its arguments. The arguments -must either both be numbers or both be sequences of the same type. In the -former case, the numbers are converted to a common type and then added together. -In the latter case, the sequences are concatenated. +The ``+`` (addition) operator yields the sum of its arguments. The arguments +must either both be numbers or both sequences of the same type. In the former +case, the numbers are converted to a common type and then added together. In +the latter case, the sequences are concatenated. -.. index:: - single: subtraction - single: operator; - (minus) - single: - (minus); binary operator +.. index:: single: subtraction The ``-`` (subtraction) operator yields the difference of its arguments. The numeric arguments are first converted to a common type. @@ -1271,24 +1000,28 @@ numeric arguments are first converted to a common type. Shifting operations =================== -.. index:: - pair: shifting; operation - operator: << - operator: >> +.. index:: pair: shifting; operation The shifting operations have lower priority than the arithmetic operations: .. productionlist:: - shift_expr: `a_expr` | `shift_expr` ("<<" | ">>") `a_expr` + shift_expr: `a_expr` | `shift_expr` ( "<<" | ">>" ) `a_expr` -These operators accept integers as arguments. They shift the first argument to -the left or right by the number of bits given by the second argument. +These operators accept plain or long integers as arguments. The arguments are +converted to a common type. They shift the first argument to the left or right +by the number of bits given by the second argument. .. index:: exception: ValueError -A right shift by *n* bits is defined as floor division by ``pow(2,n)``. A left -shift by *n* bits is defined as multiplication with ``pow(2,n)``. +A right shift by *n* bits is defined as division by ``pow(2, n)``. A left shift +by *n* bits is defined as multiplication with ``pow(2, n)``. Negative shift +counts raise a :exc:`ValueError` exception. +.. note:: + + In the current implementation, the right-hand operand is required + to be at most :attr:`sys.maxsize`. If the right-hand operand is larger than + :attr:`sys.maxsize` an :exc:`OverflowError` exception is raised. .. _bitwise: @@ -1304,28 +1037,24 @@ Each of the three bitwise operations has a different priority level: xor_expr: `and_expr` | `xor_expr` "^" `and_expr` or_expr: `xor_expr` | `or_expr` "|" `xor_expr` -.. index:: - pair: bitwise; and - operator: & (ampersand) +.. index:: pair: bitwise; and -The ``&`` operator yields the bitwise AND of its arguments, which must be -integers. +The ``&`` operator yields the bitwise AND of its arguments, which must be plain +or long integers. The arguments are converted to a common type. .. index:: pair: bitwise; xor pair: exclusive; or - operator: ^ (caret) The ``^`` operator yields the bitwise XOR (exclusive OR) of its arguments, which -must be integers. +must be plain or long integers. The arguments are converted to a common type. .. index:: pair: bitwise; or pair: inclusive; or - operator: | (vertical bar) The ``|`` operator yields the bitwise (inclusive) OR of its arguments, which -must be integers. +must be plain or long integers. The arguments are converted to a common type. .. _comparisons: @@ -1333,15 +1062,9 @@ must be integers. Comparisons =========== -.. index:: - single: comparison - pair: C; language - operator: < (less) - operator: > (greater) - operator: <= - operator: >= - operator: == - operator: != +.. index:: single: comparison + +.. index:: pair: C; language Unlike C, all comparison operations in Python have the same priority, which is lower than that of any arithmetic, shifting or bitwise operation. Also unlike @@ -1349,8 +1072,8 @@ C, expressions like ``a < b < c`` have the interpretation that is conventional in mathematics: .. productionlist:: - comparison: `or_expr` (`comp_operator` `or_expr`)* - comp_operator: "<" | ">" | "==" | ">=" | "<=" | "!=" + comparison: `or_expr` ( `comp_operator` `or_expr` )* + comp_operator: "<" | ">" | "==" | ">=" | "<=" | "<>" | "!=" : | "is" ["not"] | ["not"] "in" Comparisons yield boolean values: ``True`` or ``False``. @@ -1370,6 +1093,10 @@ Note that ``a op1 b op2 c`` doesn't imply any kind of comparison between *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal (though perhaps not pretty). +The forms ``<>`` and ``!=`` are equivalent; for consistency with C, ``!=`` is +preferred; where ``!=`` is mentioned below ``<>`` is also accepted. The ``<>`` +spelling is considered obsolescent. + Value comparisons ----------------- @@ -1385,9 +1112,8 @@ implement a particular notion of what the value of an object is. One can think of them as defining the value of an object indirectly, by means of their comparison implementation. -Because all types are (direct or indirect) subtypes of :class:`object`, they -inherit the default comparison behavior from :class:`object`. Types can -customize their comparison behavior by implementing +Types can customize their comparison behavior by implementing +a :meth:`__cmp__` method or :dfn:`rich comparison methods` like :meth:`__lt__`, described in :ref:`customization`. @@ -1398,9 +1124,13 @@ different identities results in inequality. A motivation for this default behavior is the desire that all objects should be reflexive (i.e. ``x is y`` implies ``x == y``). -A default order comparison (``<``, ``>``, ``<=``, and ``>=``) is not provided; -an attempt raises :exc:`TypeError`. A motivation for this default behavior is -the lack of a similar invariant as for equality. +The default order comparison (``<``, ``>``, ``<=``, and ``>=``) gives a +consistent but arbitrary order. + +(This unusual definition of comparison was used to simplify the definition of +operations like sorting and the :keyword:`in` and :keyword:`not in` operators. +In the future, the comparison rules for objects of different types are likely to +change.) The behavior of the default equality comparison, that instances with different identities are always unequal, may be in contrast to what types will need that @@ -1418,36 +1148,28 @@ built-in types. involved, they compare mathematically (algorithmically) correct without loss of precision. - The not-a-number values ``float('NaN')`` and ``decimal.Decimal('NaN')`` are - special. Any ordered comparison of a number to a not-a-number value is false. - A counter-intuitive implication is that not-a-number values are not equal to - themselves. For example, if ``x = float('NaN')``, ``3 < x``, ``x < 3``, ``x - == x``, ``x != x`` are all false. This behavior is compliant with IEEE 754. - -* ``None`` and ``NotImplemented`` are singletons. :PEP:`8` advises that - comparisons for singletons should always be done with ``is`` or ``is not``, - never the equality operators. - -* Binary sequences (instances of :class:`bytes` or :class:`bytearray`) can be - compared within and across their types. They compare lexicographically using - the numeric values of their elements. - -* Strings (instances of :class:`str`) compare lexicographically using the - numerical Unicode code points (the result of the built-in function - :func:`ord`) of their characters. [#]_ - - Strings and binary sequences cannot be directly compared. - -* Sequences (instances of :class:`tuple`, :class:`list`, or :class:`range`) can - be compared only within each of their types, with the restriction that ranges - do not support order comparison. Equality comparison across these types - results in inequality, and ordering comparison across these types raises - :exc:`TypeError`. - - Sequences compare lexicographically using comparison of corresponding - elements. The built-in containers typically assume identical objects are - equal to themselves. That lets them bypass equality tests for identical - objects to improve performance and to maintain their internal invariants. +* Strings (instances of :class:`str` or :class:`unicode`) + compare lexicographically using the numeric equivalents (the + result of the built-in function :func:`ord`) of their characters. [#]_ + When comparing an 8-bit string and a Unicode string, the 8-bit string + is converted to Unicode. If the conversion fails, the strings + are considered unequal. + +* Instances of :class:`tuple` or :class:`list` can be compared only + within each of their types. Equality comparison across these types + results in unequality, and ordering comparison across these types + gives an arbitrary order. + + These sequences compare lexicographically using comparison of corresponding + elements, whereby reflexivity of the elements is enforced. + + In enforcing reflexivity of elements, the comparison of collections assumes + that for a collection element ``x``, ``x == x`` is always true. Based on + that assumption, element identity is compared first, and element comparison + is performed only for distinct elements. This approach yields the same + result as a strict element comparison would, if the compared elements are + reflexive. For non-reflexive elements, the result is different than for + strict element comparison. Lexicographical comparison between built-in collections works as follows: @@ -1456,9 +1178,9 @@ built-in types. equal (for example, ``[1,2] == (1,2)`` is false because the type is not the same). - - Collections that support order comparison are ordered the same as their - first unequal elements (for example, ``[1,2,x] <= [1,2,y]`` has the same - value as ``x <= y``). If a corresponding element does not exist, the + - Collections are ordered the same as their + first unequal elements (for example, ``cmp([1,2,x], [1,2,y])`` returns the + same as ``cmp(x,y)``). If a corresponding element does not exist, the shorter collection is ordered first (for example, ``[1,2] < [1,2,3]`` is true). @@ -1466,23 +1188,13 @@ built-in types. equal `(key, value)` pairs. Equality comparison of the keys and values enforces reflexivity. - Order comparisons (``<``, ``>``, ``<=``, and ``>=``) raise :exc:`TypeError`. - -* Sets (instances of :class:`set` or :class:`frozenset`) can be compared within - and across their types. + Outcomes other than equality are resolved + consistently, but are not otherwise defined. [#]_ - They define order - comparison operators to mean subset and superset tests. Those relations do - not define total orderings (for example, the two sets ``{1,2}`` and ``{2,3}`` - are not equal, nor subsets of one another, nor supersets of one - another). Accordingly, sets are not appropriate arguments for functions - which depend on total ordering (for example, :func:`min`, :func:`max`, and - :func:`sorted` produce undefined results given a list of sets as inputs). - - Comparison of sets enforces reflexivity of its elements. - -* Most other built-in types have no comparison methods implemented, so they - inherit the default comparison behavior. +* Most other objects of built-in types compare unequal unless they are the same + object; the choice whether one object is considered smaller or larger than + another one is made arbitrarily but consistently within one execution of a + program. User-defined classes that customize their comparison behavior should follow some consistency rules, if possible: @@ -1527,8 +1239,7 @@ some consistency rules, if possible: Objects that are equal should either have the same hash value, or be marked as unhashable. -Python does not enforce these consistency rules. In fact, the not-a-number -values are an example for not following these rules. +Python does not enforce these consistency rules. .. _in: @@ -1541,7 +1252,7 @@ Membership test operations The operators :keyword:`in` and :keyword:`not in` test for membership. ``x in s`` evaluates to ``True`` if *x* is a member of *s*, and ``False`` otherwise. ``x not in s`` returns the negation of ``x in s``. All built-in sequences and -set types support this as well as dictionary, for which :keyword:`!in` tests +set types support this as well as dictionary, for which :keyword:`in` tests whether the dictionary has a given key. For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression ``x in y`` is equivalent to ``any(x is e or x == e for e in y)``. @@ -1556,15 +1267,14 @@ y`` returns ``True`` if ``y.__contains__(x)`` returns a true value, and ``False`` otherwise. For user-defined classes which do not define :meth:`__contains__` but do define -:meth:`__iter__`, ``x in y`` is ``True`` if some value ``z``, for which the -expression ``x is z or x == z`` is true, is produced while iterating over ``y``. -If an exception is raised during the iteration, it is as if :keyword:`in` raised -that exception. +:meth:`__iter__`, ``x in y`` is ``True`` if some value ``z`` with ``x == z`` is +produced while iterating over ``y``. If an exception is raised during the +iteration, it is as if :keyword:`in` raised that exception. Lastly, the old-style iteration protocol is tried: if a class defines :meth:`__getitem__`, ``x in y`` is ``True`` if and only if there is a non-negative -integer index *i* such that ``x is y[i] or x == y[i]``, and no lower integer index -raises the :exc:`IndexError` exception. (If any other exception is raised, it is as +integer index *i* such that ``x == y[i]``, and all lower integer indices do not +raise :exc:`IndexError` exception. (If any other exception is raised, it is as if :keyword:`in` raised that exception). .. index:: @@ -1573,7 +1283,7 @@ if :keyword:`in` raised that exception). pair: membership; test object: sequence -The operator :keyword:`not in` is defined to have the inverse truth value of +The operator :keyword:`not in` is defined to have the inverse true value of :keyword:`in`. .. index:: @@ -1588,10 +1298,9 @@ The operator :keyword:`not in` is defined to have the inverse truth value of Identity comparisons -------------------- -The operators :keyword:`is` and :keyword:`is not` test for an object's identity: ``x -is y`` is true if and only if *x* and *y* are the same object. An Object's identity -is determined using the :meth:`id` function. ``x is not y`` yields the inverse -truth value. [#]_ +The operators :keyword:`is` and :keyword:`is not` test for object identity: ``x +is y`` is true if and only if *x* and *y* are the same object. ``x is not y`` +yields the inverse truth value. [#]_ .. _booleans: @@ -1615,8 +1324,8 @@ In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: ``False``, ``None``, numeric zero of all types, and empty strings and containers (including strings, tuples, lists, dictionaries, sets and frozensets). All -other values are interpreted as true. User-defined objects can customize their -truth value by providing a :meth:`__bool__` method. +other values are interpreted as true. (See the :meth:`~object.__nonzero__` +special method for a way to change this.) .. index:: operator: not @@ -1633,36 +1342,33 @@ returned; otherwise, *y* is evaluated and the resulting value is returned. The expression ``x or y`` first evaluates *x*; if *x* is true, its value is returned; otherwise, *y* is evaluated and the resulting value is returned. -Note that neither :keyword:`and` nor :keyword:`or` restrict the value and type +(Note that neither :keyword:`and` nor :keyword:`or` restrict the value and type they return to ``False`` and ``True``, but rather return the last evaluated -argument. This is sometimes useful, e.g., if ``s`` is a string that should be +argument. This is sometimes useful, e.g., if ``s`` is a string that should be replaced by a default value if it is empty, the expression ``s or 'foo'`` yields -the desired value. Because :keyword:`not` has to create a new value, it -returns a boolean value regardless of the type of its argument -(for example, ``not 'foo'`` produces ``False`` rather than ``''``.) - +the desired value. Because :keyword:`not` has to invent a value anyway, it does +not bother to return a value of the same type as its argument, so e.g., ``not +'foo'`` yields ``False``, not ``''``.) -.. _if_expr: -Conditional expressions +Conditional Expressions ======================= +.. versionadded:: 2.5 + .. index:: pair: conditional; expression pair: ternary; operator - single: if; conditional expression - single: else; conditional expression .. productionlist:: conditional_expression: `or_test` ["if" `or_test` "else" `expression`] expression: `conditional_expression` | `lambda_expr` - expression_nocond: `or_test` | `lambda_expr_nocond` Conditional expressions (sometimes called a "ternary operator") have the lowest priority of all Python operations. -The expression ``x if C else y`` first evaluates the condition, *C* rather than *x*. -If *C* is true, *x* is evaluated and its value is returned; otherwise, *y* is +The expression ``x if C else y`` first evaluates the condition, *C* (*not* *x*); +if *C* is true, *x* is evaluated and its value is returned; otherwise, *y* is evaluated and its value is returned. See :pep:`308` for more details about conditional expressions. @@ -1676,26 +1382,22 @@ Lambdas .. index:: pair: lambda; expression - pair: lambda; form pair: anonymous; function - single: : (colon); lambda expression .. productionlist:: - lambda_expr: "lambda" [`parameter_list`] ":" `expression` - lambda_expr_nocond: "lambda" [`parameter_list`] ":" `expression_nocond` - -Lambda expressions (sometimes called lambda forms) are used to create anonymous -functions. The expression ``lambda parameters: expression`` yields a function -object. The unnamed object behaves like a function object defined with: + lambda_expr: "lambda" [`parameter_list`]: `expression` + old_lambda_expr: "lambda" [`parameter_list`]: `old_expression` -.. code-block:: none +Lambda expressions (sometimes called lambda forms) have the same syntactic position as +expressions. They are a shorthand to create anonymous functions; the expression +``lambda parameters: expression`` yields a function object. The unnamed object +behaves like a function object defined with :: def <lambda>(parameters): return expression See section :ref:`function` for the syntax of parameter lists. Note that -functions created with lambda expressions cannot contain statements or -annotations. +functions created with lambda expressions cannot contain statements. .. _exprlists: @@ -1703,35 +1405,17 @@ annotations. Expression lists ================ -.. index:: - pair: expression; list - single: , (comma); expression list +.. index:: pair: expression; list .. productionlist:: - expression_list: `expression` ("," `expression`)* [","] - starred_list: `starred_item` ("," `starred_item`)* [","] - starred_expression: `expression` | (`starred_item` ",")* [`starred_item`] - starred_item: `expression` | "*" `or_expr` + expression_list: `expression` ( "," `expression` )* [","] .. index:: object: tuple -Except when part of a list or set display, an expression list -containing at least one comma yields a tuple. The length of +An expression list containing at least one comma yields a tuple. The length of the tuple is the number of expressions in the list. The expressions are evaluated from left to right. -.. index:: - pair: iterable; unpacking - single: * (asterisk); in expression lists - -An asterisk ``*`` denotes :dfn:`iterable unpacking`. Its operand must be -an :term:`iterable`. The iterable is expanded into a sequence of items, -which are included in the new tuple, list, or set, at the site of -the unpacking. - -.. versionadded:: 3.5 - Iterable unpacking in expression lists, originally proposed by :pep:`448`. - .. index:: pair: trailing; comma The trailing comma is required only to create a single tuple (a.k.a. a @@ -1748,8 +1432,8 @@ Evaluation order .. index:: pair: evaluation; order -Python evaluates expressions from left to right. Notice that while evaluating -an assignment, the right-hand side is evaluated before the left-hand side. +Python evaluates expressions from left to right. Notice that while evaluating an +assignment, the right-hand side is evaluated before the left-hand side. In the following lines, expressions will be evaluated in the arithmetic order of their suffixes:: @@ -1767,28 +1451,22 @@ their suffixes:: Operator precedence =================== -.. index:: - pair: operator; precedence +.. index:: pair: operator; precedence -The following table summarizes the operator precedence in Python, from lowest -precedence (least binding) to highest precedence (most binding). Operators in +The following table summarizes the operator precedences in Python, from lowest +precedence (least binding) to highest precedence (most binding). Operators in the same box have the same precedence. Unless the syntax is explicitly given, operators are binary. Operators in the same box group left to right (except for -exponentiation, which groups from right to left). - -Note that comparisons, membership tests, and identity tests, all have the same -precedence and have a left-to-right chaining feature as described in the -:ref:`comparisons` section. - +comparisons, including tests, which all have the same precedence and chain from +left to right --- see section :ref:`comparisons` --- and exponentiation, which +groups from right to left). +-----------------------------------------------+-------------------------------------+ | Operator | Description | +===============================================+=====================================+ -| ``:=`` | Assignment expression | -+-----------------------------------------------+-------------------------------------+ | :keyword:`lambda` | Lambda expression | +-----------------------------------------------+-------------------------------------+ -| :keyword:`if <if_expr>` -- :keyword:`!else` | Conditional expression | +| :keyword:`if` -- :keyword:`else` | Conditional expression | +-----------------------------------------------+-------------------------------------+ | :keyword:`or` | Boolean OR | +-----------------------------------------------+-------------------------------------+ @@ -1798,7 +1476,7 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | :keyword:`in`, :keyword:`not in`, | Comparisons, including membership | | :keyword:`is`, :keyword:`is not`, ``<``, | tests and identity tests | -| ``<=``, ``>``, ``>=``, ``!=``, ``==`` | | +| ``<=``, ``>``, ``>=``, ``<>``, ``!=``, ``==`` | | +-----------------------------------------------+-------------------------------------+ | ``|`` | Bitwise OR | +-----------------------------------------------+-------------------------------------+ @@ -1810,29 +1488,28 @@ precedence and have a left-to-right chaining feature as described in the +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ -| ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | -| | multiplication, division, floor | -| | division, remainder [#]_ | +| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | +| | [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ | ``**`` | Exponentiation [#]_ | +-----------------------------------------------+-------------------------------------+ -| :keyword:`await` ``x`` | Await expression | -+-----------------------------------------------+-------------------------------------+ | ``x[index]``, ``x[index:index]``, | Subscription, slicing, | | ``x(arguments...)``, ``x.attribute`` | call, attribute reference | +-----------------------------------------------+-------------------------------------+ -| ``(expressions...)``, | Binding or parenthesized | -| | expression, | +| ``(expressions...)``, | Binding or tuple display, | | ``[expressions...]``, | list display, | | ``{key: value...}``, | dictionary display, | -| ``{expressions...}`` | set display | +| ```expressions...``` | string conversion | +-----------------------------------------------+-------------------------------------+ - .. rubric:: Footnotes +.. [#] In Python 2.3 and later releases, a list comprehension "leaks" the control + variables of each ``for`` it contains into the containing scope. However, this + behavior is deprecated, and relying on it will not work in Python 3. + .. [#] While ``abs(x%y) < abs(y)`` is true mathematically, for floats it may not be true numerically due to roundoff. For example, and assuming a platform on which a Python float is an IEEE 754 double-precision number, in order that ``-1e-100 % @@ -1843,7 +1520,7 @@ precedence and have a left-to-right chaining feature as described in the is more appropriate depends on the application. .. [#] If x is very close to an exact integer multiple of y, it's possible for - ``x//y`` to be one larger than ``(x-x%y)//y`` due to rounding. In such + ``floor(x/y)`` to be one larger than ``(x-x%y)/y`` due to rounding. In such cases, Python returns the latter result, in order to preserve that ``divmod(x,y)[0] * y + x % y`` be very close to ``x``. @@ -1858,14 +1535,20 @@ precedence and have a left-to-right chaining feature as described in the LETTER C), followed by a :dfn:`combining character` at code position U+0327 (COMBINING CEDILLA). - The comparison operators on strings compare at the level of Unicode code + The comparison operators on unicode strings compare at the level of Unicode code points. This may be counter-intuitive to humans. For example, - ``"\u00C7" == "\u0043\u0327"`` is ``False``, even though both strings + ``u"\u00C7" == u"\u0043\u0327"`` is ``False``, even though both strings represent the same abstract character "LATIN CAPITAL LETTER C WITH CEDILLA". To compare strings at the level of abstract characters (that is, in a way intuitive to humans), use :func:`unicodedata.normalize`. +.. [#] Earlier versions of Python used lexicographic comparison of the sorted (key, + value) lists, but this was very expensive for the common case of comparing for + equality. An even earlier version of Python compared dictionaries by identity + only, but this caused surprises because people expected to be able to test a + dictionary for emptiness by comparing it to ``{}``. + .. [#] Due to automatic garbage-collection, free lists, and the dynamic nature of descriptors, you may notice seemingly unusual behaviour in certain uses of the :keyword:`is` operator, like those involving comparisons between instance diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst deleted file mode 100644 index c958263..0000000 --- a/Doc/reference/import.rst +++ /dev/null @@ -1,1072 +0,0 @@ - -.. _importsystem: - -***************** -The import system -***************** - -.. index:: single: import machinery - -Python code in one :term:`module` gains access to the code in another module -by the process of :term:`importing` it. The :keyword:`import` statement is -the most common way of invoking the import machinery, but it is not the only -way. Functions such as :func:`importlib.import_module` and built-in -:func:`__import__` can also be used to invoke the import machinery. - -The :keyword:`import` statement combines two operations; it searches for the -named module, then it binds the results of that search to a name in the local -scope. The search operation of the :keyword:`!import` statement is defined as -a call to the :func:`__import__` function, with the appropriate arguments. -The return value of :func:`__import__` is used to perform the name -binding operation of the :keyword:`!import` statement. See the -:keyword:`!import` statement for the exact details of that name binding -operation. - -A direct call to :func:`__import__` performs only the module search and, if -found, the module creation operation. While certain side-effects may occur, -such as the importing of parent packages, and the updating of various caches -(including :data:`sys.modules`), only the :keyword:`import` statement performs -a name binding operation. - -When an :keyword:`import` statement is executed, the standard builtin -:func:`__import__` function is called. Other mechanisms for invoking the -import system (such as :func:`importlib.import_module`) may choose to bypass -:func:`__import__` and use their own solutions to implement import semantics. - -When a module is first imported, Python searches for the module and if found, -it creates a module object [#fnmo]_, initializing it. If the named module -cannot be found, a :exc:`ModuleNotFoundError` is raised. Python implements various -strategies to search for the named module when the import machinery is -invoked. These strategies can be modified and extended by using various hooks -described in the sections below. - -.. versionchanged:: 3.3 - The import system has been updated to fully implement the second phase - of :pep:`302`. There is no longer any implicit import machinery - the full - import system is exposed through :data:`sys.meta_path`. In addition, - native namespace package support has been implemented (see :pep:`420`). - - -:mod:`importlib` -================ - -The :mod:`importlib` module provides a rich API for interacting with the -import system. For example :func:`importlib.import_module` provides a -recommended, simpler API than built-in :func:`__import__` for invoking the -import machinery. Refer to the :mod:`importlib` library documentation for -additional detail. - - - -Packages -======== - -.. index:: - single: package - -Python has only one type of module object, and all modules are of this type, -regardless of whether the module is implemented in Python, C, or something -else. To help organize modules and provide a naming hierarchy, Python has a -concept of :term:`packages <package>`. - -You can think of packages as the directories on a file system and modules as -files within directories, but don't take this analogy too literally since -packages and modules need not originate from the file system. For the -purposes of this documentation, we'll use this convenient analogy of -directories and files. Like file system directories, packages are organized -hierarchically, and packages may themselves contain subpackages, as well as -regular modules. - -It's important to keep in mind that all packages are modules, but not all -modules are packages. Or put another way, packages are just a special kind of -module. Specifically, any module that contains a ``__path__`` attribute is -considered a package. - -All modules have a name. Subpackage names are separated from their parent -package name by a dot, akin to Python's standard attribute access syntax. Thus -you might have a module called :mod:`sys` and a package called :mod:`email`, -which in turn has a subpackage called :mod:`email.mime` and a module within -that subpackage called :mod:`email.mime.text`. - - -Regular packages ----------------- - -.. index:: - pair: package; regular - -Python defines two types of packages, :term:`regular packages <regular -package>` and :term:`namespace packages <namespace package>`. Regular -packages are traditional packages as they existed in Python 3.2 and earlier. -A regular package is typically implemented as a directory containing an -``__init__.py`` file. When a regular package is imported, this -``__init__.py`` file is implicitly executed, and the objects it defines are -bound to names in the package's namespace. The ``__init__.py`` file can -contain the same Python code that any other module can contain, and Python -will add some additional attributes to the module when it is imported. - -For example, the following file system layout defines a top level ``parent`` -package with three subpackages:: - - parent/ - __init__.py - one/ - __init__.py - two/ - __init__.py - three/ - __init__.py - -Importing ``parent.one`` will implicitly execute ``parent/__init__.py`` and -``parent/one/__init__.py``. Subsequent imports of ``parent.two`` or -``parent.three`` will execute ``parent/two/__init__.py`` and -``parent/three/__init__.py`` respectively. - - -Namespace packages ------------------- - -.. index:: - pair: package; namespace - pair: package; portion - -A namespace package is a composite of various :term:`portions <portion>`, -where each portion contributes a subpackage to the parent package. Portions -may reside in different locations on the file system. Portions may also be -found in zip files, on the network, or anywhere else that Python searches -during import. Namespace packages may or may not correspond directly to -objects on the file system; they may be virtual modules that have no concrete -representation. - -Namespace packages do not use an ordinary list for their ``__path__`` -attribute. They instead use a custom iterable type which will automatically -perform a new search for package portions on the next import attempt within -that package if the path of their parent package (or :data:`sys.path` for a -top level package) changes. - -With namespace packages, there is no ``parent/__init__.py`` file. In fact, -there may be multiple ``parent`` directories found during import search, where -each one is provided by a different portion. Thus ``parent/one`` may not be -physically located next to ``parent/two``. In this case, Python will create a -namespace package for the top-level ``parent`` package whenever it or one of -its subpackages is imported. - -See also :pep:`420` for the namespace package specification. - - -Searching -========= - -To begin the search, Python needs the :term:`fully qualified <qualified name>` -name of the module (or package, but for the purposes of this discussion, the -difference is immaterial) being imported. This name may come from various -arguments to the :keyword:`import` statement, or from the parameters to the -:func:`importlib.import_module` or :func:`__import__` functions. - -This name will be used in various phases of the import search, and it may be -the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python -first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``. -If any of the intermediate imports fail, a :exc:`ModuleNotFoundError` is raised. - - -The module cache ----------------- - -.. index:: - single: sys.modules - -The first place checked during import search is :data:`sys.modules`. This -mapping serves as a cache of all modules that have been previously imported, -including the intermediate paths. So if ``foo.bar.baz`` was previously -imported, :data:`sys.modules` will contain entries for ``foo``, ``foo.bar``, -and ``foo.bar.baz``. Each key will have as its value the corresponding module -object. - -During import, the module name is looked up in :data:`sys.modules` and if -present, the associated value is the module satisfying the import, and the -process completes. However, if the value is ``None``, then a -:exc:`ModuleNotFoundError` is raised. If the module name is missing, Python will -continue searching for the module. - -:data:`sys.modules` is writable. Deleting a key may not destroy the -associated module (as other modules may hold references to it), -but it will invalidate the cache entry for the named module, causing -Python to search anew for the named module upon its next -import. The key can also be assigned to ``None``, forcing the next import -of the module to result in a :exc:`ModuleNotFoundError`. - -Beware though, as if you keep a reference to the module object, -invalidate its cache entry in :data:`sys.modules`, and then re-import the -named module, the two module objects will *not* be the same. By contrast, -:func:`importlib.reload` will reuse the *same* module object, and simply -reinitialise the module contents by rerunning the module's code. - - -Finders and loaders -------------------- - -.. index:: - single: finder - single: loader - single: module spec - -If the named module is not found in :data:`sys.modules`, then Python's import -protocol is invoked to find and load the module. This protocol consists of -two conceptual objects, :term:`finders <finder>` and :term:`loaders <loader>`. -A finder's job is to determine whether it can find the named module using -whatever strategy it knows about. Objects that implement both of these -interfaces are referred to as :term:`importers <importer>` - they return -themselves when they find that they can load the requested module. - -Python includes a number of default finders and importers. The first one -knows how to locate built-in modules, and the second knows how to locate -frozen modules. A third default finder searches an :term:`import path` -for modules. The :term:`import path` is a list of locations that may -name file system paths or zip files. It can also be extended to search -for any locatable resource, such as those identified by URLs. - -The import machinery is extensible, so new finders can be added to extend the -range and scope of module searching. - -Finders do not actually load modules. If they can find the named module, they -return a :dfn:`module spec`, an encapsulation of the module's import-related -information, which the import machinery then uses when loading the module. - -The following sections describe the protocol for finders and loaders in more -detail, including how you can create and register new ones to extend the -import machinery. - -.. versionchanged:: 3.4 - In previous versions of Python, finders returned :term:`loaders <loader>` - directly, whereas now they return module specs which *contain* loaders. - Loaders are still used during import but have fewer responsibilities. - -Import hooks ------------- - -.. index:: - single: import hooks - single: meta hooks - single: path hooks - pair: hooks; import - pair: hooks; meta - pair: hooks; path - -The import machinery is designed to be extensible; the primary mechanism for -this are the *import hooks*. There are two types of import hooks: *meta -hooks* and *import path hooks*. - -Meta hooks are called at the start of import processing, before any other -import processing has occurred, other than :data:`sys.modules` cache look up. -This allows meta hooks to override :data:`sys.path` processing, frozen -modules, or even built-in modules. Meta hooks are registered by adding new -finder objects to :data:`sys.meta_path`, as described below. - -Import path hooks are called as part of :data:`sys.path` (or -``package.__path__``) processing, at the point where their associated path -item is encountered. Import path hooks are registered by adding new callables -to :data:`sys.path_hooks` as described below. - - -The meta path -------------- - -.. index:: - single: sys.meta_path - pair: finder; find_spec - -When the named module is not found in :data:`sys.modules`, Python next -searches :data:`sys.meta_path`, which contains a list of meta path finder -objects. These finders are queried in order to see if they know how to handle -the named module. Meta path finders must implement a method called -:meth:`~importlib.abc.MetaPathFinder.find_spec()` which takes three arguments: -a name, an import path, and (optionally) a target module. The meta path -finder can use any strategy it wants to determine whether it can handle -the named module or not. - -If the meta path finder knows how to handle the named module, it returns a -spec object. If it cannot handle the named module, it returns ``None``. If -:data:`sys.meta_path` processing reaches the end of its list without returning -a spec, then a :exc:`ModuleNotFoundError` is raised. Any other exceptions -raised are simply propagated up, aborting the import process. - -The :meth:`~importlib.abc.MetaPathFinder.find_spec()` method of meta path -finders is called with two or three arguments. The first is the fully -qualified name of the module being imported, for example ``foo.bar.baz``. -The second argument is the path entries to use for the module search. For -top-level modules, the second argument is ``None``, but for submodules or -subpackages, the second argument is the value of the parent package's -``__path__`` attribute. If the appropriate ``__path__`` attribute cannot -be accessed, a :exc:`ModuleNotFoundError` is raised. The third argument -is an existing module object that will be the target of loading later. -The import system passes in a target module only during reload. - -The meta path may be traversed multiple times for a single import request. -For example, assuming none of the modules involved has already been cached, -importing ``foo.bar.baz`` will first perform a top level import, calling -``mpf.find_spec("foo", None, None)`` on each meta path finder (``mpf``). After -``foo`` has been imported, ``foo.bar`` will be imported by traversing the -meta path a second time, calling -``mpf.find_spec("foo.bar", foo.__path__, None)``. Once ``foo.bar`` has been -imported, the final traversal will call -``mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)``. - -Some meta path finders only support top level imports. These importers will -always return ``None`` when anything other than ``None`` is passed as the -second argument. - -Python's default :data:`sys.meta_path` has three meta path finders, one that -knows how to import built-in modules, one that knows how to import frozen -modules, and one that knows how to import modules from an :term:`import path` -(i.e. the :term:`path based finder`). - -.. versionchanged:: 3.4 - The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path - finders replaced :meth:`~importlib.abc.MetaPathFinder.find_module`, which - is now deprecated. While it will continue to work without change, the - import machinery will try it only if the finder does not implement - ``find_spec()``. - - -Loading -======= - -If and when a module spec is found, the import machinery will use it (and -the loader it contains) when loading the module. Here is an approximation -of what happens during the loading portion of import:: - - module = None - if spec.loader is not None and hasattr(spec.loader, 'create_module'): - # It is assumed 'exec_module' will also be defined on the loader. - module = spec.loader.create_module(spec) - if module is None: - module = ModuleType(spec.name) - # The import-related module attributes get set here: - _init_module_attrs(spec, module) - - if spec.loader is None: - # unsupported - raise ImportError - if spec.origin is None and spec.submodule_search_locations is not None: - # namespace package - sys.modules[spec.name] = module - elif not hasattr(spec.loader, 'exec_module'): - module = spec.loader.load_module(spec.name) - # Set __loader__ and __package__ if missing. - else: - sys.modules[spec.name] = module - try: - spec.loader.exec_module(module) - except BaseException: - try: - del sys.modules[spec.name] - except KeyError: - pass - raise - return sys.modules[spec.name] - -Note the following details: - - * If there is an existing module object with the given name in - :data:`sys.modules`, import will have already returned it. - - * The module will exist in :data:`sys.modules` before the loader - executes the module code. This is crucial because the module code may - (directly or indirectly) import itself; adding it to :data:`sys.modules` - beforehand prevents unbounded recursion in the worst case and multiple - loading in the best. - - * If loading fails, the failing module -- and only the failing module -- - gets removed from :data:`sys.modules`. Any module already in the - :data:`sys.modules` cache, and any module that was successfully loaded - as a side-effect, must remain in the cache. This contrasts with - reloading where even the failing module is left in :data:`sys.modules`. - - * After the module is created but before execution, the import machinery - sets the import-related module attributes ("_init_module_attrs" in - the pseudo-code example above), as summarized in a - :ref:`later section <import-mod-attrs>`. - - * Module execution is the key moment of loading in which the module's - namespace gets populated. Execution is entirely delegated to the - loader, which gets to decide what gets populated and how. - - * The module created during loading and passed to exec_module() may - not be the one returned at the end of import [#fnlo]_. - -.. versionchanged:: 3.4 - The import system has taken over the boilerplate responsibilities of - loaders. These were previously performed by the - :meth:`importlib.abc.Loader.load_module` method. - -Loaders -------- - -Module loaders provide the critical function of loading: module execution. -The import machinery calls the :meth:`importlib.abc.Loader.exec_module` -method with a single argument, the module object to execute. Any value -returned from :meth:`~importlib.abc.Loader.exec_module` is ignored. - -Loaders must satisfy the following requirements: - - * If the module is a Python module (as opposed to a built-in module or a - dynamically loaded extension), the loader should execute the module's code - in the module's global name space (``module.__dict__``). - - * If the loader cannot execute the module, it should raise an - :exc:`ImportError`, although any other exception raised during - :meth:`~importlib.abc.Loader.exec_module` will be propagated. - -In many cases, the finder and loader can be the same object; in such cases the -:meth:`~importlib.abc.MetaPathFinder.find_spec` method would just return a -spec with the loader set to ``self``. - -Module loaders may opt in to creating the module object during loading -by implementing a :meth:`~importlib.abc.Loader.create_module` method. -It takes one argument, the module spec, and returns the new module object -to use during loading. ``create_module()`` does not need to set any attributes -on the module object. If the method returns ``None``, the -import machinery will create the new module itself. - -.. versionadded:: 3.4 - The :meth:`~importlib.abc.Loader.create_module` method of loaders. - -.. versionchanged:: 3.4 - The :meth:`~importlib.abc.Loader.load_module` method was replaced by - :meth:`~importlib.abc.Loader.exec_module` and the import - machinery assumed all the boilerplate responsibilities of loading. - - For compatibility with existing loaders, the import machinery will use - the ``load_module()`` method of loaders if it exists and the loader does - not also implement ``exec_module()``. However, ``load_module()`` has been - deprecated and loaders should implement ``exec_module()`` instead. - - The ``load_module()`` method must implement all the boilerplate loading - functionality described above in addition to executing the module. All - the same constraints apply, with some additional clarification: - - * If there is an existing module object with the given name in - :data:`sys.modules`, the loader must use that existing module. - (Otherwise, :func:`importlib.reload` will not work correctly.) If the - named module does not exist in :data:`sys.modules`, the loader - must create a new module object and add it to :data:`sys.modules`. - - * The module *must* exist in :data:`sys.modules` before the loader - executes the module code, to prevent unbounded recursion or multiple - loading. - - * If loading fails, the loader must remove any modules it has inserted - into :data:`sys.modules`, but it must remove **only** the failing - module(s), and only if the loader itself has loaded the module(s) - explicitly. - -.. versionchanged:: 3.5 - A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but - ``create_module()`` is not. - -.. versionchanged:: 3.6 - An :exc:`ImportError` is raised when ``exec_module()`` is defined but - ``create_module()`` is not. - -Submodules ----------- - -When a submodule is loaded using any mechanism (e.g. ``importlib`` APIs, the -``import`` or ``import-from`` statements, or built-in ``__import__()``) a -binding is placed in the parent module's namespace to the submodule object. -For example, if package ``spam`` has a submodule ``foo``, after importing -``spam.foo``, ``spam`` will have an attribute ``foo`` which is bound to the -submodule. Let's say you have the following directory structure:: - - spam/ - __init__.py - foo.py - bar.py - -and ``spam/__init__.py`` has the following lines in it:: - - from .foo import Foo - from .bar import Bar - -then executing the following puts a name binding to ``foo`` and ``bar`` in the -``spam`` module:: - - >>> import spam - >>> spam.foo - <module 'spam.foo' from '/tmp/imports/spam/foo.py'> - >>> spam.bar - <module 'spam.bar' from '/tmp/imports/spam/bar.py'> - -Given Python's familiar name binding rules this might seem surprising, but -it's actually a fundamental feature of the import system. The invariant -holding is that if you have ``sys.modules['spam']`` and -``sys.modules['spam.foo']`` (as you would after the above import), the latter -must appear as the ``foo`` attribute of the former. - -Module spec ------------ - -The import machinery uses a variety of information about each module -during import, especially before loading. Most of the information is -common to all modules. The purpose of a module's spec is to encapsulate -this import-related information on a per-module basis. - -Using a spec during import allows state to be transferred between import -system components, e.g. between the finder that creates the module spec -and the loader that executes it. Most importantly, it allows the -import machinery to perform the boilerplate operations of loading, -whereas without a module spec the loader had that responsibility. - -The module's spec is exposed as the ``__spec__`` attribute on a module object. -See :class:`~importlib.machinery.ModuleSpec` for details on the contents of -the module spec. - -.. versionadded:: 3.4 - -.. _import-mod-attrs: - -Import-related module attributes --------------------------------- - -The import machinery fills in these attributes on each module object -during loading, based on the module's spec, before the loader executes -the module. - -.. attribute:: __name__ - - The ``__name__`` attribute must be set to the fully-qualified name of - the module. This name is used to uniquely identify the module in - the import system. - -.. attribute:: __loader__ - - The ``__loader__`` attribute must be set to the loader object that - the import machinery used when loading the module. This is mostly - for introspection, but can be used for additional loader-specific - functionality, for example getting data associated with a loader. - -.. attribute:: __package__ - - The module's ``__package__`` attribute must be set. Its value must - be a string, but it can be the same value as its ``__name__``. When - the module is a package, its ``__package__`` value should be set to - its ``__name__``. When the module is not a package, ``__package__`` - should be set to the empty string for top-level modules, or for - submodules, to the parent package's name. See :pep:`366` for further - details. - - This attribute is used instead of ``__name__`` to calculate explicit - relative imports for main modules, as defined in :pep:`366`. It is - expected to have the same value as ``__spec__.parent``. - - .. versionchanged:: 3.6 - The value of ``__package__`` is expected to be the same as - ``__spec__.parent``. - -.. attribute:: __spec__ - - The ``__spec__`` attribute must be set to the module spec that was - used when importing the module. Setting ``__spec__`` - appropriately applies equally to :ref:`modules initialized during - interpreter startup <programs>`. The one exception is ``__main__``, - where ``__spec__`` is :ref:`set to None in some cases <main_spec>`. - - When ``__package__`` is not defined, ``__spec__.parent`` is used as - a fallback. - - .. versionadded:: 3.4 - - .. versionchanged:: 3.6 - ``__spec__.parent`` is used as a fallback when ``__package__`` is - not defined. - -.. attribute:: __path__ - - If the module is a package (either regular or namespace), the module - object's ``__path__`` attribute must be set. The value must be - iterable, but may be empty if ``__path__`` has no further significance. - If ``__path__`` is not empty, it must produce strings when iterated - over. More details on the semantics of ``__path__`` are given - :ref:`below <package-path-rules>`. - - Non-package modules should not have a ``__path__`` attribute. - -.. attribute:: __file__ -.. attribute:: __cached__ - - ``__file__`` is optional. If set, this attribute's value must be a - string. The import system may opt to leave ``__file__`` unset if it - has no semantic meaning (e.g. a module loaded from a database). - - If ``__file__`` is set, it may also be appropriate to set the - ``__cached__`` attribute which is the path to any compiled version of - the code (e.g. byte-compiled file). The file does not need to exist - to set this attribute; the path can simply point to where the - compiled file would exist (see :pep:`3147`). - - It is also appropriate to set ``__cached__`` when ``__file__`` is not - set. However, that scenario is quite atypical. Ultimately, the - loader is what makes use of ``__file__`` and/or ``__cached__``. So - if a loader can load from a cached module but otherwise does not load - from a file, that atypical scenario may be appropriate. - -.. _package-path-rules: - -module.__path__ ---------------- - -By definition, if a module has a ``__path__`` attribute, it is a package. - -A package's ``__path__`` attribute is used during imports of its subpackages. -Within the import machinery, it functions much the same as :data:`sys.path`, -i.e. providing a list of locations to search for modules during import. -However, ``__path__`` is typically much more constrained than -:data:`sys.path`. - -``__path__`` must be an iterable of strings, but it may be empty. -The same rules used for :data:`sys.path` also apply to a package's -``__path__``, and :data:`sys.path_hooks` (described below) are -consulted when traversing a package's ``__path__``. - -A package's ``__init__.py`` file may set or alter the package's ``__path__`` -attribute, and this was typically the way namespace packages were implemented -prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no -longer need to supply ``__init__.py`` files containing only ``__path__`` -manipulation code; the import machinery automatically sets ``__path__`` -correctly for the namespace package. - -Module reprs ------------- - -By default, all modules have a usable repr, however depending on the -attributes set above, and in the module's spec, you can more explicitly -control the repr of module objects. - -If the module has a spec (``__spec__``), the import machinery will try -to generate a repr from it. If that fails or there is no spec, the import -system will craft a default repr using whatever information is available -on the module. It will try to use the ``module.__name__``, -``module.__file__``, and ``module.__loader__`` as input into the repr, -with defaults for whatever information is missing. - -Here are the exact rules used: - - * If the module has a ``__spec__`` attribute, the information in the spec - is used to generate the repr. The "name", "loader", "origin", and - "has_location" attributes are consulted. - - * If the module has a ``__file__`` attribute, this is used as part of the - module's repr. - - * If the module has no ``__file__`` but does have a ``__loader__`` that is not - ``None``, then the loader's repr is used as part of the module's repr. - - * Otherwise, just use the module's ``__name__`` in the repr. - -.. versionchanged:: 3.4 - Use of :meth:`loader.module_repr() <importlib.abc.Loader.module_repr>` - has been deprecated and the module spec is now used by the import - machinery to generate a module repr. - - For backward compatibility with Python 3.3, the module repr will be - generated by calling the loader's - :meth:`~importlib.abc.Loader.module_repr` method, if defined, before - trying either approach described above. However, the method is deprecated. - -.. _pyc-invalidation: - -Cached bytecode invalidation ----------------------------- - -Before Python loads cached bytecode from ``.pyc`` file, it checks whether the -cache is up-to-date with the source ``.py`` file. By default, Python does this -by storing the source's last-modified timestamp and size in the cache file when -writing it. At runtime, the import system then validates the cache file by -checking the stored metadata in the cache file against the source's -metadata. - -Python also supports "hash-based" cache files, which store a hash of the source -file's contents rather than its metadata. There are two variants of hash-based -``.pyc`` files: checked and unchecked. For checked hash-based ``.pyc`` files, -Python validates the cache file by hashing the source file and comparing the -resulting hash with the hash in the cache file. If a checked hash-based cache -file is found to be invalid, Python regenerates it and writes a new checked -hash-based cache file. For unchecked hash-based ``.pyc`` files, Python simply -assumes the cache file is valid if it exists. Hash-based ``.pyc`` files -validation behavior may be overridden with the :option:`--check-hash-based-pycs` -flag. - -.. versionchanged:: 3.7 - Added hash-based ``.pyc`` files. Previously, Python only supported - timestamp-based invalidation of bytecode caches. - - -The Path Based Finder -===================== - -.. index:: - single: path based finder - -As mentioned previously, Python comes with several default meta path finders. -One of these, called the :term:`path based finder` -(:class:`~importlib.machinery.PathFinder`), searches an :term:`import path`, -which contains a list of :term:`path entries <path entry>`. Each path -entry names a location to search for modules. - -The path based finder itself doesn't know how to import anything. Instead, it -traverses the individual path entries, associating each of them with a -path entry finder that knows how to handle that particular kind of path. - -The default set of path entry finders implement all the semantics for finding -modules on the file system, handling special file types such as Python source -code (``.py`` files), Python byte code (``.pyc`` files) and -shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` -module in the standard library, the default path entry finders also handle -loading all of these file types (other than shared libraries) from zipfiles. - -Path entries need not be limited to file system locations. They can refer to -URLs, database queries, or any other location that can be specified as a -string. - -The path based finder provides additional hooks and protocols so that you -can extend and customize the types of searchable path entries. For example, -if you wanted to support path entries as network URLs, you could write a hook -that implements HTTP semantics to find modules on the web. This hook (a -callable) would return a :term:`path entry finder` supporting the protocol -described below, which was then used to get a loader for the module from the -web. - -A word of warning: this section and the previous both use the term *finder*, -distinguishing between them by using the terms :term:`meta path finder` and -:term:`path entry finder`. These two types of finders are very similar, -support similar protocols, and function in similar ways during the import -process, but it's important to keep in mind that they are subtly different. -In particular, meta path finders operate at the beginning of the import -process, as keyed off the :data:`sys.meta_path` traversal. - -By contrast, path entry finders are in a sense an implementation detail -of the path based finder, and in fact, if the path based finder were to be -removed from :data:`sys.meta_path`, none of the path entry finder semantics -would be invoked. - - -Path entry finders ------------------- - -.. index:: - single: sys.path - single: sys.path_hooks - single: sys.path_importer_cache - single: PYTHONPATH - -The :term:`path based finder` is responsible for finding and loading -Python modules and packages whose location is specified with a string -:term:`path entry`. Most path entries name locations in the file system, -but they need not be limited to this. - -As a meta path finder, the :term:`path based finder` implements the -:meth:`~importlib.abc.MetaPathFinder.find_spec` protocol previously -described, however it exposes additional hooks that can be used to -customize how modules are found and loaded from the :term:`import path`. - -Three variables are used by the :term:`path based finder`, :data:`sys.path`, -:data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` -attributes on package objects are also used. These provide additional ways -that the import machinery can be customized. - -:data:`sys.path` contains a list of strings providing search locations for -modules and packages. It is initialized from the :data:`PYTHONPATH` -environment variable and various other installation- and -implementation-specific defaults. Entries in :data:`sys.path` can name -directories on the file system, zip files, and potentially other "locations" -(see the :mod:`site` module) that should be searched for modules, such as -URLs, or database queries. Only strings and bytes should be present on -:data:`sys.path`; all other data types are ignored. The encoding of bytes -entries is determined by the individual :term:`path entry finders <path entry -finder>`. - -The :term:`path based finder` is a :term:`meta path finder`, so the import -machinery begins the :term:`import path` search by calling the path -based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method as -described previously. When the ``path`` argument to -:meth:`~importlib.machinery.PathFinder.find_spec` is given, it will be a -list of string paths to traverse - typically a package's ``__path__`` -attribute for an import within that package. If the ``path`` argument is -``None``, this indicates a top level import and :data:`sys.path` is used. - -The path based finder iterates over every entry in the search path, and -for each of these, looks for an appropriate :term:`path entry finder` -(:class:`~importlib.abc.PathEntryFinder`) for the -path entry. Because this can be an expensive operation (e.g. there may be -`stat()` call overheads for this search), the path based finder maintains -a cache mapping path entries to path entry finders. This cache is maintained -in :data:`sys.path_importer_cache` (despite the name, this cache actually -stores finder objects rather than being limited to :term:`importer` objects). -In this way, the expensive search for a particular :term:`path entry` -location's :term:`path entry finder` need only be done once. User code is -free to remove cache entries from :data:`sys.path_importer_cache` forcing -the path based finder to perform the path entry search again [#fnpic]_. - -If the path entry is not present in the cache, the path based finder iterates -over every callable in :data:`sys.path_hooks`. Each of the :term:`path entry -hooks <path entry hook>` in this list is called with a single argument, the -path entry to be searched. This callable may either return a :term:`path -entry finder` that can handle the path entry, or it may raise -:exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to -signal that the hook cannot find a :term:`path entry finder` -for that :term:`path entry`. The -exception is ignored and :term:`import path` iteration continues. The hook -should expect either a string or bytes object; the encoding of bytes objects -is up to the hook (e.g. it may be a file system encoding, UTF-8, or something -else), and if the hook cannot decode the argument, it should raise -:exc:`ImportError`. - -If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` -being returned, then the path based finder's -:meth:`~importlib.machinery.PathFinder.find_spec` method will store ``None`` -in :data:`sys.path_importer_cache` (to indicate that there is no finder for -this path entry) and return ``None``, indicating that this -:term:`meta path finder` could not find the module. - -If a :term:`path entry finder` *is* returned by one of the :term:`path entry -hook` callables on :data:`sys.path_hooks`, then the following protocol is used -to ask the finder for a module spec, which is then used when loading the -module. - -The current working directory -- denoted by an empty string -- is handled -slightly differently from other entries on :data:`sys.path`. First, if the -current working directory is found to not exist, no value is stored in -:data:`sys.path_importer_cache`. Second, the value for the current working -directory is looked up fresh for each module lookup. Third, the path used for -:data:`sys.path_importer_cache` and returned by -:meth:`importlib.machinery.PathFinder.find_spec` will be the actual current -working directory and not the empty string. - -Path entry finder protocol --------------------------- - -In order to support imports of modules and initialized packages and also to -contribute portions to namespace packages, path entry finders must implement -the :meth:`~importlib.abc.PathEntryFinder.find_spec` method. - -:meth:`~importlib.abc.PathEntryFinder.find_spec` takes two arguments: the -fully qualified name of the module being imported, and the (optional) target -module. ``find_spec()`` returns a fully populated spec for the module. -This spec will always have "loader" set (with one exception). - -To indicate to the import machinery that the spec represents a namespace -:term:`portion`. the path entry finder sets "loader" on the spec to -``None`` and "submodule_search_locations" to a list containing the -portion. - -.. versionchanged:: 3.4 - :meth:`~importlib.abc.PathEntryFinder.find_spec` replaced - :meth:`~importlib.abc.PathEntryFinder.find_loader` and - :meth:`~importlib.abc.PathEntryFinder.find_module`, both of which - are now deprecated, but will be used if ``find_spec()`` is not defined. - - Older path entry finders may implement one of these two deprecated methods - instead of ``find_spec()``. The methods are still respected for the - sake of backward compatibility. However, if ``find_spec()`` is - implemented on the path entry finder, the legacy methods are ignored. - - :meth:`~importlib.abc.PathEntryFinder.find_loader` takes one argument, the - fully qualified name of the module being imported. ``find_loader()`` - returns a 2-tuple where the first item is the loader and the second item - is a namespace :term:`portion`. When the first item (i.e. the loader) is - ``None``, this means that while the path entry finder does not have a - loader for the named module, it knows that the path entry contributes to - a namespace portion for the named module. This will almost always be the - case where Python is asked to import a namespace package that has no - physical presence on the file system. When a path entry finder returns - ``None`` for the loader, the second item of the 2-tuple return value must - be a sequence, although it can be empty. - - If ``find_loader()`` returns a non-``None`` loader value, the portion is - ignored and the loader is returned from the path based finder, terminating - the search through the path entries. - - For backwards compatibility with other implementations of the import - protocol, many path entry finders also support the same, - traditional ``find_module()`` method that meta path finders support. - However path entry finder ``find_module()`` methods are never called - with a ``path`` argument (they are expected to record the appropriate - path information from the initial call to the path hook). - - The ``find_module()`` method on path entry finders is deprecated, - as it does not allow the path entry finder to contribute portions to - namespace packages. If both ``find_loader()`` and ``find_module()`` - exist on a path entry finder, the import system will always call - ``find_loader()`` in preference to ``find_module()``. - - -Replacing the standard import system -==================================== - -The most reliable mechanism for replacing the entire import system is to -delete the default contents of :data:`sys.meta_path`, replacing them -entirely with a custom meta path hook. - -If it is acceptable to only alter the behaviour of import statements -without affecting other APIs that access the import system, then replacing -the builtin :func:`__import__` function may be sufficient. This technique -may also be employed at the module level to only alter the behaviour of -import statements within that module. - -To selectively prevent the import of some modules from a hook early on the -meta path (rather than disabling the standard import system entirely), -it is sufficient to raise :exc:`ModuleNotFoundError` directly from -:meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning -``None``. The latter indicates that the meta path search should continue, -while raising an exception terminates it immediately. - -.. _relativeimports: - -Package Relative Imports -======================== - -Relative imports use leading dots. A single leading dot indicates a relative -import, starting with the current package. Two or more leading dots indicate a -relative import to the parent(s) of the current package, one level per dot -after the first. For example, given the following package layout:: - - package/ - __init__.py - subpackage1/ - __init__.py - moduleX.py - moduleY.py - subpackage2/ - __init__.py - moduleZ.py - moduleA.py - -In either ``subpackage1/moduleX.py`` or ``subpackage1/__init__.py``, -the following are valid relative imports:: - - from .moduleY import spam - from .moduleY import spam as ham - from . import moduleY - from ..subpackage1 import moduleY - from ..subpackage2.moduleZ import eggs - from ..moduleA import foo - -Absolute imports may use either the ``import <>`` or ``from <> import <>`` -syntax, but relative imports may only use the second form; the reason -for this is that:: - - import XXX.YYY.ZZZ - -should expose ``XXX.YYY.ZZZ`` as a usable expression, but .moduleY is -not a valid expression. - - -Special considerations for __main__ -=================================== - -The :mod:`__main__` module is a special case relative to Python's import -system. As noted :ref:`elsewhere <programs>`, the ``__main__`` module -is directly initialized at interpreter startup, much like :mod:`sys` and -:mod:`builtins`. However, unlike those two, it doesn't strictly -qualify as a built-in module. This is because the manner in which -``__main__`` is initialized depends on the flags and other options with -which the interpreter is invoked. - -.. _main_spec: - -__main__.__spec__ ------------------ - -Depending on how :mod:`__main__` is initialized, ``__main__.__spec__`` -gets set appropriately or to ``None``. - -When Python is started with the :option:`-m` option, ``__spec__`` is set -to the module spec of the corresponding module or package. ``__spec__`` is -also populated when the ``__main__`` module is loaded as part of executing a -directory, zipfile or other :data:`sys.path` entry. - -In :ref:`the remaining cases <using-on-interface-options>` -``__main__.__spec__`` is set to ``None``, as the code used to populate the -:mod:`__main__` does not correspond directly with an importable module: - -- interactive prompt -- :option:`-c` option -- running from stdin -- running directly from a source or bytecode file - -Note that ``__main__.__spec__`` is always ``None`` in the last case, -*even if* the file could technically be imported directly as a module -instead. Use the :option:`-m` switch if valid module metadata is desired -in :mod:`__main__`. - -Note also that even when ``__main__`` corresponds with an importable module -and ``__main__.__spec__`` is set accordingly, they're still considered -*distinct* modules. This is due to the fact that blocks guarded by -``if __name__ == "__main__":`` checks only execute when the module is used -to populate the ``__main__`` namespace, and not during normal import. - - -Open issues -=========== - -XXX It would be really nice to have a diagram. - -XXX * (import_machinery.rst) how about a section devoted just to the -attributes of modules and packages, perhaps expanding upon or supplanting the -related entries in the data model reference page? - -XXX runpy, pkgutil, et al in the library manual should all get "See Also" -links at the top pointing to the new import system section. - -XXX Add more explanation regarding the different ways in which -``__main__`` is initialized? - -XXX Add more info on ``__main__`` quirks/pitfalls (i.e. copy from -:pep:`395`). - - -References -========== - -The import machinery has evolved considerably since Python's early days. The -original `specification for packages -<https://www.python.org/doc/essays/packages/>`_ is still available to read, -although some details have changed since the writing of that document. - -The original specification for :data:`sys.meta_path` was :pep:`302`, with -subsequent extension in :pep:`420`. - -:pep:`420` introduced :term:`namespace packages <namespace package>` for -Python 3.3. :pep:`420` also introduced the :meth:`find_loader` protocol as an -alternative to :meth:`find_module`. - -:pep:`366` describes the addition of the ``__package__`` attribute for -explicit relative imports in main modules. - -:pep:`328` introduced absolute and explicit relative imports and initially -proposed ``__name__`` for semantics :pep:`366` would eventually specify for -``__package__``. - -:pep:`338` defines executing modules as scripts. - -:pep:`451` adds the encapsulation of per-module import state in spec -objects. It also off-loads most of the boilerplate responsibilities of -loaders back onto the import machinery. These changes allow the -deprecation of several APIs in the import system and also addition of new -methods to finders and loaders. - -.. rubric:: Footnotes - -.. [#fnmo] See :class:`types.ModuleType`. - -.. [#fnlo] The importlib implementation avoids using the return value - directly. Instead, it gets the module object by looking the module name up - in :data:`sys.modules`. The indirect effect of this is that an imported - module may replace itself in :data:`sys.modules`. This is - implementation-specific behavior that is not guaranteed to work in other - Python implementations. - -.. [#fnpic] In legacy code, it is possible to find instances of - :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It - is recommended that code be changed to use ``None`` instead. See - :ref:`portingpythoncode` for more details. diff --git a/Doc/reference/index.rst b/Doc/reference/index.rst index a66673b..513e445 100644 --- a/Doc/reference/index.rst +++ b/Doc/reference/index.rst @@ -21,7 +21,6 @@ interfaces available to C/C++ programmers in detail. lexical_analysis.rst datamodel.rst executionmodel.rst - import.rst expressions.rst simple_stmts.rst compound_stmts.rst diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index bb7e390..d0959c5 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -22,12 +22,11 @@ language, maybe you could volunteer your time --- or invent a cloning machine It is dangerous to add too many implementation details to a language reference document --- the implementation may change, and other implementations of the -same language may work differently. On the other hand, CPython is the one -Python implementation in widespread use (although alternate implementations -continue to gain support), and its particular quirks are sometimes worth being -mentioned, especially where the implementation imposes additional limitations. -Therefore, you'll find short "implementation notes" sprinkled throughout the -text. +same language may work differently. On the other hand, there is currently only +one Python implementation in widespread use (although alternate implementations +exist), and its particular quirks are sometimes worth being mentioned, +especially where the implementation imposes additional limitations. Therefore, +you'll find short "implementation notes" sprinkled throughout the text. Every Python implementation comes with a number of built-in and standard modules. These are documented in :ref:`library-index`. A few built-in modules @@ -88,7 +87,11 @@ implementation you're using. Notation ======== -.. index:: BNF, grammar, syntax, notation +.. index:: + single: BNF + single: grammar + single: syntax + single: notation The descriptions of lexical analysis and syntax use a modified BNF grammar notation. This uses the following style of definition: @@ -114,7 +117,9 @@ meaningful to separate tokens. Rules are normally contained on a single line; rules with many alternatives may be formatted alternatively with each line after the first beginning with a vertical bar. -.. index:: lexical definitions, ASCII +.. index:: + single: lexical definitions + single: ASCII@ASCII In lexical definitions (as the example above), two more conventions are used: Two literal characters separated by three dots mean a choice of any single diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index c0e13b5..435dfc5 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -5,16 +5,38 @@ Lexical analysis **************** -.. index:: lexical analysis, parser, token +.. index:: + single: lexical analysis + single: parser + single: token A Python program is read by a *parser*. Input to the parser is a stream of *tokens*, generated by the *lexical analyzer*. This chapter describes how the lexical analyzer breaks a file into tokens. -Python reads program text as Unicode code points; the encoding of a source file -can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120` -for details. If the source file cannot be decoded, a :exc:`SyntaxError` is -raised. +Python uses the 7-bit ASCII character set for program text. + +.. versionadded:: 2.3 + An encoding declaration can be used to indicate that string literals and + comments use an encoding different from ASCII. + +For compatibility with older versions, Python only warns if it finds 8-bit +characters; those warnings should be corrected by either declaring an explicit +encoding, or using escape sequences if those bytes are binary data, instead of +characters. + +The run-time character set depends on the I/O devices connected to the program +but is generally a superset of ASCII. + +**Future compatibility note:** It may be tempting to assume that the character +set for 8-bit characters is ISO Latin-1 (an ASCII superset that covers most +western languages that use the Latin alphabet), but it is possible that in the +future Unicode text editors will become common. These generally use the UTF-8 +encoding, which is also an ASCII superset, but with very different use for the +characters with ordinals 128-255. While there is no consensus on this subject +yet, it is unwise to assume either Latin-1 or UTF-8, even though the current +implementation appears to favor Latin-1. This applies both to the source +character set and the run-time character set. .. _line-structure: @@ -22,17 +44,21 @@ raised. Line structure ============== -.. index:: line structure +.. index:: single: line structure A Python program is divided into a number of *logical lines*. -.. _logical-lines: +.. _logical: Logical lines ------------- -.. index:: logical line, physical line, line joining, NEWLINE token +.. index:: + single: logical line + single: physical line + single: line joining + single: NEWLINE token The end of a logical line is represented by the token NEWLINE. Statements cannot cross logical line boundaries except where NEWLINE is allowed by the @@ -41,7 +67,7 @@ constructed from one or more *physical lines* by following the explicit or implicit *line joining* rules. -.. _physical-lines: +.. _physical: Physical lines -------------- @@ -64,13 +90,14 @@ representing ASCII LF, is the line terminator). Comments -------- -.. index:: comment, hash character - single: # (hash); comment +.. index:: + single: comment + single: hash character A comment starts with a hash character (``#``) that is not part of a string literal, and ends at the end of the physical line. A comment signifies the end of the logical line unless the implicit line joining rules are invoked. Comments -are ignored by the syntax. +are ignored by the syntax; they are not tokens. .. _encodings: @@ -79,7 +106,6 @@ Encoding declarations --------------------- .. index:: source character set, encoding declarations (source file) - single: # (hash); source encoding declaration If a comment in the first or second line of the Python script matches the regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an @@ -94,16 +120,16 @@ which is recognized also by GNU Emacs, and :: # vim:fileencoding=<encoding-name> -which is recognized by Bram Moolenaar's VIM. - -If no encoding declaration is found, the default encoding is UTF-8. In -addition, if the first bytes of the file are the UTF-8 byte-order mark -(``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, -among others, by Microsoft's :program:`notepad`). +which is recognized by Bram Moolenaar's VIM. In addition, if the first bytes of +the file are the UTF-8 byte-order mark (``'\xef\xbb\xbf'``), the declared file +encoding is UTF-8 (this is supported, among others, by Microsoft's +:program:`notepad`). If an encoding is declared, the encoding name must be recognized by Python. The -encoding is used for all lexical analysis, including string literals, comments -and identifiers. +encoding is used for all lexical analysis, in particular to find the end of a +string, and to interpret the contents of Unicode literals. String literals are +converted to Unicode for syntactical analysis, then converted back to their +original encoding before interpretation starts. .. XXX there should be a list of supported encodings. @@ -113,7 +139,11 @@ and identifiers. Explicit line joining --------------------- -.. index:: physical line, line joining, line continuation, backslash character +.. index:: + single: physical line + single: line joining + single: line continuation + single: backslash character Two or more physical lines may be joined into logical lines using backslash characters (``\``), as follows: when a physical line ends in a backslash that is @@ -163,9 +193,9 @@ Blank lines A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored (i.e., no NEWLINE token is generated). During interactive input of statements, handling of a blank line may differ depending on the -implementation of the read-eval-print loop. In the standard interactive -interpreter, an entirely blank logical line (i.e. one containing not even -whitespace or a comment) terminates a multi-line statement. +implementation of the read-eval-print loop. In the standard implementation, an +entirely blank logical line (i.e. one containing not even whitespace or a +comment) terminates a multi-line statement. .. _indentation: @@ -173,24 +203,27 @@ whitespace or a comment) terminates a multi-line statement. Indentation ----------- -.. index:: indentation, leading whitespace, space, tab, grouping, statement grouping +.. index:: + single: indentation + single: whitespace + single: leading whitespace + single: space + single: tab + single: grouping + single: statement grouping Leading whitespace (spaces and tabs) at the beginning of a logical line is used to compute the indentation level of the line, which in turn is used to determine the grouping of statements. -Tabs are replaced (from left to right) by one to eight spaces such that the -total number of characters up to and including the replacement is a multiple of -eight (this is intended to be the same rule as used by Unix). The total number -of spaces preceding the first non-blank character then determines the line's -indentation. Indentation cannot be split over multiple physical lines using -backslashes; the whitespace up to the first backslash determines the +First, tabs are replaced (from left to right) by one to eight spaces such that +the total number of characters up to and including the replacement is a multiple +of eight (this is intended to be the same rule as used by Unix). The total +number of spaces preceding the first non-blank character then determines the +line's indentation. Indentation cannot be split over multiple physical lines +using backslashes; the whitespace up to the first backslash determines the indentation. -Indentation is rejected as inconsistent if a source file mixes tabs and spaces -in a way that makes the meaning dependent on the worth of a tab in spaces; a -:exc:`TabError` is raised in that case. - **Cross-platform compatibility note:** because of the nature of text editors on non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the indentation in a single source file. It should also be noted that different @@ -201,7 +234,9 @@ for the indentation calculations above. Formfeed characters occurring elsewhere in the leading whitespace have an undefined effect (for instance, they may reset the space count to zero). -.. index:: INDENT token, DEDENT token +.. index:: + single: INDENT token + single: DEDENT token The indentation levels of consecutive lines are used to generate INDENT and DEDENT tokens, using a stack, as follows. @@ -276,57 +311,22 @@ possible string that forms a legal token, when read from left to right. Identifiers and keywords ======================== -.. index:: identifier, name +.. index:: + single: identifier + single: name Identifiers (also referred to as *names*) are described by the following lexical -definitions. - -The syntax of identifiers in Python is based on the Unicode standard annex -UAX-31, with elaboration and changes as defined below; see also :pep:`3131` for -further details. +definitions: -Within the ASCII range (U+0001..U+007F), the valid characters for identifiers -are the same as in Python 2.x: the uppercase and lowercase letters ``A`` through -``Z``, the underscore ``_`` and, except for the first character, the digits -``0`` through ``9``. - -Python 3.0 introduces additional characters from outside the ASCII range (see -:pep:`3131`). For these characters, the classification uses the version of the -Unicode Character Database as included in the :mod:`unicodedata` module. +.. productionlist:: + identifier: (`letter`|"_") (`letter` | `digit` | "_")* + letter: `lowercase` | `uppercase` + lowercase: "a"..."z" + uppercase: "A"..."Z" + digit: "0"..."9" Identifiers are unlimited in length. Case is significant. -.. productionlist:: - identifier: `xid_start` `xid_continue`* - id_start: <all characters in general categories Lu, Ll, Lt, Lm, Lo, Nl, the underscore, and characters with the Other_ID_Start property> - id_continue: <all characters in `id_start`, plus characters in the categories Mn, Mc, Nd, Pc and others with the Other_ID_Continue property> - xid_start: <all characters in `id_start` whose NFKC normalization is in "id_start xid_continue*"> - xid_continue: <all characters in `id_continue` whose NFKC normalization is in "id_continue*"> - -The Unicode category codes mentioned above stand for: - -* *Lu* - uppercase letters -* *Ll* - lowercase letters -* *Lt* - titlecase letters -* *Lm* - modifier letters -* *Lo* - other letters -* *Nl* - letter numbers -* *Mn* - nonspacing marks -* *Mc* - spacing combining marks -* *Nd* - decimal numbers -* *Pc* - connector punctuations -* *Other_ID_Start* - explicit list of characters in `PropList.txt - <http://www.unicode.org/Public/12.1.0/ucd/PropList.txt>`_ to support backwards - compatibility -* *Other_ID_Continue* - likewise - -All identifiers are converted into the normal form NFKC while parsing; comparison -of identifiers is based on NFKC. - -A non-normative HTML file listing all valid identifier characters for Unicode -4.1 can be found at -https://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html. - .. _keywords: @@ -343,17 +343,27 @@ exactly as written here: .. sourcecode:: text - False await else import pass - None break except in raise - True class finally is return - and continue for lambda try - as def from nonlocal while - assert del global not with - async elif if or yield + and del from not while + as elif global or with + assert else if pass yield + break except import print + class exec in raise + continue finally is return + def for lambda try + +.. versionchanged:: 2.4 + :const:`None` became a constant and is now recognized by the compiler as a name + for the built-in object :const:`None`. Although it is not a keyword, you cannot + assign a different object to it. + +.. versionchanged:: 2.5 + Using :keyword:`as` and :keyword:`with` as identifiers triggers a warning. To + use them as keywords, enable the ``with_statement`` future feature . + +.. versionchanged:: 2.6 + :keyword:`as` and :keyword:`with` are full keywords. + -.. index:: - single: _, identifiers - single: __, identifiers .. _id-classes: Reserved classes of identifiers @@ -366,7 +376,7 @@ characters: ``_*`` Not imported by ``from module import *``. The special identifier ``_`` is used in the interactive interpreter to store the result of the last evaluation; it is - stored in the :mod:`builtins` module. When not in interactive mode, ``_`` + stored in the :mod:`__builtin__` module. When not in interactive mode, ``_`` has no special meaning and is not defined. See section :ref:`import`. .. note:: @@ -395,128 +405,83 @@ characters: Literals ======== -.. index:: literal, constant +.. index:: + single: literal + single: constant Literals are notations for constant values of some built-in types. -.. index:: string literal, bytes literal, ASCII - single: ' (single quote); string literal - single: " (double quote); string literal - single: u'; string literal - single: u"; string literal .. _strings: -String and Bytes literals -------------------------- +String literals +--------------- + +.. index:: single: string literal String literals are described by the following lexical definitions: +.. index:: single: ASCII@ASCII + .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) - stringprefix: "r" | "u" | "R" | "U" | "f" | "F" - : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" + stringprefix: "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR" + : | "b" | "B" | "br" | "Br" | "bR" | "BR" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' - longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' - shortstringitem: `shortstringchar` | `stringescapeseq` - longstringitem: `longstringchar` | `stringescapeseq` + longstring: "'''" `longstringitem`* "'''" + : | '"""' `longstringitem`* '"""' + shortstringitem: `shortstringchar` | `escapeseq` + longstringitem: `longstringchar` | `escapeseq` shortstringchar: <any source character except "\" or newline or the quote> longstringchar: <any source character except "\"> - stringescapeseq: "\" <any source character> - -.. productionlist:: - bytesliteral: `bytesprefix`(`shortbytes` | `longbytes`) - bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB" - shortbytes: "'" `shortbytesitem`* "'" | '"' `shortbytesitem`* '"' - longbytes: "'''" `longbytesitem`* "'''" | '"""' `longbytesitem`* '"""' - shortbytesitem: `shortbyteschar` | `bytesescapeseq` - longbytesitem: `longbyteschar` | `bytesescapeseq` - shortbyteschar: <any ASCII character except "\" or newline or the quote> - longbyteschar: <any ASCII character except "\"> - bytesescapeseq: "\" <any ASCII character> + escapeseq: "\" <any ASCII character> One syntactic restriction not indicated by these productions is that whitespace -is not allowed between the :token:`stringprefix` or :token:`bytesprefix` and the -rest of the literal. The source character set is defined by the encoding -declaration; it is UTF-8 if no encoding declaration is given in the source file; -see section :ref:`encodings`. +is not allowed between the :token:`stringprefix` and the rest of the string +literal. The source character set is defined by the encoding declaration; it is +ASCII if no encoding declaration is given in the source file; see section +:ref:`encodings`. -.. index:: triple-quoted string, Unicode Consortium, raw string - single: """; string literal - single: '''; string literal +.. index:: + single: triple-quoted string + single: Unicode Consortium + single: string; Unicode + single: raw string -In plain English: Both types of literals can be enclosed in matching single quotes +In plain English: String literals can be enclosed in matching single quotes (``'``) or double quotes (``"``). They can also be enclosed in matching groups of three single or double quotes (these are generally referred to as *triple-quoted strings*). The backslash (``\``) character is used to escape characters that otherwise have a special meaning, such as newline, backslash -itself, or the quote character. - -.. index:: - single: b'; bytes literal - single: b"; bytes literal - -Bytes literals are always prefixed with ``'b'`` or ``'B'``; they produce an -instance of the :class:`bytes` type instead of the :class:`str` type. They -may only contain ASCII characters; bytes with a numeric value of 128 or greater -must be expressed with escapes. +itself, or the quote character. String literals may optionally be prefixed with +a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and use +different rules for interpreting backslash escape sequences. A prefix of +``'u'`` or ``'U'`` makes the string a Unicode string. Unicode strings use the +Unicode character set as defined by the Unicode Consortium and ISO 10646. Some +additional escape sequences, described below, are available in Unicode strings. +A prefix of ``'b'`` or ``'B'`` is ignored in Python 2; it indicates that the +literal should become a bytes literal in Python 3 (e.g. when code is +automatically converted with 2to3). A ``'u'`` or ``'b'`` prefix may be followed +by an ``'r'`` prefix. + +In triple-quoted strings, unescaped newlines and quotes are allowed (and are +retained), except that three unescaped quotes in a row terminate the string. (A +"quote" is the character used to open the string, i.e. either ``'`` or ``"``.) .. index:: - single: r'; raw string literal - single: r"; raw string literal - -Both string and bytes literals may optionally be prefixed with a letter ``'r'`` -or ``'R'``; such strings are called :dfn:`raw strings` and treat backslashes as -literal characters. As a result, in string literals, ``'\U'`` and ``'\u'`` -escapes in raw strings are not treated specially. Given that Python 2.x's raw -unicode literals behave differently than Python 3.x's the ``'ur'`` syntax -is not supported. - -.. versionadded:: 3.3 - The ``'rb'`` prefix of raw bytes literals has been added as a synonym - of ``'br'``. + single: physical line + single: escape sequence + single: Standard C + single: C -.. versionadded:: 3.3 - Support for the unicode legacy literal (``u'value'``) was reintroduced - to simplify the maintenance of dual Python 2.x and 3.x codebases. - See :pep:`414` for more information. - -.. index:: - single: f'; formatted string literal - single: f"; formatted string literal - -A string literal with ``'f'`` or ``'F'`` in its prefix is a -:dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be -combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw -formatted strings are possible, but formatted bytes literals are not. - -In triple-quoted literals, unescaped newlines and quotes are allowed (and are -retained), except that three unescaped quotes in a row terminate the literal. (A -"quote" is the character used to open the literal, i.e. either ``'`` or ``"``.) - -.. index:: physical line, escape sequence, Standard C, C - single: \ (backslash); escape sequence - single: \\; escape sequence - single: \a; escape sequence - single: \b; escape sequence - single: \f; escape sequence - single: \n; escape sequence - single: \r; escape sequence - single: \t; escape sequence - single: \v; escape sequence - single: \x; escape sequence - single: \N; escape sequence - single: \u; escape sequence - single: \U; escape sequence - -Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in string and -bytes literals are interpreted according to rules similar to those used by -Standard C. The recognized escape sequences are: +Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in strings are +interpreted according to rules similar to those used by Standard C. The +recognized escape sequences are: +-----------------+---------------------------------+-------+ | Escape Sequence | Meaning | Notes | +=================+=================================+=======+ -| ``\newline`` | Backslash and newline ignored | | +| ``\newline`` | Ignored | | +-----------------+---------------------------------+-------+ | ``\\`` | Backslash (``\``) | | +-----------------+---------------------------------+-------+ @@ -532,90 +497,92 @@ Standard C. The recognized escape sequences are: +-----------------+---------------------------------+-------+ | ``\n`` | ASCII Linefeed (LF) | | +-----------------+---------------------------------+-------+ +| ``\N{name}`` | Character named *name* in the | | +| | Unicode database (Unicode only) | | ++-----------------+---------------------------------+-------+ | ``\r`` | ASCII Carriage Return (CR) | | +-----------------+---------------------------------+-------+ | ``\t`` | ASCII Horizontal Tab (TAB) | | +-----------------+---------------------------------+-------+ +| ``\uxxxx`` | Character with 16-bit hex value | \(1) | +| | *xxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ +| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(2) | +| | *xxxxxxxx* (Unicode only) | | ++-----------------+---------------------------------+-------+ | ``\v`` | ASCII Vertical Tab (VT) | | +-----------------+---------------------------------+-------+ -| ``\ooo`` | Character with octal value | (1,3) | +| ``\ooo`` | Character with octal value | (3,5) | | | *ooo* | | +-----------------+---------------------------------+-------+ -| ``\xhh`` | Character with hex value *hh* | (2,3) | +| ``\xhh`` | Character with hex value *hh* | (4,5) | +-----------------+---------------------------------+-------+ -Escape sequences only recognized in string literals are: - -+-----------------+---------------------------------+-------+ -| Escape Sequence | Meaning | Notes | -+=================+=================================+=======+ -| ``\N{name}`` | Character named *name* in the | \(4) | -| | Unicode database | | -+-----------------+---------------------------------+-------+ -| ``\uxxxx`` | Character with 16-bit hex value | \(5) | -| | *xxxx* | | -+-----------------+---------------------------------+-------+ -| ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(6) | -| | *xxxxxxxx* | | -+-----------------+---------------------------------+-------+ +.. index:: single: ASCII@ASCII Notes: (1) - As in Standard C, up to three octal digits are accepted. + Individual code units which form parts of a surrogate pair can be encoded using + this escape sequence. (2) - Unlike in Standard C, exactly two hex digits are required. + Any Unicode character can be encoded this way, but characters outside the Basic + Multilingual Plane (BMP) will be encoded using a surrogate pair if Python is + compiled to use 16-bit code units (the default). (3) - In a bytes literal, hexadecimal and octal escapes denote the byte with the - given value. In a string literal, these escapes denote a Unicode character - with the given value. + As in Standard C, up to three octal digits are accepted. (4) - .. versionchanged:: 3.3 - Support for name aliases [#]_ has been added. + Unlike in Standard C, exactly two hex digits are required. (5) - Exactly four hex digits are required. - -(6) - Any Unicode character can be encoded this way. Exactly eight hex digits - are required. - + In a string literal, hexadecimal and octal escapes denote the byte with the + given value; it is not necessary that the byte encodes a character in the source + character set. In a Unicode literal, these escapes denote a Unicode character + with the given value. -.. index:: unrecognized escape sequence +.. index:: single: unrecognized escape sequence Unlike Standard C, all unrecognized escape sequences are left in the string -unchanged, i.e., *the backslash is left in the result*. (This behavior is +unchanged, i.e., *the backslash is left in the string*. (This behavior is useful when debugging: if an escape sequence is mistyped, the resulting output is more easily recognized as broken.) It is also important to note that the -escape sequences only recognized in string literals fall into the category of -unrecognized escapes for bytes literals. - - .. versionchanged:: 3.6 - Unrecognized escape sequences produce a :exc:`DeprecationWarning`. In - a future Python version they will be a :exc:`SyntaxWarning` and - eventually a :exc:`SyntaxError`. - -Even in a raw literal, quotes can be escaped with a backslash, but the -backslash remains in the result; for example, ``r"\""`` is a valid string -literal consisting of two characters: a backslash and a double quote; ``r"\"`` -is not a valid string literal (even a raw string cannot end in an odd number of -backslashes). Specifically, *a raw literal cannot end in a single backslash* -(since the backslash would escape the following quote character). Note also -that a single backslash followed by a newline is interpreted as those two -characters as part of the literal, *not* as a line continuation. - - -.. _string-concatenation: +escape sequences marked as "(Unicode only)" in the table above fall into the +category of unrecognized escapes for non-Unicode string literals. + +When an ``'r'`` or ``'R'`` prefix is present, a character following a backslash +is included in the string without change, and *all backslashes are left in the +string*. For example, the string literal ``r"\n"`` consists of two characters: +a backslash and a lowercase ``'n'``. String quotes can be escaped with a +backslash, but the backslash remains in the string; for example, ``r"\""`` is a +valid string literal consisting of two characters: a backslash and a double +quote; ``r"\"`` is not a valid string literal (even a raw string cannot end in +an odd number of backslashes). Specifically, *a raw string cannot end in a +single backslash* (since the backslash would escape the following quote +character). Note also that a single backslash followed by a newline is +interpreted as those two characters as part of the string, *not* as a line +continuation. + +When an ``'r'`` or ``'R'`` prefix is used in conjunction with a ``'u'`` or +``'U'`` prefix, then the ``\uXXXX`` and ``\UXXXXXXXX`` escape sequences are +processed while *all other backslashes are left in the string*. For example, +the string literal ``ur"\u0062\n"`` consists of three Unicode characters: 'LATIN +SMALL LETTER B', 'REVERSE SOLIDUS', and 'LATIN SMALL LETTER N'. Backslashes can +be escaped with a preceding backslash; however, both remain in the string. As a +result, ``\uXXXX`` escape sequences are only recognized when there are an odd +number of backslashes. + + +.. _string-catenation: String literal concatenation ---------------------------- -Multiple adjacent string or bytes literals (delimited by whitespace), possibly -using different quoting conventions, are allowed, and their meaning is the same -as their concatenation. Thus, ``"hello" 'world'`` is equivalent to +Multiple adjacent string literals (delimited by whitespace), possibly using +different quoting conventions, are allowed, and their meaning is the same as +their concatenation. Thus, ``"hello" 'world'`` is equivalent to ``"helloworld"``. This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings, for example:: @@ -627,134 +594,7 @@ comments to parts of strings, for example:: Note that this feature is defined at the syntactical level, but implemented at compile time. The '+' operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting -styles for each component (even mixing raw strings and triple quoted strings), -and formatted string literals may be concatenated with plain string literals. - - -.. index:: - single: formatted string literal - single: interpolated string literal - single: string; formatted literal - single: string; interpolated literal - single: f-string - single: {} (curly brackets); in formatted string literal - single: ! (exclamation); in formatted string literal - single: : (colon); in formatted string literal -.. _f-strings: - -Formatted string literals -------------------------- - -.. versionadded:: 3.6 - -A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal -that is prefixed with ``'f'`` or ``'F'``. These strings may contain -replacement fields, which are expressions delimited by curly braces ``{}``. -While other string literals always have a constant value, formatted strings -are really expressions evaluated at run time. - -Escape sequences are decoded like in ordinary string literals (except when -a literal is also marked as a raw string). After decoding, the grammar -for the contents of the string is: - -.. productionlist:: - f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)* - replacement_field: "{" `f_expression` ["!" `conversion`] [":" `format_spec`] "}" - f_expression: (`conditional_expression` | "*" `or_expr`) - : ("," `conditional_expression` | "," "*" `or_expr`)* [","] - : | `yield_expression` - conversion: "s" | "r" | "a" - format_spec: (`literal_char` | NULL | `replacement_field`)* - literal_char: <any code point except "{", "}" or NULL> - -The parts of the string outside curly braces are treated literally, -except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced -with the corresponding single curly brace. A single opening curly -bracket ``'{'`` marks a replacement field, which starts with a -Python expression. After the expression, there may be a conversion field, -introduced by an exclamation point ``'!'``. A format specifier may also -be appended, introduced by a colon ``':'``. A replacement field ends -with a closing curly bracket ``'}'``. - -Expressions in formatted string literals are treated like regular -Python expressions surrounded by parentheses, with a few exceptions. -An empty expression is not allowed, and both :keyword:`lambda` and -assignment expressions ``:=`` must be surrounded by explicit parentheses. -Replacement expressions can contain line breaks (e.g. in triple-quoted -strings), but they cannot contain comments. Each expression is evaluated -in the context where the formatted string literal appears, in order from -left to right. - -If a conversion is specified, the result of evaluating the expression -is converted before formatting. Conversion ``'!s'`` calls :func:`str` on -the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. - -The result is then formatted using the :func:`format` protocol. The -format specifier is passed to the :meth:`__format__` method of the -expression or conversion result. An empty string is passed when the -format specifier is omitted. The formatted result is then included in -the final value of the whole string. - -Top-level format specifiers may include nested replacement fields. These nested -fields may include their own conversion fields and :ref:`format specifiers -<formatspec>`, but may not include more deeply-nested replacement fields. The -:ref:`format specifier mini-language <formatspec>` is the same as that used by -the string .format() method. - -Formatted string literals may be concatenated, but replacement fields -cannot be split across literals. - -Some examples of formatted string literals:: - - >>> name = "Fred" - >>> f"He said his name is {name!r}." - "He said his name is 'Fred'." - >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r - "He said his name is 'Fred'." - >>> width = 10 - >>> precision = 4 - >>> value = decimal.Decimal("12.34567") - >>> f"result: {value:{width}.{precision}}" # nested fields - 'result: 12.35' - >>> today = datetime(year=2017, month=1, day=27) - >>> f"{today:%B %d, %Y}" # using date format specifier - 'January 27, 2017' - >>> number = 1024 - >>> f"{number:#0x}" # using integer format specifier - '0x400' - -A consequence of sharing the same syntax as regular string literals is -that characters in the replacement fields must not conflict with the -quoting used in the outer formatted string literal:: - - f"abc {a["x"]} def" # error: outer string literal ended prematurely - f"abc {a['x']} def" # workaround: use different quoting - -Backslashes are not allowed in format expressions and will raise -an error:: - - f"newline: {ord('\n')}" # raises SyntaxError - -To include a value in which a backslash escape is required, create -a temporary variable. - - >>> newline = ord('\n') - >>> f"newline: {newline}" - 'newline: 10' - -Formatted string literals cannot be used as docstrings, even if they do not -include expressions. - -:: - - >>> def foo(): - ... f"Not a docstring" - ... - >>> foo.__doc__ is None - True - -See also :pep:`498` for the proposal that added formatted string literals, -and :meth:`str.format`, which uses a related format string mechanism. +styles for each component (even mixing raw strings and triple quoted strings). .. _numbers: @@ -762,69 +602,66 @@ and :meth:`str.format`, which uses a related format string mechanism. Numeric literals ---------------- -.. index:: number, numeric literal, integer literal - floating point literal, hexadecimal literal - octal literal, binary literal, decimal literal, imaginary literal, complex literal - -There are three types of numeric literals: integers, floating point numbers, and -imaginary numbers. There are no complex literals (complex numbers can be formed -by adding a real number and an imaginary number). +.. index:: + single: number + single: numeric literal + single: integer literal + single: plain integer literal + single: long integer literal + single: floating point literal + single: hexadecimal literal + single: binary literal + single: octal literal + single: decimal literal + single: imaginary literal + single: complex; literal + +There are four types of numeric literals: plain integers, long integers, +floating point numbers, and imaginary numbers. There are no complex literals +(complex numbers can be formed by adding a real number and an imaginary number). Note that numeric literals do not include a sign; a phrase like ``-1`` is actually an expression composed of the unary operator '``-``' and the literal ``1``. -.. index:: - single: 0b; integer literal - single: 0o; integer literal - single: 0x; integer literal - single: _ (underscore); in numeric literal - .. _integers: -Integer literals ----------------- +Integer and long integer literals +--------------------------------- -Integer literals are described by the following lexical definitions: +Integer and long integer literals are described by the following lexical +definitions: .. productionlist:: - integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger` - decinteger: `nonzerodigit` (["_"] `digit`)* | "0"+ (["_"] "0")* - bininteger: "0" ("b" | "B") (["_"] `bindigit`)+ - octinteger: "0" ("o" | "O") (["_"] `octdigit`)+ - hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+ + longinteger: `integer` ("l" | "L") + integer: `decimalinteger` | `octinteger` | `hexinteger` | `bininteger` + decimalinteger: `nonzerodigit` `digit`* | "0" + octinteger: "0" ("o" | "O") `octdigit`+ | "0" `octdigit`+ + hexinteger: "0" ("x" | "X") `hexdigit`+ + bininteger: "0" ("b" | "B") `bindigit`+ nonzerodigit: "1"..."9" - digit: "0"..."9" - bindigit: "0" | "1" octdigit: "0"..."7" + bindigit: "0" | "1" hexdigit: `digit` | "a"..."f" | "A"..."F" -There is no limit for the length of integer literals apart from what can be -stored in available memory. +Although both lower case ``'l'`` and upper case ``'L'`` are allowed as suffix +for long integers, it is strongly recommended to always use ``'L'``, since the +letter ``'l'`` looks too much like the digit ``'1'``. -Underscores are ignored for determining the numeric value of the literal. They -can be used to group digits for enhanced readability. One underscore can occur -between digits, and after base specifiers like ``0x``. +Plain integer literals that are above the largest representable plain integer +(e.g., 2147483647 when using 32-bit arithmetic) are accepted as if they were +long integers instead. [#]_ There is no limit for long integer literals apart +from what can be stored in available memory. -Note that leading zeros in a non-zero decimal number are not allowed. This is -for disambiguation with C-style octal literals, which Python used before version -3.0. +Some examples of plain integer literals (first row) and long integer literals +(second and third rows):: -Some examples of integer literals:: + 7 2147483647 0177 + 3L 79228162514264337593543950336L 0377L 0x100000000L + 79228162514264337593543950336 0xdeadbeef - 7 2147483647 0o177 0b100110111 - 3 79228162514264337593543950336 0o377 0xdeadbeef - 100_000_000_000 0b_1110_0101 -.. versionchanged:: 3.6 - Underscores are now allowed for grouping purposes in literals. - - -.. index:: - single: . (dot); in numeric literal - single: e; in numeric literal - single: _ (underscore); in numeric literal .. _floating: Floating point literals @@ -834,27 +671,25 @@ Floating point literals are described by the following lexical definitions: .. productionlist:: floatnumber: `pointfloat` | `exponentfloat` - pointfloat: [`digitpart`] `fraction` | `digitpart` "." - exponentfloat: (`digitpart` | `pointfloat`) `exponent` - digitpart: `digit` (["_"] `digit`)* - fraction: "." `digitpart` - exponent: ("e" | "E") ["+" | "-"] `digitpart` + pointfloat: [`intpart`] `fraction` | `intpart` "." + exponentfloat: (`intpart` | `pointfloat`) `exponent` + intpart: `digit`+ + fraction: "." `digit`+ + exponent: ("e" | "E") ["+" | "-"] `digit`+ -Note that the integer and exponent parts are always interpreted using radix 10. -For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The -allowed range of floating point literals is implementation-dependent. As in -integer literals, underscores are supported for digit grouping. +Note that the integer and exponent parts of floating point numbers can look like +octal integers, but are interpreted using radix 10. For example, ``077e010`` is +legal, and denotes the same number as ``77e10``. The allowed range of floating +point literals is implementation-dependent. Some examples of floating point +literals:: -Some examples of floating point literals:: + 3.14 10. .001 1e100 3.14e-10 0e0 - 3.14 10. .001 1e100 3.14e-10 0e0 3.14_15_93 - -.. versionchanged:: 3.6 - Underscores are now allowed for grouping purposes in literals. +Note that numeric literals do not include a sign; a phrase like ``-1`` is +actually an expression composed of the unary operator ``-`` and the literal +``1``. -.. index:: - single: j; in numeric literal .. _imaginary: Imaginary literals @@ -863,7 +698,7 @@ Imaginary literals Imaginary literals are described by the following lexical definitions: .. productionlist:: - imagnumber: (`floatnumber` | `digitpart`) ("j" | "J") + imagnumber: (`floatnumber` | `intpart`) ("j" | "J") An imaginary literal yields a complex number with a real part of 0.0. Complex numbers are represented as a pair of floating point numbers and have the same @@ -871,7 +706,7 @@ restrictions on their range. To create a complex number with a nonzero real part, add a floating point number to it, e.g., ``(3+4j)``. Some examples of imaginary literals:: - 3.14j 10.j 10j .001j 1e100j 3.14e-10j 3.14_15_93j + 3.14j 10.j 10j .001j 1e100j 3.14e-10j .. _operators: @@ -886,9 +721,12 @@ The following tokens are operators: .. code-block:: none - + - * ** / // % @ - << >> & | ^ ~ := - < > <= >= == != + + - * ** / // % + << >> & | ^ ~ + < > <= >= == != <> + +The comparison operators ``<>`` and ``!=`` are alternate spellings of the same +operator. ``!=`` is the preferred spelling; ``<>`` is obsolescent. .. _delimiters: @@ -902,13 +740,13 @@ The following tokens serve as delimiters in the grammar: .. code-block:: none - ( ) [ ] { } - , : . ; @ = -> - += -= *= /= //= %= @= + ( ) [ ] { } @ + , : . ` = ; + += -= *= /= //= %= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence -of three periods has a special meaning as an ellipsis literal. The second half +of three periods has a special meaning as an ellipsis in slices. The second half of the list, the augmented assignment operators, serve lexically as delimiters, but also perform an operation. @@ -919,14 +757,20 @@ tokens or are otherwise significant to the lexical analyzer: ' " # \ +.. index:: single: ASCII@ASCII + The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error: .. code-block:: none - $ ? ` - + $ ? .. rubric:: Footnotes -.. [#] http://www.unicode.org/Public/11.0.0/ucd/NameAliases.txt +.. [#] In versions of Python prior to 2.4, octal and hexadecimal literals in the range + just above the largest representable plain integer but below the largest + unsigned 32-bit number (on a machine using 32-bit arithmetic), 4294967296, were + taken as the negative plain integer obtained by subtracting 4294967296 from + their unsigned value. + diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index a8ec0fb..e2d643f 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -7,7 +7,7 @@ Simple statements .. index:: pair: simple; statement -A simple statement is comprised within a single logical line. Several simple +Simple statements are comprised within a single logical line. Several simple statements may occur on a single line separated by semicolons. The syntax for simple statements is: @@ -16,9 +16,9 @@ simple statements is: : | `assert_stmt` : | `assignment_stmt` : | `augmented_assignment_stmt` - : | `annotated_assignment_stmt` : | `pass_stmt` : | `del_stmt` + : | `print_stmt` : | `return_stmt` : | `yield_stmt` : | `raise_stmt` @@ -27,7 +27,7 @@ simple statements is: : | `import_stmt` : | `future_stmt` : | `global_stmt` - : | `nonlocal_stmt` + : | `exec_stmt` .. _exprstmts: @@ -38,7 +38,6 @@ Expression statements .. index:: pair: expression; statement pair: expression; list -.. index:: pair: expression; list Expression statements are used (mostly interactively) to compute and write a value, or (usually) to call a procedure (a function that returns no meaningful @@ -47,7 +46,7 @@ expression statements are allowed and occasionally useful. The syntax for an expression statement is: .. productionlist:: - expression_stmt: `starred_expression` + expression_stmt: `expression_list` An expression statement evaluates the expression list (which may be a single expression). @@ -63,8 +62,10 @@ expression). In interactive mode, if the value is not ``None``, it is converted to a string using the built-in :func:`repr` function and the resulting string is written to -standard output on a line by itself (except if the result is ``None``, so that -procedure calls do not cause any output.) +standard output (see section :ref:`print`) on a line by itself. (Expression +statements yielding ``None`` are not written, so that procedure calls do not +cause any output.) + .. _assignment: @@ -72,7 +73,7 @@ Assignment statements ===================== .. index:: - single: = (equals); assignment statement + single: =; assignment statement pair: assignment; statement pair: binding; name pair: rebinding; name @@ -83,18 +84,19 @@ Assignment statements are used to (re)bind names to values and to modify attributes or items of mutable objects: .. productionlist:: - assignment_stmt: (`target_list` "=")+ (`starred_expression` | `yield_expression`) + assignment_stmt: (`target_list` "=")+ (`expression_list` | `yield_expression`) target_list: `target` ("," `target`)* [","] target: `identifier` - : | "(" [`target_list`] ")" + : | "(" `target_list` ")" : | "[" [`target_list`] "]" : | `attributeref` : | `subscription` : | `slicing` - : | "*" `target` -(See section :ref:`primaries` for the syntax definitions for *attributeref*, -*subscription*, and *slicing*.) +(See section :ref:`primaries` for the syntax definitions for the last three +symbols.) + +.. index:: pair: expression; list An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and @@ -113,49 +115,36 @@ unacceptable. The rules observed by various types and the exceptions raised are given with the definition of the object types (see section :ref:`types`). .. index:: triple: target; list; assignment - single: , (comma); in target list - single: * (asterisk); in assignment target list - single: [] (square brackets); in assignment target list - single: () (parentheses); in assignment target list - -Assignment of an object to a target list, optionally enclosed in parentheses or -square brackets, is recursively defined as follows. - -* If the target list is a single target with no trailing comma, - optionally in parentheses, the object is assigned to that target. -* Else: The object must be an iterable with the same number of - items as there are targets in the target list, and the items are assigned, - from left to right, to the corresponding targets. +Assignment of an object to a target list is recursively defined as follows. - * If the target list contains one target prefixed with an asterisk, called a - "starred" target: The object must be an iterable with at least as many items - as there are targets in the target list, minus one. The first items of the - iterable are assigned, from left to right, to the targets before the starred - target. The final items of the iterable are assigned to the targets after - the starred target. A list of the remaining items in the iterable is then - assigned to the starred target (the list can be empty). +* If the target list is a single target: The object is assigned to that target. - * Else: The object must be an iterable with the same number of items as there - are targets in the target list, and the items are assigned, from left to - right, to the corresponding targets. +* If the target list is a comma-separated list of targets: The object must be an + iterable with the same number of items as there are targets in the target list, + and the items are assigned, from left to right, to the corresponding targets. Assignment of an object to a single target is recursively defined as follows. * If the target is an identifier (name): - * If the name does not occur in a :keyword:`global` or :keyword:`nonlocal` - statement in the current code block: the name is bound to the object in the - current local namespace. + .. index:: statement: global - * Otherwise: the name is bound to the object in the global namespace or the - outer namespace determined by :keyword:`nonlocal`, respectively. + * If the name does not occur in a :keyword:`global` statement in the current + code block: the name is bound to the object in the current local namespace. + + * Otherwise: the name is bound to the object in the current global namespace. .. index:: single: destructor - The name is rebound if it was already bound. This may cause the reference - count for the object previously bound to the name to reach zero, causing the - object to be deallocated and its destructor (if it has one) to be called. + The name is rebound if it was already bound. This may cause the reference count + for the object previously bound to the name to reach zero, causing the object to + be deallocated and its destructor (if it has one) to be called. + +* If the target is a target list enclosed in parentheses or in square brackets: + The object must be an iterable with the same number of items as there are + targets in the target list, and its items are assigned, from left to right, + to the corresponding targets. .. index:: pair: attribute; assignment @@ -169,12 +158,12 @@ Assignment of an object to a single target is recursively defined as follows. .. _attr-target-note: Note: If the object is a class instance and the attribute reference occurs on - both sides of the assignment operator, the right-hand side expression, ``a.x`` can access + both sides of the assignment operator, the RHS expression, ``a.x`` can access either an instance attribute or (if no instance attribute exists) a class - attribute. The left-hand side target ``a.x`` is always set as an instance attribute, + attribute. The LHS target ``a.x`` is always set as an instance attribute, creating it if necessary. Thus, the two occurrences of ``a.x`` do not - necessarily refer to the same attribute: if the right-hand side expression refers to a - class attribute, the left-hand side creates a new instance attribute as the target of the + necessarily refer to the same attribute: if the RHS expression refers to a + class attribute, the LHS creates a new instance attribute as the target of the assignment:: class Cls: @@ -190,20 +179,20 @@ Assignment of an object to a single target is recursively defined as follows. object: mutable * If the target is a subscription: The primary expression in the reference is - evaluated. It should yield either a mutable sequence object (such as a list) - or a mapping object (such as a dictionary). Next, the subscript expression is + evaluated. It should yield either a mutable sequence object (such as a list) or + a mapping object (such as a dictionary). Next, the subscript expression is evaluated. .. index:: object: sequence object: list - If the primary is a mutable sequence object (such as a list), the subscript - must yield an integer. If it is negative, the sequence's length is added to - it. The resulting value must be a nonnegative integer less than the - sequence's length, and the sequence is asked to assign the assigned object to - its item with that index. If the index is out of range, :exc:`IndexError` is - raised (assignment to a subscripted sequence cannot add new items to a list). + If the primary is a mutable sequence object (such as a list), the subscript must + yield a plain integer. If it is negative, the sequence's length is added to it. + The resulting value must be a nonnegative integer less than the sequence's + length, and the sequence is asked to assign the assigned object to its item with + that index. If the index is out of range, :exc:`IndexError` is raised + (assignment to a subscripted sequence cannot add new items to a list). .. index:: object: mapping @@ -215,22 +204,19 @@ Assignment of an object to a single target is recursively defined as follows. object. This can either replace an existing key/value pair with the same key value, or insert a new key/value pair (if no key with the same value existed). - For user-defined objects, the :meth:`__setitem__` method is called with - appropriate arguments. - .. index:: pair: slicing; assignment * If the target is a slicing: The primary expression in the reference is evaluated. It should yield a mutable sequence object (such as a list). The assigned object should be a sequence object of the same type. Next, the lower and upper bound expressions are evaluated, insofar they are present; defaults - are zero and the sequence's length. The bounds should evaluate to integers. - If either bound is negative, the sequence's length is added to it. The - resulting bounds are clipped to lie between zero and the sequence's length, - inclusive. Finally, the sequence object is asked to replace the slice with - the items of the assigned sequence. The length of the slice may be different - from the length of the assigned sequence, thus changing the length of the - target sequence, if the target sequence allows it. + are zero and the sequence's length. The bounds should evaluate to (small) + integers. If either bound is negative, the sequence's length is added to it. + The resulting bounds are clipped to lie between zero and the sequence's length, + inclusive. Finally, the sequence object is asked to replace the slice with the + items of the assigned sequence. The length of the slice may be different from + the length of the assigned sequence, thus changing the length of the target + sequence, if the object allows it. .. impl-detail:: @@ -238,22 +224,15 @@ Assignment of an object to a single target is recursively defined as follows. as for expressions, and invalid syntax is rejected during the code generation phase, causing less detailed error messages. -Although the definition of assignment implies that overlaps between the -left-hand side and the right-hand side are 'simultaneous' (for example ``a, b = -b, a`` swaps two variables), overlaps *within* the collection of assigned-to -variables occur left-to-right, sometimes resulting in confusion. For instance, -the following program prints ``[0, 2]``:: +WARNING: Although the definition of assignment implies that overlaps between the +left-hand side and the right-hand side are 'safe' (for example ``a, b = b, a`` +swaps two variables), overlaps *within* the collection of assigned-to variables +are not safe! For instance, the following program prints ``[0, 2]``:: x = [0, 1] i = 0 - i, x[i] = 1, 2 # i is updated, then x[i] is updated - print(x) - - -.. seealso:: - - :pep:`3132` - Extended Iterable Unpacking - The specification for the ``*target`` feature. + i, x[i] = 1, 2 + print x .. _augassign: @@ -283,10 +262,10 @@ operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` - augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" + augop: "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" -(See section :ref:`primaries` for the syntax definitions of the last three +(See section :ref:`primaries` for the syntax definitions for the last three symbols.) An augmented assignment evaluates the target (which, unlike normal assignment @@ -300,11 +279,6 @@ version, ``x`` is only evaluated once. Also, when possible, the actual operation is performed *in-place*, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. -Unlike normal assignments, augmented assignments evaluate the left-hand side -*before* evaluating the right-hand side. For example, ``a[i] += f(x)`` first -looks-up ``a[i]``, then it evaluates ``f(x)`` and performs the addition, and -lastly, it writes the result back to ``a[i]``. - With the exception of assigning to tuples and multiple targets in a single statement, the assignment done by augmented assignment statements is handled the same way as normal assignments. Similarly, with the exception of the possible @@ -315,72 +289,14 @@ For targets which are attribute references, the same :ref:`caveat about class and instance attributes <attr-target-note>` applies as for regular assignments. -.. _annassign: - -Annotated assignment statements -------------------------------- - -.. index:: - pair: annotated; assignment - single: statement; assignment, annotated - single: : (colon); annotated variable - -:term:`Annotation <variable annotation>` assignment is the combination, in a single -statement, of a variable or attribute annotation and an optional assignment statement: - -.. productionlist:: - annotated_assignment_stmt: `augtarget` ":" `expression` - : ["=" (`starred_expression` | `yield_expression`)] - -The difference from normal :ref:`assignment` is that only single target is allowed. - -For simple names as assignment targets, if in class or module scope, -the annotations are evaluated and stored in a special class or module -attribute :attr:`__annotations__` -that is a dictionary mapping from variable names (mangled if private) to -evaluated annotations. This attribute is writable and is automatically -created at the start of class or module body execution, if annotations -are found statically. - -For expressions as assignment targets, the annotations are evaluated if -in class or module scope, but not stored. - -If a name is annotated in a function scope, then this name is local for -that scope. Annotations are never evaluated and stored in function scopes. - -If the right hand side is present, an annotated -assignment performs the actual assignment before evaluating annotations -(where applicable). If the right hand side is not present for an expression -target, then the interpreter evaluates the target except for the last -:meth:`__setitem__` or :meth:`__setattr__` call. - -.. seealso:: - - :pep:`526` - Syntax for Variable Annotations - The proposal that added syntax for annotating the types of variables - (including class variables and instance variables), instead of expressing - them through comments. - - :pep:`484` - Type hints - The proposal that added the :mod:`typing` module to provide a standard - syntax for type annotations that can be used in static analysis tools and - IDEs. - -.. versionchanged:: 3.8 - Now annotated assignments allow same expressions in the right hand side as - the regular assignments. Previously, some expressions (like un-parenthesized - tuple expressions) caused a syntax error. - - .. _assert: -The :keyword:`!assert` statement -================================ +The :keyword:`assert` statement +=============================== .. index:: - ! statement: assert + statement: assert pair: debugging; assertions - single: , (comma); expression list Assert statements are a convenient way to insert debugging assertions into a program: @@ -405,7 +321,7 @@ The extended form, ``assert expression1, expression2``, is equivalent to :: These equivalences assume that :const:`__debug__` and :exc:`AssertionError` refer to the built-in variables with those names. In the current implementation, the built-in variable :const:`__debug__` is ``True`` under normal circumstances, -``False`` when optimization is requested (command line option :option:`-O`). The current +``False`` when optimization is requested (command line option -O). The current code generator emits no code for an assert statement when optimization is requested at compile time. Note that it is unnecessary to include the source code for the expression that failed in the error message; it will be displayed @@ -417,13 +333,12 @@ is determined when the interpreter starts. .. _pass: -The :keyword:`!pass` statement -============================== +The :keyword:`pass` statement +============================= .. index:: statement: pass pair: null; operation - pair: null; operation .. productionlist:: pass_stmt: "pass" @@ -439,11 +354,11 @@ code needs to be executed, for example:: .. _del: -The :keyword:`!del` statement -============================= +The :keyword:`del` statement +============================ .. index:: - ! statement: del + statement: del pair: deletion; target triple: deletion; target; list @@ -459,11 +374,16 @@ Deletion of a target list recursively deletes each target, from left to right. statement: global pair: unbinding; name -Deletion of a name removes the binding of that name from the local or global +Deletion of a name removes the binding of that name from the local or global namespace, depending on whether the name occurs in a :keyword:`global` statement in the same code block. If the name is unbound, a :exc:`NameError` exception will be raised. +.. index:: pair: free; variable + +It is illegal to delete a name from the local namespace if it occurs as a free +variable in a nested block. + .. index:: pair: attribute; deletion Deletion of attribute references, subscriptions and slicings is passed to the @@ -471,18 +391,74 @@ primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object). -.. versionchanged:: 3.2 - Previously it was illegal to delete a name from the local namespace if it - occurs as a free variable in a nested block. + +.. _print: + +The :keyword:`print` statement +============================== + +.. index:: statement: print + +.. productionlist:: + print_stmt: "print" ([`expression` ("," `expression`)* [","]] + : | ">>" `expression` [("," `expression`)+ [","]]) + +:keyword:`print` evaluates each expression in turn and writes the resulting +object to standard output (see below). If an object is not a string, it is +first converted to a string using the rules for string conversions. The +(resulting or original) string is then written. A space is written before each +object is (converted and) written, unless the output system believes it is +positioned at the beginning of a line. This is the case (1) when no characters +have yet been written to standard output, (2) when the last character written to +standard output is a whitespace character except ``' '``, or (3) when the last +write operation on standard output was not a :keyword:`print` statement. +(In some cases it may be functional to write an empty string to standard output +for this reason.) + +.. note:: + + Objects which act like file objects but which are not the built-in file objects + often do not properly emulate this aspect of the file object's behavior, so it + is best not to rely on this. + +.. index:: + single: output + pair: writing; values + pair: trailing; comma + pair: newline; suppression + +A ``'\n'`` character is written at the end, unless the :keyword:`print` +statement ends with a comma. This is the only action if the statement contains +just the keyword :keyword:`print`. + +.. index:: + pair: standard; output + module: sys + single: stdout (in module sys) + exception: RuntimeError + +Standard output is defined as the file object named ``stdout`` in the built-in +module :mod:`sys`. If no such object exists, or if it does not have a +:meth:`write` method, a :exc:`RuntimeError` exception is raised. + +.. index:: single: extended print statement + +:keyword:`print` also has an extended form, defined by the second portion of the +syntax described above. This form is sometimes referred to as ":keyword:`print` +chevron." In this form, the first expression after the ``>>`` must evaluate to a +"file-like" object, specifically an object that has a :meth:`write` method as +described above. With this extended form, the subsequent expressions are +printed to this file object. If the first expression evaluates to ``None``, +then ``sys.stdout`` is used as the file for output. .. _return: -The :keyword:`!return` statement -================================ +The :keyword:`return` statement +=============================== .. index:: - ! statement: return + statement: return pair: function; definition pair: class; definition @@ -500,23 +476,19 @@ If an expression list is present, it is evaluated, else ``None`` is substituted. .. index:: keyword: finally When :keyword:`return` passes control out of a :keyword:`try` statement with a -:keyword:`finally` clause, that :keyword:`!finally` clause is executed before +:keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the function. -In a generator function, the :keyword:`return` statement indicates that the -generator is done and will cause :exc:`StopIteration` to be raised. The returned -value (if any) is used as an argument to construct :exc:`StopIteration` and -becomes the :attr:`StopIteration.value` attribute. +In a generator function, the :keyword:`return` statement is not allowed to +include an :token:`expression_list`. In that context, a bare :keyword:`return` +indicates that the generator is done and will cause :exc:`StopIteration` to be +raised. -In an asynchronous generator function, an empty :keyword:`return` statement -indicates that the asynchronous generator is done and will cause -:exc:`StopAsyncIteration` to be raised. A non-empty :keyword:`!return` -statement is a syntax error in an asynchronous generator function. .. _yield: -The :keyword:`!yield` statement -=============================== +The :keyword:`yield` statement +============================== .. index:: statement: yield @@ -528,137 +500,109 @@ The :keyword:`!yield` statement .. productionlist:: yield_stmt: `yield_expression` -A :keyword:`yield` statement is semantically equivalent to a :ref:`yield -expression <yieldexpr>`. The yield statement can be used to omit the parentheses -that would otherwise be required in the equivalent yield expression -statement. For example, the yield statements :: +The :keyword:`yield` statement is only used when defining a generator function, +and is only used in the body of the generator function. Using a :keyword:`yield` +statement in a function definition is sufficient to cause that definition to +create a generator function instead of a normal function. + +When a generator function is called, it returns an iterator known as a generator +iterator, or more commonly, a generator. The body of the generator function is +executed by calling the generator's :meth:`~generator.next` method repeatedly +until it raises an exception. + +When a :keyword:`yield` statement is executed, the state of the generator is +frozen and the value of :token:`expression_list` is returned to +:meth:`~generator.next`'s caller. By "frozen" we mean that all local state is +retained, including the current bindings of local variables, the instruction +pointer, and the internal evaluation stack: enough information is saved so that +the next time :meth:`~generator.next` is invoked, the function can proceed +exactly as if the :keyword:`yield` statement were just another external call. + +As of Python version 2.5, the :keyword:`yield` statement is now allowed in the +:keyword:`try` clause of a :keyword:`try` ... :keyword:`finally` construct. If +the generator is not resumed before it is finalized (by reaching a zero +reference count or by being garbage collected), the generator-iterator's +:meth:`close` method will be called, allowing any pending :keyword:`finally` +clauses to execute. - yield <expr> - yield from <expr> +For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr` +section. -are equivalent to the yield expression statements :: +.. note:: - (yield <expr>) - (yield from <expr>) + In Python 2.2, the :keyword:`yield` statement was only allowed when the + ``generators`` feature has been enabled. This ``__future__`` + import statement was used to enable the feature:: -Yield expressions and statements are only used when defining a :term:`generator` -function, and are only used in the body of the generator function. Using yield -in a function definition is sufficient to cause that definition to create a -generator function instead of a normal function. + from __future__ import generators + + +.. seealso:: + + :pep:`255` - Simple Generators + The proposal for adding generators and the :keyword:`yield` statement to Python. + + :pep:`342` - Coroutines via Enhanced Generators + The proposal that, among other generator enhancements, proposed allowing + :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block. -For full details of :keyword:`yield` semantics, refer to the -:ref:`yieldexpr` section. .. _raise: -The :keyword:`!raise` statement -=============================== +The :keyword:`raise` statement +============================== .. index:: - ! statement: raise + statement: raise single: exception pair: raising; exception - single: __traceback__ (exception attribute) .. productionlist:: - raise_stmt: "raise" [`expression` ["from" `expression`]] + raise_stmt: "raise" [`expression` ["," `expression` ["," `expression`]]] If no expressions are present, :keyword:`raise` re-raises the last exception that was active in the current scope. If no exception is active in the current -scope, a :exc:`RuntimeError` exception is raised indicating that this is an -error. +scope, a :exc:`TypeError` exception is raised indicating that this is an error +(if running under IDLE, a :exc:`Queue.Empty` exception is raised instead). + +Otherwise, :keyword:`raise` evaluates the expressions to get three objects, +using ``None`` as the value of omitted expressions. The first two objects are +used to determine the *type* and *value* of the exception. -Otherwise, :keyword:`raise` evaluates the first expression as the exception -object. It must be either a subclass or an instance of :class:`BaseException`. -If it is a class, the exception instance will be obtained when needed by -instantiating the class with no arguments. +If the first object is an instance, the type of the exception is the class of +the instance, the instance itself is the value, and the second object must be +``None``. -The :dfn:`type` of the exception is the exception instance's class, the -:dfn:`value` is the instance itself. +If the first object is a class, it becomes the type of the exception. The second +object is used to determine the exception value: If it is an instance of the +class, the instance becomes the exception value. If the second object is a +tuple, it is used as the argument list for the class constructor; if it is +``None``, an empty argument list is used, and any other object is treated as a +single argument to the constructor. The instance so created by calling the +constructor is used as the exception value. .. index:: object: traceback -A traceback object is normally created automatically when an exception is raised -and attached to it as the :attr:`__traceback__` attribute, which is writable. -You can create an exception and set your own traceback in one step using the -:meth:`with_traceback` exception method (which returns the same exception -instance, with its traceback set to its argument), like so:: - - raise Exception("foo occurred").with_traceback(tracebackobj) - -.. index:: pair: exception; chaining - __cause__ (exception attribute) - __context__ (exception attribute) - -The ``from`` clause is used for exception chaining: if given, the second -*expression* must be another exception class or instance, which will then be -attached to the raised exception as the :attr:`__cause__` attribute (which is -writable). If the raised exception is not handled, both exceptions will be -printed:: - - >>> try: - ... print(1 / 0) - ... except Exception as exc: - ... raise RuntimeError("Something bad happened") from exc - ... - Traceback (most recent call last): - File "<stdin>", line 2, in <module> - ZeroDivisionError: division by zero - - The above exception was the direct cause of the following exception: - - Traceback (most recent call last): - File "<stdin>", line 4, in <module> - RuntimeError: Something bad happened - -A similar mechanism works implicitly if an exception is raised inside an -exception handler or a :keyword:`finally` clause: the previous exception is then -attached as the new exception's :attr:`__context__` attribute:: - - >>> try: - ... print(1 / 0) - ... except: - ... raise RuntimeError("Something bad happened") - ... - Traceback (most recent call last): - File "<stdin>", line 2, in <module> - ZeroDivisionError: division by zero - - During handling of the above exception, another exception occurred: - - Traceback (most recent call last): - File "<stdin>", line 4, in <module> - RuntimeError: Something bad happened - -Exception chaining can be explicitly suppressed by specifying :const:`None` in -the ``from`` clause:: - - >>> try: - ... print(1 / 0) - ... except: - ... raise RuntimeError("Something bad happened") from None - ... - Traceback (most recent call last): - File "<stdin>", line 4, in <module> - RuntimeError: Something bad happened +If a third object is present and not ``None``, it must be a traceback object +(see section :ref:`types`), and it is substituted instead of the current +location as the place where the exception occurred. If the third object is +present and not a traceback object or ``None``, a :exc:`TypeError` exception is +raised. The three-expression form of :keyword:`raise` is useful to re-raise an +exception transparently in an except clause, but :keyword:`raise` with no +expressions should be preferred if the exception to be re-raised was the most +recently active exception in the current scope. Additional information on exceptions can be found in section :ref:`exceptions`, and information about handling exceptions is in section :ref:`try`. -.. versionchanged:: 3.3 - :const:`None` is now permitted as ``Y`` in ``raise X from Y``. - -.. versionadded:: 3.3 - The ``__suppress_context__`` attribute to suppress automatic display of the - exception context. .. _break: -The :keyword:`!break` statement -=============================== +The :keyword:`break` statement +============================== .. index:: - ! statement: break + statement: break statement: for statement: while pair: loop; statement @@ -671,28 +615,29 @@ The :keyword:`!break` statement that loop. .. index:: keyword: else - pair: loop control; target -It terminates the nearest enclosing loop, skipping the optional :keyword:`!else` +It terminates the nearest enclosing loop, skipping the optional :keyword:`else` clause if the loop has one. +.. index:: pair: loop control; target + If a :keyword:`for` loop is terminated by :keyword:`break`, the loop control target keeps its current value. .. index:: keyword: finally When :keyword:`break` passes control out of a :keyword:`try` statement with a -:keyword:`finally` clause, that :keyword:`!finally` clause is executed before +:keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the loop. .. _continue: -The :keyword:`!continue` statement -================================== +The :keyword:`continue` statement +================================= .. index:: - ! statement: continue + statement: continue statement: for statement: while pair: loop; statement @@ -702,121 +647,193 @@ The :keyword:`!continue` statement continue_stmt: "continue" :keyword:`continue` may only occur syntactically nested in a :keyword:`for` or -:keyword:`while` loop, but not nested in a function or class definition within -that loop. It continues with the next cycle of the nearest enclosing loop. +:keyword:`while` loop, but not nested in a function or class definition or +:keyword:`finally` clause within that loop. It continues with the next +cycle of the nearest enclosing loop. When :keyword:`continue` passes control out of a :keyword:`try` statement with a -:keyword:`finally` clause, that :keyword:`!finally` clause is executed before +:keyword:`finally` clause, that :keyword:`finally` clause is executed before really starting the next loop cycle. .. _import: .. _from: -The :keyword:`!import` statement -================================ +The :keyword:`import` statement +=============================== .. index:: - ! statement: import + statement: import single: module; importing pair: name; binding keyword: from - keyword: as - exception: ImportError - single: , (comma); import statement + single: as; import statement .. productionlist:: - import_stmt: "import" `module` ["as" `identifier`] ("," `module` ["as" `identifier`])* - : | "from" `relative_module` "import" `identifier` ["as" `identifier`] - : ("," `identifier` ["as" `identifier`])* - : | "from" `relative_module` "import" "(" `identifier` ["as" `identifier`] - : ("," `identifier` ["as" `identifier`])* [","] ")" + import_stmt: "import" `module` ["as" `name`] ( "," `module` ["as" `name`] )* + : | "from" `relative_module` "import" `identifier` ["as" `name`] + : ( "," `identifier` ["as" `name`] )* + : | "from" `relative_module` "import" "(" `identifier` ["as" `name`] + : ( "," `identifier` ["as" `name`] )* [","] ")" : | "from" `module` "import" "*" module: (`identifier` ".")* `identifier` relative_module: "."* `module` | "."+ + name: `identifier` -The basic import statement (no :keyword:`from` clause) is executed in two -steps: +Import statements are executed in two steps: (1) find a module, and initialize +it if necessary; (2) define a name or names in the local namespace (of the scope +where the :keyword:`import` statement occurs). The statement comes in two +forms differing on whether it uses the :keyword:`from` keyword. The first form +(without :keyword:`from`) repeats these steps for each identifier in the list. +The form with :keyword:`from` performs step (1) once, and then performs step +(2) repeatedly. -#. find a module, loading and initializing it if necessary -#. define a name or names in the local namespace for the scope where - the :keyword:`import` statement occurs. - -When the statement contains multiple clauses (separated by -commas) the two steps are carried out separately for each clause, just -as though the clauses had been separated out into individual import -statements. +.. index:: + single: package -The details of the first step, finding and loading modules are described in -greater detail in the section on the :ref:`import system <importsystem>`, -which also describes the various types of packages and modules that can -be imported, as well as all the hooks that can be used to customize -the import system. Note that failures in this step may indicate either -that the module could not be located, *or* that an error occurred while -initializing the module, which includes execution of the module's code. +To understand how step (1) occurs, one must first understand how Python handles +hierarchical naming of modules. To help organize modules and provide a +hierarchy in naming, Python has a concept of packages. A package can contain +other packages and modules while modules cannot contain other modules or +packages. From a file system perspective, packages are directories and modules +are files. -If the requested module is retrieved successfully, it will be made -available in the local namespace in one of three ways: +.. index:: + single: sys.modules -.. index:: single: as; import statement +Once the name of the module is known (unless otherwise specified, the term +"module" will refer to both packages and modules), searching +for the module or package can begin. The first place checked is +:data:`sys.modules`, the cache of all modules that have been imported +previously. If the module is found there then it is used in step (2) of import. -* If the module name is followed by :keyword:`!as`, then the name - following :keyword:`!as` is bound directly to the imported module. -* If no other name is specified, and the module being imported is a top - level module, the module's name is bound in the local namespace as a - reference to the imported module -* If the module being imported is *not* a top level module, then the name - of the top level package that contains the module is bound in the local - namespace as a reference to the top level package. The imported module - must be accessed using its full qualified name rather than directly +.. index:: + single: sys.meta_path + single: finder + pair: finder; find_module + single: __path__ + +If the module is not found in the cache, then :data:`sys.meta_path` is searched +(the specification for :data:`sys.meta_path` can be found in :pep:`302`). +The object is a list of :term:`finder` objects which are queried in order as to +whether they know how to load the module by calling their :meth:`find_module` +method with the name of the module. If the module happens to be contained +within a package (as denoted by the existence of a dot in the name), then a +second argument to :meth:`find_module` is given as the value of the +:attr:`__path__` attribute from the parent package (everything up to the last +dot in the name of the module being imported). If a finder can find the module +it returns a :term:`loader` (discussed later) or returns ``None``. +.. index:: + single: sys.path_hooks + single: sys.path_importer_cache + single: sys.path + +If none of the finders on :data:`sys.meta_path` are able to find the module +then some implicitly defined finders are queried. Implementations of Python +vary in what implicit meta path finders are defined. The one they all do +define, though, is one that handles :data:`sys.path_hooks`, +:data:`sys.path_importer_cache`, and :data:`sys.path`. + +The implicit finder searches for the requested module in the "paths" specified +in one of two places ("paths" do not have to be file system paths). If the +module being imported is supposed to be contained within a package then the +second argument passed to :meth:`find_module`, :attr:`__path__` on the parent +package, is used as the source of paths. If the module is not contained in a +package then :data:`sys.path` is used as the source of paths. + +Once the source of paths is chosen it is iterated over to find a finder that +can handle that path. The dict at :data:`sys.path_importer_cache` caches +finders for paths and is checked for a finder. If the path does not have a +finder cached then :data:`sys.path_hooks` is searched by calling each object in +the list with a single argument of the path, returning a finder or raises +:exc:`ImportError`. If a finder is returned then it is cached in +:data:`sys.path_importer_cache` and then used for that path entry. If no finder +can be found but the path exists then a value of ``None`` is +stored in :data:`sys.path_importer_cache` to signify that an implicit, +file-based finder that handles modules stored as individual files should be +used for that path. If the path does not exist then a finder which always +returns ``None`` is placed in the cache for the path. .. index:: - pair: name; binding - single: from; import statement + single: loader + pair: loader; load_module + exception: ImportError + +If no finder can find the module then :exc:`ImportError` is raised. Otherwise +some finder returned a loader whose :meth:`load_module` method is called with +the name of the module to load (see :pep:`302` for the original definition of +loaders). A loader has several responsibilities to perform on a module it +loads. First, if the module already exists in :data:`sys.modules` (a +possibility if the loader is called outside of the import machinery) then it +is to use that module for initialization and not a new module. But if the +module does not exist in :data:`sys.modules` then it is to be added to that +dict before initialization begins. If an error occurs during loading of the +module and it was added to :data:`sys.modules` it is to be removed from the +dict. If an error occurs but the module was already in :data:`sys.modules` it +is left in the dict. -The :keyword:`from` form uses a slightly more complex process: +.. index:: + single: __name__ + single: __file__ + single: __path__ + single: __package__ + single: __loader__ + +The loader must set several attributes on the module. :data:`__name__` is to be +set to the name of the module. :data:`__file__` is to be the "path" to the file +unless the module is built-in (and thus listed in +:data:`sys.builtin_module_names`) in which case the attribute is not set. +If what is being imported is a package then :data:`__path__` is to be set to a +list of paths to be searched when looking for modules and packages contained +within the package being imported. :data:`__package__` is optional but should +be set to the name of package that contains the module or package (the empty +string is used for module not contained in a package). :data:`__loader__` is +also optional but should be set to the loader object that is loading the +module. -#. find the module specified in the :keyword:`from` clause, loading and - initializing it if necessary; -#. for each of the identifiers specified in the :keyword:`import` clauses: +.. index:: + exception: ImportError - #. check if the imported module has an attribute by that name - #. if not, attempt to import a submodule with that name and then - check the imported module again for that attribute - #. if the attribute is not found, :exc:`ImportError` is raised. - #. otherwise, a reference to that value is stored in the local namespace, - using the name in the :keyword:`!as` clause if it is present, - otherwise using the attribute name +If an error occurs during loading then the loader raises :exc:`ImportError` if +some other exception is not already being propagated. Otherwise the loader +returns the module that was loaded and initialized. -Examples:: +When step (1) finishes without raising an exception, step (2) can begin. - import foo # foo imported and bound locally - import foo.bar.baz # foo.bar.baz imported, foo bound locally - import foo.bar.baz as fbb # foo.bar.baz imported and bound as fbb - from foo.bar import baz # foo.bar.baz imported and bound as baz - from foo import attr # foo imported and foo.attr bound as attr +The first form of :keyword:`import` statement binds the module name in the local +namespace to the module object, and then goes on to import the next identifier, +if any. If the module name is followed by :keyword:`as`, the name following +:keyword:`as` is used as the local name for the module. -.. index:: single: * (asterisk); import statement +.. index:: + pair: name; binding + exception: ImportError -If the list of identifiers is replaced by a star (``'*'``), all public -names defined in the module are bound in the local namespace for the scope -where the :keyword:`import` statement occurs. +The :keyword:`from` form does not bind the module name: it goes through the list +of identifiers, looks each one of them up in the module found in step (1), and +binds the name in the local namespace to the object thus found. As with the +first form of :keyword:`import`, an alternate local name can be supplied by +specifying ":keyword:`as` localname". If a name is not found, +:exc:`ImportError` is raised. If the list of identifiers is replaced by a star +(``'*'``), all public names defined in the module are bound in the local +namespace of the :keyword:`import` statement.. .. index:: single: __all__ (optional module attribute) The *public names* defined by a module are determined by checking the module's -namespace for a variable named ``__all__``; if defined, it must be a sequence -of strings which are names defined or imported by that module. The names -given in ``__all__`` are all considered public and are required to exist. If -``__all__`` is not defined, the set of public names includes all names found -in the module's namespace which do not begin with an underscore character -(``'_'``). ``__all__`` should contain the entire public API. It is intended -to avoid accidentally exporting items that are not part of the API (such as -library modules which were imported and used within the module). - -The wild card form of import --- ``from module import *`` --- is only allowed at -the module level. Attempting to use it in class or function definitions will +namespace for a variable named ``__all__``; if defined, it must be a sequence of +strings which are names defined or imported by that module. The names given in +``__all__`` are all considered public and are required to exist. If ``__all__`` +is not defined, the set of public names includes all names found in the module's +namespace which do not begin with an underscore character (``'_'``). +``__all__`` should contain the entire public API. It is intended to avoid +accidentally exporting items that are not part of the API (such as library +modules which were imported and used within the module). + +The :keyword:`from` form with ``*`` may only occur in a module scope. If the +wild card form of import --- ``import *`` --- is used in a function and the +function contains or is a nested block with free variables, the compiler will raise a :exc:`SyntaxError`. .. index:: @@ -833,38 +850,33 @@ exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. -The specification for relative imports is contained in -the :ref:`relativeimports` section. +The specification for relative imports is contained within :pep:`328`. :func:`importlib.import_module` is provided to support applications that -determine dynamically the modules to be loaded. +determine which modules need to be loaded dynamically. -.. audit-event:: import module,filename,sys.path,sys.meta_path,sys.path_hooks import .. _future: Future statements ----------------- -.. index:: - pair: future; statement - single: __future__; future statement +.. index:: pair: future; statement A :dfn:`future statement` is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a -specified future release of Python where the feature becomes standard. - -The future statement is intended to ease migration to future versions of Python -that introduce incompatible changes to the language. It allows use of the new -features on a per-module basis before the release in which the feature becomes -standard. +specified future release of Python. The future statement is intended to ease +migration to future versions of Python that introduce incompatible changes to +the language. It allows use of the new features on a per-module basis before +the release in which the feature becomes standard. .. productionlist:: * - future_stmt: "from" "__future__" "import" `feature` ["as" `identifier`] - : ("," `feature` ["as" `identifier`])* - : | "from" "__future__" "import" "(" `feature` ["as" `identifier`] - : ("," `feature` ["as" `identifier`])* [","] ")" - feature: `identifier` + future_statement: "from" "__future__" "import" feature ["as" name] + : ("," feature ["as" name])* + : | "from" "__future__" "import" "(" feature ["as" name] + : ("," feature ["as" name])* [","] ")" + feature: identifier + name: identifier A future statement must appear near the top of the module. The only lines that can appear before a future statement are: @@ -874,15 +886,11 @@ can appear before a future statement are: * blank lines, and * other future statements. -The only feature in Python 3.7 that requires using the future statement is -``annotations``. - -All historical features enabled by the future statement are still recognized -by Python 3. The list includes ``absolute_import``, ``division``, -``generators``, ``generator_stop``, ``unicode_literals``, -``print_function``, ``nested_scopes`` and ``with_statement``. They are -all redundant because they are always enabled, and only kept for -backwards compatibility. +The features recognized by Python 2.6 are ``unicode_literals``, +``print_function``, ``absolute_import``, ``division``, ``generators``, +``nested_scopes`` and ``with_statement``. ``generators``, ``with_statement``, +``nested_scopes`` are redundant in Python version 2.6 and above because they are +always enabled. A future statement is recognized and treated specially at compile time: Changes to the semantics of core constructs are often implemented by generating @@ -909,11 +917,12 @@ Note that there is nothing special about the statement:: That is not a future statement; it's an ordinary import statement with no special semantics or syntax restrictions. -Code compiled by calls to the built-in functions :func:`exec` and :func:`compile` -that occur in a module :mod:`M` containing a future statement will, by default, -use the new syntax or semantics associated with the future statement. This can -be controlled by optional arguments to :func:`compile` --- see the documentation -of that function for details. +Code compiled by an :keyword:`exec` statement or calls to the built-in functions +:func:`compile` and :func:`execfile` that occur in a module :mod:`M` containing +a future statement will, by default, use the new syntax or semantics associated +with the future statement. This can, starting with Python 2.2 be controlled by +optional arguments to :func:`compile` --- see the documentation of that function +for details. A future statement typed at an interactive interpreter prompt will take effect for the rest of the interpreter session. If an interpreter is started with the @@ -929,13 +938,12 @@ after the script is executed. .. _global: -The :keyword:`!global` statement -================================ +The :keyword:`global` statement +=============================== .. index:: - ! statement: global + statement: global triple: global; name; binding - single: , (comma); identifier list .. productionlist:: global_stmt: "global" `identifier` ("," `identifier`)* @@ -943,71 +951,102 @@ The :keyword:`!global` statement The :keyword:`global` statement is a declaration which holds for the entire current code block. It means that the listed identifiers are to be interpreted as globals. It would be impossible to assign to a global variable without -:keyword:`!global`, although free variables may refer to globals without being +:keyword:`global`, although free variables may refer to globals without being declared global. Names listed in a :keyword:`global` statement must not be used in the same code -block textually preceding that :keyword:`!global` statement. +block textually preceding that :keyword:`global` statement. Names listed in a :keyword:`global` statement must not be defined as formal parameters or in a :keyword:`for` loop control target, :keyword:`class` -definition, function definition, :keyword:`import` statement, or variable -annotation. +definition, function definition, or :keyword:`import` statement. .. impl-detail:: - The current implementation does not enforce some of these restrictions, but + The current implementation does not enforce the latter two restrictions, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program. .. index:: - builtin: exec + statement: exec builtin: eval + builtin: execfile builtin: compile **Programmer's note:** :keyword:`global` is a directive to the parser. It -applies only to code parsed at the same time as the :keyword:`!global` statement. -In particular, a :keyword:`!global` statement contained in a string or code -object supplied to the built-in :func:`exec` function does not affect the code -block *containing* the function call, and code contained in such a string is -unaffected by :keyword:`!global` statements in the code containing the function -call. The same applies to the :func:`eval` and :func:`compile` functions. +applies only to code parsed at the same time as the :keyword:`global` statement. +In particular, a :keyword:`global` statement contained in an :keyword:`exec` +statement does not affect the code block *containing* the :keyword:`exec` +statement, and code contained in an :keyword:`exec` statement is unaffected by +:keyword:`global` statements in the code containing the :keyword:`exec` +statement. The same applies to the :func:`eval`, :func:`execfile` and +:func:`compile` functions. -.. _nonlocal: +.. _exec: -The :keyword:`!nonlocal` statement -================================== +The :keyword:`exec` statement +============================= -.. index:: statement: nonlocal - single: , (comma); identifier list +.. index:: statement: exec .. productionlist:: - nonlocal_stmt: "nonlocal" `identifier` ("," `identifier`)* + exec_stmt: "exec" `or_expr` ["in" `expression` ["," `expression`]] + +This statement supports dynamic execution of Python code. The first expression +should evaluate to either a Unicode string, a *Latin-1* encoded string, an open +file object, a code object, or a tuple. If it is a string, the string is parsed +as a suite of Python statements which is then executed (unless a syntax error +occurs). [#]_ If it is an open file, the file is parsed until EOF and executed. +If it is a code object, it is simply executed. For the interpretation of a +tuple, see below. In all cases, the code that's executed is expected to be +valid as file input (see section :ref:`file-input`). Be aware that the +:keyword:`return` and :keyword:`yield` statements may not be used outside of +function definitions even within the context of code passed to the +:keyword:`exec` statement. + +In all cases, if the optional parts are omitted, the code is executed in the +current scope. If only the first expression after ``in`` is specified, +it should be a dictionary, which will be used for both the global and the local +variables. If two expressions are given, they are used for the global and local +variables, respectively. If provided, *locals* can be any mapping object. +Remember that at module level, globals and locals are the same dictionary. If +two separate objects are given as *globals* and *locals*, the code will be +executed as if it were embedded in a class definition. + +The first expression may also be a tuple of length 2 or 3. In this case, the +optional parts must be omitted. The form ``exec(expr, globals)`` is equivalent +to ``exec expr in globals``, while the form ``exec(expr, globals, locals)`` is +equivalent to ``exec expr in globals, locals``. The tuple form of ``exec`` +provides compatibility with Python 3, where ``exec`` is a function rather than +a statement. + +.. versionchanged:: 2.4 + Formerly, *locals* was required to be a dictionary. -.. XXX add when implemented - : ["=" (`target_list` "=")+ starred_expression] - : | "nonlocal" identifier augop expression_list +.. index:: + single: __builtins__ + module: __builtin__ -The :keyword:`nonlocal` statement causes the listed identifiers to refer to -previously bound variables in the nearest enclosing scope excluding globals. -This is important because the default behavior for binding is to search the -local namespace first. The statement allows encapsulated code to rebind -variables outside of the local scope besides the global (module) scope. +As a side effect, an implementation may insert additional keys into the +dictionaries given besides those corresponding to variable names set by the +executed code. For example, the current implementation may add a reference to +the dictionary of the built-in module :mod:`__builtin__` under the key +``__builtins__`` (!). -.. XXX not implemented - The :keyword:`nonlocal` statement may prepend an assignment or augmented - assignment, but not an expression. +.. index:: + builtin: eval + builtin: globals + builtin: locals -Names listed in a :keyword:`nonlocal` statement, unlike those listed in a -:keyword:`global` statement, must refer to pre-existing bindings in an -enclosing scope (the scope in which a new binding should be created cannot -be determined unambiguously). +**Programmer's hints:** dynamic evaluation of expressions is supported by the +built-in function :func:`eval`. The built-in functions :func:`globals` and +:func:`locals` return the current global and local dictionary, respectively, +which may be useful to pass around for use by :keyword:`exec`. -Names listed in a :keyword:`nonlocal` statement must not collide with -pre-existing bindings in the local scope. -.. seealso:: +.. rubric:: Footnotes - :pep:`3104` - Access to Names in Outer Scopes - The specification for the :keyword:`nonlocal` statement. +.. [#] Note that the parser only accepts the Unix-style end of line convention. + If you are reading the code from a file, make sure to use + :term:`universal newlines` mode to convert Windows or Mac-style newlines. diff --git a/Doc/reference/toplevel_components.rst b/Doc/reference/toplevel_components.rst index d5ffb37..44196f7 100644 --- a/Doc/reference/toplevel_components.rst +++ b/Doc/reference/toplevel_components.rst @@ -23,13 +23,13 @@ Complete Python programs .. index:: module: sys module: __main__ - module: builtins + module: __builtin__ While a language specification need not prescribe how the language interpreter is invoked, it is useful to have a notion of a complete Python program. A complete Python program is executed in a minimally initialized environment: all built-in and standard modules are available, but none have been initialized, -except for :mod:`sys` (various system services), :mod:`builtins` (built-in +except for :mod:`sys` (various system services), :mod:`__builtin__` (built-in functions, exceptions and ``None``) and :mod:`__main__`. The latter is used to provide the local and global namespace for execution of the complete program. @@ -48,13 +48,12 @@ a complete program; each statement is executed in the namespace of .. index:: single: UNIX - single: Windows single: command line single: standard input A complete program can be passed to the interpreter in three forms: with the :option:`-c` *string* command line option, as a file -passed as the first command line argument, or as standard input. If the file +passed as the first command line argument, or as standard input. If the file or standard input is a tty device, the interpreter enters interactive mode; otherwise, it executes the file as a complete program. @@ -75,7 +74,7 @@ This syntax is used in the following situations: * when parsing a module; -* when parsing a string passed to the :func:`exec` function; +* when parsing a string passed to the :keyword:`exec` statement; .. _interactive: @@ -98,10 +97,29 @@ Expression input ================ .. index:: single: input + .. index:: builtin: eval -:func:`eval` is used for expression input. It ignores leading whitespace. The +There are two forms of expression input. Both ignore leading whitespace. The string argument to :func:`eval` must have the following form: .. productionlist:: eval_input: `expression_list` NEWLINE* + +.. index:: builtin: input + +The input line read by :func:`input` must have the following form: + +.. productionlist:: + input_input: `expression_list` NEWLINE + +.. index:: + object: file + single: input; raw + single: raw input + builtin: raw_input + single: readline() (file method) + +Note: to read 'raw' input line without interpretation, you can use the built-in +function :func:`raw_input` or the :meth:`readline` method of file objects. + |