diff options
33 files changed, 801 insertions, 350 deletions
diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index cb429c8..5ba66cb 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -431,6 +431,13 @@ the stack so that it is available for further iterations of the loop. Pops ``TOS`` and yields it from a :term:`generator`. +.. opcode:: YIELD_FROM + + Pops ``TOS`` and delegates to it as a subiterator from a :term:`generator`. + + .. versionadded:: 3.3 + + .. opcode:: IMPORT_STAR Loads all symbols not starting with ``'_'`` directly from the module TOS to the diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 49cec1e..a9a16d3 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -250,7 +250,16 @@ The following exceptions are the exceptions that are usually raised. .. exception:: StopIteration Raised by built-in function :func:`next` and an :term:`iterator`\'s - :meth:`__next__` method to signal that there are no further values. + :meth:`__next__` method to signal that there are no further items to be + produced by the iterator. + + The exception object has a single attribute :attr:`value`, which is + given as an argument when constructing the exception, and defaults + to :const:`None`. + + When a generator function returns, a new :exc:`StopIteration` instance is + raised, and the value returned by the function is used as the + :attr:`value` parameter to the constructor of the exception. .. exception:: SyntaxError diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 655ebde..7da54a2 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -318,7 +318,7 @@ Yield expressions .. productionlist:: yield_atom: "(" `yield_expression` ")" - yield_expression: "yield" [`expression_list`] + yield_expression: "yield" [`expression_list` | "from" `expression`] The :keyword:`yield` expression is only used when defining a generator function, and can only be used in the body of a function definition. Using a @@ -336,7 +336,10 @@ the internal evaluation stack. When the execution is resumed by calling one of the generator's methods, the function can proceed exactly as if the :keyword:`yield` expression was just another external call. The value of the :keyword:`yield` expression after resuming depends on the method which resumed -the execution. +the execution. If :meth:`__next__` is used (typically via either a +:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`, +otherwise, if :meth:`send` is used, then the result will be the value passed +in to that method. .. index:: single: coroutine @@ -346,12 +349,29 @@ suspended. The only difference is that a generator function cannot control where should the execution continue after it yields; the control is always transferred to the generator's caller. -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a +:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a :keyword:`try` ... :keyword:`finally` construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`close` method will be called, allowing any pending :keyword:`finally` clauses to execute. +When ``yield from expression`` is used, it treats the supplied expression as +a subiterator. All values produced by that subiterator are passed directly +to the caller of the current generator's methods. Any values passed in with +:meth:`send` and any exceptions passed in with :meth:`throw` are passed to +the underlying iterator if it has the appropriate methods. If this is not the +case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`, +while :meth:`throw` will just raise the passed in exception immediately. + +When the underlying iterator is complete, the :attr:`~StopIteration.value` +attribute of the raised :exc:`StopIteration` instance becomes the value of +the yield expression. It can be either set explicitly when raising +:exc:`StopIteration`, or automatically when the sub-iterator is a generator +(by returning a value from the sub-generator). + +The parentheses can be omitted when the :keyword:`yield` expression is the +sole expression on the right hand side of an assignment statement. + .. index:: object: generator The following generator's methods can be used to control the execution of a @@ -444,6 +464,10 @@ generator functions:: The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. + :pep:`0380` - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:`yield_from` syntax, making delegation + to sub-generators easy. + .. _primaries: diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 34ed92f..d98b829 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -425,10 +425,10 @@ When :keyword:`return` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the function. -In a generator function, the :keyword:`return` statement is not allowed to -include an :token:`expression_list`. In that context, a bare :keyword:`return` -indicates that the generator is done and will cause :exc:`StopIteration` to be -raised. +In a generator function, the :keyword:`return` statement indicates that the +generator is done and will cause :exc:`StopIteration` to be raised. The returned +value (if any) is used as an argument to construct :exc:`StopIteration` and +becomes the :attr:`StopIteration.value` attribute. .. _yield: @@ -450,6 +450,7 @@ The :keyword:`yield` statement is only used when defining a generator function, and is only used in the body of the generator function. Using a :keyword:`yield` statement in a function definition is sufficient to cause that definition to create a generator function instead of a normal function. + When a generator function is called, it returns an iterator known as a generator iterator, or more commonly, a generator. The body of the generator function is executed by calling the :func:`next` function on the generator repeatedly until @@ -469,14 +470,25 @@ resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`close` method will be called, allowing any pending :keyword:`finally` clauses to execute. +When ``yield from expression`` is used, it treats the supplied expression as +a subiterator, producing values from it until the underlying iterator is +exhausted. + +For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr` +section. + .. seealso:: :pep:`0255` - Simple Generators The proposal for adding generators and the :keyword:`yield` statement to Python. :pep:`0342` - Coroutines via Enhanced Generators - The proposal that, among other generator enhancements, proposed allowing - :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block. + The proposal to enhance the API and syntax of generators, making them + usable as simple coroutines. + + :pep:`0380` - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:`yield_from` syntax, making delegation + to sub-generators easy. .. _raise: diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst index 0b0f8f6..07257f0 100644 --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -195,6 +195,22 @@ inspection of exception attributes:: print("You are not allowed to read document.txt") +PEP 380: Syntax for Delegating to a Subgenerator +================================================ + +PEP 380 adds the ``yield from`` expression, allowing a generator to delegate +part of its operations to another generator. This allows a section of code +containing 'yield' to be factored out and placed in another generator. +Additionally, the subgenerator is allowed to return with a value, and the +value is made available to the delegating generator. +While designed primarily for use in delegating to a subgenerator, the ``yield +from`` expression actually allows delegation to arbitrary subiterators. + +(Implementation by Greg Ewing, integrated into 3.3 by Renaud Blanch, Ryan +Kelly and Nick Coghlan, documentation by Zbigniew Jędrzejewski-Szmek and +Nick Coghlan) + + PEP 3155: Qualified name for classes and functions ================================================== @@ -208,7 +224,6 @@ it provides better information about where they were actually defined, and how they might be accessible from the global scope. Example with (non-bound) methods:: - >>> class C: ... def meth(self): ... pass diff --git a/Grammar/Grammar b/Grammar/Grammar index 544852c..38320ef 100644 --- a/Grammar/Grammar +++ b/Grammar/Grammar @@ -121,7 +121,7 @@ arglist: (argument ',')* (argument [','] |'**' test) # The reason that keywords are test nodes instead of NAME is that using NAME # results in an ambiguity. ast.c makes sure it's a NAME. -argument: test [comp_for] | test '=' test # Really [keyword '='] test +argument: (test) [comp_for] | test '=' test # Really [keyword '='] test comp_iter: comp_for | comp_if comp_for: 'for' exprlist 'in' or_test [comp_iter] comp_if: 'if' test_nocond [comp_iter] @@ -129,4 +129,5 @@ comp_if: 'if' test_nocond [comp_iter] # not used in grammar, but may appear in "node" passed from Parser to Compiler encoding_decl: NAME -yield_expr: 'yield' [testlist] +yield_expr: 'yield' [yield_arg] +yield_arg: 'from' test | testlist diff --git a/Include/Python-ast.h b/Include/Python-ast.h index 9389049..4e21674e 100644 --- a/Include/Python-ast.h +++ b/Include/Python-ast.h @@ -245,6 +245,7 @@ struct _expr { } GeneratorExp; struct { + int is_from; expr_ty value; } Yield; @@ -487,8 +488,9 @@ expr_ty _Py_DictComp(expr_ty key, expr_ty value, asdl_seq * generators, int #define GeneratorExp(a0, a1, a2, a3, a4) _Py_GeneratorExp(a0, a1, a2, a3, a4) expr_ty _Py_GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, PyArena *arena); -#define Yield(a0, a1, a2, a3) _Py_Yield(a0, a1, a2, a3) -expr_ty _Py_Yield(expr_ty value, int lineno, int col_offset, PyArena *arena); +#define Yield(a0, a1, a2, a3, a4) _Py_Yield(a0, a1, a2, a3, a4) +expr_ty _Py_Yield(int is_from, expr_ty value, int lineno, int col_offset, + PyArena *arena); #define Compare(a0, a1, a2, a3, a4, a5) _Py_Compare(a0, a1, a2, a3, a4, a5) expr_ty _Py_Compare(expr_ty left, asdl_int_seq * ops, asdl_seq * comparators, int lineno, int col_offset, PyArena *arena); diff --git a/Include/frameobject.h b/Include/frameobject.h index 1fb64bb..55447b7 100644 --- a/Include/frameobject.h +++ b/Include/frameobject.h @@ -9,45 +9,46 @@ extern "C" { #endif typedef struct { - int b_type; /* what kind of block this is */ - int b_handler; /* where to jump to find handler */ - int b_level; /* value stack level to pop to */ + int b_type; /* what kind of block this is */ + int b_handler; /* where to jump to find handler */ + int b_level; /* value stack level to pop to */ } PyTryBlock; typedef struct _frame { PyObject_VAR_HEAD - struct _frame *f_back; /* previous frame, or NULL */ - PyCodeObject *f_code; /* code segment */ - PyObject *f_builtins; /* builtin symbol table (PyDictObject) */ - PyObject *f_globals; /* global symbol table (PyDictObject) */ - PyObject *f_locals; /* local symbol table (any mapping) */ - PyObject **f_valuestack; /* points after the last local */ + struct _frame *f_back; /* previous frame, or NULL */ + PyCodeObject *f_code; /* code segment */ + PyObject *f_builtins; /* builtin symbol table (PyDictObject) */ + PyObject *f_globals; /* global symbol table (PyDictObject) */ + PyObject *f_locals; /* local symbol table (any mapping) */ + PyObject **f_valuestack; /* points after the last local */ /* Next free slot in f_valuestack. Frame creation sets to f_valuestack. Frame evaluation usually NULLs it, but a frame that yields sets it to the current stack top. */ PyObject **f_stacktop; - PyObject *f_trace; /* Trace function */ - - /* In a generator, we need to be able to swap between the exception - state inside the generator and the exception state of the calling - frame (which shouldn't be impacted when the generator "yields" - from an except handler). - These three fields exist exactly for that, and are unused for - non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE - macros in ceval.c for details of their use. */ + PyObject *f_trace; /* Trace function */ + PyObject *f_yieldfrom; /* Iterator being delegated to by yield from */ + + /* In a generator, we need to be able to swap between the exception + state inside the generator and the exception state of the calling + frame (which shouldn't be impacted when the generator "yields" + from an except handler). + These three fields exist exactly for that, and are unused for + non-generator frames. See the SAVE_EXC_STATE and SWAP_EXC_STATE + macros in ceval.c for details of their use. */ PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; - int f_lasti; /* Last instruction if called */ + int f_lasti; /* Last instruction if called */ /* Call PyFrame_GetLineNumber() instead of reading this field directly. As of 2.3 f_lineno is only valid when tracing is active (i.e. when f_trace is set). At other times we use PyCode_Addr2Line to calculate the line from the current bytecode index. */ - int f_lineno; /* Current line number */ - int f_iblock; /* index in f_blockstack */ + int f_lineno; /* Current line number */ + int f_iblock; /* index in f_blockstack */ PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */ - PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */ + PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */ } PyFrameObject; diff --git a/Include/genobject.h b/Include/genobject.h index d29fb1e..13a3856 100644 --- a/Include/genobject.h +++ b/Include/genobject.h @@ -11,20 +11,20 @@ extern "C" { struct _frame; /* Avoid including frameobject.h */ typedef struct { - PyObject_HEAD - /* The gi_ prefix is intended to remind of generator-iterator. */ + PyObject_HEAD + /* The gi_ prefix is intended to remind of generator-iterator. */ - /* Note: gi_frame can be NULL if the generator is "finished" */ - struct _frame *gi_frame; + /* Note: gi_frame can be NULL if the generator is "finished" */ + struct _frame *gi_frame; - /* True if generator is being executed. */ - int gi_running; + /* True if generator is being executed. */ + int gi_running; - /* The code object backing the generator */ - PyObject *gi_code; + /* The code object backing the generator */ + PyObject *gi_code; - /* List of weak reference. */ - PyObject *gi_weakreflist; + /* List of weak reference. */ + PyObject *gi_weakreflist; } PyGenObject; PyAPI_DATA(PyTypeObject) PyGen_Type; @@ -34,6 +34,7 @@ PyAPI_DATA(PyTypeObject) PyGen_Type; PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *); PyAPI_FUNC(int) PyGen_NeedsFinalizing(PyGenObject *); +PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **); #ifdef __cplusplus } diff --git a/Include/graminit.h b/Include/graminit.h index e0e27f9..3ec949a 100644 --- a/Include/graminit.h +++ b/Include/graminit.h @@ -81,3 +81,4 @@ #define comp_if 334 #define encoding_decl 335 #define yield_expr 336 +#define yield_arg 337 diff --git a/Include/opcode.h b/Include/opcode.h index ece713e..a90184d 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -7,116 +7,117 @@ extern "C" { /* Instruction opcodes for compiled code */ -#define POP_TOP 1 -#define ROT_TWO 2 -#define ROT_THREE 3 -#define DUP_TOP 4 +#define POP_TOP 1 +#define ROT_TWO 2 +#define ROT_THREE 3 +#define DUP_TOP 4 #define DUP_TOP_TWO 5 -#define NOP 9 +#define NOP 9 -#define UNARY_POSITIVE 10 -#define UNARY_NEGATIVE 11 -#define UNARY_NOT 12 +#define UNARY_POSITIVE 10 +#define UNARY_NEGATIVE 11 +#define UNARY_NOT 12 -#define UNARY_INVERT 15 +#define UNARY_INVERT 15 -#define BINARY_POWER 19 +#define BINARY_POWER 19 -#define BINARY_MULTIPLY 20 +#define BINARY_MULTIPLY 20 -#define BINARY_MODULO 22 -#define BINARY_ADD 23 -#define BINARY_SUBTRACT 24 -#define BINARY_SUBSCR 25 +#define BINARY_MODULO 22 +#define BINARY_ADD 23 +#define BINARY_SUBTRACT 24 +#define BINARY_SUBSCR 25 #define BINARY_FLOOR_DIVIDE 26 #define BINARY_TRUE_DIVIDE 27 #define INPLACE_FLOOR_DIVIDE 28 #define INPLACE_TRUE_DIVIDE 29 -#define STORE_MAP 54 -#define INPLACE_ADD 55 -#define INPLACE_SUBTRACT 56 -#define INPLACE_MULTIPLY 57 - -#define INPLACE_MODULO 59 -#define STORE_SUBSCR 60 -#define DELETE_SUBSCR 61 - -#define BINARY_LSHIFT 62 -#define BINARY_RSHIFT 63 -#define BINARY_AND 64 -#define BINARY_XOR 65 -#define BINARY_OR 66 -#define INPLACE_POWER 67 -#define GET_ITER 68 -#define STORE_LOCALS 69 -#define PRINT_EXPR 70 +#define STORE_MAP 54 +#define INPLACE_ADD 55 +#define INPLACE_SUBTRACT 56 +#define INPLACE_MULTIPLY 57 + +#define INPLACE_MODULO 59 +#define STORE_SUBSCR 60 +#define DELETE_SUBSCR 61 + +#define BINARY_LSHIFT 62 +#define BINARY_RSHIFT 63 +#define BINARY_AND 64 +#define BINARY_XOR 65 +#define BINARY_OR 66 +#define INPLACE_POWER 67 +#define GET_ITER 68 +#define STORE_LOCALS 69 +#define PRINT_EXPR 70 #define LOAD_BUILD_CLASS 71 - -#define INPLACE_LSHIFT 75 -#define INPLACE_RSHIFT 76 -#define INPLACE_AND 77 -#define INPLACE_XOR 78 -#define INPLACE_OR 79 -#define BREAK_LOOP 80 +#define YIELD_FROM 72 + +#define INPLACE_LSHIFT 75 +#define INPLACE_RSHIFT 76 +#define INPLACE_AND 77 +#define INPLACE_XOR 78 +#define INPLACE_OR 79 +#define BREAK_LOOP 80 #define WITH_CLEANUP 81 -#define RETURN_VALUE 83 -#define IMPORT_STAR 84 +#define RETURN_VALUE 83 +#define IMPORT_STAR 84 -#define YIELD_VALUE 86 -#define POP_BLOCK 87 -#define END_FINALLY 88 -#define POP_EXCEPT 89 +#define YIELD_VALUE 86 +#define POP_BLOCK 87 +#define END_FINALLY 88 +#define POP_EXCEPT 89 -#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */ +#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */ -#define STORE_NAME 90 /* Index in name list */ -#define DELETE_NAME 91 /* "" */ -#define UNPACK_SEQUENCE 92 /* Number of sequence items */ -#define FOR_ITER 93 +#define STORE_NAME 90 /* Index in name list */ +#define DELETE_NAME 91 /* "" */ +#define UNPACK_SEQUENCE 92 /* Number of sequence items */ +#define FOR_ITER 93 #define UNPACK_EX 94 /* Num items before variable part + (Num items after variable part << 8) */ -#define STORE_ATTR 95 /* Index in name list */ -#define DELETE_ATTR 96 /* "" */ -#define STORE_GLOBAL 97 /* "" */ -#define DELETE_GLOBAL 98 /* "" */ - -#define LOAD_CONST 100 /* Index in const list */ -#define LOAD_NAME 101 /* Index in name list */ -#define BUILD_TUPLE 102 /* Number of tuple items */ -#define BUILD_LIST 103 /* Number of list items */ -#define BUILD_SET 104 /* Number of set items */ -#define BUILD_MAP 105 /* Always zero for now */ -#define LOAD_ATTR 106 /* Index in name list */ -#define COMPARE_OP 107 /* Comparison operator */ -#define IMPORT_NAME 108 /* Index in name list */ -#define IMPORT_FROM 109 /* Index in name list */ - -#define JUMP_FORWARD 110 /* Number of bytes to skip */ -#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */ -#define JUMP_IF_TRUE_OR_POP 112 /* "" */ -#define JUMP_ABSOLUTE 113 /* "" */ -#define POP_JUMP_IF_FALSE 114 /* "" */ -#define POP_JUMP_IF_TRUE 115 /* "" */ - -#define LOAD_GLOBAL 116 /* Index in name list */ - -#define CONTINUE_LOOP 119 /* Start of loop (absolute) */ -#define SETUP_LOOP 120 /* Target address (relative) */ -#define SETUP_EXCEPT 121 /* "" */ -#define SETUP_FINALLY 122 /* "" */ - -#define LOAD_FAST 124 /* Local variable number */ -#define STORE_FAST 125 /* Local variable number */ -#define DELETE_FAST 126 /* Local variable number */ - -#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */ +#define STORE_ATTR 95 /* Index in name list */ +#define DELETE_ATTR 96 /* "" */ +#define STORE_GLOBAL 97 /* "" */ +#define DELETE_GLOBAL 98 /* "" */ + +#define LOAD_CONST 100 /* Index in const list */ +#define LOAD_NAME 101 /* Index in name list */ +#define BUILD_TUPLE 102 /* Number of tuple items */ +#define BUILD_LIST 103 /* Number of list items */ +#define BUILD_SET 104 /* Number of set items */ +#define BUILD_MAP 105 /* Always zero for now */ +#define LOAD_ATTR 106 /* Index in name list */ +#define COMPARE_OP 107 /* Comparison operator */ +#define IMPORT_NAME 108 /* Index in name list */ +#define IMPORT_FROM 109 /* Index in name list */ + +#define JUMP_FORWARD 110 /* Number of bytes to skip */ +#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */ +#define JUMP_IF_TRUE_OR_POP 112 /* "" */ +#define JUMP_ABSOLUTE 113 /* "" */ +#define POP_JUMP_IF_FALSE 114 /* "" */ +#define POP_JUMP_IF_TRUE 115 /* "" */ + +#define LOAD_GLOBAL 116 /* Index in name list */ + +#define CONTINUE_LOOP 119 /* Start of loop (absolute) */ +#define SETUP_LOOP 120 /* Target address (relative) */ +#define SETUP_EXCEPT 121 /* "" */ +#define SETUP_FINALLY 122 /* "" */ + +#define LOAD_FAST 124 /* Local variable number */ +#define STORE_FAST 125 /* Local variable number */ +#define DELETE_FAST 126 /* Local variable number */ + +#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */ /* CALL_FUNCTION_XXX opcodes defined below depend on this definition */ -#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */ -#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ -#define BUILD_SLICE 133 /* Number of items */ +#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */ +#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ +#define BUILD_SLICE 133 /* Number of items */ #define MAKE_CLOSURE 134 /* same as MAKE_FUNCTION */ #define LOAD_CLOSURE 135 /* Load free variable from closure */ @@ -126,9 +127,9 @@ extern "C" { /* The next 3 opcodes must be contiguous and satisfy (CALL_FUNCTION_VAR - CALL_FUNCTION) & 3 == 1 */ -#define CALL_FUNCTION_VAR 140 /* #args + (#kwargs<<8) */ -#define CALL_FUNCTION_KW 141 /* #args + (#kwargs<<8) */ -#define CALL_FUNCTION_VAR_KW 142 /* #args + (#kwargs<<8) */ +#define CALL_FUNCTION_VAR 140 /* #args + (#kwargs<<8) */ +#define CALL_FUNCTION_KW 141 /* #args + (#kwargs<<8) */ +#define CALL_FUNCTION_VAR_KW 142 /* #args + (#kwargs<<8) */ #define SETUP_WITH 143 @@ -148,7 +149,7 @@ extern "C" { enum cmp_op {PyCmp_LT=Py_LT, PyCmp_LE=Py_LE, PyCmp_EQ=Py_EQ, PyCmp_NE=Py_NE, PyCmp_GT=Py_GT, PyCmp_GE=Py_GE, - PyCmp_IN, PyCmp_NOT_IN, PyCmp_IS, PyCmp_IS_NOT, PyCmp_EXC_MATCH, PyCmp_BAD}; + PyCmp_IN, PyCmp_NOT_IN, PyCmp_IS, PyCmp_IS_NOT, PyCmp_EXC_MATCH, PyCmp_BAD}; #define HAS_ARG(op) ((op) >= HAVE_ARGUMENT) diff --git a/Include/pyerrors.h b/Include/pyerrors.h index 44eb3d9..1bd0442 100644 --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -51,6 +51,11 @@ typedef struct { Py_ssize_t written; /* only for BlockingIOError, -1 otherwise */ } PyOSErrorObject; +typedef struct { + PyException_HEAD + PyObject *value; +} PyStopIterationObject; + /* Compatibility typedefs */ typedef PyOSErrorObject PyEnvironmentErrorObject; #ifdef MS_WINDOWS @@ -380,6 +385,8 @@ PyAPI_FUNC(int) PyUnicodeTranslateError_SetReason( const char *reason /* UTF-8 encoded string */ ); +/* create a StopIteration exception with the given value */ +PyAPI_FUNC(PyObject *) PyStopIteration_Create(PyObject *); /* These APIs aren't really part of the error implementation, but often needed to format error messages; the native C lib APIs are diff --git a/Lib/opcode.py b/Lib/opcode.py index b631b25..6fe20c7 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -87,6 +87,7 @@ def_op('STORE_LOCALS', 69) def_op('PRINT_EXPR', 70) def_op('LOAD_BUILD_CLASS', 71) +def_op('YIELD_FROM', 72) def_op('INPLACE_LSHIFT', 75) def_op('INPLACE_RSHIFT', 76) diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index b1656bc..2022ee2 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -813,7 +813,8 @@ class ASTValidatorTests(unittest.TestCase): self._check_comprehension(factory) def test_yield(self): - self.expr(ast.Yield(ast.Name("x", ast.Store())), "must have Load") + self.expr(ast.Yield(0, ast.Name("x", ast.Store())), "must have Load") + self.expr(ast.Yield(1, ast.Name("x", ast.Store())), "must have Load") def test_compare(self): left = ast.Name("x", ast.Load()) diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index 5f47b3e..06f67c2 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -728,29 +728,6 @@ Ye olde Fibonacci generator, tee style. syntax_tests = """ ->>> def f(): -... return 22 -... yield 1 -Traceback (most recent call last): - .. -SyntaxError: 'return' with argument inside generator - ->>> def f(): -... yield 1 -... return 22 -Traceback (most recent call last): - .. -SyntaxError: 'return' with argument inside generator - -"return None" is not the same as "return" in a generator: - ->>> def f(): -... yield 1 -... return None -Traceback (most recent call last): - .. -SyntaxError: 'return' with argument inside generator - These are fine: >>> def f(): @@ -866,20 +843,6 @@ These are fine: >>> type(f()) <class 'generator'> - ->>> def f(): -... if 0: -... lambda x: x # shouldn't trigger here -... return # or here -... def f(i): -... return 2*i # or here -... if 0: -... return 3 # but *this* sucks (line 8) -... if 0: -... yield 2 # because it's a generator (line 10) -Traceback (most recent call last): -SyntaxError: 'return' with argument inside generator - This one caused a crash (see SF bug 567538): >>> def f(): @@ -1566,11 +1529,6 @@ Traceback (most recent call last): ... SyntaxError: 'yield' outside function ->>> def f(): return lambda x=(yield): 1 -Traceback (most recent call last): - ... -SyntaxError: 'return' with argument inside generator - >>> def f(): x = yield = y Traceback (most recent call last): ... diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index eff763e..6b326bd 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -458,7 +458,39 @@ class GrammarTests(unittest.TestCase): check_syntax_error(self, "class foo:return 1") def test_yield(self): + # Allowed as standalone statement + def g(): yield 1 + def g(): yield from () + # Allowed as RHS of assignment + def g(): x = yield 1 + def g(): x = yield from () + # Ordinary yield accepts implicit tuples + def g(): yield 1, 1 + def g(): x = yield 1, 1 + # 'yield from' does not + check_syntax_error(self, "def g(): yield from (), 1") + check_syntax_error(self, "def g(): x = yield from (), 1") + # Requires parentheses as subexpression + def g(): 1, (yield 1) + def g(): 1, (yield from ()) + check_syntax_error(self, "def g(): 1, yield 1") + check_syntax_error(self, "def g(): 1, yield from ()") + # Requires parentheses as call argument + def g(): f((yield 1)) + def g(): f((yield 1), 1) + def g(): f((yield from ())) + def g(): f((yield from ()), 1) + check_syntax_error(self, "def g(): f(yield 1)") + check_syntax_error(self, "def g(): f(yield 1, 1)") + check_syntax_error(self, "def g(): f(yield from ())") + check_syntax_error(self, "def g(): f(yield from (), 1)") + # Not allowed at top level + check_syntax_error(self, "yield") + check_syntax_error(self, "yield from") + # Not allowed at class scope check_syntax_error(self, "class foo:yield 1") + check_syntax_error(self, "class foo:yield from ()") + def test_raise(self): # 'raise' test [',' test] diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py index 2b50fca..b6f81fb 100644 --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -50,6 +50,10 @@ class RoundtripLegalSyntaxTestCase(unittest.TestCase): self.check_suite("def f(): (yield 1)*2") self.check_suite("def f(): return; yield 1") self.check_suite("def f(): yield 1; return") + self.check_suite("def f(): yield from 1") + self.check_suite("def f(): x = yield from 1") + self.check_suite("def f(): f((yield from 1))") + self.check_suite("def f(): yield 1; return 1") self.check_suite("def f():\n" " for x in range(30):\n" " yield x\n") @@ -621,7 +625,6 @@ class OtherParserCase(unittest.TestCase): with self.assertRaises(TypeError): parser.expr("a", "b") - def test_main(): support.run_unittest( RoundtripLegalSyntaxTestCase, diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index f89514f..6dc0e42 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -727,7 +727,7 @@ class SizeofTest(unittest.TestCase): nfrees = len(x.f_code.co_freevars) extras = x.f_code.co_stacksize + x.f_code.co_nlocals +\ ncells + nfrees - 1 - check(x, size(vh + '12P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P')) + check(x, size(vh + '13P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P')) # function def func(): pass check(func, size(h + '12P')) @@ -96,6 +96,7 @@ Dominic Binks Philippe Biondi Stuart Bishop Roy Bixler +Renaud Blanch Mike Bland Martin Bless Pablo Bleyer @@ -482,6 +483,7 @@ Geert Jansen Jack Jansen Bill Janssen Thomas Jarosch +Zbigniew Jędrzejewski-Szmek Julien Jehannet Drew Jenkins Flemming Kjær Jensen @@ -10,6 +10,10 @@ What's New in Python 3.3 Alpha 1? Core and Builtins ----------------- +- PEP 380, Issue #11682: Add "yield from <x>" to support easy delegation to + subgenerators (initial patch by Greg Ewing, integration into 3.3 by + Renaud Blanch, Ryan Kelly, Zbigniew Jędrzejewski-Szmek and Nick Coghlan) + - Issue #13748: Raw bytes literals can now be written with the ``rb`` prefix as well as ``br``. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c index 2f2e045..9d1bca5 100644 --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -298,25 +298,25 @@ parser_richcompare(PyObject *left, PyObject *right, int op) /* Convert return value to a Boolean */ switch (op) { - case Py_EQ: + case Py_EQ: v = TEST_COND(result == 0); break; - case Py_NE: + case Py_NE: v = TEST_COND(result != 0); break; - case Py_LE: + case Py_LE: v = TEST_COND(result <= 0); break; - case Py_GE: + case Py_GE: v = TEST_COND(result >= 0); break; - case Py_LT: + case Py_LT: v = TEST_COND(result < 0); break; - case Py_GT: + case Py_GT: v = TEST_COND(result > 0); break; - default: + default: PyErr_BadArgument(); return NULL; } @@ -976,6 +976,7 @@ VALIDATER(comp_iter); VALIDATER(comp_if); VALIDATER(testlist_comp); VALIDATER(yield_expr); VALIDATER(or_test); VALIDATER(test_nocond); VALIDATER(lambdef_nocond); +VALIDATER(yield_arg); #undef VALIDATER @@ -1636,22 +1637,49 @@ validate_raise_stmt(node *tree) } -/* yield_expr: 'yield' [testlist] +/* yield_expr: 'yield' [yield_arg] */ static int validate_yield_expr(node *tree) { int nch = NCH(tree); - int res = (validate_ntype(tree, yield_expr) - && ((nch == 1) || (nch == 2)) - && validate_name(CHILD(tree, 0), "yield")); - - if (res && (nch == 2)) - res = validate_testlist(CHILD(tree, 1)); - - return (res); + if (nch < 1 || nch > 2) + return 0; + if (!validate_ntype(tree, yield_expr)) + return 0; + if (!validate_name(CHILD(tree, 0), "yield")) + return 0; + if (nch == 2) { + if (!validate_yield_arg(CHILD(tree, 1))) + return 0; + } + return 1; } +/* yield_arg: 'from' test | testlist + */ +static int +validate_yield_arg(node *tree) +{ + int nch = NCH(tree); + if (!validate_ntype(tree, yield_arg)) + return 0; + switch (nch) { + case 1: + if (!validate_testlist(CHILD(tree, nch - 1))) + return 0; + break; + case 2: + if (!validate_name(CHILD(tree, 0), "from")) + return 0; + if (!validate_test(CHILD(tree, 1))) + return 0; + break; + default: + return 0; + } + return 1; +} /* yield_stmt: yield_expr */ @@ -2120,16 +2148,16 @@ validate_comp_op(node *tree) */ tree = CHILD(tree, 0); switch (TYPE(tree)) { - case LESS: - case GREATER: - case EQEQUAL: - case EQUAL: - case LESSEQUAL: - case GREATEREQUAL: - case NOTEQUAL: + case LESS: + case GREATER: + case EQEQUAL: + case EQUAL: + case LESSEQUAL: + case GREATEREQUAL: + case NOTEQUAL: res = 1; break; - case NAME: + case NAME: res = ((strcmp(STR(tree), "in") == 0) || (strcmp(STR(tree), "is") == 0)); if (!res) { @@ -2665,9 +2693,9 @@ validate_argument(node *tree) { int nch = NCH(tree); int res = (validate_ntype(tree, argument) - && ((nch == 1) || (nch == 2) || (nch == 3)) - && validate_test(CHILD(tree, 0))); - + && ((nch == 1) || (nch == 2) || (nch == 3))); + if (res) + res = validate_test(CHILD(tree, 0)); if (res && (nch == 2)) res = validate_comp_for(CHILD(tree, 1)); else if (res && (nch == 3)) diff --git a/Objects/abstract.c b/Objects/abstract.c index a44bec6..4d73a3b 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2267,7 +2267,6 @@ PyObject_CallMethod(PyObject *o, char *name, char *format, ...) func = PyObject_GetAttrString(o, name); if (func == NULL) { - PyErr_SetString(PyExc_AttributeError, name); return 0; } @@ -2311,7 +2310,6 @@ _PyObject_CallMethod_SizeT(PyObject *o, char *name, char *format, ...) func = PyObject_GetAttrString(o, name); if (func == NULL) { - PyErr_SetString(PyExc_AttributeError, name); return 0; } va_start(va, format); diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 3318115..a529a3c 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -487,8 +487,70 @@ SimpleExtendsException(PyExc_Exception, TypeError, /* * StopIteration extends Exception */ -SimpleExtendsException(PyExc_Exception, StopIteration, - "Signal the end from iterator.__next__()."); + +static PyMemberDef StopIteration_members[] = { + {"value", T_OBJECT, offsetof(PyStopIterationObject, value), 0, + PyDoc_STR("generator return value")}, + {NULL} /* Sentinel */ +}; + +static int +StopIteration_init(PyStopIterationObject *self, PyObject *args, PyObject *kwds) +{ + Py_ssize_t size = PyTuple_GET_SIZE(args); + PyObject *value; + + if (BaseException_init((PyBaseExceptionObject *)self, args, kwds) == -1) + return -1; + Py_CLEAR(self->value); + if (size > 0) + value = PyTuple_GET_ITEM(args, 0); + else + value = Py_None; + Py_INCREF(value); + self->value = value; + return 0; +} + +static int +StopIteration_clear(PyStopIterationObject *self) +{ + Py_CLEAR(self->value); + return BaseException_clear((PyBaseExceptionObject *)self); +} + +static void +StopIteration_dealloc(PyStopIterationObject *self) +{ + _PyObject_GC_UNTRACK(self); + StopIteration_clear(self); + Py_TYPE(self)->tp_free((PyObject *)self); +} + +static int +StopIteration_traverse(PyStopIterationObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->value); + return BaseException_traverse((PyBaseExceptionObject *)self, visit, arg); +} + +PyObject * +PyStopIteration_Create(PyObject *value) +{ + return PyObject_CallFunctionObjArgs(PyExc_StopIteration, value, NULL); +} + +ComplexExtendsException( + PyExc_Exception, /* base */ + StopIteration, /* name */ + StopIteration, /* prefix for *_init, etc */ + 0, /* new */ + 0, /* methods */ + StopIteration_members, /* members */ + 0, /* getset */ + 0, /* str */ + "Signal the end from iterator.__next__()." +); /* diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 10fb8b3..9b05b9d 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -15,11 +15,12 @@ #define OFF(x) offsetof(PyFrameObject, x) static PyMemberDef frame_memberlist[] = { - {"f_back", T_OBJECT, OFF(f_back), READONLY}, - {"f_code", T_OBJECT, OFF(f_code), READONLY}, - {"f_builtins", T_OBJECT, OFF(f_builtins),READONLY}, - {"f_globals", T_OBJECT, OFF(f_globals), READONLY}, - {"f_lasti", T_INT, OFF(f_lasti), READONLY}, + {"f_back", T_OBJECT, OFF(f_back), READONLY}, + {"f_code", T_OBJECT, OFF(f_code), READONLY}, + {"f_builtins", T_OBJECT, OFF(f_builtins), READONLY}, + {"f_globals", T_OBJECT, OFF(f_globals), READONLY}, + {"f_lasti", T_INT, OFF(f_lasti), READONLY}, + {"f_yieldfrom", T_OBJECT, OFF(f_yieldfrom), READONLY}, {NULL} /* Sentinel */ }; @@ -444,6 +445,7 @@ frame_dealloc(PyFrameObject *f) Py_CLEAR(f->f_exc_type); Py_CLEAR(f->f_exc_value); Py_CLEAR(f->f_exc_traceback); + Py_CLEAR(f->f_yieldfrom); co = f->f_code; if (co->co_zombieframe == NULL) @@ -475,6 +477,7 @@ frame_traverse(PyFrameObject *f, visitproc visit, void *arg) Py_VISIT(f->f_exc_type); Py_VISIT(f->f_exc_value); Py_VISIT(f->f_exc_traceback); + Py_VISIT(f->f_yieldfrom); /* locals */ slots = f->f_code->co_nlocals + PyTuple_GET_SIZE(f->f_code->co_cellvars) + PyTuple_GET_SIZE(f->f_code->co_freevars); @@ -508,6 +511,7 @@ frame_clear(PyFrameObject *f) Py_CLEAR(f->f_exc_value); Py_CLEAR(f->f_exc_traceback); Py_CLEAR(f->f_trace); + Py_CLEAR(f->f_yieldfrom); /* locals */ slots = f->f_code->co_nlocals + PyTuple_GET_SIZE(f->f_code->co_cellvars) + PyTuple_GET_SIZE(f->f_code->co_freevars); @@ -711,6 +715,7 @@ PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals, f->f_lasti = -1; f->f_lineno = code->co_firstlineno; f->f_iblock = 0; + f->f_yieldfrom = NULL; _PyObject_GC_TRACK(f); return f; diff --git a/Objects/genobject.c b/Objects/genobject.c index c6612e2..20c926b 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -5,6 +5,9 @@ #include "structmember.h" #include "opcode.h" +static PyObject *gen_close(PyGenObject *gen, PyObject *args); +static void gen_undelegate(PyGenObject *gen); + static int gen_traverse(PyGenObject *gen, visitproc visit, void *arg) { @@ -90,12 +93,18 @@ gen_send_ex(PyGenObject *gen, PyObject *arg, int exc) /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ - if (result == Py_None && f->f_stacktop == NULL) { - Py_DECREF(result); - result = NULL; - /* Set exception if not called by gen_iternext() */ - if (arg) + if (result && f->f_stacktop == NULL) { + if (result == Py_None) { + /* Delay exception instantiation if we can */ PyErr_SetNone(PyExc_StopIteration); + } else { + PyObject *e = PyStopIteration_Create(result); + if (e != NULL) { + PyErr_SetObject(PyExc_StopIteration, e); + Py_DECREF(e); + } + } + Py_CLEAR(result); } if (!result || f->f_stacktop == NULL) { @@ -111,8 +120,8 @@ gen_send_ex(PyGenObject *gen, PyObject *arg, int exc) Py_XDECREF(t); Py_XDECREF(v); Py_XDECREF(tb); - Py_DECREF(f); gen->gi_frame = NULL; + Py_DECREF(f); } return result; @@ -125,17 +134,91 @@ return next yielded value or raise StopIteration."); static PyObject * gen_send(PyGenObject *gen, PyObject *arg) { - return gen_send_ex(gen, arg, 0); + int exc = 0; + PyObject *ret; + PyObject *yf = gen->gi_frame ? gen->gi_frame->f_yieldfrom : NULL; + /* XXX (ncoghlan): Are the incref/decref on arg and yf strictly needed? + * Or would it be valid to rely on borrowed references? + */ + Py_INCREF(arg); + if (yf) { + Py_INCREF(yf); + if (PyGen_CheckExact(yf)) { + ret = gen_send((PyGenObject *)yf, arg); + } else { + if (arg == Py_None) + ret = PyIter_Next(yf); + else + ret = PyObject_CallMethod(yf, "send", "O", arg); + } + if (ret) { + Py_DECREF(yf); + goto done; + } + gen_undelegate(gen); + Py_CLEAR(arg); + if (PyGen_FetchStopIterationValue(&arg) < 0) { + exc = 1; + } + Py_DECREF(yf); + } + ret = gen_send_ex(gen, arg, exc); +done: + Py_XDECREF(arg); + return ret; } PyDoc_STRVAR(close_doc, "close(arg) -> raise GeneratorExit inside generator."); +/* + * This helper function is used by gen_close and gen_throw to + * close a subiterator being delegated to by yield-from. + */ + +static int +gen_close_iter(PyObject *yf) +{ + PyObject *retval = NULL; + + if (PyGen_CheckExact(yf)) { + retval = gen_close((PyGenObject *)yf, NULL); + if (retval == NULL) { + return -1; + } + } else { + PyObject *meth = PyObject_GetAttrString(yf, "close"); + if (meth == NULL) { + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { + PyErr_WriteUnraisable(yf); + } + PyErr_Clear(); + } else { + retval = PyObject_CallFunction(meth, ""); + Py_DECREF(meth); + if (!retval) + return -1; + } + } + Py_XDECREF(retval); + return 0; +} + static PyObject * gen_close(PyGenObject *gen, PyObject *args) { PyObject *retval; - PyErr_SetNone(PyExc_GeneratorExit); + PyObject *yf = gen->gi_frame ? gen->gi_frame->f_yieldfrom : NULL; + int err = 0; + + if (yf) { + Py_INCREF(yf); + err = gen_close_iter(yf); + gen_undelegate(gen); + Py_DECREF(yf); + } + if (err == 0) + PyErr_SetNone(PyExc_GeneratorExit); retval = gen_send_ex(gen, Py_None, 1); if (retval) { Py_DECREF(retval); @@ -196,7 +279,7 @@ gen_del(PyObject *self) _Py_NewReference(self); self->ob_refcnt = refcnt; } - assert(PyType_IS_GC(self->ob_type) && + assert(PyType_IS_GC(Py_TYPE(self)) && _Py_AS_GC(self)->gc.gc_refs != _PyGC_REFS_UNTRACKED); /* If Py_REF_DEBUG, _Py_NewReference bumped _Py_RefTotal, so @@ -209,8 +292,8 @@ gen_del(PyObject *self) * undone. */ #ifdef COUNT_ALLOCS - --self->ob_type->tp_frees; - --self->ob_type->tp_allocs; + --(Py_TYPE(self)->tp_frees); + --(Py_TYPE(self)->tp_allocs); #endif } @@ -226,10 +309,55 @@ gen_throw(PyGenObject *gen, PyObject *args) PyObject *typ; PyObject *tb = NULL; PyObject *val = NULL; + PyObject *yf = gen->gi_frame ? gen->gi_frame->f_yieldfrom : NULL; if (!PyArg_UnpackTuple(args, "throw", 1, 3, &typ, &val, &tb)) return NULL; + if (yf) { + PyObject *ret; + int err; + Py_INCREF(yf); + if (PyErr_GivenExceptionMatches(typ, PyExc_GeneratorExit)) { + err = gen_close_iter(yf); + Py_DECREF(yf); + gen_undelegate(gen); + if (err < 0) + return gen_send_ex(gen, Py_None, 1); + goto throw_here; + } + if (PyGen_CheckExact(yf)) { + ret = gen_throw((PyGenObject *)yf, args); + } else { + PyObject *meth = PyObject_GetAttrString(yf, "throw"); + if (meth == NULL) { + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { + Py_DECREF(yf); + return NULL; + } + PyErr_Clear(); + Py_DECREF(yf); + gen_undelegate(gen); + goto throw_here; + } + ret = PyObject_CallObject(meth, args); + Py_DECREF(meth); + } + Py_DECREF(yf); + if (!ret) { + PyObject *val; + gen_undelegate(gen); + if (PyGen_FetchStopIterationValue(&val) == 0) { + ret = gen_send_ex(gen, val, 0); + Py_DECREF(val); + } else { + ret = gen_send_ex(gen, Py_None, 1); + } + } + return ret; + } + +throw_here: /* First, check the traceback argument, replacing None with NULL. */ if (tb == Py_None) { @@ -272,7 +400,7 @@ gen_throw(PyGenObject *gen, PyObject *args) PyErr_Format(PyExc_TypeError, "exceptions must be classes or instances " "deriving from BaseException, not %s", - typ->ob_type->tp_name); + Py_TYPE(typ)->tp_name); goto failed_throw; } @@ -291,9 +419,74 @@ failed_throw: static PyObject * gen_iternext(PyGenObject *gen) { - return gen_send_ex(gen, NULL, 0); + PyObject *val = NULL; + PyObject *ret; + int exc = 0; + PyObject *yf = gen->gi_frame ? gen->gi_frame->f_yieldfrom : NULL; + if (yf) { + Py_INCREF(yf); + /* ceval.c ensures that yf is an iterator */ + ret = Py_TYPE(yf)->tp_iternext(yf); + if (ret) { + Py_DECREF(yf); + return ret; + } + gen_undelegate(gen); + if (PyGen_FetchStopIterationValue(&val) < 0) + exc = 1; + Py_DECREF(yf); + } + ret = gen_send_ex(gen, val, exc); + Py_XDECREF(val); + return ret; +} + +/* + * In certain recursive situations, a generator may lose its frame + * before we get a chance to clear f_yieldfrom, so we use this + * helper function. + */ + +static void +gen_undelegate(PyGenObject *gen) { + if (gen->gi_frame) { + Py_XDECREF(gen->gi_frame->f_yieldfrom); + gen->gi_frame->f_yieldfrom = NULL; + } } +/* + * If StopIteration exception is set, fetches its 'value' + * attribute if any, otherwise sets pvalue to None. + * + * Returns 0 if no exception or StopIteration is set. + * If any other exception is set, returns -1 and leaves + * pvalue unchanged. + */ + +int +PyGen_FetchStopIterationValue(PyObject **pvalue) { + PyObject *et, *ev, *tb; + PyObject *value = NULL; + + if (PyErr_ExceptionMatches(PyExc_StopIteration)) { + PyErr_Fetch(&et, &ev, &tb); + Py_XDECREF(et); + Py_XDECREF(tb); + if (ev) { + value = ((PyStopIterationObject *)ev)->value; + Py_DECREF(ev); + } + } else if (PyErr_Occurred()) { + return -1; + } + if (value == NULL) { + value = Py_None; + } + Py_INCREF(value); + *pvalue = value; + return 0; +} static PyObject * gen_repr(PyGenObject *gen) diff --git a/Parser/Python.asdl b/Parser/Python.asdl index 6955199..8ae4e59 100644 --- a/Parser/Python.asdl +++ b/Parser/Python.asdl @@ -59,7 +59,7 @@ module Python | DictComp(expr key, expr value, comprehension* generators) | GeneratorExp(expr elt, comprehension* generators) -- the grammar constrains where yield expressions can occur - | Yield(expr? value) + | Yield(int is_from, expr? value) -- need sequences for compare to distinguish between -- x < 4 < 3 and (x < 4) < 3 | Compare(expr left, cmpop* ops, expr* comparators) diff --git a/Python/Python-ast.c b/Python/Python-ast.c index 8a101cf..3121eb1 100644 --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -231,7 +231,9 @@ static char *GeneratorExp_fields[]={ "generators", }; static PyTypeObject *Yield_type; +_Py_IDENTIFIER(is_from); static char *Yield_fields[]={ + "is_from", "value", }; static PyTypeObject *Compare_type; @@ -810,7 +812,7 @@ static int init_types(void) GeneratorExp_type = make_type("GeneratorExp", expr_type, GeneratorExp_fields, 2); if (!GeneratorExp_type) return 0; - Yield_type = make_type("Yield", expr_type, Yield_fields, 1); + Yield_type = make_type("Yield", expr_type, Yield_fields, 2); if (!Yield_type) return 0; Compare_type = make_type("Compare", expr_type, Compare_fields, 3); if (!Compare_type) return 0; @@ -1747,13 +1749,14 @@ GeneratorExp(expr_ty elt, asdl_seq * generators, int lineno, int col_offset, } expr_ty -Yield(expr_ty value, int lineno, int col_offset, PyArena *arena) +Yield(int is_from, expr_ty value, int lineno, int col_offset, PyArena *arena) { expr_ty p; p = (expr_ty)PyArena_Malloc(arena, sizeof(*p)); if (!p) return NULL; p->kind = Yield_kind; + p->v.Yield.is_from = is_from; p->v.Yield.value = value; p->lineno = lineno; p->col_offset = col_offset; @@ -2795,6 +2798,11 @@ ast2obj_expr(void* _o) case Yield_kind: result = PyType_GenericNew(Yield_type, NULL, NULL); if (!result) goto failed; + value = ast2obj_int(o->v.Yield.is_from); + if (!value) goto failed; + if (PyObject_SetAttrString(result, "is_from", value) == -1) + goto failed; + Py_DECREF(value); value = ast2obj_expr(o->v.Yield.value); if (!value) goto failed; if (_PyObject_SetAttrId(result, &PyId_value, value) == -1) @@ -5337,8 +5345,21 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena) return 1; } if (isinstance) { + int is_from; expr_ty value; + if (_PyObject_HasAttrId(obj, &PyId_is_from)) { + int res; + tmp = _PyObject_GetAttrId(obj, &PyId_is_from); + if (tmp == NULL) goto failed; + res = obj2ast_int(tmp, &is_from, arena); + if (res != 0) goto failed; + Py_XDECREF(tmp); + tmp = NULL; + } else { + PyErr_SetString(PyExc_TypeError, "required field \"is_from\" missing from Yield"); + return 1; + } if (_PyObject_HasAttrId(obj, &PyId_value)) { int res; tmp = _PyObject_GetAttrId(obj, &PyId_value); @@ -5350,7 +5371,7 @@ obj2ast_expr(PyObject* obj, expr_ty* out, PyArena* arena) } else { value = NULL; } - *out = Yield(value, lineno, col_offset, arena); + *out = Yield(is_from, value, lineno, col_offset, arena); if (*out == NULL) goto failed; return 0; } diff --git a/Python/ast.c b/Python/ast.c index 110754b..7080c65 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -2369,13 +2369,24 @@ ast_for_expr(struct compiling *c, const node *n) } return ast_for_binop(c, n); case yield_expr: { + node *an = NULL; + node *en = NULL; + int is_from = 0; expr_ty exp = NULL; - if (NCH(n) == 2) { - exp = ast_for_testlist(c, CHILD(n, 1)); + if (NCH(n) > 1) + an = CHILD(n, 1); /* yield_arg */ + if (an) { + en = CHILD(an, NCH(an) - 1); + if (NCH(an) == 2) { + is_from = 1; + exp = ast_for_expr(c, en); + } + else + exp = ast_for_testlist(c, en); if (!exp) return NULL; } - return Yield(exp, LINENO(n), n->n_col_offset, c->c_arena); + return Yield(is_from, exp, LINENO(n), n->n_col_offset, c->c_arena); } case factor: if (NCH(n) == 1) { @@ -2399,7 +2410,7 @@ ast_for_call(struct compiling *c, const node *n, expr_ty func) /* arglist: (argument ',')* (argument [',']| '*' test [',' '**' test] | '**' test) - argument: [test '='] test [comp_for] # Really [keyword '='] test + argument: [test '='] (test) [comp_for] # Really [keyword '='] test */ int i, nargs, nkeywords, ngens; @@ -2693,7 +2704,7 @@ ast_for_flow_stmt(struct compiling *c, const node *n) continue_stmt: 'continue' return_stmt: 'return' [testlist] yield_stmt: yield_expr - yield_expr: 'yield' testlist + yield_expr: 'yield' testlist | 'yield' 'from' test raise_stmt: 'raise' [test [',' test [',' test]]] */ node *ch; diff --git a/Python/ceval.c b/Python/ceval.c index ed82b94..134d1ee 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1828,6 +1828,52 @@ PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) why = WHY_RETURN; goto fast_block_end; + TARGET(YIELD_FROM) + u = POP(); + x = PyObject_GetIter(u); + Py_DECREF(u); + if (x == NULL) + break; + /* x is now the iterator, make the first next() call */ + retval = (*Py_TYPE(x)->tp_iternext)(x); + if (!retval) { + /* iter may be exhausted */ + Py_CLEAR(x); + if (!PyErr_ExceptionMatches(PyExc_StopIteration)) { + /* some other exception */ + break; + } + /* try to get return value from exception */ + PyObject *et, *ev, *tb; + PyErr_Fetch(&et, &ev, &tb); + Py_XDECREF(et); + Py_XDECREF(tb); + /* u is return value */ + u = NULL; + if (ev) { + u = PyObject_GetAttrString(ev, "value"); + Py_DECREF(ev); + if (u == NULL) { + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { + /* some other exception */ + break; + } + PyErr_Clear(); + } + } + if (u == NULL) { + u = Py_None; + Py_INCREF(u); + } + PUSH(u); + continue; + } + /* x is iterator, retval is value to be yielded */ + f->f_yieldfrom = x; + f->f_stacktop = stack_pointer; + why = WHY_YIELD; + goto fast_yield; + TARGET(YIELD_VALUE) retval = POP(); f->f_stacktop = stack_pointer; diff --git a/Python/compile.c b/Python/compile.c index 849f487..4d91f50 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -840,6 +840,7 @@ opcode_stack_effect(int opcode, int oparg) case IMPORT_STAR: return -1; case YIELD_VALUE: + case YIELD_FROM: return 0; case POP_BLOCK: @@ -3318,7 +3319,12 @@ compiler_visit_expr(struct compiler *c, expr_ty e) else { ADDOP_O(c, LOAD_CONST, Py_None, consts); } - ADDOP(c, YIELD_VALUE); + if (e->v.Yield.is_from) { + ADDOP(c, YIELD_FROM); + } + else { + ADDOP(c, YIELD_VALUE); + } break; case Compare_kind: return compiler_compare(c, e); diff --git a/Python/graminit.c b/Python/graminit.c index a8af583..e91e09f 100644 --- a/Python/graminit.c +++ b/Python/graminit.c @@ -1791,7 +1791,7 @@ static arc arcs_80_0[1] = { {167, 1}, }; static arc arcs_80_1[2] = { - {9, 2}, + {168, 2}, {0, 1}, }; static arc arcs_80_2[1] = { @@ -1802,171 +1802,188 @@ static state states_80[3] = { {2, arcs_80_1}, {1, arcs_80_2}, }; -static dfa dfas[81] = { +static arc arcs_81_0[2] = { + {73, 1}, + {9, 2}, +}; +static arc arcs_81_1[1] = { + {24, 2}, +}; +static arc arcs_81_2[1] = { + {0, 2}, +}; +static state states_81[3] = { + {2, arcs_81_0}, + {1, arcs_81_1}, + {1, arcs_81_2}, +}; +static dfa dfas[82] = { {256, "single_input", 0, 3, states_0, - "\004\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204"}, + "\004\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204\000"}, {257, "file_input", 0, 2, states_1, - "\204\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204"}, + "\204\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204\000"}, {258, "eval_input", 0, 3, states_2, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {259, "decorator", 0, 7, states_3, - "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {260, "decorators", 0, 2, states_4, - "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {261, "decorated", 0, 3, states_5, - "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\010\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {262, "funcdef", 0, 8, states_6, - "\000\000\020\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\020\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {263, "parameters", 0, 4, states_7, - "\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {264, "typedargslist", 0, 18, states_8, - "\000\000\040\200\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\200\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {265, "tfpdef", 0, 4, states_9, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {266, "varargslist", 0, 18, states_10, - "\000\000\040\200\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\200\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {267, "vfpdef", 0, 2, states_11, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {268, "stmt", 0, 2, states_12, - "\000\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204"}, + "\000\050\060\200\000\000\000\240\340\223\160\220\045\200\020\000\000\206\120\076\204\000"}, {269, "simple_stmt", 0, 4, states_13, - "\000\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200"}, + "\000\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200\000"}, {270, "small_stmt", 0, 2, states_14, - "\000\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200"}, + "\000\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200\000"}, {271, "expr_stmt", 0, 6, states_15, - "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {272, "testlist_star_expr", 0, 3, states_16, - "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {273, "augassign", 0, 2, states_17, - "\000\000\000\000\000\000\376\037\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\376\037\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {274, "del_stmt", 0, 3, states_18, - "\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {275, "pass_stmt", 0, 2, states_19, - "\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {276, "flow_stmt", 0, 2, states_20, - "\000\000\000\000\000\000\000\000\340\001\000\000\000\000\000\000\000\000\000\000\200"}, + "\000\000\000\000\000\000\000\000\340\001\000\000\000\000\000\000\000\000\000\000\200\000"}, {277, "break_stmt", 0, 2, states_21, - "\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {278, "continue_stmt", 0, 2, states_22, - "\000\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {279, "return_stmt", 0, 3, states_23, - "\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {280, "yield_stmt", 0, 2, states_24, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000"}, {281, "raise_stmt", 0, 5, states_25, - "\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000"}, {282, "import_stmt", 0, 2, states_26, - "\000\000\000\000\000\000\000\000\000\022\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\022\000\000\000\000\000\000\000\000\000\000\000\000"}, {283, "import_name", 0, 3, states_27, - "\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000\000\000"}, {284, "import_from", 0, 8, states_28, - "\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000\000"}, {285, "import_as_name", 0, 4, states_29, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {286, "dotted_as_name", 0, 4, states_30, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {287, "import_as_names", 0, 3, states_31, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {288, "dotted_as_names", 0, 2, states_32, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {289, "dotted_name", 0, 2, states_33, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {290, "global_stmt", 0, 3, states_34, - "\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000\000"}, {291, "nonlocal_stmt", 0, 3, states_35, - "\000\000\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000\000\000"}, {292, "assert_stmt", 0, 5, states_36, - "\000\000\000\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\000\000\000"}, {293, "compound_stmt", 0, 2, states_37, - "\000\010\020\000\000\000\000\000\000\000\000\220\045\000\000\000\000\000\000\000\004"}, + "\000\010\020\000\000\000\000\000\000\000\000\220\045\000\000\000\000\000\000\000\004\000"}, {294, "if_stmt", 0, 8, states_38, - "\000\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000"}, {295, "while_stmt", 0, 8, states_39, - "\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\000\000"}, {296, "for_stmt", 0, 10, states_40, - "\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000"}, {297, "try_stmt", 0, 13, states_41, - "\000\000\000\000\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\000\000"}, {298, "with_stmt", 0, 5, states_42, - "\000\000\000\000\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\040\000\000\000\000\000\000\000\000\000"}, {299, "with_item", 0, 4, states_43, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {300, "except_clause", 0, 5, states_44, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000"}, {301, "suite", 0, 5, states_45, - "\004\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200"}, + "\004\040\040\200\000\000\000\240\340\223\160\000\000\200\020\000\000\206\120\076\200\000"}, {302, "test", 0, 6, states_46, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {303, "test_nocond", 0, 2, states_47, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {304, "lambdef", 0, 5, states_48, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000"}, {305, "lambdef_nocond", 0, 5, states_49, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000"}, {306, "or_test", 0, 2, states_50, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000\000"}, {307, "and_test", 0, 2, states_51, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000\000"}, {308, "not_test", 0, 3, states_52, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\020\000\000\206\120\076\000\000"}, {309, "comparison", 0, 2, states_53, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {310, "comp_op", 0, 4, states_54, - "\000\000\000\000\000\000\000\000\000\000\000\000\002\000\220\177\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\002\000\220\177\000\000\000\000\000\000"}, {311, "star_expr", 0, 3, states_55, - "\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\200\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {312, "expr", 0, 2, states_56, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {313, "xor_expr", 0, 2, states_57, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {314, "and_expr", 0, 2, states_58, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {315, "shift_expr", 0, 2, states_59, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {316, "arith_expr", 0, 2, states_60, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {317, "term", 0, 2, states_61, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {318, "factor", 0, 3, states_62, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {319, "power", 0, 4, states_63, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\120\076\000\000"}, {320, "atom", 0, 9, states_64, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\000\000\000\000\000\120\076\000\000"}, {321, "testlist_comp", 0, 5, states_65, - "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\200\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {322, "trailer", 0, 7, states_66, - "\000\040\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\020\000\000"}, + "\000\040\000\000\000\000\000\000\000\100\000\000\000\000\000\000\000\000\020\000\000\000"}, {323, "subscriptlist", 0, 3, states_67, - "\000\040\040\002\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\002\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {324, "subscript", 0, 5, states_68, - "\000\040\040\002\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\002\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {325, "sliceop", 0, 3, states_69, - "\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\002\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {326, "exprlist", 0, 3, states_70, - "\000\040\040\200\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000"}, + "\000\040\040\200\000\000\000\000\000\200\000\000\000\000\000\000\000\206\120\076\000\000"}, {327, "testlist", 0, 3, states_71, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {328, "dictorsetmaker", 0, 11, states_72, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {329, "classdef", 0, 8, states_73, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\004"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\004\000"}, {330, "arglist", 0, 8, states_74, - "\000\040\040\200\001\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\200\001\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {331, "argument", 0, 4, states_75, - "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000"}, + "\000\040\040\000\000\000\000\000\000\200\000\000\000\200\020\000\000\206\120\076\000\000"}, {332, "comp_iter", 0, 2, states_76, - "\000\000\000\000\000\000\000\000\000\000\000\020\001\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\020\001\000\000\000\000\000\000\000\000\000"}, {333, "comp_for", 0, 6, states_77, - "\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000"}, {334, "comp_if", 0, 4, states_78, - "\000\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000"}, + "\000\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\000\000\000"}, {335, "encoding_decl", 0, 2, states_79, - "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, + "\000\000\040\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"}, {336, "yield_expr", 0, 3, states_80, - "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200"}, + "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\200\000"}, + {337, "yield_arg", 0, 3, states_81, + "\000\040\040\000\000\000\000\000\000\202\000\000\000\200\020\000\000\206\120\076\000\000"}, }; -static label labels[168] = { +static label labels[169] = { {0, "EMPTY"}, {256, 0}, {4, 0}, @@ -2135,10 +2152,11 @@ static label labels[168] = { {334, 0}, {335, 0}, {1, "yield"}, + {337, 0}, }; grammar _PyParser_Grammar = { - 81, + 82, dfas, - {168, labels}, + {169, labels}, 256 }; diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index a91da79..2d6bcda 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -71,7 +71,7 @@ static void *opcode_targets[256] = { &&TARGET_STORE_LOCALS, &&TARGET_PRINT_EXPR, &&TARGET_LOAD_BUILD_CLASS, - &&_unknown_opcode, + &&TARGET_YIELD_FROM, &&_unknown_opcode, &&_unknown_opcode, &&TARGET_INPLACE_LSHIFT, diff --git a/Python/symtable.c b/Python/symtable.c index 824a53f..1ce7f70 100644 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -19,10 +19,6 @@ #define IMPORT_STAR_WARNING "import * only allowed at module level" -#define RETURN_VAL_IN_GENERATOR \ - "'return' with argument inside generator" - - static PySTEntryObject * ste_new(struct symtable *st, identifier name, _Py_block_ty block, void *key, int lineno, int col_offset) @@ -1133,14 +1129,6 @@ symtable_visit_stmt(struct symtable *st, stmt_ty s) if (s->v.Return.value) { VISIT(st, expr, s->v.Return.value); st->st_cur->ste_returns_value = 1; - if (st->st_cur->ste_generator) { - PyErr_SetString(PyExc_SyntaxError, - RETURN_VAL_IN_GENERATOR); - PyErr_SyntaxLocationEx(st->st_filename, - s->lineno, - s->col_offset); - return 0; - } } break; case Delete_kind: @@ -1345,13 +1333,6 @@ symtable_visit_expr(struct symtable *st, expr_ty e) if (e->v.Yield.value) VISIT(st, expr, e->v.Yield.value); st->st_cur->ste_generator = 1; - if (st->st_cur->ste_returns_value) { - PyErr_SetString(PyExc_SyntaxError, - RETURN_VAL_IN_GENERATOR); - PyErr_SyntaxLocationEx(st->st_filename, - e->lineno, e->col_offset); - return 0; - } break; case Compare_kind: VISIT(st, expr, e->v.Compare.left); |