diff options
| -rw-r--r-- | Doc/reference/expressions.rst | 61 | ||||
| -rw-r--r-- | Doc/whatsnew/3.7.rst | 12 | ||||
| -rw-r--r-- | Lib/test/test_generators.py | 12 | ||||
| -rw-r--r-- | Lib/test/test_grammar.py | 35 | ||||
| -rw-r--r-- | Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst | 3 | ||||
| -rw-r--r-- | Python/symtable.c | 31 |
6 files changed, 134 insertions, 20 deletions
diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 1cff8a5..fb92ad0 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -183,8 +183,21 @@ by considering each of the :keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. -Note that the comprehension is executed in a separate scope, so names assigned -to in the target list don't "leak" into the enclosing scope. +However, aside from the iterable expression in the leftmost :keyword:`for` clause, +the comprehension is executed in a separate implicitly nested scope. This ensures +that names assigned to in the target list don't "leak" into the enclosing scope. + +The iterable expression in the leftmost :keyword:`for` clause is evaluated +directly in the enclosing scope and then passed as an argument to the implictly +nested scope. Subsequent :keyword:`for` clauses and any filter condition in the +leftmost :keyword:`for` clause cannot be evaluated in the enclosing scope as +they may depend on the values obtained from the leftmost iterable. For example: +``[x*y for x in range(10) for y in range(x, x+10)]``. + +To ensure the comprehension always results in a container of the appropriate +type, ``yield`` and ``yield from`` expressions are prohibited in the implicitly +nested scope (in Python 3.7, such expressions emit :exc:`DeprecationWarning` +when compiled, in Python 3.8+ they will emit :exc:`SyntaxError`). Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for` clause may be used to iterate over a :term:`asynchronous iterator`. @@ -198,6 +211,13 @@ or :keyword:`await` expressions it is called an suspend the execution of the coroutine function in which it appears. See also :pep:`530`. +.. versionadded:: 3.6 + Asynchronous comprehensions were introduced. + +.. deprecated:: 3.7 + ``yield`` and ``yield from`` deprecated in the implicitly nested scope. + + .. _lists: List displays @@ -316,27 +336,42 @@ brackets or curly braces. Variables used in the generator expression are evaluated lazily when the :meth:`~generator.__next__` method is called for the generator object (in the same -fashion as normal generators). However, the leftmost :keyword:`for` clause is -immediately evaluated, so that an error produced by it can be seen before any -other possible error in the code that handles the generator expression. -Subsequent :keyword:`for` clauses cannot be evaluated immediately since they -may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in -range(10) for y in bar(x))``. +fashion as normal generators). However, the iterable expression in the +leftmost :keyword:`for` clause is immediately evaluated, so that an error +produced by it will be emitted at the point where the generator expression +is defined, rather than at the point where the first value is retrieved. +Subsequent :keyword:`for` clauses and any filter condition in the leftmost +:keyword:`for` clause cannot be evaluated in the enclosing scope as they may +depend on the values obtained from the leftmost iterable. For example: +``(x*y for x in range(10) for y in range(x, x+10))``. The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for details. +To avoid interfering with the expected operation of the generator expression +itself, ``yield`` and ``yield from`` expressions are prohibited in the +implicitly defined generator (in Python 3.7, such expressions emit +:exc:`DeprecationWarning` when compiled, in Python 3.8+ they will emit +:exc:`SyntaxError`). + If a generator expression contains either :keyword:`async for` clauses or :keyword:`await` expressions it is called an :dfn:`asynchronous generator expression`. An asynchronous generator expression returns a new asynchronous generator object, which is an asynchronous iterator (see :ref:`async-iterators`). +.. versionadded:: 3.6 + Asynchronous generator expressions were introduced. + .. versionchanged:: 3.7 Prior to Python 3.7, asynchronous generator expressions could only appear in :keyword:`async def` coroutines. Starting with 3.7, any function can use asynchronous generator expressions. +.. deprecated:: 3.7 + ``yield`` and ``yield from`` deprecated in the implicitly nested scope. + + .. _yieldexpr: Yield expressions @@ -364,6 +399,16 @@ coroutine function to be an asynchronous generator. For example:: async def agen(): # defines an asynchronous generator function (PEP 525) yield 123 +Due to their side effects on the containing scope, ``yield`` expressions +are not permitted as part of the implicitly defined scopes used to +implement comprehensions and generator expressions (in Python 3.7, such +expressions emit :exc:`DeprecationWarning` when compiled, in Python 3.8+ +they will emit :exc:`SyntaxError`).. + +.. deprecated:: 3.7 + Yield expressions deprecated in the implicitly nested scopes used to + implement comprehensions and generator expressions. + Generator functions are described below, while asynchronous generator functions are described separately in section :ref:`asynchronous-generator-functions`. diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 6545a18..b6dad4e 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -570,6 +570,18 @@ Other CPython Implementation Changes Deprecated ========== +* Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated + in comprehensions and generator expressions (aside from the iterable expression + in the leftmost :keyword:`for` clause). This ensures that comprehensions + always immediately return a container of the appropriate type (rather than + potentially returning a :term:`generator iterator` object), while generator + expressions won't attempt to interleave their implicit output with the output + from any explicit yield expressions. + + In Python 3.7, such expressions emit :exc:`DeprecationWarning` when compiled, + in Python 3.8+ they will emit :exc:`SyntaxError`. (Contributed by Serhiy + Storchaka in :issue:`10544`.) + - Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with a macro if ``Py_LIMITED_API`` is not set or set to the value between ``0x03050400`` and ``0x03060000`` (not including) or ``0x03060100`` or diff --git a/Lib/test/test_generators.py b/Lib/test/test_generators.py index 7eac9d0..f88c762 100644 --- a/Lib/test/test_generators.py +++ b/Lib/test/test_generators.py @@ -1830,13 +1830,7 @@ Yield by itself yields None: [None] - -An obscene abuse of a yield expression within a generator expression: - ->>> list((yield 21) for i in range(4)) -[21, None, 21, None, 21, None, 21, None] - -And a more sane, but still weird usage: +Yield is allowed only in the outermost iterable in generator expression: >>> def f(): list(i for i in [(yield 26)]) >>> type(f()) @@ -2106,10 +2100,6 @@ enclosing function a generator: >>> type(f()) <class 'generator'> ->>> def f(): x=(i for i in (yield) if (yield)) ->>> type(f()) -<class 'generator'> - >>> def f(d): d[(yield "a")] = d[(yield "b")] = 27 >>> data = [1,2] >>> g = f(data) diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index 65e26bf..823315f 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -841,6 +841,41 @@ class GrammarTests(unittest.TestCase): # Check annotation refleak on SyntaxError check_syntax_error(self, "def g(a:(yield)): pass") + def test_yield_in_comprehensions(self): + # Check yield in comprehensions + def g(): [x for x in [(yield 1)]] + def g(): [x for x in [(yield from ())]] + + def check(code, warntext): + with self.assertWarnsRegex(DeprecationWarning, warntext): + compile(code, '<test string>', 'exec') + import warnings + with warnings.catch_warnings(): + warnings.filterwarnings('error', category=DeprecationWarning) + with self.assertRaisesRegex(SyntaxError, warntext): + compile(code, '<test string>', 'exec') + + check("def g(): [(yield x) for x in ()]", + "'yield' inside list comprehension") + check("def g(): [x for x in () if not (yield x)]", + "'yield' inside list comprehension") + check("def g(): [y for x in () for y in [(yield x)]]", + "'yield' inside list comprehension") + check("def g(): {(yield x) for x in ()}", + "'yield' inside set comprehension") + check("def g(): {(yield x): x for x in ()}", + "'yield' inside dict comprehension") + check("def g(): {x: (yield x) for x in ()}", + "'yield' inside dict comprehension") + check("def g(): ((yield x) for x in ())", + "'yield' inside generator expression") + check("def g(): [(yield from x) for x in ()]", + "'yield' inside list comprehension") + check("class C: [(yield x) for x in ()]", + "'yield' inside list comprehension") + check("[(yield x) for x in ()]", + "'yield' inside list comprehension") + def test_raise(self): # 'raise' test [',' test] try: raise RuntimeError('just testing') diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst new file mode 100644 index 0000000..555c94e --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2017-11-27-08-37-34.bpo-10544.07nioT.rst @@ -0,0 +1,3 @@ +Yield expressions are now deprecated in comprehensions and generator +expressions. They are still permitted in the definition of the outermost +iterable, as that is evaluated directly in the enclosing scope. diff --git a/Python/symtable.c b/Python/symtable.c index 55815c9..bbac25c 100644 --- a/Python/symtable.c +++ b/Python/symtable.c @@ -1734,7 +1734,6 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e, e->lineno, e->col_offset)) { return 0; } - st->st_cur->ste_generator = is_generator; if (outermost->is_async) { st->st_cur->ste_coroutine = 1; } @@ -1754,6 +1753,36 @@ symtable_handle_comprehension(struct symtable *st, expr_ty e, if (value) VISIT(st, expr, value); VISIT(st, expr, elt); + if (st->st_cur->ste_generator) { + PyObject *msg = PyUnicode_FromString( + (e->kind == ListComp_kind) ? "'yield' inside list comprehension" : + (e->kind == SetComp_kind) ? "'yield' inside set comprehension" : + (e->kind == DictComp_kind) ? "'yield' inside dict comprehension" : + "'yield' inside generator expression"); + if (msg == NULL) { + symtable_exit_block(st, (void *)e); + return 0; + } + if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, + msg, st->st_filename, st->st_cur->ste_lineno, + NULL, NULL) == -1) + { + if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) { + /* Replace the DeprecationWarning exception with a SyntaxError + to get a more accurate error report */ + PyErr_Clear(); + PyErr_SetObject(PyExc_SyntaxError, msg); + PyErr_SyntaxLocationObject(st->st_filename, + st->st_cur->ste_lineno, + st->st_cur->ste_col_offset); + } + Py_DECREF(msg); + symtable_exit_block(st, (void *)e); + return 0; + } + Py_DECREF(msg); + } + st->st_cur->ste_generator |= is_generator; return symtable_exit_block(st, (void *)e); } |