Merge p3yk branch with the trunk up to revision 45595. This breaks a fair

number of tests, all because of the codecs/_multibytecodecs issue described
here (it's not a Py3K issue, just something Py3K discovers):
http://mail.python.org/pipermail/python-dev/2006-April/064051.html

Hye-Shik Chang promised to look for a fix, so no need to fix it here. The
tests that are expected to break are:

test_codecencodings_cn
test_codecencodings_hk
test_codecencodings_jp
test_codecencodings_kr
test_codecencodings_tw
test_codecs
test_multibytecodec

This merge fixes an actual test failure (test_weakref) in this branch,
though, so I believe merging is the right thing to do anyway.

3 files changed, 156 insertions, 17 deletions

diff --git a/Doc/ref/ref3.tex b/Doc/ref/ref3.tex
index 737b861..964013f 100644
--- a/Doc/ref/ref3.tex
+++ b/Doc/ref/ref3.tex
@@ -1035,7 +1035,7 @@ by the built-in \function{classmethod()} constructor.
 %=========================================================================
 \section{New-style and classic classes}
 
-Classes and instances come in two flavours: old-style or classic, and new-style.  
+Classes and instances come in two flavors: old-style or classic, and new-style.  
 
 Up to Python 2.1, old-style classes were the only flavour available to the
 user.  The concept of (old-style) class is unrelated to the concept of type: if
@@ -1065,10 +1065,14 @@ the way special methods are invoked.  Others are "fixes" that could not be
 implemented before for compatibility concerns, like the method resolution order
 in case of multiple inheritance.
 
-This manuel is not up-to-date with respect to new-style classes.  For now,
+This manual is not up-to-date with respect to new-style classes.  For now,
 please see \url{http://www.python.org/doc/newstyle.html} for more information.
 
-The plan is to eventually drop old-style classes, leaving only the semantics of new-style classes.  This change will probably only be feasible in Python 3.0.
+The plan is to eventually drop old-style classes, leaving only the semantics of
+new-style classes.  This change will probably only be feasible in Python 3.0.
+\index{class}{new-style}
+\index{class}{classic}
+\index{class}{old-style}
 
 %=========================================================================
 \section{Special method names\label{specialnames}}
@@ -2053,14 +2057,15 @@ exception is raised.  But see the following exception:
 \item
 
 Exception to the previous item: if the left operand is an instance of
-a built-in type or a new-style class, and the right operand is an
-instance of a proper subclass of that type or class, the right
-operand's \method{__rop__()} method is tried \emph{before} the left
-operand's \method{__op__()} method.  This is done so that a subclass can
-completely override binary operators.  Otherwise, the left operand's
-__op__ method would always accept the right operand: when an instance
-of a given class is expected, an instance of a subclass of that class
-is always acceptable.
+a built-in type or a new-style class, and the right operand is an instance
+of a proper subclass of that type or class and overrides the base's
+\method{__rop__()} method, the right operand's \method{__rop__()} method
+is tried \emph{before} the left operand's \method{__op__()} method.
+
+This is done so that a subclass can completely override binary operators.
+Otherwise, the left operand's \method{__op__()} method would always
+accept the right operand: when an instance of a given class is expected,
+an instance of a subclass of that class is always acceptable.
 
 \item
 
@@ -2106,3 +2111,63 @@ implement a \method{__coerce__()} method, for use by the built-in
 \function{coerce()} function.
 
 \end{itemize}
+
+\subsection{Context Managers and Contexts\label{context-managers}}
+
+\versionadded{2.5}
+
+A \dfn{context manager} is an object that manages the entry to, and exit
+from, a \dfn{context} surrounding a block of code.  Context managers are
+normally invoked using the \keyword{with} statement (described in
+section~\ref{with}), but can also be used by directly invoking their
+methods.
+\stindex{with}
+\index{context manager}
+\index{context}
+
+Typical uses of context managers include saving and restoring various
+kinds of global state, locking and unlocking resources, closing opened
+files, etc.
+
+\begin{methoddesc}[context manager]{__context__}{self}
+Invoked when the object is used as the context expression of a
+\keyword{with} statement.  The return value must implement
+\method{__enter__()} and \method{__exit__()} methods.  Simple context
+managers that wish to directly
+implement \method{__enter__()} and \method{__exit__()} should just
+return \var{self}.
+
+Context managers written in Python can also implement this method using
+a generator function decorated with the
+\function{contextlib.contextmanager} decorator, as this can be simpler
+than writing individual \method{__enter__()} and \method{__exit__()}
+methods when the state to be managed is complex.
+\end{methoddesc}
+
+\begin{methoddesc}[context]{__enter__}{self}
+Enter the context defined by this object. The \keyword{with} statement
+will bind this method's return value to the target(s) specified in the
+\keyword{as} clause of the statement, if any.
+\end{methoddesc}
+
+\begin{methoddesc}[context]{__exit__}{exc_type, exc_value, traceback}
+Exit the context defined by this object. The parameters describe the
+exception that caused the context to be exited. If the context was
+exited without an exception, all three arguments will be
+\constant{None}.
+
+If an exception is supplied, and the method wishes to suppress the
+exception (i.e., prevent it from being propagated), it should return a
+true value. Otherwise, the exception will be processed normally upon
+exit from this method.
+
+Note that \method{__exit__} methods should not reraise the passed-in
+exception; this is the caller's responsibility.
+\end{methoddesc}
+
+\begin{seealso}
+  \seepep{0343}{The "with" statement}
+         {The specification, background, and examples for the
+          Python \keyword{with} statement.}
+\end{seealso}
+
diff --git a/Doc/ref/ref6.tex b/Doc/ref/ref6.tex
index d1d23ac..1eb1258 100644
--- a/Doc/ref/ref6.tex
+++ b/Doc/ref/ref6.tex
@@ -488,11 +488,12 @@ enough information is saved so that the next time \method{next()} is
 invoked, the function can proceed exactly as if the \keyword{yield}
 statement were just another external call.
 
-The \keyword{yield} statement is not allowed in the \keyword{try}
-clause of a \keyword{try} ...\ \keyword{finally} construct.  The
-difficulty is that there's no guarantee the generator will ever be
-resumed, hence no guarantee that the \keyword{finally} block will ever
-get executed.
+As of Python version 2.5, the \keyword{yield} statement is now
+allowed in the \keyword{try} clause of a \keyword{try} ...\ 
+\keyword{finally} construct.  If the generator is not resumed before
+it is finalized (by reaching a zero reference count or by being garbage
+collected), the generator-iterator's \method{close()} method will be
+called, allowing any pending \keyword{finally} clauses to execute.
 
 \begin{notice}
 In Python 2.2, the \keyword{yield} statement is only allowed
@@ -510,6 +511,11 @@ from __future__ import generators
   \seepep{0255}{Simple Generators}
          {The proposal for adding generators and the \keyword{yield}
           statement to Python.}
+
+  \seepep{0342}{Coroutines via Enhanced Generators}
+         {The proposal that, among other generator enhancements,
+          proposed allowing \keyword{yield} to appear inside a
+          \keyword{try} ... \keyword{finally} block.}
 \end{seealso}
 
 
diff --git a/Doc/ref/ref7.tex b/Doc/ref/ref7.tex
index 90627a4..a2d46a8 100644
--- a/Doc/ref/ref7.tex
+++ b/Doc/ref/ref7.tex
@@ -46,6 +46,7 @@ Summarizing:
   \productioncont{| \token{while_stmt}}
   \productioncont{| \token{for_stmt}}
   \productioncont{| \token{try_stmt}}
+  \productioncont{| \token{with_stmt}}
   \productioncont{| \token{funcdef}}
   \productioncont{| \token{classdef}}
   \production{suite}
@@ -305,8 +306,75 @@ statement to generate exceptions may be found in section~\ref{raise}.
 \section{The \keyword{with} statement\label{with}}
 \stindex{with}
 
-The \keyword{with} statement specifies 
+\versionadded{2.5}
 
+The \keyword{with} statement is used to wrap the execution of a block
+with methods defined by a context manager (see
+section~\ref{context-managers}). This allows common
+\keyword{try}...\keyword{except}...\keyword{finally} usage patterns to
+be encapsulated as context managers for convenient reuse.
+
+\begin{productionlist}
+  \production{with_stmt}
+  {"with" \token{expression} ["as" target_list] ":" \token{suite}}
+\end{productionlist}
+
+The execution of the \keyword{with} statement proceeds as follows:
+
+\begin{enumerate}
+
+\item The expression is evaluated, to obtain a context manager
+object.
+
+\item The context manager's \method{__context__()} method is invoked to
+obtain a context object.
+
+\item The context object's \method{__enter__()} method is invoked.
+
+\item If a target list was included in the \keyword{with}
+statement, the return value from \method{__enter__()} is assigned to it.
+
+\note{The \keyword{with} statement guarantees that if the
+\method{__enter__()} method returns without an error, then
+\method{__exit__()} will always be called. Thus, if an error occurs
+during the assignment to the target list, it will be treated the same as
+an error occurring within the suite would be. See step 6 below.}
+
+\item The suite is executed.
+
+\item The context object's \method{__exit__()} method is invoked. If an
+exception caused the suite to be exited, its type, value, and
+traceback are passed as arguments to \method{__exit__()}. Otherwise,
+three \constant{None} arguments are supplied.
+
+If the suite was exited due to an exception, and the return
+value from the \method{__exit__()} method was false, the exception is
+reraised. If the return value was true, the exception is suppressed, and
+execution continues with the statement following the \keyword{with}
+statement.
+
+If the suite was exited for any reason other than an exception, the
+return value from \method{__exit__()} is ignored, and execution proceeds
+at the normal location for the kind of exit that was taken.
+
+\end{enumerate}
+
+\begin{notice}
+In Python 2.5, the \keyword{with} statement is only allowed
+when the \code{with_statement} feature has been enabled.  It will always
+be enabled in Python 2.6.  This \code{__future__} import statement can
+be used to enable the feature:
+
+\begin{verbatim}
+from __future__ import with_statement
+\end{verbatim}
+\end{notice}
+
+\begin{seealso}
+  \seepep{0343}{The "with" statement}
+         {The specification, background, and examples for the
+          Python \keyword{with} statement.}
+\end{seealso}
 
 \section{Function definitions\label{function}}
 \indexii{function}{definition}