diff options
Diffstat (limited to 'Doc')
255 files changed, 16779 insertions, 4144 deletions
diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt index 38c3dc2..2aa0316 100644 --- a/Doc/ACKS.txt +++ b/Doc/ACKS.txt @@ -18,6 +18,7 @@ docs@python.org), and we'll be glad to correct the problem. * Oliver Andrich * Heidi Annexstad * Jesús Cea Avión + * Manuel Balsera * Daniel Barclay * Chris Barker * Don Bashford @@ -109,6 +110,7 @@ docs@python.org), and we'll be glad to correct the problem. * Andrew M. Kuchling * Dave Kuhlman * Erno Kuusela + * Ross Lagerwall * Thomas Lamb * Detlef Lannert * Piers Lauder @@ -126,6 +128,7 @@ docs@python.org), and we'll be glad to correct the problem. * Andrew MacIntyre * Vladimir Marangozov * Vincent Marchetti + * Westley Martínez * Laura Matson * Daniel May * Rebecca McCreary @@ -136,6 +139,7 @@ docs@python.org), and we'll be glad to correct the problem. * Ross Moore * Sjoerd Mullender * Dale Nagata + * Michal Nowikowski * Ng Pheng Siong * Koray Oner * Tomas Oppelstrup @@ -155,6 +159,7 @@ docs@python.org), and we'll be glad to correct the problem. * Paul Prescod * Eric S. Raymond * Edward K. Ream + * Terry J. Reedy * Sean Reifschneider * Bernhard Reiter * Armin Rigo @@ -205,6 +210,7 @@ docs@python.org), and we'll be glad to correct the problem. * Mats Wichmann * Gerry Wiener * Timothy Wild + * Paul Winkler * Collin Winter * Blake Winton * Dan Wolfe diff --git a/Doc/Makefile b/Doc/Makefile index f904e5f..2049986 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -34,12 +34,13 @@ help: @echo " dist to create a \"dist\" directory with archived docs for download" @echo " suspicious to check for suspicious markup in output text" @echo " check to run a check for frequent markup errors" + @echo " serve to serve the documentation on the localhost (8000)" # Note: if you update versions here, do the same in make.bat and README.txt checkout: @if [ ! -d tools/sphinx ]; then \ echo "Checking out Sphinx..."; \ - svn checkout $(SVNROOT)/external/Sphinx-0.6.5/sphinx tools/sphinx; \ + svn checkout $(SVNROOT)/external/Sphinx-0.6.7/sphinx tools/sphinx; \ fi @if [ ! -d tools/docutils ]; then \ echo "Checking out Docutils..."; \ @@ -86,13 +87,14 @@ changes: build linkcheck: BUILDER = linkcheck linkcheck: build - @echo "Link check complete; look for any errors in the above output " \ + @echo "Link check complete; look for any errors in the above output" \ "or in build/$(BUILDER)/output.txt" suspicious: BUILDER = suspicious suspicious: build - @echo "Suspicious check complete; look for any errors in the above output " \ - "or in build/$(BUILDER)/suspicious.txt" + @echo "Suspicious check complete; look for any errors in the above output" \ + "or in build/$(BUILDER)/suspicious.csv. If all issues are false" \ + "positives, append that file to tools/sphinxext/susp-ignored.csv." coverage: BUILDER = coverage coverage: build @@ -100,13 +102,13 @@ coverage: build doctest: BUILDER = doctest doctest: build - @echo "Testing of doctests in the sources finished, look at the " \ + @echo "Testing of doctests in the sources finished, look at the" \ "results in build/doctest/output.txt" pydoc-topics: BUILDER = pydoc-topics pydoc-topics: build - @echo "Building finished; now copy build/pydoc-topics/pydoc_topics.py " \ - "into the Lib/ directory" + @echo "Building finished; now copy build/pydoc-topics/topics.py" \ + "to Lib/pydoc_data/topics.py" htmlview: html $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" @@ -119,7 +121,7 @@ clean: -rm -rf tools/docutils dist: - -rm -rf dist + rm -rf dist mkdir -p dist # archive the HTML @@ -139,9 +141,9 @@ dist: (cd dist; zip -q -r -9 python-$(DISTVERSION)-docs-text.zip python-$(DISTVERSION)-docs-text) rm -r dist/python-$(DISTVERSION)-docs-text rm dist/python-$(DISTVERSION)-docs-text.tar - + # archive the A4 latex - -rm -r build/latex + rm -rf build/latex make latex PAPER=a4 -sed -i 's/makeindex/makeindex -q/' build/latex/Makefile (cd build/latex; make clean && make all-pdf && make FMT=pdf zip bz2) @@ -149,7 +151,7 @@ dist: cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-a4.tar.bz2 # archive the letter latex - rm -r build/latex + rm -rf build/latex make latex PAPER=letter -sed -i 's/makeindex/makeindex -q/' build/latex/Makefile (cd build/latex; make clean && make all-pdf && make FMT=pdf zip bz2) @@ -159,6 +161,9 @@ dist: check: $(PYTHON) tools/rstlint.py -i tools +serve: + ../Tools/scripts/serve.py build/html + # Targets for daily automated doc build # for development releases: always build diff --git a/Doc/README.txt b/Doc/README.txt index d32d1ed..0e70be5 100644 --- a/Doc/README.txt +++ b/Doc/README.txt @@ -73,7 +73,7 @@ Without make You'll need to install the Sphinx package, either by checking it out via :: - svn co http://svn.python.org/projects/external/Sphinx-0.6.5/sphinx tools/sphinx + svn co http://svn.python.org/projects/external/Sphinx-0.6.7/sphinx tools/sphinx or by installing it from PyPI. diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst index dbfae4f..a75b07b 100644 --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -2,10 +2,11 @@ .. _bufferobjects: -Buffer Objects --------------- +Buffers and Memoryview Objects +------------------------------ .. sectionauthor:: Greg Stein <gstein@lyra.org> +.. sectionauthor:: Benjamin Peterson .. index:: @@ -265,20 +266,6 @@ Buffer related functions :cdata:`~Py_buffer.format`. -.. cfunction:: int PyObject_CopyToObject(PyObject *obj, void *buf, Py_ssize_t len, char fortran) - - Copy *len* bytes of data pointed to by the contiguous chunk of memory - pointed to by *buf* into the buffer exported by obj. The buffer must of - course be writable. Return 0 on success and return -1 and raise an error - on failure. If the object does not have a writable buffer, then an error - is raised. If *fortran* is ``'F'``, then if the object is - multi-dimensional, then the data will be copied into the array in - Fortran-style (first dimension varies the fastest). If *fortran* is - ``'C'``, then the data will be copied into the array in C-style (last - dimension varies the fastest). If *fortran* is ``'A'``, then it does not - matter and the copy will be made in whatever way is more efficient. - - .. cfunction:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) Return 1 if the memory defined by the *view* is C-style (*fortran* is @@ -300,12 +287,56 @@ Buffer related functions length. Return 0 on success and -1 (with raising an error) on error. +MemoryView objects +================== + +.. versionadded:: 2.7 + +A :class:`memoryview` object exposes the new C level buffer interface as a +Python object which can then be passed around like any other object. + +.. cfunction:: PyObject *PyMemoryView_FromObject(PyObject *obj) + + Create a memoryview object from an object that defines the new buffer + interface. + + +.. cfunction:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) + + Create a memoryview object wrapping the given buffer-info structure *view*. + The memoryview object then owns the buffer, which means you shouldn't + try to release it yourself: it will be released on deallocation of the + memoryview object. + + +.. cfunction:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) + + Create a memoryview object to a contiguous chunk of memory (in either + 'C' or 'F'ortran *order*) from an object that defines the buffer + interface. If memory is contiguous, the memoryview object points to the + original memory. Otherwise copy is made and the memoryview points to a + new bytes object. + + +.. cfunction:: int PyMemoryView_Check(PyObject *obj) + + Return true if the object *obj* is a memoryview object. It is not + currently allowed to create subclasses of :class:`memoryview`. + + +.. cfunction:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj) + + Return a pointer to the buffer-info structure wrapped by the given + object. The object **must** be a memoryview instance; this macro doesn't + check its type, you must do it yourself or you will risk crashes. + + Old-style buffer objects ======================== .. index:: single: PyBufferProcs -More information on the buffer interface is provided in the section +More information on the old buffer interface is provided in the section :ref:`buffer-structs`, under the description for :ctype:`PyBufferProcs`. A "buffer object" is defined in the :file:`bufferobject.h` header (included by diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst new file mode 100644 index 0000000..2939314 --- /dev/null +++ b/Doc/c-api/capsule.rst @@ -0,0 +1,150 @@ +.. highlightlang:: c + +.. _capsules: + +Capsules +-------- + +.. index:: object: Capsule + +Refer to :ref:`using-capsules` for more information on using these objects. + + +.. ctype:: PyCapsule + + This subtype of :ctype:`PyObject` represents an opaque value, useful for C + extension modules who need to pass an opaque value (as a :ctype:`void\*` + pointer) through Python code to other C code. It is often used to make a C + function pointer defined in one module available to other modules, so the + regular import mechanism can be used to access C APIs defined in dynamically + loaded modules. + +.. ctype:: PyCapsule_Destructor + + The type of a destructor callback for a capsule. Defined as:: + + typedef void (*PyCapsule_Destructor)(PyObject *); + + See :cfunc:`PyCapsule_New` for the semantics of PyCapsule_Destructor + callbacks. + + +.. cfunction:: int PyCapsule_CheckExact(PyObject *p) + + Return true if its argument is a :ctype:`PyCapsule`. + + +.. cfunction:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor) + + Create a :ctype:`PyCapsule` encapsulating the *pointer*. The *pointer* + argument may not be *NULL*. + + On failure, set an exception and return *NULL*. + + The *name* string may either be *NULL* or a pointer to a valid C string. If + non-*NULL*, this string must outlive the capsule. (Though it is permitted to + free it inside the *destructor*.) + + If the *destructor* argument is not *NULL*, it will be called with the + capsule as its argument when it is destroyed. + + If this capsule will be stored as an attribute of a module, the *name* should + be specified as ``modulename.attributename``. This will enable other modules + to import the capsule using :cfunc:`PyCapsule_Import`. + + +.. cfunction:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name) + + Retrieve the *pointer* stored in the capsule. On failure, set an exception + and return *NULL*. + + The *name* parameter must compare exactly to the name stored in the capsule. + If the name stored in the capsule is *NULL*, the *name* passed in must also + be *NULL*. Python uses the C function :cfunc:`strcmp` to compare capsule + names. + + +.. cfunction:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule) + + Return the current destructor stored in the capsule. On failure, set an + exception and return *NULL*. + + It is legal for a capsule to have a *NULL* destructor. This makes a *NULL* + return code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or + :cfunc:`PyErr_Occurred` to disambiguate. + + +.. cfunction:: void* PyCapsule_GetContext(PyObject *capsule) + + Return the current context stored in the capsule. On failure, set an + exception and return *NULL*. + + It is legal for a capsule to have a *NULL* context. This makes a *NULL* + return code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or + :cfunc:`PyErr_Occurred` to disambiguate. + + +.. cfunction:: const char* PyCapsule_GetName(PyObject *capsule) + + Return the current name stored in the capsule. On failure, set an exception + and return *NULL*. + + It is legal for a capsule to have a *NULL* name. This makes a *NULL* return + code somewhat ambiguous; use :cfunc:`PyCapsule_IsValid` or + :cfunc:`PyErr_Occurred` to disambiguate. + + +.. cfunction:: void* PyCapsule_Import(const char *name, int no_block) + + Import a pointer to a C object from a capsule attribute in a module. The + *name* parameter should specify the full name to the attribute, as in + ``module.attribute``. The *name* stored in the capsule must match this + string exactly. If *no_block* is true, import the module without blocking + (using :cfunc:`PyImport_ImportModuleNoBlock`). If *no_block* is false, + import the module conventionally (using :cfunc:`PyImport_ImportModule`). + + Return the capsule's internal *pointer* on success. On failure, set an + exception and return *NULL*. However, if :cfunc:`PyCapsule_Import` failed to + import the module, and *no_block* was true, no exception is set. + +.. cfunction:: int PyCapsule_IsValid(PyObject *capsule, const char *name) + + Determines whether or not *capsule* is a valid capsule. A valid capsule is + non-*NULL*, passes :cfunc:`PyCapsule_CheckExact`, has a non-*NULL* pointer + stored in it, and its internal name matches the *name* parameter. (See + :cfunc:`PyCapsule_GetPointer` for information on how capsule names are + compared.) + + In other words, if :cfunc:`PyCapsule_IsValid` returns a true value, calls to + any of the accessors (any function starting with :cfunc:`PyCapsule_Get`) are + guaranteed to succeed. + + Return a nonzero value if the object is valid and matches the name passed in. + Return 0 otherwise. This function will not fail. + +.. cfunction:: int PyCapsule_SetContext(PyObject *capsule, void *context) + + Set the context pointer inside *capsule* to *context*. + + Return 0 on success. Return nonzero and set an exception on failure. + +.. cfunction:: int PyCapsule_SetDestructor(PyObject *capsule, PyCapsule_Destructor destructor) + + Set the destructor inside *capsule* to *destructor*. + + Return 0 on success. Return nonzero and set an exception on failure. + +.. cfunction:: int PyCapsule_SetName(PyObject *capsule, const char *name) + + Set the name inside *capsule* to *name*. If non-*NULL*, the name must + outlive the capsule. If the previous *name* stored in the capsule was not + *NULL*, no attempt is made to free it. + + Return 0 on success. Return nonzero and set an exception on failure. + +.. cfunction:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer) + + Set the void pointer inside *capsule* to *pointer*. The pointer may not be + *NULL*. + + Return 0 on success. Return nonzero and set an exception on failure. diff --git a/Doc/c-api/cobject.rst b/Doc/c-api/cobject.rst index 10f7bba..73fbbf5 100644 --- a/Doc/c-api/cobject.rst +++ b/Doc/c-api/cobject.rst @@ -7,8 +7,11 @@ CObjects .. index:: object: CObject -Refer to :ref:`using-cobjects` for more information on using these objects. +.. warning:: + + The CObject API is deprecated as of Python 2.7. Please switch to the new + :ref:`capsules` API. .. ctype:: PyCObject diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst new file mode 100644 index 0000000..c6ca8c5 --- /dev/null +++ b/Doc/c-api/code.rst @@ -0,0 +1,50 @@ +.. highlightlang:: c + +.. _codeobjects: + +Code Objects +------------ + +.. sectionauthor:: Jeffrey Yasskin <jyasskin@gmail.com> + + +.. index:: + object: code + +Code objects are a low-level detail of the CPython implementation. +Each one represents a chunk of executable code that hasn't yet been +bound into a function. + +.. ctype:: PyCodeObject + + The C structure of the objects used to describe code objects. The + fields of this type are subject to change at any time. + + +.. cvar:: PyTypeObject PyCode_Type + + This is an instance of :ctype:`PyTypeObject` representing the Python + :class:`code` type. + + +.. cfunction:: int PyCode_Check(PyObject *co) + + Return true if *co* is a :class:`code` object + +.. cfunction:: int PyCode_GetNumFree(PyObject *co) + + Return the number of free variables in *co*. + +.. cfunction:: PyCodeObject *PyCode_New(int argcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab) + + Return a new code object. If you need a dummy code object to + create a frame, use :cfunc:`PyCode_NewEmpty` instead. Calling + :cfunc:`PyCode_New` directly can bind you to a precise Python + version since the definition of the bytecode changes often. + + +.. cfunction:: int PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) + + Return a new empty code object with the specified filename, + function name, and first line number. It is illegal to + :keyword:`exec` or :func:`eval` the resulting code object. diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst new file mode 100644 index 0000000..2597d38 --- /dev/null +++ b/Doc/c-api/codec.rst @@ -0,0 +1,118 @@ +.. _codec-registry: + +Codec registry and support functions +==================================== + +.. cfunction:: int PyCodec_Register(PyObject *search_function) + + Register a new codec search function. + + As side effect, this tries to load the :mod:`encodings` package, if not yet + done, to make sure that it is always first in the list of search functions. + +.. cfunction:: int PyCodec_KnownEncoding(const char *encoding) + + Return ``1`` or ``0`` depending on whether there is a registered codec for + the given *encoding*. + +.. cfunction:: PyObject* PyCodec_Encode(PyObject *object, const char *encoding, const char *errors) + + Generic codec based encoding API. + + *object* is passed through the encoder function found for the given + *encoding* using the error handling method defined by *errors*. *errors* may + be *NULL* to use the default method defined for the codec. Raises a + :exc:`LookupError` if no encoder can be found. + +.. cfunction:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors) + + Generic codec based decoding API. + + *object* is passed through the decoder function found for the given + *encoding* using the error handling method defined by *errors*. *errors* may + be *NULL* to use the default method defined for the codec. Raises a + :exc:`LookupError` if no encoder can be found. + + +Codec lookup API +---------------- + +In the following functions, the *encoding* string is looked up converted to all +lower-case characters, which makes encodings looked up through this mechanism +effectively case-insensitive. If no codec is found, a :exc:`KeyError` is set +and *NULL* returned. + +.. cfunction:: PyObject* PyCodec_Encoder(const char *encoding) + + Get an encoder function for the given *encoding*. + +.. cfunction:: PyObject* PyCodec_Decoder(const char *encoding) + + Get a decoder function for the given *encoding*. + +.. cfunction:: PyObject* PyCodec_IncrementalEncoder(const char *encoding, const char *errors) + + Get an :class:`IncrementalEncoder` object for the given *encoding*. + +.. cfunction:: PyObject* PyCodec_IncrementalDecoder(const char *encoding, const char *errors) + + Get an :class:`IncrementalDecoder` object for the given *encoding*. + +.. cfunction:: PyObject* PyCodec_StreamReader(const char *encoding, PyObject *stream, const char *errors) + + Get a :class:`StreamReader` factory function for the given *encoding*. + +.. cfunction:: PyObject* PyCodec_StreamWriter(const char *encoding, PyObject *stream, const char *errors) + + Get a :class:`StreamWriter` factory function for the given *encoding*. + + +Registry API for Unicode encoding error handlers +------------------------------------------------ + +.. cfunction:: int PyCodec_RegisterError(const char *name, PyObject *error) + + Register the error handling callback function *error* under the given *name*. + This callback function will be called by a codec when it encounters + unencodable characters/undecodable bytes and *name* is specified as the error + parameter in the call to the encode/decode function. + + The callback gets a single argument, an instance of + :exc:`UnicodeEncodeError`, :exc:`UnicodeDecodeError` or + :exc:`UnicodeTranslateError` that holds information about the problematic + sequence of characters or bytes and their offset in the original string (see + :ref:`unicodeexceptions` for functions to extract this information). The + callback must either raise the given exception, or return a two-item tuple + containing the replacement for the problematic sequence, and an integer + giving the offset in the original string at which encoding/decoding should be + resumed. + + Return ``0`` on success, ``-1`` on error. + +.. cfunction:: PyObject* PyCodec_LookupError(const char *name) + + Lookup the error handling callback function registered under *name*. As a + special case *NULL* can be passed, in which case the error handling callback + for "strict" will be returned. + +.. cfunction:: PyObject* PyCodec_StrictErrors(PyObject *exc) + + Raise *exc* as an exception. + +.. cfunction:: PyObject* PyCodec_IgnoreErrors(PyObject *exc) + + Ignore the unicode error, skipping the faulty input. + +.. cfunction:: PyObject* PyCodec_ReplaceErrors(PyObject *exc) + + Replace the unicode encode error with ``?`` or ``U+FFFD``. + +.. cfunction:: PyObject* PyCodec_XMLCharRefReplaceErrors(PyObject *exc) + + Replace the unicode encode error with XML character references. + +.. cfunction:: PyObject* PyCodec_BackslashReplaceErrors(PyObject *exc) + + Replace the unicode encode error with backslash escapes (``\x``, ``\u`` and + ``\U``). + diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst index 6958646..5ee611b 100644 --- a/Doc/c-api/concrete.rst +++ b/Doc/c-api/concrete.rst @@ -100,8 +100,10 @@ Other Objects descriptor.rst slice.rst weakref.rst + capsule.rst cobject.rst cell.rst gen.rst datetime.rst set.rst + code.rst diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst index 0c81bc0..b32100b 100644 --- a/Doc/c-api/conversion.rst +++ b/Doc/c-api/conversion.rst @@ -51,6 +51,40 @@ The return value (*rv*) for these functions should be interpreted as follows: The following functions provide locale-independent string to number conversions. +.. cfunction:: double PyOS_string_to_double(const char *s, char **endptr, PyObject *overflow_exception) + + Convert a string ``s`` to a :ctype:`double`, raising a Python + exception on failure. The set of accepted strings corresponds to + the set of strings accepted by Python's :func:`float` constructor, + except that ``s`` must not have leading or trailing whitespace. + The conversion is independent of the current locale. + + If ``endptr`` is ``NULL``, convert the whole string. Raise + ValueError and return ``-1.0`` if the string is not a valid + representation of a floating-point number. + + If endptr is not ``NULL``, convert as much of the string as + possible and set ``*endptr`` to point to the first unconverted + character. If no initial segment of the string is the valid + representation of a floating-point number, set ``*endptr`` to point + to the beginning of the string, raise ValueError, and return + ``-1.0``. + + If ``s`` represents a value that is too large to store in a float + (for example, ``"1e500"`` is such a string on many platforms) then + if ``overflow_exception`` is ``NULL`` return ``Py_HUGE_VAL`` (with + an appropriate sign) and don't set any exception. Otherwise, + ``overflow_exception`` must point to a Python exception object; + raise that exception and return ``-1.0``. In both cases, set + ``*endptr`` to point to the first character after the converted value. + + If any other error occurs during the conversion (for example an + out-of-memory error), set the appropriate Python exception and + return ``-1.0``. + + .. versionadded:: 2.7 + + .. cfunction:: double PyOS_ascii_strtod(const char *nptr, char **endptr) Convert a string to a :ctype:`double`. This function behaves like the Standard C @@ -60,12 +94,16 @@ The following functions provide locale-independent string to number conversions. :cfunc:`PyOS_ascii_strtod` should typically be used for reading configuration files or other non-user input that should be locale independent. + See the Unix man page :manpage:`strtod(2)` for details. + .. versionadded:: 2.4 - See the Unix man page :manpage:`strtod(2)` for details. + .. deprecated:: 2.7 + Use :cfunc:`PyOS_string_to_double` instead. + -.. cfunction:: char * PyOS_ascii_formatd(char *buffer, size_t buf_len, const char *format, double d) +.. cfunction:: char* PyOS_ascii_formatd(char *buffer, size_t buf_len, const char *format, double d) Convert a :ctype:`double` to a string using the ``'.'`` as the decimal separator. *format* is a :cfunc:`printf`\ -style format string specifying the @@ -76,18 +114,58 @@ The following functions provide locale-independent string to number conversions. the conversion failed. .. versionadded:: 2.4 + .. deprecated:: 2.7 + This function is removed in Python 2.7 and 3.1. Use :func:`PyOS_double_to_string` + instead. + + +.. cfunction:: char* PyOS_double_to_string(double val, char format_code, int precision, int flags, int *ptype) + + Convert a :ctype:`double` *val* to a string using supplied + *format_code*, *precision*, and *flags*. + + *format_code* must be one of ``'e'``, ``'E'``, ``'f'``, ``'F'``, + ``'g'``, ``'G'`` or ``'r'``. For ``'r'``, the supplied *precision* + must be 0 and is ignored. The ``'r'`` format code specifies the + standard :func:`repr` format. + + *flags* can be zero or more of the values *Py_DTSF_SIGN*, + *Py_DTSF_ADD_DOT_0*, or *Py_DTSF_ALT*, or-ed together: + + * *Py_DTSF_SIGN* means to always precede the returned string with a sign + character, even if *val* is non-negative. + + * *Py_DTSF_ADD_DOT_0* means to ensure that the returned string will not look + like an integer. + + * *Py_DTSF_ALT* means to apply "alternate" formatting rules. See the + documentation for the :cfunc:`PyOS_snprintf` ``'#'`` specifier for + details. + + If *ptype* is non-NULL, then the value it points to will be set to one of + *Py_DTST_FINITE*, *Py_DTST_INFINITE*, or *Py_DTST_NAN*, signifying that + *val* is a finite number, an infinite number, or not a number, respectively. + + The return value is a pointer to *buffer* with the converted string or + *NULL* if the conversion failed. The caller is responsible for freeing the + returned string by calling :cfunc:`PyMem_Free`. + + .. versionadded:: 2.7 .. cfunction:: double PyOS_ascii_atof(const char *nptr) Convert a string to a :ctype:`double` in a locale-independent way. + See the Unix man page :manpage:`atof(2)` for details. + .. versionadded:: 2.4 - See the Unix man page :manpage:`atof(2)` for details. + .. deprecated:: 3.1 + Use :cfunc:`PyOS_string_to_double` instead. -.. cfunction:: char * PyOS_stricmp(char *s1, char *s2) +.. cfunction:: char* PyOS_stricmp(char *s1, char *s2) Case insensitive comparison of strings. The function works almost identically to :cfunc:`strcmp` except that it ignores the case. @@ -95,7 +173,7 @@ The following functions provide locale-independent string to number conversions. .. versionadded:: 2.6 -.. cfunction:: char * PyOS_strnicmp(char *s1, char *s2, Py_ssize_t size) +.. cfunction:: char* PyOS_strnicmp(char *s1, char *s2, Py_ssize_t size) Case insensitive comparison of strings. The function works almost identically to :cfunc:`strncmp` except that it ignores the case. diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index ed07f66..066e7f1 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -199,8 +199,8 @@ Dictionary Objects .. cfunction:: int PyDict_Merge(PyObject *a, PyObject *b, int override) Iterate over mapping object *b* adding key-value pairs to dictionary *a*. - *b* may be a dictionary, or any object supporting :func:`PyMapping_Keys` - and :func:`PyObject_GetItem`. If *override* is true, existing pairs in *a* + *b* may be a dictionary, or any object supporting :cfunc:`PyMapping_Keys` + and :cfunc:`PyObject_GetItem`. If *override* is true, existing pairs in *a* will be replaced if a matching key is found in *b*, otherwise pairs will only be added if there is not a matching key in *a*. Return ``0`` on success or ``-1`` if an exception was raised. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 141f357..e9ced0b 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -151,64 +151,10 @@ is a separate error indicator for each thread. .. cfunction:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...) - This function sets the error indicator and returns *NULL*. *exception* should be - a Python exception (class, not an instance). *format* should be a string, - containing format codes, similar to :cfunc:`printf`. The ``width.precision`` - before a format code is parsed, but the width part is ignored. - - .. % This should be exactly the same as the table in PyString_FromFormat. - .. % One should just refer to the other. - .. % The descriptions for %zd and %zu are wrong, but the truth is complicated - .. % because not all compilers support the %z width modifier -- we fake it - .. % when necessary via interpolating PY_FORMAT_SIZE_T. - .. % %u, %lu, %zu should have "new in Python 2.5" blurbs. - - +-------------------+---------------+--------------------------------+ - | Format Characters | Type | Comment | - +===================+===============+================================+ - | :attr:`%%` | *n/a* | The literal % character. | - +-------------------+---------------+--------------------------------+ - | :attr:`%c` | int | A single character, | - | | | represented as an C int. | - +-------------------+---------------+--------------------------------+ - | :attr:`%d` | int | Exactly equivalent to | - | | | ``printf("%d")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%u` | unsigned int | Exactly equivalent to | - | | | ``printf("%u")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%ld` | long | Exactly equivalent to | - | | | ``printf("%ld")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%lu` | unsigned long | Exactly equivalent to | - | | | ``printf("%lu")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%zd` | Py_ssize_t | Exactly equivalent to | - | | | ``printf("%zd")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%zu` | size_t | Exactly equivalent to | - | | | ``printf("%zu")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%i` | int | Exactly equivalent to | - | | | ``printf("%i")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%x` | int | Exactly equivalent to | - | | | ``printf("%x")``. | - +-------------------+---------------+--------------------------------+ - | :attr:`%s` | char\* | A null-terminated C character | - | | | array. | - +-------------------+---------------+--------------------------------+ - | :attr:`%p` | void\* | The hex representation of a C | - | | | pointer. Mostly equivalent to | - | | | ``printf("%p")`` except that | - | | | it is guaranteed to start with | - | | | the literal ``0x`` regardless | - | | | of what the platform's | - | | | ``printf`` yields. | - +-------------------+---------------+--------------------------------+ - - An unrecognized format character causes all the rest of the format string to be - copied as-is to the result string, and any extra arguments discarded. + This function sets the error indicator and returns *NULL*. *exception* + should be a Python exception class. The *format* and subsequent + parameters help format the error message; they have the same meaning and + values as in :cfunc:`PyString_FromFormat`. .. cfunction:: void PyErr_SetNone(PyObject *type) @@ -417,6 +363,15 @@ is a separate error indicator for each thread. argument can be used to specify a dictionary of class variables and methods. +.. cfunction:: PyObject* PyErr_NewExceptionWithDoc(char *name, char *doc, PyObject *base, PyObject *dict) + + Same as :cfunc:`PyErr_NewException`, except that the new exception class can + easily be given a docstring: If *doc* is non-*NULL*, it will be used as the + docstring for the exception class. + + .. versionadded:: 2.7 + + .. cfunction:: void PyErr_WriteUnraisable(PyObject *obj) This utility function prints a warning message to ``sys.stderr`` when an @@ -429,6 +384,83 @@ is a separate error indicator for each thread. the warning message. +.. _unicodeexceptions: + +Unicode Exception Objects +========================= + +The following functions are used to create and modify Unicode exceptions from C. + +.. cfunction:: PyObject* PyUnicodeDecodeError_Create(const char *encoding, const char *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) + + Create a :class:`UnicodeDecodeError` object with the attributes *encoding*, + *object*, *length*, *start*, *end* and *reason*. + +.. cfunction:: PyObject* PyUnicodeEncodeError_Create(const char *encoding, const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) + + Create a :class:`UnicodeEncodeError` object with the attributes *encoding*, + *object*, *length*, *start*, *end* and *reason*. + +.. cfunction:: PyObject* PyUnicodeTranslateError_Create(const Py_UNICODE *object, Py_ssize_t length, Py_ssize_t start, Py_ssize_t end, const char *reason) + + Create a :class:`UnicodeTranslateError` object with the attributes *object*, + *length*, *start*, *end* and *reason*. + +.. cfunction:: PyObject* PyUnicodeDecodeError_GetEncoding(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetEncoding(PyObject *exc) + + Return the *encoding* attribute of the given exception object. + +.. cfunction:: PyObject* PyUnicodeDecodeError_GetObject(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetObject(PyObject *exc) + PyObject* PyUnicodeTranslateError_GetObject(PyObject *exc) + + Return the *object* attribute of the given exception object. + +.. cfunction:: int PyUnicodeDecodeError_GetStart(PyObject *exc, Py_ssize_t *start) + int PyUnicodeEncodeError_GetStart(PyObject *exc, Py_ssize_t *start) + int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start) + + Get the *start* attribute of the given exception object and place it into + *\*start*. *start* must not be *NULL*. Return ``0`` on success, ``-1`` on + failure. + +.. cfunction:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start) + int PyUnicodeEncodeError_SetStart(PyObject *exc, Py_ssize_t start) + int PyUnicodeTranslateError_SetStart(PyObject *exc, Py_ssize_t start) + + Set the *start* attribute of the given exception object to *start*. Return + ``0`` on success, ``-1`` on failure. + +.. cfunction:: int PyUnicodeDecodeError_GetEnd(PyObject *exc, Py_ssize_t *end) + int PyUnicodeEncodeError_GetEnd(PyObject *exc, Py_ssize_t *end) + int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end) + + Get the *end* attribute of the given exception object and place it into + *\*end*. *end* must not be *NULL*. Return ``0`` on success, ``-1`` on + failure. + +.. cfunction:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end) + int PyUnicodeEncodeError_SetEnd(PyObject *exc, Py_ssize_t end) + int PyUnicodeTranslateError_SetEnd(PyObject *exc, Py_ssize_t end) + + Set the *end* attribute of the given exception object to *end*. Return ``0`` + on success, ``-1`` on failure. + +.. cfunction:: PyObject* PyUnicodeDecodeError_GetReason(PyObject *exc) + PyObject* PyUnicodeEncodeError_GetReason(PyObject *exc) + PyObject* PyUnicodeTranslateError_GetReason(PyObject *exc) + + Return the *reason* attribute of the given exception object. + +.. cfunction:: int PyUnicodeDecodeError_SetReason(PyObject *exc, const char *reason) + int PyUnicodeEncodeError_SetReason(PyObject *exc, const char *reason) + int PyUnicodeTranslateError_SetReason(PyObject *exc, const char *reason) + + Set the *reason* attribute of the given exception object to *reason*. Return + ``0`` on success, ``-1`` on failure. + + Recursion Control ================= @@ -578,14 +610,10 @@ Notes: .. versionadded:: 2.5 -Deprecation of String Exceptions -================================ - -.. index:: single: BaseException (built-in exception) - -All exceptions built into Python or provided in the standard library are derived -from :exc:`BaseException`. +String Exceptions +================= -String exceptions are still supported in the interpreter to allow existing code -to run unmodified, but this will also change in a future release. +.. versionchanged:: 2.6 + All exceptions to be raised or caught must be derived from :exc:`BaseException`. + Trying to raise a string exception now raises :exc:`TypeError`. diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst index 5d74355..bdb8c49 100644 --- a/Doc/c-api/file.rst +++ b/Doc/c-api/file.rst @@ -63,7 +63,7 @@ change in future releases of Python. Return the file object associated with *p* as a :ctype:`FILE\*`. If the caller will ever use the returned :ctype:`FILE\*` object while - the GIL is released it must also call the :cfunc:`PyFile_IncUseCount` and + the :term:`GIL` is released it must also call the :cfunc:`PyFile_IncUseCount` and :cfunc:`PyFile_DecUseCount` functions described below as appropriate. @@ -76,10 +76,19 @@ change in future releases of Python. finished with the :ctype:`FILE\*`. Otherwise the file object will never be closed by Python. - The GIL must be held while calling this function. + The :term:`GIL` must be held while calling this function. - The suggested use is to call this after :cfunc:`PyFile_AsFile` just before - you release the GIL. + The suggested use is to call this after :cfunc:`PyFile_AsFile` and before + you release the GIL:: + + FILE *fp = PyFile_AsFile(p); + PyFile_IncUseCount(p); + /* ... */ + Py_BEGIN_ALLOW_THREADS + do_something(fp); + Py_END_ALLOW_THREADS + /* ... */ + PyFile_DecUseCount(p); .. versionadded:: 2.6 @@ -90,7 +99,8 @@ change in future releases of Python. indicate that the caller is done with its own use of the :ctype:`FILE\*`. This may only be called to undo a prior call to :cfunc:`PyFile_IncUseCount`. - The GIL must be held while calling this function. + The :term:`GIL` must be held while calling this function (see the example + above). .. versionadded:: 2.6 diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst index c4099a3..295fb90 100644 --- a/Doc/c-api/float.rst +++ b/Doc/c-api/float.rst @@ -92,3 +92,27 @@ Floating Point Objects be freed. .. versionadded:: 2.6 + + +.. cfunction:: void PyFloat_AsString(char *buf, PyFloatObject *v) + + Convert the argument *v* to a string, using the same rules as + :func:`str`. The length of *buf* should be at least 100. + + This function is unsafe to call because it writes to a buffer whose + length it does not know. + + .. deprecated:: 2.7 + Use :func:`PyObject_Str` or :func:`PyOS_double_to_string` instead. + + +.. cfunction:: void PyFloat_AsReprString(char *buf, PyFloatObject *v) + + Same as PyFloat_AsString, except uses the same rules as + :func:`repr`. The length of *buf* should be at least 100. + + This function is unsafe to call because it writes to a buffer whose + length it does not know. + + .. deprecated:: 2.7 + Use :func:`PyObject_Repr` or :func:`PyOS_double_to_string` instead. diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst index 558896c..3da1415 100644 --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -241,7 +241,7 @@ Importing Modules tricks with this to provide a dynamically created collection of frozen modules. -.. cfunction:: int PyImport_AppendInittab(char *name, void (*initfunc)(void)) +.. cfunction:: int PyImport_AppendInittab(const char *name, void (*initfunc)(void)) Add a single module to the existing table of built-in modules. This is a convenience wrapper around :cfunc:`PyImport_ExtendInittab`, returning ``-1`` if diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 5d4da22..fd6a9a2 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -8,6 +8,10 @@ Initialization, Finalization, and Threads ***************************************** +Initializing and finalizing the interpreter +=========================================== + + .. cfunction:: void Py_Initialize() .. index:: @@ -83,85 +87,8 @@ Initialization, Finalization, and Threads :cfunc:`Py_Finalize` more than once. -.. cfunction:: PyThreadState* Py_NewInterpreter() - - .. index:: - module: __builtin__ - module: __main__ - module: sys - single: stdout (in module sys) - single: stderr (in module sys) - single: stdin (in module sys) - - Create a new sub-interpreter. This is an (almost) totally separate environment - for the execution of Python code. In particular, the new interpreter has - separate, independent versions of all imported modules, including the - fundamental modules :mod:`__builtin__`, :mod:`__main__` and :mod:`sys`. The - table of loaded modules (``sys.modules``) and the module search path - (``sys.path``) are also separate. The new environment has no ``sys.argv`` - variable. It has new standard I/O stream file objects ``sys.stdin``, - ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying - :ctype:`FILE` structures in the C library). - - The return value points to the first thread state created in the new - sub-interpreter. This thread state is made in the current thread state. - Note that no actual thread is created; see the discussion of thread states - below. If creation of the new interpreter is unsuccessful, *NULL* is - returned; no exception is set since the exception state is stored in the - current thread state and there may not be a current thread state. (Like all - other Python/C API functions, the global interpreter lock must be held before - calling this function and is still held when it returns; however, unlike most - other Python/C API functions, there needn't be a current thread state on - entry.) - - .. index:: - single: Py_Finalize() - single: Py_Initialize() - - Extension modules are shared between (sub-)interpreters as follows: the first - time a particular extension is imported, it is initialized normally, and a - (shallow) copy of its module's dictionary is squirreled away. When the same - extension is imported by another (sub-)interpreter, a new module is initialized - and filled with the contents of this copy; the extension's ``init`` function is - not called. Note that this is different from what happens when an extension is - imported after the interpreter has been completely re-initialized by calling - :cfunc:`Py_Finalize` and :cfunc:`Py_Initialize`; in that case, the extension's - ``initmodule`` function *is* called again. - - .. index:: single: close() (in module os) - - **Bugs and caveats:** Because sub-interpreters (and the main interpreter) are - part of the same process, the insulation between them isn't perfect --- for - example, using low-level file operations like :func:`os.close` they can - (accidentally or maliciously) affect each other's open files. Because of the - way extensions are shared between (sub-)interpreters, some extensions may not - work properly; this is especially likely when the extension makes use of - (static) global variables, or when the extension manipulates its module's - dictionary after its initialization. It is possible to insert objects created - in one sub-interpreter into a namespace of another sub-interpreter; this should - be done with great care to avoid sharing user-defined functions, methods, - instances or classes between sub-interpreters, since import operations executed - by such objects may affect the wrong (sub-)interpreter's dictionary of loaded - modules. (XXX This is a hard-to-fix bug that will be addressed in a future - release.) - - Also note that the use of this functionality is incompatible with extension - modules such as PyObjC and ctypes that use the :cfunc:`PyGILState_\*` APIs (and - this is inherent in the way the :cfunc:`PyGILState_\*` functions work). Simple - things may work, but confusing behavior will always be near. - - -.. cfunction:: void Py_EndInterpreter(PyThreadState *tstate) - - .. index:: single: Py_Finalize() - - Destroy the (sub-)interpreter represented by the given thread state. The given - thread state must be the current thread state. See the discussion of thread - states below. When the call returns, the current thread state is *NULL*. All - thread states associated with this interpreter are destroyed. (The global - interpreter lock must be held before calling this function and is still held - when it returns.) :cfunc:`Py_Finalize` will destroy all sub-interpreters that - haven't been explicitly destroyed at that point. +Process-wide parameters +======================= .. cfunction:: void Py_SetProgramName(char *name) @@ -386,14 +313,15 @@ Initialization, Finalization, and Threads .. cfunction:: void PySys_SetArgv(int argc, char **argv) - This function works like :cfunc:`PySys_SetArgv` with *updatepath* set to 1. + This function works like :cfunc:`PySys_SetArgvEx` with *updatepath* set to 1. .. cfunction:: void Py_SetPythonHome(char *home) Set the default "home" directory, that is, the location of the standard - Python libraries. The libraries are searched in - :file:`{home}/lib/python{version}` and :file:`{home}/lib/python{version}`. + Python libraries. See :envvar:`PYTHONHOME` for the meaning of the + argument string. + The argument should point to a zero-terminated character string in static storage whose contents will not change for the duration of the program's execution. No code in the Python interpreter will change the contents of @@ -413,13 +341,14 @@ Thread State and the Global Interpreter Lock ============================================ .. index:: + single: GIL single: global interpreter lock single: interpreter lock single: lock, interpreter -The Python interpreter is not fully thread safe. In order to support -multi-threaded Python programs, there's a global lock, called the :dfn:`global -interpreter lock` or :dfn:`GIL`, that must be held by the current thread before +The Python interpreter is not fully thread-safe. In order to support +multi-threaded Python programs, there's a global lock, called the :term:`global +interpreter lock` or :term:`GIL`, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the @@ -427,39 +356,38 @@ reference count could end up being incremented only once instead of twice. .. index:: single: setcheckinterval() (in module sys) -Therefore, the rule exists that only the thread that has acquired the global -interpreter lock may operate on Python objects or call Python/C API functions. -In order to support multi-threaded Python programs, the interpreter regularly -releases and reacquires the lock --- by default, every 100 bytecode instructions -(this can be changed with :func:`sys.setcheckinterval`). The lock is also -released and reacquired around potentially blocking I/O operations like reading -or writing a file, so that other threads can run while the thread that requests -the I/O is waiting for the I/O operation to complete. +Therefore, the rule exists that only the thread that has acquired the +:term:`GIL` may operate on Python objects or call Python/C API functions. +In order to emulate concurrency of execution, the interpreter regularly +tries to switch threads (see :func:`sys.setcheckinterval`). The lock is also +released around potentially blocking I/O operations like reading or writing +a file, so that other Python threads can run in the meantime. .. index:: single: PyThreadState single: PyThreadState -The Python interpreter needs to keep some bookkeeping information separate per -thread --- for this it uses a data structure called :ctype:`PyThreadState`. -There's one global variable, however: the pointer to the current -:ctype:`PyThreadState` structure. Before the addition of :dfn:`thread-local -storage` (:dfn:`TLS`) the current thread state had to be manipulated -explicitly. +The Python interpreter keeps some thread-specific bookkeeping information +inside a data structure called :ctype:`PyThreadState`. There's also one +global variable pointing to the current :ctype:`PyThreadState`: it can +be retrieved using :cfunc:`PyThreadState_Get`. -This is easy enough in most cases. Most code manipulating the global -interpreter lock has the following simple structure:: +Releasing the GIL from extension code +------------------------------------- + +Most extension code manipulating the :term:`GIL` has the following simple +structure:: Save the thread state in a local variable. Release the global interpreter lock. - ...Do some blocking I/O operation... + ... Do some blocking I/O operation ... Reacquire the global interpreter lock. Restore the thread state from the local variable. This is so common that a pair of macros exists to simplify it:: Py_BEGIN_ALLOW_THREADS - ...Do some blocking I/O operation... + ... Do some blocking I/O operation ... Py_END_ALLOW_THREADS .. index:: @@ -468,9 +396,8 @@ This is so common that a pair of macros exists to simplify it:: The :cmacro:`Py_BEGIN_ALLOW_THREADS` macro opens a new block and declares a hidden local variable; the :cmacro:`Py_END_ALLOW_THREADS` macro closes the -block. Another advantage of using these two macros is that when Python is -compiled without thread support, they are defined empty, thus saving the thread -state and GIL manipulations. +block. These two macros are still available when Python is compiled without +thread support (they simply have an empty expansion). When thread support is enabled, the block above expands to the following code:: @@ -480,65 +407,60 @@ When thread support is enabled, the block above expands to the following code:: ...Do some blocking I/O operation... PyEval_RestoreThread(_save); -Using even lower level primitives, we can get roughly the same effect as -follows:: - - PyThreadState *_save; - - _save = PyThreadState_Swap(NULL); - PyEval_ReleaseLock(); - ...Do some blocking I/O operation... - PyEval_AcquireLock(); - PyThreadState_Swap(_save); - .. index:: single: PyEval_RestoreThread() - single: errno single: PyEval_SaveThread() - single: PyEval_ReleaseLock() - single: PyEval_AcquireLock() - -There are some subtle differences; in particular, :cfunc:`PyEval_RestoreThread` -saves and restores the value of the global variable :cdata:`errno`, since the -lock manipulation does not guarantee that :cdata:`errno` is left alone. Also, -when thread support is disabled, :cfunc:`PyEval_SaveThread` and -:cfunc:`PyEval_RestoreThread` don't manipulate the GIL; in this case, -:cfunc:`PyEval_ReleaseLock` and :cfunc:`PyEval_AcquireLock` are not available. -This is done so that dynamically loaded extensions compiled with thread support -enabled can be loaded by an interpreter that was compiled with disabled thread -support. - -The global interpreter lock is used to protect the pointer to the current thread -state. When releasing the lock and saving the thread state, the current thread -state pointer must be retrieved before the lock is released (since another -thread could immediately acquire the lock and store its own thread state in the -global variable). Conversely, when acquiring the lock and restoring the thread -state, the lock must be acquired before storing the thread state pointer. - -It is important to note that when threads are created from C, they don't have -the global interpreter lock, nor is there a thread state data structure for -them. Such threads must bootstrap themselves into existence, by first -creating a thread state data structure, then acquiring the lock, and finally -storing their thread state pointer, before they can start using the Python/C -API. When they are done, they should reset the thread state pointer, release -the lock, and finally free their thread state data structure. - -Beginning with version 2.3, threads can now take advantage of the -:cfunc:`PyGILState_\*` functions to do all of the above automatically. The -typical idiom for calling into Python from a C thread is now:: + +Here is how these functions work: the global interpreter lock is used to protect the pointer to the +current thread state. When releasing the lock and saving the thread state, +the current thread state pointer must be retrieved before the lock is released +(since another thread could immediately acquire the lock and store its own thread +state in the global variable). Conversely, when acquiring the lock and restoring +the thread state, the lock must be acquired before storing the thread state +pointer. + +.. note:: + Calling system I/O functions is the most common use case for releasing + the GIL, but it can also be useful before calling long-running computations + which don't need access to Python objects, such as compression or + cryptographic functions operating over memory buffers. For example, the + standard :mod:`zlib` and :mod:`hashlib` modules release the GIL when + compressing or hashing data. + +Non-Python created threads +-------------------------- + +When threads are created using the dedicated Python APIs (such as the +:mod:`threading` module), a thread state is automatically associated to them +and the code showed above is therefore correct. However, when threads are +created from C (for example by a third-party library with its own thread +management), they don't hold the GIL, nor is there a thread state structure +for them. + +If you need to call Python code from these threads (often this will be part +of a callback API provided by the aforementioned third-party library), +you must first register these threads with the interpreter by +creating a thread state data structure, then acquiring the GIL, and finally +storing their thread state pointer, before you can start using the Python/C +API. When you are done, you should reset the thread state pointer, release +the GIL, and finally free the thread state data structure. + +The :cfunc:`PyGILState_Ensure` and :cfunc:`PyGILState_Release` functions do +all of the above automatically. The typical idiom for calling into Python +from a C thread is:: PyGILState_STATE gstate; gstate = PyGILState_Ensure(); - /* Perform Python actions here. */ + /* Perform Python actions here. */ result = CallSomeFunction(); - /* evaluate result */ + /* evaluate result or handle exception */ /* Release the thread. No Python API allowed beyond this point. */ PyGILState_Release(gstate); Note that the :cfunc:`PyGILState_\*` functions assume there is only one global -interpreter (created automatically by :cfunc:`Py_Initialize`). Python still +interpreter (created automatically by :cfunc:`Py_Initialize`). Python supports the creation of additional interpreters (using :cfunc:`Py_NewInterpreter`), but mixing multiple interpreters and the :cfunc:`PyGILState_\*` API is unsupported. @@ -560,6 +482,13 @@ being held by a thread that is defunct after the fork. :cfunc:`PyOS_AfterFork` tries to reset the necessary locks, but is not always able to. + +High-level API +-------------- + +These are the most commonly used types and functions when writing C extension +code, or when embedding the Python interpreter: + .. ctype:: PyInterpreterState This data structure represents the state shared by a number of cooperating @@ -601,21 +530,22 @@ always able to. .. index:: module: thread - When only the main thread exists, no GIL operations are needed. This is a - common situation (most Python programs do not use threads), and the lock - operations slow the interpreter down a bit. Therefore, the lock is not - created initially. This situation is equivalent to having acquired the lock: - when there is only a single thread, all object accesses are safe. Therefore, - when this function initializes the global interpreter lock, it also acquires - it. Before the Python :mod:`thread` module creates a new thread, knowing - that either it has the lock or the lock hasn't been created yet, it calls - :cfunc:`PyEval_InitThreads`. When this call returns, it is guaranteed that - the lock has been created and that the calling thread has acquired it. + .. note:: + When only the main thread exists, no GIL operations are needed. This is a + common situation (most Python programs do not use threads), and the lock + operations slow the interpreter down a bit. Therefore, the lock is not + created initially. This situation is equivalent to having acquired the lock: + when there is only a single thread, all object accesses are safe. Therefore, + when this function initializes the global interpreter lock, it also acquires + it. Before the Python :mod:`_thread` module creates a new thread, knowing + that either it has the lock or the lock hasn't been created yet, it calls + :cfunc:`PyEval_InitThreads`. When this call returns, it is guaranteed that + the lock has been created and that the calling thread has acquired it. - It is **not** safe to call this function when it is unknown which thread (if - any) currently has the global interpreter lock. + It is **not** safe to call this function when it is unknown which thread (if + any) currently has the global interpreter lock. - This function is not available when thread support is disabled at compile time. + This function is not available when thread support is disabled at compile time. .. cfunction:: int PyEval_ThreadsInitialized() @@ -628,37 +558,6 @@ always able to. .. versionadded:: 2.4 -.. cfunction:: void PyEval_AcquireLock() - - Acquire the global interpreter lock. The lock must have been created earlier. - If this thread already has the lock, a deadlock ensues. This function is not - available when thread support is disabled at compile time. - - -.. cfunction:: void PyEval_ReleaseLock() - - Release the global interpreter lock. The lock must have been created earlier. - This function is not available when thread support is disabled at compile time. - - -.. cfunction:: void PyEval_AcquireThread(PyThreadState *tstate) - - Acquire the global interpreter lock and set the current thread state to - *tstate*, which should not be *NULL*. The lock must have been created earlier. - If this thread already has the lock, deadlock ensues. This function is not - available when thread support is disabled at compile time. - - -.. cfunction:: void PyEval_ReleaseThread(PyThreadState *tstate) - - Reset the current thread state to *NULL* and release the global interpreter - lock. The lock must have been created earlier and must be held by the current - thread. The *tstate* argument, which must not be *NULL*, is only used to check - that it represents the current thread state --- if it isn't, a fatal error is - reported. This function is not available when thread support is disabled at - compile time. - - .. cfunction:: PyThreadState* PyEval_SaveThread() Release the global interpreter lock (if it has been created and thread @@ -677,6 +576,20 @@ always able to. when thread support is disabled at compile time.) +.. cfunction:: PyThreadState* PyThreadState_Get() + + Return the current thread state. The global interpreter lock must be held. + When the current thread state is *NULL*, this issues a fatal error (so that + the caller needn't check for *NULL*). + + +.. cfunction:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) + + Swap the current thread state with the thread state given by the argument + *tstate*, which may be *NULL*. The global interpreter lock must be held + and is not released. + + .. cfunction:: void PyEval_ReInitThreads() This function is called from :cfunc:`PyOS_AfterFork` to ensure that newly @@ -684,6 +597,47 @@ always able to. are not running in the child process. +The following functions use thread-local storage, and are not compatible +with sub-interpreters: + +.. cfunction:: PyGILState_STATE PyGILState_Ensure() + + Ensure that the current thread is ready to call the Python C API regardless + of the current state of Python, or of the global interpreter lock. This may + be called as many times as desired by a thread as long as each call is + matched with a call to :cfunc:`PyGILState_Release`. In general, other + thread-related APIs may be used between :cfunc:`PyGILState_Ensure` and + :cfunc:`PyGILState_Release` calls as long as the thread state is restored to + its previous state before the Release(). For example, normal usage of the + :cmacro:`Py_BEGIN_ALLOW_THREADS` and :cmacro:`Py_END_ALLOW_THREADS` macros is + acceptable. + + The return value is an opaque "handle" to the thread state when + :cfunc:`PyGILState_Ensure` was called, and must be passed to + :cfunc:`PyGILState_Release` to ensure Python is left in the same state. Even + though recursive calls are allowed, these handles *cannot* be shared - each + unique call to :cfunc:`PyGILState_Ensure` must save the handle for its call + to :cfunc:`PyGILState_Release`. + + When the function returns, the current thread will hold the GIL and be able + to call arbitrary Python code. Failure is a fatal error. + + .. versionadded:: 2.3 + + +.. cfunction:: void PyGILState_Release(PyGILState_STATE) + + Release any resources previously acquired. After this call, Python's state will + be the same as it was prior to the corresponding :cfunc:`PyGILState_Ensure` call + (but generally this state will be unknown to the caller, hence the use of the + GILState API). + + Every call to :cfunc:`PyGILState_Ensure` must be matched by a call to + :cfunc:`PyGILState_Release` on the same thread. + + .. versionadded:: 2.3 + + The following macros are normally used without a trailing semicolon; look for example usage in the Python source distribution. @@ -717,6 +671,10 @@ example usage in the Python source distribution. :cmacro:`Py_BEGIN_ALLOW_THREADS` without the opening brace and variable declaration. It is a no-op when thread support is disabled at compile time. + +Low-level API +------------- + All of the following functions are only available when thread support is enabled at compile time, and must be called only when the global interpreter lock has been created. @@ -762,19 +720,6 @@ been created. :cfunc:`PyThreadState_Clear`. -.. cfunction:: PyThreadState* PyThreadState_Get() - - Return the current thread state. The global interpreter lock must be held. - When the current thread state is *NULL*, this issues a fatal error (so that - the caller needn't check for *NULL*). - - -.. cfunction:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate) - - Swap the current thread state with the thread state given by the argument - *tstate*, which may be *NULL*. The global interpreter lock must be held. - - .. cfunction:: PyObject* PyThreadState_GetDict() Return a dictionary in which extensions can store thread-specific state @@ -801,42 +746,190 @@ been created. .. versionadded:: 2.3 -.. cfunction:: PyGILState_STATE PyGILState_Ensure() +.. cfunction:: void PyEval_AcquireThread(PyThreadState *tstate) - Ensure that the current thread is ready to call the Python C API regardless - of the current state of Python, or of the global interpreter lock. This may - be called as many times as desired by a thread as long as each call is - matched with a call to :cfunc:`PyGILState_Release`. In general, other - thread-related APIs may be used between :cfunc:`PyGILState_Ensure` and - :cfunc:`PyGILState_Release` calls as long as the thread state is restored to - its previous state before the Release(). For example, normal usage of the - :cmacro:`Py_BEGIN_ALLOW_THREADS` and :cmacro:`Py_END_ALLOW_THREADS` macros is - acceptable. + Acquire the global interpreter lock and set the current thread state to + *tstate*, which should not be *NULL*. The lock must have been created earlier. + If this thread already has the lock, deadlock ensues. - The return value is an opaque "handle" to the thread state when - :cfunc:`PyGILState_Ensure` was called, and must be passed to - :cfunc:`PyGILState_Release` to ensure Python is left in the same state. Even - though recursive calls are allowed, these handles *cannot* be shared - each - unique call to :cfunc:`PyGILState_Ensure` must save the handle for its call - to :cfunc:`PyGILState_Release`. + :cfunc:`PyEval_RestoreThread` is a higher-level function which is always + available (even when thread support isn't enabled or when threads have + not been initialized). - When the function returns, the current thread will hold the GIL. Failure is a - fatal error. - .. versionadded:: 2.3 +.. cfunction:: void PyEval_ReleaseThread(PyThreadState *tstate) + Reset the current thread state to *NULL* and release the global interpreter + lock. The lock must have been created earlier and must be held by the current + thread. The *tstate* argument, which must not be *NULL*, is only used to check + that it represents the current thread state --- if it isn't, a fatal error is + reported. -.. cfunction:: void PyGILState_Release(PyGILState_STATE) + :cfunc:`PyEval_SaveThread` is a higher-level function which is always + available (even when thread support isn't enabled or when threads have + not been initialized). - Release any resources previously acquired. After this call, Python's state will - be the same as it was prior to the corresponding :cfunc:`PyGILState_Ensure` call - (but generally this state will be unknown to the caller, hence the use of the - GILState API.) - Every call to :cfunc:`PyGILState_Ensure` must be matched by a call to - :cfunc:`PyGILState_Release` on the same thread. +.. cfunction:: void PyEval_AcquireLock() + + Acquire the global interpreter lock. The lock must have been created earlier. + If this thread already has the lock, a deadlock ensues. + + .. warning:: + This function does not change the current thread state. Please use + :cfunc:`PyEval_RestoreThread` or :cfunc:`PyEval_AcquireThread` + instead. + + +.. cfunction:: void PyEval_ReleaseLock() + + Release the global interpreter lock. The lock must have been created earlier. + + .. warning:: + This function does not change the current thread state. Please use + :cfunc:`PyEval_SaveThread` or :cfunc:`PyEval_ReleaseThread` + instead. + + +Sub-interpreter support +======================= + +While in most uses, you will only embed a single Python interpreter, there +are cases where you need to create several independent interpreters in the +same process and perhaps even in the same thread. Sub-interpreters allow +you to do that. You can switch between sub-interpreters using the +:cfunc:`PyThreadState_Swap` function. You can create and destroy them +using the following functions: + + +.. cfunction:: PyThreadState* Py_NewInterpreter() + + .. index:: + module: builtins + module: __main__ + module: sys + single: stdout (in module sys) + single: stderr (in module sys) + single: stdin (in module sys) + + Create a new sub-interpreter. This is an (almost) totally separate environment + for the execution of Python code. In particular, the new interpreter has + separate, independent versions of all imported modules, including the + fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. The + table of loaded modules (``sys.modules``) and the module search path + (``sys.path``) are also separate. The new environment has no ``sys.argv`` + variable. It has new standard I/O stream file objects ``sys.stdin``, + ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying + file descriptors). + + The return value points to the first thread state created in the new + sub-interpreter. This thread state is made in the current thread state. + Note that no actual thread is created; see the discussion of thread states + below. If creation of the new interpreter is unsuccessful, *NULL* is + returned; no exception is set since the exception state is stored in the + current thread state and there may not be a current thread state. (Like all + other Python/C API functions, the global interpreter lock must be held before + calling this function and is still held when it returns; however, unlike most + other Python/C API functions, there needn't be a current thread state on + entry.) + + .. index:: + single: Py_Finalize() + single: Py_Initialize() + + Extension modules are shared between (sub-)interpreters as follows: the first + time a particular extension is imported, it is initialized normally, and a + (shallow) copy of its module's dictionary is squirreled away. When the same + extension is imported by another (sub-)interpreter, a new module is initialized + and filled with the contents of this copy; the extension's ``init`` function is + not called. Note that this is different from what happens when an extension is + imported after the interpreter has been completely re-initialized by calling + :cfunc:`Py_Finalize` and :cfunc:`Py_Initialize`; in that case, the extension's + ``initmodule`` function *is* called again. + + .. index:: single: close() (in module os) + + +.. cfunction:: void Py_EndInterpreter(PyThreadState *tstate) + + .. index:: single: Py_Finalize() + + Destroy the (sub-)interpreter represented by the given thread state. The given + thread state must be the current thread state. See the discussion of thread + states below. When the call returns, the current thread state is *NULL*. All + thread states associated with this interpreter are destroyed. (The global + interpreter lock must be held before calling this function and is still held + when it returns.) :cfunc:`Py_Finalize` will destroy all sub-interpreters that + haven't been explicitly destroyed at that point. + + +Bugs and caveats +---------------- + +Because sub-interpreters (and the main interpreter) are part of the same +process, the insulation between them isn't perfect --- for example, using +low-level file operations like :func:`os.close` they can +(accidentally or maliciously) affect each other's open files. Because of the +way extensions are shared between (sub-)interpreters, some extensions may not +work properly; this is especially likely when the extension makes use of +(static) global variables, or when the extension manipulates its module's +dictionary after its initialization. It is possible to insert objects created +in one sub-interpreter into a namespace of another sub-interpreter; this should +be done with great care to avoid sharing user-defined functions, methods, +instances or classes between sub-interpreters, since import operations executed +by such objects may affect the wrong (sub-)interpreter's dictionary of loaded +modules. + +Also note that combining this functionality with :cfunc:`PyGILState_\*` APIs +is delicate, become these APIs assume a bijection between Python thread states +and OS-level threads, an assumption broken by the presence of sub-interpreters. +It is highly recommended that you don't switch sub-interpreters between a pair +of matching :cfunc:`PyGILState_Ensure` and :cfunc:`PyGILState_Release` calls. +Furthermore, extensions (such as :mod:`ctypes`) using these APIs to allow calling +of Python code from non-Python created threads will probably be broken when using +sub-interpreters. + + +Asynchronous Notifications +========================== + +A mechanism is provided to make asynchronous notifications to the main +interpreter thread. These notifications take the form of a function +pointer and a void argument. + +.. index:: single: setcheckinterval() (in module sys) + +Every check interval, when the global interpreter lock is released and +reacquired, Python will also call any such provided functions. This can be used +for example by asynchronous IO handlers. The notification can be scheduled from +a worker thread and the actual call than made at the earliest convenience by the +main thread where it has possession of the global interpreter lock and can +perform any Python API calls. + +.. cfunction:: void Py_AddPendingCall( int (*func)(void *, void *arg) ) + + .. index:: single: Py_AddPendingCall() + + Post a notification to the Python main thread. If successful, *func* will be + called with the argument *arg* at the earliest convenience. *func* will be + called having the global interpreter lock held and can thus use the full + Python API and can take any action such as setting object attributes to + signal IO completion. It must return 0 on success, or -1 signalling an + exception. The notification function won't be interrupted to perform another + asynchronous notification recursively, but it can still be interrupted to + switch threads if the global interpreter lock is released, for example, if it + calls back into Python code. + + This function returns 0 on success in which case the notification has been + scheduled. Otherwise, for example if the notification buffer is full, it + returns -1 without setting any exception. + + This function can be called on any thread, be it a Python thread or some + other system thread. If it is a Python thread, it doesn't matter if it holds + the global interpreter lock or not. + + .. versionadded:: 2.7 - .. versionadded:: 2.3 .. _profiling: @@ -881,13 +974,14 @@ in previous versions. +------------------------------+--------------------------------------+ | :const:`PyTrace_LINE` | Always *NULL*. | +------------------------------+--------------------------------------+ - | :const:`PyTrace_RETURN` | Value being returned to the caller. | + | :const:`PyTrace_RETURN` | Value being returned to the caller, | + | | or *NULL* if caused by an exception. | +------------------------------+--------------------------------------+ - | :const:`PyTrace_C_CALL` | Name of function being called. | + | :const:`PyTrace_C_CALL` | Function object being called. | +------------------------------+--------------------------------------+ - | :const:`PyTrace_C_EXCEPTION` | Always *NULL*. | + | :const:`PyTrace_C_EXCEPTION` | Function object being called. | +------------------------------+--------------------------------------+ - | :const:`PyTrace_C_RETURN` | Always *NULL*. | + | :const:`PyTrace_C_RETURN` | Function object being called. | +------------------------------+--------------------------------------+ @@ -932,7 +1026,7 @@ in previous versions. .. cvar:: int PyTrace_C_EXCEPTION The value for the *what* parameter to :ctype:`Py_tracefunc` functions when a C - function has thrown an exception. + function has raised an exception. .. cvar:: int PyTrace_C_RETURN diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst index c56000e..a2e47d9 100644 --- a/Doc/c-api/intro.rst +++ b/Doc/c-api/intro.rst @@ -41,8 +41,8 @@ included in your code by the following line:: #include "Python.h" This implies inclusion of the following standard headers: ``<stdio.h>``, -``<string.h>``, ``<errno.h>``, ``<limits.h>``, and ``<stdlib.h>`` (if -available). +``<string.h>``, ``<errno.h>``, ``<limits.h>``, ``<assert.h>`` and ``<stdlib.h>`` +(if available). .. note:: @@ -361,15 +361,16 @@ traceback. .. index:: single: PyErr_Occurred() -For C programmers, however, error checking always has to be explicit. All -functions in the Python/C API can raise exceptions, unless an explicit claim is -made otherwise in a function's documentation. In general, when a function -encounters an error, it sets an exception, discards any object references that -it owns, and returns an error indicator --- usually *NULL* or ``-1``. A few -functions return a Boolean true/false result, with false indicating an error. -Very few functions return no explicit error indicator or have an ambiguous -return value, and require explicit testing for errors with -:cfunc:`PyErr_Occurred`. +For C programmers, however, error checking always has to be explicit. All +functions in the Python/C API can raise exceptions, unless an explicit claim is +made otherwise in a function's documentation. In general, when a function +encounters an error, it sets an exception, discards any object references that +it owns, and returns an error indicator. If not documented otherwise, this +indicator is either *NULL* or ``-1``, depending on the function's return type. +A few functions return a Boolean true/false result, with false indicating an +error. Very few functions return no explicit error indicator or have an +ambiguous return value, and require explicit testing for errors with +:cfunc:`PyErr_Occurred`. These exceptions are always explicitly documented. .. index:: single: PyErr_SetString() @@ -524,12 +525,12 @@ the table of loaded modules, and creates the fundamental modules :mod:`__builtin__`, :mod:`__main__`, :mod:`sys`, and :mod:`exceptions`. It also initializes the module search path (``sys.path``). -.. index:: single: PySys_SetArgv() +.. index:: single: PySys_SetArgvEx() :cfunc:`Py_Initialize` does not set the "script argument list" (``sys.argv``). -If this variable is needed by Python code that will be executed later, it must -be set explicitly with a call to ``PySys_SetArgv(argc, argv)`` subsequent to -the call to :cfunc:`Py_Initialize`. +If this variable is needed by Python code that will be executed later, it must +be set explicitly with a call to ``PySys_SetArgvEx(argc, argv, updatepath)`` +after the call to :cfunc:`Py_Initialize`. On most systems (in particular, on Unix and Windows, although the details are slightly different), :cfunc:`Py_Initialize` calculates the module search path diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst index 242b4e6..d24aa72 100644 --- a/Doc/c-api/list.rst +++ b/Doc/c-api/list.rst @@ -42,7 +42,7 @@ List Objects .. note:: - If *length* is greater than zero, the returned list object's items are + If *len* is greater than zero, the returned list object's items are set to ``NULL``. Thus you cannot use abstract API functions such as :cfunc:`PySequence_SetItem` or expose the object to Python code before setting all items to a real object with :cfunc:`PyList_SetItem`. @@ -75,9 +75,9 @@ List Objects .. cfunction:: PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index) - Return the object at position *pos* in the list pointed to by *p*. The + Return the object at position *index* in the list pointed to by *list*. The position must be positive, indexing from the end of the list is not - supported. If *pos* is out of bounds, return *NULL* and set an + supported. If *index* is out of bounds, return *NULL* and set an :exc:`IndexError` exception. .. versionchanged:: 2.5 diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 9eb9f60..11d2908 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -65,6 +65,22 @@ Long Integer Objects .. versionadded:: 2.6 +.. cfunction:: PyObject* PyLong_FromSsize_t(Py_ssize_t v) + + Return a new :ctype:`PyLongObject` object with a value of *v*, or *NULL* + on failure. + + .. versionadded:: 2.6 + + +.. cfunction:: PyObject* PyLong_FromSize_t(size_t v) + + Return a new :ctype:`PyLongObject` object with a value of *v*, or *NULL* + on failure. + + .. versionadded:: 2.6 + + .. cfunction:: PyObject* PyLong_FromLongLong(PY_LONG_LONG v) Return a new :ctype:`PyLongObject` object from a C :ctype:`long long`, or *NULL* @@ -87,7 +103,7 @@ Long Integer Objects Return a new :ctype:`PyLongObject` based on the string value in *str*, which is interpreted according to the radix in *base*. If *pend* is non-*NULL*, - ``*pend`` will point to the first character in *str* which follows the + *\*pend* will point to the first character in *str* which follows the representation of the number. If *base* is ``0``, the radix will be determined based on the leading characters of *str*: if *str* starts with ``'0x'`` or ``'0X'``, radix 16 will be used; if *str* starts with ``'0'``, radix 8 will be @@ -133,6 +149,32 @@ Long Integer Objects and ``-1`` will be returned. +.. cfunction:: long PyLong_AsLongAndOverflow(PyObject *pylong, int *overflow) + + Return a C :ctype:`long` representation of the contents of + *pylong*. If *pylong* is greater than :const:`LONG_MAX` or less + than :const:`LONG_MIN`, set *\*overflow* to ``1`` or ``-1``, + respectively, and return ``-1``; otherwise, set *\*overflow* to + ``0``. If any other exception occurs (for example a TypeError or + MemoryError), then ``-1`` will be returned and *\*overflow* will + be ``0``. + + .. versionadded:: 2.7 + + +.. cfunction:: PY_LONG_LONG PyLong_AsLongLongAndOverflow(PyObject *pylong, int *overflow) + + Return a C :ctype:`long long` representation of the contents of + *pylong*. If *pylong* is greater than :const:`PY_LLONG_MAX` or less + than :const:`PY_LLONG_MIN`, set *\*overflow* to ``1`` or ``-1``, + respectively, and return ``-1``; otherwise, set *\*overflow* to + ``0``. If any other exception occurs (for example a TypeError or + MemoryError), then ``-1`` will be returned and *\*overflow* will + be ``0``. + + .. versionadded:: 2.7 + + .. cfunction:: Py_ssize_t PyLong_AsSsize_t(PyObject *pylong) .. index:: @@ -157,23 +199,46 @@ Long Integer Objects raised. +.. cfunction:: Py_ssize_t PyLong_AsSsize_t(PyObject *pylong) + + .. index:: + single: PY_SSIZE_T_MAX + + Return a :ctype:`Py_ssize_t` representation of the contents of *pylong*. If + *pylong* is greater than :const:`PY_SSIZE_T_MAX`, an :exc:`OverflowError` is + raised. + + .. versionadded:: 2.6 + + .. cfunction:: PY_LONG_LONG PyLong_AsLongLong(PyObject *pylong) - Return a C :ctype:`long long` from a Python long integer. If *pylong* cannot be - represented as a :ctype:`long long`, an :exc:`OverflowError` will be raised. + .. index:: + single: OverflowError (built-in exception) + + Return a C :ctype:`long long` from a Python long integer. If + *pylong* cannot be represented as a :ctype:`long long`, an + :exc:`OverflowError` is raised and ``-1`` is returned. .. versionadded:: 2.2 .. cfunction:: unsigned PY_LONG_LONG PyLong_AsUnsignedLongLong(PyObject *pylong) - Return a C :ctype:`unsigned long long` from a Python long integer. If *pylong* - cannot be represented as an :ctype:`unsigned long long`, an :exc:`OverflowError` - will be raised if the value is positive, or a :exc:`TypeError` will be raised if - the value is negative. + .. index:: + single: OverflowError (built-in exception) + + Return a C :ctype:`unsigned long long` from a Python long integer. If + *pylong* cannot be represented as an :ctype:`unsigned long long`, an + :exc:`OverflowError` is raised and ``(unsigned long long)-1`` is + returned. .. versionadded:: 2.2 + .. versionchanged:: 2.7 + A negative *pylong* now raises :exc:`OverflowError`, not + :exc:`TypeError`. + .. cfunction:: unsigned long PyLong_AsUnsignedLongMask(PyObject *io) diff --git a/Doc/c-api/objbuffer.rst b/Doc/c-api/objbuffer.rst index 0f4dccf..93f5ff0 100644 --- a/Doc/c-api/objbuffer.rst +++ b/Doc/c-api/objbuffer.rst @@ -2,10 +2,10 @@ .. _abstract-buffer: + Old Buffer Protocol =================== - This section describes the legacy buffer protocol, which has been introduced in Python 1.6. It is still supported but deprecated in the Python 2.x series. Python 3.0 introduces a new buffer protocol which fixes weaknesses and diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst index 822c593..3996c1f 100644 --- a/Doc/c-api/reflection.rst +++ b/Doc/c-api/reflection.rst @@ -29,6 +29,11 @@ Reflection currently executing. +.. cfunction:: int PyFrame_GetLineNumber(PyFrameObject *frame) + + Return the line number that *frame* is currently executing. + + .. cfunction:: int PyEval_GetRestricted() If there is a current frame and it is executing in restricted mode, return true, diff --git a/Doc/c-api/string.rst b/Doc/c-api/string.rst index c7d27a3..9b95859 100644 --- a/Doc/c-api/string.rst +++ b/Doc/c-api/string.rst @@ -78,6 +78,8 @@ called with a non-string parameter. .. % The descriptions for %zd and %zu are wrong, but the truth is complicated .. % because not all compilers support the %z width modifier -- we fake it .. % when necessary via interpolating PY_FORMAT_SIZE_T. + .. % Similar comments apply to the %ll width modifier and + .. % PY_FORMAT_LONG_LONG. .. % %u, %lu, %zu should have "new in Python 2.5" blurbs. +-------------------+---------------+--------------------------------+ @@ -100,6 +102,12 @@ called with a non-string parameter. | :attr:`%lu` | unsigned long | Exactly equivalent to | | | | ``printf("%lu")``. | +-------------------+---------------+--------------------------------+ + | :attr:`%lld` | long long | Exactly equivalent to | + | | | ``printf("%lld")``. | + +-------------------+---------------+--------------------------------+ + | :attr:`%llu` | unsigned | Exactly equivalent to | + | | long long | ``printf("%llu")``. | + +-------------------+---------------+--------------------------------+ | :attr:`%zd` | Py_ssize_t | Exactly equivalent to | | | | ``printf("%zd")``. | +-------------------+---------------+--------------------------------+ @@ -127,6 +135,14 @@ called with a non-string parameter. An unrecognized format character causes all the rest of the format string to be copied as-is to the result string, and any extra arguments discarded. + .. note:: + + The `"%lld"` and `"%llu"` format specifiers are only available + when :const:`HAVE_LONG_LONG` is defined. + + .. versionchanged:: 2.7 + Support for `"%lld"` and `"%llu"` added. + .. cfunction:: PyObject* PyString_FromFormatV(const char *format, va_list vargs) diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 6ef31d2..5be88ac 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -137,12 +137,10 @@ convention flags can be combined with a binding flag. This is the typical calling convention, where the methods have the type :ctype:`PyCFunction`. The function expects two :ctype:`PyObject\*` values. - The first one is the *self* object for methods; for module functions, it - has the value given to :cfunc:`Py_InitModule4` (or *NULL* if - :cfunc:`Py_InitModule` was used). The second parameter (often called - *args*) is a tuple object representing all arguments. This parameter is - typically processed using :cfunc:`PyArg_ParseTuple` or - :cfunc:`PyArg_UnpackTuple`. + The first one is the *self* object for methods; for module functions, it is + the module object. The second parameter (often called *args*) is a tuple + object representing all arguments. This parameter is typically processed + using :cfunc:`PyArg_ParseTuple` or :cfunc:`PyArg_UnpackTuple`. .. data:: METH_KEYWORDS @@ -158,9 +156,9 @@ convention flags can be combined with a binding flag. Methods without parameters don't need to check whether arguments are given if they are listed with the :const:`METH_NOARGS` flag. They need to be of type - :ctype:`PyCFunction`. When used with object methods, the first parameter is - typically named ``self`` and will hold a reference to the object instance. - In all cases the second parameter will be *NULL*. + :ctype:`PyCFunction`. The first parameter is typically named ``self`` and + will hold a reference to the module or object instance. In all cases the + second parameter will be *NULL*. .. data:: METH_O diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index f46fb9a..6788bc9 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -15,13 +15,6 @@ Operating System Utilities one of the strings ``'<stdin>'`` or ``'???'``. -.. cfunction:: long PyOS_GetLastModificationTime(char *filename) - - Return the time of last modification of the file *filename*. The result is - encoded in the same way as the timestamp returned by the standard C library - function :cfunc:`time`. - - .. cfunction:: void PyOS_AfterFork() Function to update some internal state after a process fork; this should be diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 8489c35..9af4cfc 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -820,7 +820,9 @@ The next fields, up to and including :attr:`tp_weaklist`, only exist if the This field is not inherited by subtypes (computed attributes are inherited through a different mechanism). - Docs for PyGetSetDef (XXX belong elsewhere):: + .. XXX belongs elsewhere + + Docs for PyGetSetDef:: typedef PyObject *(*getter)(PyObject *, void *); typedef int (*setter)(PyObject *, PyObject *, void *); @@ -867,7 +869,7 @@ The next fields, up to and including :attr:`tp_weaklist`, only exist if the PyObject * tp_descr_get(PyObject *self, PyObject *obj, PyObject *type); - XXX explain. + .. XXX explain. This field is inherited by subtypes. @@ -882,7 +884,7 @@ The next fields, up to and including :attr:`tp_weaklist`, only exist if the This field is inherited by subtypes. - XXX explain. + .. XXX explain. .. cmember:: long PyTypeObject.tp_dictoffset diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 0288271..964d2c8 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -208,7 +208,7 @@ APIs: .. cfunction:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size) - Create a Unicode Object from the Py_UNICODE buffer *u* of the given size. *u* + Create a Unicode object from the Py_UNICODE buffer *u* of the given size. *u* may be *NULL* which causes the contents to be undefined. It is the user's responsibility to fill in the needed data. The buffer is copied into the new object. If the buffer is not *NULL*, the return value might be a shared object. @@ -220,6 +220,111 @@ APIs: changes in your code for properly supporting 64-bit systems. +.. cfunction:: PyObject* PyUnicode_FromStringAndSize(const char *u, Py_ssize_t size) + + Create a Unicode object from the char buffer *u*. The bytes will be interpreted + as being UTF-8 encoded. *u* may also be *NULL* which + causes the contents to be undefined. It is the user's responsibility to fill in + the needed data. The buffer is copied into the new object. If the buffer is not + *NULL*, the return value might be a shared object. Therefore, modification of + the resulting Unicode object is only allowed when *u* is *NULL*. + + .. versionadded:: 2.6 + + +.. cfunction:: PyObject *PyUnicode_FromString(const char *u) + + Create a Unicode object from an UTF-8 encoded null-terminated char buffer + *u*. + + .. versionadded:: 2.6 + + +.. cfunction:: PyObject* PyUnicode_FromFormat(const char *format, ...) + + Take a C :cfunc:`printf`\ -style *format* string and a variable number of + arguments, calculate the size of the resulting Python unicode string and return + a string with the values formatted into it. The variable arguments must be C + types and must correspond exactly to the format characters in the *format* + string. The following format characters are allowed: + + .. % The descriptions for %zd and %zu are wrong, but the truth is complicated + .. % because not all compilers support the %z width modifier -- we fake it + .. % when necessary via interpolating PY_FORMAT_SIZE_T. + + +-------------------+---------------------+--------------------------------+ + | Format Characters | Type | Comment | + +===================+=====================+================================+ + | :attr:`%%` | *n/a* | The literal % character. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%c` | int | A single character, | + | | | represented as an C int. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%d` | int | Exactly equivalent to | + | | | ``printf("%d")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%u` | unsigned int | Exactly equivalent to | + | | | ``printf("%u")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%ld` | long | Exactly equivalent to | + | | | ``printf("%ld")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%lu` | unsigned long | Exactly equivalent to | + | | | ``printf("%lu")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%zd` | Py_ssize_t | Exactly equivalent to | + | | | ``printf("%zd")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%zu` | size_t | Exactly equivalent to | + | | | ``printf("%zu")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%i` | int | Exactly equivalent to | + | | | ``printf("%i")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%x` | int | Exactly equivalent to | + | | | ``printf("%x")``. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%s` | char\* | A null-terminated C character | + | | | array. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%p` | void\* | The hex representation of a C | + | | | pointer. Mostly equivalent to | + | | | ``printf("%p")`` except that | + | | | it is guaranteed to start with | + | | | the literal ``0x`` regardless | + | | | of what the platform's | + | | | ``printf`` yields. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%U` | PyObject\* | A unicode object. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%V` | PyObject\*, char \* | A unicode object (which may be | + | | | *NULL*) and a null-terminated | + | | | C character array as a second | + | | | parameter (which will be used, | + | | | if the first parameter is | + | | | *NULL*). | + +-------------------+---------------------+--------------------------------+ + | :attr:`%S` | PyObject\* | The result of calling | + | | | :func:`PyObject_Unicode`. | + +-------------------+---------------------+--------------------------------+ + | :attr:`%R` | PyObject\* | The result of calling | + | | | :func:`PyObject_Repr`. | + +-------------------+---------------------+--------------------------------+ + + An unrecognized format character causes all the rest of the format string to be + copied as-is to the result string, and any extra arguments discarded. + + .. versionadded:: 2.6 + + +.. cfunction:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs) + + Identical to :func:`PyUnicode_FromFormat` except that it takes exactly two + arguments. + + .. versionadded:: 2.6 + + .. cfunction:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode) Return a read-only pointer to the Unicode object's internal :ctype:`Py_UNICODE` @@ -579,7 +684,7 @@ These are the UTF-7 codec APIs: *s*. Return *NULL* if an exception was raised by the codec. -.. cfunction:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) +.. cfunction:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, const char *errors, Py_ssize_t *consumed) If *consumed* is *NULL*, behave like :cfunc:`PyUnicode_DecodeUTF7`. If *consumed* is not *NULL*, trailing incomplete UTF-7 base-64 sections will not diff --git a/Doc/c-api/utilities.rst b/Doc/c-api/utilities.rst index 499a510..f43933b 100644 --- a/Doc/c-api/utilities.rst +++ b/Doc/c-api/utilities.rst @@ -19,3 +19,4 @@ and parsing function arguments and constructing Python values from C values. arg.rst conversion.rst reflection.rst + codec.rst diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 1ca11cb..23f7c85 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -121,13 +121,13 @@ the same library that the Python runtime is using. .. cfunction:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) - Read and execute a single statement from a file associated with an interactive - device according to the *flags* argument. If *filename* is *NULL*, ``"???"`` is - used instead. The user will be prompted using ``sys.ps1`` and ``sys.ps2``. - Returns ``0`` when the input was executed successfully, ``-1`` if there was an - exception, or an error code from the :file:`errcode.h` include file distributed - as part of Python if there was a parse error. (Note that :file:`errcode.h` is - not included by :file:`Python.h`, so must be included specifically if needed.) + Read and execute a single statement from a file associated with an + interactive device according to the *flags* argument. The user will be + prompted using ``sys.ps1`` and ``sys.ps2``. Returns ``0`` when the input was + executed successfully, ``-1`` if there was an exception, or an error code + from the :file:`errcode.h` include file distributed as part of Python if + there was a parse error. (Note that :file:`errcode.h` is not included by + :file:`Python.h`, so must be included specifically if needed.) .. cfunction:: int PyRun_InteractiveLoop(FILE *fp, const char *filename) @@ -136,11 +136,11 @@ the same library that the Python runtime is using. leaving *flags* set to *NULL*. -.. cfunction:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) +.. cfunction:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags) Read and execute statements from a file associated with an interactive device - until EOF is reached. If *filename* is *NULL*, ``"???"`` is used instead. The - user will be prompted using ``sys.ps1`` and ``sys.ps2``. Returns ``0`` at EOF. + until EOF is reached. The user will be prompted using ``sys.ps1`` and + ``sys.ps2``. Returns ``0`` at EOF. .. cfunction:: struct _node* PyParser_SimpleParseString(const char *str, int start) diff --git a/Doc/conf.py b/Doc/conf.py index a84f9c3..b514bff 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -125,7 +125,9 @@ latex_documents = [ ('tutorial/index', 'tutorial.tex', 'Python Tutorial', _stdauthor, 'manual'), ('using/index', 'using.tex', - 'Using Python', _stdauthor, 'manual'), + 'Python Setup and Usage', _stdauthor, 'manual'), + ('faq/index', 'faq.tex', + 'Python Frequently Asked Questions', _stdauthor, 'manual'), ('whatsnew/' + version, 'whatsnew.tex', 'What\'s New in Python', 'A. M. Kuchling', 'howto'), ] diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 90ad2bf..1fc896f 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -55,6 +55,45 @@ PyBuffer_FromReadWriteMemory:int:size:: PyBuffer_New:PyObject*::+1: PyBuffer_New:int:size:: +PyCapsule_GetContext:void *::: +PyCapsule_GetContext:PyObject*:self:0: + +PyCapsule_GetDestructor:void (*)(PyObject *)::: +PyCapsule_GetDestructor:PyObject*:self:0: + +PyCapsule_GetName:const char *::: +PyCapsule_GetName:PyObject*:self:0: + +PyCapsule_GetPointer:void*::: +PyCapsule_GetPointer:PyObject*:self:0: +PyCapsule_GetPointer:const char *:name:: + +PyCapsule_Import:void *::: +PyCapsule_Import:const char *:name:: +PyCapsule_Import:int:no_block:: + +PyCapsule_New:PyObject*::+1: +PyCapsule_New:void*:pointer:: +PyCapsule_New:const char *:name:: +PyCapsule_New::void (* destructor)(PyObject* ):: + +PyCapsule_SetContext:int::: +PyCapsule_SetContext:PyObject*:self:0: +PyCapsule_SetContext:void *:context:: + +PyCapsule_SetDestructor:int::: +PyCapsule_SetDestructor:PyObject*:self:0: +PyCapsule_SetDestructor:void (*)(PyObject *):destructor:: + +PyCapsule_SetName:int::: +PyCapsule_SetName:PyObject*:self:0: +PyCapsule_SetName:const char *:name:: + +PyCapsule_SetPointer:int::: +PyCapsule_SetPointer:PyObject*:self:0: +PyCapsule_SetPointer:void*:pointer:: + + PyCObject_AsVoidPtr:void*::: PyCObject_AsVoidPtr:PyObject*:self:0: @@ -242,6 +281,12 @@ PyErr_NewException:char*:name:: PyErr_NewException:PyObject*:base:0: PyErr_NewException:PyObject*:dict:0: +PyErr_NewExceptionWithDoc:PyObject*::+1: +PyErr_NewExceptionWithDoc:char*:name:: +PyErr_NewExceptionWithDoc:char*:doc:: +PyErr_NewExceptionWithDoc:PyObject*:base:0: +PyErr_NewExceptionWithDoc:PyObject*:dict:0: + PyErr_NoMemory:PyObject*::null: PyErr_NormalizeException:void::: @@ -593,6 +638,9 @@ PyLong_AsDouble:PyObject*:pylong:0: PyLong_AsLong:long::: PyLong_AsLong:PyObject*:pylong:0: +PyLong_AsSsize_t:ssize_t::: +PyLong_AsSsize_t:PyObject*:pylong:0: + PyLong_AsUnsignedLong:unsigned long::: PyLong_AsUnsignedLong:PyObject*:pylong:0: @@ -611,6 +659,12 @@ PyLong_FromLongLong:long long:v:: PyLong_FromUnsignedLongLong:PyObject*::+1: PyLong_FromUnsignedLongLong:unsigned long long:v:: +PyLong_FromSize_t:PyObject*::+1: +PyLong_FromSize_t:size_t:v:: + +PyLong_FromSsize_t:PyObject*::+1: +PyLong_FromSsize_t:ssize_t:v:: + PyLong_FromString:PyObject*::+1: PyLong_FromString:char*:str:: PyLong_FromString:char**:pend:: @@ -853,9 +907,6 @@ PyNumber_Xor:PyObject*::+1: PyNumber_Xor:PyObject*:o1:0: PyNumber_Xor:PyObject*:o2:0: -PyOS_GetLastModificationTime:long::: -PyOS_GetLastModificationTime:char*:filename:: - PyObject_AsFileDescriptor:int::: PyObject_AsFileDescriptor:PyObject*:o:0: @@ -1431,6 +1482,21 @@ PyUnicode_FromEncodedObject:PyObject*:*obj:0: PyUnicode_FromEncodedObject:const char*:encoding:: PyUnicode_FromEncodedObject:const char*:errors:: +PyUnicode_FromFormat:PyObject*::+1: +PyUnicode_FromFormat:const char*:format:: +PyUnicode_FromFormat::...:: + +PyUnicode_FromFormatV:PyObject*::+1: +PyUnicode_FromFormatV:const char*:format:: +PyUnicode_FromFormatV:va_list:vargs:: + +PyUnicode_FromString:PyObject*::+1: +PyUnicode_FromString:const char*:u:: + +PyUnicode_FromStringAndSize:PyObject*::+1: +PyUnicode_FromStringAndSize:const char*:u:: +PyUnicode_FromStringAndSize:ssize_t:size:: + PyUnicode_FromWideChar:PyObject*::+1: PyUnicode_FromWideChar:const wchar_t*:w:: PyUnicode_FromWideChar:int:size:: diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 0a88598..b28a3af 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -847,23 +847,6 @@ This module provides the EMXCCompiler class, a subclass of :class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2. -:mod:`distutils.mwerkscompiler` --- Metrowerks CodeWarrior support -================================================================== - -.. module:: distutils.mwerkscompiler - :synopsis: Metrowerks CodeWarrior support - - -Contains :class:`MWerksCompiler`, an implementation of the abstract -:class:`CCompiler` class for MetroWerks CodeWarrior on the pre-Mac OS X -Macintosh. Needs work to support CW on Windows or Mac OS X. - -.. % \subsection{Utility modules} -.. % -.. % The following modules all provide general utility functions. They haven't -.. % all been documented yet. - - :mod:`distutils.archive_util` --- Archiving utilities ====================================================== @@ -905,7 +888,7 @@ tarballs or zipfiles. .. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) Create a zip file from all files in and under *base_dir*. The output zip file - will be named *base_dir* + :file:`.zip`. Uses either the :mod:`zipfile` Python + will be named *base_name* + :file:`.zip`. Uses either the :mod:`zipfile` Python module (if available) or the InfoZIP :file:`zip` utility (if installed and found on the default search path). If neither tool is available, raises :exc:`DistutilsExecError`. Returns the name of the output zip file. diff --git a/Doc/distutils/builtdist.rst b/Doc/distutils/builtdist.rst index f1dad77..8c83d11 100644 --- a/Doc/distutils/builtdist.rst +++ b/Doc/distutils/builtdist.rst @@ -80,7 +80,7 @@ The available formats for built distributions are: +-------------+------------------------------+---------+ | ``tar`` | tar file (:file:`.tar`) | \(3) | +-------------+------------------------------+---------+ -| ``zip`` | zip file (:file:`.zip`) | \(4) | +| ``zip`` | zip file (:file:`.zip`) | (2),(4) | +-------------+------------------------------+---------+ | ``rpm`` | RPM | \(5) | +-------------+------------------------------+---------+ @@ -90,9 +90,12 @@ The available formats for built distributions are: +-------------+------------------------------+---------+ | ``rpm`` | RPM | \(5) | +-------------+------------------------------+---------+ -| ``wininst`` | self-extracting ZIP file for | (2),(4) | +| ``wininst`` | self-extracting ZIP file for | \(4) | | | Windows | | +-------------+------------------------------+---------+ +| ``msi`` | Microsoft Installer. | | ++-------------+------------------------------+---------+ + Notes: @@ -102,8 +105,6 @@ Notes: (2) default on Windows - **\*\*** to-do! **\*\*** - (3) requires external utilities: :program:`tar` and possibly one of :program:`gzip`, :program:`bzip2`, or :program:`compress` @@ -133,6 +134,8 @@ generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and +--------------------------+-----------------------+ | :command:`bdist_wininst` | wininst | +--------------------------+-----------------------+ +| :command:`bdist_msi` | msi | ++--------------------------+-----------------------+ The following sections give details on the individual :command:`bdist_\*` commands. @@ -238,7 +241,8 @@ tedious and error-prone, so it's usually best to put them in the setup configuration file, :file:`setup.cfg`\ ---see section :ref:`setup-config`. If you distribute or package many Python module distributions, you might want to put options that apply to all of them in your personal Distutils configuration -file (:file:`~/.pydistutils.cfg`). +file (:file:`~/.pydistutils.cfg`). If you want to temporarily disable +this file, you can pass the --no-user-cfg option to setup.py. There are three steps to building a binary RPM package, all of which are handled automatically by the Distutils: @@ -318,7 +322,7 @@ the :option:`--no-target-compile` and/or the :option:`--no-target-optimize` option. By default the installer will display the cool "Python Powered" logo when it is -run, but you can also supply your own 152x161 bitmap which must be a Windows +run, but you can also supply your own 152x261 bitmap which must be a Windows :file:`.bmp` file with the :option:`--bitmap` option. The installer will also display a large title on the desktop background window @@ -354,7 +358,7 @@ would create a 64bit installation executable on your 32bit version of Windows. To cross-compile, you must download the Python source code and cross-compile Python itself for the platform you are targetting - it is not possible from a -binary installtion of Python (as the .lib etc file for other platforms are +binary installation of Python (as the .lib etc file for other platforms are not included.) In practice, this means the user of a 32 bit operating system will need to use Visual Studio 2008 to open the :file:`PCBuild/PCbuild.sln` solution in the Python source tree and build the @@ -371,7 +375,7 @@ check or modify your existing install.) The Postinstallation script --------------------------- -Starting with Python 2.3, a postinstallation script can be specified which the +Starting with Python 2.3, a postinstallation script can be specified with the :option:`--install-script` option. The basename of the script must be specified, and the script filename must also be listed in the scripts argument to the setup function. diff --git a/Doc/distutils/commandref.rst b/Doc/distutils/commandref.rst index fbe40de..7282961 100644 --- a/Doc/distutils/commandref.rst +++ b/Doc/distutils/commandref.rst @@ -48,50 +48,6 @@ This command installs all (Python) scripts in the distribution. .. % \label{clean-cmd} -.. _sdist-cmd: - -Creating a source distribution: the :command:`sdist` command -============================================================ - -**\*\*** fragment moved down from above: needs context! **\*\*** - -The manifest template commands are: - -+-------------------------------------------+-----------------------------------------------+ -| Command | Description | -+===========================================+===============================================+ -| :command:`include pat1 pat2 ...` | include all files matching any of the listed | -| | patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`exclude pat1 pat2 ...` | exclude all files matching any of the listed | -| | patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`recursive-include dir pat1 pat2 | include all files under *dir* matching any of | -| ...` | the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`recursive-exclude dir pat1 pat2 | exclude all files under *dir* matching any of | -| ...` | the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`global-include pat1 pat2 ...` | include all files anywhere in the source tree | -| | matching --- & any of the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`global-exclude pat1 pat2 ...` | exclude all files anywhere in the source tree | -| | matching --- & any of the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`prune dir` | exclude all files under *dir* | -+-------------------------------------------+-----------------------------------------------+ -| :command:`graft dir` | include all files under *dir* | -+-------------------------------------------+-----------------------------------------------+ - -The patterns here are Unix-style "glob" patterns: ``*`` matches any sequence of -regular filename characters, ``?`` matches any single regular filename -character, and ``[range]`` matches any of the characters in *range* (e.g., -``a-z``, ``a-zA-Z``, ``a-f0-9_.``). The definition of "regular filename -character" is platform-specific: on Unix it is anything except slash; on Windows -anything except backslash or colon. - -**\*\*** Windows support not there yet **\*\*** - .. % \section{Creating a built distribution: the .. % \protect\command{bdist} command family} .. % \label{bdist-cmds} diff --git a/Doc/distutils/packageindex.rst b/Doc/distutils/packageindex.rst index 912248e..1498394 100644 --- a/Doc/distutils/packageindex.rst +++ b/Doc/distutils/packageindex.rst @@ -8,17 +8,17 @@ The Python Package Index (PyPI) holds meta-data describing distributions packaged with distutils. The distutils command :command:`register` is used to submit your distribution's meta-data to the index. It is invoked as follows:: - python setup.py register + python setup.py register Distutils will respond with the following prompt:: - running register - We need to know who you are, so please choose either: - 1. use your existing login, - 2. register as a new user, - 3. have the server generate a new password for you (and email it to you), or - 4. quit - Your selection [default 1]: + running register + We need to know who you are, so please choose either: + 1. use your existing login, + 2. register as a new user, + 3. have the server generate a new password for you (and email it to you), or + 4. quit + Your selection [default 1]: Note: if your username and password are saved locally, you will not see this menu. @@ -55,39 +55,50 @@ The .pypirc file The format of the :file:`.pypirc` file is as follows:: - [distutils] - index-servers = - pypi + [distutils] + index-servers = + pypi - [pypi] - repository: <repository-url> - username: <username> - password: <password> + [pypi] + repository: <repository-url> + username: <username> + password: <password> -*repository* can be omitted and defaults to ``http://www.python.org/pypi``. +The *distutils* section defines a *index-servers* variable that lists the +name of all sections describing a repository. -If you want to define another server a new section can be created:: +Each section describing a repository defines three variables: - [distutils] - index-servers = - pypi - other +- *repository*, that defines the url of the PyPI server. Defaults to + ``http://www.python.org/pypi``. +- *username*, which is the registered username on the PyPI server. +- *password*, that will be used to authenticate. If omitted the user + will be prompt to type it when needed. - [pypi] - repository: <repository-url> - username: <username> - password: <password> +If you want to define another server a new section can be created and +listed in the *index-servers* variable:: - [other] - repository: http://example.com/pypi - username: <username> - password: <password> + [distutils] + index-servers = + pypi + other -The command can then be called with the -r option:: + [pypi] + repository: <repository-url> + username: <username> + password: <password> - python setup.py register -r http://example.com/pypi + [other] + repository: http://example.com/pypi + username: <username> + password: <password> -Or even with the section name:: +:command:`register` can then be called with the -r option to point the +repository to work with:: - python setup.py register -r other + python setup.py register -r http://example.com/pypi +For convenience, the name of the section that describes the repository +may also be used:: + + python setup.py register -r other diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst index b63e209..e457918 100644 --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -19,18 +19,18 @@ existence so that Python 1.5.2 users can use them to install other module distributions. The Distutils' own setup script, shown here, is used to install the package into Python 1.5.2.) :: - #!/usr/bin/env python + #!/usr/bin/env python - from distutils.core import setup + from distutils.core import setup - setup(name='Distutils', - version='1.0', - description='Python Distribution Utilities', - author='Greg Ward', - author_email='gward@python.net', - url='http://www.python.org/sigs/distutils-sig/', - packages=['distutils', 'distutils.command'], - ) + setup(name='Distutils', + version='1.0', + description='Python Distribution Utilities', + author='Greg Ward', + author_email='gward@python.net', + url='http://www.python.org/sigs/distutils-sig/', + packages=['distutils', 'distutils.command'], + ) There are only two differences between this and the trivial one-file distribution presented in section :ref:`distutils-simple-example`: more metadata, and the @@ -53,8 +53,8 @@ you, for example, use standard Python functions such as :func:`glob.glob` or :func:`os.listdir` to specify files, you should be careful to write portable code instead of hardcoding path separators:: - glob.glob(os.path.join('mydir', 'subdir', '*.html')) - os.listdir(os.path.join('mydir', 'subdir')) + glob.glob(os.path.join('mydir', 'subdir', '*.html')) + os.listdir(os.path.join('mydir', 'subdir')) .. _listing-packages: @@ -81,7 +81,7 @@ under :file:`lib`, so that modules in the "root package" (i.e., not in any package at all) are in :file:`lib`, modules in the :mod:`foo` package are in :file:`lib/foo`, and so forth. Then you would put :: - package_dir = {'': 'lib'} + package_dir = {'': 'lib'} in your setup script. The keys to this dictionary are package names, and an empty package name stands for the root package. The values are directory names @@ -92,7 +92,7 @@ Another possible convention is to put the :mod:`foo` package right in :file:`lib`, the :mod:`foo.bar` package in :file:`lib/bar`, etc. This would be written in the setup script as :: - package_dir = {'foo': 'lib'} + package_dir = {'foo': 'lib'} A ``package: dir`` entry in the :option:`package_dir` dictionary implicitly applies to all packages below *package*, so the :mod:`foo.bar` case is @@ -114,7 +114,7 @@ than listing packages---especially the case of a single module that goes in the "root package" (i.e., no package at all). This simplest case was shown in section :ref:`distutils-simple-example`; here is a slightly more involved example:: - py_modules = ['mod1', 'pkg.mod2'] + py_modules = ['mod1', 'pkg.mod2'] This describes two modules, one of them in the "root" package, the other in the :mod:`pkg` package. Again, the default package/directory layout implies that @@ -144,17 +144,17 @@ Suppose your distribution includes a single extension, called :mod:`foo` and implemented by :file:`foo.c`. If no additional instructions to the compiler/linker are needed, describing this extension is quite simple:: - Extension('foo', ['foo.c']) + Extension('foo', ['foo.c']) The :class:`Extension` class can be imported from :mod:`distutils.core` along with :func:`setup`. Thus, the setup script for a module distribution that contains only this one extension and nothing else might be:: - from distutils.core import setup, Extension - setup(name='foo', - version='1.0', - ext_modules=[Extension('foo', ['foo.c'])], - ) + from distutils.core import setup, Extension + setup(name='foo', + version='1.0', + ext_modules=[Extension('foo', ['foo.c'])], + ) The :class:`Extension` class (actually, the underlying extension-building machinery implemented by the :command:`build_ext` command) supports a great deal @@ -168,11 +168,11 @@ Extension names and packages The first argument to the :class:`Extension` constructor is always the name of the extension, including any package names. For example, :: - Extension('foo', ['src/foo1.c', 'src/foo2.c']) + Extension('foo', ['src/foo1.c', 'src/foo2.c']) describes an extension that lives in the root package, while :: - Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c']) + Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c']) describes the same extension in the :mod:`pkg` package. The source files and resulting object code are identical in both cases; the only difference is where @@ -183,11 +183,11 @@ If you have a number of extensions all in the same package (or all under the same base package), use the :option:`ext_package` keyword argument to :func:`setup`. For example, :: - setup(..., - ext_package='pkg', - ext_modules=[Extension('foo', ['foo.c']), - Extension('subpkg.bar', ['bar.c'])], - ) + setup(..., + ext_package='pkg', + ext_modules=[Extension('foo', ['foo.c']), + Extension('subpkg.bar', ['bar.c'])], + ) will compile :file:`foo.c` to the extension :mod:`pkg.foo`, and :file:`bar.c` to :mod:`pkg.subpkg.bar`. @@ -212,15 +212,15 @@ extension. This warning notwithstanding, options to SWIG can be currently passed like this:: - setup(..., - ext_modules=[Extension('_foo', ['foo.i'], - swig_opts=['-modern', '-I../include'])], - py_modules=['foo'], - ) + setup(..., + ext_modules=[Extension('_foo', ['foo.i'], + swig_opts=['-modern', '-I../include'])], + py_modules=['foo'], + ) Or on the commandline like this:: - > python setup.py build_ext --swig-opts="-modern -I../include" + > python setup.py build_ext --swig-opts="-modern -I../include" On some platforms, you can include non-source files that are processed by the compiler and included in your extension. Currently, this just means Windows @@ -239,18 +239,18 @@ include directories to search or preprocessor macros to define/undefine: For example, if your extension requires header files in the :file:`include` directory under your distribution root, use the ``include_dirs`` option:: - Extension('foo', ['foo.c'], include_dirs=['include']) + Extension('foo', ['foo.c'], include_dirs=['include']) You can specify absolute directories there; if you know that your extension will only be built on Unix systems with X11R6 installed to :file:`/usr`, you can get away with :: - Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11']) + Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11']) You should avoid this sort of non-portable usage if you plan to distribute your code: it's probably better to write C code like :: - #include <X11/Xlib.h> + #include <X11/Xlib.h> If you need to include header files from some other Python extension, you can take advantage of the fact that header files are installed in a consistent way @@ -262,17 +262,17 @@ directory---\ :file:`/usr/local/include/python1.5` in this case---is always included in the search path when building Python extensions, the best approach is to write C code like :: - #include <Numerical/arrayobject.h> + #include <Numerical/arrayobject.h> If you must put the :file:`Numerical` include directory right into your header search path, though, you can find that directory using the Distutils :mod:`distutils.sysconfig` module:: - from distutils.sysconfig import get_python_inc - incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') - setup(..., - Extension(..., include_dirs=[incdir]), - ) + from distutils.sysconfig import get_python_inc + incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') + setup(..., + Extension(..., include_dirs=[incdir]), + ) Even though this is quite portable---it will work on any Python installation, regardless of platform---it's probably easier to just write your C code in the @@ -288,17 +288,17 @@ just a list of macros to undefine. For example:: - Extension(..., - define_macros=[('NDEBUG', '1'), - ('HAVE_STRFTIME', None)], - undef_macros=['HAVE_FOO', 'HAVE_BAR']) + Extension(..., + define_macros=[('NDEBUG', '1'), + ('HAVE_STRFTIME', None)], + undef_macros=['HAVE_FOO', 'HAVE_BAR']) is the equivalent of having this at the top of every C source file:: - #define NDEBUG 1 - #define HAVE_STRFTIME - #undef HAVE_FOO - #undef HAVE_BAR + #define NDEBUG 1 + #define HAVE_STRFTIME + #undef HAVE_FOO + #undef HAVE_BAR Library options @@ -313,15 +313,15 @@ directories to search for shared (dynamically loaded) libraries at run-time. For example, if you need to link against libraries known to be in the standard library search path on target systems :: - Extension(..., - libraries=['gdbm', 'readline']) + Extension(..., + libraries=['gdbm', 'readline']) If you need to link with libraries in a non-standard location, you'll have to include the location in ``library_dirs``:: - Extension(..., - library_dirs=['/usr/X11R6/lib'], - libraries=['X11', 'Xt']) + Extension(..., + library_dirs=['/usr/X11R6/lib'], + libraries=['X11', 'Xt']) (Again, this sort of non-portable construct should be avoided if you intend to distribute your code.) @@ -334,6 +334,10 @@ Other options There are still some other options which can be used to handle special cases. +The :option:`optional` option is a boolean; if it is true, +a build failure in the extension will not abort the build process, but +instead simply not install the failing extension. + The :option:`extra_objects` option is a list of object files to be passed to the linker. These files must not have extensions, as the default extension for the compiler is used. @@ -347,6 +351,10 @@ symbols (functions or variables) to be exported. This option is not needed when building compiled extensions: Distutils will automatically add ``initmodule`` to the list of exported symbols. +The :option:`depends` option is a list of files that the extension depends on +(for example header files). The build command will call the compiler on the +sources to rebuild extension if any on this files has been modified since the +previous build. Relationships between Distributions and Packages ================================================ @@ -375,8 +383,8 @@ If specific versions are required, a sequence of qualifiers can be supplied in parentheses. Each qualifier may consist of a comparison operator and a version number. The accepted comparison operators are:: - < > == - <= >= != + < > == + <= >= != These can be combined by using multiple qualifiers separated by commas (and optional whitespace). In this case, all of the qualifiers must be matched; a @@ -423,6 +431,7 @@ The versions identified by the qualifiers are those that are obsoleted by the distribution being described. If no qualifiers are given, all versions of the named module or package are understood to be obsoleted. +.. _distutils-installing-scripts: Installing Scripts ================== @@ -441,10 +450,15 @@ option will allow the interpreter path to be explicitly overridden. The :option:`scripts` option simply is a list of files to be handled in this way. From the PyXML setup script:: - setup(..., - scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val'] - ) + setup(..., + scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val'] + ) +.. versionchanged:: 2.7 + All the scripts will also be added to the ``MANIFEST`` + file if no template is provided. See :ref:`manifest`. + +.. _distutils-installing-package-data: Installing Package Data ======================= @@ -468,26 +482,32 @@ created in the installation. For example, if a package should contain a subdirectory with several data files, the files can be arranged like this in the source tree:: - setup.py - src/ - mypkg/ - __init__.py - module.py - data/ - tables.dat - spoons.dat - forks.dat + setup.py + src/ + mypkg/ + __init__.py + module.py + data/ + tables.dat + spoons.dat + forks.dat The corresponding call to :func:`setup` might be:: - setup(..., - packages=['mypkg'], - package_dir={'mypkg': 'src/mypkg'}, - package_data={'mypkg': ['data/*.dat']}, - ) + setup(..., + packages=['mypkg'], + package_dir={'mypkg': 'src/mypkg'}, + package_data={'mypkg': ['data/*.dat']}, + ) .. versionadded:: 2.4 +.. versionchanged:: 2.7 + All the files that match ``package_data`` will be added to the ``MANIFEST`` + file if no template is provided. See :ref:`manifest`. + + +.. _distutils-additional-files: Installing Additional Files =========================== @@ -499,11 +519,11 @@ anything which doesn't fit in the previous categories. :option:`data_files` specifies a sequence of (*directory*, *files*) pairs in the following way:: - setup(..., - data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), - ('config', ['cfg/data.cfg']), - ('/etc/init.d', ['init-script'])] - ) + setup(..., + data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), + ('config', ['cfg/data.cfg']), + ('/etc/init.d', ['init-script'])] + ) Note that you can specify the directory names where the data files will be installed, but you cannot rename the data files themselves. @@ -523,6 +543,11 @@ without specifying a target directory, but this is not recommended, and the files directly in the target directory, an empty string should be given as the directory. +.. versionchanged:: 2.7 + All the files that match ``data_files`` will be added to the ``MANIFEST`` + file if no template is provided. See :ref:`manifest`. + + .. _meta-data: @@ -555,7 +580,7 @@ This information includes: | | description of the | | | | | package | | | +----------------------+---------------------------+-----------------+--------+ -| ``long_description`` | longer description of the | long string | | +| ``long_description`` | longer description of the | long string | \(5) | | | package | | | +----------------------+---------------------------+-----------------+--------+ | ``download_url`` | location where the | URL | \(4) | @@ -571,18 +596,22 @@ This information includes: Notes: (1) - These fields are required. + These fields are required. (2) - It is recommended that versions take the form *major.minor[.patch[.sub]]*. + It is recommended that versions take the form *major.minor[.patch[.sub]]*. (3) - Either the author or the maintainer must be identified. + Either the author or the maintainer must be identified. (4) - These fields should not be used if your package is to be compatible with Python - versions prior to 2.2.3 or 2.3. The list is available from the `PyPI website - <http://pypi.python.org/pypi>`_. + These fields should not be used if your package is to be compatible with Python + versions prior to 2.2.3 or 2.3. The list is available from the `PyPI website + <http://pypi.python.org/pypi>`_. + +(5) + The ``long_description`` field is used by PyPI when you are registering a + package, to build its home page. (6) The ``license`` field is a text indicating the license covering the @@ -592,14 +621,14 @@ Notes: acts as an alias for ``license``. 'short string' - A single line of text, not more than 200 characters. + A single line of text, not more than 200 characters. 'long string' - Multiple lines of plain text in reStructuredText format (see - http://docutils.sf.net/). + Multiple lines of plain text in reStructuredText format (see + http://docutils.sf.net/). 'list of strings' - See below. + See below. None of the string values may be Unicode. @@ -615,44 +644,44 @@ information is sometimes used to indicate sub-releases. These are (for final pre-release release testing). Some examples: 0.1.0 - the first, experimental release of a package + the first, experimental release of a package 1.0.1a2 - the second alpha release of the first patch version of 1.0 + the second alpha release of the first patch version of 1.0 :option:`classifiers` are specified in a Python list:: - setup(..., - classifiers=[ - 'Development Status :: 4 - Beta', - 'Environment :: Console', - 'Environment :: Web Environment', - 'Intended Audience :: End Users/Desktop', - 'Intended Audience :: Developers', - 'Intended Audience :: System Administrators', - 'License :: OSI Approved :: Python Software Foundation License', - 'Operating System :: MacOS :: MacOS X', - 'Operating System :: Microsoft :: Windows', - 'Operating System :: POSIX', - 'Programming Language :: Python', - 'Topic :: Communications :: Email', - 'Topic :: Office/Business', - 'Topic :: Software Development :: Bug Tracking', - ], - ) + setup(..., + classifiers=[ + 'Development Status :: 4 - Beta', + 'Environment :: Console', + 'Environment :: Web Environment', + 'Intended Audience :: End Users/Desktop', + 'Intended Audience :: Developers', + 'Intended Audience :: System Administrators', + 'License :: OSI Approved :: Python Software Foundation License', + 'Operating System :: MacOS :: MacOS X', + 'Operating System :: Microsoft :: Windows', + 'Operating System :: POSIX', + 'Programming Language :: Python', + 'Topic :: Communications :: Email', + 'Topic :: Office/Business', + 'Topic :: Software Development :: Bug Tracking', + ], + ) If you wish to include classifiers in your :file:`setup.py` file and also wish to remain backwards-compatible with Python releases prior to 2.2.3, then you can include the following code fragment in your :file:`setup.py` before the :func:`setup` call. :: - # patch distutils if it can't cope with the "classifiers" or - # "download_url" keywords - from sys import version - if version < '2.2.3': - from distutils.dist import DistributionMetadata - DistributionMetadata.classifiers = None - DistributionMetadata.download_url = None + # patch distutils if it can't cope with the "classifiers" or + # "download_url" keywords + from sys import version + if version < '2.2.3': + from distutils.dist import DistributionMetadata + DistributionMetadata.classifiers = None + DistributionMetadata.download_url = None Debugging the setup script @@ -674,5 +703,3 @@ failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set to anything except an empty string, and distutils will now print detailed information what it is doing, and prints the full traceback in case an exception occurs. - - diff --git a/Doc/distutils/sourcedist.rst b/Doc/distutils/sourcedist.rst index 9febd10..fc860de 100644 --- a/Doc/distutils/sourcedist.rst +++ b/Doc/distutils/sourcedist.rst @@ -26,16 +26,16 @@ to create a gzipped tarball and a zip file. The available formats are: +===========+=========================+=========+ | ``zip`` | zip file (:file:`.zip`) | (1),(3) | +-----------+-------------------------+---------+ -| ``gztar`` | gzip'ed tar file | (2),(4) | +| ``gztar`` | gzip'ed tar file | \(2) | | | (:file:`.tar.gz`) | | +-----------+-------------------------+---------+ -| ``bztar`` | bzip2'ed tar file | \(4) | +| ``bztar`` | bzip2'ed tar file | | | | (:file:`.tar.bz2`) | | +-----------+-------------------------+---------+ | ``ztar`` | compressed tar file | \(4) | | | (:file:`.tar.Z`) | | +-----------+-------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | \(4) | +| ``tar`` | tar file (:file:`.tar`) | | +-----------+-------------------------+---------+ Notes: @@ -51,8 +51,16 @@ Notes: of the standard Python library since Python 1.6) (4) - requires external utilities: :program:`tar` and possibly one of :program:`gzip`, - :program:`bzip2`, or :program:`compress` + requires the :program:`compress` program. Notice that this format is now + pending for deprecation and will be removed in the future versions of Python. + +When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or +``tar``) under Unix, you can specify the ``owner`` and ``group`` names +that will be set for each member of the archive. + +For example, if you want all files of the archive to be owned by root:: + + python setup.py sdist --owner=root --group=root .. _manifest: @@ -74,6 +82,7 @@ source distribution: :meth:`get_source_files` method in :file:`build_clib.py`! * scripts identified by the :option:`scripts` option + See :ref:`distutils-installing-scripts`. * anything that looks like a test script: :file:`test/test\*.py` (currently, the Distutils don't do anything with test scripts except include them in source @@ -83,6 +92,12 @@ source distribution: * :file:`README.txt` (or :file:`README`), :file:`setup.py` (or whatever you called your setup script), and :file:`setup.cfg` +* all files that matches the ``package_data`` metadata. + See :ref:`distutils-installing-package-data`. + +* all files that matches the ``data_files`` metadata. + See :ref:`distutils-additional-files`. + Sometimes this is enough, but usually you will want to specify additional files to distribute. The typical way to do this is to write a *manifest template*, called :file:`MANIFEST.in` by default. The manifest template is just a list of @@ -96,9 +111,64 @@ per line, regular files (or symlinks to them) only. If you do supply your own :file:`MANIFEST`, you must specify everything: the default set of files described above does not apply in this case. +.. versionadded:: 2.7 + :file:`MANIFEST` files start with a comment indicating they are generated. + Files without this comment are not overwritten or removed. + +See :ref:`manifest_template` section for a syntax reference. + +.. _manifest-options: + +Manifest-related options +======================== + +The normal course of operations for the :command:`sdist` command is as follows: + +* if the manifest file, :file:`MANIFEST` doesn't exist, read :file:`MANIFEST.in` + and create the manifest + +* if neither :file:`MANIFEST` nor :file:`MANIFEST.in` exist, create a manifest + with just the default file set + +* if either :file:`MANIFEST.in` or the setup script (:file:`setup.py`) are more + recent than :file:`MANIFEST`, recreate :file:`MANIFEST` by reading + :file:`MANIFEST.in` + +* use the list of files now in :file:`MANIFEST` (either just generated or read + in) to create the source distribution archive(s) + +There are a couple of options that modify this behaviour. First, use the +:option:`--no-defaults` and :option:`--no-prune` to disable the standard +"include" and "exclude" sets. + +Second, you might just want to (re)generate the manifest, but not create a +source distribution:: + + python setup.py sdist --manifest-only + +:option:`-o` is a shortcut for :option:`--manifest-only`. + +.. _manifest_template: + +The MANIFEST.in template +======================== + +A :file:`MANIFEST.in` file can be added in a project to define the list of +files to include in the distribution built by the :command:`sdist` command. + +When :command:`sdist` is run, it will look for the :file:`MANIFEST.in` file +and interpret it to generate the :file:`MANIFEST` file that contains the +list of files that will be included in the package. + +This mechanism can be used when the default list of files is not enough. +(See :ref:`manifest`). + +Principle +--------- + The manifest template has one command per line, where each command specifies a set of files to include or exclude from the source distribution. For an -example, again we turn to the Distutils' own manifest template:: +example, let's look at the Distutils' own manifest template:: include *.txt recursive-include examples *.txt *.py @@ -110,9 +180,7 @@ matching :file:`\*.txt` or :file:`\*.py`, and exclude all directories matching :file:`examples/sample?/build`. All of this is done *after* the standard include set, so you can exclude files from the standard set with explicit instructions in the manifest template. (Or, you can use the -:option:`--no-defaults` option to disable the standard set entirely.) There are -several other commands available in the manifest template mini-language; see -section :ref:`sdist-cmd`. +:option:`--no-defaults` option to disable the standard set entirely.) The order of commands in the manifest template matters: initially, we have the list of default files as described above, and each command in the template adds @@ -166,36 +234,45 @@ should always be slash-separated; the Distutils will take care of converting them to the standard representation on your platform. That way, the manifest template is portable across operating systems. - -.. _manifest-options: - -Manifest-related options -======================== - -The normal course of operations for the :command:`sdist` command is as follows: - -* if the manifest file, :file:`MANIFEST` doesn't exist, read :file:`MANIFEST.in` - and create the manifest - -* if neither :file:`MANIFEST` nor :file:`MANIFEST.in` exist, create a manifest - with just the default file set - -* if either :file:`MANIFEST.in` or the setup script (:file:`setup.py`) are more - recent than :file:`MANIFEST`, recreate :file:`MANIFEST` by reading - :file:`MANIFEST.in` - -* use the list of files now in :file:`MANIFEST` (either just generated or read - in) to create the source distribution archive(s) - -There are a couple of options that modify this behaviour. First, use the -:option:`--no-defaults` and :option:`--no-prune` to disable the standard -"include" and "exclude" sets. - -Second, you might just want to (re)generate the manifest, but not create a source -distribution:: - - python setup.py sdist --manifest-only - -:option:`-o` is a shortcut for :option:`--manifest-only`. - - +Commands +-------- + +The manifest template commands are: + ++-------------------------------------------+-----------------------------------------------+ +| Command | Description | ++===========================================+===============================================+ +| :command:`include pat1 pat2 ...` | include all files matching any of the listed | +| | patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`exclude pat1 pat2 ...` | exclude all files matching any of the listed | +| | patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`recursive-include dir pat1 pat2 | include all files under *dir* matching any of | +| ...` | the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`recursive-exclude dir pat1 pat2 | exclude all files under *dir* matching any of | +| ...` | the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`global-include pat1 pat2 ...` | include all files anywhere in the source tree | +| | matching --- & any of the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`global-exclude pat1 pat2 ...` | exclude all files anywhere in the source tree | +| | matching --- & any of the listed patterns | ++-------------------------------------------+-----------------------------------------------+ +| :command:`prune dir` | exclude all files under *dir* | ++-------------------------------------------+-----------------------------------------------+ +| :command:`graft dir` | include all files under *dir* | ++-------------------------------------------+-----------------------------------------------+ + +The patterns here are Unix-style "glob" patterns: ``*`` matches any sequence of +regular filename characters, ``?`` matches any single regular filename +character, and ``[range]`` matches any of the characters in *range* (e.g., +``a-z``, ``a-zA-Z``, ``a-f0-9_.``). The definition of "regular filename +character" is platform-specific: on Unix it is anything except slash; on Windows +anything except backslash or colon. + +.. versionchanged:: 2.7 + An existing generated :file:`MANIFEST` will be regenerated without + :command:`sdist` comparing its modification time to the one of + :file:`MANIFEST.in` or :file:`setup.py`. diff --git a/Doc/distutils/uploading.rst b/Doc/distutils/uploading.rst index 66f712b..936402b 100644 --- a/Doc/distutils/uploading.rst +++ b/Doc/distutils/uploading.rst @@ -13,7 +13,7 @@ package data if the author of the package wishes to. The distutils command The command is invoked immediately after building one or more distribution files. For example, the command :: - python setup.py sdist bdist_wininst upload + python setup.py sdist bdist_wininst upload will cause the source distribution and the Windows installer to be uploaded to PyPI. Note that these will be uploaded even if they are built using an earlier @@ -22,11 +22,14 @@ line for the invocation including the :command:`upload` command are uploaded. The :command:`upload` command uses the username, password, and repository URL from the :file:`$HOME/.pypirc` file (see section :ref:`pypirc` for more on this -file). +file). If a :command:`register` command was previously called in the same command, +and if the password was entered in the prompt, :command:`upload` will reuse the +entered password. This is useful if you do not want to store a clear text +password in the :file:`$HOME/.pypirc` file. You can specify another PyPI server with the :option:`--repository=*url*` option:: - python setup.py sdist bdist_wininst upload -r http://example.com/pypi + python setup.py sdist bdist_wininst upload -r http://example.com/pypi See section :ref:`pypirc` for more on defining several servers. @@ -41,3 +44,34 @@ Other :command:`upload` options include :option:`--repository=<url>` or :option:`--show-response` (which displays the full response text from the PyPI server for help in debugging upload problems). +PyPI package display +==================== + +The ``long_description`` field plays a special role at PyPI. It is used by +the server to display a home page for the registered package. + +If you use the `reStructuredText <http://docutils.sourceforge.net/rst.html>`_ +syntax for this field, PyPI will parse it and display an HTML output for +the package home page. + +The ``long_description`` field can be attached to a text file located +in the package:: + + from distutils.core import setup + + with open('README.txt') as file: + long_description = file.read() + + setup(name='Distutils', + long_description=long_description) + +In that case, :file:`README.txt` is a regular reStructuredText text file located +in the root of the package besides :file:`setup.py`. + +To prevent registering broken reStructuredText content, you can use the +:program:`rst2html` program that is provided by the :mod:`docutils` package +and check the ``long_description`` from the command line:: + + $ python setup.py --long-description | rst2html.py > output.html + +:mod:`docutils` will display a warning if there's something wrong with your syntax. diff --git a/Doc/documenting/markup.rst b/Doc/documenting/markup.rst index c6f5b74..d7d23cd 100644 --- a/Doc/documenting/markup.rst +++ b/Doc/documenting/markup.rst @@ -524,10 +524,6 @@ in a different style: If you don't need the "variable part" indication, use the standard ````code```` instead. -.. describe:: var - - A Python or C variable or parameter name. - The following roles generate external links: @@ -699,10 +695,10 @@ tables of contents. The ``toctree`` directive is the central element. .. toctree:: :maxdepth: 2 - intro.rst - strings.rst - datatypes.rst - numeric.rst + intro + strings + datatypes + numeric (many more files listed here) This accomplishes two things: @@ -710,8 +706,8 @@ tables of contents. The ``toctree`` directive is the central element. * Tables of contents from all those files are inserted, with a maximum depth of two, that means one nested heading. ``toctree`` directives in those files are also taken into account. - * Sphinx knows that the relative order of the files ``intro.rst``, - ``strings.rst`` and so forth, and it knows that they are children of the + * Sphinx knows that the relative order of the files ``intro``, + ``strings`` and so forth, and it knows that they are children of the shown file, the library index. From this information it generates "next chapter", "previous chapter" and "parent chapter" links. diff --git a/Doc/documenting/style.rst b/Doc/documenting/style.rst index 6145559..c3dded9 100644 --- a/Doc/documenting/style.rst +++ b/Doc/documenting/style.rst @@ -7,7 +7,7 @@ The Python documentation should follow the `Apple Publications Style Guide`_ wherever possible. This particular style guide was selected mostly because it seems reasonable and is easy to get online. -Topics which are not covered in the Apple's style guide will be discussed in +Topics which are not covered in Apple's style guide will be discussed in this document. All reST files use an indentation of 3 spaces. The maximum line length is 80 diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst index 588534a..ad6a9f3 100644 --- a/Doc/extending/extending.rst +++ b/Doc/extending/extending.rst @@ -89,10 +89,8 @@ example, the single expression ``"ls -l"``) to the arguments passed to the C function. The C function always has two arguments, conventionally named *self* and *args*. -The *self* argument is only used when the C function implements a built-in -method, not a function. In the example, *self* will always be a *NULL* pointer, -since we are defining a function, not a method. (This is done so that the -interpreter doesn't have to understand two different types of C functions.) +The *self* argument points to the module object for module-level functions; +for a method it would point to the object instance. The *args* argument will be a pointer to a Python tuple object containing the arguments. Each item of the tuple corresponds to an argument in the call's @@ -391,12 +389,7 @@ source distribution. A more substantial example module is included in the Python source distribution as :file:`Modules/xxmodule.c`. This file may be used as a template or simply -read as an example. The :program:`modulator.py` script included in the source -distribution or Windows install provides a simple graphical user interface for -declaring the functions and objects which a module should implement, and can -generate a template which can be filled in. The script lives in the -:file:`Tools/modulator/` directory; see the :file:`README` file there for more -information. +read as an example. .. _compilation: @@ -1084,7 +1077,7 @@ already if the symbol ``__cplusplus`` is defined (all recent C++ compilers define this symbol). -.. _using-cobjects: +.. _using-capsules: Providing a C API for an Extension Module ========================================= @@ -1120,23 +1113,40 @@ avoid name clashes with other extension modules (as discussed in section other extension modules must be exported in a different way. Python provides a special mechanism to pass C-level information (pointers) from -one extension module to another one: CObjects. A CObject is a Python data type -which stores a pointer (:ctype:`void \*`). CObjects can only be created and +one extension module to another one: Capsules. A Capsule is a Python data type +which stores a pointer (:ctype:`void \*`). Capsules can only be created and accessed via their C API, but they can be passed around like any other Python object. In particular, they can be assigned to a name in an extension module's namespace. Other extension modules can then import this module, retrieve the -value of this name, and then retrieve the pointer from the CObject. +value of this name, and then retrieve the pointer from the Capsule. -There are many ways in which CObjects can be used to export the C API of an -extension module. Each name could get its own CObject, or all C API pointers -could be stored in an array whose address is published in a CObject. And the +There are many ways in which Capsules can be used to export the C API of an +extension module. Each function could get its own Capsule, or all C API pointers +could be stored in an array whose address is published in a Capsule. And the various tasks of storing and retrieving the pointers can be distributed in different ways between the module providing the code and the client modules. +Whichever method you choose, it's important to name your Capsules properly. +The function :cfunc:`PyCapsule_New` takes a name parameter +(:ctype:`const char \*`); you're permitted to pass in a *NULL* name, but +we strongly encourage you to specify a name. Properly named Capsules provide +a degree of runtime type-safety; there is no feasible way to tell one unnamed +Capsule from another. + +In particular, Capsules used to expose C APIs should be given a name following +this convention:: + + modulename.attributename + +The convenience function :cfunc:`PyCapsule_Import` makes it easy to +load a C API provided via a Capsule, but only if the Capsule's name +matches this convention. This behavior gives C API users a high degree +of certainty that the Capsule they load contains the correct C API. + The following example demonstrates an approach that puts most of the burden on the writer of the exporting module, which is appropriate for commonly used library modules. It stores all C API pointers (just one in the example!) in an -array of :ctype:`void` pointers which becomes the value of a CObject. The header +array of :ctype:`void` pointers which becomes the value of a Capsule. The header file corresponding to the module provides a macro that takes care of importing the module and retrieving its C API pointers; client modules only have to call this macro before accessing the C API. @@ -1198,8 +1208,8 @@ function must take care of initializing the C API pointer array:: /* Initialize the C API pointer array */ PySpam_API[PySpam_System_NUM] = (void *)PySpam_System; - /* Create a CObject containing the API pointer array's address */ - c_api_object = PyCObject_FromVoidPtr((void *)PySpam_API, NULL); + /* Create a Capsule containing the API pointer array's address */ + c_api_object = PyCapsule_New((void *)PySpam_API, "spam._C_API", NULL); if (c_api_object != NULL) PyModule_AddObject(m, "_C_API", c_api_object); @@ -1241,28 +1251,14 @@ like this:: #define PySpam_System \ (*(PySpam_System_RETURN (*)PySpam_System_PROTO) PySpam_API[PySpam_System_NUM]) - /* Return -1 and set exception on error, 0 on success. */ + /* Return -1 on error, 0 on success. + * PyCapsule_Import will set an exception if there's an error. + */ static int import_spam(void) { - PyObject *c_api_object; - PyObject *module; - - module = PyImport_ImportModule("spam"); - if (module == NULL) - return -1; - - c_api_object = PyObject_GetAttrString(module, "_C_API"); - if (c_api_object == NULL) { - Py_DECREF(module); - return -1; - } - if (PyCObject_Check(c_api_object)) - PySpam_API = (void **)PyCObject_AsVoidPtr(c_api_object); - - Py_DECREF(c_api_object); - Py_DECREF(module); - return 0; + PySpam_API = (void **)PyCapsule_Import("spam._C_API", 0); + return (PySpam_API != NULL) ? 0 : -1; } #endif @@ -1294,11 +1290,11 @@ The main disadvantage of this approach is that the file :file:`spammodule.h` is rather complicated. However, the basic structure is the same for each function that is exported, so it has to be learned only once. -Finally it should be mentioned that CObjects offer additional functionality, +Finally it should be mentioned that Capsules offer additional functionality, which is especially useful for memory allocation and deallocation of the pointer -stored in a CObject. The details are described in the Python/C API Reference -Manual in the section :ref:`cobjects` and in the implementation of CObjects (files -:file:`Include/cobject.h` and :file:`Objects/cobject.c` in the Python source +stored in a Capsule. The details are described in the Python/C API Reference +Manual in the section :ref:`capsules` and in the implementation of Capsules (files +:file:`Include/pycapsule.h` and :file:`Objects/pycapsule.c` in the Python source code distribution). .. rubric:: Footnotes diff --git a/Doc/extending/windows.rst b/Doc/extending/windows.rst index aac1d2d..beb2ee4 100644 --- a/Doc/extending/windows.rst +++ b/Doc/extending/windows.rst @@ -114,7 +114,7 @@ described here are distributed with the Python sources in the Now your options are: #. Copy :file:`example.sln` and :file:`example.vcproj`, rename them to - :file:`spam.\*`, and edit them by hand, or + :file:`spam.\*`, and edit them by hand, or #. Create a brand new project; instructions are below. @@ -175,16 +175,16 @@ If your module creates a new type, you may have trouble with this line:: PyObject_HEAD_INIT(&PyType_Type) -Change it to:: +Static type object initializers in extension modules may cause +compiles to fail with an error message like "initializer not a +constant". This shows up when building DLL under MSVC. Change it to:: PyObject_HEAD_INIT(NULL) and add the following to the module initialization function:: - MyObject_Type.ob_type = &PyType_Type; - -Refer to section 3 of the `Python FAQ <http://www.python.org/doc/faq>`_ for -details on why you must do this. + if (PyType_Ready(&MyObject_Type) < 0) + return NULL; .. _dynamic-linking: diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 2e2ce2e..0943796 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -28,7 +28,7 @@ coding-style conflicts. In C there are many different ways to place the braces. If you're used to reading and writing code that uses one style, you will feel at least slightly uneasy when reading (or being required to write) another style. -Many coding styles place begin/end brackets on a line by themself. This makes +Many coding styles place begin/end brackets on a line by themselves. This makes programs considerably longer and wastes valuable screen space, making it harder to get a good overview of a program. Ideally, a function should fit on one screen (say, 20-30 lines). 20 lines of Python can do a lot more work than 20 @@ -75,9 +75,9 @@ necessary to make ``eval(repr(f)) == f`` true for any float f. The ``str()`` function prints fewer digits and this often results in the more sensible number that was probably intended:: - >>> 0.2 - 0.20000000000000001 - >>> print 0.2 + >>> 1.1 - 0.9 + 0.20000000000000007 + >>> print 1.1 - 0.9 0.2 One of the consequences of this is that it is error-prone to compare the result diff --git a/Doc/faq/gui.rst b/Doc/faq/gui.rst index 1f2ae09..d37f480 100644 --- a/Doc/faq/gui.rst +++ b/Doc/faq/gui.rst @@ -6,18 +6,15 @@ Graphic User Interface FAQ .. contents:: -General GUI Questions -===================== - What platform-independent GUI toolkits exist for Python? --------------------------------------------------------- +======================================================== Depending on what platform(s) you are aiming at, there are several. .. XXX check links Tkinter -''''''' +------- Standard builds of Python include an object-oriented interface to the Tcl/Tk widget set, called Tkinter. This is probably the easiest to install and use. @@ -26,22 +23,27 @@ page at http://www.tcl.tk. Tcl/Tk is fully portable to the MacOS, Windows, and Unix platforms. wxWidgets -''''''''' +--------- + +wxWidgets (http://www.wxwidgets.org) is a free, portable GUI class +library written in C++ that provides a native look and feel on a +number of platforms, with Windows, MacOS X, GTK, X11, all listed as +current stable targets. Language bindings are available for a number +of languages including Python, Perl, Ruby, etc. -wxWidgets is a GUI class library written in C++ that's a portable -interface to various platform-specific libraries, and that has a -Python interface called `wxPython <http://www.wxpython.org>`__. +wxPython (http://www.wxpython.org) is the Python binding for +wxwidgets. While it often lags slightly behind the official wxWidgets +releases, it also offers a number of features via pure Python +extensions that are not available in other language bindings. There +is an active wxPython user and developer community. -wxWidgets preserves the look and feel of the -underlying graphics toolkit, and has a large set of widgets and -collection of GDI classes. See `the wxWidgets page -<http://www.wxwidgets.org>`_ for more details. +Both wxWidgets and wxPython are free, open source, software with +permissive licences that allow their use in commercial products as +well as in freeware or shareware. -wxWidgets supports Windows and MacOS; on Unix variants, -it supports both GTk+ and Motif toolkits. Qt -''' +--- There are bindings available for the Qt toolkit (`PyQt <http://www.riverbankcomputing.co.uk/software/pyqt/>`_) and for KDE (`PyKDE <http://www.riverbankcomputing.co.uk/software/pykde/intro>`__). If @@ -52,13 +54,13 @@ Qt 4.5 upwards is licensed under the LGPL license) a Qt license from `Trolltech <http://www.trolltech.com>`_. Gtk+ -'''' +---- PyGtk bindings for the `Gtk+ toolkit <http://www.gtk.org>`_ have been implemented by James Henstridge; see <http://www.pygtk.org>. FLTK -'''' +---- Python bindings for `the FLTK toolkit <http://www.fltk.org>`_, a simple yet powerful and mature cross-platform windowing system, are available from `the @@ -66,7 +68,7 @@ PyFLTK project <http://pyfltk.sourceforge.net>`_. FOX -''' +---- A wrapper for `the FOX toolkit <http://www.fox-toolkit.org/>`_ called `FXpy <http://fxpy.sourceforge.net/>`_ is available. FOX supports both Unix variants @@ -74,13 +76,13 @@ and Windows. OpenGL -'''''' +------ For OpenGL bindings, see `PyOpenGL <http://pyopengl.sourceforge.net>`_. What platform-specific GUI toolkits exist for Python? ------------------------------------------------------ +======================================================== `The Mac port <http://python.org/download/mac>`_ by Jack Jansen has a rich and ever-growing set of modules that support the native Mac toolbox calls. The port diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index 4b0c3d3..c122d72 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -458,7 +458,7 @@ To rename a file, use ``os.rename(old_path, new_path)``. To truncate a file, open it using ``f = open(filename, "r+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's -also ```os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where +also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where ``fd`` is the file descriptor (a small integer). The :mod:`shutil` module also contains a number of functions to work on files @@ -672,9 +672,8 @@ Yes. Here's a simple example that uses httplib:: if reply != 200: sys.stdout.write(httpobj.getfile().read()) -Note that in general for URL-encoded POST operations, query strings must be -quoted by using :func:`urllib.quote`. For example to send name="Guy Steele, -Jr.":: +Note that in general for percent-encoded POST operations, query strings must be +quoted using :func:`urllib.quote`. For example to send name="Guy Steele, Jr.":: >>> from urllib import quote >>> x = quote("Guy Steele, Jr.") diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index c54bb28..e53facd 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -944,7 +944,7 @@ Is there an equivalent to Perl's chomp() for removing trailing newlines from str ------------------------------------------------------------------------------------- Starting with Python 2.2, you can use ``S.rstrip("\r\n")`` to remove all -occurences of any line terminator from the end of the string ``S`` without +occurrences of any line terminator from the end of the string ``S`` without removing other trailing whitespace. If the string ``S`` represents more than one line, with several empty lines at the end, the line terminators for all the blank lines will be removed:: @@ -979,8 +979,8 @@ and then convert decimal strings to numeric values using :func:`int` or :func:`float`. ``split()`` supports an optional "sep" parameter which is useful if the line uses something other than whitespace as a separator. -For more complicated input parsing, regular expressions more powerful than C's -:cfunc:`sscanf` and better suited for the task. +For more complicated input parsing, regular expressions are more powerful +than C's :cfunc:`sscanf` and better suited for the task. What does 'UnicodeError: ASCII [decoding,encoding] error: ordinal not in range(128)' mean? diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst index 353c400..47c09ad 100644 --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -286,20 +286,18 @@ Embedding the Python interpreter in a Windows app can be summarized as follows: 1. Do _not_ build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL's. (This is the - first key undocumented fact.) Instead, link to :file:`python{NN}.dll`; it is - typically installed in ``C:\Windows\System``. NN is the Python version, a + first key undocumented fact.) Instead, link to :file:`python{NN}.dll`; it is + typically installed in ``C:\Windows\System``. *NN* is the Python version, a number such as "23" for Python 2.3. - You can link to Python statically or dynamically. Linking statically means - linking against :file:`python{NN}.lib`, while dynamically linking means - linking against :file:`python{NN}.dll`. The drawback to dynamic linking is - that your app won't run if :file:`python{NN}.dll` does not exist on your - system. (General note: :file:`python{NN}.lib` is the so-called "import lib" - corresponding to :file:`python.dll`. It merely defines symbols for the - linker.) + You can link to Python in two different ways. Load-time linking means + linking against :file:`python{NN}.lib`, while run-time linking means linking + against :file:`python{NN}.dll`. (General note: :file:`python{NN}.lib` is the + so-called "import lib" corresponding to :file:`python{NN}.dll`. It merely + defines symbols for the linker.) - Linking dynamically greatly simplifies link options; everything happens at - run time. Your code must load :file:`python{NN}.dll` using the Windows + Run-time linking greatly simplifies link options; everything happens at run + time. Your code must load :file:`python{NN}.dll` using the Windows ``LoadLibraryEx()`` routine. The code must also use access routines and data in :file:`python{NN}.dll` (that is, Python's C API's) using pointers obtained by the Windows ``GetProcAddress()`` routine. Macros can make using these @@ -308,6 +306,8 @@ Embedding the Python interpreter in a Windows app can be summarized as follows: Borland note: convert :file:`python{NN}.lib` to OMF format using Coff2Omf.exe first. + .. XXX what about static linking? + 2. If you use SWIG, it is easy to create a Python "extension module" that will make the app's data and methods available to Python. SWIG will handle just about all the grungy details for you. The result is C code that you link @@ -441,7 +441,7 @@ present, and ``getch()`` which gets one character without echoing it. How do I emulate os.kill() in Windows? -------------------------------------- -To terminate a process, you can use ctypes:: +Prior to Python 2.7 and 3.2, to terminate a process, you can use :mod:`ctypes`:: import ctypes @@ -451,6 +451,11 @@ To terminate a process, you can use ctypes:: handle = kernel32.OpenProcess(1, 0, pid) return (0 != kernel32.TerminateProcess(handle, 0)) +In 2.7 and 3.2, :func:`os.kill` is implemented similar to the above function, +with the additional feature of being able to send CTRL+C and CTRL+BREAK +to console subprocesses which are designed to handle those signals. See +:func:`os.kill` for further details. + Why does os.path.isdir() fail on NT shared directories? ------------------------------------------------------- @@ -589,7 +594,7 @@ Warning about CTL3D32 version from installer The Python installer issues a warning like this:: - This version uses ``CTL3D32.DLL`` which is not the correct version. + This version uses CTL3D32.DLL which is not the correct version. This version is used for windows NT applications only. Tim Peters: diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 5cf3aee..7822634 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -19,7 +19,7 @@ Glossary 2to3 A tool that tries to convert Python 2.x code to Python 3.x code by - handling most of the incompatibilites which can be detected by parsing the + handling most of the incompatibilities which can be detected by parsing the source and traversing the parse tree. 2to3 is available in the standard library as :mod:`lib2to3`; a standalone @@ -27,7 +27,7 @@ Glossary :ref:`2to3-reference`. abstract base class - Abstract Base Classes (abbreviated ABCs) complement :term:`duck-typing` by + :ref:`abstract-base-classes` complement :term:`duck-typing` by providing a way to define interfaces when other techniques like :func:`hasattr` would be clumsy. Python comes with many built-in ABCs for data structures (in the :mod:`collections` module), numbers (in the @@ -63,6 +63,9 @@ Glossary "intermediate language" is said to run on a :term:`virtual machine` that executes the machine code corresponding to each bytecode. + A list of bytecode instructions can be found in the documentation for + :ref:`the dis module <bytecodes>`. + class A template for creating user-defined objects. Class definitions normally contain method definitions which operate on instances of the @@ -103,9 +106,10 @@ Glossary See :pep:`343`. CPython - The canonical implementation of the Python programming language. The - term "CPython" is used in contexts when necessary to distinguish this - implementation from others such as Jython or IronPython. + The canonical implementation of the Python programming language, as + distributed on `python.org <http://python.org>`_. The term "CPython" + is used when necessary to distinguish this implementation from others + such as Jython or IronPython. decorator A function returning another function, usually applied as a function @@ -140,10 +144,9 @@ Glossary For more information about descriptors' methods, see :ref:`descriptors`. dictionary - An associative array, where arbitrary keys are mapped to values. The use - of :class:`dict` closely resembles that for :class:`list`, but the keys can - be any object with a :meth:`__hash__` function, not just integers. - Called a hash in Perl. + An associative array, where arbitrary keys are mapped to values. The keys + can be any object with :meth:`__hash__` function and :meth:`__eq__` + methods. Called a hash in Perl. docstring A string literal which appears as the first expression in a class, @@ -154,9 +157,9 @@ Glossary object. duck-typing - A pythonic programming style which determines an object's type by inspection - of its method or attribute signature rather than by explicit relationship - to some type object ("If it looks like a duck and quacks like a duck, it + A programming style which does not look at an object's type to determine + if it has the right interface; instead, the method or attribute is simply + called or used ("If it looks like a duck and quacks like a duck, it must be a duck.") By emphasizing interfaces rather than specific types, well-designed code improves its flexibility by allowing polymorphic substitution. Duck-typing avoids tests using :func:`type` or @@ -182,21 +185,28 @@ Glossary not expressions. extension module - A module written in C or C++, using Python's C API to interact with the core and - with user code. + A module written in C or C++, using Python's C API to interact with the + core and with user code. finder An object that tries to find the :term:`loader` for a module. It must implement a method named :meth:`find_module`. See :pep:`302` for details. + floor division + Mathematical division that rounds down to nearest integer. The floor + division operator is ``//``. For example, the expression ``11 // 4`` + evaluates to ``2`` in contrast to the ``2.75`` returned by float true + division. Note that ``(-11) // 4`` is ``-3`` because that is ``-2.75`` + rounded *downward*. See :pep:`238`. + function A series of statements which returns some value to a caller. It can also be passed zero or more arguments which may be used in the execution of the body. See also :term:`argument` and :term:`method`. __future__ - A pseudo module which programmers can use to enable new language features + A pseudo-module which programmers can use to enable new language features which are not compatible with the current interpreter. For example, the expression ``11/4`` currently evaluates to ``2``. If the module in which it is executed had enabled *true division* by executing:: @@ -221,13 +231,13 @@ Glossary generator A function which returns an iterator. It looks like a normal function - except that values are returned to the caller using a :keyword:`yield` - statement instead of a :keyword:`return` statement. Generator functions - often contain one or more :keyword:`for` or :keyword:`while` loops which - :keyword:`yield` elements back to the caller. The function execution is - stopped at the :keyword:`yield` keyword (returning the result) and is - resumed there when the next element is requested by calling the - :meth:`next` method of the returned iterator. + except that it contains :keyword:`yield` statements for producing a series + a values usable in a for-loop or that can be retrieved one at a time with + the :func:`next` function. Each :keyword:`yield` temporarily suspends + processing, remembering the location execution state (including local + variables and pending try-statements). When the generator resumes, it + picks-up where it left-off (in contrast to functions which start fresh on + every invocation). .. index:: single: generator expression @@ -244,16 +254,25 @@ Glossary See :term:`global interpreter lock`. global interpreter lock - The lock used by Python threads to assure that only one thread - executes in the :term:`CPython` :term:`virtual machine` at a time. - This simplifies the CPython implementation by assuring that no two - processes can access the same memory at the same time. Locking the - entire interpreter makes it easier for the interpreter to be - multi-threaded, at the expense of much of the parallelism afforded by - multi-processor machines. Efforts have been made in the past to - create a "free-threaded" interpreter (one which locks shared data at a - much finer granularity), but so far none have been successful because - performance suffered in the common single-processor case. + The mechanism used by the :term:`CPython` interpreter to assure that + only one thread executes Python :term:`bytecode` at a time. + This simplifies the CPython implementation by making the object model + (including critical built-in types such as :class:`dict`) implicitly + safe against concurrent access. Locking the entire interpreter + makes it easier for the interpreter to be multi-threaded, at the + expense of much of the parallelism afforded by multi-processor + machines. + + However, some extension modules, either standard or third-party, + are designed so as to release the GIL when doing computationally-intensive + tasks such as compression or hashing. Also, the GIL is always released + when doing I/O. + + Past efforts to create a "free-threaded" interpreter (one which locks + shared data at a much finer granularity) have not been successful + because performance suffered in the common single-processor case. It + is believed that overcoming this performance issue would make the + implementation much more complicated and therefore costlier to maintain. hashable An object is *hashable* if it has a hash value which never changes during @@ -272,9 +291,7 @@ Glossary IDLE An Integrated Development Environment for Python. IDLE is a basic editor and interpreter environment which ships with the standard distribution of - Python. Good for beginners, it also serves as clear example code for - those wanting to implement a moderately sophisticated, multi-platform GUI - application. + Python. immutable An object with a fixed value. Immutable objects include numbers, strings and @@ -349,12 +366,52 @@ Glossary More information can be found in :ref:`typeiter`. + key function + A key function or collation function is a callable that returns a value + used for sorting or ordering. For example, :func:`locale.strxfrm` is + used to produce a sort key that is aware of locale specific sort + conventions. + + A number of tools in Python accept key functions to control how elements + are ordered or grouped. They include :func:`min`, :func:`max`, + :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, + :func:`heapq.nlargest`, and :func:`itertools.groupby`. + + There are several ways to create a key function. For example. the + :meth:`str.lower` method can serve as a key function for case insensitive + sorts. Alternatively, an ad-hoc key function can be built from a + :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, + the :mod:`operator` module provides three key function constuctors: + :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and + :func:`~operator.methodcaller`. See the :ref:`Sorting HOW TO + <sortinghowto>` for examples of how to create and use key functions. + keyword argument Arguments which are preceded with a ``variable_name=`` in the call. The variable name designates the local name in the function to which the value is assigned. ``**`` is used to accept or pass a dictionary of keyword arguments. See :term:`argument`. + key function + A key function or collation function is a callable that returns a value + used for sorting or ordering. For example, :func:`locale.strxfrm` is + used to produce a sort key that is aware of locale specific sort + conventions. + + A number of tools in Python accept key functions to control how elements + are ordered or grouped. They include :func:`min`, :func:`max`, + :func:`sorted`, :meth:`list.sort`, :func:`heapq.nsmallest`, + :func:`heapq.nlargest`, and :func:`itertools.groupby`. + + There are several ways to create a key function. For example. the + :meth:`str.lower` method can serve as a key function for case insensitive + sorts. Alternatively, an ad-hoc key function can be built from a + :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, + the :mod:`operator` module provides three key function constuctors: + :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and + :func:`~operator.methodcaller`. See the :ref:`sortinghowto` for + examples of how to create and use key functions. + lambda An anonymous inline function consisting of a single :term:`expression` which is evaluated when the function is called. The syntax to create @@ -385,8 +442,11 @@ Glossary :term:`finder`. See :pep:`302` for details. mapping - A container object (such as :class:`dict`) which supports arbitrary key - lookups using the special method :meth:`__getitem__`. + A container object that supports arbitrary key lookups and implements the + methods specified in the :class:`Mapping` or :class:`MutableMapping` + :ref:`abstract base classes <abstract-base-classes>`. Examples include + :class:`dict`, :class:`collections.defaultdict`, + :class:`collections.OrderedDict` and :class:`collections.Counter`. metaclass The class of a class. Class definitions create a class name, a class @@ -542,6 +602,13 @@ Glossary object has a type. An object's type is accessible as its :attr:`__class__` attribute or can be retrieved with ``type(obj)``. + view + The objects returned from :meth:`dict.viewkeys`, :meth:`dict.viewvalues`, + and :meth:`dict.viewitems` are called dictionary views. They are lazy + sequences that will see changes in the underlying dictionary. To force + the dictionary view to become a full list use ``list(dictview)``. See + :ref:`dict-views`. + virtual machine A computer defined entirely in software. Python's virtual machine executes the :term:`bytecode` emitted by the bytecode compiler. diff --git a/Doc/howto/cporting.rst b/Doc/howto/cporting.rst index c0b5933..8030f13 100644 --- a/Doc/howto/cporting.rst +++ b/Doc/howto/cporting.rst @@ -48,9 +48,9 @@ Python 3.0's :func:`str` (``PyString_*`` functions in C) type is equivalent to compatibility with 3.0, :ctype:`PyUnicode` should be used for textual data and :ctype:`PyBytes` for binary data. It's also important to remember that :ctype:`PyBytes` and :ctype:`PyUnicode` in 3.0 are not interchangeable like -:ctype:`PyString` and :ctype:`PyString` are in 2.x. The following example shows -best practices with regards to :ctype:`PyUnicode`, :ctype:`PyString`, and -:ctype:`PyBytes`. :: +:ctype:`PyString` and :ctype:`PyUnicode` are in 2.x. The following example +shows best practices with regards to :ctype:`PyUnicode`, :ctype:`PyString`, +and :ctype:`PyBytes`. :: #include "stdlib.h" #include "Python.h" diff --git a/Doc/howto/descriptor.rst b/Doc/howto/descriptor.rst index 3cc437f..55a1d62 100644 --- a/Doc/howto/descriptor.rst +++ b/Doc/howto/descriptor.rst @@ -296,7 +296,7 @@ Running the interpreter shows how the function descriptor works in practice:: <bound method D.f of <__main__.D object at 0x00B18C90>> The output suggests that bound and unbound methods are two different types. -While they could have been implemented that way, the actual C implemention of +While they could have been implemented that way, the actual C implementation of :ctype:`PyMethod_Type` in `Objects/classobject.c <http://svn.python.org/view/python/trunk/Objects/classobject.c?view=markup>`_ is a single object with two different representations depending on whether the diff --git a/Doc/howto/doanddont.rst b/Doc/howto/doanddont.rst index 05d8df9..2e9e0b8 100644 --- a/Doc/howto/doanddont.rst +++ b/Doc/howto/doanddont.rst @@ -144,30 +144,44 @@ except: ------- Python has the ``except:`` clause, which catches all exceptions. Since *every* -error in Python raises an exception, this makes many programming errors look -like runtime problems, and hinders the debugging process. +error in Python raises an exception, using ``except:`` can make many +programming errors look like runtime problems, which hinders the debugging +process. -The following code shows a great example:: +The following code shows a great example of why this is bad:: try: foo = opne("file") # misspelled "open" except: sys.exit("could not open file!") -The second line triggers a :exc:`NameError` which is caught by the except -clause. The program will exit, and you will have no idea that this has nothing -to do with the readability of ``"file"``. +The second line triggers a :exc:`NameError`, which is caught by the except +clause. The program will exit, and the error message the program prints will +make you think the problem is the readability of ``"file"`` when in fact +the real error has nothing to do with ``"file"``. -The example above is better written :: +A better way to write the above is :: try: - foo = opne("file") # will be changed to "open" as soon as we run it + foo = opne("file") except IOError: sys.exit("could not open file") -There are some situations in which the ``except:`` clause is useful: for -example, in a framework when running callbacks, it is good not to let any -callback disturb the framework. +When this is run, Python will produce a traceback showing the :exc:`NameError`, +and it will be immediately apparent what needs to be fixed. + +.. index:: bare except, except; bare + +Because ``except:`` catches *all* exceptions, including :exc:`SystemExit`, +:exc:`KeyboardInterrupt`, and :exc:`GeneratorExit` (which is not an error and +should not normally be caught by user code), using a bare ``except:`` is almost +never a good idea. In situations where you need to catch all "normal" errors, +such as in a framework that runs callbacks, you can catch the base class for +all normal exceptions, :exc:`Exception`. Unfortunately in Python 2.x it is +possible for third-party code to raise exceptions that do not inherit from +:exc:`Exception`, so in Python 2.x there are some cases where you may have to +use a bare ``except:`` and manually re-raise the exceptions you don't want +to catch. Exceptions @@ -185,51 +199,60 @@ The following is a very popular anti-idiom :: sys.exit(1) return open(file).readline() -Consider the case the file gets deleted between the time the call to -:func:`os.path.exists` is made and the time :func:`open` is called. That means -the last line will throw an :exc:`IOError`. The same would happen if *file* -exists but has no read permission. Since testing this on a normal machine on -existing and non-existing files make it seem bugless, that means in testing the -results will seem fine, and the code will get shipped. Then an unhandled -:exc:`IOError` escapes to the user, who has to watch the ugly traceback. +Consider the case where the file gets deleted between the time the call to +:func:`os.path.exists` is made and the time :func:`open` is called. In that +case the last line will raise an :exc:`IOError`. The same thing would happen +if *file* exists but has no read permission. Since testing this on a normal +machine on existent and non-existent files makes it seem bugless, the test +results will seem fine, and the code will get shipped. Later an unhandled +:exc:`IOError` (or perhaps some other :exc:`EnvironmentError`) escapes to the +user, who gets to watch the ugly traceback. -Here is a better way to do it. :: +Here is a somewhat better way to do it. :: def get_status(file): try: return open(file).readline() - except (IOError, OSError): - print "file not found" + except EnvironmentError as err: + print "Unable to open file: {}".format(err) sys.exit(1) -In this version, \*either\* the file gets opened and the line is read (so it -works even on flaky NFS or SMB connections), or the message is printed and the -application aborted. +In this version, *either* the file gets opened and the line is read (so it +works even on flaky NFS or SMB connections), or an error message is printed +that provides all the available information on why the open failed, and the +application is aborted. -Still, :func:`get_status` makes too many assumptions --- that it will only be -used in a short running script, and not, say, in a long running server. Sure, -the caller could do something like :: +However, even this version of :func:`get_status` makes too many assumptions --- +that it will only be used in a short running script, and not, say, in a long +running server. Sure, the caller could do something like :: try: status = get_status(log) except SystemExit: status = None -So, try to make as few ``except`` clauses in your code --- those will usually be -a catch-all in the :func:`main`, or inside calls which should always succeed. +But there is a better way. You should try to use as few ``except`` clauses in +your code as you can --- the ones you do use will usually be inside calls which +should always succeed, or a catch-all in a main function. -So, the best version is probably :: +So, an even better version of :func:`get_status()` is probably :: def get_status(file): return open(file).readline() -The caller can deal with the exception if it wants (for example, if it tries +The caller can deal with the exception if it wants (for example, if it tries several files in a loop), or just let the exception filter upwards to *its* caller. -The last version is not very good either --- due to implementation details, the -file would not be closed when an exception is raised until the handler finishes, -and perhaps not at all in non-C implementations (e.g., Jython). :: +But the last version still has a serious problem --- due to implementation +details in CPython, the file would not be closed when an exception is raised +until the exception handler finishes; and, worse, in other implementations +(e.g., Jython) it might not be closed at all regardless of whether or not +an exception is raised. + +The best version of this function uses the ``open()`` call as a context +manager, which will ensure that the file gets closed as soon as the +function returns:: def get_status(file): with open(file) as fp: @@ -258,23 +281,22 @@ Compare:: More useful functions in :mod:`os.path`: :func:`basename`, :func:`dirname` and :func:`splitext`. -There are also many useful built-in functions people seem not to be aware of for -some reason: :func:`min` and :func:`max` can find the minimum/maximum of any -sequence with comparable semantics, for example, yet many people write their own -:func:`max`/:func:`min`. Another highly useful function is :func:`reduce`. A -classical use of :func:`reduce` is something like :: - - import sys, operator - nums = map(float, sys.argv[1:]) - print reduce(operator.add, nums)/len(nums) - -This cute little script prints the average of all numbers given on the command -line. The :func:`reduce` adds up all the numbers, and the rest is just some -pre- and postprocessing. - -On the same note, note that :func:`float`, :func:`int` and :func:`long` all -accept arguments of type string, and so are suited to parsing --- assuming you -are ready to deal with the :exc:`ValueError` they raise. +There are also many useful built-in functions people seem not to be aware of +for some reason: :func:`min` and :func:`max` can find the minimum/maximum of +any sequence with comparable semantics, for example, yet many people write +their own :func:`max`/:func:`min`. Another highly useful function is +:func:`reduce` which can be used to repeatly apply a binary operation to a +sequence, reducing it to a single value. For example, compute a factorial +with a series of multiply operations:: + + >>> n = 4 + >>> import operator + >>> reduce(operator.mul, range(1, n+1)) + 24 + +When it comes to parsing numbers, note that :func:`float`, :func:`int` and +:func:`long` all accept string arguments and will reject ill-formed strings +by raising an :exc:`ValueError`. Using Backslash to Continue Statements diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index f553ceb..4ae216a 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -5,9 +5,6 @@ :Author: A. M. Kuchling :Release: 0.31 -(This is a first draft. Please send comments/error reports/suggestions to -amk@amk.ca.) - In this document, we'll take a tour of Python's features suitable for implementing programs in a functional style. After an introduction to the concepts of functional programming, we'll look at language features such as diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 1ab291a..1523c48 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -5,7 +5,6 @@ **************************** :Author: A.M. Kuchling <amk@amk.ca> -:Release: 0.05 .. TODO: Document lookbehind assertions @@ -82,7 +81,7 @@ devoted to discussing various metacharacters and what they do. Here's a complete list of the metacharacters; their meanings will be discussed in the rest of this HOWTO. :: - . ^ $ * + ? { [ ] \ | ( ) + . ^ $ * + ? { } [ ] \ | ( ) The first metacharacters we'll look at are ``[`` and ``]``. They're used for specifying a character class, which is a set of characters that you wish to @@ -113,7 +112,10 @@ meaning: ``\[`` or ``\\``. Some of the special sequences beginning with ``'\'`` represent predefined sets of characters that are often useful, such as the set of digits, the set of letters, or the set of anything that isn't whitespace. The following predefined -special sequences are available: +special sequences are a subset of those available. The equivalent classes are +for byte string patterns. For a complete list of sequences and expanded class +definitions for Unicode string patterns, see the last part of +:ref:`Regular Expression Syntax <re-syntax>`. ``\d`` Matches any decimal digit; this is equivalent to the class ``[0-9]``. @@ -264,7 +266,7 @@ performing string substitutions. :: >>> import re >>> p = re.compile('ab*') >>> print p - <_sre.SRE_Pattern object at 80b4150> + <_sre.SRE_Pattern object at 0x...> :func:`re.compile` also accepts an optional *flags* argument, used to enable various special features and syntax variations. We'll go over the available @@ -377,7 +379,7 @@ Python interpreter, import the :mod:`re` module, and compile a RE:: >>> import re >>> p = re.compile('[a-z]+') >>> p - <_sre.SRE_Pattern object at 80c3c28> + <_sre.SRE_Pattern object at 0x...> Now, you can try matching various strings against the RE ``[a-z]+``. An empty string shouldn't match at all, since ``+`` means 'one or more repetitions'. @@ -395,7 +397,7 @@ result in a variable for later use. :: >>> m = p.match('tempo') >>> print m - <_sre.SRE_Match object at 80c4f68> + <_sre.SRE_Match object at 0x...> Now you can query the :class:`MatchObject` for information about the matching string. :class:`MatchObject` instances also have several methods and @@ -434,7 +436,7 @@ case. :: >>> print p.match('::: message') None >>> m = p.search('::: message') ; print m - <re.MatchObject instance at 80c9650> + <_sre.SRE_Match object at 0x...> >>> m.group() 'message' >>> m.span() @@ -485,7 +487,7 @@ the RE string added as the first argument, and still return either ``None`` or a >>> print re.match(r'From\s+', 'Fromage amk') None >>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998') - <re.MatchObject instance at 80c5978> + <_sre.SRE_Match object at 0x...> Under the hood, these functions simply create a pattern object for you and call the appropriate method on it. They also store the compiled object in a @@ -686,7 +688,7 @@ given location, they can obviously be matched an infinite number of times. line, the RE to use is ``^From``. :: >>> print re.search('^From', 'From Here to Eternity') - <re.MatchObject instance at 80c1520> + <_sre.SRE_Match object at 0x...> >>> print re.search('^From', 'Reciting From Memory') None @@ -698,11 +700,11 @@ given location, they can obviously be matched an infinite number of times. or any location followed by a newline character. :: >>> print re.search('}$', '{block}') - <re.MatchObject instance at 80adfa8> + <_sre.SRE_Match object at 0x...> >>> print re.search('}$', '{block} ') None >>> print re.search('}$', '{block}\n') - <re.MatchObject instance at 80adfa8> + <_sre.SRE_Match object at 0x...> To match a literal ``'$'``, use ``\$`` or enclose it inside a character class, as in ``[$]``. @@ -727,7 +729,7 @@ given location, they can obviously be matched an infinite number of times. >>> p = re.compile(r'\bclass\b') >>> print p.search('no class at all') - <re.MatchObject instance at 80c8f28> + <_sre.SRE_Match object at 0x...> >>> print p.search('the declassified algorithm') None >>> print p.search('one subclass is') @@ -745,7 +747,7 @@ given location, they can obviously be matched an infinite number of times. >>> print p.search('no class at all') None >>> print p.search('\b' + 'class' + '\b') - <re.MatchObject instance at 80c3ee0> + <_sre.SRE_Match object at 0x...> Second, inside a character class, where there's no use for this assertion, ``\b`` represents the backspace character, for compatibility with Python's @@ -1315,7 +1317,7 @@ a regular expression that handles all of the possible cases, the patterns will be *very* complicated. Use an HTML or XML parser module for such tasks.) -Not Using re.VERBOSE +Using re.VERBOSE -------------------- By now you've probably noticed that regular expressions are a very compact diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index 1928c2a..4002a3d 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -314,7 +314,7 @@ process is likely to be screwed up. Non-blocking Sockets ==================== -If you've understood the preceeding, you already know most of what you need to +If you've understood the preceding, you already know most of what you need to know about the mechanics of using sockets. You'll still use the same calls, in much the same ways. It's just that, if you do it right, your app will be almost inside-out. diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index 79008a7..746e40c 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -1,3 +1,5 @@ +.. _sortinghowto: + Sorting HOW TO ************** @@ -6,8 +8,8 @@ Sorting HOW TO Python lists have a built-in :meth:`list.sort` method that modifies the list -in-place and a :func:`sorted` built-in function that builds a new sorted list -from an iterable. +in-place. There is also a :func:`sorted` built-in function that builds a new +sorted list from an iterable. In this document, we explore the various techniques for sorting data using Python. diff --git a/Doc/howto/unicode.rst b/Doc/howto/unicode.rst index 4e4921c..ff3c721 100644 --- a/Doc/howto/unicode.rst +++ b/Doc/howto/unicode.rst @@ -2,10 +2,12 @@ Unicode HOWTO ***************** -:Release: 1.02 +:Release: 1.03 -This HOWTO discusses Python's support for Unicode, and explains various problems -that people commonly encounter when trying to work with Unicode. +This HOWTO discusses Python 2.x's support for Unicode, and explains +various problems that people commonly encounter when trying to work +with Unicode. (This HOWTO has not yet been updated to cover the 3.x +versions of Python.) Introduction to Unicode ======================= @@ -144,8 +146,9 @@ problems. 4. Many Internet standards are defined in terms of textual data, and can't handle content with embedded zero bytes. -Generally people don't use this encoding, instead choosing other encodings that -are more efficient and convenient. +Generally people don't use this encoding, instead choosing other +encodings that are more efficient and convenient. UTF-8 is probably +the most commonly supported encoding; it will be discussed below. Encodings don't have to handle every possible Unicode character, and most encodings don't. For example, Python's default encoding is the 'ascii' @@ -222,8 +225,8 @@ Wikipedia entries are often helpful; see the entries for "character encoding" <http://en.wikipedia.org/wiki/UTF-8>, for example. -Python's Unicode Support -======================== +Python 2.x's Unicode Support +============================ Now that you've learned the rudiments of Unicode, we can look at Python's Unicode features. @@ -272,7 +275,7 @@ Unicode result). The following examples show the differences:: >>> unicode('\x80abc', errors='ignore') u'abc' -Encodings are specified as strings containing the encoding's name. Python 2.4 +Encodings are specified as strings containing the encoding's name. Python 2.7 comes with roughly 100 different encodings; see the Python Library Reference at :ref:`standard-encodings` for a list. Some encodings have multiple names; for example, 'latin-1', 'iso_8859_1' and '8859' are all @@ -427,11 +430,19 @@ encoding declaration:: When you run it with Python 2.4, it will output the following warning:: - amk:~$ python p263.py + amk:~$ python2.4 p263.py sys:1: DeprecationWarning: Non-ASCII character '\xe9' in file p263.py on line 2, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details +Python 2.5 and higher are stricter and will produce a syntax error:: + + amk:~$ python2.5 p263.py + File "/tmp/p263.py", line 2 + SyntaxError: Non-ASCII character '\xc3' in file /tmp/p263.py + on line 2, but no encoding declared; see + http://www.python.org/peps/pep-0263.html for details + Unicode Properties ------------------ @@ -472,7 +483,7 @@ These are grouped into categories such as "Letter", "Number", "Punctuation", or from the above output, ``'Ll'`` means 'Letter, lowercase', ``'No'`` means "Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol, other". See -<http://unicode.org/Public/5.1.0/ucd/UCD.html#General_Category_Values> for a +<http://www.unicode.org/reports/tr44/#General_Category_Values> for a list of category codes. References @@ -693,7 +704,11 @@ several links. Version 1.02: posted August 16 2005. Corrects factual errors. +Version 1.03: posted June 20 2010. Notes that Python 3.x is not covered, +and that the HOWTO only covers 2.x. + +.. comment Describe Python 3.x support (new section? new document?) .. comment Additional topic: building Python w/ UCS2 or UCS4 support .. comment Describe obscure -U switch somewhere? .. comment Describe use of codecs.StreamRecoder and StreamReaderWriter diff --git a/Doc/howto/webservers.rst b/Doc/howto/webservers.rst index 5a7b16a..03853f4 100644 --- a/Doc/howto/webservers.rst +++ b/Doc/howto/webservers.rst @@ -530,7 +530,7 @@ Popular template engines include: .. seealso:: - There are many template engines competing for attention, becuase it is + There are many template engines competing for attention, because it is pretty easy to create them in Python. The page `Templating <http://wiki.python.org/moin/Templating>`_ in the wiki lists a big, ever-growing number of these. The three listed above are considered "second diff --git a/Doc/includes/email-alternative.py b/Doc/includes/email-alternative.py index 82e3ffa..82d605e 100644 --- a/Doc/includes/email-alternative.py +++ b/Doc/includes/email-alternative.py @@ -1,4 +1,4 @@ -#! /usr/bin/python +#!/usr/bin/env python import smtplib diff --git a/Doc/includes/email-headers.py b/Doc/includes/email-headers.py new file mode 100644 index 0000000..664c3ff --- /dev/null +++ b/Doc/includes/email-headers.py @@ -0,0 +1,17 @@ +# Import the email modules we'll need +from email.parser import Parser + +# If the e-mail headers are in a file, uncomment this line: +#headers = Parser().parse(open(messagefile, 'r')) + +# Or for parsing headers in a string, use: +headers = Parser().parsestr('From: <user@example.com>\n' + 'To: <someone_else@example.com>\n' + 'Subject: Test message\n' + '\n' + 'Body would go here\n') + +# Now the header items can be accessed as a dictionary: +print 'To: %s' % headers['to'] +print 'From: %s' % headers['from'] +print 'Subject: %s' % headers['subject'] diff --git a/Doc/includes/sqlite3/load_extension.py b/Doc/includes/sqlite3/load_extension.py new file mode 100644 index 0000000..7f893c9 --- /dev/null +++ b/Doc/includes/sqlite3/load_extension.py @@ -0,0 +1,26 @@ +import sqlite3 + +con = sqlite3.connect(":memory:") + +# enable extension loading +con.enable_load_extension(True) + +# Load the fulltext search extension +con.execute("select load_extension('./fts3.so')") + +# alternatively you can load the extension using an API call: +# con.load_extension("./fts3.so") + +# disable extension laoding again +con.enable_load_extension(False) + +# example from SQLite wiki +con.execute("create virtual table recipe using fts3(name, ingredients)") +con.executescript(""" + insert into recipe (name, ingredients) values ('broccoli stew', 'broccoli peppers cheese tomatoes'); + insert into recipe (name, ingredients) values ('pumpkin stew', 'pumpkin onions garlic celery'); + insert into recipe (name, ingredients) values ('broccoli pie', 'broccoli cheese onions flour'); + insert into recipe (name, ingredients) values ('pumpkin pie', 'pumpkin sugar flour butter'); + """) +for row in con.execute("select rowid, name, ingredients from recipe where name match 'pie'"): + print row diff --git a/Doc/includes/sqlite3/shared_cache.py b/Doc/includes/sqlite3/shared_cache.py index bf1d7b4..30e71c9 100644 --- a/Doc/includes/sqlite3/shared_cache.py +++ b/Doc/includes/sqlite3/shared_cache.py @@ -1,6 +1,6 @@ import sqlite3 # The shared cache is only available in SQLite versions 3.3.3 or later -# See the SQLite documentaton for details. +# See the SQLite documentation for details. sqlite3.enable_shared_cache(True) diff --git a/Doc/includes/tzinfo-examples.py b/Doc/includes/tzinfo-examples.py index 2de95d4..5132429 100644 --- a/Doc/includes/tzinfo-examples.py +++ b/Doc/includes/tzinfo-examples.py @@ -71,7 +71,7 @@ class LocalTimezone(tzinfo): def _isdst(self, dt): tt = (dt.year, dt.month, dt.day, dt.hour, dt.minute, dt.second, - dt.weekday(), 0, -1) + dt.weekday(), 0, 0) stamp = _time.mktime(tt) tt = _time.localtime(stamp) return tt.tm_isdst > 0 diff --git a/Doc/install/index.rst b/Doc/install/index.rst index a5d2671..af113ee 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -694,6 +694,9 @@ And on Windows, the configuration files are: | local | :file:`setup.cfg` | \(3) | +--------------+-------------------------------------------------+-------+ +On all platforms, the "personal" file can be temporarily disabled by +passing the `--no-user-cfg` option. + Notes: (1) @@ -929,15 +932,34 @@ section :ref:`inst-config-files`.) GNU C / Cygwin / MinGW ^^^^^^^^^^^^^^^^^^^^^^ -These instructions only apply if you're using a version of Python prior to -2.4.1 with a MinGW prior to 3.0.0 (with binutils-2.13.90-20030111-1). - This section describes the necessary steps to use Distutils with the GNU C/C++ compilers in their Cygwin and MinGW distributions. [#]_ For a Python interpreter that was built with Cygwin, everything should work without any of these following steps. -These compilers require some special libraries. This task is more complex than +Not all extensions can be built with MinGW or Cygwin, but many can. Extensions +most likely to not work are those that use C++ or depend on Microsoft Visual C +extensions. + +To let Distutils compile your extension with Cygwin you have to type:: + + python setup.py build --compiler=cygwin + +and for Cygwin in no-cygwin mode [#]_ or for MinGW type:: + + python setup.py build --compiler=mingw32 + +If you want to use any of these options/compilers as default, you should +consider writing it in your personal or system-wide configuration file for +Distutils (see section :ref:`inst-config-files`.) + +Older Versions of Python and MinGW +"""""""""""""""""""""""""""""""""" +The following instructions only apply if you're using a version of Python +inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with +binutils-2.13.90-20030111-1). + +These compilers require some special libraries. This task is more complex than for Borland's C++, because there is no program to convert the library. First you have to create a list of symbols which the Python DLL exports. (You can find a good program for this task at @@ -967,18 +989,6 @@ If your extension uses other libraries (zlib,...) you might have to convert them too. The converted files have to reside in the same directories as the normal libraries do. -To let Distutils compile your extension with Cygwin you now have to type :: - - python setup.py build --compiler=cygwin - -and for Cygwin in no-cygwin mode [#]_ or for MinGW type:: - - python setup.py build --compiler=mingw32 - -If you want to use any of these options/compilers as default, you should -consider to write it in your personal or system-wide configuration file for -Distutils (see section :ref:`inst-config-files`.) - .. seealso:: diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst index f3be8fa..de31251 100644 --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -89,7 +89,7 @@ process. Since some print statements can be parsed as function calls or statements, 2to3 cannot always read files containing the print function. When 2to3 detects the presence of the ``from __future__ import print_function`` compiler directive, it -modifies its internal grammar to interpert :func:`print` as a function. This +modifies its internal grammar to interpret :func:`print` as a function. This change can also be enabled manually with the :option:`-p` flag. Use :option:`-p` to run fixers on code that already has had its print statements converted. diff --git a/Doc/library/_winreg.rst b/Doc/library/_winreg.rst index 3627130..825ce1f 100644 --- a/Doc/library/_winreg.rst +++ b/Doc/library/_winreg.rst @@ -65,6 +65,34 @@ This module offers the following functions: :exc:`WindowsError` exception is raised. +.. function:: CreateKeyEx(key, sub_key[, res[, sam]]) + + Creates or opens the specified key, returning a + :ref:`handle object <handle-object>`. + + *key* is an already open key, or one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that names the key this method opens or creates. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_ALL_ACCESS`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + + If *key* is one of the predefined keys, *sub_key* may be ``None``. In that + case, the handle returned is the same key handle passed in to the function. + + If the key already exists, this function opens the existing key. + + The return value is the handle of the opened key. If the function fails, a + :exc:`WindowsError` exception is raised. + +.. versionadded:: 2.7 + + .. function:: DeleteKey(key, sub_key) Deletes the specified key. @@ -72,8 +100,8 @@ This module offers the following functions: *key* is an already open key, or any one of the predefined :ref:`HKEY_* constants <hkey-constants>`. - *sub_key* is a string that must be a subkey of the key identified by the *key* - parameter. This value must not be ``None``, and the key may not have subkeys. + *sub_key* is a string that must be a subkey of the key identified by the *key* + parameter. This value must not be ``None``, and the key may not have subkeys. *This method can not delete keys with subkeys.* @@ -81,6 +109,40 @@ This module offers the following functions: If the method fails, a :exc:`WindowsError` exception is raised. +.. function:: DeleteKeyEx(key, sub_key[, sam[, res]]) + + Deletes the specified key. + + .. note:: + The :func:`DeleteKeyEx` function is implemented with the RegDeleteKeyEx + Windows API function, which is specific to 64-bit versions of Windows. + See the `RegDeleteKeyEx documentation + <http://msdn.microsoft.com/en-us/library/ms724847%28VS.85%29.aspx>`__. + + *key* is an already open key, or any one of the predefined + :ref:`HKEY_* constants <hkey-constants>`. + + *sub_key* is a string that must be a subkey of the key identified by the + *key* parameter. This value must not be ``None``, and the key may not have + subkeys. + + *res* is a reserved integer, and must be zero. The default is zero. + + *sam* is an integer that specifies an access mask that describes the desired + security access for the key. Default is :const:`KEY_WOW64_64KEY`. See + :ref:`Access Rights <access-rights>` for other allowed values. + + + *This method can not delete keys with subkeys.* + + If the method succeeds, the entire key, including all of its values, is + removed. If the method fails, a :exc:`WindowsError` exception is raised. + + On unsupported Windows versions, :exc:`NotImplementedError` is raised. + +.. versionadded:: 2.7 + + .. function:: DeleteValue(key, value) Removes a named value from a registry key. @@ -183,7 +245,7 @@ This module offers the following functions: A call to :func:`LoadKey` fails if the calling process does not have the :const:`SE_RESTORE_PRIVILEGE` privilege. Note that privileges are different from permissions -- see the `RegLoadKey documentation - <http://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx>`_ for + <http://msdn.microsoft.com/en-us/library/ms724889%28v=VS.85%29.aspx>`__ for more details. If *key* is a handle returned by :func:`ConnectRegistry`, then the path diff --git a/Doc/library/allos.rst b/Doc/library/allos.rst index 6c5837d..b135485 100644 --- a/Doc/library/allos.rst +++ b/Doc/library/allos.rst @@ -16,6 +16,7 @@ but they are available on most other systems as well. Here's an overview: os.rst io.rst time.rst + argparse.rst optparse.rst getopt.rst logging.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst new file mode 100644 index 0000000..b34be8a --- /dev/null +++ b/Doc/library/argparse.rst @@ -0,0 +1,1791 @@ +:mod:`argparse` --- Parser for command line options, arguments and sub-commands +=============================================================================== + +.. module:: argparse + :synopsis: Command-line option and argument parsing library. +.. moduleauthor:: Steven Bethard <steven.bethard@gmail.com> +.. versionadded:: 2.7 +.. sectionauthor:: Steven Bethard <steven.bethard@gmail.com> + + +The :mod:`argparse` module makes it easy to write user friendly command line +interfaces. The program defines what arguments it requires, and :mod:`argparse` +will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` +module also automatically generates help and usage messages and issues errors +when users give the program invalid arguments. + + +Example +------- + +The following code is a Python program that takes a list of integers and +produces either the sum or the max:: + + import argparse + + parser = argparse.ArgumentParser(description='Process some integers.') + parser.add_argument('integers', metavar='N', type=int, nargs='+', + help='an integer for the accumulator') + parser.add_argument('--sum', dest='accumulate', action='store_const', + const=sum, default=max, + help='sum the integers (default: find the max)') + + args = parser.parse_args() + print args.accumulate(args.integers) + +Assuming the Python code above is saved into a file called ``prog.py``, it can +be run at the command line and provides useful help messages:: + + $ prog.py -h + usage: prog.py [-h] [--sum] N [N ...] + + Process some integers. + + positional arguments: + N an integer for the accumulator + + optional arguments: + -h, --help show this help message and exit + --sum sum the integers (default: find the max) + +When run with the appropriate arguments, it prints either the sum or the max of +the command-line integers:: + + $ prog.py 1 2 3 4 + 4 + + $ prog.py 1 2 3 4 --sum + 10 + +If invalid arguments are passed in, it will issue an error:: + + $ prog.py a b c + usage: prog.py [-h] [--sum] N [N ...] + prog.py: error: argument N: invalid int value: 'a' + +The following sections walk you through this example. + + +Creating a parser +^^^^^^^^^^^^^^^^^ + +The first step in using the :mod:`argparse` is creating an +:class:`ArgumentParser` object:: + + >>> parser = argparse.ArgumentParser(description='Process some integers.') + +The :class:`ArgumentParser` object will hold all the information necessary to +parse the command line into python data types. + + +Adding arguments +^^^^^^^^^^^^^^^^ + +Filling an :class:`ArgumentParser` with information about program arguments is +done by making calls to the :meth:`~ArgumentParser.add_argument` method. +Generally, these calls tell the :class:`ArgumentParser` how to take the strings +on the command line and turn them into objects. This information is stored and +used when :meth:`~ArgumentParser.parse_args` is called. For example:: + + >>> parser.add_argument('integers', metavar='N', type=int, nargs='+', + ... help='an integer for the accumulator') + >>> parser.add_argument('--sum', dest='accumulate', action='store_const', + ... const=sum, default=max, + ... help='sum the integers (default: find the max)') + +Later, calling :meth:`parse_args` will return an object with +two attributes, ``integers`` and ``accumulate``. The ``integers`` attribute +will be a list of one or more ints, and the ``accumulate`` attribute will be +either the :func:`sum` function, if ``--sum`` was specified at the command line, +or the :func:`max` function if it was not. + + +Parsing arguments +^^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` parses args through the +:meth:`~ArgumentParser.parse_args` method. This will inspect the command-line, +convert each arg to the appropriate type and then invoke the appropriate action. +In most cases, this means a simple namespace object will be built up from +attributes parsed out of the command-line:: + + >>> parser.parse_args(['--sum', '7', '-1', '42']) + Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42]) + +In a script, :meth:`~ArgumentParser.parse_args` will typically be called with no +arguments, and the :class:`ArgumentParser` will automatically determine the +command-line args from :data:`sys.argv`. + + +ArgumentParser objects +---------------------- + +.. class:: ArgumentParser([description], [epilog], [prog], [usage], [add_help], \ + [argument_default], [parents], [prefix_chars], \ + [conflict_handler], [formatter_class]) + + Create a new :class:`ArgumentParser` object. Each parameter has its own more + detailed description below, but in short they are: + + * description_ - Text to display before the argument help. + + * epilog_ - Text to display after the argument help. + + * add_help_ - Add a -h/--help option to the parser. (default: ``True``) + + * argument_default_ - Set the global default value for arguments. + (default: ``None``) + + * parents_ - A list of :class:`ArgumentParser` objects whose arguments should + also be included. + + * prefix_chars_ - The set of characters that prefix optional arguments. + (default: '-') + + * fromfile_prefix_chars_ - The set of characters that prefix files from + which additional arguments should be read. (default: ``None``) + + * formatter_class_ - A class for customizing the help output. + + * conflict_handler_ - Usually unnecessary, defines strategy for resolving + conflicting optionals. + + * prog_ - The name of the program (default: + :data:`sys.argv[0]`) + + * usage_ - The string describing the program usage (default: generated) + +The following sections describe how each of these are used. + + +description +^^^^^^^^^^^ + +Most calls to the :class:`ArgumentParser` constructor will use the +``description=`` keyword argument. This argument gives a brief description of +what the program does and how it works. In help messages, the description is +displayed between the command-line usage string and the help messages for the +various arguments:: + + >>> parser = argparse.ArgumentParser(description='A foo that bars') + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + +By default, the description will be line-wrapped so that it fits within the +given space. To change this behavior, see the formatter_class_ argument. + + +epilog +^^^^^^ + +Some programs like to display additional description of the program after the +description of the arguments. Such text can be specified using the ``epilog=`` +argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser( + ... description='A foo that bars', + ... epilog="And that's how you'd foo a bar") + >>> parser.print_help() + usage: argparse.py [-h] + + A foo that bars + + optional arguments: + -h, --help show this help message and exit + + And that's how you'd foo a bar + +As with the description_ argument, the ``epilog=`` text is by default +line-wrapped, but this behavior can be adjusted with the formatter_class_ +argument to :class:`ArgumentParser`. + + +add_help +^^^^^^^^ + +By default, ArgumentParser objects add an option which simply displays +the parser's help message. For example, consider a file named +``myprogram.py`` containing the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +If ``-h`` or ``--help`` is supplied is at the command-line, the ArgumentParser +help will be printed:: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +Occasionally, it may be useful to disable the addition of this help option. +This can be achieved by passing ``False`` as the ``add_help=`` argument to +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> parser.add_argument('--foo', help='foo help') + >>> parser.print_help() + usage: PROG [--foo FOO] + + optional arguments: + --foo FOO foo help + +The help option is typically ``-h/--help``. The exception to this is +if the ``prefix_chars=`` is specified and does not include ``'-'``, in +which case ``-h`` and ``--help`` are not valid options. In +this case, the first character in ``prefix_chars`` is used to prefix +the help options:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='+/') + >>> parser.print_help() + usage: PROG [+h] + + optional arguments: + +h, ++help show this help message and exit + + +prefix_chars +^^^^^^^^^^^^ + +Most command-line options will use ``'-'`` as the prefix, e.g. ``-f/--foo``. +Parsers that need to support different or additional prefix +characters, e.g. for options +like ``+f`` or ``/foo``, may specify them using the ``prefix_chars=`` argument +to the ArgumentParser constructor:: + + >>> parser = argparse.ArgumentParser(prog='PROG', prefix_chars='-+') + >>> parser.add_argument('+f') + >>> parser.add_argument('++bar') + >>> parser.parse_args('+f X ++bar Y'.split()) + Namespace(bar='Y', f='X') + +The ``prefix_chars=`` argument defaults to ``'-'``. Supplying a set of +characters that does not include ``'-'`` will cause ``-f/--foo`` options to be +disallowed. + + +fromfile_prefix_chars +^^^^^^^^^^^^^^^^^^^^^ + +Sometimes, for example when dealing with a particularly long argument lists, it +may make sense to keep the list of arguments in a file rather than typing it out +at the command line. If the ``fromfile_prefix_chars=`` argument is given to the +:class:`ArgumentParser` constructor, then arguments that start with any of the +specified characters will be treated as files, and will be replaced by the +arguments they contain. For example:: + + >>> with open('args.txt', 'w') as fp: + ... fp.write('-f\nbar') + >>> parser = argparse.ArgumentParser(fromfile_prefix_chars='@') + >>> parser.add_argument('-f') + >>> parser.parse_args(['-f', 'foo', '@args.txt']) + Namespace(f='bar') + +Arguments read from a file must by default be one per line (but see also +:meth:`convert_arg_line_to_args`) and are treated as if they were in the same +place as the original file referencing argument on the command line. So in the +example above, the expression ``['-f', 'foo', '@args.txt']`` is considered +equivalent to the expression ``['-f', 'foo', '-f', 'bar']``. + +The ``fromfile_prefix_chars=`` argument defaults to ``None``, meaning that +arguments will never be treated as file references. + + +argument_default +^^^^^^^^^^^^^^^^ + +Generally, argument defaults are specified either by passing a default to +:meth:`add_argument` or by calling the :meth:`set_defaults` methods with a +specific set of name-value pairs. Sometimes however, it may be useful to +specify a single parser-wide default for arguments. This can be accomplished by +passing the ``argument_default=`` keyword argument to :class:`ArgumentParser`. +For example, to globally suppress attribute creation on :meth:`parse_args` +calls, we supply ``argument_default=SUPPRESS``:: + + >>> parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS) + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar', nargs='?') + >>> parser.parse_args(['--foo', '1', 'BAR']) + Namespace(bar='BAR', foo='1') + >>> parser.parse_args([]) + Namespace() + + +parents +^^^^^^^ + +Sometimes, several parsers share a common set of arguments. Rather than +repeating the definitions of these arguments, a single parser with all the +shared arguments and passed to ``parents=`` argument to :class:`ArgumentParser` +can be used. The ``parents=`` argument takes a list of :class:`ArgumentParser` +objects, collects all the positional and optional actions from them, and adds +these actions to the :class:`ArgumentParser` object being constructed:: + + >>> parent_parser = argparse.ArgumentParser(add_help=False) + >>> parent_parser.add_argument('--parent', type=int) + + >>> foo_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> foo_parser.add_argument('foo') + >>> foo_parser.parse_args(['--parent', '2', 'XXX']) + Namespace(foo='XXX', parent=2) + + >>> bar_parser = argparse.ArgumentParser(parents=[parent_parser]) + >>> bar_parser.add_argument('--bar') + >>> bar_parser.parse_args(['--bar', 'YYY']) + Namespace(bar='YYY', parent=None) + +Note that most parent parsers will specify ``add_help=False``. Otherwise, the +:class:`ArgumentParser` will see two ``-h/--help`` options (one in the parent +and one in the child) and raise an error. + +.. note:: + You must fully initialize the parsers before passing them via ``parents=``. + If you change the parent parsers after the child parser, those changes will + not be reflected in the child. + + +formatter_class +^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects allow the help formatting to be customized by +specifying an alternate formatting class. Currently, there are three such +classes: :class:`argparse.RawDescriptionHelpFormatter`, +:class:`argparse.RawTextHelpFormatter` and +:class:`argparse.ArgumentDefaultsHelpFormatter`. The first two allow more +control over how textual descriptions are displayed, while the last +automatically adds information about argument default values. + +By default, :class:`ArgumentParser` objects line-wrap the description_ and +epilog_ texts in command-line help messages:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... description='''this description + ... was indented weird + ... but that is okay''', + ... epilog=''' + ... likewise for this epilog whose whitespace will + ... be cleaned up and whose words will be wrapped + ... across a couple lines''') + >>> parser.print_help() + usage: PROG [-h] + + this description was indented weird but that is okay + + optional arguments: + -h, --help show this help message and exit + + likewise for this epilog whose whitespace will be cleaned up and whose words + will be wrapped across a couple lines + +Passing :class:`argparse.RawDescriptionHelpFormatter` as ``formatter_class=`` +indicates that description_ and epilog_ are already correctly formatted and +should not be line-wrapped:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.RawDescriptionHelpFormatter, + ... description=textwrap.dedent('''\ + ... Please do not mess up this text! + ... -------------------------------- + ... I have indented it + ... exactly the way + ... I want it + ... ''')) + >>> parser.print_help() + usage: PROG [-h] + + Please do not mess up this text! + -------------------------------- + I have indented it + exactly the way + I want it + + optional arguments: + -h, --help show this help message and exit + +:class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text +including argument descriptions. + +The other formatter class available, :class:`ArgumentDefaultsHelpFormatter`, +will add information about the default value of each of the arguments:: + + >>> parser = argparse.ArgumentParser( + ... prog='PROG', + ... formatter_class=argparse.ArgumentDefaultsHelpFormatter) + >>> parser.add_argument('--foo', type=int, default=42, help='FOO!') + >>> parser.add_argument('bar', nargs='*', default=[1, 2, 3], help='BAR!') + >>> parser.print_help() + usage: PROG [-h] [--foo FOO] [bar [bar ...]] + + positional arguments: + bar BAR! (default: [1, 2, 3]) + + optional arguments: + -h, --help show this help message and exit + --foo FOO FOO! (default: 42) + + +conflict_handler +^^^^^^^^^^^^^^^^ + +:class:`ArgumentParser` objects do not allow two actions with the same option +string. By default, :class:`ArgumentParser` objects raises an exception if an +attempt is made to create an argument with an option string that is already in +use:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + Traceback (most recent call last): + .. + ArgumentError: argument --foo: conflicting option string(s): --foo + +Sometimes (e.g. when using parents_) it may be useful to simply override any +older arguments with the same option string. To get this behavior, the value +``'resolve'`` can be supplied to the ``conflict_handler=`` argument of +:class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='PROG', conflict_handler='resolve') + >>> parser.add_argument('-f', '--foo', help='old foo help') + >>> parser.add_argument('--foo', help='new foo help') + >>> parser.print_help() + usage: PROG [-h] [-f FOO] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + -f FOO old foo help + --foo FOO new foo help + +Note that :class:`ArgumentParser` objects only remove an action if all of its +option strings are overridden. So, in the example above, the old ``-f/--foo`` +action is retained as the ``-f`` action, because only the ``--foo`` option +string was overridden. + + +prog +^^^^ + +By default, :class:`ArgumentParser` objects uses ``sys.argv[0]`` to determine +how to display the name of the program in help messages. This default is almost +always desirable because it will make the help messages match how the program was +invoked on the command line. For example, consider a file named +``myprogram.py`` with the following code:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--foo', help='foo help') + args = parser.parse_args() + +The help for this program will display ``myprogram.py`` as the program name +(regardless of where the program was invoked from):: + + $ python myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + $ cd .. + $ python subdir\myprogram.py --help + usage: myprogram.py [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo help + +To change this default behavior, another value can be supplied using the +``prog=`` argument to :class:`ArgumentParser`:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.print_help() + usage: myprogram [-h] + + optional arguments: + -h, --help show this help message and exit + +Note that the program name, whether determined from ``sys.argv[0]`` or from the +``prog=`` argument, is available to help messages using the ``%(prog)s`` format +specifier. + +:: + + >>> parser = argparse.ArgumentParser(prog='myprogram') + >>> parser.add_argument('--foo', help='foo of the %(prog)s program') + >>> parser.print_help() + usage: myprogram [-h] [--foo FOO] + + optional arguments: + -h, --help show this help message and exit + --foo FOO foo of the myprogram program + + +usage +^^^^^ + +By default, :class:`ArgumentParser` calculates the usage message from the +arguments it contains:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [-h] [--foo [FOO]] bar [bar ...] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The default message can be overridden with the ``usage=`` keyword argument:: + + >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]') + >>> parser.add_argument('--foo', nargs='?', help='foo help') + >>> parser.add_argument('bar', nargs='+', help='bar help') + >>> parser.print_help() + usage: PROG [options] + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + --foo [FOO] foo help + +The ``%(prog)s`` format specifier is available to fill in the program name in +your usage messages. + + +The add_argument() method +------------------------- + +.. method:: ArgumentParser.add_argument(name or flags..., [action], [nargs], \ + [const], [default], [type], [choices], [required], \ + [help], [metavar], [dest]) + + Define how a single command line argument should be parsed. Each parameter + has its own more detailed description below, but in short they are: + + * `name or flags`_ - Either a name or a list of option strings, e.g. ``foo`` + or ``-f, --foo`` + + * action_ - The basic type of action to be taken when this argument is + encountered at the command-line. + + * nargs_ - The number of command-line arguments that should be consumed. + + * const_ - A constant value required by some action_ and nargs_ selections. + + * default_ - The value produced if the argument is absent from the + command-line. + + * type_ - The type to which the command-line arg should be converted. + + * choices_ - A container of the allowable values for the argument. + + * required_ - Whether or not the command-line option may be omitted + (optionals only). + + * help_ - A brief description of what the argument does. + + * metavar_ - A name for the argument in usage messages. + + * dest_ - The name of the attribute to be added to the object returned by + :meth:`parse_args`. + +The following sections describe how each of these are used. + + +name or flags +^^^^^^^^^^^^^ + +The :meth:`add_argument` method must know whether an optional argument, like +``-f`` or ``--foo``, or a positional argument, like a list of filenames, is +expected. The first arguments passed to :meth:`add_argument` must therefore be +either a series of flags, or a simple argument name. For example, an optional +argument could be created like:: + + >>> parser.add_argument('-f', '--foo') + +while a positional argument could be created like:: + + >>> parser.add_argument('bar') + +When :meth:`parse_args` is called, optional arguments will be identified by the +``-`` prefix, and the remaining arguments will be assumed to be positional:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-f', '--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args(['BAR']) + Namespace(bar='BAR', foo=None) + >>> parser.parse_args(['BAR', '--foo', 'FOO']) + Namespace(bar='BAR', foo='FOO') + >>> parser.parse_args(['--foo', 'FOO']) + usage: PROG [-h] [-f FOO] bar + PROG: error: too few arguments + + +action +^^^^^^ + +:class:`ArgumentParser` objects associate command-line args with actions. These +actions can do just about anything with the command-line args associated with +them, though most actions simply add an attribute to the object returned by +:meth:`parse_args`. The ``action`` keyword argument specifies how the +command-line args should be handled. The supported actions are: + +* ``'store'`` - This just stores the argument's value. This is the default + action. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args('--foo 1'.split()) + Namespace(foo='1') + +* ``'store_const'`` - This stores the value specified by the const_ keyword + argument. (Note that the const_ keyword argument defaults to the rather + unhelpful ``None``.) The ``'store_const'`` action is most commonly used with + optional arguments that specify some sort of flag. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_const', const=42) + >>> parser.parse_args('--foo'.split()) + Namespace(foo=42) + +* ``'store_true'`` and ``'store_false'`` - These store the values ``True`` and + ``False`` respectively. These are special cases of ``'store_const'``. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('--bar', action='store_false') + >>> parser.parse_args('--foo --bar'.split()) + Namespace(bar=False, foo=True) + +* ``'append'`` - This stores a list, and appends each argument value to the + list. This is useful to allow an option to be specified multiple times. + Example usage:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='append') + >>> parser.parse_args('--foo 1 --foo 2'.split()) + Namespace(foo=['1', '2']) + +* ``'append_const'`` - This stores a list, and appends the value specified by + the const_ keyword argument to the list. (Note that the const_ keyword + argument defaults to ``None``.) The ``'append_const'`` action is typically + useful when multiple arguments need to store constants to the same list. For + example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--str', dest='types', action='append_const', const=str) + >>> parser.add_argument('--int', dest='types', action='append_const', const=int) + >>> parser.parse_args('--str --int'.split()) + Namespace(types=[<type 'str'>, <type 'int'>]) + +* ``'version'`` - This expects a ``version=`` keyword argument in the + :meth:`add_argument` call, and prints version information and exits when + invoked. + + >>> import argparse + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--version', action='version', version='%(prog)s 2.0') + >>> parser.parse_args(['--version']) + PROG 2.0 + +You can also specify an arbitrary action by passing an object that implements +the Action API. The easiest way to do this is to extend +:class:`argparse.Action`, supplying an appropriate ``__call__`` method. The +``__call__`` method should accept four parameters: + +* ``parser`` - The ArgumentParser object which contains this action. + +* ``namespace`` - The namespace object that will be returned by + :meth:`parse_args`. Most actions add an attribute to this object. + +* ``values`` - The associated command-line args, with any type-conversions + applied. (Type-conversions are specified with the type_ keyword argument to + :meth:`add_argument`. + +* ``option_string`` - The option string that was used to invoke this action. + The ``option_string`` argument is optional, and will be absent if the action + is associated with a positional argument. + +An example of a custom action:: + + >>> class FooAction(argparse.Action): + ... def __call__(self, parser, namespace, values, option_string=None): + ... print '%r %r %r' % (namespace, values, option_string) + ... setattr(namespace, self.dest, values) + ... + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action=FooAction) + >>> parser.add_argument('bar', action=FooAction) + >>> args = parser.parse_args('1 --foo 2'.split()) + Namespace(bar=None, foo=None) '1' None + Namespace(bar='1', foo=None) '2' '--foo' + >>> args + Namespace(bar='1', foo='2') + + +nargs +^^^^^ + +ArgumentParser objects usually associate a single command-line argument with a +single action to be taken. The ``nargs`` keyword argument associates a +different number of command-line arguments with a single action.. The supported +values are: + +* N (an integer). N args from the command-line will be gathered together into a + list. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs=2) + >>> parser.add_argument('bar', nargs=1) + >>> parser.parse_args('c --foo a b'.split()) + Namespace(bar=['c'], foo=['a', 'b']) + + Note that ``nargs=1`` produces a list of one item. This is different from + the default, in which the item is produced by itself. + +* ``'?'``. One arg will be consumed from the command-line if possible, and + produced as a single item. If no command-line arg is present, the value from + default_ will be produced. Note that for optional arguments, there is an + additional case - the option string is present but not followed by a + command-line arg. In this case the value from const_ will be produced. Some + examples to illustrate this:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='?', const='c', default='d') + >>> parser.add_argument('bar', nargs='?', default='d') + >>> parser.parse_args('XX --foo YY'.split()) + Namespace(bar='XX', foo='YY') + >>> parser.parse_args('XX --foo'.split()) + Namespace(bar='XX', foo='c') + >>> parser.parse_args(''.split()) + Namespace(bar='d', foo='d') + + One of the more common uses of ``nargs='?'`` is to allow optional input and + output files:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), + ... default=sys.stdin) + >>> parser.add_argument('outfile', nargs='?', type=argparse.FileType('w'), + ... default=sys.stdout) + >>> parser.parse_args(['input.txt', 'output.txt']) + Namespace(infile=<open file 'input.txt', mode 'r' at 0x...>, + outfile=<open file 'output.txt', mode 'w' at 0x...>) + >>> parser.parse_args([]) + Namespace(infile=<open file '<stdin>', mode 'r' at 0x...>, + outfile=<open file '<stdout>', mode 'w' at 0x...>) + +* ``'*'``. All command-line args present are gathered into a list. Note that + it generally doesn't make much sense to have more than one positional argument + with ``nargs='*'``, but multiple optional arguments with ``nargs='*'`` is + possible. For example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', nargs='*') + >>> parser.add_argument('--bar', nargs='*') + >>> parser.add_argument('baz', nargs='*') + >>> parser.parse_args('a b --foo x y --bar 1 2'.split()) + Namespace(bar=['1', '2'], baz=['a', 'b'], foo=['x', 'y']) + +* ``'+'``. Just like ``'*'``, all command-line args present are gathered into a + list. Additionally, an error message will be generated if there wasn't at + least one command-line arg present. For example:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', nargs='+') + >>> parser.parse_args('a b'.split()) + Namespace(foo=['a', 'b']) + >>> parser.parse_args(''.split()) + usage: PROG [-h] foo [foo ...] + PROG: error: too few arguments + +If the ``nargs`` keyword argument is not provided, the number of args consumed +is determined by the action_. Generally this means a single command-line arg +will be consumed and a single item (not a list) will be produced. + + +const +^^^^^ + +The ``const`` argument of :meth:`add_argument` is used to hold constant values +that are not read from the command line but are required for the various +ArgumentParser actions. The two most common uses of it are: + +* When :meth:`add_argument` is called with ``action='store_const'`` or + ``action='append_const'``. These actions add the ``const`` value to one of + the attributes of the object returned by :meth:`parse_args`. See the action_ + description for examples. + +* When :meth:`add_argument` is called with option strings (like ``-f`` or + ``--foo``) and ``nargs='?'``. This creates an optional argument that can be + followed by zero or one command-line args. When parsing the command-line, if + the option string is encountered with no command-line arg following it, the + value of ``const`` will be assumed instead. See the nargs_ description for + examples. + +The ``const`` keyword argument defaults to ``None``. + + +default +^^^^^^^ + +All optional arguments and some positional arguments may be omitted at the +command-line. The ``default`` keyword argument of :meth:`add_argument`, whose +value defaults to ``None``, specifies what value should be used if the +command-line arg is not present. For optional arguments, the ``default`` value +is used when the option string was not present at the command line:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=42) + >>> parser.parse_args('--foo 2'.split()) + Namespace(foo='2') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + +For positional arguments with nargs_ ``='?'`` or ``'*'``, the ``default`` value +is used when no command-line arg was present:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', nargs='?', default=42) + >>> parser.parse_args('a'.split()) + Namespace(foo='a') + >>> parser.parse_args(''.split()) + Namespace(foo=42) + + +Providing ``default=argparse.SUPPRESS`` causes no attribute to be added if the +command-line argument was not present.:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default=argparse.SUPPRESS) + >>> parser.parse_args([]) + Namespace() + >>> parser.parse_args(['--foo', '1']) + Namespace(foo='1') + + +type +^^^^ + +By default, ArgumentParser objects read command-line args in as simple strings. +However, quite often the command-line string should instead be interpreted as +another type, like a :class:`float`, :class:`int` or :class:`file`. The +``type`` keyword argument of :meth:`add_argument` allows any necessary +type-checking and type-conversions to be performed. Many common built-in types +can be used directly as the value of the ``type`` argument:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.add_argument('bar', type=file) + >>> parser.parse_args('2 temp.txt'.split()) + Namespace(bar=<open file 'temp.txt', mode 'r' at 0x...>, foo=2) + +To ease the use of various types of files, the argparse module provides the +factory FileType which takes the ``mode=`` and ``bufsize=`` arguments of the +``file`` object. For example, ``FileType('w')`` can be used to create a +writable file:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar', type=argparse.FileType('w')) + >>> parser.parse_args(['out.txt']) + Namespace(bar=<open file 'out.txt', mode 'w' at 0x...>) + +``type=`` can take any callable that takes a single string argument and returns +the type-converted value:: + + >>> def perfect_square(string): + ... value = int(string) + ... sqrt = math.sqrt(value) + ... if sqrt != int(sqrt): + ... msg = "%r is not a perfect square" % string + ... raise argparse.ArgumentTypeError(msg) + ... return value + ... + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=perfect_square) + >>> parser.parse_args('9'.split()) + Namespace(foo=9) + >>> parser.parse_args('7'.split()) + usage: PROG [-h] foo + PROG: error: argument foo: '7' is not a perfect square + +The choices_ keyword argument may be more convenient for type checkers that +simply check against a range of values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=int, choices=xrange(5, 10)) + >>> parser.parse_args('7'.split()) + Namespace(foo=7) + >>> parser.parse_args('11'.split()) + usage: PROG [-h] {5,6,7,8,9} + PROG: error: argument foo: invalid choice: 11 (choose from 5, 6, 7, 8, 9) + +See the choices_ section for more details. + + +choices +^^^^^^^ + +Some command-line args should be selected from a restricted set of values. +These can be handled by passing a container object as the ``choices`` keyword +argument to :meth:`add_argument`. When the command-line is parsed, arg values +will be checked, and an error message will be displayed if the arg was not one +of the acceptable values:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', choices='abc') + >>> parser.parse_args('c'.split()) + Namespace(foo='c') + >>> parser.parse_args('X'.split()) + usage: PROG [-h] {a,b,c} + PROG: error: argument foo: invalid choice: 'X' (choose from 'a', 'b', 'c') + +Note that inclusion in the ``choices`` container is checked after any type_ +conversions have been performed, so the type of the objects in the ``choices`` +container should match the type_ specified:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('foo', type=complex, choices=[1, 1j]) + >>> parser.parse_args('1j'.split()) + Namespace(foo=1j) + >>> parser.parse_args('-- -4'.split()) + usage: PROG [-h] {1,1j} + PROG: error: argument foo: invalid choice: (-4+0j) (choose from 1, 1j) + +Any object that supports the ``in`` operator can be passed as the ``choices`` +value, so :class:`dict` objects, :class:`set` objects, custom containers, +etc. are all supported. + + +required +^^^^^^^^ + +In general, the argparse module assumes that flags like ``-f`` and ``--bar`` +indicate *optional* arguments, which can always be omitted at the command-line. +To make an option *required*, ``True`` can be specified for the ``required=`` +keyword argument to :meth:`add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', required=True) + >>> parser.parse_args(['--foo', 'BAR']) + Namespace(foo='BAR') + >>> parser.parse_args([]) + usage: argparse.py [-h] [--foo FOO] + argparse.py: error: option --foo is required + +As the example shows, if an option is marked as ``required``, :meth:`parse_args` +will report an error if that option is not present at the command line. + +.. note:: + + Required options are generally considered bad form because users expect + *options* to be *optional*, and thus they should be avoided when possible. + + +help +^^^^ + +The ``help`` value is a string containing a brief description of the argument. +When a user requests help (usually by using ``-h`` or ``--help`` at the +command-line), these ``help`` descriptions will be displayed with each +argument:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('--foo', action='store_true', + ... help='foo the bars before frobbling') + >>> parser.add_argument('bar', nargs='+', + ... help='one of the bars to be frobbled') + >>> parser.parse_args('-h'.split()) + usage: frobble [-h] [--foo] bar [bar ...] + + positional arguments: + bar one of the bars to be frobbled + + optional arguments: + -h, --help show this help message and exit + --foo foo the bars before frobbling + +The ``help`` strings can include various format specifiers to avoid repetition +of things like the program name or the argument default_. The available +specifiers include the program name, ``%(prog)s`` and most keyword arguments to +:meth:`add_argument`, e.g. ``%(default)s``, ``%(type)s``, etc.:: + + >>> parser = argparse.ArgumentParser(prog='frobble') + >>> parser.add_argument('bar', nargs='?', type=int, default=42, + ... help='the bar to %(prog)s (default: %(default)s)') + >>> parser.print_help() + usage: frobble [-h] [bar] + + positional arguments: + bar the bar to frobble (default: 42) + + optional arguments: + -h, --help show this help message and exit + + +metavar +^^^^^^^ + +When :class:`ArgumentParser` generates help messages, it need some way to refer +to each expected argument. By default, ArgumentParser objects use the dest_ +value as the "name" of each object. By default, for positional argument +actions, the dest_ value is used directly, and for optional argument actions, +the dest_ value is uppercased. So, a single positional argument with +``dest='bar'`` will that argument will be referred to as ``bar``. A single +optional argument ``--foo`` that should be followed by a single command-line arg +will be referred to as ``FOO``. An example:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.add_argument('bar') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo FOO] bar + + positional arguments: + bar + + optional arguments: + -h, --help show this help message and exit + --foo FOO + +An alternative name can be specified with ``metavar``:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', metavar='YYY') + >>> parser.add_argument('bar', metavar='XXX') + >>> parser.parse_args('X --foo Y'.split()) + Namespace(bar='X', foo='Y') + >>> parser.print_help() + usage: [-h] [--foo YYY] XXX + + positional arguments: + XXX + + optional arguments: + -h, --help show this help message and exit + --foo YYY + +Note that ``metavar`` only changes the *displayed* name - the name of the +attribute on the :meth:`parse_args` object is still determined by the dest_ +value. + +Different values of ``nargs`` may cause the metavar to be used multiple times. +Providing a tuple to ``metavar`` specifies a different display for each of the +arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', nargs=2) + >>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz')) + >>> parser.print_help() + usage: PROG [-h] [-x X X] [--foo bar baz] + + optional arguments: + -h, --help show this help message and exit + -x X X + --foo bar baz + + +dest +^^^^ + +Most :class:`ArgumentParser` actions add some value as an attribute of the +object returned by :meth:`parse_args`. The name of this attribute is determined +by the ``dest`` keyword argument of :meth:`add_argument`. For positional +argument actions, ``dest`` is normally supplied as the first argument to +:meth:`add_argument`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('bar') + >>> parser.parse_args('XXX'.split()) + Namespace(bar='XXX') + +For optional argument actions, the value of ``dest`` is normally inferred from +the option strings. :class:`ArgumentParser` generates the value of ``dest`` by +taking the first long option string and stripping away the initial ``'--'`` +string. If no long option strings were supplied, ``dest`` will be derived from +the first short option string by stripping the initial ``'-'`` character. Any +internal ``'-'`` characters will be converted to ``'_'`` characters to make sure +the string is a valid attribute name. The examples below illustrate this +behavior:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('-f', '--foo-bar', '--foo') + >>> parser.add_argument('-x', '-y') + >>> parser.parse_args('-f 1 -x 2'.split()) + Namespace(foo_bar='1', x='2') + >>> parser.parse_args('--foo 1 -y 2'.split()) + Namespace(foo_bar='1', x='2') + +``dest`` allows a custom attribute name to be provided:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', dest='bar') + >>> parser.parse_args('--foo XXX'.split()) + Namespace(bar='XXX') + + +The parse_args() method +----------------------- + +.. method:: ArgumentParser.parse_args(args=None, namespace=None) + + Convert argument strings to objects and assign them as attributes of the + namespace. Return the populated namespace. + + Previous calls to :meth:`add_argument` determine exactly what objects are + created and how they are assigned. See the documentation for + :meth:`add_argument` for details. + + By default, the arg strings are taken from :data:`sys.argv`, and a new empty + :class:`Namespace` object is created for the attributes. + + +Option value syntax +^^^^^^^^^^^^^^^^^^^ + +The :meth:`parse_args` method supports several ways of specifying the value of +an option (if it takes one). In the simplest case, the option and its value are +passed as two separate arguments:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('--foo') + >>> parser.parse_args('-x X'.split()) + Namespace(foo=None, x='X') + >>> parser.parse_args('--foo FOO'.split()) + Namespace(foo='FOO', x=None) + +For long options (options with names longer than a single character), the option +and value can also be passed as a single command line argument, using ``=`` to +separate them:: + + >>> parser.parse_args('--foo=FOO'.split()) + Namespace(foo='FOO', x=None) + +For short options (options only one character long), the option and its value +can be concatenated:: + + >>> parser.parse_args('-xX'.split()) + Namespace(foo=None, x='X') + +Several short options can be joined together, using only a single ``-`` prefix, +as long as only the last option (or none of them) requires a value:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x', action='store_true') + >>> parser.add_argument('-y', action='store_true') + >>> parser.add_argument('-z') + >>> parser.parse_args('-xyzZ'.split()) + Namespace(x=True, y=True, z='Z') + + +Invalid arguments +^^^^^^^^^^^^^^^^^ + +While parsing the command-line, ``parse_args`` checks for a variety of errors, +including ambiguous options, invalid types, invalid options, wrong number of +positional arguments, etc. When it encounters such an error, it exits and +prints the error along with a usage message:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', type=int) + >>> parser.add_argument('bar', nargs='?') + + >>> # invalid type + >>> parser.parse_args(['--foo', 'spam']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: argument --foo: invalid int value: 'spam' + + >>> # invalid option + >>> parser.parse_args(['--bar']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: no such option: --bar + + >>> # wrong number of arguments + >>> parser.parse_args(['spam', 'badger']) + usage: PROG [-h] [--foo FOO] [bar] + PROG: error: extra arguments found: badger + + +Arguments containing ``"-"`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``parse_args`` method attempts to give errors whenever the user has clearly +made a mistake, but some situations are inherently ambiguous. For example, the +command-line arg ``'-1'`` could either be an attempt to specify an option or an +attempt to provide a positional argument. The ``parse_args`` method is cautious +here: positional arguments may only begin with ``'-'`` if they look like +negative numbers and there are no options in the parser that look like negative +numbers:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-x') + >>> parser.add_argument('foo', nargs='?') + + >>> # no negative number options, so -1 is a positional argument + >>> parser.parse_args(['-x', '-1']) + Namespace(foo=None, x='-1') + + >>> # no negative number options, so -1 and -5 are positional arguments + >>> parser.parse_args(['-x', '-1', '-5']) + Namespace(foo='-5', x='-1') + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-1', dest='one') + >>> parser.add_argument('foo', nargs='?') + + >>> # negative number options present, so -1 is an option + >>> parser.parse_args(['-1', 'X']) + Namespace(foo=None, one='X') + + >>> # negative number options present, so -2 is an option + >>> parser.parse_args(['-2']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: no such option: -2 + + >>> # negative number options present, so both -1s are options + >>> parser.parse_args(['-1', '-1']) + usage: PROG [-h] [-1 ONE] [foo] + PROG: error: argument -1: expected one argument + +If you have positional arguments that must begin with ``'-'`` and don't look +like negative numbers, you can insert the pseudo-argument ``'--'`` which tells +``parse_args`` that everything after that is a positional argument:: + + >>> parser.parse_args(['--', '-f']) + Namespace(foo='-f', one=None) + + +Argument abbreviations +^^^^^^^^^^^^^^^^^^^^^^ + +The :meth:`parse_args` method allows long options to be abbreviated if the +abbreviation is unambiguous:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('-bacon') + >>> parser.add_argument('-badger') + >>> parser.parse_args('-bac MMM'.split()) + Namespace(bacon='MMM', badger=None) + >>> parser.parse_args('-bad WOOD'.split()) + Namespace(bacon=None, badger='WOOD') + >>> parser.parse_args('-ba BA'.split()) + usage: PROG [-h] [-bacon BACON] [-badger BADGER] + PROG: error: ambiguous option: -ba could match -badger, -bacon + +An error is produced for arguments that could produce more than one options. + + +Beyond ``sys.argv`` +^^^^^^^^^^^^^^^^^^^ + +Sometimes it may be useful to have an ArgumentParser parse args other than those +of :data:`sys.argv`. This can be accomplished by passing a list of strings to +``parse_args``. This is useful for testing at the interactive prompt:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument( + ... 'integers', metavar='int', type=int, choices=xrange(10), + ... nargs='+', help='an integer in the range 0..9') + >>> parser.add_argument( + ... '--sum', dest='accumulate', action='store_const', const=sum, + ... default=max, help='sum the integers (default: find the max)') + >>> parser.parse_args(['1', '2', '3', '4']) + Namespace(accumulate=<built-in function max>, integers=[1, 2, 3, 4]) + >>> parser.parse_args('1 2 3 4 --sum'.split()) + Namespace(accumulate=<built-in function sum>, integers=[1, 2, 3, 4]) + + +The Namespace object +^^^^^^^^^^^^^^^^^^^^ + +By default, :meth:`parse_args` will return a new object of type :class:`Namespace` +where the necessary attributes have been set. This class is deliberately simple, +just an :class:`object` subclass with a readable string representation. If you +prefer to have dict-like view of the attributes, you can use the standard Python +idiom via :func:`vars`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> args = parser.parse_args(['--foo', 'BAR']) + >>> vars(args) + {'foo': 'BAR'} + +It may also be useful to have an :class:`ArgumentParser` assign attributes to an +already existing object, rather than a new :class:`Namespace` object. This can +be achieved by specifying the ``namespace=`` keyword argument:: + + >>> class C(object): + ... pass + ... + >>> c = C() + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) + >>> c.foo + 'BAR' + + +Other utilities +--------------- + +Sub-commands +^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_subparsers() + + Many programs split up their functionality into a number of sub-commands, + for example, the ``svn`` program can invoke sub-commands like ``svn + checkout``, ``svn update``, and ``svn commit``. Splitting up functionality + this way can be a particularly good idea when a program performs several + different functions which require different kinds of command-line arguments. + :class:`ArgumentParser` supports the creation of such sub-commands with the + :meth:`add_subparsers` method. The :meth:`add_subparsers` method is normally + called with no arguments and returns an special action object. This object + has a single method, ``add_parser``, which takes a command name and any + :class:`ArgumentParser` constructor arguments, and returns an + :class:`ArgumentParser` object that can be modified as usual. + + Some example usage:: + + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> parser.add_argument('--foo', action='store_true', help='foo help') + >>> subparsers = parser.add_subparsers(help='sub-command help') + >>> + >>> # create the parser for the "a" command + >>> parser_a = subparsers.add_parser('a', help='a help') + >>> parser_a.add_argument('bar', type=int, help='bar help') + >>> + >>> # create the parser for the "b" command + >>> parser_b = subparsers.add_parser('b', help='b help') + >>> parser_b.add_argument('--baz', choices='XYZ', help='baz help') + >>> + >>> # parse some arg lists + >>> parser.parse_args(['a', '12']) + Namespace(bar=12, foo=False) + >>> parser.parse_args(['--foo', 'b', '--baz', 'Z']) + Namespace(baz='Z', foo=True) + + Note that the object returned by :meth:`parse_args` will only contain + attributes for the main parser and the subparser that was selected by the + command line (and not any other subparsers). So in the example above, when + the ``"a"`` command is specified, only the ``foo`` and ``bar`` attributes are + present, and when the ``"b"`` command is specified, only the ``foo`` and + ``baz`` attributes are present. + + Similarly, when a help message is requested from a subparser, only the help + for that particular parser will be printed. The help message will not + include parent parser or sibling parser messages. (A help message for each + subparser command, however, can be given by supplying the ``help=`` argument + to ``add_parser`` as above.) + + :: + + >>> parser.parse_args(['--help']) + usage: PROG [-h] [--foo] {a,b} ... + + positional arguments: + {a,b} sub-command help + a a help + b b help + + optional arguments: + -h, --help show this help message and exit + --foo foo help + + >>> parser.parse_args(['a', '--help']) + usage: PROG a [-h] bar + + positional arguments: + bar bar help + + optional arguments: + -h, --help show this help message and exit + + >>> parser.parse_args(['b', '--help']) + usage: PROG b [-h] [--baz {X,Y,Z}] + + optional arguments: + -h, --help show this help message and exit + --baz {X,Y,Z} baz help + + The :meth:`add_subparsers` method also supports ``title`` and ``description`` + keyword arguments. When either is present, the subparser's commands will + appear in their own group in the help output. For example:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(title='subcommands', + ... description='valid subcommands', + ... help='additional help') + >>> subparsers.add_parser('foo') + >>> subparsers.add_parser('bar') + >>> parser.parse_args(['-h']) + usage: [-h] {foo,bar} ... + + optional arguments: + -h, --help show this help message and exit + + subcommands: + valid subcommands + + {foo,bar} additional help + + + One particularly effective way of handling sub-commands is to combine the use + of the :meth:`add_subparsers` method with calls to :meth:`set_defaults` so + that each subparser knows which Python function it should execute. For + example:: + + >>> # sub-command functions + >>> def foo(args): + ... print args.x * args.y + ... + >>> def bar(args): + ... print '((%s))' % args.z + ... + >>> # create the top-level parser + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers() + >>> + >>> # create the parser for the "foo" command + >>> parser_foo = subparsers.add_parser('foo') + >>> parser_foo.add_argument('-x', type=int, default=1) + >>> parser_foo.add_argument('y', type=float) + >>> parser_foo.set_defaults(func=foo) + >>> + >>> # create the parser for the "bar" command + >>> parser_bar = subparsers.add_parser('bar') + >>> parser_bar.add_argument('z') + >>> parser_bar.set_defaults(func=bar) + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('foo 1 -x 2'.split()) + >>> args.func(args) + 2.0 + >>> + >>> # parse the args and call whatever function was selected + >>> args = parser.parse_args('bar XYZYX'.split()) + >>> args.func(args) + ((XYZYX)) + + This way, you can let :meth:`parse_args` does the job of calling the + appropriate function after argument parsing is complete. Associating + functions with actions like this is typically the easiest way to handle the + different actions for each of your subparsers. However, if it is necessary + to check the name of the subparser that was invoked, the ``dest`` keyword + argument to the :meth:`add_subparsers` call will work:: + + >>> parser = argparse.ArgumentParser() + >>> subparsers = parser.add_subparsers(dest='subparser_name') + >>> subparser1 = subparsers.add_parser('1') + >>> subparser1.add_argument('-x') + >>> subparser2 = subparsers.add_parser('2') + >>> subparser2.add_argument('y') + >>> parser.parse_args(['2', 'frobble']) + Namespace(subparser_name='2', y='frobble') + + +FileType objects +^^^^^^^^^^^^^^^^ + +.. class:: FileType(mode='r', bufsize=None) + + The :class:`FileType` factory creates objects that can be passed to the type + argument of :meth:`ArgumentParser.add_argument`. Arguments that have + :class:`FileType` objects as their type will open command-line args as files + with the requested modes and buffer sizes: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--output', type=argparse.FileType('wb', 0)) + >>> parser.parse_args(['--output', 'out']) + Namespace(output=<open file 'out', mode 'wb' at 0x...>) + + FileType objects understand the pseudo-argument ``'-'`` and automatically + convert this into ``sys.stdin`` for readable :class:`FileType` objects and + ``sys.stdout`` for writable :class:`FileType` objects: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('infile', type=argparse.FileType('r')) + >>> parser.parse_args(['-']) + Namespace(infile=<open file '<stdin>', mode 'r' at 0x...>) + + +Argument groups +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.add_argument_group(title=None, description=None) + + By default, :class:`ArgumentParser` groups command-line arguments into + "positional arguments" and "optional arguments" when displaying help + messages. When there is a better conceptual grouping of arguments than this + default one, appropriate groups can be created using the + :meth:`add_argument_group` method:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group = parser.add_argument_group('group') + >>> group.add_argument('--foo', help='foo help') + >>> group.add_argument('bar', help='bar help') + >>> parser.print_help() + usage: PROG [--foo FOO] bar + + group: + bar bar help + --foo FOO foo help + + The :meth:`add_argument_group` method returns an argument group object which + has an :meth:`~ArgumentParser.add_argument` method just like a regular + :class:`ArgumentParser`. When an argument is added to the group, the parser + treats it just like a normal argument, but displays the argument in a + separate group for help messages. The :meth:`add_argument_group` method + accepts *title* and *description* arguments which can be used to + customize this display:: + + >>> parser = argparse.ArgumentParser(prog='PROG', add_help=False) + >>> group1 = parser.add_argument_group('group1', 'group1 description') + >>> group1.add_argument('foo', help='foo help') + >>> group2 = parser.add_argument_group('group2', 'group2 description') + >>> group2.add_argument('--bar', help='bar help') + >>> parser.print_help() + usage: PROG [--bar BAR] foo + + group1: + group1 description + + foo foo help + + group2: + group2 description + + --bar BAR bar help + + Note that any arguments not your user defined groups will end up back in the + usual "positional arguments" and "optional arguments" sections. + + +Mutual exclusion +^^^^^^^^^^^^^^^^ + +.. method:: add_mutually_exclusive_group(required=False) + + Create a mutually exclusive group. argparse will make sure that only one of + the arguments in the mutually exclusive group was present on the command + line:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group() + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args(['--foo']) + Namespace(bar=True, foo=True) + >>> parser.parse_args(['--bar']) + Namespace(bar=False, foo=False) + >>> parser.parse_args(['--foo', '--bar']) + usage: PROG [-h] [--foo | --bar] + PROG: error: argument --bar: not allowed with argument --foo + + The :meth:`add_mutually_exclusive_group` method also accepts a *required* + argument, to indicate that at least one of the mutually exclusive arguments + is required:: + + >>> parser = argparse.ArgumentParser(prog='PROG') + >>> group = parser.add_mutually_exclusive_group(required=True) + >>> group.add_argument('--foo', action='store_true') + >>> group.add_argument('--bar', action='store_false') + >>> parser.parse_args([]) + usage: PROG [-h] (--foo | --bar) + PROG: error: one of the arguments --foo --bar is required + + Note that currently mutually exclusive argument groups do not support the + *title* and *description* arguments of :meth:`add_argument_group`. + + +Parser defaults +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.set_defaults(**kwargs) + + Most of the time, the attributes of the object returned by :meth:`parse_args` + will be fully determined by inspecting the command-line args and the argument + actions. :meth:`ArgumentParser.set_defaults` allows some additional + attributes that are determined without any inspection of the command-line to + be added:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('foo', type=int) + >>> parser.set_defaults(bar=42, baz='badger') + >>> parser.parse_args(['736']) + Namespace(bar=42, baz='badger', foo=736) + + Note that parser-level defaults always override argument-level defaults:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='bar') + >>> parser.set_defaults(foo='spam') + >>> parser.parse_args([]) + Namespace(foo='spam') + + Parser-level defaults can be particularly useful when working with multiple + parsers. See the :meth:`~ArgumentParser.add_subparsers` method for an + example of this type. + +.. method:: ArgumentParser.get_default(dest) + + Get the default value for a namespace attribute, as set by either + :meth:`~ArgumentParser.add_argument` or by + :meth:`~ArgumentParser.set_defaults`:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', default='badger') + >>> parser.get_default('foo') + 'badger' + + +Printing help +^^^^^^^^^^^^^ + +In most typical applications, :meth:`parse_args` will take care of formatting +and printing any usage or error messages. However, several formatting methods +are available: + +.. method:: ArgumentParser.print_usage(file=None) + + Print a brief description of how the :class:`ArgumentParser` should be + invoked on the command line. If *file* is ``None``, :data:`sys.stdout` is + assumed. + +.. method:: ArgumentParser.print_help(file=None) + + Print a help message, including the program usage and information about the + arguments registered with the :class:`ArgumentParser`. If *file* is + ``None``, :data:`sys.stdout` is assumed. + +There are also variants of these methods that simply return a string instead of +printing it: + +.. method:: ArgumentParser.format_usage() + + Return a string containing a brief description of how the + :class:`ArgumentParser` should be invoked on the command line. + +.. method:: ArgumentParser.format_help() + + Return a string containing a help message, including the program usage and + information about the arguments registered with the :class:`ArgumentParser`. + + +Partial parsing +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.parse_known_args(args=None, namespace=None) + +Sometimes a script may only parse a few of the command line arguments, passing +the remaining arguments on to another script or program. In these cases, the +:meth:`parse_known_args` method can be useful. It works much like +:meth:`~ArgumentParser.parse_args` except that it does not produce an error when +extra arguments are present. Instead, it returns a two item tuple containing +the populated namespace and the list of remaining argument strings. + +:: + + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('bar') + >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) + (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) + + +Customizing file parsing +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.convert_arg_line_to_args(arg_line) + + Arguments that are read from a file (see the *fromfile_prefix_chars* + keyword argument to the :class:`ArgumentParser` constructor) are read one + argument per line. :meth:`convert_arg_line_to_args` can be overriden for + fancier reading. + + This method takes a single argument *arg_line* which is a string read from + the argument file. It returns a list of arguments parsed from this string. + The method is called once per line read from the argument file, in order. + + A useful override of this method is one that treats each space-separated word + as an argument:: + + def convert_arg_line_to_args(self, arg_line): + for arg in arg_line.split(): + if not arg.strip(): + continue + yield arg + + +Exiting methods +^^^^^^^^^^^^^^^ + +.. method:: ArgumentParser.exit(status=0, message=None) + + This method terminates the program, exiting with the specified *status* + and, if given, it prints a *message* before that. + +.. method:: ArgumentParser.error(message) + + This method prints a usage message including the *message* to the + standard output and terminates the program with a status code of 2. + + +.. _argparse-from-optparse: + +Upgrading optparse code +----------------------- + +Originally, the argparse module had attempted to maintain compatibility with +optparse. However, optparse was difficult to extend transparently, particularly +with the changes required to support the new ``nargs=`` specifiers and better +usage messages. When most everything in optparse had either been copy-pasted +over or monkey-patched, it no longer seemed practical to try to maintain the +backwards compatibility. + +A partial upgrade path from optparse to argparse: + +* Replace all ``add_option()`` calls with :meth:`ArgumentParser.add_argument` + calls. + +* Replace ``options, args = parser.parse_args()`` with ``args = + parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` + calls for the positional arguments. + +* Replace callback actions and the ``callback_*`` keyword arguments with + ``type`` or ``action`` arguments. + +* Replace string names for ``type`` keyword arguments with the corresponding + type objects (e.g. int, float, complex, etc). + +* Replace :class:`optparse.Values` with :class:`Namespace` and + :exc:`optparse.OptionError` and :exc:`optparse.OptionValueError` with + :exc:`ArgumentError`. + +* Replace strings with implicit arguments such as ``%default`` or ``%prog`` with + the standard python syntax to use dictionaries to format strings, that is, + ``%(default)s`` and ``%(prog)s``. + +* Replace the OptionParser constructor ``version`` argument with a call to + ``parser.add_argument('--version', action='version', version='<the version>')`` diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 82ca6e3..ef36f50 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -28,6 +28,11 @@ classes all inherit from :class:`ast.AST`. An abstract syntax tree can be compiled into a Python code object using the built-in :func:`compile` function. +.. seealso:: + + Latest version of the `ast module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/ast.py?view=markup>`_ + Node classes ------------ @@ -122,9 +127,9 @@ The abstract grammar is currently defined as follows: Apart from the node classes, :mod:`ast` module defines these utility functions and classes for traversing abstract syntax trees: -.. function:: parse(expr, filename='<unknown>', mode='exec') +.. function:: parse(source, filename='<unknown>', mode='exec') - Parse an expression into an AST node. Equivalent to ``compile(expr, + Parse the source into an AST node. Equivalent to ``compile(source, filename, mode, ast.PyCF_ONLY_AST)``. @@ -182,9 +187,9 @@ and classes for traversing abstract syntax trees: .. function:: walk(node) - Recursively yield all child nodes of *node*, in no specified order. This is - useful if you only want to modify nodes in place and don't care about the - context. + Recursively yield all descendant nodes in the tree starting at *node* + (including *node* itself), in no specified order. This is useful if you only + want to modify nodes in place and don't care about the context. .. class:: NodeVisitor() diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst index 468e5b5..813ac21 100644 --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -211,10 +211,13 @@ any that have been added to the map during asynchronous service) is closed. .. method:: accept() Accept a connection. The socket must be bound to an address and listening - for connections. The return value is a pair ``(conn, address)`` where - *conn* is a *new* socket object usable to send and receive data on the - connection, and *address* is the address bound to the socket on the other - end of the connection. + for connections. The return value can be either ``None`` or a pair + ``(conn, address)`` where *conn* is a *new* socket object usable to send + and receive data on the connection, and *address* is the address bound to + the socket on the other end of the connection. + When ``None`` is returned it means the connection didn't take place, in + which case the server should just ignore this event and keep listening + for further incoming connections. .. method:: close() @@ -224,6 +227,12 @@ any that have been added to the map during asynchronous service) is closed. flushed). Sockets are automatically closed when they are garbage-collected. +.. class:: dispatcher_with_send() + + A :class:`dispatcher` subclass which adds simple buffered output capability, + useful for simple clients. For more sophisticated usage use + :class:`asynchat.async_chat`. + .. class:: file_dispatcher() A file_dispatcher takes a file descriptor or file object along with an @@ -240,7 +249,7 @@ any that have been added to the map during asynchronous service) is closed. socket for use by the :class:`file_dispatcher` class. Availability: UNIX. -.. _asyncore-example: +.. _asyncore-example-1: asyncore Example basic HTTP client ---------------------------------- @@ -250,7 +259,7 @@ implement its socket handling:: import asyncore, socket - class http_client(asyncore.dispatcher): + class HTTPClient(asyncore.dispatcher): def __init__(self, host, path): asyncore.dispatcher.__init__(self) @@ -274,6 +283,46 @@ implement its socket handling:: sent = self.send(self.buffer) self.buffer = self.buffer[sent:] - c = http_client('www.python.org', '/') + client = HTTPClient('www.python.org', '/') asyncore.loop() + +.. _asyncore-example-2: + +asyncore Example basic echo server +---------------------------------- + +Here is a basic echo server that uses the :class:`dispatcher` class to accept +connections and dispatches the incoming connections to a handler:: + + import asyncore + import socket + + class EchoHandler(asyncore.dispatcher_with_send): + + def handle_read(self): + data = self.recv(8192) + if data: + self.send(data) + + class EchoServer(asyncore.dispatcher): + + def __init__(self, host, port): + asyncore.dispatcher.__init__(self) + self.create_socket(socket.AF_INET, socket.SOCK_STREAM) + self.set_reuse_addr() + self.bind((host, port)) + self.listen(5) + + def handle_accept(self): + pair = self.accept() + if pair is None: + pass + else: + sock, addr = pair + print 'Incoming connection from %s' % repr(addr) + handler = EchoHandler(sock) + + server = EchoServer('localhost', 8080) + asyncore.loop() + diff --git a/Doc/library/atexit.rst b/Doc/library/atexit.rst index 8b33d5c..eab8cd9 100644 --- a/Doc/library/atexit.rst +++ b/Doc/library/atexit.rst @@ -14,9 +14,14 @@ The :mod:`atexit` module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. +.. seealso:: + + Latest version of the `atexit Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/atexit.py?view=markup>`_ + Note: the functions registered via this module are not called when the program -is killed by a signal, when a Python fatal internal error is detected, or when -:func:`os._exit` is called. +is killed by a signal not handled by Python, when a Python fatal internal error +is detected, or when :func:`os._exit` is called. .. index:: single: exitfunc (in sys) diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst index b497e1f..29c3914 100644 --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -236,8 +236,8 @@ and recombined later. Here is an example of how to do that:: def mul_stereo(sample, width, lfactor, rfactor): lsample = audioop.tomono(sample, width, 1, 0) rsample = audioop.tomono(sample, width, 0, 1) - lsample = audioop.mul(sample, width, lfactor) - rsample = audioop.mul(sample, width, rfactor) + lsample = audioop.mul(lsample, width, lfactor) + rsample = audioop.mul(rsample, width, rfactor) lsample = audioop.tostereo(lsample, width, 1, 0) rsample = audioop.tostereo(rsample, width, 0, 1) return audioop.add(lsample, rsample, width) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 084660d..ab85436 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -1,4 +1,3 @@ - :mod:`base64` --- RFC 3548: Base16, Base32, Base64 Data Encodings ================================================================= diff --git a/Doc/library/bdb.rst b/Doc/library/bdb.rst index 4ee2f19..93304de 100644 --- a/Doc/library/bdb.rst +++ b/Doc/library/bdb.rst @@ -108,7 +108,7 @@ The :mod:`bdb` module also defines two classes: * ``"exception"``: An exception has occurred. * ``"c_call"``: A C function is about to be called. * ``"c_return"``: A C function has returned. - * ``"c_exception"``: A C function has thrown an exception. + * ``"c_exception"``: A C function has raised an exception. For the Python events, specialized functions (see below) are called. For the C events, no action is taken. @@ -342,12 +342,10 @@ Finally, the module defines the following functions: .. function:: effective(file, line, frame) Determine if there is an effective (active) breakpoint at this line of code. - Return breakpoint number or 0 if none. - - Called only if we know there is a breakpoint at this location. Returns the - breakpoint that was triggered and a flag that indicates if it is ok to delete - a temporary breakpoint. + Return a tuple of the breakpoint and a boolean that indicates if it is ok + to delete a temporary breakpoint. Return ``(None, None)`` if there is no + matching breakpoint. .. function:: set_trace() - Starts debugging with a :class:`Bdb` instance from caller's frame. + Start debugging with a :class:`Bdb` instance from caller's frame. diff --git a/Doc/library/bisect.rst b/Doc/library/bisect.rst index eb32354..48cb331 100644 --- a/Doc/library/bisect.rst +++ b/Doc/library/bisect.rst @@ -1,10 +1,10 @@ - :mod:`bisect` --- Array bisection algorithm =========================================== .. module:: bisect :synopsis: Array bisection algorithms for binary searching. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> +.. sectionauthor:: Raymond Hettinger <python at rcn.com> .. example based on the PyModules FAQ entry by Aaron Watters <arw@pythonpros.com> This module provides support for maintaining a list in sorted order without @@ -14,80 +14,125 @@ approach. The module is called :mod:`bisect` because it uses a basic bisection algorithm to do its work. The source code may be most useful as a working example of the algorithm (the boundary conditions are already right!). -The following functions are provided: +.. versionadded:: 2.1 + +.. seealso:: + Latest version of the `bisect module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/bisect.py?view=markup>`_ + +The following functions are provided: -.. function:: bisect_left(list, item[, lo[, hi]]) - Locate the proper insertion point for *item* in *list* to maintain sorted order. - The parameters *lo* and *hi* may be used to specify a subset of the list which - should be considered; by default the entire list is used. If *item* is already - present in *list*, the insertion point will be before (to the left of) any - existing entries. The return value is suitable for use as the first parameter - to ``list.insert()``. This assumes that *list* is already sorted. +.. function:: bisect_left(a, x, lo=0, hi=len(a)) - .. versionadded:: 2.1 + Locate the insertion point for *x* in *a* to maintain sorted order. + The parameters *lo* and *hi* may be used to specify a subset of the list + which should be considered; by default the entire list is used. If *x* is + already present in *a*, the insertion point will be before (to the left of) + any existing entries. The return value is suitable for use as the first + parameter to ``list.insert()`` assuming that *a* is already sorted. + The returned insertion point *i* partitions the array *a* into two halves so + that ``all(val < x for val in a[lo:i])`` for the left side and + ``all(val >= x for val in a[i:hi])`` for the right side. -.. function:: bisect_right(list, item[, lo[, hi]]) +.. function:: bisect_right(a, x, lo=0, hi=len(a)) + bisect(a, x, lo=0, hi=len(a)) - Similar to :func:`bisect_left`, but returns an insertion point which comes after - (to the right of) any existing entries of *item* in *list*. + Similar to :func:`bisect_left`, but returns an insertion point which comes + after (to the right of) any existing entries of *x* in *a*. - .. versionadded:: 2.1 + The returned insertion point *i* partitions the array *a* into two halves so + that ``all(val <= x for val in a[lo:i])`` for the left side and + ``all(val > x for val in a[i:hi])`` for the right side. +.. function:: insort_left(a, x, lo=0, hi=len(a)) -.. function:: bisect(...) + Insert *x* in *a* in sorted order. This is equivalent to + ``a.insert(bisect.bisect_left(a, x, lo, hi), x)`` assuming that *a* is + already sorted. Keep in mind that the O(log n) search is dominated by + the slow O(n) insertion step. - Alias for :func:`bisect_right`. +.. function:: insort_right(a, x, lo=0, hi=len(a)) + insort(a, x, lo=0, hi=len(a)) + Similar to :func:`insort_left`, but inserting *x* in *a* after any existing + entries of *x*. -.. function:: insort_left(list, item[, lo[, hi]]) +.. seealso:: - Insert *item* in *list* in sorted order. This is equivalent to - ``list.insert(bisect.bisect_left(list, item, lo, hi), item)``. This assumes - that *list* is already sorted. + `SortedCollection recipe + <http://code.activestate.com/recipes/577197-sortedcollection/>`_ that uses + bisect to build a full-featured collection class with straight-forward search + methods and support for a key-function. The keys are precomputed to save + unnecessary calls to the key function during searches. - .. versionadded:: 2.1 +Searching Sorted Lists +---------------------- -.. function:: insort_right(list, item[, lo[, hi]]) +The above :func:`bisect` functions are useful for finding insertion points but +can be tricky or awkward to use for common searching tasks. The following five +functions show how to transform them into the standard lookups for sorted +lists:: - Similar to :func:`insort_left`, but inserting *item* in *list* after any - existing entries of *item*. + def index(a, x): + 'Locate the leftmost value exactly equal to x' + i = bisect_left(a, x) + if i != len(a) and a[i] == x: + return i + raise ValueError - .. versionadded:: 2.1 + def find_lt(a, x): + 'Find rightmost value less than x' + i = bisect_left(a, x) + if i: + return a[i-1] + raise ValueError + def find_le(a, x): + 'Find rightmost value less than or equal to x' + i = bisect_right(a, x) + if i: + return a[i-1] + raise ValueError -.. function:: insort(...) + def find_gt(a, x): + 'Find leftmost value greater than x' + i = bisect_right(a, x) + if i != len(a): + return a[i] + raise ValueError - Alias for :func:`insort_right`. + def find_ge(a, x): + 'Find leftmost item greater than or equal to x' + i = bisect_left(a, x) + if i != len(a): + return a[i] + raise ValueError -Examples --------- +Other Examples +-------------- .. _bisect-example: -The :func:`bisect` function is generally useful for categorizing numeric data. -This example uses :func:`bisect` to look up a letter grade for an exam total -(say) based on a set of ordered numeric breakpoints: 85 and up is an 'A', 75..84 -is a 'B', etc. +The :func:`bisect` function can be useful for numeric table lookups. This +example uses :func:`bisect` to look up a letter grade for an exam score (say) +based on a set of ordered numeric breakpoints: 90 and up is an 'A', 80 to 89 is +a 'B', and so on:: - >>> grades = "FEDCBA" - >>> breakpoints = [30, 44, 66, 75, 85] - >>> from bisect import bisect - >>> def grade(total): - ... return grades[bisect(breakpoints, total)] + >>> def grade(score, breakpoints=[60, 70, 80, 90], grades='FDCBA'): + ... i = bisect(breakpoints, score) + ... return grades[i] ... - >>> grade(66) - 'C' - >>> map(grade, [33, 99, 77, 44, 12, 88]) - ['E', 'A', 'B', 'D', 'F', 'A'] + >>> [grade(score) for score in [33, 99, 77, 70, 89, 90, 100]] + ['F', 'A', 'C', 'C', 'B', 'A', 'A'] Unlike the :func:`sorted` function, it does not make sense for the :func:`bisect` functions to have *key* or *reversed* arguments because that would lead to an -inefficent design (successive calls to bisect functions would not "remember" +inefficient design (successive calls to bisect functions would not "remember" all of the previous key lookups). Instead, it is better to search a list of precomputed keys to find the index @@ -104,3 +149,4 @@ of the record in question:: ('red', 5) >>> data[bisect_left(keys, 8)] ('yellow', 8) + diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst index bf8ebc0..ed281bd 100644 --- a/Doc/library/bz2.rst +++ b/Doc/library/bz2.rst @@ -61,6 +61,11 @@ Handling of compressed files is offered by the :class:`BZ2File` class. reading. Instances support iteration in the same way as normal :class:`file` instances. + :class:`BZ2File` supports the :keyword:`with` statement. + + .. versionchanged:: 2.7 + Support for the :keyword:`with` statement was added. + .. method:: close() diff --git a/Doc/library/calendar.rst b/Doc/library/calendar.rst index d650ff4..2245e51 100644 --- a/Doc/library/calendar.rst +++ b/Doc/library/calendar.rst @@ -16,12 +16,16 @@ the week to Sunday (6) or to any other weekday. Parameters that specify dates are given as integers. For related functionality, see also the :mod:`datetime` and :mod:`time` modules. -Most of these functions and classses rely on the :mod:`datetime` module which +Most of these functions and classes rely on the :mod:`datetime` module which uses an idealized calendar, the current Gregorian calendar indefinitely extended in both directions. This matches the definition of the "proleptic Gregorian" calendar in Dershowitz and Reingold's book "Calendrical Calculations", where it's the base calendar for all computations. +.. seealso:: + + Latest version of the `calendar module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/calendar.py?view=markup>`_ .. class:: Calendar([firstweekday]) @@ -178,9 +182,9 @@ it's the base calendar for all computations. .. class:: LocaleTextCalendar([firstweekday[, locale]]) This subclass of :class:`TextCalendar` can be passed a locale name in the - constructor and will return month and weekday names in the specified - locale. If this locale includes an encoding all strings containing month and - weekday names will be returned as unicode. + constructor and will return month and weekday names in the specified locale. + If this locale includes an encoding all strings containing month and weekday + names will be returned as unicode. .. versionadded:: 2.5 @@ -194,6 +198,13 @@ it's the base calendar for all computations. .. versionadded:: 2.5 +.. note:: + + The :meth:`formatweekday` and :meth:`formatmonthname` methods of these two + classes temporarily change the current locale to the given *locale*. Because + the current locale is a process-wide setting, they are not thread-safe. + + For simple text calendars this module provides the following functions. diff --git a/Doc/library/cgi.rst b/Doc/library/cgi.rst index f0c0a07..149cf23 100644 --- a/Doc/library/cgi.rst +++ b/Doc/library/cgi.rst @@ -354,7 +354,7 @@ algorithms implemented in this module in other circumstances. that single quotes are never translated. If the value to be quoted might include single- or double-quote characters, - or both, consider using the :func:`quoteattr` function in the + or both, consider using the :func:`~xml.sax.saxutils.quoteattr` function in the :mod:`xml.sax.saxutils` module instead. diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index cc49990..b35373c 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -12,6 +12,10 @@ command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface. +.. seealso:: + + Latest version of the `cmd module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/cmd.py?view=markup>`_ .. class:: Cmd([completekey[, stdin[, stdout]]]) diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index d966262..71bf25a 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -760,7 +760,7 @@ Encodings and Unicode Unicode strings are stored internally as sequences of codepoints (to be precise as :ctype:`Py_UNICODE` arrays). Depending on the way Python is compiled (either -via :option:`--enable-unicode=ucs2` or :option:`--enable-unicode=ucs4`, with the +via ``--enable-unicode=ucs2`` or ``--enable-unicode=ucs4``, with the former being the default) :ctype:`Py_UNICODE` is either a 16-bit or 32-bit data type. Once a Unicode object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a @@ -908,6 +908,8 @@ particular, the following variants typically exist: | cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | | | IBM500 | | +-----------------+--------------------------------+--------------------------------+ +| cp720 | | Arabic | ++-----------------+--------------------------------+--------------------------------+ | cp737 | | Greek | +-----------------+--------------------------------+--------------------------------+ | cp775 | IBM775 | Baltic languages | @@ -923,6 +925,8 @@ particular, the following variants typically exist: +-----------------+--------------------------------+--------------------------------+ | cp857 | 857, IBM857 | Turkish | +-----------------+--------------------------------+--------------------------------+ +| cp858 | 858, IBM858 | Western Europe | ++-----------------+--------------------------------+--------------------------------+ | cp860 | 860, IBM860 | Portuguese | +-----------------+--------------------------------+--------------------------------+ | cp861 | 861, CP-IS, IBM861 | Icelandic | diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index b74c24d..48cac2f 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -15,144 +15,206 @@ import itertools __name__ = '<doctest>' -This module implements high-performance container datatypes. Currently, -there are two datatypes, :class:`deque` and :class:`defaultdict`, and -one datatype factory function, :func:`namedtuple`. +This module implements specialized container datatypes providing alternatives to +Python's general purpose built-in containers, :class:`dict`, :class:`list`, +:class:`set`, and :class:`tuple`. + +===================== ==================================================================== =========================== +:func:`namedtuple` factory function for creating tuple subclasses with named fields .. versionadded:: 2.6 +:class:`deque` list-like container with fast appends and pops on either end .. versionadded:: 2.4 +:class:`Counter` dict subclass for counting hashable objects .. versionadded:: 2.7 +:class:`OrderedDict` dict subclass that remembers the order entries were added .. versionadded:: 2.7 +:class:`defaultdict` dict subclass that calls a factory function to supply missing values .. versionadded:: 2.5 +===================== ==================================================================== =========================== + +In addition to the concrete container classes, the collections module provides +:ref:`abstract-base-classes` that can be used to test whether a class provides a +particular interface, for example, whether it is hashable or a mapping. -.. versionchanged:: 2.5 - Added :class:`defaultdict`. +.. seealso:: -.. versionchanged:: 2.6 - Added :func:`namedtuple`. + Latest version of the `collections module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/collections.py?view=markup>`_ -The specialized containers provided in this module provide alternatives -to Python's general purpose built-in containers, :class:`dict`, -:class:`list`, :class:`set`, and :class:`tuple`. +:class:`Counter` objects +------------------------ -Besides the containers provided here, the optional :mod:`bsddb` -module offers the ability to create in-memory or file based ordered -dictionaries with string keys using the :meth:`bsddb.btopen` method. +A counter tool is provided to support convenient and rapid tallies. +For example:: -In addition to containers, the collections module provides some ABCs -(abstract base classes) that can be used to test whether a class -provides a particular interface, for example, is it hashable or -a mapping. + >>> # Tally occurrences of words in a list + >>> cnt = Counter() + >>> for word in ['red', 'blue', 'red', 'green', 'blue', 'blue']: + ... cnt[word] += 1 + >>> cnt + Counter({'blue': 3, 'red': 2, 'green': 1}) -.. versionchanged:: 2.6 - Added abstract base classes. + >>> # Find the ten most common words in Hamlet + >>> import re + >>> words = re.findall('\w+', open('hamlet.txt').read().lower()) + >>> Counter(words).most_common(10) + [('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631), + ('you', 554), ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)] -ABCs - abstract base classes ----------------------------- +.. class:: Counter([iterable-or-mapping]) -The collections module offers the following ABCs: + A :class:`Counter` is a :class:`dict` subclass for counting hashable objects. + It is an unordered collection where elements are stored as dictionary keys + and their counts are stored as dictionary values. Counts are allowed to be + any integer value including zero or negative counts. The :class:`Counter` + class is similar to bags or multisets in other languages. -========================= ===================== ====================== ==================================================== -ABC Inherits Abstract Methods Mixin Methods -========================= ===================== ====================== ==================================================== -:class:`Container` ``__contains__`` -:class:`Hashable` ``__hash__`` -:class:`Iterable` ``__iter__`` -:class:`Iterator` :class:`Iterable` ``next`` ``__iter__`` -:class:`Sized` ``__len__`` -:class:`Callable` ``__call__`` + Elements are counted from an *iterable* or initialized from another + *mapping* (or counter): -:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``. - :class:`Iterable`, ``index``, and ``count`` - :class:`Container` + >>> c = Counter() # a new, empty counter + >>> c = Counter('gallahad') # a new counter from an iterable + >>> c = Counter({'red': 4, 'blue': 2}) # a new counter from a mapping + >>> c = Counter(cats=4, dogs=8) # a new counter from keyword args -:class:`MutableSequence` :class:`Sequence` ``__setitem__`` Inherited Sequence methods and - ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``, - and ``insert`` ``remove``, and ``__iadd__`` + Counter objects have a dictionary interface except that they return a zero + count for missing items instead of raising a :exc:`KeyError`: -:class:`Set` :class:`Sized`, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, - :class:`Iterable`, ``__gt__``, ``__ge__``, ``__and__``, ``__or__`` - :class:`Container` ``__sub__``, ``__xor__``, and ``isdisjoint`` + >>> c = Counter(['eggs', 'ham']) + >>> c['bacon'] # count of a missing element is zero + 0 -:class:`MutableSet` :class:`Set` ``add`` and Inherited Set methods and - ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``, - ``__iand__``, ``__ixor__``, and ``__isub__`` + Setting a count to zero does not remove an element from a counter. + Use ``del`` to remove it entirely: -:class:`Mapping` :class:`Sized`, ``__getitem__`` ``__contains__``, ``keys``, ``items``, ``values``, - :class:`Iterable`, ``get``, ``__eq__``, and ``__ne__`` - :class:`Container` + >>> c['sausage'] = 0 # counter entry with a zero count + >>> del c['sausage'] # del actually removes the entry -:class:`MutableMapping` :class:`Mapping` ``__setitem__`` and Inherited Mapping methods and - ``__delitem__`` ``pop``, ``popitem``, ``clear``, ``update``, - and ``setdefault`` + .. versionadded:: 2.7 -:class:`MappingView` :class:`Sized` ``__len__`` -:class:`KeysView` :class:`MappingView`, ``__contains__``, - :class:`Set` ``__iter__`` -:class:`ItemsView` :class:`MappingView`, ``__contains__``, - :class:`Set` ``__iter__`` -:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` -========================= ===================== ====================== ==================================================== + Counter objects support three methods beyond those available for all + dictionaries: -These ABCs allow us to ask classes or instances if they provide -particular functionality, for example:: + .. method:: elements() - size = None - if isinstance(myvar, collections.Sized): - size = len(myvar) + Return an iterator over elements repeating each as many times as its + count. Elements are returned in arbitrary order. If an element's count + is less than one, :meth:`elements` will ignore it. -Several of the ABCs are also useful as mixins that make it easier to develop -classes supporting container APIs. For example, to write a class supporting -the full :class:`Set` API, it only necessary to supply the three underlying -abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`. -The ABC supplies the remaining methods such as :meth:`__and__` and -:meth:`isdisjoint` :: + >>> c = Counter(a=4, b=2, c=0, d=-2) + >>> list(c.elements()) + ['a', 'a', 'a', 'a', 'b', 'b'] - class ListBasedSet(collections.Set): - ''' Alternate set implementation favoring space over speed - and not requiring the set elements to be hashable. ''' - def __init__(self, iterable): - self.elements = lst = [] - for value in iterable: - if value not in lst: - lst.append(value) - def __iter__(self): - return iter(self.elements) - def __contains__(self, value): - return value in self.elements - def __len__(self): - return len(self.elements) + .. method:: most_common([n]) - s1 = ListBasedSet('abcdef') - s2 = ListBasedSet('defghi') - overlap = s1 & s2 # The __and__() method is supported automatically + Return a list of the *n* most common elements and their counts from the + most common to the least. If *n* is not specified, :func:`most_common` + returns *all* elements in the counter. Elements with equal counts are + ordered arbitrarily: -Notes on using :class:`Set` and :class:`MutableSet` as a mixin: + >>> Counter('abracadabra').most_common(3) + [('a', 5), ('r', 2), ('b', 2)] -(1) - Since some set operations create new sets, the default mixin methods need - a way to create new instances from an iterable. The class constructor is - assumed to have a signature in the form ``ClassName(iterable)``. - That assumption is factored-out to an internal classmethod called - :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set. - If the :class:`Set` mixin is being used in a class with a different - constructor signature, you will need to override :meth:`from_iterable` - with a classmethod that can construct new instances from - an iterable argument. + .. method:: subtract([iterable-or-mapping]) -(2) - To override the comparisons (presumably for speed, as the - semantics are fixed), redefine :meth:`__le__` and - then the other operations will automatically follow suit. + Elements are subtracted from an *iterable* or from another *mapping* + (or counter). Like :meth:`dict.update` but subtracts counts instead + of replacing them. Both inputs and outputs may be zero or negative. -(3) - The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value - for the set; however, :meth:`__hash__` is not defined because not all sets - are hashable or immutable. To add set hashabilty using mixins, - inherit from both :meth:`Set` and :meth:`Hashable`, then define - ``__hash__ = Set._hash``. + >>> c = Counter(a=4, b=2, c=0, d=-2) + >>> d = Counter(a=1, b=2, c=3, d=4) + >>> c.subtract(d) + Counter({'a': 3, 'b': 0, 'c': -3, 'd': -6}) + + The usual dictionary methods are available for :class:`Counter` objects + except for two which work differently for counters. + + .. method:: fromkeys(iterable) + + This class method is not implemented for :class:`Counter` objects. + + .. method:: update([iterable-or-mapping]) + + Elements are counted from an *iterable* or added-in from another + *mapping* (or counter). Like :meth:`dict.update` but adds counts + instead of replacing them. Also, the *iterable* is expected to be a + sequence of elements, not a sequence of ``(key, value)`` pairs. + +Common patterns for working with :class:`Counter` objects:: + + sum(c.values()) # total of all counts + c.clear() # reset all counts + list(c) # list unique elements + set(c) # convert to a set + dict(c) # convert to a regular dictionary + c.items() # convert to a list of (elem, cnt) pairs + Counter(dict(list_of_pairs)) # convert from a list of (elem, cnt) pairs + c.most_common()[:-n:-1] # n least common elements + c += Counter() # remove zero and negative counts + +Several mathematical operations are provided for combining :class:`Counter` +objects to produce multisets (counters that have counts greater than zero). +Addition and subtraction combine counters by adding or subtracting the counts +of corresponding elements. Intersection and union return the minimum and +maximum of corresponding counts. Each operation can accept inputs with signed +counts, but the output will exclude results with counts of zero or less. + + >>> c = Counter(a=3, b=1) + >>> d = Counter(a=1, b=2) + >>> c + d # add two counters together: c[x] + d[x] + Counter({'a': 4, 'b': 3}) + >>> c - d # subtract (keeping only positive counts) + Counter({'a': 2}) + >>> c & d # intersection: min(c[x], d[x]) + Counter({'a': 1, 'b': 1}) + >>> c | d # union: max(c[x], d[x]) + Counter({'a': 3, 'b': 2}) + +.. note:: + + Counters were primarily designed to work with positive integers to represent + running counts; however, care was taken to not unnecessarily preclude use + cases needing other types or negative values. To help with those use cases, + this section documents the minimum range and type restrictions. + + * The :class:`Counter` class itself is a dictionary subclass with no + restrictions on its keys and values. The values are intended to be numbers + representing counts, but you *could* store anything in the value field. + + * The :meth:`most_common` method requires only that the values be orderable. + + * For in-place operations such as ``c[key] += 1``, the value type need only + support addition and subtraction. So fractions, floats, and decimals would + work and negative values are supported. The same is also true for + :meth:`update` and :meth:`subtract` which allow negative and zero values + for both inputs and outputs. + + * The multiset methods are designed only for use cases with positive values. + The inputs may be negative or zero, but only outputs with positive values + are created. There are no type restrictions, but the value type needs to + support support addition, subtraction, and comparison. + + * The :meth:`elements` method requires integer counts. It ignores zero and + negative counts. .. seealso:: - * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an - example built on :class:`MutableSet`. + * `Counter class <http://code.activestate.com/recipes/576611/>`_ + adapted for Python 2.5 and an early `Bag recipe + <http://code.activestate.com/recipes/259174/>`_ for Python 2.4. - * For more about ABCs, see the :mod:`abc` module and :pep:`3119`. + * `Bag class <http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html>`_ + in Smalltalk. + + * Wikipedia entry for `Multisets <http://en.wikipedia.org/wiki/Multiset>`_\. + + * `C++ multisets <http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm>`_ + tutorial with examples. + + * For mathematical operations on multisets and their use cases, see + *Knuth, Donald. The Art of Computer Programming Volume II, + Section 4.6.3, Exercise 19*\. + + * To enumerate all distinct multisets of a given size over a given set of + elements, see :func:`itertools.combinations_with_replacement`. + + map(Counter, combinations_with_replacement('ABC', 2)) --> AA AB AC BB BC CC :class:`deque` objects @@ -204,6 +266,12 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: Remove all elements from the deque leaving it with length 0. + .. method:: count(x) + + Count the number of deque elements equal to *x*. + + .. versionadded:: 2.7 + .. method:: extend(iterable) Extend the right side of the deque by appending elements from the iterable @@ -236,6 +304,11 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: .. versionadded:: 2.5 + .. method:: reverse() + + Reverse the elements of the deque in-place and then return ``None``. + + .. versionadded:: 2.7 .. method:: rotate(n) @@ -244,6 +317,15 @@ Notes on using :class:`Set` and :class:`MutableSet` as a mixin: ``d.appendleft(d.pop())``. + Deque objects also provide one read-only attribute: + + .. attribute:: maxlen + + Maximum size of a deque or *None* if unbounded. + + .. versionadded:: 2.7 + + In addition to the above, deques support iteration, pickling, ``len(d)``, ``reversed(d)``, ``copy.copy(d)``, ``copy.deepcopy(d)``, membership testing with the :keyword:`in` operator, and subscript references such as ``d[-1]``. Indexed @@ -475,7 +557,7 @@ Named tuples assign meaning to each position in a tuple and allow for more reada self-documenting code. They can be used wherever regular tuples are used, and they add the ability to access fields by name instead of position index. -.. function:: namedtuple(typename, field_names, [verbose]) +.. function:: namedtuple(typename, field_names, [verbose=False], [rename=False]) Returns a new tuple subclass named *typename*. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as @@ -483,9 +565,9 @@ they add the ability to access fields by name instead of position index. helpful docstring (with typename and field_names) and a helpful :meth:`__repr__` method which lists the tuple contents in a ``name=value`` format. - The *field_names* are a single string with each fieldname separated by whitespace - and/or commas, for example ``'x y'`` or ``'x, y'``. Alternatively, *field_names* - can be a sequence of strings such as ``['x', 'y']``. + The *field_names* are a sequence of strings such as ``['x', 'y']``. + Alternatively, *field_names* can be a single string with each fieldname + separated by whitespace and/or commas, for example ``'x y'`` or ``'x, y'``. Any valid Python identifier may be used for a fieldname except for names starting with an underscore. Valid identifiers consist of letters, digits, @@ -493,6 +575,11 @@ they add the ability to access fields by name instead of position index. a :mod:`keyword` such as *class*, *for*, *return*, *global*, *pass*, *print*, or *raise*. + If *rename* is true, invalid fieldnames are automatically replaced + with positional names. For example, ``['abc', 'def', 'ghi', 'abc']`` is + converted to ``['abc', '_1', 'ghi', '_3']``, eliminating the keyword + ``def`` and the duplicate fieldname ``abc``. + If *verbose* is true, the class definition is printed just before being built. Named tuple instances do not have per-instance dictionaries, so they are @@ -500,24 +587,15 @@ they add the ability to access fields by name instead of position index. .. versionadded:: 2.6 + .. versionchanged:: 2.7 + added support for *rename*. + Example: .. doctest:: :options: +NORMALIZE_WHITESPACE - >>> Point = namedtuple('Point', 'x y') - >>> p = Point(11, y=22) # instantiate with positional or keyword arguments - >>> p[0] + p[1] # indexable like the plain tuple (11, 22) - 33 - >>> x, y = p # unpack like a regular tuple - >>> x, y - (11, 22) - >>> p.x + p.y # fields also accessible by name - 33 - >>> p # readable __repr__ with a name=value style - Point(x=11, y=22) - - >>> Point = namedtuple('Point', 'x y', verbose=True) # show the class definition + >>> Point = namedtuple('Point', ['x', 'y'], verbose=True) class Point(tuple): 'Point(x, y)' <BLANKLINE> @@ -526,6 +604,7 @@ Example: _fields = ('x', 'y') <BLANKLINE> def __new__(_cls, x, y): + 'Create a new instance of Point(x, y)' return _tuple.__new__(_cls, (x, y)) <BLANKLINE> @classmethod @@ -537,11 +616,12 @@ Example: return result <BLANKLINE> def __repr__(self): + 'Return a nicely formatted representation string' return 'Point(x=%r, y=%r)' % self <BLANKLINE> - def _asdict(t): - 'Return a new dict which maps field names to their values' - return {'x': t[0], 'y': t[1]} + def _asdict(self): + 'Return a new OrderedDict which maps field names to their values' + return OrderedDict(zip(self._fields, self)) <BLANKLINE> def _replace(_self, **kwds): 'Return a new Point object replacing specified fields with new values' @@ -551,10 +631,22 @@ Example: return result <BLANKLINE> def __getnewargs__(self): + 'Return self as a plain tuple. Used by copy and pickle.' return tuple(self) <BLANKLINE> - x = _property(_itemgetter(0)) - y = _property(_itemgetter(1)) + x = _property(_itemgetter(0), doc='Alias for field number 0') + y = _property(_itemgetter(1), doc='Alias for field number 1') + + >>> p = Point(11, y=22) # instantiate with positional or keyword arguments + >>> p[0] + p[1] # indexable like the plain tuple (11, 22) + 33 + >>> x, y = p # unpack like a regular tuple + >>> x, y + (11, 22) + >>> p.x + p.y # fields also accessible by name + 33 + >>> p # readable __repr__ with a name=value style + Point(x=11, y=22) Named tuples are especially useful for assigning field names to result tuples returned by the :mod:`csv` or :mod:`sqlite3` modules:: @@ -588,10 +680,14 @@ field names, the method and attribute names start with an underscore. .. method:: somenamedtuple._asdict() - Return a new dict which maps field names to their corresponding values:: + Return a new :class:`OrderedDict` which maps field names to their corresponding + values:: >>> p._asdict() - {'x': 11, 'y': 22} + OrderedDict([('x', 11), ('y', 22)]) + + .. versionchanged:: 2.7 + Returns an :class:`OrderedDict` instead of a regular :class:`dict`. .. method:: somenamedtuple._replace(kwargs) @@ -603,7 +699,7 @@ field names, the method and attribute names start with an underscore. Point(x=33, y=22) >>> for partnum, record in inventory.items(): - ... inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now()) + inventory[partnum] = record._replace(price=newprices[partnum], timestamp=time.now()) .. attribute:: somenamedtuple._fields @@ -638,15 +734,15 @@ functionality with a subclass. Here is how to add a calculated field and a fixed-width print format: >>> class Point(namedtuple('Point', 'x y')): - ... __slots__ = () - ... @property - ... def hypot(self): - ... return (self.x ** 2 + self.y ** 2) ** 0.5 - ... def __str__(self): - ... return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) + __slots__ = () + @property + def hypot(self): + return (self.x ** 2 + self.y ** 2) ** 0.5 + def __str__(self): + return 'Point: x=%6.3f y=%6.3f hypot=%6.3f' % (self.x, self.y, self.hypot) >>> for p in Point(3, 4), Point(14, 5/7.): - ... print p + print p Point: x= 3.000 y= 4.000 hypot= 5.000 Point: x=14.000 y= 0.714 hypot=14.018 @@ -672,9 +768,247 @@ and more efficient to use a simple class declaration: >>> Status.open, Status.pending, Status.closed (0, 1, 2) >>> class Status: - ... open, pending, closed = range(3) + open, pending, closed = range(3) .. seealso:: `Named tuple recipe <http://code.activestate.com/recipes/500261/>`_ adapted for Python 2.4. + + +:class:`OrderedDict` objects +---------------------------- + +Ordered dictionaries are just like regular dictionaries but they remember the +order that items were inserted. When iterating over an ordered dictionary, +the items are returned in the order their keys were first added. + +.. class:: OrderedDict([items]) + + Return an instance of a dict subclass, supporting the usual :class:`dict` + methods. An *OrderedDict* is a dict that remembers the order that keys + were first inserted. If a new entry overwrites an existing entry, the + original insertion position is left unchanged. Deleting an entry and + reinserting it will move it to the end. + + .. versionadded:: 2.7 + +.. method:: OrderedDict.popitem(last=True) + + The :meth:`popitem` method for ordered dictionaries returns and removes + a (key, value) pair. The pairs are returned in LIFO order if *last* is + true or FIFO order if false. + +In addition to the usual mapping methods, ordered dictionaries also support +reverse iteration using :func:`reversed`. + +Equality tests between :class:`OrderedDict` objects are order-sensitive +and are implemented as ``list(od1.items())==list(od2.items())``. +Equality tests between :class:`OrderedDict` objects and other +:class:`Mapping` objects are order-insensitive like regular dictionaries. +This allows :class:`OrderedDict` objects to be substituted anywhere a +regular dictionary is used. + +The :class:`OrderedDict` constructor and :meth:`update` method both accept +keyword arguments, but their order is lost because Python's function call +semantics pass-in keyword arguments using a regular unordered dictionary. + +.. seealso:: + + `Equivalent OrderedDict recipe <http://code.activestate.com/recipes/576693/>`_ + that runs on Python 2.4 or later. + +Since an ordered dictionary remembers its insertion order, it can be used +in conjuction with sorting to make a sorted dictionary:: + + >>> # regular unsorted dictionary + >>> d = {'banana': 3, 'apple':4, 'pear': 1, 'orange': 2} + + >>> # dictionary sorted by key + >>> OrderedDict(sorted(d.items(), key=lambda t: t[0])) + OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)]) + + >>> # dictionary sorted by value + >>> OrderedDict(sorted(d.items(), key=lambda t: t[1])) + OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)]) + + >>> # dictionary sorted by length of the key string + >>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0]))) + OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)]) + +The new sorted dictionaries maintain their sort order when entries +are deleted. But when new keys are added, the keys are appended +to the end and the sort is not maintained. + +It is also straight-forward to create an ordered dictionary variant +that the remembers the order the keys were *last* inserted. +If a new entry overwrites an existing entry, the +original insertion position is changed and moved to the end:: + + class LastUpdatedOrderedDict(OrderedDict): + 'Store items in the order the keys were last added' + def __setitem__(self, key, value): + if key in self: + del self[key] + OrderedDict.__setitem__(self, key, value) + + +.. _abstract-base-classes: + +ABCs - abstract base classes +---------------------------- + +The collections module offers the following :term:`ABCs <abstract base class>`: + +========================= ===================== ====================== ==================================================== +ABC Inherits from Abstract Methods Mixin Methods +========================= ===================== ====================== ==================================================== +:class:`Container` ``__contains__`` +:class:`Hashable` ``__hash__`` +:class:`Iterable` ``__iter__`` +:class:`Iterator` :class:`Iterable` ``next`` ``__iter__`` +:class:`Sized` ``__len__`` +:class:`Callable` ``__call__`` + +:class:`Sequence` :class:`Sized`, ``__getitem__`` ``__contains__``. ``__iter__``, ``__reversed__``, + :class:`Iterable`, ``index``, and ``count`` + :class:`Container` + +:class:`MutableSequence` :class:`Sequence` ``__setitem__``, Inherited :class:`Sequence` methods and + ``__delitem__``, ``append``, ``reverse``, ``extend``, ``pop``, + ``insert`` ``remove``, and ``__iadd__`` + +:class:`Set` :class:`Sized`, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``, + :class:`Iterable`, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``, + :class:`Container` ``__sub__``, ``__xor__``, and ``isdisjoint`` + +:class:`MutableSet` :class:`Set` ``add``, Inherited :class:`Set` methods and + ``discard`` ``clear``, ``pop``, ``remove``, ``__ior__``, + ``__iand__``, ``__ixor__``, and ``__isub__`` + +:class:`Mapping` :class:`Sized`, ``__getitem__`` ``__contains__``, ``keys``, ``items``, ``values``, + :class:`Iterable`, ``get``, ``__eq__``, and ``__ne__`` + :class:`Container` + +:class:`MutableMapping` :class:`Mapping` ``__setitem__``, Inherited :class:`Mapping` methods and + ``__delitem__`` ``pop``, ``popitem``, ``clear``, ``update``, + and ``setdefault`` + + +:class:`MappingView` :class:`Sized` ``__len__`` +:class:`ItemsView` :class:`MappingView`, ``__contains__``, + :class:`Set` ``__iter__`` +:class:`KeysView` :class:`MappingView`, ``__contains__``, + :class:`Set` ``__iter__`` +:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__`` +========================= ===================== ====================== ==================================================== + + +.. class:: Container + Hashable + Sized + Callable + + ABCs for classes that provide respectively the methods :meth:`__contains__`, + :meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`. + +.. class:: Iterable + + ABC for classes that provide the :meth:`__iter__` method. + See also the definition of :term:`iterable`. + +.. class:: Iterator + + ABC for classes that provide the :meth:`__iter__` and :meth:`next` methods. + See also the definition of :term:`iterator`. + +.. class:: Sequence + MutableSequence + + ABCs for read-only and mutable :term:`sequences <sequence>`. + +.. class:: Set + MutableSet + + ABCs for read-only and mutable sets. + +.. class:: Mapping + MutableMapping + + ABCs for read-only and mutable :term:`mappings <mapping>`. + +.. class:: MappingView + ItemsView + KeysView + ValuesView + + ABCs for mapping, items, keys, and values :term:`views <view>`. + + +These ABCs allow us to ask classes or instances if they provide +particular functionality, for example:: + + size = None + if isinstance(myvar, collections.Sized): + size = len(myvar) + +Several of the ABCs are also useful as mixins that make it easier to develop +classes supporting container APIs. For example, to write a class supporting +the full :class:`Set` API, it only necessary to supply the three underlying +abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`. +The ABC supplies the remaining methods such as :meth:`__and__` and +:meth:`isdisjoint` :: + + class ListBasedSet(collections.Set): + ''' Alternate set implementation favoring space over speed + and not requiring the set elements to be hashable. ''' + def __init__(self, iterable): + self.elements = lst = [] + for value in iterable: + if value not in lst: + lst.append(value) + def __iter__(self): + return iter(self.elements) + def __contains__(self, value): + return value in self.elements + def __len__(self): + return len(self.elements) + + s1 = ListBasedSet('abcdef') + s2 = ListBasedSet('defghi') + overlap = s1 & s2 # The __and__() method is supported automatically + +Notes on using :class:`Set` and :class:`MutableSet` as a mixin: + +(1) + Since some set operations create new sets, the default mixin methods need + a way to create new instances from an iterable. The class constructor is + assumed to have a signature in the form ``ClassName(iterable)``. + That assumption is factored-out to an internal classmethod called + :meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set. + If the :class:`Set` mixin is being used in a class with a different + constructor signature, you will need to override :meth:`_from_iterable` + with a classmethod that can construct new instances from + an iterable argument. + +(2) + To override the comparisons (presumably for speed, as the + semantics are fixed), redefine :meth:`__le__` and + then the other operations will automatically follow suit. + +(3) + The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value + for the set; however, :meth:`__hash__` is not defined because not all sets + are hashable or immutable. To add set hashabilty using mixins, + inherit from both :meth:`Set` and :meth:`Hashable`, then define + ``__hash__ = Set._hash``. + +.. seealso:: + + * Latest version of the `Python source code for the collections abstract base classes + <http://svn.python.org/view/python/branches/release27-maint/Lib/_abcoll.py?view=markup>`_ + + * `OrderedSet recipe <http://code.activestate.com/recipes/576694/>`_ for an + example built on :class:`MutableSet`. + + * For more about ABCs, see the :mod:`abc` module and :pep:`3119`. diff --git a/Doc/library/commands.rst b/Doc/library/commands.rst index 0b347f7..916e4a5 100644 --- a/Doc/library/commands.rst +++ b/Doc/library/commands.rst @@ -5,6 +5,12 @@ .. module:: commands :platform: Unix :synopsis: Utility functions for running external commands. + :deprecated: + +.. deprecated:: 2.6 + The :mod:`commands` module has been removed in Python 3.0. Use the + :mod:`subprocess` module instead. + .. sectionauthor:: Sue Williams <sbw@provis.com> diff --git a/Doc/library/compileall.rst b/Doc/library/compileall.rst index 86f60fb..cf0d5f8 100644 --- a/Doc/library/compileall.rst +++ b/Doc/library/compileall.rst @@ -1,4 +1,3 @@ - :mod:`compileall` --- Byte-compile Python libraries =================================================== @@ -7,41 +6,117 @@ This module provides some utility functions to support installing Python -libraries. These functions compile Python source files in a directory tree, -allowing users without permission to write to the libraries to take advantage of -cached byte-code files. +libraries. These functions compile Python source files in a directory tree. +This module can be used to create the cached byte-code files at library +installation time, which makes them available for use even by users who don't +have write permission to the library directories. + + +Command-line use +---------------- + +This module can work as a script (using :program:`python -m compileall`) to +compile Python sources. + +.. program:: compileall + +.. cmdoption:: [directory|file]... + + Positional arguments are files to compile or directories that contain + source files, traversed recursively. If no argument is given, behave as if + the command line was ``-l <directories from sys.path>``. + +.. cmdoption:: -l + + Do not recurse into subdirectories, only compile source code files directly + contained in the named or implied directories. + +.. cmdoption:: -f + + Force rebuild even if timestamps are up-to-date. + +.. cmdoption:: -q + + Do not print the list of files compiled, print only error messages. + +.. cmdoption:: -d destdir + + Directory prepended to the path to each file being compiled. This will + appear in compilation time tracebacks, and is also compiled in to the + byte-code file, where it will be used in tracebacks and other messages in + cases where the source file does not exist at the time the byte-code file is + executed. + +.. cmdoption:: -x regex -This module may also be used as a script (using the :option:`-m` Python flag) to -compile Python sources. Directories to recursively traverse (passing -:option:`-l` stops the recursive behavior) for sources are listed on the command -line. If no arguments are given, the invocation is equivalent to ``-l -sys.path``. Printing lists of the files compiled can be disabled with the -:option:`-q` flag. In addition, the :option:`-x` option takes a regular -expression argument. All files that match the expression will be skipped. + regex is used to search the full path to each file considered for + compilation, and if the regex produces a match, the file is skipped. +.. cmdoption:: -i list -.. function:: compile_dir(dir[, maxlevels[, ddir[, force[, rx[, quiet]]]]]) + Read the file ``list`` and add each line that it contains to the list of + files and directories to compile. If ``list`` is ``-``, read lines from + ``stdin``. + +.. versionchanged:: 2.7 + Added the ``-i`` option. + + +Public functions +---------------- + +.. function:: compile_dir(dir[, maxlevels[, ddir[, force[, rx[, quiet]]]]]) Recursively descend the directory tree named by *dir*, compiling all :file:`.py` - files along the way. The *maxlevels* parameter is used to limit the depth of - the recursion; it defaults to ``10``. If *ddir* is given, it is used as the - base path from which the filenames used in error messages will be generated. + files along the way. + + The *maxlevels* parameter is used to limit the depth of the recursion; it + defaults to ``10``. + + If *ddir* is given, it is prepended to the path to each file being compiled + for use in compilation time tracebacks, and is also compiled in to the + byte-code file, where it will be used in tracebacks and other messages in + cases where the source file does not exist at the time the byte-code file is + executed. + If *force* is true, modules are re-compiled even if the timestamps are up to date. - If *rx* is given, it specifies a regular expression of file names to exclude - from the search; that expression is searched for in the full path. + If *rx* is given, its search method is called on the complete path to each + file considered for compilation, and if it returns a true value, the file + is skipped. + + If *quiet* is true, nothing is printed to the standard output unless errors + occur. - If *quiet* is true, nothing is printed to the standard output in normal - operation. + +.. function:: compile_file(fullname[, ddir[, force[, rx[, quiet]]]]) + + Compile the file with path *fullname*. + + If *ddir* is given, it is prepended to the path to the file being compiled + for use in compilation time tracebacks, and is also compiled in to the + byte-code file, where it will be used in tracebacks and other messages in + cases where the source file does not exist at the time the byte-code file is + executed. + + If *rx* is given, its search method is passed the full path name to the + file being compiled, and if it returns a true value, the file is not + compiled and ``True`` is returned. + + If *quiet* is true, nothing is printed to the standard output unless errors + occur. + + .. versionadded:: 2.7 .. function:: compile_path([skip_curdir[, maxlevels[, force]]]) Byte-compile all the :file:`.py` files found along ``sys.path``. If - *skip_curdir* is true (the default), the current directory is not included in - the search. The *maxlevels* and *force* parameters default to ``0`` and are - passed to the :func:`compile_dir` function. + *skip_curdir* is true (the default), the current directory is not included + in the search. All other parameters are passed to the :func:`compile_dir` + function. Note that unlike the other compile functions, ``maxlevels`` + defaults to ``0``. To force a recompile of all the :file:`.py` files in the :file:`Lib/` subdirectory and all its subdirectories:: @@ -59,4 +134,3 @@ subdirectory and all its subdirectories:: Module :mod:`py_compile` Byte-compile a single source file. - diff --git a/Doc/library/compiler.rst b/Doc/library/compiler.rst index 991628a..458e653 100644 --- a/Doc/library/compiler.rst +++ b/Doc/library/compiler.rst @@ -18,7 +18,7 @@ abstract syntax tree from Python source code and to generate Python The :mod:`compiler` package is a Python source to bytecode translator written in Python. It uses the built-in parser and standard :mod:`parser` module to -generated a concrete syntax tree. This tree is used to generate an abstract +generate a concrete syntax tree. This tree is used to generate an abstract syntax tree (AST) and then Python bytecode. The full functionality of the package duplicates the built-in compiler provided diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst index 867f091..4b1e7a0 100644 --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -43,7 +43,7 @@ provided on initialization and retrieval. Lines beginning with ``'#'`` or Configuration files may include comments, prefixed by specific characters (``#`` and ``;``). Comments may appear on their own in an otherwise empty line, or may -be entered in lines holding values or spection names. In the latter case, they +be entered in lines holding values or section names. In the latter case, they need to be preceded by a whitespace character to be recognized as a comment. (For backwards compatibility, only ``;`` starts an inline comment, while ``#`` does not.) @@ -74,12 +74,16 @@ dictionary type is passed that sorts its keys, the sections will be sorted on write-back, as will be the keys within each section. -.. class:: RawConfigParser([defaults[, dict_type]]) +.. class:: RawConfigParser([defaults[, dict_type[, allow_no_value]]]) The basic configuration object. When *defaults* is given, it is initialized into the dictionary of intrinsic defaults. When *dict_type* is given, it will be used to create the dictionary objects for the list of sections, for the - options within a section, and for the default values. This class does not + options within a section, and for the default values. When *allow_no_value* + is true (default: ``False``), options without values are accepted; the value + presented for these is ``None``. + + This class does not support the magical interpolation behavior. .. versionadded:: 2.3 @@ -87,8 +91,12 @@ write-back, as will be the keys within each section. .. versionchanged:: 2.6 *dict_type* was added. + .. versionchanged:: 2.7 + The default *dict_type* is :class:`collections.OrderedDict`. + *allow_no_value* was added. + -.. class:: ConfigParser([defaults[, dict_type]]) +.. class:: ConfigParser([defaults[, dict_type[, allow_no_value]]]) Derived class of :class:`RawConfigParser` that implements the magical interpolation feature and adds optional arguments to the :meth:`get` and @@ -103,8 +111,17 @@ write-back, as will be the keys within each section. option names to lower case), the values ``foo %(bar)s`` and ``foo %(BAR)s`` are equivalent. + .. versionadded:: 2.3 + + .. versionchanged:: 2.6 + *dict_type* was added. + + .. versionchanged:: 2.7 + The default *dict_type* is :class:`collections.OrderedDict`. + *allow_no_value* was added. -.. class:: SafeConfigParser([defaults[, dict_type]]) + +.. class:: SafeConfigParser([defaults[, dict_type[, allow_no_value]]]) Derived class of :class:`ConfigParser` that implements a more-sane variant of the magical interpolation feature. This implementation is more predictable as @@ -115,6 +132,13 @@ write-back, as will be the keys within each section. .. versionadded:: 2.3 + .. versionchanged:: 2.6 + *dict_type* was added. + + .. versionchanged:: 2.7 + The default *dict_type* is :class:`collections.OrderedDict`. + *allow_no_value* was added. + .. exception:: Error @@ -486,3 +510,38 @@ The function ``opt_move`` below can be used to move options between sections:: opt_move(config, section1, section2, option) else: config.remove_option(section1, option) + +Some configuration files are known to include settings without values, but which +otherwise conform to the syntax supported by :mod:`ConfigParser`. The +*allow_no_value* parameter to the constructor can be used to indicate that such +values should be accepted: + +.. doctest:: + + >>> import ConfigParser + >>> import io + + >>> sample_config = """ + ... [mysqld] + ... user = mysql + ... pid-file = /var/run/mysqld/mysqld.pid + ... skip-external-locking + ... old_passwords = 1 + ... skip-bdb + ... skip-innodb + ... """ + >>> config = ConfigParser.RawConfigParser(allow_no_value=True) + >>> config.readfp(io.BytesIO(sample_config)) + + >>> # Settings with values are treated as before: + >>> config.get("mysqld", "user") + 'mysql' + + >>> # Settings without values provide None: + >>> config.get("mysqld", "skip-bdb") + + >>> # Settings which aren't specified still raise an error: + >>> config.get("mysqld", "does-not-exist") + Traceback (most recent call last): + ... + ConfigParser.NoOptionError: No option 'does-not-exist' in section: 'mysqld' diff --git a/Doc/library/constants.rst b/Doc/library/constants.rst index e6d927a..80e792c 100644 --- a/Doc/library/constants.rst +++ b/Doc/library/constants.rst @@ -42,14 +42,17 @@ A small number of constants live in the built-in namespace. They are: .. data:: __debug__ This constant is true if Python was not started with an :option:`-O` option. - It cannot be reassigned. See also the :keyword:`assert` statement. + See also the :keyword:`assert` statement. .. note:: - The name :data:`None` cannot be reassigned (assignments to it, even as an - attribute name, raise :exc:`SyntaxError`), so it can be considered a "true" - constant. + The names :data:`None` and :data:`__debug__` cannot be reassigned + (assignments to them, even as an attribute name, raise :exc:`SyntaxError`), + so they can be considered "true" constants. + + .. versionchanged:: 2.7 + Assignments to ``__debug__`` as an attribute became illegal. Constants added by the :mod:`site` module diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index b5e9aa8..c435082 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -11,6 +11,11 @@ This module provides utilities for common tasks involving the :keyword:`with` statement. For more information see also :ref:`typecontextmanager` and :ref:`context-managers`. +.. seealso:: + + Latest version of the `contextlib Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/contextlib.py?view=markup>`_ + Functions provided: @@ -58,21 +63,18 @@ Functions provided: Combine multiple context managers into a single nested context manager. - Code like this:: + This function has been deprecated in favour of the multiple manager form + of the :keyword:`with` statement. + + The one advantage of this function over the multiple manager form of the + :keyword:`with` statement is that argument unpacking allows it to be + used with a variable number of context managers as follows:: from contextlib import nested - with nested(A(), B(), C()) as (X, Y, Z): + with nested(*managers): do_something() - is equivalent to this:: - - m1, m2, m3 = A(), B(), C() - with m1 as X: - with m2 as Y: - with m3 as Z: - do_something() - Note that if the :meth:`__exit__` method of one of the nested context managers indicates an exception should be suppressed, no exception information will be passed to any remaining outer context managers. Similarly, if the @@ -82,6 +84,28 @@ Functions provided: :meth:`__exit__` methods should avoid raising exceptions, and in particular they should not re-raise a passed-in exception. + This function has two major quirks that have led to it being deprecated. Firstly, + as the context managers are all constructed before the function is invoked, the + :meth:`__new__` and :meth:`__init__` methods of the inner context managers are + not actually covered by the scope of the outer context managers. That means, for + example, that using :func:`nested` to open two files is a programming error as the + first file will not be closed promptly if an exception is thrown when opening + the second file. + + Secondly, if the :meth:`__enter__` method of one of the inner context managers + raises an exception that is caught and suppressed by the :meth:`__exit__` method + of one of the outer context managers, this construct will raise + :exc:`RuntimeError` rather than skipping the body of the :keyword:`with` + statement. + + Developers that need to support nesting of a variable number of context managers + can either use the :mod:`warnings` module to suppress the DeprecationWarning + raised by this function or else use this function as a model for an application + specific implementation. + + .. deprecated:: 2.7 + The with-statement now supports this functionality directly (without the + confusing error prone quirks). .. function:: closing(thing) diff --git a/Doc/library/cookie.rst b/Doc/library/cookie.rst index b6a85f5..3bdc61a 100644 --- a/Doc/library/cookie.rst +++ b/Doc/library/cookie.rst @@ -235,8 +235,6 @@ The following example demonstrates how to use the :mod:`Cookie` module. >>> import Cookie >>> C = Cookie.SimpleCookie() - >>> C = Cookie.SerialCookie() - >>> C = Cookie.SmartCookie() >>> C["fig"] = "newton" >>> C["sugar"] = "wafer" >>> print C # generate HTTP headers @@ -245,28 +243,27 @@ The following example demonstrates how to use the :mod:`Cookie` module. >>> print C.output() # same thing Set-Cookie: fig=newton Set-Cookie: sugar=wafer - >>> C = Cookie.SmartCookie() + >>> C = Cookie.SimpleCookie() >>> C["rocky"] = "road" >>> C["rocky"]["path"] = "/cookie" >>> print C.output(header="Cookie:") Cookie: rocky=road; Path=/cookie >>> print C.output(attrs=[], header="Cookie:") Cookie: rocky=road - >>> C = Cookie.SmartCookie() + >>> C = Cookie.SimpleCookie() >>> C.load("chips=ahoy; vienna=finger") # load from a string (HTTP header) >>> print C Set-Cookie: chips=ahoy Set-Cookie: vienna=finger - >>> C = Cookie.SmartCookie() + >>> C = Cookie.SimpleCookie() >>> C.load('keebler="E=everybody; L=\\"Loves\\"; fudge=\\012;";') >>> print C Set-Cookie: keebler="E=everybody; L=\"Loves\"; fudge=\012;" - >>> C = Cookie.SmartCookie() + >>> C = Cookie.SimpleCookie() >>> C["oreo"] = "doublestuff" >>> C["oreo"]["path"] = "/" >>> print C Set-Cookie: oreo=doublestuff; Path=/ - >>> C = Cookie.SmartCookie() >>> C["twix"] = "none for you" >>> C["twix"].value 'none for you' @@ -280,6 +277,8 @@ The following example demonstrates how to use the :mod:`Cookie` module. >>> print C Set-Cookie: number=7 Set-Cookie: string=seven + >>> # SerialCookie and SmartCookie are deprecated + >>> # using it can cause security loopholes in your code. >>> C = Cookie.SerialCookie() >>> C["number"] = 7 >>> C["string"] = "seven" diff --git a/Doc/library/cookielib.rst b/Doc/library/cookielib.rst index 47211c9..4ca0471 100644 --- a/Doc/library/cookielib.rst +++ b/Doc/library/cookielib.rst @@ -750,7 +750,7 @@ cookies (assumes Unix/Netscape convention for location of the cookies file):: import os, cookielib, urllib2 cj = cookielib.MozillaCookieJar() - cj.load(os.path.join(os.environ["HOME"], ".netscape/cookies.txt")) + cj.load(os.path.join(os.path.expanduser("~"), ".netscape", "cookies.txt")) opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) r = opener.open("http://example.com/") diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index c399bb7..f4e9d7c 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -424,6 +424,16 @@ Writer objects have the following public attribute: A read-only description of the dialect in use by the writer. +DictWriter objects have the following public method: + + +.. method:: DictWriter.writeheader() + + Write a row with the field names (as specified in the constructor). + + .. versionadded:: 2.7 + + .. _csv-examples: Examples @@ -432,41 +442,44 @@ Examples The simplest example of reading a CSV file:: import csv - reader = csv.reader(open("some.csv", "rb")) - for row in reader: - print row + with open('some.csv', 'rb') as f: + reader = csv.reader(f) + for row in reader: + print row Reading a file with an alternate format:: import csv - reader = csv.reader(open("passwd", "rb"), delimiter=':', quoting=csv.QUOTE_NONE) - for row in reader: - print row + with open('passwd', 'rb') as f: + reader = csv.reader(f, delimiter=':', quoting=csv.QUOTE_NONE) + for row in reader: + print row The corresponding simplest possible writing example is:: import csv - writer = csv.writer(open("some.csv", "wb")) - writer.writerows(someiterable) + with open('some.csv', 'wb') as f: + writer = csv.writer(f) + writer.writerows(someiterable) Registering a new dialect:: import csv - csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE) - - reader = csv.reader(open("passwd", "rb"), 'unixpwd') + with open('passwd', 'rb') as f: + reader = csv.reader(f, 'unixpwd') A slightly more advanced use of the reader --- catching and reporting errors:: import csv, sys - filename = "some.csv" - reader = csv.reader(open(filename, "rb")) - try: - for row in reader: - print row - except csv.Error, e: - sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e)) + filename = 'some.csv' + with open(filename, 'rb') as f: + reader = csv.reader(f) + try: + for row in reader: + print row + except csv.Error, e: + sys.exit('file %s, line %d: %s' % (filename, reader.line_num, e)) And while the module doesn't directly support parsing strings, it can easily be done:: diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 582300c..f43e166 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -220,6 +220,8 @@ Fundamental data types +----------------------+----------------------------------------+----------------------------+ | ctypes type | C type | Python type | +======================+========================================+============================+ +| :class:`c_bool` | :ctype:`_Bool` | bool (1) | ++----------------------+----------------------------------------+----------------------------+ | :class:`c_char` | :ctype:`char` | 1-character string | +----------------------+----------------------------------------+----------------------------+ | :class:`c_wchar` | :ctype:`wchar_t` | 1-character unicode string | @@ -258,6 +260,9 @@ Fundamental data types | :class:`c_void_p` | :ctype:`void *` | int/long or ``None`` | +----------------------+----------------------------------------+----------------------------+ +(1) + The constructor accepts any object with a truth value. + All these types can be created by calling them with an optional initializer of the correct type and value:: @@ -2248,6 +2253,13 @@ These are the fundamental ctypes data types: Represents the C :ctype:`size_t` datatype. +.. class:: c_ssize_t + + Represents the C :ctype:`ssize_t` datatype. + + .. versionadded:: 2.7 + + .. class:: c_ubyte Represents the C :ctype:`unsigned char` datatype, it interprets the value as diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index a2519f4..63691db 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -1176,6 +1176,9 @@ Several constants are available to specify character cell attributes: +------------------+-------------------------------+ | ``A_NORMAL`` | Normal attribute. | +------------------+-------------------------------+ +| ``A_REVERSE`` | Reverse background and | +| | foreground colors. | ++------------------+-------------------------------+ | ``A_STANDOUT`` | Standout mode. | +------------------+-------------------------------+ | ``A_UNDERLINE`` | Underline mode. | diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 15a6616..24229bb 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -284,12 +284,28 @@ comparison is ``==`` or ``!=``. The latter cases return :const:`False` or efficient pickling, and in Boolean contexts, a :class:`timedelta` object is considered to be true if and only if it isn't equal to ``timedelta(0)``. +Instance methods: + +.. method:: timedelta.total_seconds() + + Return the total number of seconds contained in the duration. + Equivalent to ``(td.microseconds + (td.seconds + td.days * 24 * + 3600) * 10**6) / 10**6`` computed with true division enabled. + + Note that for very large time intervals (greater than 270 years on + most platforms) this method will lose microsecond accuracy. + + .. versionadded:: 2.7 + + Example usage: >>> from datetime import timedelta >>> year = timedelta(days=365) >>> another_year = timedelta(weeks=40, days=84, hours=23, ... minutes=50, seconds=600) # adds up to 365 days + >>> year.total_seconds() + 31536000.0 >>> year == another_year True >>> ten_years = 10 * year @@ -457,7 +473,9 @@ Instance methods: Return a :class:`time.struct_time` such as returned by :func:`time.localtime`. The hours, minutes and seconds are 0, and the DST flag is -1. ``d.timetuple()`` is equivalent to ``time.struct_time((d.year, d.month, d.day, 0, 0, 0, - d.weekday(), d.toordinal() - date(d.year, 1, 1).toordinal() + 1, -1))`` + d.weekday(), yday, -1))``, where ``yday = d.toordinal() - date(d.year, 1, + 1).toordinal() + 1`` is the day number within the current year starting with + ``1`` for January 1st. .. method:: date.toordinal() @@ -922,12 +940,13 @@ Instance methods: Return a :class:`time.struct_time` such as returned by :func:`time.localtime`. ``d.timetuple()`` is equivalent to ``time.struct_time((d.year, d.month, d.day, - d.hour, d.minute, d.second, d.weekday(), d.toordinal() - date(d.year, 1, - 1).toordinal() + 1, dst))`` The :attr:`tm_isdst` flag of the result is set - according to the :meth:`dst` method: :attr:`tzinfo` is ``None`` or :meth:`dst` - returns ``None``, :attr:`tm_isdst` is set to ``-1``; else if :meth:`dst` - returns a non-zero value, :attr:`tm_isdst` is set to ``1``; else ``tm_isdst`` is - set to ``0``. + d.hour, d.minute, d.second, d.weekday(), yday, dst))``, where ``yday = + d.toordinal() - date(d.year, 1, 1).toordinal() + 1`` is the day number within + the current year starting with ``1`` for January 1st. The :attr:`tm_isdst` flag + of the result is set according to the :meth:`dst` method: :attr:`tzinfo` is + ``None`` or :meth:`dst` returns ``None``, :attr:`tm_isdst` is set to ``-1``; + else if :meth:`dst` returns a non-zero value, :attr:`tm_isdst` is set to ``1``; + else :attr:`tm_isdst` is set to ``0``. .. method:: datetime.utctimetuple() diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index ad44bda..357882f 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -35,9 +35,9 @@ arithmetic. It offers several advantages over the :class:`float` datatype: people learn at school." -- excerpt from the decimal arithmetic specification. * Decimal numbers can be represented exactly. In contrast, numbers like - :const:`1.1` do not have an exact representation in binary floating point. End - users typically would not expect :const:`1.1` to display as - :const:`1.1000000000000001` as it does with binary floating point. + :const:`1.1` and :const:`2.2` do not have an exact representations in binary + floating point. End users typically would not expect ``1.1 + 2.2`` to display + as :const:`3.3000000000000003` as it does with binary floating point. * The exactness carries over into arithmetic. In decimal floating point, ``0.1 + 0.1 + 0.1 - 0.3`` is exactly equal to zero. In binary floating point, the result @@ -57,6 +57,7 @@ arithmetic. It offers several advantages over the :class:`float` datatype: alterable precision (defaulting to 28 places) which can be as large as needed for a given problem: + >>> from decimal import * >>> getcontext().prec = 6 >>> Decimal(1) / Decimal(7) Decimal('0.142857') @@ -133,10 +134,9 @@ precision, rounding, or enabled traps:: >>> getcontext().prec = 7 # Set a new precision -Decimal instances can be constructed from integers, strings, or tuples. To -create a Decimal from a :class:`float`, first convert it to a string. This -serves as an explicit reminder of the details of the conversion (including -representation error). Decimal numbers include special values such as +Decimal instances can be constructed from integers, strings, floats, or tuples. +Construction from an integer or a float performs an exact conversion of the +value of that integer or float. Decimal numbers include special values such as :const:`NaN` which stands for "Not a number", positive and negative :const:`Infinity`, and :const:`-0`. @@ -145,6 +145,8 @@ representation error). Decimal numbers include special values such as Decimal('10') >>> Decimal('3.14') Decimal('3.14') + >>> Decimal(3.14) + Decimal('3.140000000000000124344978758017532527446746826171875') >>> Decimal((0, (3, 1, 4), -2)) Decimal('3.14') >>> Decimal(str(2.0 ** 0.5)) @@ -193,7 +195,7 @@ floating point flying circus: >>> str(a) '1.34' >>> float(a) - 1.3400000000000001 + 1.34 >>> round(a, 1) # round() first converts to binary floating point 1.3 >>> int(a) @@ -314,7 +316,7 @@ Decimal objects Construct a new :class:`Decimal` object based from *value*. - *value* can be an integer, string, tuple, or another :class:`Decimal` + *value* can be an integer, string, tuple, :class:`float`, or another :class:`Decimal` object. If no *value* is given, returns ``Decimal('0')``. If *value* is a string, it should conform to the decimal numeric string syntax after leading and trailing whitespace characters are removed:: @@ -341,6 +343,12 @@ Decimal objects digits, and an integer exponent. For example, ``Decimal((0, (1, 4, 1, 4), -3))`` returns ``Decimal('1.414')``. + If *value* is a :class:`float`, the binary floating point value is losslessly + converted to its exact decimal equivalent. This conversion can often require + 53 or more digits of precision. For example, ``Decimal(float('1.1'))`` + converts to + ``Decimal('1.100000000000000088817841970012523233890533447265625')``. + The *context* precision does not affect how many digits are stored. That is determined exclusively by the number of digits in *value*. For example, ``Decimal('3.00000')`` records all five zeros even if the context precision is @@ -357,6 +365,9 @@ Decimal objects leading and trailing whitespace characters are permitted when creating a Decimal instance from a string. + .. versionchanged:: 2.7 + The argument to the constructor is now permitted to be a :class:`float` instance. + Decimal floating point objects share many properties with the other built-in numeric types such as :class:`float` and :class:`int`. All of the usual math operations and special methods apply. Likewise, decimal objects can be @@ -364,6 +375,24 @@ Decimal objects compared, sorted, and coerced to another type (such as :class:`float` or :class:`long`). + Decimal objects cannot generally be combined with floats in + arithmetic operations: an attempt to add a :class:`Decimal` to a + :class:`float`, for example, will raise a :exc:`TypeError`. + There's one exception to this rule: it's possible to use Python's + comparison operators to compare a :class:`float` instance ``x`` + with a :class:`Decimal` instance ``y``. Without this exception, + comparisons between :class:`Decimal` and :class:`float` instances + would follow the general rules for comparing objects of different + types described in the :ref:`expressions` section of the reference + manual, leading to confusing results. + + .. versionchanged:: 2.7 + A comparison between a :class:`float` instance ``x`` and a + :class:`Decimal` instance ``y`` now returns a result based on + the values of ``x`` and ``y``. In earlier versions ``x < y`` + returned the same (arbitrary) result for any :class:`Decimal` + instance ``x`` and any :class:`float` instance ``y``. + In addition to the standard numeric properties, decimal floating point objects also have a number of specialized methods: @@ -490,6 +519,32 @@ Decimal objects .. versionadded:: 2.6 + .. method:: from_float(f) + + Classmethod that converts a float to a decimal number, exactly. + + Note `Decimal.from_float(0.1)` is not the same as `Decimal('0.1')`. + Since 0.1 is not exactly representable in binary floating point, the + value is stored as the nearest representable value which is + `0x1.999999999999ap-4`. That equivalent value in decimal is + `0.1000000000000000055511151231257827021181583404541015625`. + + .. note:: From Python 2.7 onwards, a :class:`Decimal` instance + can also be constructed directly from a :class:`float`. + + .. doctest:: + + >>> Decimal.from_float(0.1) + Decimal('0.1000000000000000055511151231257827021181583404541015625') + >>> Decimal.from_float(float('nan')) + Decimal('NaN') + >>> Decimal.from_float(float('inf')) + Decimal('Infinity') + >>> Decimal.from_float(float('-inf')) + Decimal('-Infinity') + + .. versionadded:: 2.7 + .. method:: fma(other, third[, context]) Fused multiply-add. Return self*other+third with no rounding of the @@ -976,8 +1031,11 @@ In addition to the three supplied contexts, new contexts can be created with the a large number of methods for doing arithmetic directly in a given context. In addition, for each of the :class:`Decimal` methods described above (with the exception of the :meth:`adjusted` and :meth:`as_tuple` methods) there is - a corresponding :class:`Context` method. For example, ``C.exp(x)`` is - equivalent to ``x.exp(context=C)``. + a corresponding :class:`Context` method. For example, for a :class:`Context` + instance ``C`` and :class:`Decimal` instance ``x``, ``C.exp(x)`` is + equivalent to ``x.exp(context=C)``. Each :class:`Context` method accepts a + Python integer (an instance of :class:`int` or :class:`long`) anywhere that a + Decimal instance is accepted. .. method:: clear_flags() @@ -1016,6 +1074,26 @@ In addition to the three supplied contexts, new contexts can be created with the If the argument is a string, no leading or trailing whitespace is permitted. + .. method:: create_decimal_from_float(f) + + Creates a new Decimal instance from a float *f* but rounding using *self* + as the context. Unlike the :meth:`Decimal.from_float` class method, + the context precision, rounding method, flags, and traps are applied to + the conversion. + + .. doctest:: + + >>> context = Context(prec=5, rounding=ROUND_DOWN) + >>> context.create_decimal_from_float(math.pi) + Decimal('3.1415') + >>> context = Context(prec=5, traps=[Inexact]) + >>> context.create_decimal_from_float(math.pi) + Traceback (most recent call last): + ... + Inexact: None + + .. versionadded:: 2.7 + .. method:: Etiny() Returns a value equal to ``Emin - prec + 1`` which is the minimum exponent @@ -1872,49 +1950,29 @@ of significant places in the coefficient. For example, expressing original's two-place significance. If an application does not care about tracking significance, it is easy to -remove the exponent and trailing zeroes, losing significance, but keeping the -value unchanged: - - >>> def remove_exponent(d): - ... return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize() +remove the exponent and trailing zeros, losing significance, but keeping the +value unchanged:: - >>> remove_exponent(Decimal('5E+3')) - Decimal('5000') + def remove_exponent(d): + '''Remove exponent and trailing zeros. -Q. Is there a way to convert a regular float to a :class:`Decimal`? + >>> remove_exponent(Decimal('5E+3')) + Decimal('5000') -A. Yes, all binary floating point numbers can be exactly expressed as a -Decimal. An exact conversion may take more precision than intuition would -suggest, so we trap :const:`Inexact` to signal a need for more precision: + ''' + return d.quantize(Decimal(1)) if d == d.to_integral() else d.normalize() -.. testcode:: +Q. Is there a way to convert a regular float to a :class:`Decimal`? - def float_to_decimal(f): - "Convert a floating point number to a Decimal with no loss of information" - n, d = f.as_integer_ratio() - numerator, denominator = Decimal(n), Decimal(d) - ctx = Context(prec=60) - result = ctx.divide(numerator, denominator) - while ctx.flags[Inexact]: - ctx.flags[Inexact] = False - ctx.prec *= 2 - result = ctx.divide(numerator, denominator) - return result +A. Yes, any binary floating point number can be exactly expressed as a +Decimal though an exact conversion may take more precision than intuition would +suggest: .. doctest:: - >>> float_to_decimal(math.pi) + >>> Decimal(math.pi) Decimal('3.141592653589793115997963468544185161590576171875') -Q. Why isn't the :func:`float_to_decimal` routine included in the module? - -A. There is some question about whether it is advisable to mix binary and -decimal floating point. Also, its use requires some care to avoid the -representation issues associated with binary floating point: - - >>> float_to_decimal(1.1) - Decimal('1.100000000000000088817841970012523233890533447265625') - Q. Within a complex calculation, how can I make sure that I haven't gotten a spurious result because of insufficient precision or rounding anomalies. diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst index 9cd76e3..225b486 100644 --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -37,11 +37,16 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. complicated way on how many elements the sequences have in common; best case time is linear. - **Heuristic:** To speed-up matching, items that appear more than 1% of the - time in sequences of at least 200 items are treated as junk. This has the - unfortunate side-effect of giving bad results for sequences constructed from - a small set of items. An option to turn off the heuristic will be added to a - future version. + **Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that + automatically treats certain sequence items as junk. The heuristic counts how many + times each individual item appears in the sequence. If an item's duplicates (after + the first one) account for more than 1% of the sequence and the sequence is at least + 200 items long, this item is marked as "popular" and is treated as junk for + the purpose of sequence matching. This heuristic can be turned off by setting + the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`. + + .. versionadded:: 2.7.1 + The *autojunk* parameter. .. class:: Differ @@ -151,8 +156,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] @@ -286,8 +291,8 @@ diffs. For comparing directories and files, see also, the :mod:`filecmp` module. The context diff format normally has a header for filenames and modification times. Any or all of these may be specified using strings for *fromfile*, - *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally - expressed in the format returned by :func:`time.ctime`. If not specified, the + *tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally + expressed in the ISO 8601 format. If not specified, the strings default to blanks. >>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n'] @@ -339,7 +344,7 @@ SequenceMatcher Objects The :class:`SequenceMatcher` class has this constructor: -.. class:: SequenceMatcher([isjunk[, a[, b]]]) +.. class:: SequenceMatcher([isjunk[, a[, b[, autojunk=True]]]]) Optional argument *isjunk* must be ``None`` (the default) or a one-argument function that takes a sequence element and returns true if and only if the @@ -355,8 +360,13 @@ The :class:`SequenceMatcher` class has this constructor: The optional arguments *a* and *b* are sequences to be compared; both default to empty strings. The elements of both sequences must be :term:`hashable`. - :class:`SequenceMatcher` objects have the following methods: + The optional argument *autojunk* can be used to disable the automatic junk + heuristic. + .. versionadded:: 2.7.1 + The *autojunk* parameter. + + :class:`SequenceMatcher` objects have the following methods: .. method:: set_seqs(a, b) @@ -517,16 +527,11 @@ The :class:`SequenceMatcher` class has this constructor: Return an upper bound on :meth:`ratio` relatively quickly. - This isn't defined beyond that it is an upper bound on :meth:`ratio`, and - is faster to compute. - .. method:: real_quick_ratio() Return an upper bound on :meth:`ratio` very quickly. - This isn't defined beyond that it is an upper bound on :meth:`ratio`, and - is faster to compute than either :meth:`ratio` or :meth:`quick_ratio`. The three methods that return the ratio of matching to total characters can give different results due to differing levels of approximation, although diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index e3f30cb..9addfe7 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -11,6 +11,11 @@ disassembling it. The CPython bytecode which this module takes as an input is defined in the file :file:`Include/opcode.h` and used by the compiler and the interpreter. +.. seealso:: + + Latest version of the `dis module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/dis.py?view=markup>`_ + .. impl-detail:: Bytecode is an implementation detail of the CPython interpreter! No @@ -70,8 +75,21 @@ The :mod:`dis` module defines the following functions and constants: .. function:: disco(code[, lasti]) - A synonym for disassemble. It is more convenient to type, and kept for - compatibility with earlier Python releases. + A synonym for :func:`disassemble`. It is more convenient to type, and kept + for compatibility with earlier Python releases. + + +.. function:: findlinestarts(code) + + This generator function uses the ``co_firstlineno`` and ``co_lnotab`` + attributes of the code object *code* to find the offsets which are starts of + lines in the source code. They are generated as ``(offset, lineno)`` pairs. + + +.. function:: findlabels(code) + + Detect all offsets in the code object *code* which are jump targets, and + return a list of these offsets. .. data:: opname @@ -81,7 +99,7 @@ The :mod:`dis` module defines the following functions and constants: .. data:: opmap - Dictionary mapping bytecodes to operation names. + Dictionary mapping operation names to bytecodes. .. data:: cmp_op @@ -469,9 +487,11 @@ Miscellaneous opcodes. address to jump to (which should be a ``FOR_ITER`` instruction). -.. opcode:: LIST_APPEND () +.. opcode:: LIST_APPEND (i) - Calls ``list.append(TOS1, TOS)``. Used to implement list comprehensions. + Calls ``list.append(TOS[-i], TOS)``. Used to implement list comprehensions. + While the appended value is popped off, the list object remains on the + stack so that it is available for further iterations of the loop. .. opcode:: LOAD_LOCALS () @@ -523,6 +543,18 @@ Miscellaneous opcodes. the names of the base classes, and TOS2 the class name. +.. opcode:: SETUP_WITH (delta) + + This opcode performs several operations before a with block starts. First, + it loads :meth:`~object.__exit__` from the context manager and pushes it onto + the stack for later use by :opcode:`WITH_CLEANUP`. Then, + :meth:`~object.__enter__` is called, and a finally block pointing to *delta* + is pushed. Finally, the result of calling the enter method is pushed onto + the stack. The next opcode will either ignore it (:opcode:`POP_TOP`), or + store it in (a) variable(s) (:opcode:`STORE_FAST`, :opcode:`STORE_NAME`, or + :opcode:`UNPACK_SEQUENCE`). + + .. opcode:: WITH_CLEANUP () Cleans up the stack when a :keyword:`with` statement block exits. On top of @@ -655,16 +687,26 @@ the more significant byte last. Increments bytecode counter by *delta*. -.. opcode:: JUMP_IF_TRUE (delta) +.. opcode:: POP_JUMP_IF_TRUE (target) + + If TOS is true, sets the bytecode counter to *target*. TOS is popped. + + +.. opcode:: POP_JUMP_IF_FALSE (target) + + If TOS is false, sets the bytecode counter to *target*. TOS is popped. + + +.. opcode:: JUMP_IF_TRUE_OR_POP (target) - If TOS is true, increment the bytecode counter by *delta*. TOS is left on the - stack. + If TOS is true, sets the bytecode counter to *target* and leaves TOS + on the stack. Otherwise (TOS is false), TOS is popped. -.. opcode:: JUMP_IF_FALSE (delta) +.. opcode:: JUMP_IF_FALSE_OR_POP (target) - If TOS is false, increment the bytecode counter by *delta*. TOS is not - changed. + If TOS is false, sets the bytecode counter to *target* and leaves + TOS on the stack. Otherwise (TOS is true), TOS is popped. .. opcode:: JUMP_ABSOLUTE (target) diff --git a/Doc/library/doctest.rst b/Doc/library/doctest.rst index 1a43a33..86804ca 100644 --- a/Doc/library/doctest.rst +++ b/Doc/library/doctest.rst @@ -95,7 +95,7 @@ works its magic:: $ There's no output! That's normal, and it means all the examples worked. Pass -:option:`-v` to the script, and :mod:`doctest` prints a detailed log of what +``-v`` to the script, and :mod:`doctest` prints a detailed log of what it's trying, and prints a summary at the end:: $ python example.py -v @@ -163,7 +163,7 @@ example(s) and the cause(s) of the failure(s) are printed to stdout, and the final line of output is ``***Test Failed*** N failures.``, where *N* is the number of examples that failed. -Run it with the :option:`-v` switch instead:: +Run it with the ``-v`` switch instead:: python M.py -v @@ -172,7 +172,7 @@ with assorted summaries at the end. You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or prohibit it by passing ``verbose=False``. In either of those cases, -``sys.argv`` is not examined by :func:`testmod` (so passing :option:`-v` or not +``sys.argv`` is not examined by :func:`testmod` (so passing ``-v`` or not has no effect). Since Python 2.6, there is also a command line shortcut for running @@ -242,7 +242,7 @@ See section :ref:`doctest-basic-api` for a description of the optional arguments that can be used to tell it to look for files in other locations. Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the -:option:`-v` command-line switch or with the optional keyword argument +``-v`` command-line switch or with the optional keyword argument *verbose*. Since Python 2.6, there is also a command line shortcut for running @@ -331,7 +331,7 @@ The fine print: blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line is expected. - .. versionchanged:: 2.4 + .. versionadded:: 2.4 ``<BLANKLINE>`` was added; there was no way to use expected output containing empty lines in previous versions. @@ -446,6 +446,9 @@ multi-line detail:: The last three lines (starting with :exc:`ValueError`) are compared against the exception's type and detail, and the rest are ignored. +.. versionchanged:: 2.4 + Previous versions were unable to handle multi-line exception details. + Best practice is to omit the traceback stack, unless it adds significant documentation value to the example. So the last example is probably better as:: @@ -477,8 +480,9 @@ Some details you should read once, but won't need to remember: with an alphanumeric is taken to be the start of the exception detail. Of course this does the right thing for genuine tracebacks. -* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is is specified, - everything following the leftmost colon is ignored. +* When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is specified, + everything following the leftmost colon and any module information in the + exception name is ignored. * The interactive shell omits the traceback header line for some :exc:`SyntaxError`\ s. But doctest uses the traceback header line to @@ -506,10 +510,6 @@ Some details you should read once, but won't need to remember: ^ SyntaxError: invalid syntax -.. versionchanged:: 2.4 - The ability to handle a multi-line exception detail, and the - :const:`IGNORE_EXCEPTION_DETAIL` doctest option, were added. - .. _doctest-options: @@ -572,20 +572,38 @@ doctest decides whether actual output matches an example's expected output: exception raised is ``ValueError: 3*14``, but will fail, e.g., if :exc:`TypeError` is raised. - Note that a similar effect can be obtained using :const:`ELLIPSIS`, and - :const:`IGNORE_EXCEPTION_DETAIL` may go away when Python releases prior to 2.4 - become uninteresting. Until then, :const:`IGNORE_EXCEPTION_DETAIL` is the only - clear way to write a doctest that doesn't care about the exception detail yet - continues to pass under Python releases prior to 2.4 (doctest directives appear - to be comments to them). For example, :: + It will also ignore the module name used in Python 3 doctest reports. Hence + both these variations will work regardless of whether the test is run under + Python 2.7 or Python 3.2 (or later versions): + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + CustomError: message + + >>> raise CustomError('message') #doctest: +IGNORE_EXCEPTION_DETAIL + Traceback (most recent call last): + my_module.CustomError: message + + Note that :const:`ELLIPSIS` can also be used to ignore the + details of the exception message, but such a test may still fail based + on whether or not the module details are printed as part of the + exception name. Using :const:`IGNORE_EXCEPTION_DETAIL` and the details + from Python 2.3 is also the only clear way to write a doctest that doesn't + care about the exception detail yet continues to pass under Python 2.3 or + earlier (those releases do not support doctest directives and ignore them + as irrelevant comments). For example, :: >>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: object doesn't support item assignment - passes under Python 2.4 and Python 2.3. The detail changed in 2.4, to say "does - not" instead of "doesn't". + passes under Python 2.3 and later Python versions, even though the detail + changed in Python 2.4 to say "does not" instead of "doesn't". + + .. versionchanged:: 2.7 + :const:`IGNORE_EXCEPTION_DETAIL` now also ignores any information + relating to the module containing the exception under test .. data:: SKIP @@ -598,6 +616,8 @@ doctest decides whether actual output matches an example's expected output: The SKIP flag can also be used for temporarily "commenting out" examples. +.. versionadded:: 2.5 + .. data:: COMPARISON_FLAGS @@ -700,17 +720,13 @@ usually the only meaningful choice. However, option flags can also be passed to functions that run doctests, establishing different defaults. In such cases, disabling an option via ``-`` in a directive can be useful. -.. versionchanged:: 2.4 - Constants :const:`DONT_ACCEPT_BLANKLINE`, :const:`NORMALIZE_WHITESPACE`, +.. versionadded:: 2.4 + Doctest directives and the associated constants + :const:`DONT_ACCEPT_BLANKLINE`, :const:`NORMALIZE_WHITESPACE`, :const:`ELLIPSIS`, :const:`IGNORE_EXCEPTION_DETAIL`, :const:`REPORT_UDIFF`, :const:`REPORT_CDIFF`, :const:`REPORT_NDIFF`, :const:`REPORT_ONLY_FIRST_FAILURE`, :const:`COMPARISON_FLAGS` and - :const:`REPORTING_FLAGS` were added; by default ``<BLANKLINE>`` in expected - output matches an empty line in actual output; and doctest directives were - added. - -.. versionchanged:: 2.5 - Constant :const:`SKIP` was added. + :const:`REPORTING_FLAGS` were added. There's also a way to register new option flag names, although this isn't useful unless you intend to extend :mod:`doctest` internals via subclassing: @@ -956,18 +972,17 @@ serious Python testing frameworks build on the :mod:`unittest` module, which supplies many flexible ways to combine tests from multiple sources. So, in Python 2.4, :mod:`doctest`'s :class:`Tester` class is deprecated, and :mod:`doctest` provides two functions that can be used to create :mod:`unittest` -test suites from modules and text files containing doctests. These test suites -can then be run using :mod:`unittest` test runners:: +test suites from modules and text files containing doctests. To integrate with +:mod:`unittest` test discovery, include a :func:`load_tests` function in your +test module:: import unittest import doctest - import my_module_with_doctests, and_another + import my_module_with_doctests - suite = unittest.TestSuite() - for mod in my_module_with_doctests, and_another: - suite.addTest(doctest.DocTestSuite(mod)) - runner = unittest.TextTestRunner() - runner.run(suite) + def load_tests(loader, tests, ignore): + tests.addTests(doctest.DocTestSuite(my_module_with_doctests)) + return tests There are two main functions for creating :class:`unittest.TestSuite` instances from text files and modules with doctests: @@ -1438,7 +1453,7 @@ DocTestRunner objects verbosity. If *verbose* is ``True``, then information is printed about each example, as it is run. If *verbose* is ``False``, then only failures are printed. If *verbose* is unspecified, or ``None``, then verbose output is used - iff the command-line switch :option:`-v` is used. + iff the command-line switch ``-v`` is used. The optional keyword argument *optionflags* can be used to control how the test runner compares expected output to actual output, and how it displays failures. @@ -1753,7 +1768,7 @@ There are two exceptions that may be raised by :class:`DebugRunner` instances: .. exception:: DocTestFailure(test, example, got) - An exception thrown by :class:`DocTestRunner` to signal that a doctest example's + An exception raised by :class:`DocTestRunner` to signal that a doctest example's actual output did not match its expected output. The constructor arguments are used to initialize the member variables of the same names. @@ -1777,9 +1792,9 @@ There are two exceptions that may be raised by :class:`DebugRunner` instances: .. exception:: UnexpectedException(test, example, exc_info) - An exception thrown by :class:`DocTestRunner` to signal that a doctest example - raised an unexpected exception. The constructor arguments are used to - initialize the member variables of the same names. + An exception raised by :class:`DocTestRunner` to signal that a doctest + example raised an unexpected exception. The constructor arguments are used + to initialize the member variables of the same names. :exc:`UnexpectedException` defines the following member variables: diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst index c1b16da..32cecf3 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -11,6 +11,12 @@ First, let's see how to create and send a simple text message: .. literalinclude:: ../includes/email-simple.py +And parsing RFC822 headers can easily be done by the parse(filename) or +parsestr(message_as_string) methods of the Parser() class: + +.. literalinclude:: ../includes/email-headers.py + + Here's an example of how to send a MIME message containing a bunch of family pictures that may be residing in a directory: diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst index 7a0c52a..06598d5 100644 --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -17,7 +17,7 @@ The following exception classes are defined in the :mod:`email.errors` module: .. exception:: MessageParseError() - This is the base class for exceptions thrown by the :class:`~email.parser.Parser` + This is the base class for exceptions raised by the :class:`~email.parser.Parser` class. It is derived from :exc:`MessageError`. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst index 7182005..fe09de5 100644 --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -65,7 +65,7 @@ Here is the :class:`Header` class description: character set is used both as *s*'s initial charset and as the default for subsequent :meth:`append` calls. - The maximum line length can be specified explicit via *maxlinelen*. For + The maximum line length can be specified explicitly via *maxlinelen*. For splitting the first line to a shorter value (to account for the field header which isn't included in *s*, e.g. :mailheader:`Subject`) pass in the name of the field in *header_name*. The default *maxlinelen* is 76, and the default value @@ -74,7 +74,8 @@ Here is the :class:`Header` class description: Optional *continuation_ws* must be :rfc:`2822`\ -compliant folding whitespace, and is usually either a space or a hard tab character. This character will be - prepended to continuation lines. + prepended to continuation lines. *continuation_ws* defaults to a single + space character (" "). Optional *errors* is passed straight through to the :meth:`append` method. diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 4cc3880..a570d81 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -133,8 +133,22 @@ Here are the methods of the :class:`Message` class: string naming a character set, or ``None``. If it is a string, it will be converted to a :class:`~email.charset.Charset` instance. If *charset* is ``None``, the ``charset`` parameter will be removed from the - :mailheader:`Content-Type` header. Anything else will generate a - :exc:`TypeError`. + :mailheader:`Content-Type` header (the message will not be otherwise + modified). Anything else will generate a :exc:`TypeError`. + + If there is no existing :mailheader:`MIME-Version` header one will be + added. If there is no existing :mailheader:`Content-Type` header, one + will be added with a value of :mimetype:`text/plain`. Whether the + :mailheader:`Content-Type` header already exists or not, its ``charset`` + parameter will be set to *charset.output_charset*. If + *charset.input_charset* and *charset.output_charset* differ, the payload + will be re-encoded to the *output_charset*. If there is no existing + :mailheader:`Content-Transfer-Encoding` header, then the payload will be + transfer-encoded, if needed, using the specified + :class:`~email.charset.Charset`, and a header with the appropriate value + will be added. If a :mailheader:`Content-Transfer-Encoding` header + already exists, the payload is assumed to already be correctly encoded + using that :mailheader:`Content-Transfer-Encoding` and is not modified. The message will be assumed to be of type :mimetype:`text/\*`, with the payload either in unicode or encoded with *charset.input_charset*. @@ -266,7 +280,12 @@ Here are the methods of the :class:`Message` class: taken as the parameter name, with underscores converted to dashes (since dashes are illegal in Python identifiers). Normally, the parameter will be added as ``key="value"`` unless the value is ``None``, in which case - only the key will be added. + only the key will be added. If the value contains non-ASCII characters, + it must be specified as a three tuple in the format + ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the + charset to be used to encode the value, ``LANGUAGE`` can usually be set + to ``None`` or the empty string (see :RFC:`2231` for other possibilities), + and ``VALUE`` is the string value containing non-ASCII code points. Here's an example:: @@ -276,6 +295,15 @@ Here are the methods of the :class:`Message` class: Content-Disposition: attachment; filename="bud.gif" + An example with with non-ASCII characters:: + + msg.add_header('Content-Disposition', 'attachment', + filename=('iso-8859-1', '', 'Fußballer.ppt')) + + Which produces :: + + Content-Disposition: attachment; filename*="iso-8859-1''Fu%DFballer.ppt" + .. method:: replace_header(_name, _value) @@ -380,7 +408,7 @@ Here are the methods of the :class:`Message` class: :rfc:`2231`, you can collapse the parameter value by calling :func:`email.utils.collapse_rfc2231_value`, passing in the return value from :meth:`get_param`. This will return a suitably decoded Unicode - string whn the value is a tuple, or the original string unquoted if it + string when the value is a tuple, or the original string unquoted if it isn't. For example:: rawparam = msg.get_param('foo') @@ -448,9 +476,10 @@ Here are the methods of the :class:`Message` class: Return the value of the ``filename`` parameter of the :mailheader:`Content-Disposition` header of the message. If the header does not have a ``filename`` parameter, this method falls back to looking - for the ``name`` parameter. If neither is found, or the header is - missing, then *failobj* is returned. The returned string will always be - unquoted as per :func:`email.utils.unquote`. + for the ``name`` parameter on the :mailheader:`Content-Type` header. If + neither is found, or the header is missing, then *failobj* is returned. + The returned string will always be unquoted as per + :func:`email.utils.unquote`. .. method:: get_boundary([failobj]) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 2f07d29..8bd903a 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -26,7 +26,7 @@ equivalent, even if they have the same name. The built-in exceptions listed below can be generated by the interpreter or built-in functions. Except where mentioned, they have an "associated value" -indicating the detailed cause of the error. This may be a string or a tuple +indicating the detailed cause of the error. This may be a string or a tuple containing several items of information (e.g., an error code and a string explaining the code). The associated value is the second argument to the :keyword:`raise` statement. If the exception class is derived from the standard @@ -46,18 +46,23 @@ defining exceptions is available in the Python Tutorial under The following exceptions are only used as base classes for other exceptions. - .. exception:: BaseException The base class for all built-in exceptions. It is not meant to be directly - inherited by user-defined classes (for that use :exc:`Exception`). If + inherited by user-defined classes (for that, use :exc:`Exception`). If :func:`str` or :func:`unicode` is called on an instance of this class, the - representation of the argument(s) to the instance are returned or the empty - string when there were no arguments. All arguments are stored in :attr:`args` - as a tuple. + representation of the argument(s) to the instance are returned, or the empty + string when there were no arguments. .. versionadded:: 2.5 + .. attribute:: args + + The tuple of arguments given to the exception constructor. Some built-in + exceptions (like :exc:`IOError`) expect a certain number of arguments and + assign a special meaning to the elements of this tuple, while others are + usually called only with a single string giving an error message. + .. exception:: Exception @@ -82,6 +87,12 @@ The following exceptions are only used as base classes for other exceptions. :exc:`FloatingPointError`. +.. exception:: BufferError + + Raised when a :ref:`buffer <bufferobjects>` related operation cannot be + performed. + + .. exception:: LookupError The base class for the exceptions that are raised when a key or index used on @@ -141,7 +152,7 @@ The following exceptions are the exceptions that are actually raised. Raised when a floating point operation fails. This exception is always defined, but can only be raised when Python is configured with the - :option:`--with-fpectl` option, or the :const:`WANT_SIGFPE_HANDLER` symbol is + ``--with-fpectl`` option, or the :const:`WANT_SIGFPE_HANDLER` symbol is defined in the :file:`pyconfig.h` file. @@ -301,6 +312,18 @@ The following exceptions are the exceptions that are actually raised. of the exception instance returns only the message. +.. exception:: IndentationError + + Base class for syntax errors related to incorrect indentation. This is a + subclass of :exc:`SyntaxError`. + + +.. exception:: TabError + + Raised when indentation contains an inconsistent use of tabs and spaces. + This is a subclass of :exc:`IndentationError`. + + .. exception:: SystemError Raised when the interpreter finds an internal error, but the situation does not diff --git a/Doc/library/filecmp.rst b/Doc/library/filecmp.rst index 11d74ba..699e510 100644 --- a/Doc/library/filecmp.rst +++ b/Doc/library/filecmp.rst @@ -11,6 +11,11 @@ The :mod:`filecmp` module defines functions to compare files and directories, with various optional time/correctness trade-offs. For comparing files, see also the :mod:`difflib` module. +.. seealso:: + + Latest version of the `filecmp Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/filecmp.py?view=markup>`_ + The :mod:`filecmp` module defines the following functions: diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index ffd109c..709237e 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -44,6 +44,11 @@ hook must be a function that takes two arguments, *filename* and *mode*, and returns an accordingly opened file-like object. Two useful hooks are already provided by this module. +.. seealso:: + + Latest version of the `fileinput Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/fileinput.py?view=markup>`_ + The following function is the primary interface of this module: diff --git a/Doc/library/fnmatch.rst b/Doc/library/fnmatch.rst index 67aa2f4..cb1fa10 100644 --- a/Doc/library/fnmatch.rst +++ b/Doc/library/fnmatch.rst @@ -34,6 +34,10 @@ module. See module :mod:`glob` for pathname expansion (:mod:`glob` uses a period are not special for this module, and are matched by the ``*`` and ``?`` patterns. +.. seealso:: + + Latest version of the `fnmatch Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/fnmatch.py?view=markup>`_ .. function:: fnmatch(filename, pattern) @@ -73,6 +77,8 @@ patterns. Return the shell-style *pattern* converted to a regular expression. + Be aware there is no way to quote meta-characters. + Example: >>> import fnmatch, re diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index 36df11c..1ca9a9c 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -17,30 +17,34 @@ another rational number, or from a string. .. class:: Fraction(numerator=0, denominator=1) Fraction(other_fraction) + Fraction(float) + Fraction(decimal) Fraction(string) - The first version requires that *numerator* and *denominator* are - instances of :class:`numbers.Integral` and returns a new - :class:`Fraction` instance with value ``numerator/denominator``. If - *denominator* is :const:`0`, it raises a - :exc:`ZeroDivisionError`. The second version requires that - *other_fraction* is an instance of :class:`numbers.Rational` and - returns an :class:`Fraction` instance with the same value. The - last version of the constructor expects a string or unicode - instance in one of two possible forms. The first form is:: + The first version requires that *numerator* and *denominator* are instances + of :class:`numbers.Rational` and returns a new :class:`Fraction` instance + with value ``numerator/denominator``. If *denominator* is :const:`0`, it + raises a :exc:`ZeroDivisionError`. The second version requires that + *other_fraction* is an instance of :class:`numbers.Rational` and returns a + :class:`Fraction` instance with the same value. The next two versions accept + either a :class:`float` or a :class:`decimal.Decimal` instance, and return a + :class:`Fraction` instance with exactly the same value. Note that due to the + usual issues with binary floating-point (see :ref:`tut-fp-issues`), the + argument to ``Fraction(1.1)`` is not exactly equal to 11/10, and so + ``Fraction(1.1)`` does *not* return ``Fraction(11, 10)`` as one might expect. + (But see the documentation for the :meth:`limit_denominator` method below.) + The last version of the constructor expects a string or unicode instance. + The usual form for this instance is:: [sign] numerator ['/' denominator] where the optional ``sign`` may be either '+' or '-' and ``numerator`` and ``denominator`` (if present) are strings of - decimal digits. The second permitted form is that of a number - containing a decimal point:: - - [sign] integer '.' [fraction] | [sign] '.' fraction - - where ``integer`` and ``fraction`` are strings of digits. In - either form the input string may also have leading and/or trailing - whitespace. Here are some examples:: + decimal digits. In addition, any string that represents a finite + value and is accepted by the :class:`float` constructor is also + accepted by the :class:`Fraction` constructor. In either form the + input string may also have leading and/or trailing whitespace. + Here are some examples:: >>> from fractions import Fraction >>> Fraction(16, -10) @@ -58,6 +62,15 @@ another rational number, or from a string. Fraction(1414213, 1000000) >>> Fraction('-.125') Fraction(-1, 8) + >>> Fraction('7e-6') + Fraction(7, 1000000) + >>> Fraction(2.25) + Fraction(9, 4) + >>> Fraction(1.1) + Fraction(2476979795053773, 2251799813685248) + >>> from decimal import Decimal + >>> Fraction(Decimal('1.1')) + Fraction(11, 10) The :class:`Fraction` class inherits from the abstract base class @@ -66,6 +79,10 @@ another rational number, or from a string. and should be treated as immutable. In addition, :class:`Fraction` has the following methods: + .. versionchanged:: 2.7 + The :class:`Fraction` constructor now accepts :class:`float` and + :class:`decimal.Decimal` instances. + .. method:: from_float(flt) @@ -73,12 +90,19 @@ another rational number, or from a string. value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` + .. note:: From Python 2.7 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`float`. + .. method:: from_decimal(dec) This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal`. + .. note:: From Python 2.7 onwards, you can also construct a + :class:`Fraction` instance directly from a :class:`decimal.Decimal` + instance. + .. method:: limit_denominator(max_denominator=1000000) @@ -93,10 +117,12 @@ another rational number, or from a string. or for recovering a rational number that's represented as a float: >>> from math import pi, cos - >>> Fraction.from_float(cos(pi/3)) + >>> Fraction(cos(pi/3)) Fraction(4503599627370497, 9007199254740992) - >>> Fraction.from_float(cos(pi/3)).limit_denominator() + >>> Fraction(cos(pi/3)).limit_denominator() Fraction(1, 2) + >>> Fraction(1.1).limit_denominator() + Fraction(11, 10) .. function:: gcd(a, b) diff --git a/Doc/library/ftplib.rst b/Doc/library/ftplib.rst index f489469..b6db983 100644 --- a/Doc/library/ftplib.rst +++ b/Doc/library/ftplib.rst @@ -50,6 +50,40 @@ The module defines the following items: *timeout* was added. +.. class:: FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, timeout]]]]]]]) + + A :class:`FTP` subclass which adds TLS support to FTP as described in + :rfc:`4217`. + Connect as usual to port 21 implicitly securing the FTP control connection + before authenticating. Securing the data connection requires the user to + explicitly ask for it by calling the :meth:`prot_p` method. + *keyfile* and *certfile* are optional -- they can contain a PEM formatted + private key and certificate chain file name for the SSL connection. + + .. versionadded:: 2.7 + + Here's a sample session using the :class:`FTP_TLS` class: + + >>> from ftplib import FTP_TLS + >>> ftps = FTP_TLS('ftp.python.org') + >>> ftps.login() # login anonymously before securing control channel + >>> ftps.prot_p() # switch to secure data connection + >>> ftps.retrlines('LIST') # list directory content securely + total 9 + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . + drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin + drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc + d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming + drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib + drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub + drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr + -rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg + '226 Transfer complete.' + >>> ftps.quit() + >>> + + .. exception:: error_reply Exception raised when an unexpected reply is received from the server. @@ -57,18 +91,21 @@ The module defines the following items: .. exception:: error_temp - Exception raised when an error code in the range 400--499 is received. + Exception raised when an error code signifying a temporary error (response + codes in the range 400--499) is received. .. exception:: error_perm - Exception raised when an error code in the range 500--599 is received. + Exception raised when an error code signifying a permanent error (response + codes in the range 500--599) is received. .. exception:: error_proto - Exception raised when a reply is received from the server that does not - begin with a digit in the range 1--5. + Exception raised when a reply is received from the server that does not fit + the response specifications of the File Transfer Protocol, i.e. begin with a + digit in the range 1--5. .. data:: all_errors @@ -76,7 +113,7 @@ The module defines the following items: The set of all exceptions (as a tuple) that methods of :class:`FTP` instances may raise as a result of problems with the FTP connection (as opposed to programming errors made by the caller). This set includes the - four exceptions listed below as well as :exc:`socket.error` and + four exceptions listed above as well as :exc:`socket.error` and :exc:`IOError`. @@ -164,9 +201,9 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. .. method:: FTP.voidcmd(command) - Send a simple command string to the server and handle the response. Return - nothing if a response code in the range 200--299 is received. Raise an exception - otherwise. + Send a simple command string to the server and handle the response. Return + nothing if a response code corresponding to success (codes in the range + 200--299) is received. Raise :exc:`error_reply` otherwise. .. method:: FTP.retrbinary(command, callback[, maxblocksize[, rest]]) @@ -186,9 +223,11 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. Retrieve a file or directory listing in ASCII transfer mode. *command* should be an appropriate ``RETR`` command (see :meth:`retrbinary`) or a command such as ``LIST``, ``NLST`` or ``MLSD`` (usually just the string - ``'LIST'``). The *callback* function is called for each line with a - string argument containing the line with the trailing CRLF stripped. - The default *callback* prints the line to ``sys.stdout``. + ``'LIST'``). ``LIST`` retrieves a list of files and information about those files. + ``NLST`` retrieves a list of file names. On some servers, ``MLSD`` retrieves + a machine readable list of files and information about those files. The *callback* + function is called for each line with a string argument containing the line with + the trailing CRLF stripped. The default *callback* prints the line to ``sys.stdout``. .. method:: FTP.set_pasv(boolean) @@ -198,14 +237,15 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. it is on by default.) -.. method:: FTP.storbinary(command, file[, blocksize, callback]) +.. method:: FTP.storbinary(command, file[, blocksize, callback, rest]) Store a file in binary transfer mode. *command* should be an appropriate ``STOR`` command: ``"STOR filename"``. *file* is an open file object which is read until EOF using its :meth:`read` method in blocks of size *blocksize* to provide the data to be stored. The *blocksize* argument defaults to 8192. *callback* is an optional single parameter callable that is called - on each block of data after it is sent. + on each block of data after it is sent. *rest* means the same thing as in + the :meth:`transfercmd` method. .. versionchanged:: 2.1 default for *blocksize* added. @@ -213,6 +253,8 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. .. versionchanged:: 2.6 *callback* parameter added. + .. versionchanged:: 2.7 + *rest* parameter added. .. method:: FTP.storlines(command, file[, callback]) @@ -256,10 +298,10 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. .. method:: FTP.nlst(argument[, ...]) - Return a list of files as returned by the ``NLST`` command. The optional - *argument* is a directory to list (default is the current server directory). - Multiple arguments can be used to pass non-standard options to the ``NLST`` - command. + Return a list of file names as returned by the ``NLST`` command. The + optional *argument* is a directory to list (default is the current server + directory). Multiple arguments can be used to pass non-standard options to + the ``NLST`` command. .. method:: FTP.dir(argument[, ...]) @@ -329,3 +371,26 @@ followed by ``lines`` for the text version or ``binary`` for the binary version. :meth:`close` or :meth:`quit` you cannot reopen the connection by issuing another :meth:`login` method). + +FTP_TLS Objects +--------------- + +:class:`FTP_TLS` class inherits from :class:`FTP`, defining these additional objects: + +.. attribute:: FTP_TLS.ssl_version + + The SSL version to use (defaults to *TLSv1*). + +.. method:: FTP_TLS.auth() + + Set up secure control connection by using TLS or SSL, depending on what specified in :meth:`ssl_version` attribute. + +.. method:: FTP_TLS.prot_p() + + Set up secure data connection. + +.. method:: FTP_TLS.prot_c() + + Set up clear text data connection. + + diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index f24c46d..4b63c9f 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -7,6 +7,26 @@ Built-in Functions The Python interpreter has a number of functions built into it that are always available. They are listed here in alphabetical order. +=================== ================= ================== ================= ==================== +.. .. Built-in Functions .. .. +=================== ================= ================== ================= ==================== +:func:`abs` :func:`divmod` :func:`input` :func:`open` :func:`staticmethod` +:func:`all` :func:`enumerate` :func:`int` :func:`ord` :func:`str` +:func:`any` :func:`eval` :func:`isinstance` :func:`pow` :func:`sum` +:func:`basestring` :func:`execfile` :func:`issubclass` :func:`print` :func:`super` +:func:`bin` :func:`file` :func:`iter` :func:`property` :func:`tuple` +:func:`bool` :func:`filter` :func:`len` :func:`range` :func:`type` +:func:`bytearray` :func:`float` :func:`list` :func:`raw_input` :func:`unichr` +:func:`callable` :func:`format` :func:`locals` :func:`reduce` :func:`unicode` +:func:`chr` :func:`frozenset` :func:`long` :func:`reload` :func:`vars` +:func:`classmethod` :func:`getattr` :func:`map` :func:`repr` :func:`xrange` +:func:`cmp` :func:`globals` :func:`max` :func:`reversed` :func:`zip` +:func:`compile` :func:`hasattr` :func:`memoryview` :func:`round` :func:`__import__` +:func:`complex` :func:`hash` :func:`min` :func:`set` :func:`apply` +:func:`delattr` :func:`help` :func:`next` :func:`setattr` :func:`buffer` +:func:`dict` :func:`hex` :func:`object` :func:`slice` :func:`coerce` +:func:`dir` :func:`id` :func:`oct` :func:`sorted` :func:`intern` +=================== ================= ================== ================= ==================== .. function:: abs(x) @@ -78,6 +98,32 @@ available. They are listed here in alphabetical order. If no argument is given, this function returns :const:`False`. +.. function:: bytearray([source[, encoding[, errors]]]) + + Return a new array of bytes. The :class:`bytearray` type is a mutable + sequence of integers in the range 0 <= x < 256. It has most of the usual + methods of mutable sequences, described in :ref:`typesseq-mutable`, as well + as most methods that the :class:`str` type has, see :ref:`string-methods`. + + The optional *source* parameter can be used to initialize the array in a few + different ways: + + * If it is a *string*, you must also give the *encoding* (and optionally, + *errors*) parameters; :func:`bytearray` then converts the string to + bytes using :meth:`str.encode`. + + * If it is an *integer*, the array will have that size and will be + initialized with null bytes. + + * If it is an object conforming to the *buffer* interface, a read-only buffer + of the object will be used to initialize the bytes array. + + * If it is an *iterable*, it must be an iterable of integers in the range + ``0 <= x < 256``, which are used as the initial contents of the array. + + Without an argument, an array of size 0 is created. + + .. function:: callable(object) Return :const:`True` if the *object* argument appears callable, @@ -173,11 +219,10 @@ available. They are listed here in alphabetical order. .. note:: - When compiling a string with multi-line code, line endings must be - represented by a single newline character (``'\n'``), and the input must - be terminated by at least one newline character. If line endings are - represented by ``'\r\n'``, use :meth:`str.replace` to change them into - ``'\n'``. + When compiling a string with multi-line code in ``'single'`` or + ``'eval'`` mode, input must be terminated by at least one newline + character. This is to facilitate detection of incomplete and complete + statements in the :mod:`code` module. .. versionchanged:: 2.3 The *flags* and *dont_inherit* arguments were added. @@ -185,6 +230,10 @@ available. They are listed here in alphabetical order. .. versionchanged:: 2.6 Support for compiling AST objects. + .. versionchanged:: 2.7 + Allowed use of Windows and Mac newlines. Also input in ``'exec'`` mode + does not have to end in a newline anymore. + .. function:: complex([real[, imag]]) @@ -344,6 +393,9 @@ available. They are listed here in alphabetical order. returns the current global and local dictionary, respectively, which may be useful to pass around for use by :func:`eval` or :func:`execfile`. + See :func:`ast.literal_eval` for a function that can safely evaluate strings + with expressions containing only literals. + .. function:: execfile(filename[, globals[, locals]]) @@ -462,7 +514,7 @@ available. They are listed here in alphabetical order. .. function:: getattr(object, name[, default]) - Return the value of the named attributed of *object*. *name* must be a string. + Return the value of the named attribute of *object*. *name* must be a string. If the string is the name of one of the object's attributes, the result is the value of that attribute. For example, ``getattr(x, 'foobar')`` is equivalent to ``x.foobar``. If the named attribute does not exist, *default* is returned if @@ -689,6 +741,13 @@ available. They are listed here in alphabetical order. Added support for the optional *key* argument. +.. function:: memoryview(obj) + :noindex: + + Return a "memory view" object created from the given argument. See + :ref:`typememoryview` for more information. + + .. function:: min(iterable[, args...][key]) With a single argument *iterable*, return the smallest item of a non-empty @@ -1088,6 +1147,14 @@ available. They are listed here in alphabetical order. example, ``round(0.5)`` is ``1.0`` and ``round(-0.5)`` is ``-1.0``). + .. note:: + + The behavior of :func:`round` for floats can be surprising: for example, + ``round(2.675, 2)`` gives ``2.67`` instead of the expected ``2.68``. + This is not a bug: it's a result of the fact that most decimal fractions + can't be represented exactly as a float. See :ref:`tut-fp-issues` for + more information. + .. function:: set([iterable]) :noindex: @@ -1148,9 +1215,8 @@ available. They are listed here in alphabetical order. In general, the *key* and *reverse* conversion processes are much faster than specifying an equivalent *cmp* function. This is because *cmp* is called multiple times for each list element while *key* and *reverse* touch - each element only once. To convert an old-style *cmp* function to a *key* - function, see the `CmpToKey recipe in the ASPN cookbook - <http://code.activestate.com/recipes/576653/>`_\. + each element only once. Use :func:`functools.cmp_to_key` to convert an + old-style *cmp* function to a *key* function. For sorting examples and a brief sorting tutorial, see `Sorting HowTo <http://wiki.python.org/moin/HowTo/Sorting/>`_\. @@ -1207,10 +1273,13 @@ available. They are listed here in alphabetical order. Sums *start* and the items of an *iterable* from left to right and returns the total. *start* defaults to ``0``. The *iterable*'s items are normally numbers, - and are not allowed to be strings. The fast, correct way to concatenate a - sequence of strings is by calling ``''.join(sequence)``. Note that - ``sum(range(n), m)`` is equivalent to ``reduce(operator.add, range(n), m)`` - To add floating point values with extended precision, see :func:`math.fsum`\. + and the start value is not allowed to be a string. + + For some use cases, there are good alternatives to :func:`sum`. + The preferred, fast way to concatenate a sequence of strings is by calling + ``''.join(sequence)``. To add floating point values with extended precision, + see :func:`math.fsum`\. To concatenate a series of iterables, consider using + :func:`itertools.chain`. .. versionadded:: 2.3 @@ -1394,8 +1463,8 @@ available. They are listed here in alphabetical order. restricts all arguments to native C longs ("short" Python integers), and also requires that the number of elements fit in a native C long. If a larger range is needed, an alternate version can be crafted using the - :mod:`itertools` module: ``takewhile(lambda x: x<stop, (start+i*step - for i in count()))``. + :mod:`itertools` module: ``islice(count(start, step), + (stop-start+step-1)//step)``. .. function:: zip([iterable, ...]) diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index a09d3cf..787c000 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -15,8 +15,56 @@ The :mod:`functools` module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a function for the purposes of this module. +.. seealso:: + + Latest version of the `functools Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/functools.py?view=markup>`_ + The :mod:`functools` module defines the following functions: +.. function:: cmp_to_key(func) + + Transform an old-style comparison function to a key-function. Used with + tools that accept key functions (such as :func:`sorted`, :func:`min`, + :func:`max`, :func:`heapq.nlargest`, :func:`heapq.nsmallest`, + :func:`itertools.groupby`). This function is primarily used as a transition + tool for programs being converted to Py3.x where comparison functions are no + longer supported. + + A compare function is any callable that accept two arguments, compares them, + and returns a negative number for less-than, zero for equality, or a positive + number for greater-than. A key function is a callable that accepts one + argument and returns another value that indicates the position in the desired + collation sequence. + + Example:: + + sorted(iterable, key=cmp_to_key(locale.strcoll)) # locale-aware sort order + + .. versionadded:: 2.7 + +.. function:: total_ordering(cls) + + Given a class defining one or more rich comparison ordering methods, this + class decorator supplies the rest. This simplifies the effort involved + in specifying all of the possible rich comparison operations: + + The class must define one of :meth:`__lt__`, :meth:`__le__`, + :meth:`__gt__`, or :meth:`__ge__`. + In addition, the class should supply an :meth:`__eq__` method. + + For example:: + + @total_ordering + class Student: + def __eq__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) == + (other.lastname.lower(), other.firstname.lower())) + def __lt__(self, other): + return ((self.lastname.lower(), self.firstname.lower()) < + (other.lastname.lower(), other.firstname.lower())) + + .. versionadded:: 2.7 .. function:: reduce(function, iterable[, initializer]) diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst index 7074776..80a2d92 100644 --- a/Doc/library/gc.rst +++ b/Doc/library/gc.rst @@ -140,6 +140,31 @@ The :mod:`gc` module provides the following functions: .. versionadded:: 2.3 +.. function:: is_tracked(obj) + + Returns True if the object is currently tracked by the garbage collector, + False otherwise. As a general rule, instances of atomic types aren't + tracked and instances of non-atomic types (containers, user-defined + objects...) are. However, some type-specific optimizations can be present + in order to suppress the garbage collector footprint of simple instances + (e.g. dicts containing only atomic keys and values):: + + >>> gc.is_tracked(0) + False + >>> gc.is_tracked("a") + False + >>> gc.is_tracked([]) + True + >>> gc.is_tracked({}) + False + >>> gc.is_tracked({"a": 1}) + False + >>> gc.is_tracked({"a": []}) + True + + .. versionadded:: 2.7 + + The following variable is provided for read-only access (you can mutate its value but should not rebind it): diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst index 927953c..4d65c79 100644 --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -1,11 +1,17 @@ -:mod:`getopt` --- Parser for command line options -================================================= +:mod:`getopt` --- C-style parser for command line options +========================================================= .. module:: getopt :synopsis: Portable parser for command line options; support both short and long option names. +.. note:: + The :mod:`getopt` module is a parser for command line options whose API is + designed to be familiar to users of the C :cfunc:`getopt` function. Users who + are unfamiliar with the C :cfunc:`getopt` function or who would like to write + less code and get better help and error messages should consider using the + :mod:`argparse` module instead. This module helps scripts to parse the command line arguments in ``sys.argv``. It supports the same conventions as the Unix :cfunc:`getopt` function (including @@ -35,14 +41,14 @@ exception: non-GNU Unix systems work. *long_options*, if specified, must be a list of strings with the names of the - long options which should be supported. The leading ``'-``\ ``-'`` + long options which should be supported. The leading ``'--'`` characters should not be included in the option name. Long options which require an argument should be followed by an equal sign (``'='``). Optional arguments are not supported. To accept only long options, *options* should be an empty string. Long options on the command line can be recognized so long as they provide a prefix of the option name that matches exactly one of the accepted options. For example, if *long_options* is ``['foo', 'frob']``, - the option :option:`--fo` will match as :option:`--foo`, but :option:`--f` + the option ``--fo`` will match as ``--foo``, but ``--f`` will not match uniquely, so :exc:`GetoptError` will be raised. The return value consists of two elements: the first is a list of ``(option, @@ -50,7 +56,7 @@ exception: option list was stripped (this is a trailing slice of *args*). Each option-and-value pair returned has the option as its first element, prefixed with a hyphen for short options (e.g., ``'-x'``) or two hyphens for long - options (e.g., ``'-``\ ``-long-option'``), and the option argument as its + options (e.g., ``'--long-option'``), and the option argument as its second element, or an empty string if the option has no argument. The options occur in the list in the same order in which they were found, thus allowing multiple occurrences. Long and short options may be mixed. @@ -63,7 +69,7 @@ exception: intermixed. The :func:`getopt` function stops processing options as soon as a non-option argument is encountered. - If the first character of the option string is '+', or if the environment + If the first character of the option string is ``'+'``, or if the environment variable :envvar:`POSIXLY_CORRECT` is set, then option processing stops as soon as a non-option argument is encountered. @@ -142,9 +148,21 @@ In a script, typical usage is something like this:: if __name__ == "__main__": main() +Note that an equivalent command line interface could be produced with less code +and more informative help and error messages by using the :mod:`argparse` module:: + + import argparse + + if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-o', '--output') + parser.add_argument('-v', dest='verbose', action='store_true') + args = parser.parse_args() + # ... do something with args.output ... + # ... do something with args.verbose .. .. seealso:: - Module :mod:`optparse` - More object-oriented command line option parsing. + Module :mod:`argparse` + Alternative command line option and argument parsing library. diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst index 5f28480..1ef9a31 100644 --- a/Doc/library/glob.rst +++ b/Doc/library/glob.rst @@ -16,6 +16,10 @@ matched. This is done by using the :func:`os.listdir` and subshell. (For tilde and shell variable expansion, use :func:`os.path.expanduser` and :func:`os.path.expandvars`.) +.. seealso:: + + Latest version of the `glob module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/glob.py?view=markup>`_ .. function:: glob(pathname) diff --git a/Doc/library/grp.rst b/Doc/library/grp.rst index a71c308..f20a212 100644 --- a/Doc/library/grp.rst +++ b/Doc/library/grp.rst @@ -31,7 +31,9 @@ correspond to the members of the ``group`` structure (Attribute field below, see The gid is an integer, name and password are strings, and the member list is a list of strings. (Note that most users are not explicitly listed as members of the group they are in according to the password database. Check both databases -to get complete membership information.) +to get complete membership information. Also note that a ``gr_name`` that +starts with a ``+`` or ``-`` is likely to be a YP/NIS reference and may not be +accessible via :func:`getgrnam` or :func:`getgrgid`.) It defines the following items: diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst index 3493162..17896c3 100644 --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -24,7 +24,7 @@ For other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and The module defines the following items: -.. class:: GzipFile([filename[, mode[, compresslevel[, fileobj]]]]) +.. class:: GzipFile([filename[, mode[, compresslevel[, fileobj[, mtime]]]]]) Constructor for the :class:`GzipFile` class, which simulates most of the methods of a file object, with the exception of the :meth:`readinto` and @@ -52,13 +52,29 @@ The module defines the following items: level of compression; ``1`` is fastest and produces the least compression, and ``9`` is slowest and produces the most compression. The default is ``9``. + The *mtime* argument is an optional numeric timestamp to be written to + the stream when compressing. All :program:`gzip` compressed streams are + required to contain a timestamp. If omitted or ``None``, the current + time is used. This module ignores the timestamp when decompressing; + however, some programs, such as :program:`gunzip`\ , make use of it. + The format of the timestamp is the same as that of the return value of + ``time.time()`` and of the ``st_mtime`` member of the object returned + by ``os.stat()``. + Calling a :class:`GzipFile` object's :meth:`close` method does not close *fileobj*, since you might wish to append more material after the compressed data. This also allows you to pass a :class:`StringIO` object opened for writing as *fileobj*, and retrieve the resulting memory buffer using the :class:`StringIO` object's :meth:`getvalue` method. - :class:`GzipFile` supports iteration. + :class:`GzipFile` supports iteration and the :keyword:`with` statement. + + .. versionchanged:: 2.7 + Support for the :keyword:`with` statement was added. + + .. versionchanged:: 2.7 + Support for zero-padded files was added. + .. function:: open(filename[, mode[, compresslevel]]) diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index a889a2f..51ac93b 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -74,6 +74,15 @@ Using :func:`new` with an algorithm provided by OpenSSL: >>> h.hexdigest() 'cc4a5ce1b3df48aec5d22d1f16b894a0b894eccc' +This module provides the following constant attribute: + +.. data:: hashlib.algorithms + + A tuple providing the names of the hash algorithms guaranteed to be + supported by this module. + + .. versionadded:: 2.7 + The following values are provided as constant attributes of the hash objects returned by the constructors: @@ -95,6 +104,12 @@ A hash object has the following methods: a single call with the concatenation of all the arguments: ``m.update(a); m.update(b)`` is equivalent to ``m.update(a+b)``. + .. versionchanged:: 2.7 + + The Python GIL is released to allow other threads to run while + hash updates on data larger than 2048 bytes is taking place when + using hash algorithms supplied by OpenSSL. + .. method:: hash.digest() diff --git a/Doc/library/heapq.rst b/Doc/library/heapq.rst index 9a44047..a6b9ba2 100644 --- a/Doc/library/heapq.rst +++ b/Doc/library/heapq.rst @@ -6,17 +6,24 @@ .. moduleauthor:: Kevin O'Connor .. sectionauthor:: Guido van Rossum <guido@python.org> .. sectionauthor:: François Pinard +.. sectionauthor:: Raymond Hettinger .. versionadded:: 2.3 This module provides an implementation of the heap queue algorithm, also known as the priority queue algorithm. -Heaps are arrays for which ``heap[k] <= heap[2*k+1]`` and ``heap[k] <= -heap[2*k+2]`` for all *k*, counting elements from zero. For the sake of -comparison, non-existing elements are considered to be infinite. The -interesting property of a heap is that ``heap[0]`` is always its smallest -element. +.. seealso:: + + Latest version of the `heapq Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/heapq.py?view=markup>`_ + +Heaps are binary trees for which every parent node has a value less than or +equal to any of its children. This implementation uses arrays for which +``heap[k] <= heap[2*k+1]`` and ``heap[k] <= heap[2*k+2]`` for all *k*, counting +elements from zero. For the sake of comparison, non-existing elements are +considered to be infinite. The interesting property of a heap is that its +smallest element is always the root, ``heap[0]``. The API below differs from textbook heap algorithms in two aspects: (a) We use zero-based indexing. This makes the relationship between the index for a node @@ -62,45 +69,16 @@ The following functions are provided: Pop and return the smallest item from the *heap*, and also push the new *item*. The heap size doesn't change. If the heap is empty, :exc:`IndexError` is raised. - This is more efficient than :func:`heappop` followed by :func:`heappush`, and - can be more appropriate when using a fixed-size heap. Note that the value - returned may be larger than *item*! That constrains reasonable uses of this - routine unless written as part of a conditional replacement:: - - if item > heap[0]: - item = heapreplace(heap, item) -Example of use: + This one step operation is more efficient than a :func:`heappop` followed by + :func:`heappush` and can be more appropriate when using a fixed-size heap. + The pop/push combination always returns an element from the heap and replaces + it with *item*. - >>> from heapq import heappush, heappop - >>> heap = [] - >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] - >>> for item in data: - ... heappush(heap, item) - ... - >>> ordered = [] - >>> while heap: - ... ordered.append(heappop(heap)) - ... - >>> print ordered - [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] - >>> data.sort() - >>> print data == ordered - True - -Using a heap to insert items at the correct place in a priority queue: - - >>> heap = [] - >>> data = [(1, 'J'), (4, 'N'), (3, 'H'), (2, 'O')] - >>> for item in data: - ... heappush(heap, item) - ... - >>> while heap: - ... print heappop(heap)[1] - J - O - H - N + The value returned may be larger than the *item* added. If that isn't + desired, consider using :func:`heappushpop` instead. Its push/pop + combination returns the smaller of the two values, leaving the larger value + on the heap. The module also offers three general purpose functions based on heaps. @@ -151,12 +129,100 @@ values, it is more efficient to use the :func:`sorted` function. Also, when functions. +Basic Examples +-------------- + +A `heapsort <http://en.wikipedia.org/wiki/Heapsort>`_ can be implemented by +pushing all values onto a heap and then popping off the smallest values one at a +time:: + + >>> def heapsort(iterable): + ... 'Equivalent to sorted(iterable)' + ... h = [] + ... for value in iterable: + ... heappush(h, value) + ... return [heappop(h) for i in range(len(h))] + ... + >>> heapsort([1, 3, 5, 7, 9, 2, 4, 6, 8, 0]) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + +Heap elements can be tuples. This is useful for assigning comparison values +(such as task priorities) alongside the main record being tracked:: + + >>> h = [] + >>> heappush(h, (5, 'write code')) + >>> heappush(h, (7, 'release product')) + >>> heappush(h, (1, 'write spec')) + >>> heappush(h, (3, 'create tests')) + >>> heappop(h) + (1, 'write spec') + + +Priority Queue Implementation Notes +----------------------------------- + +A `priority queue <http://en.wikipedia.org/wiki/Priority_queue>`_ is common use +for a heap, and it presents several implementation challenges: + +* Sort stability: how do you get two tasks with equal priorities to be returned + in the order they were originally added? + +* In the future with Python 3, tuple comparison breaks for (priority, task) + pairs if the priorities are equal and the tasks do not have a default + comparison order. + +* If the priority of a task changes, how do you move it to a new position in + the heap? + +* Or if a pending task needs to be deleted, how do you find it and remove it + from the queue? + +A solution to the first two challenges is to store entries as 3-element list +including the priority, an entry count, and the task. The entry count serves as +a tie-breaker so that two tasks with the same priority are returned in the order +they were added. And since no two entry counts are the same, the tuple +comparison will never attempt to directly compare two tasks. + +The remaining challenges revolve around finding a pending task and making +changes to its priority or removing it entirely. Finding a task can be done +with a dictionary pointing to an entry in the queue. + +Removing the entry or changing its priority is more difficult because it would +break the heap structure invariants. So, a possible solution is to mark an +entry as invalid and optionally add a new entry with the revised priority:: + + pq = [] # the priority queue list + counter = itertools.count(1) # unique sequence count + task_finder = {} # mapping of tasks to entries + INVALID = 0 # mark an entry as deleted + + def add_task(priority, task, count=None): + if count is None: + count = next(counter) + entry = [priority, count, task] + task_finder[task] = entry + heappush(pq, entry) + + def get_top_priority(): + while True: + priority, count, task = heappop(pq) + del task_finder[task] + if count is not INVALID: + return task + + def delete_task(task): + entry = task_finder[task] + entry[1] = INVALID + + def reprioritize(priority, task): + entry = task_finder[task] + add_task(priority, task, entry[1]) + entry[1] = INVALID + + Theory ------ -(This explanation is due to François Pinard. The Python code for this module -was contributed by Kevin O'Connor.) - Heaps are arrays for which ``a[k] <= a[2*k+1]`` and ``a[k] <= a[2*k+2]`` for all *k*, counting elements from 0. For the sake of comparison, non-existing elements are considered to be infinite. The interesting property of a heap is diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index 8b3b92b..480d0a7 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -19,10 +19,6 @@ This module implements the HMAC algorithm as described by :rfc:`2104`. is made. *digestmod* is the digest constructor or module for the HMAC object to use. It defaults to the :func:`hashlib.md5` constructor. - .. note:: - - The md5 hash has known weaknesses but remains the default for backwards - compatibility. Choose a better one for your application. An HMAC object has the following methods: diff --git a/Doc/library/htmlparser.rst b/Doc/library/htmlparser.rst index 7b0ce03..c114bf8 100644 --- a/Doc/library/htmlparser.rst +++ b/Doc/library/htmlparser.rst @@ -158,7 +158,7 @@ An exception is defined as well: Method called when an unrecognized SGML declaration is read by the parser. The *data* parameter will be the entire contents of the declaration inside - the ``<!...>`` markup. It is sometimes useful to be be overridden by a + the ``<!...>`` markup. It is sometimes useful to be overridden by a derived class; the base class implementation throws an :exc:`HTMLParseError`. diff --git a/Doc/library/httplib.rst b/Doc/library/httplib.rst index 0689b0e..436857c 100644 --- a/Doc/library/httplib.rst +++ b/Doc/library/httplib.rst @@ -34,7 +34,7 @@ uses it to handle URLs that use HTTP and HTTPS. The module provides the following classes: -.. class:: HTTPConnection(host[, port[, strict[, timeout]]]) +.. class:: HTTPConnection(host[, port[, strict[, timeout[, source_address]]]]) An :class:`HTTPConnection` instance represents one transaction with an HTTP server. It should be instantiated passing it a host and optional port @@ -46,6 +46,8 @@ The module provides the following classes: status line. If the optional *timeout* parameter is given, blocking operations (like connection attempts) will timeout after that many seconds (if it is not given, the global default timeout setting is used). + The optional *source_address* parameter may be a tuple of a (host, port) + to use as the source address the HTTP connection is made from. For example, the following calls all create instances that connect to the server at the same host and port:: @@ -60,23 +62,28 @@ The module provides the following classes: .. versionchanged:: 2.6 *timeout* was added. + .. versionchanged:: 2.7 + *source_address* was added. -.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout]]]]]) + +.. class:: HTTPSConnection(host[, port[, key_file[, cert_file[, strict[, timeout[, source_address]]]]]]) A subclass of :class:`HTTPConnection` that uses SSL for communication with secure servers. Default port is ``443``. *key_file* is the name of a PEM formatted file that contains your private key. *cert_file* is a PEM formatted certificate chain file. - .. note:: - - This does not do any certificate verification. + .. warning:: + This does not do any verification of the server's certificate. .. versionadded:: 2.0 .. versionchanged:: 2.6 *timeout* was added. + .. versionchanged:: 2.7 + *source_address* was added. + .. class:: HTTPResponse(sock[, debuglevel=0][, strict=0]) @@ -436,6 +443,17 @@ HTTPConnection Objects debug level is ``0``, meaning no debugging output is printed. +.. method:: HTTPConnection.set_tunnel(host,port=None, headers=None) + + Set the host and the port for HTTP Connect Tunnelling. Normally used when + it is required to do HTTPS Conection through a proxy server. + + The headers argument should be a mapping of extra HTTP headers to to sent + with the CONNECT request. + + .. versionadded:: 2.7 + + .. method:: HTTPConnection.connect() Connect to the server specified when the object was created. @@ -507,6 +525,9 @@ HTTPResponse Objects .. versionadded:: 2.4 +.. method:: HTTPResponse.fileno() + + Returns the ``fileno`` of the underlying socket. .. attribute:: HTTPResponse.msg @@ -549,9 +570,8 @@ Here is an example session that uses the ``GET`` method:: >>> data2 = r2.read() >>> conn.close() -Here is an example session that uses ``HEAD`` method. Note that ``HEAD`` method -never returns any data. :: - +Here is an example session that uses the ``HEAD`` method. Note that the +``HEAD`` method never returns any data. :: >>> import httplib >>> conn = httplib.HTTPConnection("www.python.org") diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst index 1b78fb9..6bd1898 100644 --- a/Doc/library/idle.rst +++ b/Doc/library/idle.rst @@ -286,13 +286,13 @@ Command line usage If there are arguments: -#. If :option:`-e` is used, arguments are files opened for editing and +#. If ``-e`` is used, arguments are files opened for editing and ``sys.argv`` reflects the arguments passed to IDLE itself. -#. Otherwise, if :option:`-c` is used, all arguments are placed in +#. Otherwise, if ``-c`` is used, all arguments are placed in ``sys.argv[1:...]``, with ``sys.argv[0]`` set to ``'-c'``. -#. Otherwise, if neither :option:`-e` nor :option:`-c` is used, the first +#. Otherwise, if neither ``-e`` nor ``-c`` is used, the first argument is a script which is executed with the remaining arguments in ``sys.argv[1:...]`` and ``sys.argv[0]`` set to the script name. If the script name is '-', no script is executed but an interactive Python session is started; diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst index e18d7d5..95bdc70 100644 --- a/Doc/library/imaplib.rst +++ b/Doc/library/imaplib.rst @@ -84,9 +84,9 @@ The following utility functions are defined: .. function:: Internaldate2tuple(datestr) - Converts an IMAP4 INTERNALDATE string to Coordinated Universal Time. Returns a - :mod:`time` module tuple. - + Parse an IMAP4 ``INTERNALDATE`` string and return corresponding local + time. The return value is a :class:`time.struct_time` instance or + None if the string has wrong format. .. function:: Int2AP(num) @@ -101,9 +101,13 @@ The following utility functions are defined: .. function:: Time2Internaldate(date_time) - Converts a :mod:`time` module tuple to an IMAP4 ``INTERNALDATE`` representation. - Returns a string in the form: ``"DD-Mmm-YYYY HH:MM:SS +HHMM"`` (including - double-quotes). + Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation. The + return value is a string in the form: ``"DD-Mmm-YYYY HH:MM:SS + +HHMM"`` (including double-quotes). The *date_time* argument can be a + number (int or float) representing seconds since epoch (as returned + by :func:`time.time`), a 9-tuple representing local time (as returned by + :func:`time.localtime`), or a double-quoted string. In the last case, it + is assumed to already be in the correct format. Note that IMAP4 message numbers change as the mailbox changes; in particular, after an ``EXPUNGE`` command performs deletions the remaining messages are @@ -303,9 +307,10 @@ An :class:`IMAP4` instance has the following methods: .. method:: IMAP4.open(host, port) - Opens socket to *port* at *host*. The connection objects established by this + Opens socket to *port* at *host*. This method is implicitly called by + the :class:`IMAP4` constructor. The connection objects established by this method will be used in the ``read``, ``readline``, ``send``, and ``shutdown`` - methods. You may override this method. + methods. You may override this method. .. method:: IMAP4.partial(message_num, message_part, start, length) @@ -401,7 +406,8 @@ An :class:`IMAP4` instance has the following methods: .. method:: IMAP4.shutdown() - Close connection established in ``open``. You may override this method. + Close connection established in ``open``. This method is implicitly + called by :meth:`IMAP4.logout`. You may override this method. .. method:: IMAP4.socket() diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst index cf4655b..607dd14 100644 --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -116,8 +116,7 @@ This module provides an interface to the mechanisms used to implement the .. function:: acquire_lock() Acquire the interpreter's import lock for the current thread. This lock should - be used by import hooks to ensure thread-safety when importing modules. On - platforms without threads, this function does nothing. + be used by import hooks to ensure thread-safety when importing modules. Once a thread has acquired the import lock, the same thread may acquire it again without blocking; the thread must release it once for each time it has diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst new file mode 100644 index 0000000..c4307b5 --- /dev/null +++ b/Doc/library/importlib.rst @@ -0,0 +1,27 @@ +:mod:`importlib` -- Convenience wrappers for :func:`__import__` +=============================================================== + +.. module:: importlib + :synopsis: Convenience wrappers for __import__ + +.. moduleauthor:: Brett Cannon <brett@python.org> +.. sectionauthor:: Brett Cannon <brett@python.org> + +.. versionadded:: 2.7 + +This module is a minor subset of what is available in the more full-featured +package of the same name from Python 3.1 that provides a complete +implementation of :keyword:`import`. What is here has been provided to +help ease in transitioning from 2.7 to 3.1. + + +.. function:: import_module(name, package=None) + + Import a module. The *name* argument specifies what module to + import in absolute or relative terms + (e.g. either ``pkg.mod`` or ``..mod``). If the name is + specified in relative terms, then the *package* argument must be + specified to the package which is to act as the anchor for resolving the + package name (e.g. ``import_module('..mod', 'pkg.subpkg')`` will import + ``pkg.mod``). The specified module will be inserted into + :data:`sys.modules` and returned. diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 29dc6ae..850ff08 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -234,14 +234,14 @@ Note: Return a tuple of values that describe how Python will interpret the file identified by *path* if it is a module, or ``None`` if it would not be - identified as a module. The return tuple is ``(name, suffix, mode, mtype)``, - where *name* is the name of the module without the name of any enclosing - package, *suffix* is the trailing part of the file name (which may not be a - dot-delimited extension), *mode* is the :func:`open` mode that would be used - (``'r'`` or ``'rb'``), and *mtype* is an integer giving the type of the - module. *mtype* will have a value which can be compared to the constants - defined in the :mod:`imp` module; see the documentation for that module for - more information on module types. + identified as a module. The return tuple is ``(name, suffix, mode, + module_type)``, where *name* is the name of the module without the name of + any enclosing package, *suffix* is the trailing part of the file name (which + may not be a dot-delimited extension), *mode* is the :func:`open` mode that + would be used (``'r'`` or ``'rb'``), and *module_type* is an integer giving + the type of the module. *module_type* will have a value which can be + compared to the constants defined in the :mod:`imp` module; see the + documentation for that module for more information on module types. .. versionchanged:: 2.6 Returns a :term:`named tuple` ``ModuleInfo(name, suffix, mode, @@ -263,17 +263,20 @@ Note: .. function:: isclass(object) - Return true if the object is a class. + Return true if the object is a class, whether built-in or created in Python + code. .. function:: ismethod(object) - Return true if the object is a method. + Return true if the object is a bound method written in Python. .. function:: isfunction(object) - Return true if the object is a Python function or unnamed (:term:`lambda`) function. + Return true if the object is a Python function, which includes functions + created by a :term:`lambda` expression. + .. function:: isgeneratorfunction(object) @@ -281,12 +284,14 @@ Note: .. versionadded:: 2.6 + .. function:: isgenerator(object) Return true if the object is a generator. .. versionadded:: 2.6 + .. function:: istraceback(object) Return true if the object is a traceback. @@ -304,13 +309,14 @@ Note: .. function:: isbuiltin(object) - Return true if the object is a built-in function. + Return true if the object is a built-in function or a bound built-in method. .. function:: isroutine(object) Return true if the object is a user-defined or built-in function or method. + .. function:: isabstract(object) Return true if the object is an abstract base class. @@ -320,8 +326,9 @@ Note: .. function:: ismethoddescriptor(object) - Return true if the object is a method descriptor, but not if :func:`ismethod` - or :func:`isclass` or :func:`isfunction` are true. + Return true if the object is a method descriptor, but not if + :func:`ismethod`, :func:`isclass`, :func:`isfunction` or :func:`isbuiltin` + are true. This is new as of Python 2.2, and, for example, is true of ``int.__add__``. An object passing this test has a :attr:`__get__` attribute @@ -457,12 +464,13 @@ Classes and functions .. function:: getargspec(func) - Get the names and default values of a Python function's arguments. A tuple of four - things is returned: ``(args, varargs, varkw, defaults)``. *args* is a list of - the argument names (it may contain nested lists). *varargs* and *varkw* are the - names of the ``*`` and ``**`` arguments or ``None``. *defaults* is a tuple of - default argument values or None if there are no default arguments; if this tuple - has *n* elements, they correspond to the last *n* elements listed in *args*. + Get the names and default values of a Python function's arguments. A tuple of + four things is returned: ``(args, varargs, keywords, defaults)``. *args* is a + list of the argument names (it may contain nested lists). *varargs* and + *keywords* are the names of the ``*`` and ``**`` arguments or + ``None``. *defaults* is a tuple of default argument values or None if there + are no default arguments; if this tuple has *n* elements, they correspond to + the last *n* elements listed in *args*. .. versionchanged:: 2.6 Returns a :term:`named tuple` ``ArgSpec(args, varargs, keywords, @@ -471,11 +479,11 @@ Classes and functions .. function:: getargvalues(frame) - Get information about arguments passed into a particular frame. A tuple of four - things is returned: ``(args, varargs, varkw, locals)``. *args* is a list of the - argument names (it may contain nested lists). *varargs* and *varkw* are the - names of the ``*`` and ``**`` arguments or ``None``. *locals* is the locals - dictionary of the given frame. + Get information about arguments passed into a particular frame. A tuple of + four things is returned: ``(args, varargs, keywords, locals)``. *args* is a + list of the argument names (it may contain nested lists). *varargs* and + *keywords* are the names of the ``*`` and ``**`` arguments or ``None``. + *locals* is the locals dictionary of the given frame. .. versionchanged:: 2.6 Returns a :term:`named tuple` ``ArgInfo(args, varargs, keywords, @@ -504,6 +512,32 @@ Classes and functions metatype is in use, cls will be the first element of the tuple. +.. function:: getcallargs(func[, *args][, **kwds]) + + Bind the *args* and *kwds* to the argument names of the Python function or + method *func*, as if it was called with them. For bound methods, bind also the + first argument (typically named ``self``) to the associated instance. A dict + is returned, mapping the argument names (including the names of the ``*`` and + ``**`` arguments, if any) to their values from *args* and *kwds*. In case of + invoking *func* incorrectly, i.e. whenever ``func(*args, **kwds)`` would raise + an exception because of incompatible signature, an exception of the same type + and the same or similar message is raised. For example:: + + >>> from inspect import getcallargs + >>> def f(a, b=1, *pos, **named): + ... pass + >>> getcallargs(f, 1, 2, 3) + {'a': 1, 'named': {}, 'b': 2, 'pos': (3,)} + >>> getcallargs(f, a=2, x=4) + {'a': 2, 'named': {'x': 4}, 'b': 1, 'pos': ()} + >>> getcallargs(f) + Traceback (most recent call last): + ... + TypeError: f() takes at least 1 argument (0 given) + + .. versionadded:: 2.7 + + .. _inspect-stack: The interpreter stack diff --git a/Doc/library/io.rst b/Doc/library/io.rst index e3f8b36..3cfaaed 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -6,16 +6,30 @@ .. moduleauthor:: Guido van Rossum <guido@python.org> .. moduleauthor:: Mike Verdone <mike.verdone@gmail.com> .. moduleauthor:: Mark Russell <mark.russell@zen.co.uk> +.. moduleauthor:: Antoine Pitrou <solipsis@pitrou.net> +.. moduleauthor:: Amaury Forgeot d'Arc <amauryfa@gmail.com> +.. moduleauthor:: Benjamin Peterson <benjamin@python.org> .. sectionauthor:: Benjamin Peterson <benjamin@python.org> + .. versionadded:: 2.6 -The :mod:`io` module provides the Python interfaces to stream handling. The -built-in :func:`open` function is defined in this module. +The :mod:`io` module provides the Python interfaces to stream handling. +Under Python 2.x, this is proposed as an alternative to the built-in +:class:`file` object, but in Python 3.x it is the default interface to +access files and streams. + +.. note:: + + Since this module has been designed primarily for Python 3.x, you have to + be aware that all uses of "bytes" in this document refer to the + :class:`str` type (of which :class:`bytes` is an alias), and all uses + of "text" refer to the :class:`unicode` type. Furthermore, those two + types are not interchangeable in the :mod:`io` APIs. At the top of the I/O hierarchy is the abstract base class :class:`IOBase`. It defines the basic interface to a stream. Note, however, that there is no separation between reading and writing to streams; implementations are allowed -to throw an :exc:`IOError` if they do not support a given operation. +to raise an :exc:`IOError` if they do not support a given operation. Extending :class:`IOBase` is :class:`RawIOBase` which deals simply with the reading and writing of raw bytes to a stream. :class:`FileIO` subclasses @@ -31,10 +45,10 @@ streams. :class:`BytesIO` is a simple stream of in-memory bytes. Another :class:`IOBase` subclass, :class:`TextIOBase`, deals with streams whose bytes represent text, and handles encoding and decoding -from and to strings. :class:`TextIOWrapper`, which extends it, is a -buffered text interface to a buffered raw stream +from and to :class:`unicode` strings. :class:`TextIOWrapper`, which extends +it, is a buffered text interface to a buffered raw stream (:class:`BufferedIOBase`). Finally, :class:`StringIO` is an in-memory -stream for text. +stream for unicode text. Argument names are not part of the specification, and only the arguments of :func:`.open` are intended to be used as keyword arguments. @@ -49,16 +63,16 @@ Module Interface classes. :func:`.open` uses the file's blksize (as obtained by :func:`os.stat`) if possible. -.. function:: open(file[, mode[, buffering[, encoding[, errors[, newline[, closefd=True]]]]]]) +.. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True) - Open *file* and return a stream. If the file cannot be opened, an - :exc:`IOError` is raised. + Open *file* and return a corresponding stream. If the file cannot be opened, + an :exc:`IOError` is raised. - *file* is either a string giving the name (and the path if the file isn't in - the current working directory) of the file to be opened or a file - descriptor of the file to be opened. (If a file descriptor is given, - for example, from :func:`os.fdopen`, it is closed when the returned - I/O object is closed, unless *closefd* is set to ``False``.) + *file* is either a string giving the pathname (absolute or + relative to the current working directory) of the file to be opened or + an integer file descriptor of the file to be wrapped. (If a file descriptor + is given, it is closed when the returned I/O object is closed, unless + *closefd* is set to ``False``.) *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. @@ -88,11 +102,11 @@ Module Interface Python distinguishes between files opened in binary and text modes, even when the underlying operating system doesn't. Files opened in binary mode - (including ``'b'`` in the *mode* argument) return contents as ``bytes`` + (including ``'b'`` in the *mode* argument) return contents as :class:`bytes` objects without any decoding. In text mode (the default, or when ``'t'`` is included in the *mode* argument), the contents of the file are returned as - strings, the bytes having been first decoded using a platform-dependent - encoding or using the specified *encoding* if given. + :class:`unicode` strings, the bytes having been first decoded using a + platform-dependent encoding or using the specified *encoding* if given. *buffering* is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select @@ -111,19 +125,21 @@ Module Interface *encoding* is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform - dependent, but any encoding supported by Python can be used. See the - :mod:`codecs` module for the list of supported encodings. + dependent (whatever :func:`locale.getpreferredencoding` returns), but any + encoding supported by Python can be used. See the :mod:`codecs` module for + the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding - errors are to be handled. Pass ``'strict'`` to raise a :exc:`ValueError` - exception if there is an encoding error (the default of ``None`` has the same - effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding - errors can lead to data loss.) ``'replace'`` causes a replacement marker - (such as ``'?'``) to be inserted where there is malformed data. When - writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character - reference) or ``'backslashreplace'`` (replace with backslashed escape - sequences) can be used. Any other error handling name that has been - registered with :func:`codecs.register_error` is also valid. + errors are to be handled--this cannot be used in binary mode. Pass + ``'strict'`` to raise a :exc:`ValueError` exception if there is an encoding + error (the default of ``None`` has the same effect), or pass ``'ignore'`` to + ignore errors. (Note that ignoring encoding errors can lead to data loss.) + ``'replace'`` causes a replacement marker (such as ``'?'``) to be inserted + where there is malformed data. When writing, ``'xmlcharrefreplace'`` + (replace with the appropriate XML character reference) or + ``'backslashreplace'`` (replace with backslashed escape sequences) can be + used. Any other error handling name that has been registered with + :func:`codecs.register_error` is also valid. *newline* controls how universal newlines works (it only applies to text mode). It can be ``None``, ``''``, ``'\n'``, ``'\r'``, and ``'\r\n'``. It @@ -143,23 +159,26 @@ Module Interface the other legal values, any ``'\n'`` characters written are translated to the given string. - If *closefd* is ``False`` and a file descriptor rather than a - filename was given, the underlying file descriptor will be kept open - when the file is closed. If a filename is given *closefd* has no - effect but must be ``True`` (the default). - - The type of file object returned by the :func:`.open` function depends - on the mode. When :func:`.open` is used to open a file in a text mode - (``'w'``, ``'r'``, ``'wt'``, ``'rt'``, etc.), it returns a - :class:`TextIOWrapper`. When used to open a file in a binary mode, - the returned class varies: in read binary mode, it returns a - :class:`BufferedReader`; in write binary and append binary modes, it - returns a :class:`BufferedWriter`, and in read/write mode, it returns - a :class:`BufferedRandom`. - - It is also possible to use a string or bytearray as a file for both reading - and writing. For strings :class:`StringIO` can be used like a file opened in - a text mode, and for bytearrays a :class:`BytesIO` can be used like a + If *closefd* is ``False`` and a file descriptor rather than a filename was + given, the underlying file descriptor will be kept open when the file is + closed. If a filename is given *closefd* has no effect and must be ``True`` + (the default). + + The type of file object returned by the :func:`.open` function depends on the + mode. When :func:`.open` is used to open a file in a text mode (``'w'``, + ``'r'``, ``'wt'``, ``'rt'``, etc.), it returns a subclass of + :class:`TextIOBase` (specifically :class:`TextIOWrapper`). When used to open + a file in a binary mode with buffering, the returned class is a subclass of + :class:`BufferedIOBase`. The exact class varies: in read binary mode, it + returns a :class:`BufferedReader`; in write binary and append binary modes, + it returns a :class:`BufferedWriter`, and in read/write mode, it returns a + :class:`BufferedRandom`. When buffering is disabled, the raw stream, a + subclass of :class:`RawIOBase`, :class:`FileIO`, is returned. + + It is also possible to use an :class:`unicode` or :class:`bytes` string + as a file for both reading and writing. For :class:`unicode` strings + :class:`StringIO` can be used like a file opened in text mode, + and for :class:`bytes` a :class:`BytesIO` can be used like a file opened in a binary mode. @@ -203,22 +222,25 @@ I/O Base Classes support are called. The basic type used for binary data read from or written to a file is - :class:`bytes`. :class:`bytearray`\s are accepted too, and in some cases - (such as :class:`readinto`) required. Text I/O classes work with - :class:`str` data. + :class:`bytes` (also known as :class:`str`). :class:`bytearray`\s are + accepted too, and in some cases (such as :class:`readinto`) required. + Text I/O classes work with :class:`unicode` data. Note that calling any method (even inquiries) on a closed stream is undefined. Implementations may raise :exc:`IOError` in this case. IOBase (and its subclasses) support the iterator protocol, meaning that an :class:`IOBase` object can be iterated over yielding the lines in a stream. + Lines are defined slightly differently depending on whether the stream is + a binary stream (yielding :class:`bytes`), or a text stream (yielding + :class:`unicode` strings). See :meth:`readline` below. IOBase is also a context manager and therefore supports the :keyword:`with` statement. In this example, *file* is closed after the :keyword:`with` statement's suite is finished---even if an exception occurs:: - with open('spam.txt', 'w') as file: - file.write('Spam and eggs!') + with io.open('spam.txt', 'w') as file: + file.write(u'Spam and eggs!') :class:`IOBase` provides these data attributes and methods: @@ -226,7 +248,7 @@ I/O Base Classes Flush and close this stream. This method has no effect if the file is already closed. Once the file is closed, any operation on the file - (e.g. reading or writing) will raise an :exc:`ValueError`. + (e.g. reading or writing) will raise a :exc:`ValueError`. As a convenience, it is allowed to call this method more than once; only the first call, however, will have an effect. @@ -256,7 +278,7 @@ I/O Base Classes Return ``True`` if the stream can be read from. If False, :meth:`read` will raise :exc:`IOError`. - .. method:: readline([limit]) + .. method:: readline(limit=-1) Read and return one line from the stream. If *limit* is specified, at most *limit* bytes will be read. @@ -265,24 +287,30 @@ I/O Base Classes the *newlines* argument to :func:`.open` can be used to select the line terminator(s) recognized. - .. method:: readlines([hint]) + .. method:: readlines(hint=-1) Read and return a list of lines from the stream. *hint* can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds *hint*. - .. method:: seek(offset[, whence]) + .. method:: seek(offset, whence=SEEK_SET) Change the stream position to the given byte *offset*. *offset* is interpreted relative to the position indicated by *whence*. Values for *whence* are: - * ``0`` -- start of the stream (the default); *offset* should be zero or positive - * ``1`` -- current stream position; *offset* may be negative - * ``2`` -- end of the stream; *offset* is usually negative + * :data:`SEEK_SET` or ``0`` -- start of the stream (the default); + *offset* should be zero or positive + * :data:`SEEK_CUR` or ``1`` -- current stream position; *offset* may + be negative + * :data:`SEEK_END` or ``2`` -- end of the stream; *offset* is usually + negative Return the new absolute position. + .. versionadded:: 2.7 + The ``SEEK_*`` constants + .. method:: seekable() Return ``True`` if the stream supports random access. If ``False``, @@ -292,7 +320,7 @@ I/O Base Classes Return the current stream position. - .. method:: truncate([size]) + .. method:: truncate(size=None) Resize the stream to the given *size* in bytes (or the current position if *size* is not specified). The current stream position isn't changed. @@ -318,15 +346,23 @@ I/O Base Classes Base class for raw binary I/O. It inherits :class:`IOBase`. There is no public constructor. + Raw binary I/O typically provides low-level access to an underlying OS + device or API, and does not try to encapsulate it in high-level primitives + (this is left to Buffered I/O and Text I/O, described later in this page). + In addition to the attributes and methods from :class:`IOBase`, RawIOBase provides the following methods: - .. method:: read([n]) + .. method:: read(n=-1) - Read and return all the bytes from the stream until EOF, or if *n* is - specified, up to *n* bytes. Only one system call is ever made. An empty - bytes object is returned on EOF; ``None`` is returned if the object is set - not to block and has no data to read. + Read up to *n* bytes from the object and return them. As a convenience, + if *n* is unspecified or -1, :meth:`readall` is called. Otherwise, + only one system call is ever made. Fewer than *n* bytes may be + returned if the operating system call returns fewer than *n* bytes. + + If 0 bytes are returned, and *n* was not 0, this indicates end of file. + If the object is in non-blocking mode and no bytes are available, + ``None`` is returned. .. method:: readall() @@ -335,38 +371,65 @@ I/O Base Classes .. method:: readinto(b) - Read up to len(b) bytes into bytearray *b* and return the number of bytes - read. + Read up to len(b) bytes into bytearray *b* and return the number + of bytes read. If the object is in non-blocking mode and no + bytes are available, ``None`` is returned. .. method:: write(b) Write the given bytes or bytearray object, *b*, to the underlying raw - stream and return the number of bytes written (This is never less than - ``len(b)``, since if the write fails, an :exc:`IOError` will be raised). + stream and return the number of bytes written. This can be less than + ``len(b)``, depending on specifics of the underlying raw stream, and + especially if it is in non-blocking mode. ``None`` is returned if the + raw stream is set not to block and no single byte could be readily + written to it. .. class:: BufferedIOBase - Base class for streams that support buffering. It inherits :class:`IOBase`. - There is no public constructor. + Base class for binary streams that support some kind of buffering. + It inherits :class:`IOBase`. There is no public constructor. - The main difference with :class:`RawIOBase` is that the :meth:`read` method - supports omitting the *size* argument, and does not have a default - implementation that defers to :meth:`readinto`. + The main difference with :class:`RawIOBase` is that methods :meth:`read`, + :meth:`readinto` and :meth:`write` will try (respectively) to read as much + input as requested or to consume all given output, at the expense of + making perhaps more than one system call. - In addition, :meth:`read`, :meth:`readinto`, and :meth:`write` may raise - :exc:`BlockingIOError` if the underlying raw stream is in non-blocking mode - and not ready; unlike their raw counterparts, they will never return - ``None``. + In addition, those methods can raise :exc:`BlockingIOError` if the + underlying raw stream is in non-blocking mode and cannot take or give + enough data; unlike their :class:`RawIOBase` counterparts, they will + never return ``None``. - A typical implementation should not inherit from a :class:`RawIOBase` - implementation, but wrap one like :class:`BufferedWriter` and - :class:`BufferedReader`. + Besides, the :meth:`read` method does not have a default + implementation that defers to :meth:`readinto`. + + A typical :class:`BufferedIOBase` implementation should not inherit from a + :class:`RawIOBase` implementation, but wrap one, like + :class:`BufferedWriter` and :class:`BufferedReader` do. - :class:`BufferedIOBase` provides or overrides these methods in addition to + :class:`BufferedIOBase` provides or overrides these members in addition to those from :class:`IOBase`: - .. method:: read([n]) + .. attribute:: raw + + The underlying raw stream (a :class:`RawIOBase` instance) that + :class:`BufferedIOBase` deals with. This is not part of the + :class:`BufferedIOBase` API and may not exist on some implementations. + + .. method:: detach() + + Separate the underlying raw stream from the buffer and return it. + + After the raw stream has been detached, the buffer is in an unusable + state. + + Some buffers, like :class:`BytesIO`, do not have the concept of a single + raw stream to return from this method. They raise + :exc:`UnsupportedOperation`. + + .. versionadded:: 2.7 + + .. method:: read(n=-1) Read and return up to *n* bytes. If the argument is omitted, ``None``, or negative, data is read and returned until EOF is reached. An empty bytes @@ -378,8 +441,15 @@ I/O Base Classes one raw read will be issued, and a short result does not imply that EOF is imminent. - A :exc:`BlockingIOError` is raised if the underlying raw stream has no - data at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in + non blocking-mode, and has no data available at the moment. + + .. method:: read1(n=-1) + + Read and return up to *n* bytes, with at most one call to the underlying + raw stream's :meth:`~RawIOBase.read` method. This can be useful if you + are implementing your own buffering on top of a :class:`BufferedIOBase` + object. .. method:: readinto(b) @@ -387,35 +457,47 @@ I/O Base Classes read. Like :meth:`read`, multiple reads may be issued to the underlying raw - stream, unless the latter is 'interactive.' + stream, unless the latter is 'interactive'. - A :exc:`BlockingIOError` is raised if the underlying raw stream has no - data at the moment. + A :exc:`BlockingIOError` is raised if the underlying raw stream is in + non blocking-mode, and has no data available at the moment. .. method:: write(b) - Write the given bytes or bytearray object, *b*, to the underlying raw - stream and return the number of bytes written (never less than ``len(b)``, - since if the write fails an :exc:`IOError` will be raised). + Write the given bytes or bytearray object, *b* and return the number + of bytes written (never less than ``len(b)``, since if the write fails + an :exc:`IOError` will be raised). Depending on the actual + implementation, these bytes may be readily written to the underlying + stream, or held in a buffer for performance and latency reasons. - A :exc:`BlockingIOError` is raised if the buffer is full, and the - underlying raw stream cannot accept more data at the moment. + When in non-blocking mode, a :exc:`BlockingIOError` is raised if the + data needed to be written to the raw stream but it couldn't accept + all the data without blocking. Raw File I/O ------------ -.. class:: FileIO(name[, mode]) +.. class:: FileIO(name, mode='r', closefd=True) + + :class:`FileIO` represents an OS-level file containing bytes data. + It implements the :class:`RawIOBase` interface (and therefore the + :class:`IOBase` interface, too). + + The *name* can be one of two things: - :class:`FileIO` represents a file containing bytes data. It implements - the :class:`RawIOBase` interface (and therefore the :class:`IOBase` - interface, too). + * a string representing the path to the file which will be opened; + * an integer representing the number of an existing OS-level file descriptor + to which the resulting :class:`FileIO` object will give access. The *mode* can be ``'r'``, ``'w'`` or ``'a'`` for reading (default), writing, or appending. The file will be created if it doesn't exist when opened for writing or appending; it will be truncated when opened for writing. Add a ``'+'`` to the mode to allow simultaneous reading and writing. + The :meth:`read` (when called with a positive argument), :meth:`readinto` + and :meth:`write` methods on this class will only make one system call. + In addition to the attributes and methods from :class:`IOBase` and :class:`RawIOBase`, :class:`FileIO` provides the following data attributes and methods: @@ -429,38 +511,19 @@ Raw File I/O The file name. This is the file descriptor of the file when no name is given in the constructor. - .. method:: read([n]) - - Read and return at most *n* bytes. Only one system call is made, so it is - possible that less data than was requested is returned. Use :func:`len` - on the returned bytes object to see how many bytes were actually returned. - (In non-blocking mode, ``None`` is returned when no data is available.) - - .. method:: readall() - - Read and return the entire file's contents in a single bytes object. As - much as immediately available is returned in non-blocking mode. If the - EOF has been reached, ``b''`` is returned. - - .. method:: write(b) - - Write the bytes or bytearray object, *b*, to the file, and return - the number actually written. Only one system call is made, so it - is possible that only some of the data is written. - - Note that the inherited ``readinto()`` method should not be used on - :class:`FileIO` objects. - Buffered Streams ---------------- +Buffered I/O streams provide a higher-level interface to an I/O device +than raw I/O does. + .. class:: BytesIO([initial_bytes]) A stream implementation using an in-memory bytes buffer. It inherits :class:`BufferedIOBase`. - The argument *initial_bytes* is an optional initial bytearray. + The argument *initial_bytes* is an optional initial :class:`bytes`. :class:`BytesIO` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: @@ -474,10 +537,13 @@ Buffered Streams In :class:`BytesIO`, this is the same as :meth:`read`. -.. class:: BufferedReader(raw[, buffer_size]) +.. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) - A buffer for a readable, sequential :class:`RawIOBase` object. It inherits - :class:`BufferedIOBase`. + A buffer providing higher-level access to a readable, sequential + :class:`RawIOBase` object. It inherits :class:`BufferedIOBase`. + When reading data from this object, a larger amount of data may be + requested from the underlying raw stream, and kept in an internal buffer. + The buffered data can then be returned directly on subsequent reads. The constructor creates a :class:`BufferedReader` for the given readable *raw* stream and *buffer_size*. If *buffer_size* is omitted, @@ -488,11 +554,9 @@ Buffered Streams .. method:: peek([n]) - Return 1 (or *n* if specified) bytes from a buffer without advancing the - position. Only a single read on the raw stream is done to satisfy the - call. The number of bytes returned may be less than requested since at - most all the buffer's bytes from the current position to the end are - returned. + Return bytes from the stream without advancing the position. At most one + single read on the raw stream is done to satisfy the call. The number of + bytes returned may be less or more than requested. .. method:: read([n]) @@ -506,15 +570,24 @@ Buffered Streams Otherwise, one raw stream read call is made. -.. class:: BufferedWriter(raw[, buffer_size[, max_buffer_size]]) +.. class:: BufferedWriter(raw, buffer_size=DEFAULT_BUFFER_SIZE) - A buffer for a writeable sequential RawIO object. It inherits - :class:`BufferedIOBase`. + A buffer providing higher-level access to a writeable, sequential + :class:`RawIOBase` object. It inherits :class:`BufferedIOBase`. + When writing to this object, data is normally held into an internal + buffer. The buffer will be written out to the underlying :class:`RawIOBase` + object under various conditions, including: + + * when the buffer gets too small for all pending data; + * when :meth:`flush()` is called; + * when a :meth:`seek()` is requested (for :class:`BufferedRandom` objects); + * when the :class:`BufferedWriter` object is closed or destroyed. The constructor creates a :class:`BufferedWriter` for the given writeable *raw* stream. If the *buffer_size* is not given, it defaults to - :data:`DEAFULT_BUFFER_SIZE`. If *max_buffer_size* is omitted, it defaults to - twice the buffer size. + :data:`DEFAULT_BUFFER_SIZE`. + + A third argument, *max_buffer_size*, is supported, but unused and deprecated. :class:`BufferedWriter` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: @@ -526,35 +599,41 @@ Buffered Streams .. method:: write(b) - Write the bytes or bytearray object, *b*, onto the raw stream and return - the number of bytes written. A :exc:`BlockingIOError` is raised when the - raw stream blocks. + Write the bytes or bytearray object, *b* and return the number of bytes + written. When in non-blocking mode, a :exc:`BlockingIOError` is raised + if the buffer needs to be written out but the raw stream blocks. -.. class:: BufferedRWPair(reader, writer[, buffer_size[, max_buffer_size]]) +.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) - A combined buffered writer and reader object for a raw stream that can be - written to and read from. It has and supports both :meth:`read`, :meth:`write`, - and their variants. This is useful for sockets and two-way pipes. - It inherits :class:`BufferedIOBase`. + A buffered I/O object giving a combined, higher-level access to two + sequential :class:`RawIOBase` objects: one readable, the other writeable. + It is useful for pairs of unidirectional communication channels + (pipes, for instance). It inherits :class:`BufferedIOBase`. *reader* and *writer* are :class:`RawIOBase` objects that are readable and writeable respectively. If the *buffer_size* is omitted it defaults to - :data:`DEFAULT_BUFFER_SIZE`. The *max_buffer_size* (for the buffered writer) - defaults to twice the buffer size. + :data:`DEFAULT_BUFFER_SIZE`. + + A fourth argument, *max_buffer_size*, is supported, but unused and + deprecated. - :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods. + :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods + except for :meth:`~BufferedIOBase.detach`, which raises + :exc:`UnsupportedOperation`. -.. class:: BufferedRandom(raw[, buffer_size[, max_buffer_size]]) +.. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffered interface to random access streams. It inherits - :class:`BufferedReader` and :class:`BufferedWriter`. + :class:`BufferedReader` and :class:`BufferedWriter`, and further supports + :meth:`seek` and :meth:`tell` functionality. The constructor creates a reader and writer for a seekable raw stream, given in the first argument. If the *buffer_size* is omitted it defaults to - :data:`DEFAULT_BUFFER_SIZE`. The *max_buffer_size* (for the buffered writer) - defaults to twice the buffer size. + :data:`DEFAULT_BUFFER_SIZE`. + + A third argument, *max_buffer_size*, is supported, but unused and deprecated. :class:`BufferedRandom` is capable of anything :class:`BufferedReader` or :class:`BufferedWriter` can do. @@ -565,10 +644,10 @@ Text I/O .. class:: TextIOBase - Base class for text streams. This class provides a character and line based - interface to stream I/O. There is no :meth:`readinto` method because - Python's character strings are immutable. It inherits :class:`IOBase`. - There is no public constructor. + Base class for text streams. This class provides an unicode character + and line based interface to stream I/O. There is no :meth:`readinto` + method because Python's :class:`unicode` strings are immutable. + It inherits :class:`IOBase`. There is no public constructor. :class:`TextIOBase` provides or overrides these data attributes and methods in addition to those from :class:`IOBase`: @@ -578,30 +657,55 @@ Text I/O The name of the encoding used to decode the stream's bytes into strings, and to encode strings into bytes. + .. attribute:: errors + + The error setting of the decoder or encoder. + .. attribute:: newlines A string, a tuple of strings, or ``None``, indicating the newlines - translated so far. + translated so far. Depending on the implementation and the initial + constructor flags, this may not be available. + + .. attribute:: buffer + + The underlying binary buffer (a :class:`BufferedIOBase` instance) that + :class:`TextIOBase` deals with. This is not part of the + :class:`TextIOBase` API and may not exist on some implementations. + + .. method:: detach() + + Separate the underlying binary buffer from the :class:`TextIOBase` and + return it. + + After the underlying buffer has been detached, the :class:`TextIOBase` is + in an unusable state. + + Some :class:`TextIOBase` implementations, like :class:`StringIO`, may not + have the concept of an underlying buffer and calling this method will + raise :exc:`UnsupportedOperation`. + + .. versionadded:: 2.7 .. method:: read(n) Read and return at most *n* characters from the stream as a single - :class:`str`. If *n* is negative or ``None``, reads to EOF. + :class:`unicode`. If *n* is negative or ``None``, reads until EOF. .. method:: readline() - Read until newline or EOF and return a single ``str``. If the stream is - already at EOF, an empty string is returned. + Read until newline or EOF and return a single ``unicode``. If the + stream is already at EOF, an empty string is returned. .. method:: write(s) - Write the string *s* to the stream and return the number of characters - written. + Write the :class:`unicode` string *s* to the stream and return the + number of characters written. -.. class:: TextIOWrapper(buffer[, encoding[, errors[, newline[, line_buffering]]]]) +.. class:: TextIOWrapper(buffer, encoding=None, errors=None, newline=None, line_buffering=False) - A buffered text stream over a :class:`BufferedIOBase` raw stream, *buffer*. + A buffered text stream over a :class:`BufferedIOBase` binary stream. It inherits :class:`TextIOBase`. *encoding* gives the name of the encoding that the stream will be decoded or @@ -630,32 +734,47 @@ Text I/O If *line_buffering* is ``True``, :meth:`flush` is implied when a call to write contains a newline character. - :class:`TextIOWrapper` provides these data attributes in addition to those of + :class:`TextIOWrapper` provides one attribute in addition to those of :class:`TextIOBase` and its parents: - .. attribute:: errors - - The encoding and decoding error setting. - .. attribute:: line_buffering Whether line buffering is enabled. -.. class:: StringIO([initial_value[, encoding[, errors[, newline]]]]) +.. class:: StringIO(initial_value=u'', newline=None) - An in-memory stream for text. It inherits :class:`TextIOWrapper`. + An in-memory stream for unicode text. It inherits :class:`TextIOWrapper`. - Create a new StringIO stream with an inital value, encoding, error handling, - and newline setting. See :class:`TextIOWrapper`\'s constructor for more - information. + The initial value of the buffer (an empty unicode string by default) can + be set by providing *initial_value*. The *newline* argument works like + that of :class:`TextIOWrapper`. The default is to do no newline + translation. :class:`StringIO` provides this method in addition to those from :class:`TextIOWrapper` and its parents: .. method:: getvalue() - Return a ``str`` containing the entire contents of the buffer. + Return a ``unicode`` containing the entire contents of the buffer at any + time before the :class:`StringIO` object's :meth:`close` method is + called. + + Example usage:: + + import io + + output = io.StringIO() + output.write(u'First line.\n') + output.write(u'Second line.\n') + + # Retrieve file contents -- this will be + # u'First line.\nSecond line.\n' + contents = output.getvalue() + + # Close object and discard memory buffer -- + # .getvalue() will now raise an exception. + output.close() .. class:: IncrementalNewlineDecoder @@ -663,3 +782,68 @@ Text I/O A helper codec that decodes newlines for universal newlines mode. It inherits :class:`codecs.IncrementalDecoder`. + +Advanced topics +--------------- + +Here we will discuss several advanced topics pertaining to the concrete +I/O implementations described above. + +Performance +^^^^^^^^^^^ + +Binary I/O +"""""""""" + +By reading and writing only large chunks of data even when the user asks +for a single byte, buffered I/O is designed to hide any inefficiency in +calling and executing the operating system's unbuffered I/O routines. The +gain will vary very much depending on the OS and the kind of I/O which is +performed (for example, on some contemporary OSes such as Linux, unbuffered +disk I/O can be as fast as buffered I/O). The bottom line, however, is +that buffered I/O will offer you predictable performance regardless of the +platform and the backing device. Therefore, it is most always preferable to +use buffered I/O rather than unbuffered I/O. + +Text I/O +"""""""" + +Text I/O over a binary storage (such as a file) is significantly slower than +binary I/O over the same storage, because it implies conversions from +unicode to binary data using a character codec. This can become noticeable +if you handle huge amounts of text data (for example very large log files). +Also, :meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both +quite slow due to the reconstruction algorithm used. + +:class:`StringIO`, however, is a native in-memory unicode container and will +exhibit similar speed to :class:`BytesIO`. + +Multi-threading +^^^^^^^^^^^^^^^ + +:class:`FileIO` objects are thread-safe to the extent that the operating +system calls (such as ``read(2)`` under Unix) they are wrapping are thread-safe +too. + +Binary buffered objects (instances of :class:`BufferedReader`, +:class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) +protect their internal structures using a lock; it is therefore safe to call +them from multiple threads at once. + +:class:`TextIOWrapper` objects are not thread-safe. + +Reentrancy +^^^^^^^^^^ + +Binary buffered objects (instances of :class:`BufferedReader`, +:class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) +are not reentrant. While reentrant calls will not happen in normal situations, +they can arise if you are doing I/O in a :mod:`signal` handler. If it is +attempted to enter a buffered object again while already being accessed +*from the same thread*, then a :exc:`RuntimeError` is raised. + +The above implicitly extends to text files, since the :func:`open()` +function will wrap a buffered object inside a :class:`TextIOWrapper`. This +includes standard streams and therefore affects the built-in function +:func:`print()` as well. + diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index 33630a5..b7f407d 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -38,7 +38,7 @@ operator can be mapped across two vectors to form an efficient dot-product: ================== ================= ================================================= ========================================= Iterator Arguments Results Example ================== ================= ================================================= ========================================= -:func:`count` start start, start+1, start+2, ... ``count(10) --> 10 11 12 13 14 ...`` +:func:`count` start, [step] start, start+step, start+2*step, ... ``count(10) --> 10 11 12 13 14 ...`` :func:`cycle` p p0, p1, ... plast, p0, p1, ... ``cycle('ABCD') --> A B C D A B C D ...`` :func:`repeat` elem [,n] elem, elem, elem, ... endlessly or up to n times ``repeat(10, 3) --> 10 10 10`` ================== ================= ================================================= ========================================= @@ -49,6 +49,7 @@ Iterator Arguments Results Iterator Arguments Results Example ==================== ============================ ================================================= ============================================================= :func:`chain` p, q, ... p0, p1, ... plast, q0, q1, ... ``chain('ABC', 'DEF') --> A B C D E F`` +:func:`compress` data, selectors (d[0] if s[0]), (d[1] if s[1]), ... ``compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F`` :func:`dropwhile` pred, seq seq[n], seq[n+1], starting when pred fails ``dropwhile(lambda x: x<5, [1,4,6,4,1]) --> 6 4 1`` :func:`groupby` iterable[, keyfunc] sub-iterators grouped by value of keyfunc(v) :func:`ifilter` pred, seq elements of seq where pred(elem) is True ``ifilter(lambda x: x%2, range(10)) --> 1 3 5 7 9`` @@ -70,10 +71,11 @@ Iterator Arguments Resu :func:`product` p, q, ... [repeat=1] cartesian product, equivalent to a nested for-loop :func:`permutations` p[, r] r-length tuples, all possible orderings, no repeated elements :func:`combinations` p, r r-length tuples, in sorted order, no repeated elements -| +:func:`combinations_with_replacement` p, r r-length tuples, in sorted order, with repeated elements ``product('ABCD', repeat=2)`` ``AA AB AC AD BA BB BC BD CA CB CC CD DA DB DC DD`` ``permutations('ABCD', 2)`` ``AB AC AD BA BC BD CA CB CD DA DB DC`` ``combinations('ABCD', 2)`` ``AB AC AD BC BD CD`` +``combinations_with_replacement('ABCD', 2)`` ``AA AB AC AD BB BC BD CC CD DD`` ============================================== ==================== ============================================================= @@ -166,19 +168,87 @@ loops that truncate the stream. .. versionadded:: 2.6 -.. function:: count([n]) +.. function:: combinations_with_replacement(iterable, r) - Make an iterator that returns consecutive integers starting with *n*. If not - specified *n* defaults to zero. Often used as an argument to :func:`imap` to - generate consecutive data points. Also, used with :func:`izip` to add sequence - numbers. Equivalent to:: + Return *r* length subsequences of elements from the input *iterable* + allowing individual elements to be repeated more than once. - def count(n=0): + Combinations are emitted in lexicographic sort order. So, if the + input *iterable* is sorted, the combination tuples will be produced + in sorted order. + + Elements are treated as unique based on their position, not on their + value. So if the input elements are unique, the generated combinations + will also be unique. + + Equivalent to:: + + def combinations_with_replacement(iterable, r): + # combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC + pool = tuple(iterable) + n = len(pool) + if not n and r: + return + indices = [0] * r + yield tuple(pool[i] for i in indices) + while True: + for i in reversed(range(r)): + if indices[i] != n - 1: + break + else: + return + indices[i:] = [indices[i] + 1] * (r - i) + yield tuple(pool[i] for i in indices) + + The code for :func:`combinations_with_replacement` can be also expressed as + a subsequence of :func:`product` after filtering entries where the elements + are not in sorted order (according to their position in the input pool):: + + def combinations_with_replacement(iterable, r): + pool = tuple(iterable) + n = len(pool) + for indices in product(range(n), repeat=r): + if sorted(indices) == list(indices): + yield tuple(pool[i] for i in indices) + + The number of items returned is ``(n+r-1)! / r! / (n-1)!`` when ``n > 0``. + + .. versionadded:: 2.7 + +.. function:: compress(data, selectors) + + Make an iterator that filters elements from *data* returning only those that + have a corresponding element in *selectors* that evaluates to ``True``. + Stops when either the *data* or *selectors* iterables has been exhausted. + Equivalent to:: + + def compress(data, selectors): + # compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F + return (d for d, s in izip(data, selectors) if s) + + .. versionadded:: 2.7 + + +.. function:: count(start=0, step=1) + + Make an iterator that returns evenly spaced values starting with *n*. Often + used as an argument to :func:`imap` to generate consecutive data points. + Also, used with :func:`izip` to add sequence numbers. Equivalent to:: + + def count(start=0, step=1): # count(10) --> 10 11 12 13 14 ... + # count(2.5, 0.5) -> 2.5 3.0 3.5 ... + n = start while True: yield n - n += 1 + n += step + + When counting with floating point numbers, better accuracy can sometimes be + achieved by substituting multiplicative code such as: ``(start + step * i + for i in count())``. + .. versionchanged:: 2.7 + added *step* argument and allowed non-integer arguments. .. function:: cycle(iterable) @@ -573,43 +643,6 @@ loops that truncate the stream. .. versionadded:: 2.4 -.. _itertools-example: - -Examples --------- - -The following examples show common uses for each tool and demonstrate ways they -can be combined. - -.. doctest:: - - >>> # Show a dictionary sorted and grouped by value - >>> from operator import itemgetter - >>> d = dict(a=1, b=2, c=1, d=2, e=1, f=2, g=3) - >>> di = sorted(d.iteritems(), key=itemgetter(1)) - >>> for k, g in groupby(di, key=itemgetter(1)): - ... print k, map(itemgetter(0), g) - ... - 1 ['a', 'c', 'e'] - 2 ['b', 'd', 'f'] - 3 ['g'] - - >>> # Find runs of consecutive numbers using groupby. The key to the solution - >>> # is differencing with a range so that consecutive numbers all appear in - >>> # same group. - >>> data = [ 1, 4,5,6, 10, 15,16,17,18, 22, 25,26,27,28] - >>> for k, g in groupby(enumerate(data), lambda (i,x):i-x): - ... print map(itemgetter(1), g) - ... - [1] - [4, 5, 6] - [10] - [15, 16, 17, 18] - [22] - [25, 26, 27, 28] - - - .. _itertools-recipes: Recipes @@ -638,12 +671,12 @@ which incur interpreter overhead. def consume(iterator, n): "Advance the iterator n-steps ahead. If n is none, consume entirely." - # The technique uses objects that consume iterators at C speed. + # Use functions that consume iterators at C speed. if n is None: # feed the entire iterator into a zero-length deque collections.deque(iterator, maxlen=0) else: - # advance to the emtpy slice starting at position n + # advance to the empty slice starting at position n next(islice(iterator, n, n), None) def nth(iterable, n, default=None): @@ -705,28 +738,6 @@ which incur interpreter overhead. pending -= 1 nexts = cycle(islice(nexts, pending)) - def compress(data, selectors): - "compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F" - return (d for d, s in izip(data, selectors) if s) - - def combinations_with_replacement(iterable, r): - "combinations_with_replacement('ABC', 2) --> AA AB AC BB BC CC" - # number items returned: (n+r-1)! / r! / (n-1)! - pool = tuple(iterable) - n = len(pool) - if not n and r: - return - indices = [0] * r - yield tuple(pool[i] for i in indices) - while True: - for i in reversed(range(r)): - if indices[i] != n - 1: - break - else: - return - indices[i:] = [indices[i] + 1] * (r - i) - yield tuple(pool[i] for i in indices) - def powerset(iterable): "powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)" s = list(iterable) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 8fa790e..be54eec 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -7,7 +7,7 @@ .. sectionauthor:: Bob Ippolito <bob@redivi.com> .. versionadded:: 2.6 -JSON (JavaScript Object Notation) <http://json.org> is a subset of JavaScript +`JSON (JavaScript Object Notation) <http://json.org>`_ is a subset of JavaScript syntax (ECMA-262 3rd edition) used as a lightweight data interchange format. :mod:`json` exposes an API familiar to users of the standard library @@ -154,7 +154,7 @@ Basic Usage To use a custom :class:`JSONEncoder` subclass (e.g. one that overrides the :meth:`default` method to serialize additional types), specify it with the - *cls* kwarg. + *cls* kwarg; otherwise :class:`JSONEncoder` is used. .. function:: dumps(obj[, skipkeys[, ensure_ascii[, check_circular[, allow_nan[, cls[, indent[, separators[, encoding[, default[, **kw]]]]]]]]]]) @@ -166,7 +166,7 @@ Basic Usage :func:`dump`. -.. function:: load(fp[, encoding[, cls[, object_hook[, parse_float[, parse_int[, parse_constant[, **kw]]]]]]]) +.. function:: load(fp[, encoding[, cls[, object_hook[, parse_float[, parse_int[, parse_constant[, object_pairs_hook[, **kw]]]]]]]]) Deserialize *fp* (a ``.read()``-supporting file-like object containing a JSON document) to a Python object. @@ -182,6 +182,17 @@ Basic Usage *object_hook* will be used instead of the :class:`dict`. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting). + *object_pairs_hook* is an optional function that will be called with the + result of any object literal decoded with an ordered list of pairs. The + return value of *object_pairs_hook* will be used instead of the + :class:`dict`. This feature can be used to implement custom decoders that + rely on the order that the key and value pairs are decoded (for example, + :func:`collections.OrderedDict` will remember the order of insertion). If + *object_hook* is also defined, the *object_pairs_hook* takes priority. + + .. versionchanged:: 2.7 + Added support for *object_pairs_hook*. + *parse_float*, if specified, will be called with the string of every JSON float to be decoded. By default, this is equivalent to ``float(num_str)``. This can be used to use another datatype or parser for JSON floats @@ -198,11 +209,11 @@ Basic Usage are encountered. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` - kwarg. Additional keyword arguments will be passed to the constructor of the - class. + kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments + will be passed to the constructor of the class. -.. function:: loads(s[, encoding[, cls[, object_hook[, parse_float[, parse_int[, parse_constant[, **kw]]]]]]]) +.. function:: loads(s[, encoding[, cls[, object_hook[, parse_float[, parse_int[, parse_constant[, object_pairs_hook[, **kw]]]]]]]]) Deserialize *s* (a :class:`str` or :class:`unicode` instance containing a JSON document) to a Python object. @@ -212,13 +223,13 @@ Basic Usage specified. Encodings that are not ASCII based (such as UCS-2) are not allowed and should be decoded to :class:`unicode` first. - The other arguments have the same meaning as in :func:`dump`. + The other arguments have the same meaning as in :func:`load`. Encoders and decoders --------------------- -.. class:: JSONDecoder([encoding[, object_hook[, parse_float[, parse_int[, parse_constant[, strict]]]]]]) +.. class:: JSONDecoder([encoding[, object_hook[, parse_float[, parse_int[, parse_constant[, strict[, object_pairs_hook]]]]]]]) Simple JSON decoder. @@ -259,6 +270,17 @@ Encoders and decoders :class:`dict`. This can be used to provide custom deserializations (e.g. to support JSON-RPC class hinting). + *object_pairs_hook*, if specified will be called with the result of every + JSON object decoded with an ordered list of pairs. The return value of + *object_pairs_hook* will be used instead of the :class:`dict`. This + feature can be used to implement custom decoders that rely on the order + that the key and value pairs are decoded (for example, + :func:`collections.OrderedDict` will remember the order of insertion). If + *object_hook* is also defined, the *object_pairs_hook* takes priority. + + .. versionchanged:: 2.7 + Added support for *object_pairs_hook*. + *parse_float*, if specified, will be called with the string of every JSON float to be decoded. By default, this is equivalent to ``float(num_str)``. This can be used to use another datatype or parser for JSON floats @@ -274,6 +296,11 @@ Encoders and decoders ``'false'``. This can be used to raise an exception if invalid JSON numbers are encountered. + If *strict* is ``False`` (``True`` is the default), then control characters + will be allowed inside strings. Control characters in this context are + those with character codes in the 0-31 range, including ``'\t'`` (tab), + ``'\n'``, ``'\r'`` and ``'\0'``. + .. method:: decode(s) @@ -338,7 +365,7 @@ Encoders and decoders encoders and decoders. Otherwise, it will be a :exc:`ValueError` to encode such floats. - If *sort_keys* is ``True`` (the default), then the output of dictionaries + If *sort_keys* is ``True`` (default ``False``), then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis. diff --git a/Doc/library/keyword.rst b/Doc/library/keyword.rst index 32a2d34..3b69d0b 100644 --- a/Doc/library/keyword.rst +++ b/Doc/library/keyword.rst @@ -20,3 +20,8 @@ This module allows a Python program to determine if a string is a keyword. keywords are defined to only be active when particular :mod:`__future__` statements are in effect, these will be included as well. + +.. seealso:: + + Latest version of the `keyword module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/keyword.py?view=markup>`_ diff --git a/Doc/library/linecache.rst b/Doc/library/linecache.rst index f3d8379..8ebee05 100644 --- a/Doc/library/linecache.rst +++ b/Doc/library/linecache.rst @@ -12,12 +12,17 @@ attempting to optimize internally, using a cache, the common case where many lines are read from a single file. This is used by the :mod:`traceback` module to retrieve source lines for inclusion in the formatted traceback. +.. seealso:: + + Latest version of the `linecache module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/linecache.py?view=markup>`_ + The :mod:`linecache` module defines the following functions: .. function:: getline(filename, lineno[, module_globals]) - Get line *lineno* from file named *filename*. This function will never throw an + Get line *lineno* from file named *filename*. This function will never raise an exception --- it will return ``''`` on errors (the terminating newline character will be included for lines that are found). diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 7b4a05f..85dad99 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -40,7 +40,7 @@ The :mod:`locale` module defines the following exception and functions: If *locale* is omitted or ``None``, the current setting for *category* is returned. - :func:`setlocale` is not thread safe on most systems. Applications typically + :func:`setlocale` is not thread-safe on most systems. Applications typically start with a call of :: import locale @@ -165,7 +165,7 @@ The :mod:`locale` module defines the following exception and functions: .. data:: D_T_FMT Get a string that can be used as a format string for :func:`strftime` to - represent time and date in a locale-specific way. + represent date and time in a locale-specific way. .. data:: D_FMT @@ -250,12 +250,17 @@ The :mod:`locale` module defines the following exception and functions: .. data:: ERA_D_T_FMT - Get a format string for :func:`strftime` to represent dates and times in a + Get a format string for :func:`strftime` to represent date and time in a locale-specific era-based way. .. data:: ERA_D_FMT - Get a format string for :func:`strftime` to represent time in a + Get a format string for :func:`strftime` to represent a date in a + locale-specific era-based way. + + .. data:: ERA_T_FMT + + Get a format string for :func:`strftime` to represent a time in a locale-specific era-based way. .. data:: ALT_DIGITS @@ -554,7 +559,7 @@ catalogs, and the C library's search algorithms for locating message catalogs. Python applications should normally find no need to invoke these functions, and should use :mod:`gettext` instead. A known exception to this rule are -applications that link use additional C libraries which internally invoke +applications that link with additional C libraries which internally invoke :cfunc:`gettext` or :func:`dcgettext`. For these applications, it may be necessary to bind the text domain, so that the libraries can properly locate their message catalogs. diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 6674dfa..6b6c7ea 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -321,28 +321,46 @@ order:: "%(asctime)s - %(levelname)s - %(message)s" +Formatters use a user-configurable function to convert the creation time of a +record to a tuple. By default, :func:`time.localtime` is used; to change this +for a particular formatter instance, set the ``converter`` attribute of the +instance to a function with the same signature as :func:`time.localtime` or +:func:`time.gmtime`. To change it for all formatters, for example if you want +all logging times to be shown in GMT, set the ``converter`` attribute in the +Formatter class (to ``time.gmtime`` for GMT display). + Configuring Logging ^^^^^^^^^^^^^^^^^^^ -Programmers can configure logging either by creating loggers, handlers, and -formatters explicitly in a main module with the configuration methods listed -above (using Python code), or by creating a logging config file. The following -code is an example of configuring a very simple logger, a console handler, and a -simple formatter in a Python module:: +Programmers can configure logging in three ways: + +1. Creating loggers, handlers, and formatters explicitly using Python + code that calls the configuration methods listed above. +2. Creating a logging config file and reading it using the :func:`fileConfig` + function. +3. Creating a dictionary of configuration information and passing it + to the :func:`dictConfig` function. + +The following example configures a very simple logger, a console +handler, and a simple formatter using Python code:: import logging # create logger logger = logging.getLogger("simple_example") logger.setLevel(logging.DEBUG) + # create console handler and set level to debug ch = logging.StreamHandler() ch.setLevel(logging.DEBUG) + # create formatter formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") + # add formatter to ch ch.setFormatter(formatter) + # add ch to logger logger.addHandler(ch) @@ -425,6 +443,52 @@ You can see that the config file approach has a few advantages over the Python code approach, mainly separation of configuration and code and the ability of noncoders to easily modify the logging properties. +Note that the class names referenced in config files need to be either relative +to the logging module, or absolute values which can be resolved using normal +import mechanisms. Thus, you could use either :class:`handlers.WatchedFileHandler` +(relative to the logging module) or :class:`mypackage.mymodule.MyHandler` (for a +class defined in package :mod:`mypackage` and module :mod:`mymodule`, where +:mod:`mypackage` is available on the Python import path). + +.. versionchanged:: 2.7 + +In Python 2.7, a new means of configuring logging has been introduced, using +dictionaries to hold configuration information. This provides a superset of the +functionality of the config-file-based approach outlined above, and is the +recommended configuration method for new applications and deployments. Because +a Python dictionary is used to hold configuration information, and since you +can populate that dictionary using different means, you have more options for +configuration. For example, you can use a configuration file in JSON format, +or, if you have access to YAML processing functionality, a file in YAML +format, to populate the configuration dictionary. Or, of course, you can +construct the dictionary in Python code, receive it in pickled form over a +socket, or use whatever approach makes sense for your application. + +Here's an example of the same configuration as above, in YAML format for +the new dictionary-based approach:: + + version: 1 + formatters: + simple: + format: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s + handlers: + console: + class: logging.StreamHandler + level: DEBUG + formatter: simple + stream: ext://sys.stdout + loggers: + simpleExample: + level: DEBUG + handlers: [console] + propagate: no + root: + level: DEBUG + handlers: [console] + +For more information about logging using a dictionary, see +:ref:`logging-config-api`. + .. _library-config: Configuring Logging for a Library @@ -466,6 +530,17 @@ should have the desired effect. If an organisation produces a number of libraries, then the logger name specified can be "orgname.foo" rather than just "foo". +**PLEASE NOTE:** It is strongly advised that you *do not add any handlers other +than* :class:`NullHandler` *to your library's loggers*. This is because the +configuration of handlers is the prerogative of the application developer who +uses your library. The application developer knows their target audience and +what handlers are most appropriate for their application: if you add handlers +"under the hood", you might well interfere with their ability to carry out +unit tests and deliver logs which suit their requirements. + +.. versionadded:: 2.7 + The :class:`NullHandler` class. + Logging Levels -------------- @@ -523,6 +598,22 @@ decides to actually dispatch an event, the :meth:`emit` method is used to send the message to its destination. Most user-defined subclasses of :class:`Handler` will need to override this :meth:`emit`. +.. _custom-levels: + +Custom Levels +^^^^^^^^^^^^^ + +Defining your own levels is possible, but should not be necessary, as the +existing levels have been chosen on the basis of practical experience. +However, if you are convinced that you need custom levels, great care should +be exercised when doing this, and it is possibly *a very bad idea to define +custom levels if you are developing a library*. That's because if multiple +library authors all define their own custom levels, there is a chance that +the logging output from such multiple libraries used together will be +difficult for the using developer to control and/or interpret, because a +given numeric value might mean different things for different libraries. + + Useful Handlers --------------- @@ -571,7 +662,16 @@ provided: name. This handler is only useful on Unix-like systems; Windows does not support the underlying mechanism used. -The :class:`StreamHandler` and :class:`FileHandler` +#. :ref:`null-handler` instances do nothing with error messages. They are used + by library developers who want to use logging, but want to avoid the "No + handlers could be found for logger XXX" message which can be displayed if + the library user has not configured logging. See :ref:`library-config` for + more information. + +.. versionadded:: 2.7 + The :class:`NullHandler` class. + +The :class:`NullHandler`, :class:`StreamHandler` and :class:`FileHandler` classes are defined in the core logging package. The other handlers are defined in a sub- module, :mod:`logging.handlers`. (There is also another sub-module, :mod:`logging.config`, for configuration functionality.) @@ -650,7 +750,7 @@ functions. d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logging.warning("Protocol problem: %s", "connection reset", extra=d) - would print something like :: + would print something like:: 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset @@ -712,6 +812,14 @@ functions. Logs a message with level *level* on the root logger. The other arguments are interpreted as for :func:`debug`. + PLEASE NOTE: The above module-level functions which delegate to the root + logger should *not* be used in threads, in versions of Python earlier than + 2.7.1 and 3.2, unless at least one handler has been added to the root + logger *before* the threads are started. These convenience functions call + :func:`basicConfig` to ensure that at least one handler is available; in + earlier versions of Python, this can (under rare circumstances) lead to + handlers being added multiple times to the root logger, which can in turn + lead to multiple messages for the same event. .. function:: disable(lvl) @@ -733,6 +841,8 @@ functions. registered using this function, levels should be positive integers and they should increase in increasing order of severity. + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. .. function:: getLevelName(lvl) @@ -767,6 +877,13 @@ functions. .. versionchanged:: 2.4 Formerly, :func:`basicConfig` did not take any keyword arguments. + PLEASE NOTE: This function should be called from the main thread + before other threads are started. In versions of Python prior to + 2.7.1 and 3.2, if this function is called from multiple threads, + it is possible (in rare circumstances) that a handler will be added + to the root logger more than once, leading to unexpected results + such as messages being duplicated in the log. + The following keyword arguments are supported. +--------------+---------------------------------------------+ @@ -811,6 +928,7 @@ functions. which need to use custom logger behavior. + .. seealso:: :pep:`282` - A Logging System @@ -876,6 +994,16 @@ instantiated directly, but always through the module-level function :const:`NOTSET` is found, and that value is returned. +.. method:: Logger.getChild(suffix) + + Returns a logger which is a descendant to this logger, as determined by the suffix. + Thus, ``logging.getLogger('abc').getChild('def.ghi')`` would return the same + logger as would be returned by ``logging.getLogger('abc.def.ghi')``. This is a + convenience method, useful when the parent logger is named using e.g. ``__name__`` + rather than a literal string. + + .. versionadded:: 2.7 + .. method:: Logger.debug(msg[, *args[, **kwargs]]) Logs a message with level :const:`DEBUG` on this logger. The *msg* is the @@ -1245,6 +1373,10 @@ level of granularity you want to use in logging an application, it could be hard to manage if the number of :class:`Logger` instances becomes effectively unbounded. + +Using LoggerAdapters to impart contextual information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + An easy way in which you can pass contextual information to be output along with logging event information is to use the :class:`LoggerAdapter` class. This class is designed to look like a :class:`Logger`, so that you can call @@ -1352,6 +1484,80 @@ When this script is run, the output should look something like this:: The :class:`LoggerAdapter` class was not present in previous versions. +.. _filters-contextual: + +Using Filters to impart contextual information +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can also add contextual information to log output using a user-defined +:class:`Filter`. ``Filter`` instances are allowed to modify the ``LogRecords`` +passed to them, including adding additional attributes which can then be output +using a suitable format string, or if needed a custom :class:`Formatter`. + +For example in a web application, the request being processed (or at least, +the interesting parts of it) can be stored in a threadlocal +(:class:`threading.local`) variable, and then accessed from a ``Filter`` to +add, say, information from the request - say, the remote IP address and remote +user's username - to the ``LogRecord``, using the attribute names 'ip' and +'user' as in the ``LoggerAdapter`` example above. In that case, the same format +string can be used to get similar output to that shown above. Here's an example +script:: + + import logging + from random import choice + + class ContextFilter(logging.Filter): + """ + This is a filter which injects contextual information into the log. + + Rather than use actual contextual information, we just use random + data in this demo. + """ + + USERS = ['jim', 'fred', 'sheila'] + IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1'] + + def filter(self, record): + + record.ip = choice(ContextFilter.IPS) + record.user = choice(ContextFilter.USERS) + return True + + if __name__ == "__main__": + levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) + a1 = logging.LoggerAdapter(logging.getLogger("a.b.c"), + { "ip" : "123.231.231.123", "user" : "sheila" }) + logging.basicConfig(level=logging.DEBUG, + format="%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s") + a1 = logging.getLogger("a.b.c") + a2 = logging.getLogger("d.e.f") + + f = ContextFilter() + a1.addFilter(f) + a2.addFilter(f) + a1.debug("A debug message") + a1.info("An info message with %s", "some parameters") + for x in range(10): + lvl = choice(levels) + lvlname = logging.getLevelName(lvl) + a2.log(lvl, "A message at %s level with %d %s", lvlname, 2, "parameters") + +which, when run, produces something like:: + + 2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message + 2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with some parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters + 2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level with 2 parameters + 2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level with 2 parameters + + .. _multiple-processes: Logging to a single file from multiple processes @@ -1513,6 +1719,8 @@ these affect you, you can use an alternative serialization scheme by overriding the :meth:`makePickle` method and implementing your alternative there, as well as adapting the above script to use your alternative serialization. +.. _arbitrary-object-messages: + Using arbitrary objects as messages ----------------------------------- @@ -1681,12 +1889,14 @@ and :meth:`flush` methods). .. currentmodule:: logging -.. class:: StreamHandler([strm]) +.. class:: StreamHandler([stream]) - Returns a new instance of the :class:`StreamHandler` class. If *strm* is + Returns a new instance of the :class:`StreamHandler` class. If *stream* is specified, the instance will use it for logging output; otherwise, *sys.stderr* will be used. + .. versionchanged:: 2.7 + The ``stream`` parameter was called ``strm`` in earlier versions. .. method:: emit(record) @@ -1733,8 +1943,36 @@ sends logging output to a disk file. It inherits the output functionality from Outputs the record to the file. + .. _null-handler: +NullHandler +^^^^^^^^^^^ + +.. versionadded:: 2.7 + +The :class:`NullHandler` class, located in the core :mod:`logging` package, +does not do any formatting or output. It is essentially a "no-op" handler +for use by library developers. + +.. class:: NullHandler() + + Returns a new instance of the :class:`NullHandler` class. + + .. method:: emit(record) + + This method does nothing. + + .. method:: handle(record) + + This method does nothing. + + .. method:: createLock() + + This method returns `None` for the lock, since there is no + underlying I/O to which access needs to be serialized. + + See :ref:`library-config` for more information on how to use :class:`NullHandler`. @@ -1745,6 +1983,8 @@ WatchedFileHandler .. versionadded:: 2.6 +.. currentmodule:: logging.handlers + The :class:`WatchedFileHandler` class, located in the :mod:`logging.handlers` module, is a :class:`FileHandler` which watches the file it is logging to. If the file changes, it is closed and reopened using the file name. @@ -1997,16 +2237,22 @@ The :class:`SysLogHandler` class, located in the :mod:`logging.handlers` module, supports sending logging messages to a remote or local Unix syslog. -.. class:: SysLogHandler([address[, facility]]) +.. class:: SysLogHandler([address[, facility[, socktype]]]) Returns a new instance of the :class:`SysLogHandler` class intended to communicate with a remote Unix machine whose address is given by *address* in the form of a ``(host, port)`` tuple. If *address* is not specified, - ``('localhost', 514)`` is used. The address is used to open a UDP socket. An + ``('localhost', 514)`` is used. The address is used to open a socket. An alternative to providing a ``(host, port)`` tuple is providing an address as a string, for example "/dev/log". In this case, a Unix domain socket is used to send the message to the syslog. If *facility* is not specified, - :const:`LOG_USER` is used. + :const:`LOG_USER` is used. The type of socket opened depends on the + *socktype* argument, which defaults to :const:`socket.SOCK_DGRAM` and thus + opens a UDP socket. To open a TCP socket (for use with the newer syslog + daemons such as rsyslog), specify a value of :const:`socket.SOCK_STREAM`. + + .. versionchanged:: 2.7 + *socktype* was added. .. method:: close() @@ -2296,7 +2542,7 @@ supports sending logging messages to a Web server, using either ``GET`` or .. method:: emit(record) - Sends the record to the Web server as an URL-encoded dictionary. + Sends the record to the Web server as a percent-encoded dictionary. .. _formatter: @@ -2369,6 +2615,8 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: +-------------------------+-----------------------------------------------+ | ``%(process)d`` | Process ID (if available). | +-------------------------+-----------------------------------------------+ +| ``%(processName)s`` | Process name (if available). | ++-------------------------+-----------------------------------------------+ | ``%(message)s`` | The logged message, computed as ``msg % | | | args``. | +-------------------------+-----------------------------------------------+ @@ -2376,15 +2624,17 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: .. versionchanged:: 2.5 *funcName* was added. +.. versionchanged:: 2.6 + *processName* was added. + .. class:: Formatter([fmt[, datefmt]]) Returns a new instance of the :class:`Formatter` class. The instance is - initialized with a format string for the message as a whole, as well as a format - string for the date/time portion of a message. If no *fmt* is specified, - ``'%(message)s'`` is used. If no *datefmt* is specified, the ISO8601 date format - is used. - + initialized with a format string for the message as a whole, as well as a + format string for the date/time portion of a message. If no *fmt* is + specified, ``'%(message)s'`` is used. If no *datefmt* is specified, the + ISO8601 date format is used. .. method:: format(record) @@ -2428,7 +2678,7 @@ Currently, the useful mapping keys in a :class:`LogRecord` are: Filter Objects -------------- -Filters can be used by :class:`Handler`\ s and :class:`Logger`\ s for +:class:`Filter`\ s can be used by :class:`Handler`\ s and :class:`Logger`\ s for more sophisticated filtering than is provided by levels. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with "A.B" will allow events logged by loggers @@ -2440,7 +2690,7 @@ initialized with the empty string, all events are passed. Returns an instance of the :class:`Filter` class. If *name* is specified, it names a logger which, together with its children, will have its events allowed - through the filter. If no name is specified, allows every event. + through the filter. If *name* is the empty string, allows every event. .. method:: filter(record) @@ -2449,40 +2699,99 @@ initialized with the empty string, all events are passed. yes. If deemed appropriate, the record may be modified in-place by this method. +Note that filters attached to handlers are consulted whenever an event is +emitted by the handler, whereas filters attached to loggers are consulted +whenever an event is logged to the handler (using :meth:`debug`, :meth:`info`, +etc.) This means that events which have been generated by descendant loggers +will not be filtered by a logger's filter setting, unless the filter has also +been applied to those descendant loggers. + +You don't actually need to subclass ``Filter``: you can pass any instance +which has a ``filter`` method with the same semantics. + +Other uses for filters +^^^^^^^^^^^^^^^^^^^^^^ + +Although filters are used primarily to filter records based on more +sophisticated criteria than levels, they get to see every record which is +processed by the handler or logger they're attached to: this can be useful if +you want to do things like counting how many records were processed by a +particular logger or handler, or adding, changing or removing attributes in +the LogRecord being processed. Obviously changing the LogRecord needs to be +done with some care, but it does allow the injection of contextual information +into logs (see :ref:`filters-contextual`). + .. _log-record: LogRecord Objects ----------------- -:class:`LogRecord` instances are created every time something is logged. They -contain all the information pertinent to the event being logged. The main -information passed in is in msg and args, which are combined using msg % args to -create the message field of the record. The record also includes information -such as when the record was created, the source line where the logging call was -made, and any exception information to be logged. +:class:`LogRecord` instances are created automatically by the :class:`Logger` +every time something is logged, and can be created manually via +:func:`makeLogRecord` (for example, from a pickled event received over the +wire). -.. class:: LogRecord(name, lvl, pathname, lineno, msg, args, exc_info [, func]) +.. class:: + LogRecord(name, lvl, pathname, lineno, msg, args, exc_info [, func=None]) - Returns an instance of :class:`LogRecord` initialized with interesting - information. The *name* is the logger name; *lvl* is the numeric level; - *pathname* is the absolute pathname of the source file in which the logging - call was made; *lineno* is the line number in that file where the logging - call is found; *msg* is the user-supplied message (a format string); *args* - is the tuple which, together with *msg*, makes up the user message; and - *exc_info* is the exception tuple obtained by calling :func:`sys.exc_info` - (or :const:`None`, if no exception information is available). The *func* is - the name of the function from which the logging call was made. If not - specified, it defaults to ``None``. + Contains all the information pertinent to the event being logged. - .. versionchanged:: 2.5 - *func* was added. + The primary information is passed in :attr:`msg` and :attr:`args`, which + are combined using ``msg % args`` to create the :attr:`message` field of the + record. + + .. attribute:: args + + Tuple of arguments to be used in formatting :attr:`msg`. + + .. attribute:: exc_info + + Exception tuple (à la `sys.exc_info`) or `None` if no exception + information is available. + + .. attribute:: func + + Name of the function of origin (i.e. in which the logging call was made). + .. attribute:: lineno + + Line number in the source file of origin. + + .. attribute:: lvl + + Numeric logging level. + + .. attribute:: message + + Bound to the result of :meth:`getMessage` when + :meth:`Formatter.format(record)<Formatter.format>` is invoked. + + .. attribute:: msg + + User-supplied :ref:`format string<string-formatting>` or arbitrary object + (see :ref:`arbitrary-object-messages`) used in :meth:`getMessage`. + + .. attribute:: name + + Name of the logger that emitted the record. + + .. attribute:: pathname + + Absolute pathname of the source file of origin. .. method:: getMessage() Returns the message for this :class:`LogRecord` instance after merging any - user-supplied arguments with the message. + user-supplied arguments with the message. If the user-supplied message + argument to the logging call is not a string, :func:`str` is called on it to + convert it to a string. This allows use of user-defined classes as + messages, whose ``__str__`` method can return the actual format string to + be used. + + .. versionchanged:: 2.5 + *func* was added. + .. _logger-adapter: @@ -2493,22 +2802,21 @@ LoggerAdapter Objects :class:`LoggerAdapter` instances are used to conveniently pass contextual information into logging calls. For a usage example , see the section on -`adding contextual information to your logging output`__. +:ref:`adding contextual information to your logging output <context-info>`. -__ context-info_ .. class:: LoggerAdapter(logger, extra) - Returns an instance of :class:`LoggerAdapter` initialized with an - underlying :class:`Logger` instance and a dict-like object. + Returns an instance of :class:`LoggerAdapter` initialized with an + underlying :class:`Logger` instance and a dict-like object. - .. method:: process(msg, kwargs) + .. method:: process(msg, kwargs) - Modifies the message and/or keyword arguments passed to a logging call in - order to insert contextual information. This implementation takes the object - passed as *extra* to the constructor and adds it to *kwargs* using key - 'extra'. The return value is a (*msg*, *kwargs*) tuple which has the - (possibly modified) versions of the arguments passed in. + Modifies the message and/or keyword arguments passed to a logging call in + order to insert contextual information. This implementation takes the object + passed as *extra* to the constructor and adds it to *kwargs* using key + 'extra'. The return value is a (*msg*, *kwargs*) tuple which has the + (possibly modified) versions of the arguments passed in. In addition to the above, :class:`LoggerAdapter` supports all the logging methods of :class:`Logger`, i.e. :meth:`debug`, :meth:`info`, :meth:`warning`, @@ -2516,6 +2824,11 @@ methods of :class:`Logger`, i.e. :meth:`debug`, :meth:`info`, :meth:`warning`, methods have the same signatures as their counterparts in :class:`Logger`, so you can use the two types of instances interchangeably. +.. versionchanged:: 2.7 + +The :meth:`isEnabledFor` method was added to :class:`LoggerAdapter`. This method +delegates to the underlying logger. + Thread Safety ------------- @@ -2530,6 +2843,28 @@ module, you may not be able to use logging from within such handlers. This is because lock implementations in the :mod:`threading` module are not always re-entrant, and so cannot be invoked from such signal handlers. + +Integration with the warnings module +------------------------------------ + +The :func:`captureWarnings` function can be used to integrate :mod:`logging` +with the :mod:`warnings` module. + +.. function:: captureWarnings(capture) + + This function is used to turn the capture of warnings by logging on and + off. + + If *capture* is ``True``, warnings issued by the :mod:`warnings` module + will be redirected to the logging system. Specifically, a warning will be + formatted using :func:`warnings.formatwarning` and the resulting string + logged to a logger named "py.warnings" with a severity of ``WARNING``. + + If *capture* is ``False``, the redirection of warnings to the logging system + will stop, and warnings will be redirected to their original destinations + (i.e. those in effect before ``captureWarnings(True)`` was called). + + Configuration ------------- @@ -2545,18 +2880,65 @@ logging module using these functions or by making calls to the main API (defined in :mod:`logging` itself) and defining handlers which are declared either in :mod:`logging` or :mod:`logging.handlers`. +<<<<<<< local +.. function:: dictConfig(config) +======= .. currentmodule:: logging.config - +>>>>>>> other + +<<<<<<< local + Takes the logging configuration from a dictionary. The contents of + this dictionary are described in :ref:`logging-config-dictschema` + below. + + If an error is encountered during configuration, this function will + raise a :exc:`ValueError`, :exc:`TypeError`, :exc:`AttributeError` + or :exc:`ImportError` with a suitably descriptive message. The + following is a (possibly incomplete) list of conditions which will + raise an error: + + * A ``level`` which is not a string or which is a string not + corresponding to an actual logging level. + * A ``propagate`` value which is not a boolean. + * An id which does not have a corresponding destination. + * A non-existent handler id found during an incremental call. + * An invalid logger name. + * Inability to resolve to an internal or external object. + + Parsing is performed by the :class:`DictConfigurator` class, whose + constructor is passed the dictionary used for configuration, and + has a :meth:`configure` method. The :mod:`logging.config` module + has a callable attribute :attr:`dictConfigClass` + which is initially set to :class:`DictConfigurator`. + You can replace the value of :attr:`dictConfigClass` with a + suitable implementation of your own. + + :func:`dictConfig` calls :attr:`dictConfigClass` passing + the specified dictionary, and then calls the :meth:`configure` method on + the returned object to put the configuration into effect:: + + def dictConfig(config): + dictConfigClass(config).configure() + + For example, a subclass of :class:`DictConfigurator` could call + ``DictConfigurator.__init__()`` in its own :meth:`__init__()`, then + set up custom prefixes which would be usable in the subsequent + :meth:`configure` call. :attr:`dictConfigClass` would be bound to + this new subclass, and then :func:`dictConfig` could be called exactly as + in the default, uncustomized state. + + .. versionadded:: 2.7 +======= +>>>>>>> other .. function:: fileConfig(fname[, defaults]) - Reads the logging configuration from a ConfigParser-format file named *fname*. - This function can be called several times from an application, allowing an end - user the ability to select from various pre-canned configurations (if the - developer provides a mechanism to present the choices and load the chosen - configuration). Defaults to be passed to ConfigParser can be specified in the - *defaults* argument. - + Reads the logging configuration from a :mod:`ConfigParser`\-format file named + *fname*. This function can be called several times from an application, + allowing an end user to select from various pre-canned + configurations (if the developer provides a mechanism to present the choices + and load the chosen configuration). Defaults to be passed to the ConfigParser + can be specified in the *defaults* argument. .. function:: listen([port]) @@ -2580,8 +2962,409 @@ in :mod:`logging` itself) and defining handlers which are declared either in :func:`listen`. +<<<<<<< local +.. _logging-config-dictschema: +======= .. currentmodule:: logging +>>>>>>> other + +<<<<<<< local +Configuration dictionary schema +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Describing a logging configuration requires listing the various +objects to create and the connections between them; for example, you +may create a handler named "console" and then say that the logger +named "startup" will send its messages to the "console" handler. +These objects aren't limited to those provided by the :mod:`logging` +module because you might write your own formatter or handler class. +The parameters to these classes may also need to include external +objects such as ``sys.stderr``. The syntax for describing these +objects and connections is defined in :ref:`logging-config-dict-connections` +below. + +Dictionary Schema Details +""""""""""""""""""""""""" + +The dictionary passed to :func:`dictConfig` must contain the following +keys: + +* `version` - to be set to an integer value representing the schema + version. The only valid value at present is 1, but having this key + allows the schema to evolve while still preserving backwards + compatibility. + +All other keys are optional, but if present they will be interpreted +as described below. In all cases below where a 'configuring dict' is +mentioned, it will be checked for the special ``'()'`` key to see if a +custom instantiation is required. If so, the mechanism described in +:ref:`logging-config-dict-userdef` below is used to create an instance; +otherwise, the context is used to determine what to instantiate. + +* `formatters` - the corresponding value will be a dict in which each + key is a formatter id and each value is a dict describing how to + configure the corresponding Formatter instance. + The configuring dict is searched for keys ``format`` and ``datefmt`` + (with defaults of ``None``) and these are used to construct a + :class:`logging.Formatter` instance. + +* `filters` - the corresponding value will be a dict in which each key + is a filter id and each value is a dict describing how to configure + the corresponding Filter instance. + + The configuring dict is searched for the key ``name`` (defaulting to the + empty string) and this is used to construct a :class:`logging.Filter` + instance. + +* `handlers` - the corresponding value will be a dict in which each + key is a handler id and each value is a dict describing how to + configure the corresponding Handler instance. + + The configuring dict is searched for the following keys: + + * ``class`` (mandatory). This is the fully qualified name of the + handler class. + + * ``level`` (optional). The level of the handler. + + * ``formatter`` (optional). The id of the formatter for this + handler. + + * ``filters`` (optional). A list of ids of the filters for this + handler. + + All *other* keys are passed through as keyword arguments to the + handler's constructor. For example, given the snippet:: + + handlers: + console: + class : logging.StreamHandler + formatter: brief + level : INFO + filters: [allow_foo] + stream : ext://sys.stdout + file: + class : logging.handlers.RotatingFileHandler + formatter: precise + filename: logconfig.log + maxBytes: 1024 + backupCount: 3 + + the handler with id ``console`` is instantiated as a + :class:`logging.StreamHandler`, using ``sys.stdout`` as the underlying + stream. The handler with id ``file`` is instantiated as a + :class:`logging.handlers.RotatingFileHandler` with the keyword arguments + ``filename='logconfig.log', maxBytes=1024, backupCount=3``. + +* `loggers` - the corresponding value will be a dict in which each key + is a logger name and each value is a dict describing how to + configure the corresponding Logger instance. + + The configuring dict is searched for the following keys: + + * ``level`` (optional). The level of the logger. + + * ``propagate`` (optional). The propagation setting of the logger. + + * ``filters`` (optional). A list of ids of the filters for this + logger. + + * ``handlers`` (optional). A list of ids of the handlers for this + logger. + + The specified loggers will be configured according to the level, + propagation, filters and handlers specified. + +* `root` - this will be the configuration for the root logger. + Processing of the configuration will be as for any logger, except + that the ``propagate`` setting will not be applicable. + +* `incremental` - whether the configuration is to be interpreted as + incremental to the existing configuration. This value defaults to + ``False``, which means that the specified configuration replaces the + existing configuration with the same semantics as used by the + existing :func:`fileConfig` API. + + If the specified value is ``True``, the configuration is processed + as described in the section on :ref:`logging-config-dict-incremental`. + +* `disable_existing_loggers` - whether any existing loggers are to be + disabled. This setting mirrors the parameter of the same name in + :func:`fileConfig`. If absent, this parameter defaults to ``True``. + This value is ignored if `incremental` is ``True``. + +.. _logging-config-dict-incremental: + +Incremental Configuration +""""""""""""""""""""""""" + +It is difficult to provide complete flexibility for incremental +configuration. For example, because objects such as filters +and formatters are anonymous, once a configuration is set up, it is +not possible to refer to such anonymous objects when augmenting a +configuration. + +Furthermore, there is not a compelling case for arbitrarily altering +the object graph of loggers, handlers, filters, formatters at +run-time, once a configuration is set up; the verbosity of loggers and +handlers can be controlled just by setting levels (and, in the case of +loggers, propagation flags). Changing the object graph arbitrarily in +a safe way is problematic in a multi-threaded environment; while not +impossible, the benefits are not worth the complexity it adds to the +implementation. + +Thus, when the ``incremental`` key of a configuration dict is present +and is ``True``, the system will completely ignore any ``formatters`` and +``filters`` entries, and process only the ``level`` +settings in the ``handlers`` entries, and the ``level`` and +``propagate`` settings in the ``loggers`` and ``root`` entries. + +Using a value in the configuration dict lets configurations to be sent +over the wire as pickled dicts to a socket listener. Thus, the logging +verbosity of a long-running application can be altered over time with +no need to stop and restart the application. + +.. _logging-config-dict-connections: + +Object connections +"""""""""""""""""" + +The schema describes a set of logging objects - loggers, +handlers, formatters, filters - which are connected to each other in +an object graph. Thus, the schema needs to represent connections +between the objects. For example, say that, once configured, a +particular logger has attached to it a particular handler. For the +purposes of this discussion, we can say that the logger represents the +source, and the handler the destination, of a connection between the +two. Of course in the configured objects this is represented by the +logger holding a reference to the handler. In the configuration dict, +this is done by giving each destination object an id which identifies +it unambiguously, and then using the id in the source object's +configuration to indicate that a connection exists between the source +and the destination object with that id. + +So, for example, consider the following YAML snippet:: + + formatters: + brief: + # configuration for formatter with id 'brief' goes here + precise: + # configuration for formatter with id 'precise' goes here + handlers: + h1: #This is an id + # configuration of handler with id 'h1' goes here + formatter: brief + h2: #This is another id + # configuration of handler with id 'h2' goes here + formatter: precise + loggers: + foo.bar.baz: + # other configuration for logger 'foo.bar.baz' + handlers: [h1, h2] + +(Note: YAML used here because it's a little more readable than the +equivalent Python source form for the dictionary.) + +The ids for loggers are the logger names which would be used +programmatically to obtain a reference to those loggers, e.g. +``foo.bar.baz``. The ids for Formatters and Filters can be any string +value (such as ``brief``, ``precise`` above) and they are transient, +in that they are only meaningful for processing the configuration +dictionary and used to determine connections between objects, and are +not persisted anywhere when the configuration call is complete. + +The above snippet indicates that logger named ``foo.bar.baz`` should +have two handlers attached to it, which are described by the handler +ids ``h1`` and ``h2``. The formatter for ``h1`` is that described by id +``brief``, and the formatter for ``h2`` is that described by id +``precise``. + + +.. _logging-config-dict-userdef: + +User-defined objects +"""""""""""""""""""" + +The schema supports user-defined objects for handlers, filters and +formatters. (Loggers do not need to have different types for +different instances, so there is no support in this configuration +schema for user-defined logger classes.) + +Objects to be configured are described by dictionaries +which detail their configuration. In some places, the logging system +will be able to infer from the context how an object is to be +instantiated, but when a user-defined object is to be instantiated, +the system will not know how to do this. In order to provide complete +flexibility for user-defined object instantiation, the user needs +to provide a 'factory' - a callable which is called with a +configuration dictionary and which returns the instantiated object. +This is signalled by an absolute import path to the factory being +made available under the special key ``'()'``. Here's a concrete +example:: + + formatters: + brief: + format: '%(message)s' + default: + format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s' + datefmt: '%Y-%m-%d %H:%M:%S' + custom: + (): my.package.customFormatterFactory + bar: baz + spam: 99.9 + answer: 42 + +The above YAML snippet defines three formatters. The first, with id +``brief``, is a standard :class:`logging.Formatter` instance with the +specified format string. The second, with id ``default``, has a +longer format and also defines the time format explicitly, and will +result in a :class:`logging.Formatter` initialized with those two format +strings. Shown in Python source form, the ``brief`` and ``default`` +formatters have configuration sub-dictionaries:: + + { + 'format' : '%(message)s' + } + +and:: + + { + 'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s', + 'datefmt' : '%Y-%m-%d %H:%M:%S' + } + +respectively, and as these dictionaries do not contain the special key +``'()'``, the instantiation is inferred from the context: as a result, +standard :class:`logging.Formatter` instances are created. The +configuration sub-dictionary for the third formatter, with id +``custom``, is:: + + { + '()' : 'my.package.customFormatterFactory', + 'bar' : 'baz', + 'spam' : 99.9, + 'answer' : 42 + } + +and this contains the special key ``'()'``, which means that +user-defined instantiation is wanted. In this case, the specified +factory callable will be used. If it is an actual callable it will be +used directly - otherwise, if you specify a string (as in the example) +the actual callable will be located using normal import mechanisms. +The callable will be called with the **remaining** items in the +configuration sub-dictionary as keyword arguments. In the above +example, the formatter with id ``custom`` will be assumed to be +returned by the call:: + + my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42) + +The key ``'()'`` has been used as the special key because it is not a +valid keyword parameter name, and so will not clash with the names of +the keyword arguments used in the call. The ``'()'`` also serves as a +mnemonic that the corresponding value is a callable. + + +.. _logging-config-dict-externalobj: + +Access to external objects +"""""""""""""""""""""""""" + +There are times where a configuration needs to refer to objects +external to the configuration, for example ``sys.stderr``. If the +configuration dict is constructed using Python code, this is +straightforward, but a problem arises when the configuration is +provided via a text file (e.g. JSON, YAML). In a text file, there is +no standard way to distinguish ``sys.stderr`` from the literal string +``'sys.stderr'``. To facilitate this distinction, the configuration +system looks for certain special prefixes in string values and +treat them specially. For example, if the literal string +``'ext://sys.stderr'`` is provided as a value in the configuration, +then the ``ext://`` will be stripped off and the remainder of the +value processed using normal import mechanisms. + +The handling of such prefixes is done in a way analogous to protocol +handling: there is a generic mechanism to look for prefixes which +match the regular expression ``^(?P<prefix>[a-z]+)://(?P<suffix>.*)$`` +whereby, if the ``prefix`` is recognised, the ``suffix`` is processed +in a prefix-dependent manner and the result of the processing replaces +the string value. If the prefix is not recognised, then the string +value will be left as-is. + + +.. _logging-config-dict-internalobj: + +Access to internal objects +"""""""""""""""""""""""""" + +As well as external objects, there is sometimes also a need to refer +to objects in the configuration. This will be done implicitly by the +configuration system for things that it knows about. For example, the +string value ``'DEBUG'`` for a ``level`` in a logger or handler will +automatically be converted to the value ``logging.DEBUG``, and the +``handlers``, ``filters`` and ``formatter`` entries will take an +object id and resolve to the appropriate destination object. + +However, a more generic mechanism is needed for user-defined +objects which are not known to the :mod:`logging` module. For +example, consider :class:`logging.handlers.MemoryHandler`, which takes +a ``target`` argument which is another handler to delegate to. Since +the system already knows about this class, then in the configuration, +the given ``target`` just needs to be the object id of the relevant +target handler, and the system will resolve to the handler from the +id. If, however, a user defines a ``my.package.MyHandler`` which has +an ``alternate`` handler, the configuration system would not know that +the ``alternate`` referred to a handler. To cater for this, a generic +resolution system allows the user to specify:: + + handlers: + file: + # configuration of file handler goes here + + custom: + (): my.package.MyHandler + alternate: cfg://handlers.file + +The literal string ``'cfg://handlers.file'`` will be resolved in an +analogous way to strings with the ``ext://`` prefix, but looking +in the configuration itself rather than the import namespace. The +mechanism allows access by dot or by index, in a similar way to +that provided by ``str.format``. Thus, given the following snippet:: + + handlers: + email: + class: logging.handlers.SMTPHandler + mailhost: localhost + fromaddr: my_app@domain.tld + toaddrs: + - support_team@domain.tld + - dev_team@domain.tld + subject: Houston, we have a problem. + +in the configuration, the string ``'cfg://handlers'`` would resolve to +the dict with key ``handlers``, the string ``'cfg://handlers.email`` +would resolve to the dict with key ``email`` in the ``handlers`` dict, +and so on. The string ``'cfg://handlers.email.toaddrs[1]`` would +resolve to ``'dev_team.domain.tld'`` and the string +``'cfg://handlers.email.toaddrs[0]'`` would resolve to the value +``'support_team@domain.tld'``. The ``subject`` value could be accessed +using either ``'cfg://handlers.email.subject'`` or, equivalently, +``'cfg://handlers.email[subject]'``. The latter form only needs to be +used if the key contains spaces or non-alphanumeric characters. If an +index value consists only of decimal digits, access will be attempted +using the corresponding integer value, falling back to the string +value if needed. + +Given a string ``cfg://handlers.myhandler.mykey.123``, this will +resolve to ``config_dict['handlers']['myhandler']['mykey']['123']``. +If the string is specified as ``cfg://handlers.myhandler.mykey[123]``, +the system will attempt to retrieve the value from +``config_dict['handlers']['myhandler']['mykey'][123]``, and fall back +to ``config_dict['handlers']['myhandler']['mykey']['123']`` if that +fails. + +======= +>>>>>>> other .. _logging-config-fileformat: Configuration file format diff --git a/Doc/library/macosa.rst b/Doc/library/macosa.rst index 250a923..54e62f2 100644 --- a/Doc/library/macosa.rst +++ b/Doc/library/macosa.rst @@ -9,7 +9,8 @@ This chapter describes the current implementation of the Open Scripting Architecture (OSA, also commonly referred to as AppleScript) for Python, allowing you to control scriptable applications from your Python program, and with a fairly pythonic interface. Development on this set of modules has -stopped, and a replacement is expected for Python 2.5. +stopped. For more up-to-date implementation of AppleScript support for Python, +see the third-party py-appscript project: <http://pypi.python.org/pypi/appscript/>. For a description of the various components of AppleScript and OSA, and to get an understanding of the architecture and terminology, you should read Apple's diff --git a/Doc/library/macostools.rst b/Doc/library/macostools.rst index fc26d2f..f2a2643 100644 --- a/Doc/library/macostools.rst +++ b/Doc/library/macostools.rst @@ -75,7 +75,7 @@ have incompatible behaviour in some cases. :mod:`findertools` --- The :program:`finder`'s Apple Events interface -===================================================================== +====================================================================== .. module:: findertools :platform: Mac diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 789eccc..e5ffba0 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -87,7 +87,7 @@ Number-theoretic and representation functions loss of precision by tracking multiple intermediate partial sums:: >>> sum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1]) - 0.99999999999999989 + 0.9999999999999999 >>> fsum([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1]) 1.0 @@ -159,6 +159,22 @@ Power and logarithmic functions Return ``e**x``. +.. function:: expm1(x) + + Return ``e**x - 1``. For small floats *x*, the subtraction in + ``exp(x) - 1`` can result in a significant loss of precision; the + :func:`expm1` function provides a way to compute this quantity to + full precision:: + + >>> from math import exp, expm1 + >>> exp(1e-5) - 1 # gives result accurate to 11 places + 1.0000050000069649e-05 + >>> expm1(1e-5) # result accurate to full precision + 1.0000050000166668e-05 + + .. versionadded:: 2.7 + + .. function:: log(x[, base]) With one argument, return the natural logarithm of *x* (to base *e*). @@ -303,6 +319,38 @@ Hyperbolic functions Return the hyperbolic tangent of *x*. +Special functions +----------------- + +.. function:: erf(x) + + Return the error function at *x*. + + .. versionadded:: 2.7 + + +.. function:: erfc(x) + + Return the complementary error function at *x*. + + .. versionadded:: 2.7 + + +.. function:: gamma(x) + + Return the Gamma function at *x*. + + .. versionadded:: 2.7 + + +.. function:: lgamma(x) + + Return the natural logarithm of the absolute value of the Gamma + function at *x*. + + .. versionadded:: 2.7 + + Constants --------- diff --git a/Doc/library/mimetypes.rst b/Doc/library/mimetypes.rst index 818f39f..956a1f1 100644 --- a/Doc/library/mimetypes.rst +++ b/Doc/library/mimetypes.rst @@ -77,9 +77,13 @@ behavior of the module. Initialize the internal data structures. If given, *files* must be a sequence of file names which should be used to augment the default type map. If omitted, - the file names to use are taken from :const:`knownfiles`. Each file named in - *files* or :const:`knownfiles` takes precedence over those named before it. - Calling :func:`init` repeatedly is allowed. + the file names to use are taken from :const:`knownfiles`; on Windows, the + current registry settings are loaded. Each file named in *files* or + :const:`knownfiles` takes precedence over those named before it. Calling + :func:`init` repeatedly is allowed. + + .. versionchanged:: 2.7 + Previously, Windows registry settings were ignored. .. function:: read_mime_types(filename) @@ -213,6 +217,12 @@ MimeTypes Objects of the object. +.. method:: MimeTypes.guess_all_extensions(type[, strict]) + + Similar to the :func:`guess_all_extensions` function, using the tables stored as part + of the object. + + .. method:: MimeTypes.guess_type(url[, strict]) Similar to the :func:`guess_type` function, using the tables stored as part of @@ -230,3 +240,9 @@ MimeTypes Objects Load MIME type information from an open file. The file must have the format of the standard :file:`mime.types` files. + +.. method:: MimeTypes.read_windows_registry() + + Load MIME type information from the Windows registry. Availability: Windows. + + .. versionadded:: 2.7 diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index a6e9bdc..125b34f 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -173,7 +173,7 @@ memory but does not update the underlying file. Copy the *count* bytes starting at offset *src* to the destination index *dest*. If the mmap was created with :const:`ACCESS_READ`, then calls to - move will throw a :exc:`TypeError` exception. + move will raise a :exc:`TypeError` exception. .. method:: read(num) @@ -199,7 +199,7 @@ memory but does not update the underlying file. Resizes the map and the underlying file, if any. If the mmap was created with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will - throw a :exc:`TypeError` exception. + raise a :exc:`TypeError` exception. .. method:: rfind(string[, start[, end]]) @@ -234,7 +234,7 @@ memory but does not update the underlying file. Write the bytes in *string* into memory at the current position of the file pointer; the file position is updated to point after the bytes that were written. If the mmap was created with :const:`ACCESS_READ`, then - writing to it will throw a :exc:`TypeError` exception. + writing to it will raise a :exc:`TypeError` exception. .. method:: write_byte(byte) @@ -242,6 +242,4 @@ memory but does not update the underlying file. Write the single-character string *byte* into memory at the current position of the file pointer; the file position is advanced by ``1``. If the mmap was created with :const:`ACCESS_READ`, then writing to it will - throw a :exc:`TypeError` exception. - - + raise a :exc:`TypeError` exception. diff --git a/Doc/library/modules.rst b/Doc/library/modules.rst index ec6f7cd..b5543e6 100644 --- a/Doc/library/modules.rst +++ b/Doc/library/modules.rst @@ -14,6 +14,7 @@ The full list of modules described in this chapter is: .. toctree:: imp.rst + importlib.rst imputil.rst zipimport.rst pkgutil.rst diff --git a/Doc/library/msilib.rst b/Doc/library/msilib.rst index 7d64de3..a412609 100644 --- a/Doc/library/msilib.rst +++ b/Doc/library/msilib.rst @@ -353,7 +353,7 @@ Directory Objects ----------------- -.. class:: Directory(database, cab, basedir, physical, logical, default, component, [componentflags]) +.. class:: Directory(database, cab, basedir, physical, logical, default, [componentflags]) Create a new directory in the Directory table. There is a current component at each point in time for the directory, which is either explicitly created through diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 9bdaf6f..26d0b3d 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -216,7 +216,7 @@ However, if you really do need to use some shared data then The ``'d'`` and ``'i'`` arguments used when creating ``num`` and ``arr`` are typecodes of the kind used by the :mod:`array` module: ``'d'`` indicates a double precision float and ``'i'`` indicates a signed integer. These shared - objects will be process and thread safe. + objects will be process and thread-safe. For more flexibility in using shared memory one can use the :mod:`multiprocessing.sharedctypes` module which supports the creation of @@ -376,7 +376,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits. Additionally, these are **not** Unix daemons or services, they are normal processes that will be - terminated (and not joined) if non-dameonic processes have exited. + terminated (and not joined) if non-daemonic processes have exited. In addition to the :class:`Threading.Thread` API, :class:`Process` objects also support the following attributes and methods: @@ -721,7 +721,8 @@ Connection objects usually created using :func:`Pipe` -- see also Send an object to the other end of the connection which should be read using :meth:`recv`. - The object must be picklable. + The object must be picklable. Very large pickles (approximately 32 MB+, + though it depends on the OS) may raise a ValueError exception. .. method:: recv() @@ -753,7 +754,9 @@ Connection objects usually created using :func:`Pipe` -- see also complete message. If *offset* is given then data is read from that position in *buffer*. If - *size* is given then that many bytes will be read from buffer. + *size* is given then that many bytes will be read from buffer. Very large + buffers (approximately 32 MB+, though it depends on the OS) may raise a + ValueError exception .. method:: recv_bytes([maxlength]) @@ -849,6 +852,12 @@ object -- see :ref:`multiprocessing-managers`. .. class:: Event() A clone of :class:`threading.Event`. + This method returns the state of the internal semaphore on exit, so it + will always return ``True`` except if a timeout is given and the operation + times out. + + .. versionchanged:: 2.7 + Previously, the method always returned ``None``. .. class:: Lock() @@ -860,7 +869,7 @@ object -- see :ref:`multiprocessing-managers`. .. class:: Semaphore([value]) - A bounded semaphore object: a clone of :class:`threading.Semaphore`. + A semaphore object: a clone of :class:`threading.Semaphore`. .. note:: @@ -1134,9 +1143,10 @@ their parent process exits. The manager classes are defined in the ``current_process().authkey``. Otherwise *authkey* is used and it must be a string. - .. method:: start() + .. method:: start([initializer[, initargs]]) - Start a subprocess to start the manager. + Start a subprocess to start the manager. If *initializer* is not ``None`` + then the subprocess will call ``initializer(*initargs)`` when it starts. .. method:: get_server() @@ -1276,6 +1286,24 @@ their parent process exits. The manager classes are defined in the Create a shared ``list`` object and return a proxy for it. + .. note:: + + Modifications to mutable values or items in dict and list proxies will not + be propagated through the manager, because the proxy has no way of knowing + when its values or items are modified. To modify such an item, you can + re-assign the modified object to the container proxy:: + + # create a list proxy and append a mutable object (a dictionary) + lproxy = manager.list() + lproxy.append({}) + # now mutate the dictionary + d = lproxy[0] + d['a'] = 1 + d['b'] = 2 + # at this point, the changes to d are not yet synced, but by + # reassigning the dictionary, the proxy is notified of the change + lproxy[0] = d + Namespace objects >>>>>>>>>>>>>>>>> @@ -1522,7 +1550,7 @@ Process Pools One can create a pool of processes which will carry out tasks submitted to it with the :class:`Pool` class. -.. class:: multiprocessing.Pool([processes[, initializer[, initargs]]]) +.. class:: multiprocessing.Pool([processes[, initializer[, initargs[, maxtasksperchild]]]]) A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and @@ -1533,6 +1561,22 @@ with the :class:`Pool` class. *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + .. versionadded:: 2.7 + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + .. note:: + + Worker processes within a :class:`Pool` typically live for the complete + duration of the Pool's work queue. A frequent pattern found in other + systems (such as Apache, mod_wsgi, etc) to free resources held by + workers is to allow a worker within a pool to complete only a set + amount of work before being exiting, being cleaned up and a new + process spawned to replace the old one. The *maxtasksperchild* + argument to the :class:`Pool` exposes this ability to the end user. + .. method:: apply(func[, args[, kwds]]) Equivalent of the :func:`apply` built-in function. It blocks till the @@ -2198,8 +2242,8 @@ Synchronization types like locks, conditions and queues: .. literalinclude:: ../includes/mp_synchronize.py -An showing how to use queues to feed tasks to a collection of worker process and -collect the results: +An example showing how to use queues to feed tasks to a collection of worker +process and collect the results: .. literalinclude:: ../includes/mp_workers.py diff --git a/Doc/library/nntplib.rst b/Doc/library/nntplib.rst index 6c16a43..7a461b4 100644 --- a/Doc/library/nntplib.rst +++ b/Doc/library/nntplib.rst @@ -18,35 +18,35 @@ Protocol), see Internet :rfc:`977`. Here are two small examples of how it can be used. To list some statistics about a newsgroup and print the subjects of the last 10 articles:: - >>> s = NNTP('news.cwi.nl') - >>> resp, count, first, last, name = s.group('comp.lang.python') + >>> s = NNTP('news.gmane.org') + >>> resp, count, first, last, name = s.group('gmane.comp.python.committers') >>> print 'Group', name, 'has', count, 'articles, range', first, 'to', last - Group comp.lang.python has 59 articles, range 3742 to 3803 + Group gmane.comp.python.committers has 1071 articles, range 1 to 1071 >>> resp, subs = s.xhdr('subject', first + '-' + last) >>> for id, sub in subs[-10:]: print id, sub ... - 3792 Re: Removing elements from a list while iterating... - 3793 Re: Who likes Info files? - 3794 Emacs and doc strings - 3795 a few questions about the Mac implementation - 3796 Re: executable python scripts - 3797 Re: executable python scripts - 3798 Re: a few questions about the Mac implementation - 3799 Re: PROPOSAL: A Generic Python Object Interface for Python C Modules - 3802 Re: executable python scripts - 3803 Re: \POSIX{} wait and SIGCHLD + 1062 Re: Mercurial Status? + 1063 Re: [python-committers] (Windows) buildbots on 3.x + 1064 Re: Mercurial Status? + 1065 Re: Mercurial Status? + 1066 Python 2.6.6 status + 1067 Commit Privileges for Ask Solem + 1068 Re: Commit Privileges for Ask Solem + 1069 Re: Commit Privileges for Ask Solem + 1070 Re: Commit Privileges for Ask Solem + 1071 2.6.6 rc 2 >>> s.quit() - '205 news.cwi.nl closing connection. Goodbye.' + '205 Bye!' To post an article from a file (this assumes that the article has valid -headers):: +headers, and that you have right to post on the particular newsgroup):: - >>> s = NNTP('news.cwi.nl') + >>> s = NNTP('news.gmane.org') >>> f = open('/tmp/article') >>> s.post(f) '240 Article posted successfully.' >>> s.quit() - '205 news.cwi.nl closing connection. Goodbye.' + '205 Bye!' The module itself defines the following items: diff --git a/Doc/library/numbers.rst b/Doc/library/numbers.rst index 300a6f3..a0a825f 100644 --- a/Doc/library/numbers.rst +++ b/Doc/library/numbers.rst @@ -47,7 +47,7 @@ The numeric tower To :class:`Complex`, :class:`Real` adds the operations that work on real numbers. - In short, those are: a conversion to :class:`float`, :func:`trunc`, + In short, those are: a conversion to :class:`float`, :func:`math.trunc`, :func:`round`, :func:`math.floor`, :func:`math.ceil`, :func:`divmod`, ``//``, ``%``, ``<``, ``<=``, ``>``, and ``>=``. diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index db813d8..854f14d 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -241,7 +241,7 @@ Operations which work with sequences (some of them with mappings too) include: Delete the slice of *a* from index *b* to index *c-1*. .. deprecated:: 2.6 - This function is removed in Python 3.0. Use :func:`delitem` with a slice + This function is removed in Python 3.x. Use :func:`delitem` with a slice index. @@ -257,7 +257,7 @@ Operations which work with sequences (some of them with mappings too) include: Return the slice of *a* from index *b* to index *c-1*. .. deprecated:: 2.6 - This function is removed in Python 3.0. Use :func:`getitem` with a slice + This function is removed in Python 3.x. Use :func:`getitem` with a slice index. @@ -269,8 +269,8 @@ Operations which work with sequences (some of them with mappings too) include: .. function:: repeat(a, b) __repeat__(a, b) - .. deprecated:: 2.6 - This function is removed in Python 3.0. Use :func:`__mul__` instead. + .. deprecated:: 2.7 + Use :func:`__mul__` instead. Return ``a * b`` where *a* is a sequence and *b* is an integer. @@ -295,7 +295,7 @@ Operations which work with sequences (some of them with mappings too) include: Set the slice of *a* from index *b* to index *c-1* to the sequence *v*. .. deprecated:: 2.6 - This function is removed in Python 3.0. Use :func:`setitem` with a slice + This function is removed in Python 3.x. Use :func:`setitem` with a slice index. Example use of operator functions:: @@ -399,8 +399,8 @@ example, the :term:`statement` ``x += y`` is equivalent to .. function:: irepeat(a, b) __irepeat__(a, b) - .. deprecated:: 2.6 - This function is removed in Python 3.0. Use :func:`__imul__` instead. + .. deprecated:: 2.7 + Use :func:`__imul__` instead. ``a = irepeat(a, b)`` is equivalent to ``a *= b`` where *a* is a sequence and *b* is an integer. @@ -458,8 +458,8 @@ abstract base classes instead (see :mod:`collections` and .. function:: isMappingType(obj) - .. deprecated:: 2.6 - This function is removed in Python 3.0. Use ``isinstance(x, collections.Mapping)`` instead. + .. deprecated:: 2.7 + Use ``isinstance(x, collections.Mapping)`` instead. Returns true if the object *obj* supports the mapping interface. This is true for dictionaries and all instance objects defining :meth:`__getitem__`. @@ -467,8 +467,8 @@ abstract base classes instead (see :mod:`collections` and .. function:: isNumberType(obj) - .. deprecated:: 2.6 - This function is removed in Python 3.0. Use ``isinstance(x, numbers.Number)`` instead. + .. deprecated:: 2.7 + Use ``isinstance(x, numbers.Number)`` instead. Returns true if the object *obj* represents a number. This is true for all numeric types implemented in C. @@ -476,8 +476,8 @@ abstract base classes instead (see :mod:`collections` and .. function:: isSequenceType(obj) - .. deprecated:: 2.6 - This function is removed in Python 3.0. Use ``isinstance(x, collections.Sequence)`` instead. + .. deprecated:: 2.7 + Use ``isinstance(x, collections.Sequence)`` instead. Returns true if the object *obj* supports the sequence protocol. This returns true for all objects which define sequence methods in C, and for all instance objects @@ -496,7 +496,23 @@ expect a function argument. attribute is requested, returns a tuple of attributes. After, ``f = attrgetter('name')``, the call ``f(b)`` returns ``b.name``. After, ``f = attrgetter('name', 'date')``, the call ``f(b)`` returns ``(b.name, - b.date)``. + b.date)``. Equivalent to:: + + def attrgetter(*items): + if len(items) == 1: + attr = items[0] + def g(obj): + return resolve_attr(obj, attr) + else: + def g(obj): + return tuple(resolve_att(obj, attr) for attr in items) + return g + + def resolve_attr(obj, attr): + for name in attr.split("."): + obj = getattr(obj, name) + return obj + The attribute names can also contain dots; after ``f = attrgetter('date.month')``, the call ``f(b)`` returns ``b.date.month``. @@ -516,15 +532,15 @@ expect a function argument. operand's :meth:`__getitem__` method. If multiple items are specified, returns a tuple of lookup values. Equivalent to:: - def itemgetter(*items): - if len(items) == 1: - item = items[0] - def g(obj): - return obj[item] - else: - def g(obj): - return tuple(obj[item] for item in items) - return g + def itemgetter(*items): + if len(items) == 1: + item = items[0] + def g(obj): + return obj[item] + else: + def g(obj): + return tuple(obj[item] for item in items) + return g The items can be any type accepted by the operand's :meth:`__getitem__` method. Dictionaries accept any hashable value. Lists, tuples, and @@ -545,12 +561,12 @@ expect a function argument. Example of using :func:`itemgetter` to retrieve specific fields from a tuple record: - >>> inventory = [('apple', 3), ('banana', 2), ('pear', 5), ('orange', 1)] - >>> getcount = itemgetter(1) - >>> map(getcount, inventory) - [3, 2, 5, 1] - >>> sorted(inventory, key=getcount) - [('orange', 1), ('banana', 2), ('apple', 3), ('pear', 5)] + >>> inventory = [('apple', 3), ('banana', 2), ('pear', 5), ('orange', 1)] + >>> getcount = itemgetter(1) + >>> map(getcount, inventory) + [3, 2, 5, 1] + >>> sorted(inventory, key=getcount) + [('orange', 1), ('banana', 2), ('apple', 3), ('pear', 5)] .. function:: methodcaller(name[, args...]) @@ -559,7 +575,12 @@ expect a function argument. additional arguments and/or keyword arguments are given, they will be given to the method as well. After ``f = methodcaller('name')``, the call ``f(b)`` returns ``b.name()``. After ``f = methodcaller('name', 'foo', bar=1)``, the - call ``f(b)`` returns ``b.name('foo', bar=1)``. + call ``f(b)`` returns ``b.name('foo', bar=1)``. Equivalent to:: + + def methodcaller(name, *args, **kwargs): + def caller(obj): + return getattr(obj, name)(*args, **kwargs) + return caller .. versionadded:: 2.6 diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst index c543bd9..d4b121b 100644 --- a/Doc/library/optparse.rst +++ b/Doc/library/optparse.rst @@ -1,10 +1,15 @@ -:mod:`optparse` --- More powerful command line option parser -============================================================ +:mod:`optparse` --- Parser for command line options +=================================================== .. module:: optparse - :synopsis: More convenient, flexible, and powerful command-line parsing library. -.. moduleauthor:: Greg Ward <gward@python.net> + :synopsis: Command-line option parsing library. + :deprecated: + +.. deprecated:: 2.7 + The :mod:`optparse` module is deprecated and will not be developed further; + development will continue with the :mod:`argparse` module. +.. moduleauthor:: Greg Ward <gward@python.net> .. versionadded:: 2.3 @@ -59,9 +64,9 @@ and :mod:`optparse` will print out a brief summary of your script's options: .. code-block:: text - usage: <yourscript> [options] + Usage: <yourscript> [options] - options: + Options: -h, --help show this help message and exit -f FILE, --file=FILE write report to FILE -q, --quiet don't print status messages to stdout @@ -102,26 +107,26 @@ option an argument used to supply extra information to guide or customize the execution of a program. There are many different syntaxes for options; the traditional Unix syntax is a hyphen ("-") followed by a single letter, - e.g. ``"-x"`` or ``"-F"``. Also, traditional Unix syntax allows multiple - options to be merged into a single argument, e.g. ``"-x -F"`` is equivalent - to ``"-xF"``. The GNU project introduced ``"--"`` followed by a series of - hyphen-separated words, e.g. ``"--file"`` or ``"--dry-run"``. These are the + e.g. ``-x`` or ``-F``. Also, traditional Unix syntax allows multiple + options to be merged into a single argument, e.g. ``-x -F`` is equivalent + to ``-xF``. The GNU project introduced ``--`` followed by a series of + hyphen-separated words, e.g. ``--file`` or ``--dry-run``. These are the only two option syntaxes provided by :mod:`optparse`. Some other option syntaxes that the world has seen include: - * a hyphen followed by a few letters, e.g. ``"-pf"`` (this is *not* the same + * a hyphen followed by a few letters, e.g. ``-pf`` (this is *not* the same as multiple options merged into a single argument) - * a hyphen followed by a whole word, e.g. ``"-file"`` (this is technically + * a hyphen followed by a whole word, e.g. ``-file`` (this is technically equivalent to the previous syntax, but they aren't usually seen in the same program) * a plus sign followed by a single letter, or a few letters, or a word, e.g. - ``"+f"``, ``"+rgb"`` + ``+f``, ``+rgb`` - * a slash followed by a letter, or a few letters, or a word, e.g. ``"/f"``, - ``"/file"`` + * a slash followed by a letter, or a few letters, or a word, e.g. ``/f``, + ``/file`` These option syntaxes are not supported by :mod:`optparse`, and they never will be. This is deliberate: the first three are non-standard on any @@ -149,9 +154,9 @@ option argument Typically, a given option either takes an argument or it doesn't. Lots of people want an "optional option arguments" feature, meaning that some options will take an argument if they see it, and won't if they don't. This is - somewhat controversial, because it makes parsing ambiguous: if ``"-a"`` takes - an optional argument and ``"-b"`` is another option entirely, how do we - interpret ``"-ab"``? Because of this ambiguity, :mod:`optparse` does not + somewhat controversial, because it makes parsing ambiguous: if ``-a`` takes + an optional argument and ``-b`` is another option entirely, how do we + interpret ``-ab``? Because of this ambiguity, :mod:`optparse` does not support this feature. positional argument @@ -169,9 +174,9 @@ For example, consider this hypothetical command-line:: prog -v --report /tmp/report.txt foo bar -``"-v"`` and ``"--report"`` are both options. Assuming that :option:`--report` -takes one argument, ``"/tmp/report.txt"`` is an option argument. ``"foo"`` and -``"bar"`` are positional arguments. +``-v`` and ``--report`` are both options. Assuming that ``--report`` +takes one argument, ``/tmp/report.txt`` is an option argument. ``foo`` and +``bar`` are positional arguments. .. _optparse-what-options-for: @@ -256,7 +261,7 @@ Then you can start defining options. The basic syntax is:: parser.add_option(opt_str, ..., attr=value, ...) -Each option has one or more option strings, such as ``"-f"`` or ``"--file"``, +Each option has one or more option strings, such as ``-f`` or ``--file``, and several option attributes that tell :mod:`optparse` what to expect and what to do when it encounters that option on the command line. @@ -285,7 +290,7 @@ that's rarely necessary: by default it uses ``sys.argv[1:]``.) :meth:`parse_args` returns two values: * ``options``, an object containing values for all of your options---e.g. if - ``"--file"`` takes a single string argument, then ``options.file`` will be the + ``--file`` takes a single string argument, then ``options.file`` will be the filename supplied by the user, or ``None`` if the user did not supply that option @@ -331,8 +336,8 @@ Now let's make up a fake command line and ask :mod:`optparse` to parse it:: args = ["-f", "foo.txt"] (options, args) = parser.parse_args(args) -When :mod:`optparse` sees the option string ``"-f"``, it consumes the next -argument, ``"foo.txt"``, and stores it in ``options.filename``. So, after this +When :mod:`optparse` sees the option string ``-f``, it consumes the next +argument, ``foo.txt``, and stores it in ``options.filename``. So, after this call to :meth:`parse_args`, ``options.filename`` is ``"foo.txt"``. Some other option types supported by :mod:`optparse` are ``int`` and ``float``. @@ -344,13 +349,13 @@ Note that this option has no long option string, which is perfectly acceptable. Also, there's no explicit action, since the default is ``store``. Let's parse another fake command-line. This time, we'll jam the option argument -right up against the option: since ``"-n42"`` (one argument) is equivalent to -``"-n 42"`` (two arguments), the code :: +right up against the option: since ``-n42`` (one argument) is equivalent to +``-n 42`` (two arguments), the code :: (options, args) = parser.parse_args(["-n42"]) print options.num -will print ``"42"``. +will print ``42``. If you don't specify a type, :mod:`optparse` assumes ``string``. Combined with the fact that the default action is ``store``, that means our first example can @@ -360,9 +365,9 @@ be a lot shorter:: If you don't supply a destination, :mod:`optparse` figures out a sensible default from the option strings: if the first long option string is -``"--foo-bar"``, then the default destination is ``foo_bar``. If there are no +``--foo-bar``, then the default destination is ``foo_bar``. If there are no long option strings, :mod:`optparse` looks at the first short option string: the -default destination for ``"-f"`` is ``f``. +default destination for ``-f`` is ``f``. :mod:`optparse` also includes built-in ``long`` and ``complex`` types. Adding types is covered in section :ref:`optparse-extending-optparse`. @@ -376,7 +381,7 @@ Handling boolean (flag) options Flag options---set a variable to true or false when a particular option is seen ---are quite common. :mod:`optparse` supports them with two separate actions, ``store_true`` and ``store_false``. For example, you might have a ``verbose`` -flag that is turned on with ``"-v"`` and off with ``"-q"``:: +flag that is turned on with ``-v`` and off with ``-q``:: parser.add_option("-v", action="store_true", dest="verbose") parser.add_option("-q", action="store_false", dest="verbose") @@ -385,8 +390,8 @@ Here we have two different options with the same destination, which is perfectly OK. (It just means you have to be a bit careful when setting default values--- see below.) -When :mod:`optparse` encounters ``"-v"`` on the command line, it sets -``options.verbose`` to ``True``; when it encounters ``"-q"``, +When :mod:`optparse` encounters ``-v`` on the command line, it sets +``options.verbose`` to ``True``; when it encounters ``-q``, ``options.verbose`` is set to ``False``. @@ -426,7 +431,7 @@ supply a default value for each destination, which is assigned before the command line is parsed. First, consider the verbose/quiet example. If we want :mod:`optparse` to set -``verbose`` to ``True`` unless ``"-q"`` is seen, then we can do this:: +``verbose`` to ``True`` unless ``-q`` is seen, then we can do this:: parser.add_option("-v", action="store_true", dest="verbose", default=True) parser.add_option("-q", action="store_false", dest="verbose") @@ -484,15 +489,15 @@ user-friendly (documented) options:: help="interaction mode: novice, intermediate, " "or expert [default: %default]") -If :mod:`optparse` encounters either ``"-h"`` or ``"--help"`` on the +If :mod:`optparse` encounters either ``-h`` or ``--help`` on the command-line, or if you just call :meth:`parser.print_help`, it prints the following to standard output: .. code-block:: text - usage: <yourscript> [options] arg1 arg2 + Usage: <yourscript> [options] arg1 arg2 - options: + Options: -h, --help show this help message and exit -v, --verbose make lots of noise [default] -q, --quiet be vewwy quiet (I'm hunting wabbits) @@ -511,12 +516,12 @@ help message: usage = "usage: %prog [options] arg1 arg2" - :mod:`optparse` expands ``"%prog"`` in the usage string to the name of the + :mod:`optparse` expands ``%prog`` in the usage string to the name of the current program, i.e. ``os.path.basename(sys.argv[0])``. The expanded string is then printed before the detailed option help. If you don't supply a usage string, :mod:`optparse` uses a bland but sensible - default: ``"usage: %prog [options]"``, which is fine if your script doesn't + default: ``"Usage: %prog [options]"``, which is fine if your script doesn't take any positional arguments. * every option defines a help string, and doesn't worry about line-wrapping--- @@ -529,17 +534,17 @@ help message: -m MODE, --mode=MODE Here, "MODE" is called the meta-variable: it stands for the argument that the - user is expected to supply to :option:`-m`/:option:`--mode`. By default, + user is expected to supply to ``-m``/``--mode``. By default, :mod:`optparse` converts the destination variable name to uppercase and uses that for the meta-variable. Sometimes, that's not what you want---for - example, the :option:`--filename` option explicitly sets ``metavar="FILE"``, + example, the ``--filename`` option explicitly sets ``metavar="FILE"``, resulting in this automatically-generated option description:: -f FILE, --filename=FILE This is important for more than just saving space, though: the manually - written help text uses the meta-variable "FILE" to clue the user in that - there's a connection between the semi-formal syntax "-f FILE" and the informal + written help text uses the meta-variable ``FILE`` to clue the user in that + there's a connection between the semi-formal syntax ``-f FILE`` and the informal semantic description "write output to FILE". This is a simple but effective way to make your help text a lot clearer and more useful for end users. @@ -549,12 +554,33 @@ help message: default value. If an option has no default value (or the default value is ``None``), ``%default`` expands to ``none``. +Grouping Options +++++++++++++++++ + When dealing with many options, it is convenient to group these options for better help output. An :class:`OptionParser` can contain several option groups, each of which can contain several options. -Continuing with the parser defined above, adding an :class:`OptionGroup` to a -parser is easy:: +An option group is obtained using the class :class:`OptionGroup`: + +.. class:: OptionGroup(parser, title, description=None) + + where + + * parser is the :class:`OptionParser` instance the group will be insterted in + to + * title is the group title + * description, optional, is a long description of the group + +:class:`OptionGroup` inherits from :class:`OptionContainer` (like +:class:`OptionParser`) and so the :meth:`add_option` method can be used to add +an option to the group. + +Once all the options are declared, using the :class:`OptionParser` method +:meth:`add_option_group` the group is added to the previously defined parser. + +Continuing with the parser defined in the previous section, adding an +:class:`OptionGroup` to a parser is easy:: group = OptionGroup(parser, "Dangerous Options", "Caution: use these options at your own risk. " @@ -566,20 +592,73 @@ This would result in the following help output: .. code-block:: text - usage: [options] arg1 arg2 + Usage: <yourscript> [options] arg1 arg2 + + Options: + -h, --help show this help message and exit + -v, --verbose make lots of noise [default] + -q, --quiet be vewwy quiet (I'm hunting wabbits) + -f FILE, --filename=FILE + write output to FILE + -m MODE, --mode=MODE interaction mode: novice, intermediate, or + expert [default: intermediate] + + Dangerous Options: + Caution: use these options at your own risk. It is believed that some + of them bite. + + -g Group option. + +A bit more complete example might invole using more than one group: still +extendind the previous example:: + + group = OptionGroup(parser, "Dangerous Options", + "Caution: use these options at your own risk. " + "It is believed that some of them bite.") + group.add_option("-g", action="store_true", help="Group option.") + parser.add_option_group(group) + + group = OptionGroup(parser, "Debug Options") + group.add_option("-d", "--debug", action="store_true", + help="Print debug information") + group.add_option("-s", "--sql", action="store_true", + help="Print all SQL statements executed") + group.add_option("-e", action="store_true", help="Print every action done") + parser.add_option_group(group) + +that results in the following output: + +.. code-block:: text + + Usage: <yourscript> [options] arg1 arg2 + + Options: + -h, --help show this help message and exit + -v, --verbose make lots of noise [default] + -q, --quiet be vewwy quiet (I'm hunting wabbits) + -f FILE, --filename=FILE + write output to FILE + -m MODE, --mode=MODE interaction mode: novice, intermediate, or expert + [default: intermediate] + + Dangerous Options: + Caution: use these options at your own risk. It is believed that some + of them bite. + + -g Group option. + + Debug Options: + -d, --debug Print debug information + -s, --sql Print all SQL statements executed + -e Print every action done + +Another interesting method, in particular when working programmatically with +option groups is: - options: - -h, --help show this help message and exit - -v, --verbose make lots of noise [default] - -q, --quiet be vewwy quiet (I'm hunting wabbits) - -fFILE, --file=FILE write output to FILE - -mMODE, --mode=MODE interaction mode: one of 'novice', 'intermediate' - [default], 'expert' +.. method:: OptionParser.get_option_group(opt_str) - Dangerous Options: - Caution: use of these options is at your own risk. It is believed that - some of them bite. - -g Group option. + Return, if defined, the :class:`OptionGroup` that has the title or the long + description equals to *opt_str* .. _optparse-printing-version-string: @@ -592,11 +671,11 @@ argument to OptionParser:: parser = OptionParser(usage="%prog [-f] [-q]", version="%prog 1.0") -``"%prog"`` is expanded just like it is in ``usage``. Apart from that, +``%prog`` is expanded just like it is in ``usage``. Apart from that, ``version`` can contain anything you like. When you supply it, :mod:`optparse` -automatically adds a ``"--version"`` option to your parser. If it encounters +automatically adds a ``--version`` option to your parser. If it encounters this option on the command line, it expands your ``version`` string (by -replacing ``"%prog"``), prints it to stdout, and exits. +replacing ``%prog``), prints it to stdout, and exits. For example, if your script is called ``/usr/bin/foo``:: @@ -609,7 +688,7 @@ The following two methods can be used to print and get the ``version`` string: Print the version message for the current program (``self.version``) to *file* (default stdout). As with :meth:`print_usage`, any occurrence - of ``"%prog"`` in ``self.version`` is replaced with the name of the current + of ``%prog`` in ``self.version`` is replaced with the name of the current program. Does nothing if ``self.version`` is empty or undefined. .. method:: OptionParser.get_version() @@ -632,9 +711,9 @@ usual way: raise an exception (either :exc:`optparse.OptionError` or Handling user errors is much more important, since they are guaranteed to happen no matter how stable your code is. :mod:`optparse` can automatically detect -some user errors, such as bad option arguments (passing ``"-n 4x"`` where -:option:`-n` takes an integer argument), missing arguments (``"-n"`` at the end -of the command line, where :option:`-n` takes an argument of any type). Also, +some user errors, such as bad option arguments (passing ``-n 4x`` where +``-n`` takes an integer argument), missing arguments (``-n`` at the end +of the command line, where ``-n`` takes an argument of any type). Also, you can call :func:`OptionParser.error` to signal an application-defined error condition:: @@ -647,18 +726,18 @@ In either case, :mod:`optparse` handles the error the same way: it prints the program's usage message and an error message to standard error and exits with error status 2. -Consider the first example above, where the user passes ``"4x"`` to an option +Consider the first example above, where the user passes ``4x`` to an option that takes an integer:: $ /usr/bin/foo -n 4x - usage: foo [options] + Usage: foo [options] foo: error: option -n: invalid integer value: '4x' Or, where the user fails to pass a value at all:: $ /usr/bin/foo -n - usage: foo [options] + Usage: foo [options] foo: error: -n option requires an argument @@ -740,8 +819,8 @@ The first step in using :mod:`optparse` is to create an OptionParser instance. ``version`` (default: ``None``) A version string to print when the user supplies a version option. If you supply a true value for ``version``, :mod:`optparse` automatically adds a - version option with the single option string ``"--version"``. The - substring ``"%prog"`` is expanded the same as for ``usage``. + version option with the single option string ``--version``. The + substring ``%prog`` is expanded the same as for ``usage``. ``conflict_handler`` (default: ``"error"``) Specifies what to do when options with conflicting option strings are @@ -760,11 +839,11 @@ The first step in using :mod:`optparse` is to create an OptionParser instance. IndentedHelpFormatter and TitledHelpFormatter. ``add_help_option`` (default: ``True``) - If true, :mod:`optparse` will add a help option (with option strings ``"-h"`` - and ``"--help"``) to the parser. + If true, :mod:`optparse` will add a help option (with option strings ``-h`` + and ``--help``) to the parser. ``prog`` - The string to use when expanding ``"%prog"`` in ``usage`` and ``version`` + The string to use when expanding ``%prog`` in ``usage`` and ``version`` instead of ``os.path.basename(sys.argv[0])``. ``epilog`` (default: ``None``) @@ -808,7 +887,7 @@ Defining options ^^^^^^^^^^^^^^^^ Each Option instance represents a set of synonymous command-line option strings, -e.g. :option:`-f` and :option:`--file`. You can specify any number of short or +e.g. ``-f`` and ``--file``. You can specify any number of short or long option strings, but you must specify at least one overall option string. The canonical way to create an :class:`Option` instance is with the @@ -971,7 +1050,7 @@ relevant to a particular option, or fail to pass a required option attribute, .. attribute:: Option.help Help text to print for this option when listing all available options after - the user supplies a :attr:`~Option.help` option (such as ``"--help"``). If + the user supplies a :attr:`~Option.help` option (such as ``--help``). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value :data:`optparse.SUPPRESS_HELP`. @@ -1009,9 +1088,9 @@ must specify for any option using that action. If :attr:`~Option.type` is not supplied, it defaults to ``"string"``. If :attr:`~Option.dest` is not supplied, :mod:`optparse` derives a destination - from the first long option string (e.g., ``"--foo-bar"`` implies + from the first long option string (e.g., ``--foo-bar`` implies ``foo_bar``). If there are no long option strings, :mod:`optparse` derives a - destination from the first short option string (e.g., ``"-f"`` implies ``f``). + destination from the first short option string (e.g., ``-f`` implies ``f``). Example:: @@ -1042,7 +1121,7 @@ must specify for any option using that action. parser.add_option("--noisy", action="store_const", const=2, dest="verbose") - If ``"--noisy"`` is seen, :mod:`optparse` will set :: + If ``--noisy`` is seen, :mod:`optparse` will set :: options.verbose = 2 @@ -1077,13 +1156,13 @@ must specify for any option using that action. parser.add_option("-t", "--tracks", action="append", type="int") - If ``"-t3"`` is seen on the command-line, :mod:`optparse` does the equivalent + If ``-t3`` is seen on the command-line, :mod:`optparse` does the equivalent of:: options.tracks = [] options.tracks.append(int("3")) - If, a little later on, ``"--tracks=4"`` is seen, it does:: + If, a little later on, ``--tracks=4`` is seen, it does:: options.tracks.append(int("4")) @@ -1105,13 +1184,13 @@ must specify for any option using that action. parser.add_option("-v", action="count", dest="verbosity") - The first time ``"-v"`` is seen on the command line, :mod:`optparse` does the + The first time ``-v`` is seen on the command line, :mod:`optparse` does the equivalent of:: options.verbosity = 0 options.verbosity += 1 - Every subsequent occurrence of ``"-v"`` results in :: + Every subsequent occurrence of ``-v`` results in :: options.verbosity += 1 @@ -1154,15 +1233,15 @@ must specify for any option using that action. help="Input file to read data from") parser.add_option("--secret", help=SUPPRESS_HELP) - If :mod:`optparse` sees either ``"-h"`` or ``"--help"`` on the command line, + If :mod:`optparse` sees either ``-h`` or ``--help`` on the command line, it will print something like the following help message to stdout (assuming ``sys.argv[0]`` is ``"foo.py"``): .. code-block:: text - usage: foo.py [options] + Usage: foo.py [options] - options: + Options: -h, --help Show this help message and exit -v Be moderately verbose --file=FILENAME Input file to read data from @@ -1211,7 +1290,7 @@ although with a more useful error message. :func:`float` and :func:`complex`, with similar error-handling. ``"choice"`` options are a subtype of ``"string"`` options. The -:attr:`~Option.choices`` option attribute (a sequence of strings) defines the +:attr:`~Option.choices` option attribute (a sequence of strings) defines the set of allowed option arguments. :func:`optparse.check_choice` compares user-supplied option arguments against this master list and raises :exc:`OptionValueError` if an invalid string is given. @@ -1268,8 +1347,8 @@ provides several methods to help you out: .. method:: OptionParser.disable_interspersed_args() - Set parsing to stop on the first non-option. For example, if ``"-a"`` and - ``"-b"`` are both simple options that take no arguments, :mod:`optparse` + Set parsing to stop on the first non-option. For example, if ``-a`` and + ``-b`` are both simple options that take no arguments, :mod:`optparse` normally accepts this syntax:: prog -a arg1 -b arg2 @@ -1299,7 +1378,7 @@ provides several methods to help you out: .. method:: OptionParser.has_option(opt_str) Return true if the OptionParser has an option with option string *opt_str* - (e.g., ``"-q"`` or ``"--verbose"``). + (e.g., ``-q`` or ``--verbose``). .. method:: OptionParser.remove_option(opt_str) @@ -1352,12 +1431,12 @@ intelligently and add conflicting options to it:: parser.add_option("-n", "--noisy", ..., help="be noisy") At this point, :mod:`optparse` detects that a previously-added option is already -using the ``"-n"`` option string. Since ``conflict_handler`` is ``"resolve"``, -it resolves the situation by removing ``"-n"`` from the earlier option's list of -option strings. Now ``"--dry-run"`` is the only way for the user to activate +using the ``-n`` option string. Since ``conflict_handler`` is ``"resolve"``, +it resolves the situation by removing ``-n`` from the earlier option's list of +option strings. Now ``--dry-run`` is the only way for the user to activate that option. If the user asks for help, the help message will reflect that:: - options: + Options: --dry-run do no harm [...] -n, --noisy be noisy @@ -1370,10 +1449,10 @@ existing OptionParser:: parser.add_option("--dry-run", ..., help="new dry-run option") -At this point, the original :option:`-n/--dry-run` option is no longer +At this point, the original ``-n``/``--dry-run`` option is no longer accessible, so :mod:`optparse` removes it, leaving this help text:: - options: + Options: [...] -n, --noisy be noisy --dry-run new dry-run option @@ -1408,7 +1487,7 @@ OptionParser supports several other public methods: .. method:: OptionParser.print_usage(file=None) Print the usage message for the current program (``self.usage``) to *file* - (default stdout). Any occurrence of the string ``"%prog"`` in ``self.usage`` + (default stdout). Any occurrence of the string ``%prog`` in ``self.usage`` is replaced with the name of the current program. Does nothing if ``self.usage`` is empty or not defined. @@ -1472,9 +1551,9 @@ only option attribute you must specify is ``callback``, the function to call:: ``callback`` is a function (or other callable object), so you must have already defined ``my_callback()`` when you create this callback option. In this simple -case, :mod:`optparse` doesn't even know if :option:`-c` takes any arguments, +case, :mod:`optparse` doesn't even know if ``-c`` takes any arguments, which usually means that the option takes no arguments---the mere presence of -:option:`-c` on the command-line is all it needs to know. In some +``-c`` on the command-line is all it needs to know. In some circumstances, though, you might want your callback to consume an arbitrary number of command-line arguments. This is where writing callbacks gets tricky; it's covered later in this section. @@ -1527,8 +1606,8 @@ where ``opt_str`` is the option string seen on the command-line that's triggering the callback. (If an abbreviated long option was used, ``opt_str`` will be the full, - canonical option string---e.g. if the user puts ``"--foo"`` on the - command-line as an abbreviation for ``"--foobar"``, then ``opt_str`` will be + canonical option string---e.g. if the user puts ``--foo`` on the + command-line as an abbreviation for ``--foobar``, then ``opt_str`` will be ``"--foobar"``.) ``value`` @@ -1603,8 +1682,8 @@ Of course, you could do that with the ``"store_true"`` action. Callback example 2: check option order ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Here's a slightly more interesting example: record the fact that ``"-a"`` is -seen, but blow up if it comes after ``"-b"`` in the command-line. :: +Here's a slightly more interesting example: record the fact that ``-a`` is +seen, but blow up if it comes after ``-b`` in the command-line. :: def check_order(option, opt_str, value, parser): if parser.values.b: @@ -1621,7 +1700,7 @@ Callback example 3: check option order (generalized) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you want to re-use this callback for several similar options (set a flag, but -blow up if ``"-b"`` has already been seen), it needs a bit of work: the error +blow up if ``-b`` has already been seen), it needs a bit of work: the error message and the flag that it sets must be generalized. :: def check_order(option, opt_str, value, parser): @@ -1691,15 +1770,15 @@ For this case, you must write a callback, as :mod:`optparse` doesn't provide any built-in capabilities for it. And you have to deal with certain intricacies of conventional Unix command-line parsing that :mod:`optparse` normally handles for you. In particular, callbacks should implement the conventional rules for bare -``"--"`` and ``"-"`` arguments: +``--`` and ``-`` arguments: -* either ``"--"`` or ``"-"`` can be option arguments +* either ``--`` or ``-`` can be option arguments -* bare ``"--"`` (if not the argument to some option): halt command-line - processing and discard the ``"--"`` +* bare ``--`` (if not the argument to some option): halt command-line + processing and discard the ``--`` -* bare ``"-"`` (if not the argument to some option): halt command-line - processing but keep the ``"-"`` (append it to ``parser.largs``) +* bare ``-`` (if not the argument to some option): halt command-line + processing but keep the ``-`` (append it to ``parser.largs``) If you want an option that takes a variable number of arguments, there are several subtle, tricky issues to worry about. The exact implementation you @@ -1770,7 +1849,7 @@ To add new types, you need to define your own subclass of :mod:`optparse`'s def check_mytype(option, opt, value) where ``option`` is an :class:`Option` instance, ``opt`` is an option string - (e.g., ``"-f"``), and ``value`` is the string from the command line that must + (e.g., ``-f``), and ``value`` is the string from the command line that must be checked and converted to your desired type. ``check_mytype()`` should return an object of the hypothetical type ``mytype``. The value returned by a type-checking function will wind up in the OptionValues instance returned @@ -1882,7 +1961,7 @@ For example, let's add an ``"extend"`` action. This is similar to the standard ``"append"`` action, but instead of taking a single value from the command-line and appending it to an existing list, ``"extend"`` will take multiple values in a single comma-delimited string, and extend an existing list with them. That -is, if ``"--names"`` is an ``"extend"`` option of type ``"string"``, the command +is, if ``--names`` is an ``"extend"`` option of type ``"string"``, the command line :: --names=foo,bar --names blah --names ding,dong diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index ae55dd8..4867e8b 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -268,14 +268,14 @@ write files see :func:`open`, and for accessing the filesystem see the .. function:: split(path) - Split the pathname *path* into a pair, ``(head, tail)`` where *tail* is the last - pathname component and *head* is everything leading up to that. The *tail* part - will never contain a slash; if *path* ends in a slash, *tail* will be empty. If - there is no slash in *path*, *head* will be empty. If *path* is empty, both - *head* and *tail* are empty. Trailing slashes are stripped from *head* unless - it is the root (one or more slashes only). In nearly all cases, ``join(head, - tail)`` equals *path* (the only exception being when there were multiple slashes - separating *head* from *tail*). + Split the pathname *path* into a pair, ``(head, tail)`` where *tail* is the + last pathname component and *head* is everything leading up to that. The + *tail* part will never contain a slash; if *path* ends in a slash, *tail* + will be empty. If there is no slash in *path*, *head* will be empty. If + *path* is empty, both *head* and *tail* are empty. Trailing slashes are + stripped from *head* unless it is the root (one or more slashes only). In + all cases, ``join(head, tail)`` returns a path to the same location as *path* + (but the strings may differ). .. function:: splitdrive(path) @@ -337,8 +337,7 @@ write files see :func:`open`, and for accessing the filesystem see the .. data:: supports_unicode_filenames True if arbitrary Unicode strings can be used as file names (within limitations - imposed by the file system), and if :func:`os.listdir` returns Unicode strings - for a Unicode argument. + imposed by the file system). .. versionadded:: 2.3 diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 5b3a922..b16fb1f 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -50,7 +50,7 @@ Notes on the availability of these functions: .. data:: name The name of the operating system dependent module imported. The following - names have currently been registered: ``'posix'``, ``'nt'``, ``'mac'``, + names have currently been registered: ``'posix'``, ``'nt'``, ``'os2'``, ``'ce'``, ``'java'``, ``'riscos'``. @@ -151,6 +151,17 @@ process and user. Availability: Unix. +.. function:: initgroups(username, gid) + + Call the system initgroups() to initialize the group access list with all of + the groups of which the specified username is a member, plus the specified + group id. + + Availability: Unix. + + .. versionadded:: 2.7 + + .. function:: getlogin() Return the name of the user logged in on the controlling terminal of the @@ -199,6 +210,26 @@ process and user. Availability: Unix. +.. function:: getresuid() + + Return a tuple (ruid, euid, suid) denoting the current process's + real, effective, and saved user ids. + + Availability: Unix. + + .. versionadded:: 2.7 + + +.. function:: getresgid() + + Return a tuple (rgid, egid, sgid) denoting the current process's + real, effective, and saved group ids. + + Availability: Unix. + + .. versionadded:: 2.7 + + .. function:: getuid() .. index:: single: user; id @@ -286,16 +317,34 @@ process and user. Availability: Unix. -.. function:: setreuid(ruid, euid) +.. function:: setregid(rgid, egid) - Set the current process's real and effective user ids. + Set the current process's real and effective group ids. Availability: Unix. -.. function:: setregid(rgid, egid) +.. function:: setresgid(rgid, egid, sgid) - Set the current process's real and effective group ids. + Set the current process's real, effective, and saved group ids. + + Availability: Unix. + + .. versionadded:: 2.7 + + +.. function:: setresuid(ruid, euid, suid) + + Set the current process's real, effective, and saved user ids. + + Availability: Unix. + + .. versionadded:: 2.7 + + +.. function:: setreuid(ruid, euid) + + Set the current process's real and effective user ids. Availability: Unix. @@ -625,7 +674,7 @@ as internal buffering of data. .. function:: fstat(fd) - Return status for file descriptor *fd*, like :func:`stat`. + Return status for file descriptor *fd*, like :func:`~os.stat`. Availability: Unix, Windows. @@ -682,7 +731,9 @@ as internal buffering of data. SEEK_END Parameters to the :func:`lseek` function. Their values are 0, 1, and 2, - respectively. Availability: Windows, Unix. + respectively. + + Availability: Windows, Unix. .. versionadded:: 2.5 @@ -939,16 +990,16 @@ Files and Directories Set the flags of *path* to the numeric *flags*. *flags* may take a combination (bitwise OR) of the following values (as defined in the :mod:`stat` module): - * ``UF_NODUMP`` - * ``UF_IMMUTABLE`` - * ``UF_APPEND`` - * ``UF_OPAQUE`` - * ``UF_NOUNLINK`` - * ``SF_ARCHIVED`` - * ``SF_IMMUTABLE`` - * ``SF_APPEND`` - * ``SF_NOUNLINK`` - * ``SF_SNAPSHOT`` + * :data:`stat.UF_NODUMP` + * :data:`stat.UF_IMMUTABLE` + * :data:`stat.UF_APPEND` + * :data:`stat.UF_OPAQUE` + * :data:`stat.UF_NOUNLINK` + * :data:`stat.SF_ARCHIVED` + * :data:`stat.SF_IMMUTABLE` + * :data:`stat.SF_APPEND` + * :data:`stat.SF_NOUNLINK` + * :data:`stat.SF_SNAPSHOT` Availability: Unix. @@ -1063,9 +1114,10 @@ Files and Directories .. function:: lstat(path) - Like :func:`stat`, but do not follow symbolic links. This is an alias for - :func:`stat` on platforms that do not support symbolic links, such as - Windows. + Perform the equivalent of an :cfunc:`lstat` system call on the given path. + Similar to :func:`~os.stat`, but does not follow symbolic links. On + platforms that do not support symbolic links, this is an alias for + :func:`~os.stat`. .. function:: mkfifo(path[, mode]) @@ -1124,7 +1176,8 @@ Files and Directories Create a directory named *path* with numeric mode *mode*. The default *mode* is ``0777`` (octal). On some systems, *mode* is ignored. Where it is used, the - current umask value is first masked out. + current umask value is first masked out. If the directory already exists, + :exc:`OSError` is raised. It is also possible to create temporary directories; see the :mod:`tempfile` module's :func:`tempfile.mkdtemp` function. @@ -1139,7 +1192,7 @@ Files and Directories single: UNC paths; and os.makedirs() Recursive directory creation function. Like :func:`mkdir`, but makes all - intermediate-level directories needed to contain the leaf directory. Throws an + intermediate-level directories needed to contain the leaf directory. Raises an :exc:`error` exception if the leaf directory already exists or cannot be created. The default *mode* is ``0777`` (octal). On some systems, *mode* is ignored. Where it is used, the current umask value is first masked out. @@ -1262,23 +1315,23 @@ Files and Directories .. function:: stat(path) - Perform a :cfunc:`stat` system call on the given path. The return value is an - object whose attributes correspond to the members of the :ctype:`stat` - structure, namely: :attr:`st_mode` (protection bits), :attr:`st_ino` (inode - number), :attr:`st_dev` (device), :attr:`st_nlink` (number of hard links), - :attr:`st_uid` (user id of owner), :attr:`st_gid` (group id of owner), - :attr:`st_size` (size of file, in bytes), :attr:`st_atime` (time of most recent - access), :attr:`st_mtime` (time of most recent content modification), - :attr:`st_ctime` (platform dependent; time of most recent metadata change on - Unix, or the time of creation on Windows):: + Perform the equivalent of a :cfunc:`stat` system call on the given path. + (This function follows symlinks; to stat a symlink use :func:`lstat`.) - >>> import os - >>> statinfo = os.stat('somefile.txt') - >>> statinfo - (33188, 422511L, 769L, 1, 1032, 100, 926L, 1105022698,1105022732, 1105022732) - >>> statinfo.st_size - 926L - >>> + The return value is an object whose attributes correspond to the members + of the :ctype:`stat` structure, namely: + + * :attr:`st_mode` - protection bits, + * :attr:`st_ino` - inode number, + * :attr:`st_dev` - device, + * :attr:`st_nlink` - number of hard links, + * :attr:`st_uid` - user id of owner, + * :attr:`st_gid` - group id of owner, + * :attr:`st_size` - size of file, in bytes, + * :attr:`st_atime` - time of most recent access, + * :attr:`st_mtime` - time of most recent content modification, + * :attr:`st_ctime` - platform dependent; time of most recent metadata change on + Unix, or the time of creation on Windows) .. versionchanged:: 2.3 If :func:`stat_float_times` returns ``True``, the time values are floats, measuring @@ -1287,39 +1340,60 @@ Files and Directories discussion. On some Unix systems (such as Linux), the following attributes may also be - available: :attr:`st_blocks` (number of blocks allocated for file), - :attr:`st_blksize` (filesystem blocksize), :attr:`st_rdev` (type of device if an - inode device). :attr:`st_flags` (user defined flags for file). + available: + + * :attr:`st_blocks` - number of blocks allocated for file + * :attr:`st_blksize` - filesystem blocksize + * :attr:`st_rdev` - type of device if an inode device + * :attr:`st_flags` - user defined flags for file On other Unix systems (such as FreeBSD), the following attributes may be - available (but may be only filled out if root tries to use them): :attr:`st_gen` - (file generation number), :attr:`st_birthtime` (time of file creation). + available (but may be only filled out if root tries to use them): + + * :attr:`st_gen` - file generation number + * :attr:`st_birthtime` - time of file creation On Mac OS systems, the following attributes may also be available: - :attr:`st_rsize`, :attr:`st_creator`, :attr:`st_type`. - On RISCOS systems, the following attributes are also available: :attr:`st_ftype` - (file type), :attr:`st_attrs` (attributes), :attr:`st_obtype` (object type). + * :attr:`st_rsize` + * :attr:`st_creator` + * :attr:`st_type` - .. index:: module: stat + On RISCOS systems, the following attributes are also available: + + * :attr:`st_ftype` (file type) + * :attr:`st_attrs` (attributes) + * :attr:`st_obtype` (object type). + + .. note:: + + The exact meaning and resolution of the :attr:`st_atime`, :attr:`st_mtime`, and + :attr:`st_ctime` members depends on the operating system and the file system. + For example, on Windows systems using the FAT or FAT32 file systems, + :attr:`st_mtime` has 2-second resolution, and :attr:`st_atime` has only 1-day + resolution. See your operating system documentation for details. - For backward compatibility, the return value of :func:`stat` is also accessible + For backward compatibility, the return value of :func:`~os.stat` is also accessible as a tuple of at least 10 integers giving the most important (and portable) members of the :ctype:`stat` structure, in the order :attr:`st_mode`, :attr:`st_ino`, :attr:`st_dev`, :attr:`st_nlink`, :attr:`st_uid`, :attr:`st_gid`, :attr:`st_size`, :attr:`st_atime`, :attr:`st_mtime`, :attr:`st_ctime`. More items may be added at the end by some implementations. + + .. index:: module: stat + The standard module :mod:`stat` defines functions and constants that are useful for extracting information from a :ctype:`stat` structure. (On Windows, some items are filled with dummy values.) - .. note:: + Example:: - The exact meaning and resolution of the :attr:`st_atime`, :attr:`st_mtime`, and - :attr:`st_ctime` members depends on the operating system and the file system. - For example, on Windows systems using the FAT or FAT32 file systems, - :attr:`st_mtime` has 2-second resolution, and :attr:`st_atime` has only 1-day - resolution. See your operating system documentation for details. + >>> import os + >>> statinfo = os.stat('somefile.txt') + >>> statinfo + (33188, 422511, 769, 1, 1032, 100, 926, 1105022698,1105022732, 1105022732) + >>> statinfo.st_size + 926 Availability: Unix, Windows. @@ -1333,7 +1407,7 @@ Files and Directories .. function:: stat_float_times([newvalue]) Determine whether :class:`stat_result` represents time stamps as float objects. - If *newvalue* is ``True``, future calls to :func:`stat` return floats, if it is + If *newvalue* is ``True``, future calls to :func:`~os.stat` return floats, if it is ``False``, future calls return ints. If *newvalue* is omitted, return the current setting. @@ -1453,8 +1527,8 @@ Files and Directories respectively. Whether a directory can be given for *path* depends on whether the operating system implements directories as files (for example, Windows does not). Note that the exact times you set here may not be returned by a - subsequent :func:`stat` call, depending on the resolution with which your - operating system records access and modification times; see :func:`stat`. + subsequent :func:`~os.stat` call, depending on the resolution with which your + operating system records access and modification times; see :func:`~os.stat`. .. versionchanged:: 2.0 Added support for ``None`` for *times*. @@ -1627,15 +1701,15 @@ to be ignored. .. function:: _exit(n) - Exit to the system with status *n*, without calling cleanup handlers, flushing + Exit the process with status *n*, without calling cleanup handlers, flushing stdio buffers, etc. Availability: Unix, Windows. .. note:: - The standard way to exit is ``sys.exit(n)``. :func:`_exit` should normally only - be used in the child process after a :func:`fork`. + The standard way to exit is ``sys.exit(n)``. :func:`_exit` should + normally only be used in the child process after a :func:`fork`. The following exit codes are defined and can be used with :func:`_exit`, although they are not required. These are typically used for system programs @@ -1838,7 +1912,16 @@ written in Python, such as a mail server's external command delivery program. Send signal *sig* to the process *pid*. Constants for the specific signals available on the host platform are defined in the :mod:`signal` module. - Availability: Unix. + + Windows: The :data:`signal.CTRL_C_EVENT` and + :data:`signal.CTRL_BREAK_EVENT` signals are special signals which can + only be sent to console processes which share a common console window, + e.g., some subprocesses. Any other value for *sig* will cause the process + to be unconditionally killed by the TerminateProcess API, and the exit code + will be set to *sig*. The Windows version of :func:`kill` additionally takes + process handles to be killed. + + .. versionadded:: 2.7 Windows support .. function:: killpg(pgid, sig) diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst index b1cbd12..c46aeae 100644 --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -121,7 +121,7 @@ and ``'exec'`` forms. The :func:`expr` function parses the parameter *source* as if it were an input to ``compile(source, 'file.py', 'eval')``. If the parse succeeds, an ST object is created to hold the internal parse tree representation, otherwise an - appropriate exception is thrown. + appropriate exception is raised. .. function:: suite(source) @@ -129,7 +129,7 @@ and ``'exec'`` forms. The :func:`suite` function parses the parameter *source* as if it were an input to ``compile(source, 'file.py', 'exec')``. If the parse succeeds, an ST object is created to hold the internal parse tree representation, otherwise an - appropriate exception is thrown. + appropriate exception is raised. .. function:: sequence2st(sequence) @@ -139,9 +139,9 @@ and ``'exec'`` forms. to the Python grammar and all nodes are valid node types in the host version of Python, an ST object is created from the internal representation and returned to the called. If there is a problem creating the internal representation, or - if the tree cannot be validated, a :exc:`ParserError` exception is thrown. An + if the tree cannot be validated, a :exc:`ParserError` exception is raised. An ST object created this way should not be assumed to compile correctly; normal - exceptions thrown by compilation may still be initiated when the ST object is + exceptions raised by compilation may still be initiated when the ST object is passed to :func:`compilest`. This may indicate problems not related to syntax (such as a :exc:`MemoryError` exception), but may also be due to constructs such as the result of parsing ``del f(0)``, which escapes the Python parser but is @@ -264,8 +264,8 @@ function for information about the exceptions it can raise. .. exception:: ParserError Exception raised when a failure occurs within the parser module. This is - generally produced for validation failures rather than the built in - :exc:`SyntaxError` thrown during normal parsing. The exception argument is + generally produced for validation failures rather than the built-in + :exc:`SyntaxError` raised during normal parsing. The exception argument is either a string describing the reason of the failure or a tuple containing a sequence causing the failure from a parse tree passed to :func:`sequence2st` and an explanatory string. Calls to :func:`sequence2st` need to be able to @@ -273,7 +273,7 @@ function for information about the exceptions it can raise. will only need to be aware of the simple string values. Note that the functions :func:`compilest`, :func:`expr`, and :func:`suite` may -throw exceptions which are normally thrown by the parsing and compilation +raise exceptions which are normally raised by the parsing and compilation process. These include the built in exceptions :exc:`MemoryError`, :exc:`OverflowError`, :exc:`SyntaxError`, and :exc:`SystemError`. In these cases, these exceptions carry all the meaning normally associated with them. @@ -322,22 +322,8 @@ ST objects have the following methods: Same as ``st2tuple(st, line_info)``. -.. _st-examples: - -Examples --------- - -.. index:: builtin: compile - -The parser modules allows operations to be performed on the parse tree of Python -source code before the :term:`bytecode` is generated, and provides for inspection of the -parse tree for information gathering purposes. Two examples are presented. The -simple example demonstrates emulation of the :func:`compile` built-in function -and the complex example shows the use of a parse tree for information discovery. - - -Emulation of :func:`compile` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Example: Emulation of :func:`compile` +------------------------------------- While many useful operations may take place between parsing and bytecode generation, the simplest operation is to do nothing. For this purpose, using @@ -371,322 +357,3 @@ readily available functions:: def load_expression(source_string): st = parser.expr(source_string) return st, st.compile() - - -Information Discovery -^^^^^^^^^^^^^^^^^^^^^ - -.. index:: - single: string; documentation - single: docstrings - -Some applications benefit from direct access to the parse tree. The remainder -of this section demonstrates how the parse tree provides access to module -documentation defined in docstrings without requiring that the code being -examined be loaded into a running interpreter via :keyword:`import`. This can -be very useful for performing analyses of untrusted code. - -Generally, the example will demonstrate how the parse tree may be traversed to -distill interesting information. Two functions and a set of classes are -developed which provide programmatic access to high level function and class -definitions provided by a module. The classes extract information from the -parse tree and provide access to the information at a useful semantic level, one -function provides a simple low-level pattern matching capability, and the other -function defines a high-level interface to the classes by handling file -operations on behalf of the caller. All source files mentioned here which are -not part of the Python installation are located in the :file:`Demo/parser/` -directory of the distribution. - -The dynamic nature of Python allows the programmer a great deal of flexibility, -but most modules need only a limited measure of this when defining classes, -functions, and methods. In this example, the only definitions that will be -considered are those which are defined in the top level of their context, e.g., -a function defined by a :keyword:`def` statement at column zero of a module, but -not a function defined within a branch of an :keyword:`if` ... :keyword:`else` -construct, though there are some good reasons for doing so in some situations. -Nesting of definitions will be handled by the code developed in the example. - -To construct the upper-level extraction methods, we need to know what the parse -tree structure looks like and how much of it we actually need to be concerned -about. Python uses a moderately deep parse tree so there are a large number of -intermediate nodes. It is important to read and understand the formal grammar -used by Python. This is specified in the file :file:`Grammar/Grammar` in the -distribution. Consider the simplest case of interest when searching for -docstrings: a module consisting of a docstring and nothing else. (See file -:file:`docstring.py`.) :: - - """Some documentation. - """ - -Using the interpreter to take a look at the parse tree, we find a bewildering -mass of numbers and parentheses, with the documentation buried deep in nested -tuples. :: - - >>> import parser - >>> import pprint - >>> st = parser.suite(open('docstring.py').read()) - >>> tup = st.totuple() - >>> pprint.pprint(tup) - (257, - (264, - (265, - (266, - (267, - (307, - (287, - (288, - (289, - (290, - (292, - (293, - (294, - (295, - (296, - (297, - (298, - (299, - (300, (3, '"""Some documentation.\n"""'))))))))))))))))), - (4, ''))), - (4, ''), - (0, '')) - -The numbers at the first element of each node in the tree are the node types; -they map directly to terminal and non-terminal symbols in the grammar. -Unfortunately, they are represented as integers in the internal representation, -and the Python structures generated do not change that. However, the -:mod:`symbol` and :mod:`token` modules provide symbolic names for the node types -and dictionaries which map from the integers to the symbolic names for the node -types. - -In the output presented above, the outermost tuple contains four elements: the -integer ``257`` and three additional tuples. Node type ``257`` has the symbolic -name :const:`file_input`. Each of these inner tuples contains an integer as the -first element; these integers, ``264``, ``4``, and ``0``, represent the node -types :const:`stmt`, :const:`NEWLINE`, and :const:`ENDMARKER`, respectively. -Note that these values may change depending on the version of Python you are -using; consult :file:`symbol.py` and :file:`token.py` for details of the -mapping. It should be fairly clear that the outermost node is related primarily -to the input source rather than the contents of the file, and may be disregarded -for the moment. The :const:`stmt` node is much more interesting. In -particular, all docstrings are found in subtrees which are formed exactly as -this node is formed, with the only difference being the string itself. The -association between the docstring in a similar tree and the defined entity -(class, function, or module) which it describes is given by the position of the -docstring subtree within the tree defining the described structure. - -By replacing the actual docstring with something to signify a variable component -of the tree, we allow a simple pattern matching approach to check any given -subtree for equivalence to the general pattern for docstrings. Since the -example demonstrates information extraction, we can safely require that the tree -be in tuple form rather than list form, allowing a simple variable -representation to be ``['variable_name']``. A simple recursive function can -implement the pattern matching, returning a Boolean and a dictionary of variable -name to value mappings. (See file :file:`example.py`.) :: - - from types import ListType, TupleType - - def match(pattern, data, vars=None): - if vars is None: - vars = {} - if type(pattern) is ListType: - vars[pattern[0]] = data - return 1, vars - if type(pattern) is not TupleType: - return (pattern == data), vars - if len(data) != len(pattern): - return 0, vars - for pattern, data in map(None, pattern, data): - same, vars = match(pattern, data, vars) - if not same: - break - return same, vars - -Using this simple representation for syntactic variables and the symbolic node -types, the pattern for the candidate docstring subtrees becomes fairly readable. -(See file :file:`example.py`.) :: - - import symbol - import token - - DOCSTRING_STMT_PATTERN = ( - symbol.stmt, - (symbol.simple_stmt, - (symbol.small_stmt, - (symbol.expr_stmt, - (symbol.testlist, - (symbol.test, - (symbol.and_test, - (symbol.not_test, - (symbol.comparison, - (symbol.expr, - (symbol.xor_expr, - (symbol.and_expr, - (symbol.shift_expr, - (symbol.arith_expr, - (symbol.term, - (symbol.factor, - (symbol.power, - (symbol.atom, - (token.STRING, ['docstring']) - )))))))))))))))), - (token.NEWLINE, '') - )) - -Using the :func:`match` function with this pattern, extracting the module -docstring from the parse tree created previously is easy:: - - >>> found, vars = match(DOCSTRING_STMT_PATTERN, tup[1]) - >>> found - 1 - >>> vars - {'docstring': '"""Some documentation.\n"""'} - -Once specific data can be extracted from a location where it is expected, the -question of where information can be expected needs to be answered. When -dealing with docstrings, the answer is fairly simple: the docstring is the first -:const:`stmt` node in a code block (:const:`file_input` or :const:`suite` node -types). A module consists of a single :const:`file_input` node, and class and -function definitions each contain exactly one :const:`suite` node. Classes and -functions are readily identified as subtrees of code block nodes which start -with ``(stmt, (compound_stmt, (classdef, ...`` or ``(stmt, (compound_stmt, -(funcdef, ...``. Note that these subtrees cannot be matched by :func:`match` -since it does not support multiple sibling nodes to match without regard to -number. A more elaborate matching function could be used to overcome this -limitation, but this is sufficient for the example. - -Given the ability to determine whether a statement might be a docstring and -extract the actual string from the statement, some work needs to be performed to -walk the parse tree for an entire module and extract information about the names -defined in each context of the module and associate any docstrings with the -names. The code to perform this work is not complicated, but bears some -explanation. - -The public interface to the classes is straightforward and should probably be -somewhat more flexible. Each "major" block of the module is described by an -object providing several methods for inquiry and a constructor which accepts at -least the subtree of the complete parse tree which it represents. The -:class:`ModuleInfo` constructor accepts an optional *name* parameter since it -cannot otherwise determine the name of the module. - -The public classes include :class:`ClassInfo`, :class:`FunctionInfo`, and -:class:`ModuleInfo`. All objects provide the methods :meth:`get_name`, -:meth:`get_docstring`, :meth:`get_class_names`, and :meth:`get_class_info`. The -:class:`ClassInfo` objects support :meth:`get_method_names` and -:meth:`get_method_info` while the other classes provide -:meth:`get_function_names` and :meth:`get_function_info`. - -Within each of the forms of code block that the public classes represent, most -of the required information is in the same form and is accessed in the same way, -with classes having the distinction that functions defined at the top level are -referred to as "methods." Since the difference in nomenclature reflects a real -semantic distinction from functions defined outside of a class, the -implementation needs to maintain the distinction. Hence, most of the -functionality of the public classes can be implemented in a common base class, -:class:`SuiteInfoBase`, with the accessors for function and method information -provided elsewhere. Note that there is only one class which represents function -and method information; this parallels the use of the :keyword:`def` statement -to define both types of elements. - -Most of the accessor functions are declared in :class:`SuiteInfoBase` and do not -need to be overridden by subclasses. More importantly, the extraction of most -information from a parse tree is handled through a method called by the -:class:`SuiteInfoBase` constructor. The example code for most of the classes is -clear when read alongside the formal grammar, but the method which recursively -creates new information objects requires further examination. Here is the -relevant part of the :class:`SuiteInfoBase` definition from :file:`example.py`:: - - class SuiteInfoBase: - _docstring = '' - _name = '' - - def __init__(self, tree = None): - self._class_info = {} - self._function_info = {} - if tree: - self._extract_info(tree) - - def _extract_info(self, tree): - # extract docstring - if len(tree) == 2: - found, vars = match(DOCSTRING_STMT_PATTERN[1], tree[1]) - else: - found, vars = match(DOCSTRING_STMT_PATTERN, tree[3]) - if found: - self._docstring = eval(vars['docstring']) - # discover inner definitions - for node in tree[1:]: - found, vars = match(COMPOUND_STMT_PATTERN, node) - if found: - cstmt = vars['compound'] - if cstmt[0] == symbol.funcdef: - name = cstmt[2][1] - self._function_info[name] = FunctionInfo(cstmt) - elif cstmt[0] == symbol.classdef: - name = cstmt[2][1] - self._class_info[name] = ClassInfo(cstmt) - -After initializing some internal state, the constructor calls the -:meth:`_extract_info` method. This method performs the bulk of the information -extraction which takes place in the entire example. The extraction has two -distinct phases: the location of the docstring for the parse tree passed in, and -the discovery of additional definitions within the code block represented by the -parse tree. - -The initial :keyword:`if` test determines whether the nested suite is of the -"short form" or the "long form." The short form is used when the code block is -on the same line as the definition of the code block, as in :: - - def square(x): "Square an argument."; return x ** 2 - -while the long form uses an indented block and allows nested definitions:: - - def make_power(exp): - "Make a function that raises an argument to the exponent `exp`." - def raiser(x, y=exp): - return x ** y - return raiser - -When the short form is used, the code block may contain a docstring as the -first, and possibly only, :const:`small_stmt` element. The extraction of such a -docstring is slightly different and requires only a portion of the complete -pattern used in the more common case. As implemented, the docstring will only -be found if there is only one :const:`small_stmt` node in the -:const:`simple_stmt` node. Since most functions and methods which use the short -form do not provide a docstring, this may be considered sufficient. The -extraction of the docstring proceeds using the :func:`match` function as -described above, and the value of the docstring is stored as an attribute of the -:class:`SuiteInfoBase` object. - -After docstring extraction, a simple definition discovery algorithm operates on -the :const:`stmt` nodes of the :const:`suite` node. The special case of the -short form is not tested; since there are no :const:`stmt` nodes in the short -form, the algorithm will silently skip the single :const:`simple_stmt` node and -correctly not discover any nested definitions. - -Each statement in the code block is categorized as a class definition, function -or method definition, or something else. For the definition statements, the -name of the element defined is extracted and a representation object appropriate -to the definition is created with the defining subtree passed as an argument to -the constructor. The representation objects are stored in instance variables -and may be retrieved by name using the appropriate accessor methods. - -The public classes provide any accessors required which are more specific than -those provided by the :class:`SuiteInfoBase` class, but the real extraction -algorithm remains common to all forms of code blocks. A high-level function can -be used to extract the complete set of information from a source file. (See -file :file:`example.py`.) :: - - def get_docs(fileName): - import os - import parser - - source = open(fileName).read() - basename = os.path.basename(os.path.splitext(fileName)[0]) - st = parser.suite(source) - return ModuleInfo(st.totuple(), basename) - -This provides an easy-to-use interface to the documentation of a module. If -information is required which is not extracted by the code of this example, the -code may be extended at clearly defined points to provide additional -capabilities. - diff --git a/Doc/library/pdb.rst b/Doc/library/pdb.rst index 7f2197e..a475099 100644 --- a/Doc/library/pdb.rst +++ b/Doc/library/pdb.rst @@ -1,4 +1,3 @@ - .. _debugger: :mod:`pdb` --- The Python Debugger @@ -79,10 +78,10 @@ The typical usage to inspect a crashed program is:: -> print spam (Pdb) + The module defines the following functions; each enters the debugger in a slightly different way: - .. function:: run(statement[, globals[, locals]]) Execute the *statement* (given as a string) under debugger control. The @@ -126,7 +125,38 @@ slightly different way: .. function:: pm() - Enter post-mortem debugging of the traceback found in ``sys.last_traceback``. + Enter post-mortem debugging of the traceback found in + :data:`sys.last_traceback`. + + +The ``run*`` functions and :func:`set_trace` are aliases for instantiating the +:class:`Pdb` class and calling the method of the same name. If you want to +access further features, you have to do this yourself: + +.. class:: Pdb(completekey='tab', stdin=None, stdout=None, skip=None) + + :class:`Pdb` is the debugger class. + + The *completekey*, *stdin* and *stdout* arguments are passed to the + underlying :class:`cmd.Cmd` class; see the description there. + + The *skip* argument, if given, must be an iterable of glob-style module name + patterns. The debugger will not step into frames that originate in a module + that matches one of these patterns. [1]_ + + Example call to enable tracing with *skip*:: + + import pdb; pdb.Pdb(skip=['django.*']).set_trace() + + .. versionadded:: 2.7 + The *skip* argument. + + .. method:: run(statement[, globals[, locals]]) + runeval(expression[, globals[, locals]]) + runcall(function[, argument, ...]) + set_trace() + + See the documentation for the functions explained above. .. _debugger-commands: @@ -209,7 +239,8 @@ tbreak [[*filename*:]\ *lineno* | *function*\ [, *condition*]] Temporary breakpoint, which is removed automatically when it is first hit. The arguments are the same as break. -cl(ear) [*bpnumber* [*bpnumber ...*]] +cl(ear) [*filename:lineno* | *bpnumber* [*bpnumber ...*]] + With a *filename:lineno* argument, clear all the breakpoints at this line. With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation). @@ -360,3 +391,9 @@ run [*args* ...] q(uit) Quit from the debugger. The program being executed is aborted. + + +.. rubric:: Footnotes + +.. [1] Whether a frame is considered to originate in a certain module + is determined by the ``__name__`` in the frame globals. diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index a3e5ad4..22165f9 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -25,6 +25,12 @@ confusion, the terms used here are "pickling" and "unpickling". This documentation describes both the :mod:`pickle` module and the :mod:`cPickle` module. +.. warning:: + + The :mod:`pickle` module is not intended to be secure against erroneous or + maliciously constructed data. Never unpickle data received from an untrusted + or unauthenticated source. + Relationship to other Python modules ------------------------------------ @@ -47,7 +53,7 @@ general :mod:`pickle` should always be the preferred way to serialize Python objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc` files. -The :mod:`pickle` module differs from :mod:`marshal` several significant ways: +The :mod:`pickle` module differs from :mod:`marshal` in several significant ways: * The :mod:`pickle` module keeps track of the objects it has already serialized, so that later references to the same object won't be serialized again. @@ -74,12 +80,6 @@ The :mod:`pickle` module differs from :mod:`marshal` several significant ways: The :mod:`pickle` serialization format is guaranteed to be backwards compatible across Python releases. -.. warning:: - - The :mod:`pickle` module is not intended to be secure against erroneous or - maliciously constructed data. Never unpickle data received from an untrusted - or unauthenticated source. - Note that serialization is a more primitive notion than persistence; although :mod:`pickle` reads and writes file objects, it does not handle the issue of naming persistent objects, nor the (even more complicated) issue of concurrent @@ -444,7 +444,7 @@ Pickling and unpickling normal class instances instance's dictionary. If there is no :meth:`__getstate__` method, the instance's :attr:`__dict__` is pickled. -.. method:: object.__setstate__() +.. method:: object.__setstate__(state) Upon unpickling, if the class also defines the method :meth:`__setstate__`, it is called with the unpickled state. [#]_ If there is no diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 5189377..9e634d6 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -3,62 +3,187 @@ ============================================ .. module:: pkgutil - :synopsis: Utilities to support extension of packages. + :synopsis: Utilities for the import system. +This module provides utilities for the import system, in particular package +support. .. versionadded:: 2.3 -This module provides functions to manipulate packages: - .. function:: extend_path(path, name) - Extend the search path for the modules which comprise a package. Intended use is - to place the following code in a package's :file:`__init__.py`:: + Extend the search path for the modules which comprise a package. Intended + use is to place the following code in a package's :file:`__init__.py`:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) - This will add to the package's ``__path__`` all subdirectories of directories on - ``sys.path`` named after the package. This is useful if one wants to distribute - different parts of a single logical package as multiple directories. + This will add to the package's ``__path__`` all subdirectories of directories + on ``sys.path`` named after the package. This is useful if one wants to + distribute different parts of a single logical package as multiple + directories. - It also looks for :file:`\*.pkg` files beginning where ``*`` matches the *name* - argument. This feature is similar to :file:`\*.pth` files (see the :mod:`site` - module for more information), except that it doesn't special-case lines starting - with ``import``. A :file:`\*.pkg` file is trusted at face value: apart from - checking for duplicates, all entries found in a :file:`\*.pkg` file are added to - the path, regardless of whether they exist on the filesystem. (This is a - feature.) + It also looks for :file:`\*.pkg` files beginning where ``*`` matches the + *name* argument. This feature is similar to :file:`\*.pth` files (see the + :mod:`site` module for more information), except that it doesn't special-case + lines starting with ``import``. A :file:`\*.pkg` file is trusted at face + value: apart from checking for duplicates, all entries found in a + :file:`\*.pkg` file are added to the path, regardless of whether they exist + on the filesystem. (This is a feature.) If the input path is not a list (as is the case for frozen packages) it is returned unchanged. The input path is not modified; an extended copy is returned. Items are only appended to the copy at the end. - It is assumed that ``sys.path`` is a sequence. Items of ``sys.path`` that are - not (Unicode or 8-bit) strings referring to existing directories are ignored. - Unicode items on ``sys.path`` that cause errors when used as filenames may cause - this function to raise an exception (in line with :func:`os.path.isdir` - behavior). + It is assumed that :data:`sys.path` is a sequence. Items of :data:`sys.path` + that are not (Unicode or 8-bit) strings referring to existing directories are + ignored. Unicode items on :data:`sys.path` that cause errors when used as + filenames may cause this function to raise an exception (in line with + :func:`os.path.isdir` behavior). + + +.. class:: ImpImporter(dirname=None) + + :pep:`302` Importer that wraps Python's "classic" import algorithm. + + If *dirname* is a string, a :pep:`302` importer is created that searches that + directory. If *dirname* is ``None``, a :pep:`302` importer is created that + searches the current :data:`sys.path`, plus any modules that are frozen or + built-in. + + Note that :class:`ImpImporter` does not currently support being used by + placement on :data:`sys.meta_path`. + + +.. class:: ImpLoader(fullname, file, filename, etc) + + :pep:`302` Loader that wraps Python's "classic" import algorithm. + + +.. function:: find_loader(fullname) + + Find a :pep:`302` "loader" object for *fullname*. + + If *fullname* contains dots, path must be the containing package's + ``__path__``. Returns ``None`` if the module cannot be found or imported. + This function uses :func:`iter_importers`, and is thus subject to the same + limitations regarding platform-specific special import locations such as the + Windows registry. + + +.. function:: get_importer(path_item) + + Retrieve a :pep:`302` importer for the given *path_item*. + + The returned importer is cached in :data:`sys.path_importer_cache` if it was + newly created by a path hook. + + If there is no importer, a wrapper around the basic import machinery is + returned. This wrapper is never inserted into the importer cache (``None`` + is inserted instead). + + The cache (or part of it) can be cleared manually if a rescan of + :data:`sys.path_hooks` is necessary. + + +.. function:: get_loader(module_or_name) + + Get a :pep:`302` "loader" object for *module_or_name*. + + If the module or package is accessible via the normal import mechanism, a + wrapper around the relevant part of that machinery is returned. Returns + ``None`` if the module cannot be found or imported. If the named module is + not already imported, its containing package (if any) is imported, in order + to establish the package ``__path__``. + + This function uses :func:`iter_importers`, and is thus subject to the same + limitations regarding platform-specific special import locations such as the + Windows registry. + + +.. function:: iter_importers(fullname='') + + Yield :pep:`302` importers for the given module name. + + If fullname contains a '.', the importers will be for the package containing + fullname, otherwise they will be importers for :data:`sys.meta_path`, + :data:`sys.path`, and Python's "classic" import machinery, in that order. If + the named module is in a package, that package is imported as a side effect + of invoking this function. + + Non-:pep:`302` mechanisms (e.g. the Windows registry) used by the standard + import machinery to find files in alternative locations are partially + supported, but are searched *after* :data:`sys.path`. Normally, these + locations are searched *before* :data:`sys.path`, preventing :data:`sys.path` + entries from shadowing them. + + For this to cause a visible difference in behaviour, there must be a module + or package name that is accessible via both :data:`sys.path` and one of the + non-:pep:`302` file system mechanisms. In this case, the emulation will find + the former version, while the builtin import mechanism will find the latter. + + Items of the following types can be affected by this discrepancy: + ``imp.C_EXTENSION``, ``imp.PY_SOURCE``, ``imp.PY_COMPILED``, + ``imp.PKG_DIRECTORY``. + + +.. function:: iter_modules(path=None, prefix='') + + Yields ``(module_loader, name, ispkg)`` for all submodules on *path*, or, if + path is ``None``, all top-level modules on ``sys.path``. + + *path* should be either ``None`` or a list of paths to look for modules in. + + *prefix* is a string to output on the front of every module name on output. + + +.. function:: walk_packages(path=None, prefix='', onerror=None) + + Yields ``(module_loader, name, ispkg)`` for all modules recursively on + *path*, or, if path is ``None``, all accessible modules. + + *path* should be either ``None`` or a list of paths to look for modules in. + + *prefix* is a string to output on the front of every module name on output. + + Note that this function must import all *packages* (*not* all modules!) on + the given *path*, in order to access the ``__path__`` attribute to find + submodules. + + *onerror* is a function which gets called with one argument (the name of the + package which was being imported) if any exception occurs while trying to + import a package. If no *onerror* function is supplied, :exc:`ImportError`\s + are caught and ignored, while all other exceptions are propagated, + terminating the search. + + Examples:: + + # list all modules python can access + walk_packages() + + # list all submodules of ctypes + walk_packages(ctypes.__path__, ctypes.__name__ + '.') + .. function:: get_data(package, resource) Get a resource from a package. - This is a wrapper for the :pep:`302` loader :func:`get_data` API. The package - argument should be the name of a package, in standard module format - (foo.bar). The resource argument should be in the form of a relative - filename, using ``/`` as the path separator. The parent directory name + This is a wrapper for the :pep:`302` loader :func:`get_data` API. The + *package* argument should be the name of a package, in standard module format + (``foo.bar``). The *resource* argument should be in the form of a relative + filename, using ``/`` as the path separator. The parent directory name ``..`` is not allowed, and nor is a rooted name (starting with a ``/``). - The function returns a binary string that is the contents of the - specified resource. + The function returns a binary string that is the contents of the specified + resource. For packages located in the filesystem, which have already been imported, this is the rough equivalent of:: - d = os.path.dirname(sys.modules[package].__file__) - data = open(os.path.join(d, resource), 'rb').read() + d = os.path.dirname(sys.modules[package].__file__) + data = open(os.path.join(d, resource), 'rb').read() If the package cannot be located or loaded, or it uses a :pep:`302` loader - which does not support :func:`get_data`, then None is returned. + which does not support :func:`get_data`, then ``None`` is returned. diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index b98d992..7802422 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -38,6 +38,16 @@ Cross Platform and then only if the executable points to the Python interpreter. Reasonable defaults are used when the above needs are not met. + .. note:: + + On Mac OS X (and perhaps other platforms), executable files may be + universal files containing multiple architectures. + + To get at the "64-bitness" of the current interpreter, it is more + reliable to query the :attr:`sys.maxsize` attribute:: + + is_64bits = sys.maxsize > 2**32 + .. function:: machine() @@ -194,7 +204,7 @@ Windows Platform .. note:: - Note: this function works best with Mark Hammond's + This function works best with Mark Hammond's :mod:`win32all` package installed, but also on Python 2.3 and later (support for this was added in Python 2.6). It obviously only runs on Win32 compatible platforms. diff --git a/Doc/library/pprint.rst b/Doc/library/pprint.rst index 36a2bc4..7f13029 100644 --- a/Doc/library/pprint.rst +++ b/Doc/library/pprint.rst @@ -28,6 +28,11 @@ width constraint. .. versionchanged:: 2.6 Added support for :class:`set` and :class:`frozenset`. +.. seealso:: + + Latest version of the `pprint module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/pprint.py?view=markup>`_ + The :mod:`pprint` module defines one class: .. First the implementation class: diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index a69a0da..1e2350f 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -620,7 +620,7 @@ The resulting profiler will then call :func:`your_time_func`. integers, you can also invoke the class constructor with a second argument specifying the real duration of one unit of time. For example, if :func:`your_integer_time_func` returns times measured in thousands of seconds, - you would constuct the :class:`Profile` instance as follows:: + you would construct the :class:`Profile` instance as follows:: pr = profile.Profile(your_integer_time_func, 0.001) diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst index 40ca588..ca86143 100644 --- a/Doc/library/py_compile.rst +++ b/Doc/library/py_compile.rst @@ -40,6 +40,11 @@ byte-code cache files in the directory containing the source code. line, if *args* is not specified) are compiled and the resulting bytecode is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. + If ``'-'`` is the only parameter in args, the list of files is taken from + standard input. + + .. versionchanged:: 2.7 + Added support for ``'-'``. When this module is run as a script, the :func:`main` is used to compile all the files named on the command line. The exit status is nonzero if one of the files diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst index 3784515..e5f8fe2 100644 --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -43,25 +43,25 @@ produced for that file. executed on that occasion. Use an ``if __name__ == '__main__':`` guard to only execute code when a file is invoked as a script and not just imported. -Specifying a :option:`-w` flag before the argument will cause HTML documentation +Specifying a ``-w`` flag before the argument will cause HTML documentation to be written out to a file in the current directory, instead of displaying text on the console. -Specifying a :option:`-k` flag before the argument will search the synopsis +Specifying a ``-k`` flag before the argument will search the synopsis lines of all available modules for the keyword given as the argument, again in a manner similar to the Unix :program:`man` command. The synopsis line of a module is the first line of its documentation string. You can also use :program:`pydoc` to start an HTTP server on the local machine -that will serve documentation to visiting Web browsers. :program:`pydoc` -:option:`-p 1234` will start a HTTP server on port 1234, allowing you to browse +that will serve documentation to visiting Web browsers. :program:`pydoc -p 1234` +will start a HTTP server on port 1234, allowing you to browse the documentation at ``http://localhost:1234/`` in your preferred Web browser. -:program:`pydoc` :option:`-g` will start the server and additionally bring up a +:program:`pydoc -g` will start the server and additionally bring up a small :mod:`Tkinter`\ -based graphical interface to help you search for documentation pages. When :program:`pydoc` generates documentation, it uses the current environment -and path to locate modules. Thus, invoking :program:`pydoc` :option:`spam` +and path to locate modules. Thus, invoking :program:`pydoc spam` documents precisely the version of the module you would get if you started the Python interpreter and typed ``import spam``. diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 04179a4..8299739 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -157,6 +157,13 @@ XMLParser Objects :attr:`ordered_attributes`, :attr:`returns_unicode` and :attr:`specified_attributes` set to the values of this parser. +.. method:: xmlparser.SetParamEntityParsing(flag) + + Control parsing of parameter entities (including the external DTD subset). + Possible *flag* values are :const:`XML_PARAM_ENTITY_PARSING_NEVER`, + :const:`XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE` and + :const:`XML_PARAM_ENTITY_PARSING_ALWAYS`. Return true if setting the flag + was successful. .. method:: xmlparser.UseForeignDTD([flag]) @@ -464,7 +471,7 @@ otherwise stated. Called if the XML document hasn't been declared as being a standalone document. This happens when there is an external subset or a reference to a parameter entity, but the XML declaration does not set standalone to ``yes`` in an XML - declaration. If this handler returns ``0``, then the parser will throw an + declaration. If this handler returns ``0``, then the parser will raise an :const:`XML_ERROR_NOT_STANDALONE` error. If this handler is not set, no exception is raised by the parser for this condition. @@ -481,7 +488,7 @@ otherwise stated. responsible for creating the sub-parser using ``ExternalEntityParserCreate(context)``, initializing it with the appropriate callbacks, and parsing the entity. This handler should return an integer; if it - returns ``0``, the parser will throw an + returns ``0``, the parser will raise an :const:`XML_ERROR_EXTERNAL_ENTITY_HANDLING` error, otherwise parsing will continue. diff --git a/Doc/library/python.rst b/Doc/library/python.rst index 6f4ecb7..aaf1abd 100644 --- a/Doc/library/python.rst +++ b/Doc/library/python.rst @@ -13,6 +13,7 @@ overview: .. toctree:: sys.rst + sysconfig.rst __builtin__.rst future_builtins.rst __main__.rst diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 9765102..a471bb0 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -24,6 +24,11 @@ the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the lowest valued entry is retrieved first. +.. seealso:: + + Latest version of the `queue module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/Queue.py?view=markup>`_. + The :mod:`Queue` module defines the following classes and exceptions: .. class:: Queue(maxsize=0) @@ -183,7 +188,7 @@ Example of how to wait for enqueued tasks to be completed:: q = Queue() for i in range(num_worker_threads): t = Thread(target=worker) - t.setDaemon(True) + t.daemon = True t.start() for item in source(): diff --git a/Doc/library/quopri.rst b/Doc/library/quopri.rst index f4674dc..87d70c3 100644 --- a/Doc/library/quopri.rst +++ b/Doc/library/quopri.rst @@ -18,6 +18,10 @@ few nonprintable characters; the base64 encoding scheme available via the :mod:`base64` module is more compact if there are many such characters, as when sending a graphics file. +.. seealso:: + + Latest version of the `quopri module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/quopri.py?view=markup>`_ .. function:: decode(input, output[,header]) diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 486e475..4306911 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -9,6 +9,11 @@ This module implements pseudo-random number generators for various distributions. +.. seealso:: + + Latest version of the `random module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/random.py?view=markup>`_ + For integers, uniform selection from a range. For sequences, uniform selection of a random element, a function to generate a random permutation of a list in-place, and a function for random sampling without replacement. @@ -73,10 +78,6 @@ Bookkeeping functions: .. versionchanged:: 2.4 formerly, operating system resources were not used. - If *x* is not ``None`` or an int or long, ``hash(x)`` is used instead. If *x* is - an int or long, *x* is used directly. - - .. function:: getstate() Return an object capturing the current internal state of the generator. This diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 9821047..f789601 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -224,7 +224,7 @@ The special characters are: undefined. ``(?:...)`` - A non-grouping version of regular parentheses. Matches whatever regular + A non-capturing version of regular parentheses. Matches whatever regular expression is inside the parentheses, but the substring matched by the group *cannot* be retrieved after performing a match or referenced later in the pattern. @@ -332,7 +332,8 @@ the second character. For example, ``\$`` matches the character ``'$'``. ``\d`` When the :const:`UNICODE` flag is not specified, matches any decimal digit; this is equivalent to the set ``[0-9]``. With :const:`UNICODE`, it will match - whatever is classified as a digit in the Unicode character properties database. + whatever is classified as a decimal digit in the Unicode character properties + database. ``\D`` When the :const:`UNICODE` flag is not specified, matches any non-digit @@ -536,7 +537,7 @@ form. instead. -.. function:: split(pattern, string[, maxsplit=0]) +.. function:: split(pattern, string[, maxsplit=0, flags=0]) Split *string* by the occurrences of *pattern*. If capturing parentheses are used in *pattern*, then the text of all groups in the pattern are also returned @@ -551,6 +552,8 @@ form. ['Words', ', ', 'words', ', ', 'words', '.', ''] >>> re.split('\W+', 'Words, words, words.', 1) ['Words', 'words, words.'] + >>> re.split('[a-f]+', '0a3B9', flags=re.IGNORECASE) + ['0', '3', '9'] If there are capturing groups in the separator and it matches at the start of the string, the result will start with an empty string. The same holds for @@ -571,6 +574,9 @@ form. >>> re.split("(?m)^$", "foo\n\nbar\n") ['foo\n\nbar\n'] + .. versionchanged:: 2.7 + Added the optional flags argument. + .. function:: findall(pattern, string[, flags]) @@ -601,7 +607,7 @@ form. Added the optional flags argument. -.. function:: sub(pattern, repl, string[, count]) +.. function:: sub(pattern, repl, string[, count, flags]) Return the string obtained by replacing the leftmost non-overlapping occurrences of *pattern* in *string* by the replacement *repl*. If the pattern isn't found, @@ -626,10 +632,10 @@ form. ... else: return '-' >>> re.sub('-{1,2}', dashrepl, 'pro----gram-files') 'pro--gram files' + >>> re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam', flags=re.IGNORECASE) + 'Baked Beans & Spam' - The pattern may be a string or an RE object; if you need to specify regular - expression flags, you must use a RE object, or use embedded modifiers in a - pattern; for example, ``sub("(?i)b+", "x", "bbbb BBBB")`` returns ``'x x'``. + The pattern may be a string or an RE object. The optional argument *count* is the maximum number of pattern occurrences to be replaced; *count* must be a non-negative integer. If omitted or zero, all @@ -646,12 +652,18 @@ form. character ``'0'``. The backreference ``\g<0>`` substitutes in the entire substring matched by the RE. + .. versionchanged:: 2.7 + Added the optional flags argument. + -.. function:: subn(pattern, repl, string[, count]) +.. function:: subn(pattern, repl, string[, count, flags]) Perform the same operation as :func:`sub`, but return a tuple ``(new_string, number_of_subs_made)``. + .. versionchanged:: 2.7 + Added the optional flags argument. + .. function:: escape(string) diff --git a/Doc/library/readline.rst b/Doc/library/readline.rst index c58fd26..64c61a2 100644 --- a/Doc/library/readline.rst +++ b/Doc/library/readline.rst @@ -20,7 +20,7 @@ interactive prompt and the prompts offered by the :func:`raw_input` and the ``libedit`` library instead of GNU readline. The configuration file for ``libedit`` is different from that - of GNU readline. If you programmaticly load configuration strings + of GNU readline. If you programmatically load configuration strings you can check for the text "libedit" in :const:`readline.__doc__` to differentiate between GNU readline and libedit. @@ -210,7 +210,8 @@ normally be executed automatically during interactive sessions from the user's :envvar:`PYTHONSTARTUP` file. :: import os - histfile = os.path.join(os.environ["HOME"], ".pyhist") + import readline + histfile = os.path.join(os.path.expanduser("~"), ".pyhist") try: readline.read_history_file(histfile) except IOError: diff --git a/Doc/library/repr.rst b/Doc/library/repr.rst index 78284f0..5bf5bf2 100644 --- a/Doc/library/repr.rst +++ b/Doc/library/repr.rst @@ -15,6 +15,11 @@ The :mod:`repr` module provides a means for producing object representations with limits on the size of the resulting strings. This is used in the Python debugger and may be useful in other contexts as well. +.. seealso:: + + Latest version of the `repr module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/repr.py?view=markup>`_ + This module provides a class, an instance, and a function: diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index a0f3035..905c64b 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -9,61 +9,129 @@ .. versionadded:: 2.5 The :mod:`runpy` module is used to locate and run Python modules without -importing them first. Its main use is to implement the :option:`-m` command line -switch that allows scripts to be located using the Python module namespace -rather than the filesystem. +importing them first. Its main use is to implement the :option:`-m` command +line switch that allows scripts to be located using the Python module +namespace rather than the filesystem. -When executed as a script, the module effectively operates as follows:: +The :mod:`runpy` module provides two functions: - del sys.argv[0] # Remove the runpy module from the arguments - run_module(sys.argv[0], run_name="__main__", alter_sys=True) -The :mod:`runpy` module provides a single function: +.. function:: run_module(mod_name, init_globals=None, run_name=None, alter_sys=False) + Execute the code of the specified module and return the resulting module + globals dictionary. The module's code is first located using the standard + import mechanism (refer to :pep:`302` for details) and then executed in a + fresh module namespace. -.. function:: run_module(mod_name[, init_globals] [, run_name][, alter_sys]) + If the supplied module name refers to a package rather than a normal + module, then that package is imported and the ``__main__`` submodule within + that package is then executed and the resulting module globals dictionary + returned. - Execute the code of the specified module and return the resulting module globals - dictionary. The module's code is first located using the standard import - mechanism (refer to :pep:`302` for details) and then executed in a fresh module - namespace. + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_module`. - The optional dictionary argument *init_globals* may be used to pre-populate the - globals dictionary before the code is executed. The supplied dictionary will not - be modified. If any of the special global variables below are defined in the - supplied dictionary, those definitions are overridden by the ``run_module`` - function. + The special global variables ``__name__``, ``__file__``, ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). - The special global variables ``__name__``, ``__file__``, ``__loader__`` and - ``__builtins__`` are set in the globals dictionary before the module code is - executed. + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None`, to ``mod_name + '.__main__'`` if the named module is a + package and to the *mod_name* argument otherwise. - ``__name__`` is set to *run_name* if this optional argument is supplied, and the - *mod_name* argument otherwise. + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. - ``__loader__`` is set to the :pep:`302` module loader used to retrieve the code for - the module (This loader may be a wrapper around the standard import mechanism). + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). - ``__file__`` is set to the name provided by the module loader. If the loader - does not make filename information available, this variable is set to ``None``. + ``__package__`` is set to *mod_name* if the named module is a package and + to ``mod_name.rpartition('.')[0]`` otherwise. - ``__builtins__`` is automatically initialised with a reference to the top level - namespace of the :mod:`__builtin__` module. - - If the argument *alter_sys* is supplied and evaluates to ``True``, then - ``sys.argv[0]`` is updated with the value of ``__file__`` and + If the argument *alter_sys* is supplied and evaluates to :const:`True`, + then ``sys.argv[0]`` is updated with the value of ``__file__`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. Both ``sys.argv[0]`` and ``sys.modules[__name__]`` are restored to their original values before the function returns. - Note that this manipulation of :mod:`sys` is not thread-safe. Other threads may - see the partially initialised module, as well as the altered list of arguments. - It is recommended that the :mod:`sys` module be left alone when invoking this - function from threaded code. + Note that this manipulation of :mod:`sys` is not thread-safe. Other threads + may see the partially initialised module, as well as the altered list of + arguments. It is recommended that the :mod:`sys` module be left alone when + invoking this function from threaded code. + + + .. versionchanged:: 2.7 + Added ability to execute packages by looking for a ``__main__`` + submodule + + +.. function:: run_path(file_path, init_globals=None, run_name=None) + + Execute the code at the named filesystem location and return the resulting + module globals dictionary. As with a script name supplied to the CPython + command line, the supplied path may refer to a Python source file, a + compiled bytecode file or a valid sys.path entry containing a ``__main__`` + module (e.g. a zipfile containing a top-level ``__main__.py`` file). + + For a simple script, the specified code is simply executed in a fresh + module namespace. For a valid sys.path entry (typically a zipfile or + directory), the entry is first added to the beginning of ``sys.path``. The + function then looks for and executes a :mod:`__main__` module using the + updated path. Note that there is no special protection against invoking + an existing :mod:`__main__` entry located elsewhere on ``sys.path`` if + there is no such module at the specified location. + + The optional dictionary argument *init_globals* may be used to pre-populate + the module's globals dictionary before the code is executed. The supplied + dictionary will not be modified. If any of the special global variables + below are defined in the supplied dictionary, those definitions are + overridden by :func:`run_path`. + The special global variables ``__name__``, ``__file__``, ``__loader__`` + and ``__package__`` are set in the globals dictionary before the module + code is executed (Note that this is a minimal set of variables - other + variables may be set implicitly as an interpreter implementation detail). + + ``__name__`` is set to *run_name* if this optional argument is not + :const:`None` and to ``'<run_path>'`` otherwise. + + ``__file__`` is set to the name provided by the module loader. If the + loader does not make filename information available, this variable is set + to :const:`None`. For a simple script, this will be set to ``file_path``. + + ``__loader__`` is set to the :pep:`302` module loader used to retrieve the + code for the module (This loader may be a wrapper around the standard + import mechanism). For a simple script, this will be set to :const:`None`. + + ``__package__`` is set to ``__name__.rpartition('.')[0]``. + + A number of alterations are also made to the :mod:`sys` module. Firstly, + ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated + with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with a temporary module object for the module being executed. All + modifications to items in :mod:`sys` are reverted before the function + returns. + + Note that, unlike :func:`run_module`, the alterations made to :mod:`sys` + are not optional in this function as these adjustments are essential to + allowing the execution of sys.path entries. As the thread-safety + limitations still apply, use of this function in threaded code should be + either serialised with the import lock or delegated to a separate process. + + .. versionadded:: 2.7 .. seealso:: :pep:`338` - Executing modules as scripts - PEP written and implemented by Nick Coghlan. + PEP written and implemented by Nick Coghlan. + + :pep:`366` - Main module explicit relative imports + PEP written and implemented by Nick Coghlan. + :ref:`using-on-general` - CPython command line details diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst index 51b9d3f..1efe47a 100644 --- a/Doc/library/sched.rst +++ b/Doc/library/sched.rst @@ -10,6 +10,10 @@ The :mod:`sched` module defines a class which implements a general purpose event scheduler: +.. seealso:: + + Latest version of the `sched module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/sched.py?view=markup>`_ .. class:: scheduler(timefunc, delayfunc) diff --git a/Doc/library/select.rst b/Doc/library/select.rst index 034756e..2e08b74 100644 --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -100,6 +100,15 @@ The module defines the following: library, and does not handle file descriptors that don't originate from WinSock. +.. attribute:: select.PIPE_BUF + + Files reported as ready for writing by :func:`select`, :func:`poll` or + similar interfaces in this module are guaranteed to not block on a write + of up to :const:`PIPE_BUF` bytes. + This value is guaranteed by POSIX to be at least 512. Availability: Unix. + + .. versionadded:: 2.7 + .. _epoll-objects: @@ -129,15 +138,15 @@ Edge and Level Trigger Polling (epoll) Objects | :const:`EPOLLONESHOT` | Set one-shot behavior. After one event is | | | pulled out, the fd is internally disabled | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDNORM` | ??? | + | :const:`EPOLLRDNORM` | Equivalent to :const:`EPOLLIN` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLRDBAND` | ??? | + | :const:`EPOLLRDBAND` | Priority data band can be read. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRNORM` | ??? | + | :const:`EPOLLWRNORM` | Equivalent to :const:`EPOLLOUT` | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLWRBAND` | ??? | + | :const:`EPOLLWRBAND` | Priority data may be written. | +-----------------------+-----------------------------------------------+ - | :const:`EPOLLMSG` | ??? | + | :const:`EPOLLMSG` | Ignored. | +-----------------------+-----------------------------------------------+ @@ -231,7 +240,7 @@ linearly scanned again. :cfunc:`select` is O(highest file descriptor), while .. method:: poll.modify(fd, eventmask) Modifies an already registered fd. This has the same effect as - :meth:`register(fd, eventmask)`. Attempting to modify a file descriptor + ``register(fd, eventmask)``. Attempting to modify a file descriptor that was never registered causes an :exc:`IOError` exception with errno :const:`ENOENT` to be raised. diff --git a/Doc/library/sets.rst b/Doc/library/sets.rst index 47b2d76..7bd2931 100644 --- a/Doc/library/sets.rst +++ b/Doc/library/sets.rst @@ -14,7 +14,7 @@ .. versionadded:: 2.3 .. deprecated:: 2.6 - The built-in ``set``/``frozenset`` types replace this module. + The built-in :class:`set`/:class:`frozenset` types replace this module. The :mod:`sets` module provides classes for constructing and manipulating unordered collections of unique elements. Common uses include membership diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst index c0bcb80..4f2d94b 100644 --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -13,6 +13,10 @@ arbitrary Python objects --- anything that the :mod:`pickle` module can handle. This includes most class instances, recursive data types, and objects containing lots of shared sub-objects. The keys are ordinary strings. +.. seealso:: + + Latest version of the `shelve module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/shelve.py?view=markup>`_ .. function:: open(filename[, flag='c'[, protocol=None[, writeback=False]]]) @@ -46,6 +50,11 @@ lots of shared sub-objects. The keys are ordinary strings. :meth:`close` explicitly when you don't need it any more, or use a :keyword:`with` statement with :func:`contextlib.closing`. +.. warning:: + + Because the :mod:`shelve` module is backed by :mod:`pickle`, it is insecure + to load a shelf from an untrusted source. Like with pickle, loading a shelf + can execute arbitrary code. Shelf objects support all methods supported by dictionaries. This eases the transition from dictionary based scripts to those requiring persistent storage. diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index ad3ab57..1b160d8 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -16,6 +16,11 @@ collections of files. In particular, functions are provided which support file copying and removal. For operations on individual files, see also the :mod:`os` module. +.. seealso:: + + Latest version of the `shutil module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/shutil.py?view=markup>`_ + .. warning:: Even the higher-level file copying functions (:func:`copy`, :func:`copy2`) @@ -28,6 +33,9 @@ copying and removal. For operations on individual files, see also the are not copied. +Directory and files operations +------------------------------ + .. function:: copyfileobj(fsrc, fdst[, length]) Copy the contents of the file-like object *fsrc* to the file-like object *fdst*. @@ -169,11 +177,10 @@ copying and removal. For operations on individual files, see also the .. versionadded:: 2.3 - .. _shutil-example: -Example -------- +copytree example +:::::::::::::::: This example is the implementation of the :func:`copytree` function, described above, with the docstring omitted. It demonstrates many of the other functions @@ -238,3 +245,96 @@ Another example that uses the *ignore* argument to add a logging call:: copytree(source, destination, ignore=_logpath) + +Archives operations +------------------- + +.. function:: make_archive(base_name, format, [root_dir, [base_dir, [verbose, [dry_run, [owner, [group, [logger]]]]]]]) + + Create an archive file (eg. zip or tar) and returns its name. + + *base_name* is the name of the file to create, including the path, minus + any format-specific extension. *format* is the archive format: one of + "zip", "tar", "bztar" or "gztar". + + *root_dir* is a directory that will be the root directory of the + archive; ie. we typically chdir into *root_dir* before creating the + archive. + + *base_dir* is the directory where we start archiving from; + ie. *base_dir* will be the common prefix of all files and + directories in the archive. + + *root_dir* and *base_dir* both default to the current directory. + + *owner* and *group* are used when creating a tar archive. By default, + uses the current owner and group. + + .. versionadded:: 2.7 + + +.. function:: get_archive_formats() + + Returns a list of supported formats for archiving. + Each element of the returned sequence is a tuple ``(name, description)`` + + By default :mod:`shutil` provides these formats: + + - *gztar*: gzip'ed tar-file + - *bztar*: bzip2'ed tar-file + - *tar*: uncompressed tar file + - *zip*: ZIP file + + You can register new formats or provide your own archiver for any existing + formats, by using :func:`register_archive_format`. + + .. versionadded:: 2.7 + + +.. function:: register_archive_format(name, function, [extra_args, [description]]) + + Registers an archiver for the format *name*. *function* is a callable that + will be used to invoke the archiver. + + If given, *extra_args* is a sequence of ``(name, value)`` that will be + used as extra keywords arguments when the archiver callable is used. + + *description* is used by :func:`get_archive_formats` which returns the + list of archivers. Defaults to an empty list. + + .. versionadded:: 2.7 + + +.. function:: unregister_archive_format(name) + + Remove the archive format *name* from the list of supported formats. + + .. versionadded:: 2.7 + + +Archiving example +::::::::::::::::: + +In this example, we create a gzip'ed tar-file archive containing all files +found in the :file:`.ssh` directory of the user:: + + >>> from shutil import make_archive + >>> import os + >>> archive_name = os.path.expanduser(os.path.join('~', 'myarchive')) + >>> root_dir = os.path.expanduser(os.path.join('~', '.ssh')) + >>> make_archive(archive_name, 'gztar', root_dir) + '/Users/tarek/myarchive.tar.gz' + +The resulting archive contains:: + + $ tar -tzvf /Users/tarek/myarchive.tar.gz + drwx------ tarek/staff 0 2010-02-01 16:23:40 ./ + -rw-r--r-- tarek/staff 609 2008-06-09 13:26:54 ./authorized_keys + -rwxr-xr-x tarek/staff 65 2008-06-09 13:26:54 ./config + -rwx------ tarek/staff 668 2008-06-09 13:26:54 ./id_dsa + -rwxr-xr-x tarek/staff 609 2008-06-09 13:26:54 ./id_dsa.pub + -rw------- tarek/staff 1675 2008-06-09 13:26:54 ./id_rsa + -rw-r--r-- tarek/staff 397 2008-06-09 13:26:54 ./id_rsa.pub + -rw-r--r-- tarek/staff 37192 2010-02-06 18:23:10 ./known_hosts + + diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 84f08b3..74ff78d 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -75,6 +75,26 @@ The variables defined in the :mod:`signal` module are: the system are defined by this module. +.. data:: CTRL_C_EVENT + + The signal corresponding to the CTRL+C keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 2.7 + + +.. data:: CTRL_BREAK_EVENT + + The signal corresponding to the CTRL+BREAK keystroke event. This signal can + only be used with :func:`os.kill`. + + Availability: Windows. + + .. versionadded:: 2.7 + + .. data:: NSIG One more than the number of the highest signal number. @@ -216,6 +236,10 @@ The :mod:`signal` module defines the following functions: see the :ref:`description in the type hierarchy <frame-objects>` or see the attribute descriptions in the :mod:`inspect` module). + On Windows, :func:`signal` can only be called with :const:`SIGABRT`, + :const:`SIGFPE`, :const:`SIGILL`, :const:`SIGINT`, :const:`SIGSEGV`, or + :const:`SIGTERM`. A :exc:`ValueError` will be raised in any other case. + .. _signal-example: diff --git a/Doc/library/simplehttpserver.rst b/Doc/library/simplehttpserver.rst index ecb9340..a92c7c9 100644 --- a/Doc/library/simplehttpserver.rst +++ b/Doc/library/simplehttpserver.rst @@ -82,34 +82,35 @@ The :mod:`SimpleHTTPServer` module defines the following class: ``text/`` the file is opened in text mode; otherwise binary mode is used. The :func:`test` function in the :mod:`SimpleHTTPServer` module is an - example which creates a server using the - :class:`SimpleHTTPRequestHandler` as the Handler. + example which creates a server using the :class:`SimpleHTTPRequestHandler` + as the Handler. .. versionadded:: 2.5 The ``'Last-Modified'`` header. -The :mod:`SimpleHTTPServer` module can be used the following manner in order to -set up a very basic web server serving files relative to the current -directory.:: +The :mod:`SimpleHTTPServer` module can be used in the following manner in order +to set up a very basic web server serving files relative to the current +directory. :: - import SimpleHTTPServer - import SocketServer + import SimpleHTTPServer + import SocketServer - PORT = 8000 + PORT = 8000 - Handler = SimpleHTTPServer.SimpleHTTPRequestHandler + Handler = SimpleHTTPServer.SimpleHTTPRequestHandler - httpd = SocketServer.TCPServer(("", PORT), Handler) + httpd = SocketServer.TCPServer(("", PORT), Handler) - print "serving at port", PORT - httpd.serve_forever() + print "serving at port", PORT + httpd.serve_forever() -:mod:`SimpleHTTPServer` module can also be invoked directly using the ``-m`` -switch of interpreter with a ``port number`` argument. Similar to previous -example, this serves the files relative to the current directory.:: +The :mod:`SimpleHTTPServer` module can also be invoked directly using the +:option:`-m` switch of the interpreter with a ``port number`` argument. +Similar to the previous example, this serves the files relative to the +current directory. :: - python -m SimpleHTTPServer 8000 + python -m SimpleHTTPServer 8000 .. seealso:: diff --git a/Doc/library/simplexmlrpcserver.rst b/Doc/library/simplexmlrpcserver.rst index 7cbb5f0..c0819bf 100644 --- a/Doc/library/simplexmlrpcserver.rst +++ b/Doc/library/simplexmlrpcserver.rst @@ -133,6 +133,15 @@ alone XML-RPC servers. .. versionadded:: 2.5 +.. attribute:: SimpleXMLRPCRequestHandler.encode_threshold + + If this attribute is not ``None``, responses larger than this value + will be encoded using the *gzip* transfer encoding, if permitted by + the client. The default is ``1400`` which corresponds roughly + to a single TCP packet. + + .. versionadded:: 2.7 + .. _simplexmlrpcserver-example: SimpleXMLRPCServer Example diff --git a/Doc/library/site.rst b/Doc/library/site.rst index 692d077..6ad156e 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -131,6 +131,32 @@ empty, and the path manipulations are skipped; however the import of Adds a directory to sys.path and processes its pth files. +.. function:: getsitepackages() + + Returns a list containing all global site-packages directories + (and possibly site-python). + + .. versionadded:: 2.7 + +.. function:: getuserbase() + + Returns the "user base" directory path. + + The "user base" directory can be used to store data. If the global + variable ``USER_BASE`` is not initialized yet, this function will also set + it. + + .. versionadded:: 2.7 + +.. function:: getusersitepackages() + + Returns the user-specific site-packages directory path. + + If the global variable ``USER_SITE`` is not initialized yet, this + function will also set it. + + .. versionadded:: 2.7 .. XXX Update documentation .. XXX document python -m site --user-base --user-site + diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index c8f4ed1..d59ebf7 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -48,8 +48,7 @@ Protocol) and :rfc:`1869` (SMTP Service Extensions). connection attempt (if not specified, the global default timeout setting will be used). - .. versionchanged:: 2.6 - *timeout* was added. + .. versionadded:: 2.6 .. class:: LMTP([host[, port[, local_hostname]]]) @@ -298,9 +297,9 @@ An :class:`SMTP` instance has the following methods: and ESMTP options suppressed. This method will return normally if the mail is accepted for at least one - recipient. Otherwise it will throw an exception. That is, if this method does - not throw an exception, then someone should get your mail. If this method does - not throw an exception, it returns a dictionary, with one entry for each + recipient. Otherwise it will raise an exception. That is, if this method does + not raise an exception, then someone should get your mail. If this method does + not raise an exception, it returns a dictionary, with one entry for each recipient that was refused. Each entry contains a tuple of the SMTP error code and the accompanying error message sent by the server. diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 7fe07be..96152f9 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -205,7 +205,7 @@ The module :mod:`socket` exports the following constants and functions: .. versionadded:: 2.3 -.. function:: create_connection(address[, timeout]) +.. function:: create_connection(address[, timeout[, source_address]]) Convenience function. Connect to *address* (a 2-tuple ``(host, port)``), and return the socket object. Passing the optional *timeout* parameter will @@ -213,8 +213,15 @@ The module :mod:`socket` exports the following constants and functions: *timeout* is supplied, the global default timeout setting returned by :func:`getdefaulttimeout` is used. + If supplied, *source_address* must be a 2-tuple ``(host, port)`` for the + socket to bind to as its source address before connecting. If host or port + are '' or 0 respectively the OS default behavior will be used. + .. versionadded:: 2.6 + .. versionchanged:: 2.7 + *source_address* was added. + .. function:: getaddrinfo(host, port, family=0, socktype=0, proto=0, flags=0) @@ -508,6 +515,9 @@ The module :mod:`socket` exports the following constants and functions: Module :mod:`SocketServer` Classes that simplify writing network servers. + Module :mod:`ssl` + A TLS/SSL wrapper for socket objects. + .. _socket-objects: @@ -544,6 +554,12 @@ correspond to Unix system calls applicable to sockets. remote end will receive no more data (after queued data is flushed). Sockets are automatically closed when they are garbage-collected. + .. note:: + :meth:`close()` releases the resource associated with a connection but + does not necessarily close the connection immediately. If you want + to close the connection in a timely fashion, call :meth:`shutdown()` + before :meth:`close()`. + .. method:: socket.connect(address) @@ -644,6 +660,12 @@ correspond to Unix system calls applicable to sockets. *mode* and *bufsize* arguments are interpreted the same way as by the built-in :func:`file` function. + .. note:: + + On Windows, the file-like object created by :meth:`makefile` cannot be + used where a file object with a file descriptor is expected, such as the + stream arguments of :meth:`subprocess.Popen`. + .. method:: socket.recv(bufsize[, flags]) @@ -733,7 +755,7 @@ correspond to Unix system calls applicable to sockets. Set a timeout on blocking socket operations. The *value* argument can be a nonnegative float expressing seconds, or ``None``. If a float is given, - subsequent socket operations will raise an :exc:`timeout` exception if the + subsequent socket operations will raise a :exc:`timeout` exception if the timeout period *value* has elapsed before the operation has completed. Setting a timeout of ``None`` disables timeouts on socket operations. ``s.settimeout(0.0)`` is equivalent to ``s.setblocking(0)``; @@ -791,7 +813,9 @@ timeout error of its own regardless of any Python socket timeout setting. Shut down one or both halves of the connection. If *how* is :const:`SHUT_RD`, further receives are disallowed. If *how* is :const:`SHUT_WR`, further sends are disallowed. If *how* is :const:`SHUT_RDWR`, further sends and receives are - disallowed. + disallowed. Depending on the platform, shutting down one half of the connection + can also close the opposite half (e.g. on Mac OS X, ``shutdown(SHUT_WR)`` does + not allow further reads on the other end of the connection). Note that there are no methods :meth:`read` or :meth:`write`; use :meth:`~socket.recv` and :meth:`~socket.send` without *flags* argument instead. diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst index 5f68d33..fce216e 100644 --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -223,7 +223,6 @@ The server classes support the following class variables: desired. If :meth:`handle_request` receives no incoming requests within the timeout period, the :meth:`handle_timeout` method is called. - There are various server methods that can be overridden by subclasses of base server classes like :class:`TCPServer`; these methods aren't useful to external users of the server object. diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index dc5f239..59ed097 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -83,7 +83,7 @@ This example uses the iterator form:: >>> for row in c: ... print row ... - (u'2006-01-05', u'BUY', u'RHAT', 100, 35.140000000000001) + (u'2006-01-05', u'BUY', u'RHAT', 100, 35.14) (u'2006-03-28', u'BUY', u'IBM', 1000, 45.0) (u'2006-04-06', u'SELL', u'IBM', 500, 53.0) (u'2006-04-05', u'BUY', u'MSOFT', 1000, 72.0) @@ -138,7 +138,7 @@ Module functions and constants first blank for the column name: the column name would simply be "x". -.. function:: connect(database[, timeout, isolation_level, detect_types, factory]) +.. function:: connect(database[, timeout, detect_types, isolation_level, check_same_thread, factory, cached_statements]) Opens a connection to the SQLite database file *database*. You can use ``":memory:"`` to open a database connection to a database that resides in RAM @@ -258,22 +258,21 @@ Connection Objects .. method:: Connection.execute(sql, [parameters]) This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's - :meth:`execute<Cursor.execute>` method with the parameters given. + calling the cursor method, then calls the cursor's :meth:`execute + <Cursor.execute>` method with the parameters given. .. method:: Connection.executemany(sql, [parameters]) This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's - :meth:`executemany<Cursor.executemany>` method with the parameters given. + calling the cursor method, then calls the cursor's :meth:`executemany + <Cursor.executemany>` method with the parameters given. .. method:: Connection.executescript(sql_script) This is a nonstandard shortcut that creates an intermediate cursor object by - calling the cursor method, then calls the cursor's - :meth:`executescript<Cursor.executescript>` method with the parameters - given. + calling the cursor method, then calls the cursor's :meth:`executescript + <Cursor.executescript>` method with the parameters given. .. method:: Connection.create_function(name, num_params, func) @@ -368,6 +367,25 @@ Connection Objects method with :const:`None` for *handler*. +.. method:: Connection.enable_load_extension(enabled) + + .. versionadded:: 2.7 + + This routine allows/disallows the SQLite engine to load SQLite extensions + from shared libraries. SQLite extensions can define new functions, + aggregates or whole new virtual table implementations. One well-known + extension is the fulltext-search extension distributed with SQLite. + + .. literalinclude:: ../includes/sqlite3/load_extension.py + +.. method:: Connection.load_extension(path) + + .. versionadded:: 2.7 + + This routine loads a SQLite extension from a shared library. You have to + enable extension loading with :meth:`enable_load_extension` before you can + use this routine. + .. attribute:: Connection.row_factory You can change this attribute to a callable that accepts the cursor and the @@ -441,7 +459,7 @@ Cursor Objects .. class:: Cursor - A SQLite database cursor has the following attributes and methods: + A :class:`Cursor` instance has the following attributes and methods. .. method:: Cursor.execute(sql, [parameters]) @@ -602,7 +620,7 @@ Now we plug :class:`Row` in:: >>> type(r) <type 'sqlite3.Row'> >>> r - (u'2006-01-05', u'BUY', u'RHAT', 100.0, 35.140000000000001) + (u'2006-01-05', u'BUY', u'RHAT', 100.0, 35.14) >>> len(r) 5 >>> r[2] @@ -861,3 +879,17 @@ exception, the transaction is rolled back; otherwise, the transaction is committed: .. literalinclude:: ../includes/sqlite3/ctx_manager.py + + +Common issues +------------- + +Multithreading +^^^^^^^^^^^^^^ + +Older SQLite versions had issues with sharing connections between threads. +That's why the Python module disallows sharing connections and cursors between +threads. If you still try to do so, you will get an exception at runtime. + +The only exception is calling the :meth:`~Connection.interrupt` method, which +only makes sense to call from a different thread. diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 5c76df9..a7e78bd 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1,8 +1,8 @@ -:mod:`ssl` --- SSL wrapper for socket objects -============================================= +:mod:`ssl` --- TLS/SSL wrapper for socket objects +================================================= .. module:: ssl - :synopsis: SSL wrapper for socket objects + :synopsis: TLS/SSL wrapper for socket objects .. moduleauthor:: Bill Janssen <bill.janssen@gmail.com> @@ -50,7 +50,7 @@ Functions, Constants, and Exceptions is a subtype of :exc:`socket.error`, which in turn is a subtype of :exc:`IOError`. -.. function:: wrap_socket (sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True) +.. function:: wrap_socket (sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version={see docs}, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None) Takes an instance ``sock`` of :class:`socket.socket`, and returns an instance of :class:`ssl.SSLSocket`, a subtype of :class:`socket.socket`, which wraps @@ -113,14 +113,26 @@ Functions, Constants, and Exceptions ======================== ========= ========= ========== ========= *client* / **server** **SSLv2** **SSLv3** **SSLv23** **TLSv1** ------------------------ --------- --------- ---------- --------- - *SSLv2* yes no yes* no + *SSLv2* yes no yes no *SSLv3* yes yes yes no *SSLv23* yes no yes no *TLSv1* no no yes yes ======================== ========= ========= ========== ========= - In some older versions of OpenSSL (for instance, 0.9.7l on OS X 10.4), an - SSLv2 client could not connect to an SSLv23 server. + .. note:: + + Which connections succeed will vary depending on the version of + OpenSSL. For instance, in some older versions of OpenSSL (such + as 0.9.7l on OS X 10.4), an SSLv2 client could not connect to an + SSLv23 server. Another example: beginning with OpenSSL 1.0.0, + an SSLv23 client will not actually attempt SSLv2 connections + unless you explicitly enable SSLv2 ciphers; for example, you + might specify ``"ALL"`` or ``"SSLv2"`` as the *ciphers* parameter + to enable them. + + The *ciphers* parameter sets the available ciphers for this SSL object. + It should be a string in the `OpenSSL cipher list format + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`_. The parameter ``do_handshake_on_connect`` specifies whether to do the SSL handshake automatically after doing a :meth:`socket.connect`, or whether the @@ -135,6 +147,9 @@ Functions, Constants, and Exceptions normal EOF in response to unexpected EOF errors raised from the underlying socket; if :const:`False`, it will raise the exceptions back to the caller. + .. versionchanged:: 2.7 + New optional argument *ciphers*. + .. function:: RAND_status() Returns True if the SSL pseudo-random number generator has been seeded with @@ -244,6 +259,36 @@ Functions, Constants, and Exceptions modern version, and probably the best choice for maximum protection, if both sides can speak it. +.. data:: OPENSSL_VERSION + + The version string of the OpenSSL library loaded by the interpreter:: + + >>> ssl.OPENSSL_VERSION + 'OpenSSL 0.9.8k 25 Mar 2009' + + .. versionadded:: 2.7 + +.. data:: OPENSSL_VERSION_INFO + + A tuple of five integers representing version information about the + OpenSSL library:: + + >>> ssl.OPENSSL_VERSION_INFO + (0, 9, 8, 11, 15) + + .. versionadded:: 2.7 + +.. data:: OPENSSL_VERSION_NUMBER + + The raw version number of the OpenSSL library, as a single integer:: + + >>> ssl.OPENSSL_VERSION_NUMBER + 9470143L + >>> hex(ssl.OPENSSL_VERSION_NUMBER) + '0x9080bfL' + + .. versionadded:: 2.7 + SSLSocket Objects ----------------- @@ -460,11 +505,11 @@ To test for the presence of SSL support in a Python installation, user code should use the following idiom:: try: - import ssl + import ssl except ImportError: - pass + pass else: - [ do something that requires SSL support ] + ... # do something that requires SSL support Client-side operation ^^^^^^^^^^^^^^^^^^^^^ @@ -537,29 +582,31 @@ the other end, and use :func:`wrap_socket` to create a server-side SSL context for it:: while True: - newsocket, fromaddr = bindsocket.accept() - connstream = ssl.wrap_socket(newsocket, - server_side=True, - certfile="mycertfile", - keyfile="mykeyfile", - ssl_version=ssl.PROTOCOL_TLSv1) - deal_with_client(connstream) + newsocket, fromaddr = bindsocket.accept() + connstream = ssl.wrap_socket(newsocket, + server_side=True, + certfile="mycertfile", + keyfile="mykeyfile", + ssl_version=ssl.PROTOCOL_TLSv1) + try: + deal_with_client(connstream) + finally: + connstream.shutdown(socket.SHUT_RDWR) + connstream.close() Then you'd read data from the ``connstream`` and do something with it till you are finished with the client (or the client is finished with you):: def deal_with_client(connstream): - - data = connstream.read() - # null data means the client is finished with us - while data: - if not do_something(connstream, data): - # we'll assume do_something returns False - # when we're finished with client - break - data = connstream.read() - # finished with client - connstream.close() + data = connstream.read() + # null data means the client is finished with us + while data: + if not do_something(connstream, data): + # we'll assume do_something returns False + # when we're finished with client + break + data = connstream.read() + # finished with client And go back to listening for new client connections. diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index a48a02d..417ba8d 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -73,6 +73,34 @@ for each test. These are also useful when checking for information about a file that isn't handled by :mod:`os.path`, like the tests for block and character devices. +Example:: + + import os, sys + from stat import * + + def walktree(top, callback): + '''recursively descend the directory tree rooted at top, + calling the callback function for each regular file''' + + for f in os.listdir(top): + pathname = os.path.join(top, f) + mode = os.stat(pathname)[ST_MODE] + if S_ISDIR(mode): + # It's a directory, recurse into it + walktree(pathname, callback) + elif S_ISREG(mode): + # It's a file, call the callback function + callback(pathname) + else: + # Unknown file type, print a message + print 'Skipping %s' % pathname + + def visitfile(file): + print 'visiting', file + + if __name__ == '__main__': + walktree(sys.argv[1], visitfile) + All the variables below are simply symbolic indexes into the 10-tuple returned by :func:`os.stat`, :func:`os.fstat` or :func:`os.lstat`. @@ -262,31 +290,47 @@ The following flags can also be used in the *mode* argument of :func:`os.chmod`: Unix V7 synonym for :data:`S_IXUSR`. -Example:: +The following flags can be used in the *flags* argument of :func:`os.chflags`: - import os, sys - from stat import * +.. data:: UF_NODUMP - def walktree(top, callback): - '''recursively descend the directory tree rooted at top, - calling the callback function for each regular file''' + Do not dump the file. - for f in os.listdir(top): - pathname = os.path.join(top, f) - mode = os.stat(pathname)[ST_MODE] - if S_ISDIR(mode): - # It's a directory, recurse into it - walktree(pathname, callback) - elif S_ISREG(mode): - # It's a file, call the callback function - callback(pathname) - else: - # Unknown file type, print a message - print 'Skipping %s' % pathname +.. data:: UF_IMMUTABLE - def visitfile(file): - print 'visiting', file + The file may not be changed. - if __name__ == '__main__': - walktree(sys.argv[1], visitfile) +.. data:: UF_APPEND + + The file may only be appended to. + +.. data:: UF_OPAQUE + + The file may not be renamed or deleted. + +.. data:: UF_NOUNLINK + + The directory is opaque when viewed through a union stack. + +.. data:: SF_ARCHIVED + + The file may be archived. + +.. data:: SF_IMMUTABLE + + The file may not be changed. + +.. data:: SF_APPEND + + The file may only be appended to. + +.. data:: SF_NOUNLINK + + The file may not be renamed or deleted. + +.. data:: SF_SNAPSHOT + + The file is a snapshot file. + +See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 6999a5b..da5db1d 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -224,18 +224,20 @@ Numeric Types --- :class:`int`, :class:`float`, :class:`long`, :class:`complex` pair: C; language There are four distinct numeric types: :dfn:`plain integers`, :dfn:`long -integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In +integers`, :dfn:`floating point numbers`, and :dfn:`complex numbers`. In addition, Booleans are a subtype of plain integers. Plain integers (also just called :dfn:`integers`) are implemented using :ctype:`long` in C, which gives them at least 32 bits of precision (``sys.maxint`` is always set to the maximum plain integer value for the current platform, the minimum value is -``-sys.maxint - 1``). Long integers have unlimited precision. Floating point -numbers are implemented using :ctype:`double` in C. All bets on their precision -are off unless you happen to know the machine you are working with. - -Complex numbers have a real and imaginary part, which are each implemented using -:ctype:`double` in C. To extract these parts from a complex number *z*, use -``z.real`` and ``z.imag``. +``-sys.maxint - 1``). Long integers have unlimited precision. Floating point +numbers are usually implemented using :ctype:`double` in C; information about +the precision and internal representation of floating point numbers for the +machine on which your program is running is available in +:data:`sys.float_info`. Complex numbers have a real and imaginary part, which +are each a floating point number. To extract these parts from a complex number +*z*, use ``z.real`` and ``z.imag``. (The standard library includes additional +numeric types, :mod:`fractions` that hold rationals, and :mod:`decimal` that +hold floating-point numbers with user-definable precision.) .. index:: pair: numeric; literals @@ -358,10 +360,10 @@ Notes: See :ref:`built-in-funcs` for a full description. (4) - Complex floor division operator, modulo operator, and :func:`divmod`. - .. deprecated:: 2.3 - Instead convert to float using :func:`abs` if appropriate. + The floor division operator, the modulo operator, and the :func:`divmod` + function are no longer defined for complex numbers. Instead, convert to + a floating point number using the :func:`abs` function if appropriate. (5) Also referred to as integer division. The resultant value is a whole integer, @@ -455,6 +457,37 @@ Notes: A right shift by *n* bits is equivalent to division by ``pow(2, n)``. +Additional Methods on Integer Types +----------------------------------- + +.. method:: int.bit_length() +.. method:: long.bit_length() + + Return the number of bits necessary to represent an integer in binary, + excluding the sign and leading zeros:: + + >>> n = -37 + >>> bin(n) + '-0b100101' + >>> n.bit_length() + 6 + + More precisely, if ``x`` is nonzero, then ``x.bit_length()`` is the + unique positive integer ``k`` such that ``2**(k-1) <= abs(x) < 2**k``. + Equivalently, when ``abs(x)`` is small enough to have a correctly + rounded logarithm, then ``k = 1 + int(log(abs(x), 2))``. + If ``x`` is zero, then ``x.bit_length()`` returns ``0``. + + Equivalent to:: + + def bit_length(self): + s = bin(self) # binary representation: bin(-37) --> '-0b100101' + s = s.lstrip('-0b') # remove leading zeros and minus sign + return len(s) # len('100101') --> 6 + + .. versionadded:: 2.7 + + Additional Methods on Float --------------------------- @@ -462,12 +495,24 @@ The float type has some additional methods. .. method:: float.as_integer_ratio() - Return a pair of integers whose ratio is exactly equal to the - original float and with a positive denominator. Raises - :exc:`OverflowError` on infinities and a :exc:`ValueError` on - NaNs. + Return a pair of integers whose ratio is exactly equal to the + original float and with a positive denominator. Raises + :exc:`OverflowError` on infinities and a :exc:`ValueError` on + NaNs. + + .. versionadded:: 2.6 + +.. method:: float.is_integer() + + Return ``True`` if the float instance is finite with integral + value, and ``False`` otherwise:: - .. versionadded:: 2.6 + >>> (-2.0).is_integer() + True + >>> (3.2).is_integer() + False + + .. versionadded:: 2.6 Two methods support conversion to and from hexadecimal strings. Since Python's floats are stored @@ -614,11 +659,11 @@ yield expression <yieldexpr>`. .. _typesseq: -Sequence Types --- :class:`str`, :class:`unicode`, :class:`list`, :class:`tuple`, :class:`buffer`, :class:`xrange` -================================================================================================================== +Sequence Types --- :class:`str`, :class:`unicode`, :class:`list`, :class:`tuple`, :class:`bytearray`, :class:`buffer`, :class:`xrange` +====================================================================================================================================== -There are six sequence types: strings, Unicode strings, lists, tuples, buffers, -and xrange objects. +There are seven sequence types: strings, Unicode strings, lists, tuples, +bytearrays, buffers, and xrange objects. For other containers see the built in :class:`dict` and :class:`set` classes, and the :mod:`collections` module. @@ -630,6 +675,7 @@ and the :mod:`collections` module. object: Unicode object: tuple object: list + object: bytearray object: buffer object: xrange @@ -645,6 +691,8 @@ brackets), with or without enclosing parentheses, but an empty tuple must have the enclosing parentheses, such as ``a, b, c`` or ``()``. A single item tuple must have a trailing comma, such as ``(d,)``. +Bytearray objects are created with the built-in function :func:`bytearray`. + Buffer objects are not directly supported by Python syntax, but can be created by calling the built-in function :func:`buffer`. They don't support concatenation or repetition. @@ -691,6 +739,12 @@ are sequences of the same type; *n*, *i* and *j* are integers: +------------------+--------------------------------+----------+ | ``max(s)`` | largest item of *s* | | +------------------+--------------------------------+----------+ +| ``s.index(i)`` | index of the first occurence | | +| | of *i* in *s* | | ++------------------+--------------------------------+----------+ +| ``s.count(i)`` | total number of occurences of | | +| | *i* in *s* | | ++------------------+--------------------------------+----------+ Sequence types also support comparisons. In particular, tuples and lists are compared lexicographically by comparing corresponding @@ -788,8 +842,9 @@ String Methods .. index:: pair: string; methods -Below are listed the string methods which both 8-bit strings and Unicode objects -support. Note that none of these methods take keyword arguments. +Below are listed the string methods which both 8-bit strings and +Unicode objects support. Some of them are also available on :class:`bytearray` +objects. In addition, Python's strings support the sequence type methods described in the :ref:`typesseq` section. To output formatted strings @@ -835,6 +890,8 @@ string functions based on regular expressions. .. versionchanged:: 2.3 Support for other error handling schemes added. + .. versionchanged:: 2.7 + Support for keyword arguments added. .. method:: str.encode([encoding[,errors]]) @@ -853,6 +910,8 @@ string functions based on regular expressions. Support for ``'xmlcharrefreplace'`` and ``'backslashreplace'`` and other error handling schemes added. + .. versionchanged:: 2.7 + Support for keyword arguments added. .. method:: str.endswith(suffix[, start[, end]]) @@ -1297,8 +1356,8 @@ formats in the string *must* include a parenthesised mapping key into that dictionary inserted immediately after the ``'%'`` character. The mapping key selects the value to be formatted from the mapping. For example: - >>> print '%(language)s has %(#)03d quote types.' % \ - ... {'language': "Python", "#": 2} + >>> print '%(language)s has %(number)03d quote types.' % \ + ... {"language": "Python", "number": 2} Python has 002 quote types. In this case no ``*`` specifiers may occur in a format (since they require a @@ -1419,9 +1478,9 @@ that ``'\0'`` is the end of the string. .. XXX Examples? -For safety reasons, floating point precisions are clipped to 50; ``%f`` -conversions for numbers whose absolute value is over 1e50 are replaced by ``%g`` -conversions. [#]_ All other errors raise exceptions. +.. versionchanged:: 2.7 + ``%f`` conversions for numbers whose absolute value is over 1e50 are no + longer replaced by ``%g`` conversions. .. index:: module: string @@ -1456,11 +1515,29 @@ Mutable Sequence Types triple: mutable; sequence; types object: list -List objects support additional operations that allow in-place modification of -the object. Other mutable sequence types (when added to the language) should -also support these operations. Strings and tuples are immutable sequence types: -such objects cannot be modified once created. The following operations are -defined on mutable sequence types (where *x* is an arbitrary object): +List and :class:`bytearray` objects support additional operations that allow +in-place modification of the object. Other mutable sequence types (when added +to the language) should also support these operations. Strings and tuples +are immutable sequence types: such objects cannot be modified once created. +The following operations are defined on mutable sequence types (where *x* is +an arbitrary object): + +.. index:: + triple: operations on; sequence; types + triple: operations on; list; type + pair: subscript; assignment + pair: slice; assignment + pair: extended slice; assignment + statement: del + single: append() (list method) + single: extend() (list method) + single: count() (list method) + single: index() (list method) + single: insert() (list method) + single: pop() (list method) + single: remove() (list method) + single: reverse() (list method) + single: sort() (list method) +------------------------------+--------------------------------+---------------------+ | Operation | Result | Notes | @@ -1507,23 +1584,6 @@ defined on mutable sequence types (where *x* is an arbitrary object): | reverse]]])`` | | | +------------------------------+--------------------------------+---------------------+ -.. index:: - triple: operations on; sequence; types - triple: operations on; list; type - pair: subscript; assignment - pair: slice; assignment - pair: extended slice; assignment - statement: del - single: append() (list method) - single: extend() (list method) - single: count() (list method) - single: index() (list method) - single: insert() (list method) - single: pop() (list method) - single: remove() (list method) - single: reverse() (list method) - single: sort() (list method) - Notes: (1) @@ -1584,7 +1644,8 @@ Notes: In general, the *key* and *reverse* conversion processes are much faster than specifying an equivalent *cmp* function. This is because *cmp* is called multiple times for each list element while *key* and *reverse* touch each - element only once. + element only once. Use :func:`functools.cmp_to_key` to convert an + old-style *cmp* function to a *key* function. .. versionchanged:: 2.3 Support for ``None`` as an equivalent to omitting *cmp* was added. @@ -1634,9 +1695,13 @@ There are currently two built-in set types, :class:`set` and :class:`frozenset`. The :class:`set` type is mutable --- the contents can be changed using methods like :meth:`add` and :meth:`remove`. Since it is mutable, it has no hash value and cannot be used as either a dictionary key or as an element of another set. -The :class:`frozenset` type is immutable and :term:`hashable` --- its contents cannot be -altered after it is created; it can therefore be used as a dictionary key or as -an element of another set. +The :class:`frozenset` type is immutable and :term:`hashable` --- its contents +cannot be altered after it is created; it can therefore be used as a dictionary +key or as an element of another set. + +As of Python 2.7, non-empty sets (not frozensets) can be created by placing a +comma-separated list of elements within braces, for example: ``{'jack', +'sjoerd'}``, in addition to the :class:`set` constructor. The constructors for both classes work the same: @@ -1879,15 +1944,12 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: values are added as items to the dictionary. If a key is specified both in the positional argument and as a keyword argument, the value associated with the keyword is retained in the dictionary. For example, these all return a - dictionary equal to ``{"one": 2, "two": 3}``: + dictionary equal to ``{"one": 1, "two": 2}``: - * ``dict(one=2, two=3)`` - - * ``dict({'one': 2, 'two': 3})`` - - * ``dict(zip(('one', 'two'), (2, 3)))`` - - * ``dict([['two', 3], ['one', 2]])`` + * ``dict(one=1, two=2)`` + * ``dict({'one': 1, 'two': 2})`` + * ``dict(zip(('one', 'two'), (1, 2)))`` + * ``dict([['two', 2], ['one', 1]])`` The first example only works for keys that are valid Python identifiers; the others work with any valid keys. @@ -2059,7 +2121,7 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: existing keys. Return ``None``. :func:`update` accepts either another dictionary object or an iterable of - key/value pairs (as a tuple or other iterable of length two). If keyword + key/value pairs (as tuples or other iterables of length two). If keyword arguments are specified, the dictionary is then updated with those key/value pairs: ``d.update(red=1, blue=2)``. @@ -2072,6 +2134,121 @@ pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: Return a copy of the dictionary's list of values. See the note for :meth:`dict.items`. + .. method:: viewitems() + + Return a new view of the dictionary's items (``(key, value)`` pairs). See + below for documentation of view objects. + + .. versionadded:: 2.7 + + .. method:: viewkeys() + + Return a new view of the dictionary's keys. See below for documentation of + view objects. + + .. versionadded:: 2.7 + + .. method:: viewvalues() + + Return a new view of the dictionary's values. See below for documentation of + view objects. + + .. versionadded:: 2.7 + + +.. _dict-views: + +Dictionary view objects +----------------------- + +The objects returned by :meth:`dict.viewkeys`, :meth:`dict.viewvalues` and +:meth:`dict.viewitems` are *view objects*. They provide a dynamic view on the +dictionary's entries, which means that when the dictionary changes, the view +reflects these changes. + +Dictionary views can be iterated over to yield their respective data, and +support membership tests: + +.. describe:: len(dictview) + + Return the number of entries in the dictionary. + +.. describe:: iter(dictview) + + Return an iterator over the keys, values or items (represented as tuples of + ``(key, value)``) in the dictionary. + + Keys and values are iterated over in an arbitrary order which is non-random, + varies across Python implementations, and depends on the dictionary's history + of insertions and deletions. If keys, values and items views are iterated + over with no intervening modifications to the dictionary, the order of items + will directly correspond. This allows the creation of ``(value, key)`` pairs + using :func:`zip`: ``pairs = zip(d.values(), d.keys())``. Another way to + create the same list is ``pairs = [(v, k) for (k, v) in d.items()]``. + + Iterating views while adding or deleting entries in the dictionary may raise + a :exc:`RuntimeError` or fail to iterate over all entries. + +.. describe:: x in dictview + + Return ``True`` if *x* is in the underlying dictionary's keys, values or + items (in the latter case, *x* should be a ``(key, value)`` tuple). + + +Keys views are set-like since their entries are unique and hashable. If all +values are hashable, so that (key, value) pairs are unique and hashable, then +the items view is also set-like. (Values views are not treated as set-like +since the entries are generally not unique.) Then these set operations are +available ("other" refers either to another view or a set): + +.. describe:: dictview & other + + Return the intersection of the dictview and the other object as a new set. + +.. describe:: dictview | other + + Return the union of the dictview and the other object as a new set. + +.. describe:: dictview - other + + Return the difference between the dictview and the other object (all elements + in *dictview* that aren't in *other*) as a new set. + +.. describe:: dictview ^ other + + Return the symmetric difference (all elements either in *dictview* or + *other*, but not in both) of the dictview and the other object as a new set. + + +An example of dictionary view usage:: + + >>> dishes = {'eggs': 2, 'sausage': 1, 'bacon': 1, 'spam': 500} + >>> keys = dishes.viewkeys() + >>> values = dishes.viewvalues() + + >>> # iteration + >>> n = 0 + >>> for val in values: + ... n += val + >>> print(n) + 504 + + >>> # keys and values are iterated over in the same order + >>> list(keys) + ['eggs', 'bacon', 'sausage', 'spam'] + >>> list(values) + [2, 1, 1, 500] + + >>> # view objects are dynamic and reflect dict changes + >>> del dishes['eggs'] + >>> del dishes['sausage'] + >>> list(keys) + ['spam', 'bacon'] + + >>> # set operations + >>> keys & {'eggs', 'bacon', 'salad'} + {'bacon'} + .. _bltin-file-objects: @@ -2211,11 +2388,12 @@ Files have the following methods: .. method:: file.readline([size]) - Read one entire line from the file. A trailing newline character is kept in the - string (but may be absent when a file ends with an incomplete line). [#]_ If - the *size* argument is present and non-negative, it is a maximum byte count - (including the trailing newline) and an incomplete line may be returned. An - empty string is returned *only* when EOF is encountered immediately. + Read one entire line from the file. A trailing newline character is kept in + the string (but may be absent when a file ends with an incomplete line). [#]_ + If the *size* argument is present and non-negative, it is a maximum byte + count (including the trailing newline) and an incomplete line may be + returned. When *size* is not 0, an empty string is returned *only* when EOF + is encountered immediately. .. note:: @@ -2357,14 +2535,14 @@ the particular object. .. attribute:: file.newlines - If Python was built with the :option:`--with-universal-newlines` option to - :program:`configure` (the default) this read-only attribute exists, and for - files opened in universal newline read mode it keeps track of the types of - newlines encountered while reading the file. The values it can take are - ``'\r'``, ``'\n'``, ``'\r\n'``, ``None`` (unknown, no newlines read yet) or a - tuple containing all the newline types seen, to indicate that multiple newline - conventions were encountered. For files not opened in universal newline read - mode the value of this attribute will be ``None``. + If Python was built with universal newlines enabled (the default) this + read-only attribute exists, and for files opened in universal newline read + mode it keeps track of the types of newlines encountered while reading the + file. The values it can take are ``'\r'``, ``'\n'``, ``'\r\n'``, ``None`` + (unknown, no newlines read yet) or a tuple containing all the newline types + seen, to indicate that multiple newline conventions were encountered. For + files not opened in universal newline read mode the value of this attribute + will be ``None``. .. attribute:: file.softspace @@ -2384,6 +2562,117 @@ the particular object. state. +.. _typememoryview: + +memoryview type +=============== + +.. versionadded:: 2.7 + +:class:`memoryview` objects allow Python code to access the internal data +of an object that supports the buffer protocol without copying. Memory +is generally interpreted as simple bytes. + +.. class:: memoryview(obj) + + Create a :class:`memoryview` that references *obj*. *obj* must support the + buffer protocol. Built-in objects that support the buffer protocol include + :class:`str` and :class:`bytearray` (but not :class:`unicode`). + + A :class:`memoryview` has the notion of an *element*, which is the + atomic memory unit handled by the originating object *obj*. For many + simple types such as :class:`str` and :class:`bytearray`, an element + is a single byte, but other third-party types may expose larger elements. + + ``len(view)`` returns the total number of elements in the memoryview, + *view*. The :class:`~memoryview.itemsize` attribute will give you the + number of bytes in a single element. + + A :class:`memoryview` supports slicing to expose its data. Taking a single + index will return a single element as a :class:`str` object. Full + slicing will result in a subview:: + + >>> v = memoryview('abcefg') + >>> v[1] + 'b' + >>> v[-1] + 'g' + >>> v[1:4] + <memory at 0x77ab28> + >>> v[1:4].tobytes() + 'bce' + + If the object the memoryview is over supports changing its data, the + memoryview supports slice assignment:: + + >>> data = bytearray('abcefg') + >>> v = memoryview(data) + >>> v.readonly + False + >>> v[0] = 'z' + >>> data + bytearray(b'zbcefg') + >>> v[1:4] = '123' + >>> data + bytearray(b'z123fg') + >>> v[2] = 'spam' + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + ValueError: cannot modify size of memoryview object + + Notice how the size of the memoryview object cannot be changed. + + :class:`memoryview` has two methods: + + .. method:: tobytes() + + Return the data in the buffer as a bytestring (an object of class + :class:`str`). :: + + >>> m = memoryview("abc") + >>> m.tobytes() + 'abc' + + .. method:: tolist() + + Return the data in the buffer as a list of integers. :: + + >>> memoryview("abc").tolist() + [97, 98, 99] + + There are also several readonly attributes available: + + .. attribute:: format + + A string containing the format (in :mod:`struct` module style) for each + element in the view. This defaults to ``'B'``, a simple bytestring. + + .. attribute:: itemsize + + The size in bytes of each element of the memoryview. + + .. attribute:: shape + + A tuple of integers the length of :attr:`ndim` giving the shape of the + memory as a N-dimensional array. + + .. attribute:: ndim + + An integer indicating how many dimensions of a multi-dimensional array the + memory represents. + + .. attribute:: strides + + A tuple of integers the length of :attr:`ndim` giving the size in bytes to + access each element for each dimension of the array. + + .. attribute:: readonly + + A bool indicating whether the memory is read only. + + .. memoryview.suboffsets isn't documented because it only seems useful for C + + .. _typecontextmanager: Context Manager Types @@ -2742,10 +3031,6 @@ The following attributes are only supported by :term:`new-style class`\ es. .. [#] To format only a tuple you should therefore provide a singleton tuple whose only element is the tuple to be formatted. -.. [#] These numbers are fairly arbitrary. They are intended to avoid printing endless - strings of meaningless digits without hampering correct use and without having - to know the exact precision of floating point values on a particular machine. - .. [#] The advantage of leaving the newline on is that returning an empty string is then an unambiguous EOF indication. It is also possible (in cases where it might matter, for example, if you want to make an exact copy of a file while diff --git a/Doc/library/string.rst b/Doc/library/string.rst index bb0bcc5..59ac116 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -17,6 +17,11 @@ template strings or the ``%`` operator described in the :ref:`string-formatting` section. Also, see the :mod:`re` module for string functions based on regular expressions. +.. seealso:: + + Latest version of the `string module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/string.py?view=markup>`_ + String constants ---------------- @@ -118,7 +123,7 @@ string formatting behaviors using the same implementation as the built-in The :class:`Formatter` class has the following public methods: - .. method:: format(format_string, *args, *kwargs) + .. method:: format(format_string, *args, **kwargs) :meth:`format` is the primary API method. It takes a format template string, and an arbitrary set of positional and keyword argument. @@ -141,7 +146,7 @@ string formatting behaviors using the same implementation as the built-in Loop over the format_string and return an iterable of tuples (*literal_text*, *field_name*, *format_spec*, *conversion*). This is used - by :meth:`vformat` to break the string in to either literal text, or + by :meth:`vformat` to break the string into either literal text, or replacement fields. The values in the tuple conceptually represent a span of literal text @@ -190,7 +195,7 @@ string formatting behaviors using the same implementation as the built-in the format string (integers for positional arguments, and strings for named arguments), and a reference to the *args* and *kwargs* that was passed to vformat. The set of unused args can be calculated from these - parameters. :meth:`check_unused_args` is assumed to throw an exception if + parameters. :meth:`check_unused_args` is assumed to raise an exception if the check fails. .. method:: format_field(value, format_spec) @@ -222,32 +227,43 @@ literal text, it can be escaped by doubling: ``{{`` and ``}}``. The grammar for a replacement field is as follows: .. productionlist:: sf - replacement_field: "{" `field_name` ["!" `conversion`] [":" `format_spec`] "}" - field_name: (`identifier` | `integer`) ("." `attribute_name` | "[" `element_index` "]")* + replacement_field: "{" [`field_name`] ["!" `conversion`] [":" `format_spec`] "}" + field_name: arg_name ("." `attribute_name` | "[" `element_index` "]")* + arg_name: [`identifier` | `integer`] attribute_name: `identifier` element_index: `integer` | `index_string` index_string: <any source character except "]"> + conversion: "r" | "s" format_spec: <described in the next section> -In less formal terms, the replacement field starts with a *field_name*, which -can either be a number (for a positional argument), or an identifier (for -keyword arguments). Following this is an optional *conversion* field, which is +In less formal terms, the replacement field can start with a *field_name* that specifies +the object whose value is to be formatted and inserted +into the output instead of the replacement field. +The *field_name* is optionally followed by a *conversion* field, which is preceded by an exclamation point ``'!'``, and a *format_spec*, which is preceded -by a colon ``':'``. +by a colon ``':'``. These specify a non-default format for the replacement value. See also the :ref:`formatspec` section. -The *field_name* itself begins with either a number or a keyword. If it's a -number, it refers to a positional argument, and if it's a keyword it refers to a -named keyword argument. This can be followed by any number of index or +The *field_name* itself begins with an *arg_name* that is either either a number or a +keyword. If it's a number, it refers to a positional argument, and if it's a keyword, +it refers to a named keyword argument. If the numerical arg_names in a format string +are 0, 1, 2, ... in sequence, they can all be omitted (not just some) +and the numbers 0, 1, 2, ... will be automatically inserted in that order. +The *arg_name* can be followed by any number of index or attribute expressions. An expression of the form ``'.name'`` selects the named attribute using :func:`getattr`, while an expression of the form ``'[index]'`` does an index lookup using :func:`__getitem__`. +.. versionchanged:: 2.7 + The positional argument specifiers can be omitted, so ``'{} {}'`` is + equivalent to ``'{0} {1}'``. + Some simple format string examples:: "First, thou shalt count to {0}" # References first positional argument + "Bring me a {}" # Implicitly references the first positional argument + "From {} to {}" # Same as "From {0} to {1}" "My quest is {name}" # References keyword argument 'name' "Weight in tons {0.weight}" # 'weight' attribute of first positional arg "Units destroyed: {players[0]}" # First element of keyword argument 'players'. @@ -305,7 +321,7 @@ non-empty format string typically modifies the result. The general form of a *standard format specifier* is: .. productionlist:: sf - format_spec: [[`fill`]`align`][`sign`][#][0][`width`][.`precision`][`type`] + format_spec: [[`fill`]`align`][`sign`][#][0][`width`][,][.`precision`][`type`] fill: <a character other than '}'> align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " @@ -313,11 +329,11 @@ The general form of a *standard format specifier* is: precision: `integer` type: "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%" -The *fill* character can be any character other than '}' (which signifies the -end of the field). The presence of a fill character is signaled by the *next* -character, which must be one of the alignment options. If the second character -of *format_spec* is not a valid alignment option, then it is assumed that both -the fill character and the alignment option are absent. +The *fill* character can be any character other than '{' or '}'. The presence +of a fill character is signaled by the character following it, which must be +one of the alignment options. If the second character of *format_spec* is not +a valid alignment option, then it is assumed that both the fill character and +the alignment option are absent. The meaning of the various alignment options is as follows: @@ -325,10 +341,10 @@ The meaning of the various alignment options is as follows: | Option | Meaning | +=========+==========================================================+ | ``'<'`` | Forces the field to be left-aligned within the available | - | | space (this is the default). | + | | space (this is the default for most objects). | +---------+----------------------------------------------------------+ | ``'>'`` | Forces the field to be right-aligned within the | - | | available space. | + | | available space (this is the default for numbers). | +---------+----------------------------------------------------------+ | ``'='`` | Forces the padding to be placed after the sign (if any) | | | but before the digits. This is used for printing fields | @@ -363,6 +379,13 @@ The ``'#'`` option is only valid for integers, and only for binary, octal, or hexadecimal output. If present, it specifies that the output will be prefixed by ``'0b'``, ``'0o'``, or ``'0x'``, respectively. +The ``','`` option signals the use of a comma for a thousands separator. +For a locale aware separator, use the ``'n'`` integer presentation type +instead. + +.. versionchanged:: 2.7 + Added the ``','`` option (see also :pep:`378`). + *width* is a decimal integer defining the minimum field width. If not specified, then the field width will be determined by the content. @@ -454,7 +477,7 @@ The available presentation types for floating point and decimal values are: | | from the significand, and the decimal point is also | | | removed if there are no remaining digits following it. | | | | - | | Postive and negative infinity, positive and negative | + | | Positive and negative infinity, positive and negative | | | zero, and nans, are formatted as ``inf``, ``-inf``, | | | ``0``, ``-0`` and ``nan`` respectively, regardless of | | | the precision. | @@ -488,7 +511,7 @@ the old ``%``-formatting. In most of the cases the syntax is similar to the old ``%``-formatting, with the addition of the ``{}`` and with ``:`` used instead of ``%``. -For example, ``'%03.2f'`` can be translated to ``'{0:03.2f}'``. +For example, ``'%03.2f'`` can be translated to ``'{:03.2f}'``. The new format syntax also supports new and different options, shown in the follow examples. @@ -497,6 +520,8 @@ Accessing arguments by position:: >>> '{0}, {1}, {2}'.format('a', 'b', 'c') 'a, b, c' + >>> '{}, {}, {}'.format('a', 'b', 'c') # 2.7+ only + 'a, b, c' >>> '{2}, {1}, {0}'.format('a', 'b', 'c') 'c, b, a' >>> '{2}, {1}, {0}'.format(*'abc') # unpacking argument sequence @@ -536,27 +561,27 @@ Accessing arguments' items:: Replacing ``%s`` and ``%r``:: - >>> "repr() shows quotes: {0!r}; str() doesn't: {1!s}".format('test1', 'test2') + >>> "repr() shows quotes: {!r}; str() doesn't: {!s}".format('test1', 'test2') "repr() shows quotes: 'test1'; str() doesn't: test2" Aligning the text and specifying a width:: - >>> '{0:<30}'.format('left aligned') + >>> '{:<30}'.format('left aligned') 'left aligned ' - >>> '{0:>30}'.format('right aligned') + >>> '{:>30}'.format('right aligned') ' right aligned' - >>> '{0:^30}'.format('centered') + >>> '{:^30}'.format('centered') ' centered ' - >>> '{0:*^30}'.format('centered') # use '*' as a fill char + >>> '{:*^30}'.format('centered') # use '*' as a fill char '***********centered***********' Replacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:: - >>> '{0:+f}; {0:+f}'.format(3.14, -3.14) # show it always + >>> '{:+f}; {:+f}'.format(3.14, -3.14) # show it always '+3.140000; -3.140000' - >>> '{0: f}; {0: f}'.format(3.14, -3.14) # show a space for positive numbers + >>> '{: f}; {: f}'.format(3.14, -3.14) # show a space for positive numbers ' 3.140000; -3.140000' - >>> '{0:-f}; {0:-f}'.format(3.14, -3.14) # show only the minus -- same as '{0:f}; {0:f}' + >>> '{:-f}; {:-f}'.format(3.14, -3.14) # show only the minus -- same as '{:f}; {:f}' '3.140000; -3.140000' Replacing ``%x`` and ``%o`` and converting the value to different bases:: @@ -568,31 +593,36 @@ Replacing ``%x`` and ``%o`` and converting the value to different bases:: >>> "int: {0:d}; hex: {0:#x}; oct: {0:#o}; bin: {0:#b}".format(42) 'int: 42; hex: 0x2a; oct: 0o52; bin: 0b101010' +Using the comma as a thousands separator:: + + >>> '{:,}'.format(1234567890) + '1,234,567,890' + Expressing a percentage:: >>> points = 19.5 >>> total = 22 - >>> 'Correct answers: {0:.2%}.'.format(points/total) + >>> 'Correct answers: {:.2%}.'.format(points/total) 'Correct answers: 88.64%' Using type-specific formatting:: >>> import datetime >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58) - >>> '{0:%Y-%m-%d %H:%M:%S}'.format(d) + >>> '{:%Y-%m-%d %H:%M:%S}'.format(d) '2010-07-04 12:15:58' Nesting arguments and more complex examples:: >>> for align, text in zip('<^>', ['left', 'center', 'right']): - ... '{0:{align}{fill}16}'.format(text, fill=align, align=align) + ... '{0:{fill}{align}16}'.format(text, fill=align, align=align) ... 'left<<<<<<<<<<<<' '^^^^^center^^^^^' '>>>>>>>>>>>right' >>> >>> octets = [192, 168, 0, 1] - >>> '{0:02X}{1:02X}{2:02X}{3:02X}'.format(*octets) + >>> '{:02X}{:02X}{:02X}{:02X}'.format(*octets) 'C0A80001' >>> int(_, 16) 3232235521 diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst index f1f019d..0b8052c 100644 --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -169,38 +169,38 @@ platform-dependent. +--------+-------------------------+--------------------+----------------+------------+ | ``c`` | :ctype:`char` | string of length 1 | 1 | | +--------+-------------------------+--------------------+----------------+------------+ -| ``b`` | :ctype:`signed char` | integer | 1 | | +| ``b`` | :ctype:`signed char` | integer | 1 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``B`` | :ctype:`unsigned char` | integer | 1 | | +| ``B`` | :ctype:`unsigned char` | integer | 1 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ | ``?`` | :ctype:`_Bool` | bool | 1 | \(1) | +--------+-------------------------+--------------------+----------------+------------+ -| ``h`` | :ctype:`short` | integer | 2 | | +| ``h`` | :ctype:`short` | integer | 2 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``H`` | :ctype:`unsigned short` | integer | 2 | | +| ``H`` | :ctype:`unsigned short` | integer | 2 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``i`` | :ctype:`int` | integer | 4 | | +| ``i`` | :ctype:`int` | integer | 4 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``I`` | :ctype:`unsigned int` | integer | 4 | | +| ``I`` | :ctype:`unsigned int` | integer | 4 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``l`` | :ctype:`long` | integer | 4 | | +| ``l`` | :ctype:`long` | integer | 4 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``L`` | :ctype:`unsigned long` | integer | 4 | | +| ``L`` | :ctype:`unsigned long` | integer | 4 | \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``q`` | :ctype:`long long` | integer | 8 | \(2) | +| ``q`` | :ctype:`long long` | integer | 8 | \(2), \(3) | +--------+-------------------------+--------------------+----------------+------------+ -| ``Q`` | :ctype:`unsigned long | integer | 8 | \(2) | +| ``Q`` | :ctype:`unsigned long | integer | 8 | \(2), \(3) | | | long` | | | | +--------+-------------------------+--------------------+----------------+------------+ -| ``f`` | :ctype:`float` | float | 4 | \(3) | +| ``f`` | :ctype:`float` | float | 4 | \(4) | +--------+-------------------------+--------------------+----------------+------------+ -| ``d`` | :ctype:`double` | float | 8 | \(3) | +| ``d`` | :ctype:`double` | float | 8 | \(4) | +--------+-------------------------+--------------------+----------------+------------+ | ``s`` | :ctype:`char[]` | string | | | +--------+-------------------------+--------------------+----------------+------------+ | ``p`` | :ctype:`char[]` | string | | | +--------+-------------------------+--------------------+----------------+------------+ -| ``P`` | :ctype:`void \*` | integer | | \(4) | +| ``P`` | :ctype:`void \*` | integer | | \(5), \(3) | +--------+-------------------------+--------------------+----------------+------------+ Notes: @@ -220,11 +220,27 @@ Notes: .. versionadded:: 2.2 (3) + When attempting to pack a non-integer using any of the integer conversion + codes, if the non-integer has a :meth:`__index__` method then that method is + called to convert the argument to an integer before packing. If no + :meth:`__index__` method exists, or the call to :meth:`__index__` raises + :exc:`TypeError`, then the :meth:`__int__` method is tried. However, the use + of :meth:`__int__` is deprecated, and will raise :exc:`DeprecationWarning`. + + .. versionchanged:: 2.7 + Use of the :meth:`__index__` method for non-integers is new in 2.7. + + .. versionchanged:: 2.7 + Prior to version 2.7, not all integer conversion codes would use the + :meth:`__int__` method to convert, and :exc:`DeprecationWarning` was + raised only for float arguments. + +(4) For the ``'f'`` and ``'d'`` conversion codes, the packed representation uses the IEEE 754 binary32 (for ``'f'``) or binary64 (for ``'d'``) format, regardless of the floating-point format used by the platform. -(4) +(5) The ``'P'`` format character is only available for the native byte ordering (selected as the default or with the ``'@'`` byte order character). The byte order character ``'='`` chooses to use little- or big-endian ordering based @@ -247,14 +263,14 @@ specified number of bytes. As a special case, ``'0s'`` means a single, empty string (while ``'0c'`` means 0 characters). The ``'p'`` format character encodes a "Pascal string", meaning a short -variable-length string stored in a fixed number of bytes. The count is the total -number of bytes stored. The first byte stored is the length of the string, or -255, whichever is smaller. The bytes of the string follow. If the string -passed in to :func:`pack` is too long (longer than the count minus 1), only the -leading count-1 bytes of the string are stored. If the string is shorter than -count-1, it is padded with null bytes so that exactly count bytes in all are -used. Note that for :func:`unpack`, the ``'p'`` format character consumes count -bytes, but that the string returned can never contain more than 255 characters. +variable-length string stored in a *fixed number of bytes*, given by the count. +The first byte stored is the length of the string, or 255, whichever is smaller. +The bytes of the string follow. If the string passed in to :func:`pack` is too +long (longer than the count minus 1), only the leading ``count-1`` bytes of the +string are stored. If the string is shorter than ``count-1``, it is padded with +null bytes so that exactly count bytes in all are used. Note that for +:func:`unpack`, the ``'p'`` format character consumes count bytes, but that the +string returned can never contain more than 255 characters. For the ``'P'`` format character, the return value is a Python integer or long integer, depending on the size needed to hold a pointer when it has been cast to diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 6f2d55b..425c526 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -81,6 +81,24 @@ This module defines one class called :class:`Popen`: Popen(['/bin/sh', '-c', args[0], args[1], ...]) + .. warning:: + + Executing shell commands that incorporate unsanitized input from an + untrusted source makes a program vulnerable to `shell injection + <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_, + a serious security flaw which can result in arbitrary command execution. + For this reason, the use of *shell=True* is **strongly discouraged** in cases + where the command string is constructed from external input:: + + >>> from subprocess import call + >>> filename = input("What file would you like to display?\n") + What file would you like to display? + non_existent; rm -rf / # + >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly... + + *shell=False* does not suffer from this vulnerability; the above Note may be + helpful in getting code using *shell=False* to work. + On Windows: the :class:`Popen` class uses CreateProcess() to execute the child program, which operates on strings. If *args* is a sequence, it will be converted to a string using the :meth:`list2cmdline` method. Please note that @@ -196,6 +214,13 @@ This module also defines two shortcut functions: >>> retcode = subprocess.call(["ls", "-l"]) + .. warning:: + + Like :meth:`Popen.wait`, this will deadlock when using + ``stdout=PIPE`` and/or ``stderr=PIPE`` and the child process + generates enough output to a pipe such that it blocks waiting + for the OS pipe buffer to accept more data. + .. function:: check_call(*popenargs, **kwargs) @@ -211,6 +236,35 @@ This module also defines two shortcut functions: .. versionadded:: 2.5 + .. warning:: + + See the warning for :func:`call`. + + +.. function:: check_output(*popenargs, **kwargs) + + Run command with arguments and return its output as a byte string. + + If the exit code was non-zero it raises a :exc:`CalledProcessError`. The + :exc:`CalledProcessError` object will have the return code in the + :attr:`returncode` + attribute and output in the :attr:`output` attribute. + + The arguments are the same as for the :class:`Popen` constructor. Example:: + + >>> subprocess.check_output(["ls", "-l", "/dev/null"]) + 'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n' + + The stdout argument is not allowed as it is used internally. + To capture standard error in the result, use ``stderr=subprocess.STDOUT``:: + + >>> subprocess.check_output( + ... ["/bin/sh", "-c", "ls non_existent_file; exit 0"], + ... stderr=subprocess.STDOUT) + 'ls: non_existent_file: No such file or directory\n' + + .. versionadded:: 2.7 + Exceptions ^^^^^^^^^^ @@ -218,7 +272,7 @@ Exceptions Exceptions raised in the child process, before the new program has started to execute, will be re-raised in the parent. Additionally, the exception object will have one extra attribute called :attr:`child_traceback`, which is a string -containing traceback information from the childs point of view. +containing traceback information from the child's point of view. The most common exception raised is :exc:`OSError`. This occurs, for example, when trying to execute a non-existent file. Applications should prepare for @@ -258,9 +312,10 @@ Instances of the :class:`Popen` class have the following methods: .. warning:: - This will deadlock if the child process generates enough output to a - stdout or stderr pipe such that it blocks waiting for the OS pipe buffer - to accept more data. Use :meth:`communicate` to avoid that. + This will deadlock when using ``stdout=PIPE`` and/or + ``stderr=PIPE`` and the child process generates enough output to + a pipe such that it blocks waiting for the OS pipe buffer to + accept more data. Use :meth:`communicate` to avoid that. .. method:: Popen.communicate(input=None) @@ -289,8 +344,9 @@ Instances of the :class:`Popen` class have the following methods: .. note:: - On Windows only SIGTERM is supported so far. It's an alias for - :meth:`terminate`. + On Windows, SIGTERM is an alias for :meth:`terminate`. CTRL_C_EVENT and + CTRL_BREAK_EVENT can be sent to processes started with a *creationflags* + parameter which includes `CREATE_NEW_PROCESS_GROUP`. .. versionadded:: 2.6 @@ -394,8 +450,11 @@ Replacing shell pipeline ==> p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) + p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate()[0] +The p1.stdout.close() call after starting the p2 is important in order for p1 +to receive a SIGPIPE if p2 exits before p1. Replacing :func:`os.system` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index f7a0a5d..12f861b 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -53,6 +53,13 @@ always available. ``modules.keys()`` only lists the imported modules.) +.. function:: call_tracing(func, args) + + Call ``func(*args)``, while tracing is enabled. The tracing state is saved, + and restored afterwards. This is intended to be called from a debugger from + a checkpoint, to recursively debug some other code. + + .. data:: copyright A string containing the copyright pertaining to the Python interpreter. @@ -201,7 +208,7 @@ always available. A string giving the site-specific directory prefix where the platform-dependent Python files are installed; by default, this is also ``'/usr/local'``. This can - be set at build time with the :option:`--exec-prefix` argument to the + be set at build time with the ``--exec-prefix`` argument to the :program:`configure` script. Specifically, all configuration files (e.g. the :file:`pyconfig.h` header file) are installed in the directory ``exec_prefix + '/lib/pythonversion/config'``, and shared library modules are installed in @@ -219,19 +226,25 @@ always available. Exit from Python. This is implemented by raising the :exc:`SystemExit` exception, so cleanup actions specified by finally clauses of :keyword:`try` - statements are honored, and it is possible to intercept the exit attempt at an - outer level. The optional argument *arg* can be an integer giving the exit - status (defaulting to zero), or another type of object. If it is an integer, - zero is considered "successful termination" and any nonzero value is considered - "abnormal termination" by shells and the like. Most systems require it to be in - the range 0-127, and produce undefined results otherwise. Some systems have a - convention for assigning specific meanings to specific exit codes, but these are - generally underdeveloped; Unix programs generally use 2 for command line syntax - errors and 1 for all other kind of errors. If another type of object is passed, - ``None`` is equivalent to passing zero, and any other object is printed to - ``sys.stderr`` and results in an exit code of 1. In particular, - ``sys.exit("some error message")`` is a quick way to exit a program when an - error occurs. + statements are honored, and it is possible to intercept the exit attempt at + an outer level. + + The optional argument *arg* can be an integer giving the exit status + (defaulting to zero), or another type of object. If it is an integer, zero + is considered "successful termination" and any nonzero value is considered + "abnormal termination" by shells and the like. Most systems require it to be + in the range 0-127, and produce undefined results otherwise. Some systems + have a convention for assigning specific meanings to specific exit codes, but + these are generally underdeveloped; Unix programs generally use 2 for command + line syntax errors and 1 for all other kind of errors. If another type of + object is passed, ``None`` is equivalent to passing zero, and any other + object is printed to :data:`stderr` and results in an exit code of 1. In + particular, ``sys.exit("some error message")`` is a quick way to exit a + program when an error occurs. + + Since :func:`exit` ultimately "only" raises an exception, it will only exit + the process when called from the main thread, and the exception is not + intercepted. .. data:: exitfunc @@ -256,39 +269,25 @@ always available. The struct sequence *flags* exposes the status of command line flags. The attributes are read only. - +------------------------------+------------------------------------------+ - | attribute | flag | - +==============================+==========================================+ - | :const:`debug` | -d | - +------------------------------+------------------------------------------+ - | :const:`py3k_warning` | -3 | - +------------------------------+------------------------------------------+ - | :const:`division_warning` | -Q | - +------------------------------+------------------------------------------+ - | :const:`division_new` | -Qnew | - +------------------------------+------------------------------------------+ - | :const:`inspect` | -i | - +------------------------------+------------------------------------------+ - | :const:`interactive` | -i | - +------------------------------+------------------------------------------+ - | :const:`optimize` | -O or -OO | - +------------------------------+------------------------------------------+ - | :const:`dont_write_bytecode` | -B | - +------------------------------+------------------------------------------+ - | :const:`no_user_site` | -s | - +------------------------------+------------------------------------------+ - | :const:`no_site` | -S | - +------------------------------+------------------------------------------+ - | :const:`ignore_environment` | -E | - +------------------------------+------------------------------------------+ - | :const:`tabcheck` | -t or -tt | - +------------------------------+------------------------------------------+ - | :const:`verbose` | -v | - +------------------------------+------------------------------------------+ - | :const:`unicode` | -U | - +------------------------------+------------------------------------------+ - | :const:`bytes_warning` | -b | - +------------------------------+------------------------------------------+ + ============================= =================================== + attribute flag + ============================= =================================== + :const:`debug` :option:`-d` + :const:`py3k_warning` :option:`-3` + :const:`division_warning` :option:`-Q` + :const:`division_new` :option:`-Qnew <-Q>` + :const:`inspect` :option:`-i` + :const:`interactive` :option:`-i` + :const:`optimize` :option:`-O` or :option:`-OO` + :const:`dont_write_bytecode` :option:`-B` + :const:`no_user_site` :option:`-s` + :const:`no_site` :option:`-S` + :const:`ignore_environment` :option:`-E` + :const:`tabcheck` :option:`-t` or :option:`-tt <-t>` + :const:`verbose` :option:`-v` + :const:`unicode` :option:`-U` + :const:`bytes_warning` :option:`-b` + ============================= =================================== .. versionadded:: 2.6 @@ -358,6 +357,18 @@ always available. .. versionadded:: 2.6 +.. data:: float_repr_style + + A string indicating how the :func:`repr` function behaves for + floats. If the string has value ``'short'`` then for a finite + float ``x``, ``repr(x)`` aims to produce a short string with the + property that ``float(repr(x)) == x``. This is the usual behaviour + in Python 2.7 and later. Otherwise, ``float_repr_style`` has value + ``'legacy'`` and ``repr(x)`` behaves in the same way as it did in + versions of Python prior to 2.7. + + .. versionadded:: 2.7 + .. function:: getcheckinterval() @@ -482,9 +493,15 @@ always available. .. function:: getwindowsversion() - Return a tuple containing five components, describing the Windows version - currently running. The elements are *major*, *minor*, *build*, *platform*, and - *text*. *text* contains a string while all other values are integers. + Return a named tuple describing the Windows version + currently running. The named elements are *major*, *minor*, + *build*, *platform*, *service_pack*, *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. + *service_pack* contains a string while all other values are + integers. The components can also be accessed by name, so + ``sys.getwindowsversion()[0]`` is equivalent to + ``sys.getwindowsversion().major``. For compatibility with prior + versions, only the first 5 elements are retrievable by indexing. *platform* may be one of the following values: @@ -500,12 +517,31 @@ always available. | :const:`3 (VER_PLATFORM_WIN32_CE)` | Windows CE | +-----------------------------------------+-------------------------+ - This function wraps the Win32 :cfunc:`GetVersionEx` function; see the Microsoft - documentation for more information about these fields. + *product_type* may be one of the following values: + + +---------------------------------------+---------------------------------+ + | Constant | Meaning | + +=======================================+=================================+ + | :const:`1 (VER_NT_WORKSTATION)` | The system is a workstation. | + +---------------------------------------+---------------------------------+ + | :const:`2 (VER_NT_DOMAIN_CONTROLLER)` | The system is a domain | + | | controller. | + +---------------------------------------+---------------------------------+ + | :const:`3 (VER_NT_SERVER)` | The system is a server, but not | + | | a domain controller. | + +---------------------------------------+---------------------------------+ + + + This function wraps the Win32 :cfunc:`GetVersionEx` function; see the + Microsoft documentation on :cfunc:`OSVERSIONINFOEX` for more information + about these fields. Availability: Windows. .. versionadded:: 2.3 + .. versionchanged:: 2.7 + Changed to a named tuple and added *service_pack_minor*, + *service_pack_major*, *suite_mask*, and *product_type*. .. data:: hexversion @@ -529,6 +565,25 @@ always available. .. versionadded:: 1.5.2 +.. data:: long_info + + A struct sequence that holds information about Python's + internal representation of integers. The attributes are read only. + + +-------------------------+----------------------------------------------+ + | attribute | explanation | + +=========================+==============================================+ + | :const:`bits_per_digit` | number of bits held in each digit. Python | + | | integers are stored internally in base | + | | ``2**long_info.bits_per_digit`` | + +-------------------------+----------------------------------------------+ + | :const:`sizeof_digit` | size in bytes of the C type used to | + | | represent a digit | + +-------------------------+----------------------------------------------+ + + .. versionadded:: 2.7 + + .. data:: last_type last_value last_traceback @@ -666,7 +721,7 @@ always available. A string giving the site-specific directory prefix where the platform independent Python files are installed; by default, this is the string - ``'/usr/local'``. This can be set at build time with the :option:`--prefix` + ``'/usr/local'``. This can be set at build time with the ``--prefix`` argument to the :program:`configure` script. The main collection of Python library modules is installed in the directory ``prefix + '/lib/pythonversion'`` while the platform independent header files (all except :file:`pyconfig.h`) are @@ -810,14 +865,17 @@ always available. specifies the local trace function. ``'line'`` - The interpreter is about to execute a new line of code (sometimes multiple - line events on one line exist). The local trace function is called; *arg* - is ``None``; the return value specifies the new local trace function. + The interpreter is about to execute a new line of code or re-execute the + condition of a loop. The local trace function is called; *arg* is + ``None``; the return value specifies the new local trace function. See + :file:`Objects/lnotab_notes.txt` for a detailed explanation of how this + works. ``'return'`` A function (or other code block) is about to return. The local trace - function is called; *arg* is the value that will be returned. The trace - function's return value is ignored. + function is called; *arg* is the value that will be returned, or ``None`` + if the event is caused by an exception being raised. The trace function's + return value is ignored. ``'exception'`` An exception has occurred. The local trace function is called; *arg* is a @@ -829,10 +887,10 @@ always available. a built-in. *arg* is the C function object. ``'c_return'`` - A C function has returned. *arg* is ``None``. + A C function has returned. *arg* is the C function object. ``'c_exception'`` - A C function has thrown an exception. *arg* is ``None``. + A C function has raised an exception. *arg* is the C function object. Note that as an exception is propagated down the chain of callers, an ``'exception'`` event is generated at each level. @@ -851,7 +909,7 @@ always available. Activate dumping of VM measurements using the Pentium timestamp counter, if *on_flag* is true. Deactivate these dumps if *on_flag* is off. The function is - available only if Python was compiled with :option:`--with-tsc`. To understand + available only if Python was compiled with ``--with-tsc``. To understand the output of this dump, read :file:`Python/ceval.c` in the Python sources. .. versionadded:: 2.4 @@ -929,9 +987,13 @@ always available. *micro*, *releaselevel*, and *serial*. All values except *releaselevel* are integers; the release level is ``'alpha'``, ``'beta'``, ``'candidate'``, or ``'final'``. The ``version_info`` value corresponding to the Python version 2.0 - is ``(2, 0, 0, 'final', 0)``. + is ``(2, 0, 0, 'final', 0)``. The components can also be accessed by name, + so ``sys.version_info[0]`` is equivalent to ``sys.version_info.major`` + and so on. .. versionadded:: 2.0 + .. versionchanged:: 2.7 + Added named component attributes .. data:: warnoptions diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst new file mode 100644 index 0000000..38aae0a --- /dev/null +++ b/Doc/library/sysconfig.rst @@ -0,0 +1,218 @@ +:mod:`sysconfig` --- Provide access to Python's configuration information +========================================================================= + +.. module:: sysconfig + :synopsis: Python's configuration information +.. moduleauthor:: Tarek Ziade <tarek@ziade.org> +.. sectionauthor:: Tarek Ziade <tarek@ziade.org> +.. versionadded:: 2.7 +.. index:: + single: configuration information + +The :mod:`sysconfig` module provides access to Python's configuration +information like the list of installation paths and the configuration variables +relevant for the current platform. + +Configuration variables +----------------------- + +A Python distribution contains a :file:`Makefile` and a :file:`pyconfig.h` +header file that are necessary to build both the Python binary itself and +third-party C extensions compiled using :mod:`distutils`. + +:mod:`sysconfig` puts all variables found in these files in a dictionary that +can be accessed using :func:`get_config_vars` or :func:`get_config_var`. + +Notice that on Windows, it's a much smaller set. + +.. function:: get_config_vars(\*args) + + With no arguments, return a dictionary of all configuration variables + relevant for the current platform. + + With arguments, return a list of values that result from looking up each + argument in the configuration variable dictionary. + + For each argument, if the value is not found, return ``None``. + + +.. function:: get_config_var(name) + + Return the value of a single variable *name*. Equivalent to + ``get_config_vars().get(name)``. + + If *name* is not found, return ``None``. + +Example of usage:: + + >>> import sysconfig + >>> sysconfig.get_config_var('Py_ENABLE_SHARED') + 0 + >>> sysconfig.get_config_var('LIBDIR') + '/usr/local/lib' + >>> sysconfig.get_config_vars('AR', 'CXX') + ['ar', 'g++'] + + +Installation paths +------------------ + +Python uses an installation scheme that differs depending on the platform and on +the installation options. These schemes are stored in :mod:`sysconfig` under +unique identifiers based on the value returned by :const:`os.name`. + +Every new component that is installed using :mod:`distutils` or a +Distutils-based system will follow the same scheme to copy its file in the right +places. + +Python currently supports seven schemes: + +- *posix_prefix*: scheme for Posix platforms like Linux or Mac OS X. This is + the default scheme used when Python or a component is installed. +- *posix_home*: scheme for Posix platforms used when a *home* option is used + upon installation. This scheme is used when a component is installed through + Distutils with a specific home prefix. +- *posix_user*: scheme for Posix platforms used when a component is installed + through Distutils and the *user* option is used. This scheme defines paths + located under the user home directory. +- *nt*: scheme for NT platforms like Windows. +- *nt_user*: scheme for NT platforms, when the *user* option is used. +- *os2*: scheme for OS/2 platforms. +- *os2_home*: scheme for OS/2 patforms, when the *user* option is used. + +Each scheme is itself composed of a series of paths and each path has a unique +identifier. Python currently uses eight paths: + +- *stdlib*: directory containing the standard Python library files that are not + platform-specific. +- *platstdlib*: directory containing the standard Python library files that are + platform-specific. +- *platlib*: directory for site-specific, platform-specific files. +- *purelib*: directory for site-specific, non-platform-specific files. +- *include*: directory for non-platform-specific header files. +- *platinclude*: directory for platform-specific header files. +- *scripts*: directory for script files. +- *data*: directory for data files. + +:mod:`sysconfig` provides some functions to determine these paths. + +.. function:: get_scheme_names() + + Return a tuple containing all schemes currently supported in + :mod:`sysconfig`. + + +.. function:: get_path_names() + + Return a tuple containing all path names currently supported in + :mod:`sysconfig`. + + +.. function:: get_path(name, [scheme, [vars, [expand]]]) + + Return an installation path corresponding to the path *name*, from the + install scheme named *scheme*. + + *name* has to be a value from the list returned by :func:`get_path_names`. + + :mod:`sysconfig` stores installation paths corresponding to each path name, + for each platform, with variables to be expanded. For instance the *stdlib* + path for the *nt* scheme is: ``{base}/Lib``. + + :func:`get_path` will use the variables returned by :func:`get_config_vars` + to expand the path. All variables have default values for each platform so + one may call this function and get the default value. + + If *scheme* is provided, it must be a value from the list returned by + :func:`get_path_names`. Otherwise, the default scheme for the current + platform is used. + + If *vars* is provided, it must be a dictionary of variables that will update + the dictionary return by :func:`get_config_vars`. + + If *expand* is set to ``False``, the path will not be expanded using the + variables. + + If *name* is not found, return ``None``. + + +.. function:: get_paths([scheme, [vars, [expand]]]) + + Return a dictionary containing all installation paths corresponding to an + installation scheme. See :func:`get_path` for more information. + + If *scheme* is not provided, will use the default scheme for the current + platform. + + If *vars* is provided, it must be a dictionary of variables that will + update the dictionary used to expand the paths. + + If *expand* is set to False, the paths will not be expanded. + + If *scheme* is not an existing scheme, :func:`get_paths` will raise a + :exc:`KeyError`. + + +Other functions +--------------- + +.. function:: get_python_version() + + Return the ``MAJOR.MINOR`` Python version number as a string. Similar to + ``sys.version[:3]``. + + +.. function:: get_platform() + + Return a string that identifies the current platform. + + This is used mainly to distinguish platform-specific build directories and + platform-specific built distributions. Typically includes the OS name and + version and the architecture (as supplied by :func:`os.uname`), although the + exact information included depends on the OS; e.g. for IRIX the architecture + isn't particularly important (IRIX only runs on SGI hardware), but for Linux + the kernel version isn't particularly important. + + Examples of returned values: + + - linux-i586 + - linux-alpha (?) + - solaris-2.6-sun4u + - irix-5.3 + - irix64-6.2 + + Windows will return one of: + + - win-amd64 (64bit Windows on AMD64 (aka x86_64, Intel64, EM64T, etc) + - win-ia64 (64bit Windows on Itanium) + - win32 (all others - specifically, sys.platform is returned) + + Mac OS X can return: + + - macosx-10.6-ppc + - macosx-10.4-ppc64 + - macosx-10.3-i386 + - macosx-10.4-fat + + For other non-POSIX platforms, currently just returns :data:`sys.platform`. + + +.. function:: is_python_build() + + Return ``True`` if the current Python installation was built from source. + + +.. function:: parse_config_h(fp[, vars]) + + Parse a :file:`config.h`\-style file. + + *fp* is a file-like object pointing to the :file:`config.h`\-like file. + + A dictionary containing name/value pairs is returned. If an optional + dictionary is passed in as the second argument, it is used instead of a new + dictionary, and updated with the values read in the file. + + +.. function:: get_config_h_filename() + + Return the path of :file:`pyconfig.h`. diff --git a/Doc/library/syslog.rst b/Doc/library/syslog.rst index 549f26b..7b1a340 100644 --- a/Doc/library/syslog.rst +++ b/Doc/library/syslog.rst @@ -1,4 +1,3 @@ - :mod:`syslog` --- Unix syslog library routines ============================================== @@ -11,42 +10,58 @@ This module provides an interface to the Unix ``syslog`` library routines. Refer to the Unix manual pages for a detailed description of the ``syslog`` facility. +This module wraps the system ``syslog`` family of routines. A pure Python +library that can speak to a syslog server is available in the +:mod:`logging.handlers` module as :class:`SysLogHandler`. + The module defines the following functions: .. function:: syslog([priority,] message) - Send the string *message* to the system logger. A trailing newline is added if - necessary. Each message is tagged with a priority composed of a *facility* and - a *level*. The optional *priority* argument, which defaults to - :const:`LOG_INFO`, determines the message priority. If the facility is not - encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the value - given in the :func:`openlog` call is used. + Send the string *message* to the system logger. A trailing newline is added + if necessary. Each message is tagged with a priority composed of a + *facility* and a *level*. The optional *priority* argument, which defaults + to :const:`LOG_INFO`, determines the message priority. If the facility is + not encoded in *priority* using logical-or (``LOG_INFO | LOG_USER``), the + value given in the :func:`openlog` call is used. + + If :func:`openlog` has not been called prior to the call to :func:`syslog`, + ``openlog()`` will be called with no arguments. -.. function:: openlog(ident[, logopt[, facility]]) +.. function:: openlog([ident[, logopt[, facility]]]) - Logging options other than the defaults can be set by explicitly opening the log - file with :func:`openlog` prior to calling :func:`syslog`. The defaults are - (usually) *ident* = ``'syslog'``, *logopt* = ``0``, *facility* = - :const:`LOG_USER`. The *ident* argument is a string which is prepended to every - message. The optional *logopt* argument is a bit field - see below for possible - values to combine. The optional *facility* argument sets the default facility - for messages which do not have a facility explicitly encoded. + Logging options of subsequent :func:`syslog` calls can be set by calling + :func:`openlog`. :func:`syslog` will call :func:`openlog` with no arguments + if the log is not currently open. + + The optional *ident* keyword argument is a string which is prepended to every + message, and defaults to ``sys.argv[0]`` with leading path components + stripped. The optional *logopt* keyword argument (default is 0) is a bit + field -- see below for possible values to combine. The optional *facility* + keyword argument (default is :const:`LOG_USER`) sets the default facility for + messages which do not have a facility explicitly encoded. .. function:: closelog() - Close the log file. + Reset the syslog module values and call the system library ``closelog()``. + + This causes the module to behave as it does when initially imported. For + example, :func:`openlog` will be called on the first :func:`syslog` call (if + :func:`openlog` hasn't already been called), and *ident* and other + :func:`openlog` parameters are reset to defaults. .. function:: setlogmask(maskpri) - Set the priority mask to *maskpri* and return the previous mask value. Calls to - :func:`syslog` with a priority level not set in *maskpri* are ignored. The - default is to log all priorities. The function ``LOG_MASK(pri)`` calculates the - mask for the individual priority *pri*. The function ``LOG_UPTO(pri)`` - calculates the mask for all priorities up to and including *pri*. + Set the priority mask to *maskpri* and return the previous mask value. Calls + to :func:`syslog` with a priority level not set in *maskpri* are ignored. + The default is to log all priorities. The function ``LOG_MASK(pri)`` + calculates the mask for the individual priority *pri*. The function + ``LOG_UPTO(pri)`` calculates the mask for all priorities up to and including + *pri*. The module defines the following constants: @@ -64,3 +79,24 @@ Log options: :const:`LOG_PID`, :const:`LOG_CONS`, :const:`LOG_NDELAY`, :const:`LOG_NOWAIT` and :const:`LOG_PERROR` if defined in ``<syslog.h>``. + +Examples +-------- + +Simple example +~~~~~~~~~~~~~~ + +A simple set of examples:: + + import syslog + + syslog.syslog('Processing started') + if error: + syslog.syslog(syslog.LOG_ERR, 'Processing started') + +An example of setting some log options, these would include the process ID in +logged messages, and write the messages to the destination facility used for +mail logging:: + + syslog.openlog(logopt=syslog.LOG_PID, facility=syslog.LOG_MAIL) + syslog.syslog('E-mail processing initiated...') diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst index 2996839..a8e1d56 100644 --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -234,6 +234,14 @@ a header block followed by data blocks. It is possible to store a file in a tar archive several times. Each archive member is represented by a :class:`TarInfo` object, see :ref:`tarinfo-objects` for details. +A :class:`TarFile` object can be used as a context manager in a :keyword:`with` +statement. It will automatically be closed when the block is completed. Please +note that in the event of an exception an archive opened for writing will not +be finalized; only the internally used file object will be closed. See the +:ref:`tar-examples` section for a use case. + +.. versionadded:: 2.7 + Added support for the context manager protocol. .. class:: TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors=None, pax_headers=None, debug=0, errorlevel=0) @@ -389,7 +397,7 @@ object, see :ref:`tarinfo-objects` for details. and :meth:`close`, and also supports iteration over its lines. -.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None) +.. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None, filter=None) Add the file *name* to the archive. *name* may be any type of file (directory, fifo, symbolic link, etc.). If given, *arcname* specifies an alternative name @@ -397,11 +405,24 @@ object, see :ref:`tarinfo-objects` for details. can be avoided by setting *recursive* to :const:`False`. If *exclude* is given it must be a function that takes one filename argument and returns a boolean value. Depending on this value the respective file is either excluded - (:const:`True`) or added (:const:`False`). + (:const:`True`) or added (:const:`False`). If *filter* is specified it must + be a function that takes a :class:`TarInfo` object argument and returns the + changed :class:`TarInfo` object. If it instead returns :const:`None` the :class:`TarInfo` + object will be excluded from the archive. See :ref:`tar-examples` for an + example. .. versionchanged:: 2.6 Added the *exclude* parameter. + .. versionchanged:: 2.7 + Added the *filter* parameter. + + .. deprecated:: 2.7 + The *exclude* parameter is deprecated, please use the *filter* parameter + instead. For maximum portability, *filter* should be used as a keyword + argument rather than as a positional argument so that code won't be + affected when *exclude* is ultimately removed. + .. method:: TarFile.addfile(tarinfo, fileobj=None) @@ -639,6 +660,13 @@ How to create an uncompressed tar archive from a list of filenames:: tar.add(name) tar.close() +The same example using the :keyword:`with` statement:: + + import tarfile + with tarfile.open("sample.tar", "w") as tar: + for name in ["foo", "bar", "quux"]: + tar.add(name) + How to read a gzip compressed tar archive and display some member information:: import tarfile @@ -653,6 +681,18 @@ How to read a gzip compressed tar archive and display some member information:: print "something else." tar.close() +How to create an archive and reset the user information using the *filter* +parameter in :meth:`TarFile.add`:: + + import tarfile + def reset(tarinfo): + tarinfo.uid = tarinfo.gid = 0 + tarinfo.uname = tarinfo.gname = "root" + return tarinfo + tar = tarfile.open("sample.tar.gz", "w:gz") + tar.add("foo", filter=reset) + tar.close() + .. _tar-formats: diff --git a/Doc/library/telnetlib.rst b/Doc/library/telnetlib.rst index b1528bd..925eccc 100644 --- a/Doc/library/telnetlib.rst +++ b/Doc/library/telnetlib.rst @@ -198,8 +198,8 @@ Telnet Objects received so far (may be the empty string if a timeout happened). If a regular expression ends with a greedy match (such as ``.*``) or if more - than one expression can match the same input, the results are indeterministic, - and may depend on the I/O timing. + than one expression can match the same input, the results are + non-deterministic, and may depend on the I/O timing. .. method:: Telnet.set_option_negotiation_callback(callback) diff --git a/Doc/library/test.rst b/Doc/library/test.rst index c73e23e..3a0b5cb 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -97,17 +97,17 @@ The goal for regression testing is to try to break code. This leads to a few guidelines to be followed: * The testing suite should exercise all classes, functions, and constants. This - includes not just the external API that is to be presented to the outside world - but also "private" code. + includes not just the external API that is to be presented to the outside + world but also "private" code. * Whitebox testing (examining the code being tested when the tests are being written) is preferred. Blackbox testing (testing only the published user - interface) is not complete enough to make sure all boundary and edge cases are - tested. + interface) is not complete enough to make sure all boundary and edge cases + are tested. * Make sure all possible values are tested including invalid ones. This makes - sure that not only all valid values are acceptable but also that improper values - are handled correctly. + sure that not only all valid values are acceptable but also that improper + values are handled correctly. * Exhaust as many code paths as possible. Test where branching occurs and thus tailor input to make sure as many different paths through the code are taken. @@ -127,8 +127,8 @@ guidelines to be followed: behavior from side-effects of importing a module. * Try to maximize code reuse. On occasion, tests will vary by something as small - as what type of input is used. Minimize code duplication by subclassing a basic - test class with a class that specifies the input:: + as what type of input is used. Minimize code duplication by subclassing a + basic test class with a class that specifies the input:: class TestFuncAcceptsSequences(unittest.TestCase): @@ -138,13 +138,13 @@ guidelines to be followed: self.func(self.arg) class AcceptLists(TestFuncAcceptsSequences): - arg = [1,2,3] + arg = [1, 2, 3] class AcceptStrings(TestFuncAcceptsSequences): arg = 'abc' class AcceptTuples(TestFuncAcceptsSequences): - arg = (1,2,3) + arg = (1, 2, 3) .. seealso:: @@ -155,33 +155,33 @@ guidelines to be followed: .. _regrtest: -Running tests using :mod:`test.regrtest` ----------------------------------------- +Running tests using the command-line interface +---------------------------------------------- -:mod:`test.regrtest` can be used as a script to drive Python's regression test -suite. Running the script by itself automatically starts running all regression +The :mod:`test.regrtest` module can be run as a script to drive Python's regression +test suite, thanks to the :option:`-m` option: :program:`python -m test.regrtest`. +Running the script by itself automatically starts running all regression tests in the :mod:`test` package. It does this by finding all modules in the package whose name starts with ``test_``, importing them, and executing the -function :func:`test_main` if present. The names of tests to execute may also be -passed to the script. Specifying a single regression test (:program:`python -regrtest.py` :option:`test_spam.py`) will minimize output and only print whether +function :func:`test_main` if present. The names of tests to execute may also +be passed to the script. Specifying a single regression test (:program:`python +-m test.regrtest test_spam`) will minimize output and only print whether the test passed or failed and thus minimize output. Running :mod:`test.regrtest` directly allows what resources are available for tests to use to be set. You do this by using the :option:`-u` command-line -option. Run :program:`python regrtest.py` :option:`-uall` to turn on all -resources; specifying :option:`all` as an option for :option:`-u` enables all +option. Run :program:`python -m test.regrtest -uall` to turn on all +resources; specifying ``all`` as an option for ``-u`` enables all possible resources. If all but one resource is desired (a more common case), a comma-separated list of resources that are not desired may be listed after -:option:`all`. The command :program:`python regrtest.py` -:option:`-uall,-audio,-largefile` will run :mod:`test.regrtest` with all -resources except the :option:`audio` and :option:`largefile` resources. For a -list of all resources and more command-line options, run :program:`python -regrtest.py` :option:`-h`. +``all``. The command :program:`python -m test.regrtest -uall,-audio,-largefile` +will run :mod:`test.regrtest` with all resources except the ``audio`` and +``largefile`` resources. For a list of all resources and more command-line +options, run :program:`python -m test.regrtest -h`. Some other ways to execute the regression tests depend on what platform the -tests are being executed on. On Unix, you can run :program:`make` :option:`test` -at the top-level directory where Python was built. On Windows, executing +tests are being executed on. On Unix, you can run :program:`make test` at the +top-level directory where Python was built. On Windows, executing :program:`rt.bat` from your :file:`PCBuild` directory will run all regression tests. @@ -210,17 +210,11 @@ This module defines the following exceptions: methods. -.. exception:: TestSkipped - - Subclass of :exc:`TestFailed`. Raised when a test is skipped. This occurs when a - needed resource (such as a network connection) is not available at the time of - testing. - - .. exception:: ResourceDenied - Subclass of :exc:`TestSkipped`. Raised when a resource (such as a network - connection) is not available. Raised by the :func:`requires` function. + Subclass of :exc:`unittest.SkipTest`. Raised when a resource (such as a + network connection) is not available. Raised by the :func:`requires` + function. The :mod:`test.test_support` module defines the following constants: @@ -244,44 +238,45 @@ The :mod:`test.test_support` module defines the following constants: .. data:: TESTFN - Set to the path that a temporary file may be created at. Any temporary that is - created should be closed and unlinked (removed). + Set to a name that is safe to use as the name of a temporary file. Any + temporary file that is created should be closed and unlinked (removed). The :mod:`test.test_support` module defines the following functions: .. function:: forget(module_name) - Removes the module named *module_name* from ``sys.modules`` and deletes any + Remove the module named *module_name* from ``sys.modules`` and delete any byte-compiled files of the module. .. function:: is_resource_enabled(resource) - Returns :const:`True` if *resource* is enabled and available. The list of + Return :const:`True` if *resource* is enabled and available. The list of available resources is only set when :mod:`test.regrtest` is executing the tests. .. function:: requires(resource[, msg]) - Raises :exc:`ResourceDenied` if *resource* is not available. *msg* is the - argument to :exc:`ResourceDenied` if it is raised. Always returns true if called - by a function whose ``__name__`` is ``'__main__'``. Used when tests are executed - by :mod:`test.regrtest`. + Raise :exc:`ResourceDenied` if *resource* is not available. *msg* is the + argument to :exc:`ResourceDenied` if it is raised. Always returns + :const:`True` if called by a function whose ``__name__`` is ``'__main__'``. + Used when tests are executed by :mod:`test.regrtest`. .. function:: findfile(filename) - Return the path to the file named *filename*. If no match is found *filename* is - returned. This does not equal a failure since it could be the path to the file. + Return the path to the file named *filename*. If no match is found + *filename* is returned. This does not equal a failure since it could be the + path to the file. .. function:: run_unittest(*classes) Execute :class:`unittest.TestCase` subclasses passed to the function. The - function scans the classes for methods starting with the prefix ``test_`` and - executes the tests individually. + function scans the classes for methods starting with the prefix ``test_`` + and executes the tests individually. It is also legal to pass strings as parameters; these should be keys in ``sys.modules``. Each associated module will be scanned by @@ -294,37 +289,85 @@ The :mod:`test.test_support` module defines the following functions: This will run all tests defined in the named module. -.. function:: check_warnings() +.. function:: check_warnings(*filters, quiet=True) + + A convenience wrapper for :func:`warnings.catch_warnings()` that makes it + easier to test that a warning was correctly raised. It is approximately + equivalent to calling ``warnings.catch_warnings(record=True)`` with + :meth:`warnings.simplefilter` set to ``always`` and with the option to + automatically validate the results that are recorded. + + ``check_warnings`` accepts 2-tuples of the form ``("message regexp", + WarningCategory)`` as positional arguments. If one or more *filters* are + provided, or if the optional keyword argument *quiet* is :const:`False`, + it checks to make sure the warnings are as expected: each specified filter + must match at least one of the warnings raised by the enclosed code or the + test fails, and if any warnings are raised that do not match any of the + specified filters the test fails. To disable the first of these checks, + set *quiet* to :const:`True`. + + If no arguments are specified, it defaults to:: + + check_warnings(("", Warning), quiet=True) + + In this case all warnings are caught and no errors are raised. - A convenience wrapper for ``warnings.catch_warnings()`` that makes - it easier to test that a warning was correctly raised with a single - assertion. It is approximately equivalent to calling - ``warnings.catch_warnings(record=True)``. + On entry to the context manager, a :class:`WarningRecorder` instance is + returned. The underlying warnings list from + :func:`~warnings.catch_warnings` is available via the recorder object's + :attr:`warnings` attribute. As a convenience, the attributes of the object + representing the most recent warning can also be accessed directly through + the recorder object (see example below). If no warning has been raised, + then any of the attributes that would otherwise be expected on an object + representing a warning will return :const:`None`. - The main difference is that on entry to the context manager, a - :class:`WarningRecorder` instance is returned instead of a simple list. - The underlying warnings list is available via the recorder object's - :attr:`warnings` attribute, while the attributes of the last raised - warning are also accessible directly on the object. If no warning has - been raised, then the latter attributes will all be :const:`None`. + The recorder object also has a :meth:`reset` method, which clears the + warnings list. - A :meth:`reset` method is also provided on the recorder object. This - method simply clears the warning list. + The context manager is designed to be used like this:: - The context manager is used like this:: + with check_warnings(("assertion is always true", SyntaxWarning), + ("", UserWarning)): + exec('assert(False, "Hey!")') + warnings.warn(UserWarning("Hide me!")) - with check_warnings() as w: - warnings.simplefilter("always") + In this case if either warning was not raised, or some other warning was + raised, :func:`check_warnings` would raise an error. + + When a test needs to look more deeply into the warnings, rather than + just checking whether or not they occurred, code like this can be used:: + + with check_warnings(quiet=True) as w: warnings.warn("foo") - assert str(w.message) == "foo" + assert str(w.args[0]) == "foo" warnings.warn("bar") - assert str(w.message) == "bar" - assert str(w.warnings[0].message) == "foo" - assert str(w.warnings[1].message) == "bar" + assert str(w.args[0]) == "bar" + assert str(w.warnings[0].args[0]) == "foo" + assert str(w.warnings[1].args[0]) == "bar" w.reset() assert len(w.warnings) == 0 + Here all warnings will be caught, and the test code tests the captured + warnings directly. + .. versionadded:: 2.6 + .. versionchanged:: 2.7 + New optional arguments *filters* and *quiet*. + + +.. function:: check_py3k_warnings(*filters, quiet=False) + + Similar to :func:`check_warnings`, but for Python 3 compatibility warnings. + If ``sys.py3kwarning == 1``, it checks if the warning is effectively raised. + If ``sys.py3kwarning == 0``, it checks that no warning is raised. It + accepts 2-tuples of the form ``("message regexp", WarningCategory)`` as + positional arguments. When the optional keyword argument *quiet* is + :const:`True`, it does not fail if a filter catches nothing. Without + arguments, it defaults to:: + + check_py3k_warnings(("", DeprecationWarning), quiet=False) + + .. versionadded:: 2.7 .. function:: captured_stdout() @@ -342,6 +385,54 @@ The :mod:`test.test_support` module defines the following functions: .. versionadded:: 2.6 +.. function:: import_module(name, deprecated=False) + + This function imports and returns the named module. Unlike a normal + import, this function raises :exc:`unittest.SkipTest` if the module + cannot be imported. + + Module and package deprecation messages are suppressed during this import + if *deprecated* is :const:`True`. + + .. versionadded:: 2.7 + + +.. function:: import_fresh_module(name, fresh=(), blocked=(), deprecated=False) + + This function imports and returns a fresh copy of the named Python module + by removing the named module from ``sys.modules`` before doing the import. + Note that unlike :func:`reload`, the original module is not affected by + this operation. + + *fresh* is an iterable of additional module names that are also removed + from the ``sys.modules`` cache before doing the import. + + *blocked* is an iterable of module names that are replaced with :const:`0` + in the module cache during the import to ensure that attempts to import + them raise :exc:`ImportError`. + + The named module and any modules named in the *fresh* and *blocked* + parameters are saved before starting the import and then reinserted into + ``sys.modules`` when the fresh import is complete. + + Module and package deprecation messages are suppressed during this import + if *deprecated* is :const:`True`. + + This function will raise :exc:`unittest.SkipTest` is the named module + cannot be imported. + + Example use:: + + # Get copies of the warnings module for testing without + # affecting the version being used by the rest of the test suite + # One copy uses the C implementation, the other is forced to use + # the pure Python fallback implementation + py_warnings = import_fresh_module('warnings', blocked=['_warnings']) + c_warnings = import_fresh_module('warnings', fresh=['_warnings']) + + .. versionadded:: 2.7 + + The :mod:`test.test_support` module defines the following classes: .. class:: TransientResource(exc[, **kwargs]) @@ -355,25 +446,31 @@ The :mod:`test.test_support` module defines the following classes: .. versionadded:: 2.6 .. class:: EnvironmentVarGuard() - Class used to temporarily set or unset environment variables. Instances can be - used as a context manager. + Class used to temporarily set or unset environment variables. Instances can + be used as a context manager and have a complete dictionary interface for + querying/modifying the underlying ``os.environ``. After exit from the + context manager all changes to environment variables done through this + instance will be rolled back. .. versionadded:: 2.6 + .. versionchanged:: 2.7 + Added dictionary interface. .. method:: EnvironmentVarGuard.set(envvar, value) - Temporarily set the environment variable ``envvar`` to the value of ``value``. + Temporarily set the environment variable ``envvar`` to the value of + ``value``. .. method:: EnvironmentVarGuard.unset(envvar) Temporarily unset the environment variable ``envvar``. + .. class:: WarningsRecorder() Class used to record warnings for unit tests. See documentation of :func:`check_warnings` above for more details. .. versionadded:: 2.6 - diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index a2db567..8fbfe9c 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -16,6 +16,10 @@ and a utility function :func:`dedent`. If you're just wrapping or filling one or two text strings, the convenience functions should be good enough; otherwise, you should use an instance of :class:`TextWrapper` for efficiency. +.. seealso:: + + Latest version of the `textwrap module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/textwrap.py?view=markup>`_ .. function:: wrap(text[, width[, ...]]) @@ -121,6 +125,13 @@ indentation from strings that have unwanted whitespace to the left of the text. each tab character will be replaced by a single space, which is *not* the same as tab expansion. + .. note:: + + If :attr:`replace_whitespace` is false, newlines may appear in the + middle of a line and cause strange output. For this reason, text should + be split into paragraphs (using :meth:`str.splitlines` or similar) + which are wrapped separately. + .. attribute:: drop_whitespace diff --git a/Doc/library/thread.rst b/Doc/library/thread.rst index f8b5850..7e8d5c8 100644 --- a/Doc/library/thread.rst +++ b/Doc/library/thread.rst @@ -112,6 +112,7 @@ It defines the following constant and functions: .. versionadded:: 2.5 + Lock objects have the following methods: diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst index 6a8ed0d..0b46349 100644 --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -26,6 +26,21 @@ The :mod:`dummy_threading` module is provided for situations where Starting with Python 2.5, several Thread methods raise :exc:`RuntimeError` instead of :exc:`AssertionError` if called erroneously. +.. impl-detail:: + + Due to the :term:`Global Interpreter Lock`, in CPython only one thread + can execute Python code at once (even though certain performance-oriented + libraries might overcome this limitation). + If you want your application to make better of use of the computational + resources of multi-core machines, you are advised to use + :mod:`multiprocessing`. However, threading is still an appropriate model + if you want to run multiple I/O-bound tasks simultaneously. + +.. seealso:: + + Latest version of the `threading module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/threading.py?view=markup>`_ + This module defines the following functions and objects: @@ -43,6 +58,8 @@ This module defines the following functions and objects: variable allows one or more threads to wait until they are notified by another thread. + See :ref:`condition-objects`. + .. function:: current_thread() currentThread() @@ -69,6 +86,8 @@ This module defines the following functions and objects: with the :meth:`clear` method. The :meth:`wait` method blocks until the flag is true. + See :ref:`event-objects`. + .. class:: local @@ -93,6 +112,8 @@ This module defines the following functions and objects: acquired it, subsequent attempts to acquire it block, until it is released; any thread may release it. + See :ref:`lock-objects`. + .. function:: RLock() @@ -101,6 +122,8 @@ This module defines the following functions and objects: reentrant lock, the same thread may acquire it again without blocking; the thread must release it once for each time it has acquired it. + See :ref:`rlock-objects`. + .. function:: Semaphore([value]) :noindex: @@ -111,6 +134,8 @@ This module defines the following functions and objects: if necessary until it can return without making the counter negative. If not given, *value* defaults to 1. + See :ref:`semaphore-objects`. + .. function:: BoundedSemaphore([value]) @@ -122,15 +147,21 @@ This module defines the following functions and objects: .. class:: Thread + :noindex: A class that represents a thread of control. This class can be safely subclassed in a limited fashion. + See :ref:`thread-objects`. + .. class:: Timer + :noindex: A thread that executes a function after a specified interval has passed. + See :ref:`timer-objects`. + .. function:: settrace(func) @@ -259,7 +290,7 @@ impossible to detect the termination of alien threads. It must be called at most once per thread object. It arranges for the object's :meth:`run` method to be invoked in a separate thread of control. - This method will raise a :exc:`RuntimeException` if called more than once + This method will raise a :exc:`RuntimeError` if called more than once on the same thread object. .. method:: run() @@ -368,7 +399,7 @@ and may vary across implementations. All methods are executed atomically. -.. method:: Lock.acquire([blocking=1]) +.. method:: Lock.acquire([blocking]) Acquire a lock, blocking or non-blocking. @@ -624,9 +655,9 @@ waiting until some other thread calls :meth:`release`. ^^^^^^^^^^^^^^^^^^^^^^^^^^ Semaphores are often used to guard resources with limited capacity, for example, -a database server. In any situation where the size of the resource size is -fixed, you should use a bounded semaphore. Before spawning any worker threads, -your main thread would initialize the semaphore:: +a database server. In any situation where the size of the resource is fixed, +you should use a bounded semaphore. Before spawning any worker threads, your +main thread would initialize the semaphore:: maxconnections = 5 ... @@ -693,7 +724,11 @@ An event object manages an internal flag that can be set to true with the floating point number specifying a timeout for the operation in seconds (or fractions thereof). - This method always returns ``None``. + This method returns the internal flag on exit, so it will always return + ``True`` except if a timeout is given and the operation times out. + + .. versionchanged:: 2.7 + Previously, the method always returned ``None``. .. _timer-objects: @@ -757,9 +792,9 @@ Currently, :class:`Lock`, :class:`RLock`, :class:`Condition`, Importing in threaded code -------------------------- -While the import machinery is thread safe, there are two key -restrictions on threaded imports due to inherent limitations in the way -that thread safety is provided: +While the import machinery is thread-safe, there are two key restrictions on +threaded imports due to inherent limitations in the way that thread-safety is +provided: * Firstly, other than in the main module, an import should not have the side effect of spawning a new thread and then waiting for that thread in diff --git a/Doc/library/time.rst b/Doc/library/time.rst index 1ffef2b..d6efcee 100644 --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -17,21 +17,23 @@ semantics of these functions varies among platforms. An explanation of some terminology and conventions is in order. - .. index:: single: epoch +.. index:: single: epoch * The :dfn:`epoch` is the point where the time starts. On January 1st of that year, at 0 hours, the "time since the epoch" is zero. For Unix, the epoch is 1970. To find out what the epoch is, look at ``gmtime(0)``. - .. index:: single: Year 2038 +.. index:: single: Year 2038 * The functions in this module do not handle dates and times before the epoch or far in the future. The cut-off point in the future is determined by the C library; for Unix, it is typically in 2038. - .. index:: - single: Year 2000 - single: Y2K +.. index:: + single: Year 2000 + single: Y2K + +.. _time-y2kissues: * **Year 2000 (Y2K) issues**: Python depends on the platform's C library, which generally doesn't have year 2000 issues, since all dates and times are @@ -48,16 +50,16 @@ An explanation of some terminology and conventions is in order. Note that this is new as of Python 1.5.2(a2); earlier versions, up to Python 1.5.1 and 1.5.2a1, would add 1900 to year values below 1900. - .. index:: - single: UTC - single: Coordinated Universal Time - single: Greenwich Mean Time +.. index:: + single: UTC + single: Coordinated Universal Time + single: Greenwich Mean Time * UTC is Coordinated Universal Time (formerly known as Greenwich Mean Time, or GMT). The acronym UTC is not a mistake but a compromise between English and French. - .. index:: single: Daylight Saving Time +.. index:: single: Daylight Saving Time * DST is Daylight Saving Time, an adjustment of the timezone by (usually) one hour during part of the year. DST rules are magic (determined by local law) and @@ -82,37 +84,7 @@ An explanation of some terminology and conventions is in order. values of :func:`gmtime`, :func:`localtime`, and :func:`strptime` also offer attribute names for individual fields. - +-------+-------------------+---------------------------------+ - | Index | Attribute | Values | - +=======+===================+=================================+ - | 0 | :attr:`tm_year` | (for example, 1993) | - +-------+-------------------+---------------------------------+ - | 1 | :attr:`tm_mon` | range [1,12] | - +-------+-------------------+---------------------------------+ - | 2 | :attr:`tm_mday` | range [1,31] | - +-------+-------------------+---------------------------------+ - | 3 | :attr:`tm_hour` | range [0,23] | - +-------+-------------------+---------------------------------+ - | 4 | :attr:`tm_min` | range [0,59] | - +-------+-------------------+---------------------------------+ - | 5 | :attr:`tm_sec` | range [0,61]; see **(1)** in | - | | | :func:`strftime` description | - +-------+-------------------+---------------------------------+ - | 6 | :attr:`tm_wday` | range [0,6], Monday is 0 | - +-------+-------------------+---------------------------------+ - | 7 | :attr:`tm_yday` | range [1,366] | - +-------+-------------------+---------------------------------+ - | 8 | :attr:`tm_isdst` | 0, 1 or -1; see below | - +-------+-------------------+---------------------------------+ - - Note that unlike the C structure, the month value is a range of 1-12, not 0-11. - A year value will be handled as described under "Year 2000 (Y2K) issues" above. - A ``-1`` argument as the daylight savings flag, passed to :func:`mktime` will - usually result in the correct daylight savings state to be filled in. - - When a tuple with an incorrect length is passed to a function expecting a - :class:`struct_time`, or having elements of the wrong type, a :exc:`TypeError` - is raised. + See :class:`struct_time` for a description of these objects. .. versionchanged:: 2.2 The time value sequence was changed from a tuple to a :class:`struct_time`, with @@ -418,13 +390,48 @@ The module defines the following functions and data items: documented as supported. -.. data:: struct_time +.. class:: struct_time The type of the time value sequence returned by :func:`gmtime`, - :func:`localtime`, and :func:`strptime`. + :func:`localtime`, and :func:`strptime`. It is an object with a :term:`named + tuple` interface: values can be accessed by index and by attribute name. The + following values are present: + + +-------+-------------------+---------------------------------+ + | Index | Attribute | Values | + +=======+===================+=================================+ + | 0 | :attr:`tm_year` | (for example, 1993) | + +-------+-------------------+---------------------------------+ + | 1 | :attr:`tm_mon` | range [1, 12] | + +-------+-------------------+---------------------------------+ + | 2 | :attr:`tm_mday` | range [1, 31] | + +-------+-------------------+---------------------------------+ + | 3 | :attr:`tm_hour` | range [0, 23] | + +-------+-------------------+---------------------------------+ + | 4 | :attr:`tm_min` | range [0, 59] | + +-------+-------------------+---------------------------------+ + | 5 | :attr:`tm_sec` | range [0, 61]; see **(1)** in | + | | | :func:`strftime` description | + +-------+-------------------+---------------------------------+ + | 6 | :attr:`tm_wday` | range [0, 6], Monday is 0 | + +-------+-------------------+---------------------------------+ + | 7 | :attr:`tm_yday` | range [1, 366] | + +-------+-------------------+---------------------------------+ + | 8 | :attr:`tm_isdst` | 0, 1 or -1; see below | + +-------+-------------------+---------------------------------+ .. versionadded:: 2.2 + Note that unlike the C structure, the month value is a range of [1, 12], not + [0, 11]. A year value will be handled as described under :ref:`Year 2000 + (Y2K) issues <time-y2kissues>` above. A ``-1`` argument as the daylight + savings flag, passed to :func:`mktime` will usually result in the correct + daylight savings state to be filled in. + + When a tuple with an incorrect length is passed to a function expecting a + :class:`struct_time`, or having elements of the wrong type, a + :exc:`TypeError` is raised. + .. function:: time() diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst index 4f930b3..29a941c 100644 --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -125,27 +125,36 @@ When called as a program from the command line, the following form is used:: python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...] -where the following options are understood: +Where the following options are understood: + +.. program:: timeit + +.. cmdoption:: -n N, --number=N --n N/:option:`--number=N` how many times to execute 'statement' --r N/:option:`--repeat=N` +.. cmdoption:: -r N, --repeat=N + how many times to repeat the timer (default 3) --s S/:option:`--setup=S` - statement to be executed once initially (default ``'pass'``) +.. cmdoption:: -s S, --setup=S + + statement to be executed once initially (default ``pass``) + +.. cmdoption:: -t, --time --t/:option:`--time` use :func:`time.time` (default on all platforms but Windows) --c/:option:`--clock` +.. cmdoption:: -c, --clock + use :func:`time.clock` (default on Windows) --v/:option:`--verbose` +.. cmdoption:: -v, --verbose + print raw timing results; repeat for more digits precision --h/:option:`--help` +.. cmdoption:: -h, --help + print a short usage message and exit A multi-line statement may be given by specifying each line as a separate diff --git a/Doc/library/tk.rst b/Doc/library/tk.rst index e1c25b6..f04c3ec 100644 --- a/Doc/library/tk.rst +++ b/Doc/library/tk.rst @@ -12,7 +12,8 @@ Graphical User Interfaces with Tk Tk/Tcl has long been an integral part of Python. It provides a robust and platform independent windowing toolkit, that is available to Python programmers -using the :mod:`Tkinter` module, and its extension, the :mod:`Tix` module. +using the :mod:`Tkinter` module, and its extensions, the :mod:`Tix` and +the :mod:`ttk` modules. The :mod:`Tkinter` module is a thin object-oriented layer on top of Tcl/Tk. To use :mod:`Tkinter`, you don't need to write Tcl code, but you will need to @@ -32,6 +33,7 @@ alternatives, see the :ref:`other-gui-packages` section. .. toctree:: tkinter.rst + ttk.rst tix.rst scrolledtext.rst turtle.rst diff --git a/Doc/library/token.rst b/Doc/library/token.rst index 5bf0ea8..00972b0 100644 --- a/Doc/library/token.rst +++ b/Doc/library/token.rst @@ -13,8 +13,8 @@ in the Python distribution for the definitions of the names in the context of the language grammar. The specific numeric values which the names map to may change between Python versions. -This module also provides one data object and some functions. The functions -mirror definitions in the Python C header files. +The module also provides a mapping from numeric codes to names and some +functions. The functions mirror definitions in the Python C header files. .. data:: tok_name @@ -39,6 +39,65 @@ mirror definitions in the Python C header files. Return true if *x* is the marker indicating the end of input. +The token constants are: + +.. data:: ENDMARKER + NAME + NUMBER + STRING + NEWLINE + INDENT + DEDENT + LPAR + RPAR + LSQB + RSQB + COLON + COMMA + SEMI + PLUS + MINUS + STAR + SLASH + VBAR + AMPER + LESS + GREATER + EQUAL + DOT + PERCENT + BACKQUOTE + LBRACE + RBRACE + EQEQUAL + NOTEQUAL + LESSEQUAL + GREATEREQUAL + TILDE + CIRCUMFLEX + LEFTSHIFT + RIGHTSHIFT + DOUBLESTAR + PLUSEQUAL + MINEQUAL + STAREQUAL + SLASHEQUAL + PERCENTEQUAL + AMPEREQUAL + VBAREQUAL + CIRCUMFLEXEQUAL + LEFTSHIFTEQUAL + RIGHTSHIFTEQUAL + DOUBLESTAREQUAL + DOUBLESLASH + DOUBLESLASHEQUAL + AT + OP + ERRORTOKEN + N_TOKENS + NT_OFFSET + + .. seealso:: Module :mod:`parser` diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 36f5838..2a7dea1 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -13,6 +13,11 @@ implemented in Python. The scanner in this module returns comments as tokens as well, making it useful for implementing "pretty-printers," including colorizers for on-screen displays. +.. seealso:: + + Latest version of the `tokenize module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/tokenize.py?view=markup>`_ + The primary entry point is a :term:`generator`: .. function:: generate_tokens(readline) diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst index 303cb1e..2e9d2b4 100644 --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -1,3 +1,4 @@ + :mod:`trace` --- Trace or track Python statement execution ========================================================== @@ -10,10 +11,14 @@ annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line. +.. seealso:: + + Latest version of the `trace module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/trace.py?view=markup>`_ .. _trace-cli: -Command Line Usage +Command-Line Usage ------------------ The :mod:`trace` module can be invoked from the command line. It can be as @@ -21,82 +26,89 @@ simple as :: python -m trace --count -C . somefile.py ... -The above will execute :file:`somefile.py` and generate annotated listings of all -Python modules imported during the execution into the current directory. +The above will execute :file:`somefile.py` and generate annotated listings of +all Python modules imported during the execution into the current directory. -Meta-options -^^^^^^^^^^^^ +.. program:: trace -``--help`` +.. cmdoption:: --help Display usage and exit. -``--version`` +.. cmdoption:: --version Display the version of the module and exit. Main options ^^^^^^^^^^^^ -The ``--listfuncs`` option is mutually exclusive with the ``--trace`` and -``--count`` options . When ``--listfuncs`` is provided, neither ``--counts`` -nor ``--trace`` are accepted, and vice versa. +At least one of the following options must be specified when invoking +:mod:`trace`. The :option:`--listfuncs <-l>` option is mutually exclusive with +the :option:`--trace <-t>` and :option:`--counts <-c>` options . When +:option:`--listfuncs <-l>` is provided, neither :option:`--counts <-c>` nor +:option:`--trace <-t>` are accepted, and vice versa. -``--count, -c`` +.. program:: trace + +.. cmdoption:: -c, --count Produce a set of annotated listing files upon program completion that shows - how many times each statement was executed. - See also ``--coverdir``, ``--file``, ``--no-report`` below. + how many times each statement was executed. See also + :option:`--coverdir <-C>`, :option:`--file <-f>` and + :option:`--no-report <-R>` below. -``--trace, -t`` +.. cmdoption:: -t, --trace Display lines as they are executed. -``--listfuncs, -l`` +.. cmdoption:: -l, --listfuncs Display the functions executed by running the program. -``--report, -r`` +.. cmdoption:: -r, --report - Produce an annotated list from an earlier program run that used the ``--count`` - and ``--file`` option. Do not execute any code. + Produce an annotated list from an earlier program run that used the + :option:`--count <-c>` and :option:`--file <-f>` option. This does not + execute any code. -``--trackcalls, -T`` +.. cmdoption:: -T, --trackcalls Display the calling relationships exposed by running the program. Modifiers ^^^^^^^^^ -``--file=<file>, -f`` +.. program:: trace + +.. cmdoption:: -f, --file=<file> - Name of a file to accumulate counts over several tracing runs. Should be used - with the ``--count`` option. + Name of a file to accumulate counts over several tracing runs. Should be + used with the :option:`--count <-c>` option. -``--coverdir=<dir>, -C`` +.. cmdoption:: -C, --coverdir=<dir> - Directory where the report files go. The coverage report for - ``package.module`` is written to file ``dir/package/module.cover``. + Directory where the report files go. The coverage report for + ``package.module`` is written to file :file:`{dir}/{package}/{module}.cover`. -``--missing, -m`` +.. cmdoption:: -m, --missing When generating annotated listings, mark lines which were not executed with - '``>>>>>>``'. + ``>>>>>>``. -``--summary, -s`` +.. cmdoption:: -s, --summary - When using ``--count`` or ``--report``, write a brief summary to - stdout for each file processed. + When using :option:`--count <-c>` or :option:`--report <-r>`, write a brief + summary to stdout for each file processed. -``--no-report, -R`` +.. cmdoption:: -R, --no-report Do not generate annotated listings. This is useful if you intend to make - several runs with ``--count`` then produce a single set of annotated - listings at the end. + several runs with :option:`--count <-c>`, and then produce a single set of + annotated listings at the end. -``--timing, -g`` +.. cmdoption:: -g, --timing - Prefix each line with the time since the program started. Only used while + Prefix each line with the time since the program started. Only used while tracing. Filters @@ -104,78 +116,79 @@ Filters These options may be repeated multiple times. -``--ignore-module=<mod>`` +.. program:: trace - Accepts comma separated list of module names. Ignore each of the named - modules and its submodules (if it is a package). +.. cmdoption:: --ignore-module=<mod> -``--ignore-dir=<dir>`` + Ignore each of the given module names and its submodules (if it is a + package). The argument can be a list of names separated by a comma. - Ignore all modules and packages in the named directory and subdirectories - (multiple directories can be joined by ``os.pathsep``). +.. cmdoption:: --ignore-dir=<dir> -.. _trace-api: + Ignore all modules and packages in the named directory and subdirectories. + The argument can be a list of directories separated by :data:`os.pathsep`. -Programming Interface ---------------------- - - -.. class:: Trace(count=1, trace=1, countfuncs=0, countcallers=0, ignoremods=(), ignoredirs=(), infile=None, outfile=None, timing=False) - - Create an object to trace execution of a single statement or expression. All - parameters are optional. *count* enables counting of line numbers. *trace* - enables line execution tracing. *countfuncs* enables listing of the functions - called during the run. *countcallers* enables call relationship tracking. - *ignoremods* is a list of modules or packages to ignore. *ignoredirs* is a list - of directories whose modules or packages should be ignored. *infile* is the - name of the file from which to read stored count information. *outfile* is - the name of the file in which to write updated count information. *timing* - enables a timestamp relative to when tracing was started to be displayed. +.. _trace-api: +Programmatic Interface +---------------------- -.. method:: Trace.run(cmd) +.. class:: Trace([count=1[, trace=1[, countfuncs=0[, countcallers=0[, ignoremods=()[, ignoredirs=()[, infile=None[, outfile=None[, timing=False]]]]]]]]]) - Run *cmd* under control of the :class:`Trace` object with the current tracing parameters. - *cmd* must be a string or code object, suitable for passing into :func:`exec`. + Create an object to trace execution of a single statement or expression. All + parameters are optional. *count* enables counting of line numbers. *trace* + enables line execution tracing. *countfuncs* enables listing of the + functions called during the run. *countcallers* enables call relationship + tracking. *ignoremods* is a list of modules or packages to ignore. + *ignoredirs* is a list of directories whose modules or packages should be + ignored. *infile* is the name of the file from which to read stored count + information. *outfile* is the name of the file in which to write updated + count information. *timing* enables a timestamp relative to when tracing was + started to be displayed. + .. method:: run(cmd) -.. method:: Trace.runctx(cmd, globals=None, locals=None) + Execute the command and gather statistics from the execution with + the current tracing parameters. *cmd* must be a string or code object, + suitable for passing into :func:`exec`. - Run *cmd* under control of the :class:`Trace` object with the current tracing parameters - in the defined global and local environments. If not defined, *globals* and - *locals* default to empty dictionaries. + .. method:: runctx(cmd[, globals=None[, locals=None]]) + Execute the command and gather statistics from the execution with the + current tracing parameters, in the defined global and local + environments. If not defined, *globals* and *locals* default to empty + dictionaries. -.. method:: Trace.runfunc(func, *args, **kwds) + .. method:: runfunc(func, *args, **kwds) - Call *func* with the given arguments under control of the :class:`Trace` object - with the current tracing parameters. + Call *func* with the given arguments under control of the :class:`Trace` + object with the current tracing parameters. -.. method:: Trace.results() + .. method:: results() - Return a :class:`CoverageResults` object that contains the cumulative results - of all previous calls to ``run``, ``runctx`` and ``runfunc`` for the given - :class:`Trace` instance. Does not reset the accumulated trace results. + Return a :class:`CoverageResults` object that contains the cumulative + results of all previous calls to ``run``, ``runctx`` and ``runfunc`` + for the given :class:`Trace` instance. Does not reset the accumulated + trace results. .. class:: CoverageResults - A container for coverage results, created by :meth:`Trace.results`. Should not - be created directly by the user. - -.. method:: CoverageResults.update(other) + A container for coverage results, created by :meth:`Trace.results`. Should + not be created directly by the user. - Merge in data from another :class:`CoverageResults` object. + .. method:: update(other) -.. method:: CoverageResults.write_results(show_missing=True, summary=False, coverdir=None) + Merge in data from another :class:`CoverageResults` object. - Write coverage results. Set *show_missing* to show lines that had no hits. - Set *summary* to include in the output the coverage summary per module. *coverdir* - specifies the directory into which the coverage result files will be output. - If ``None``, the results for each source file are placed in its directory. + .. method:: write_results([show_missing=True[, summary=False[, coverdir=None]]]) -.. + Write coverage results. Set *show_missing* to show lines that had no + hits. Set *summary* to include in the output the coverage summary per + module. *coverdir* specifies the directory into which the coverage + result files will be output. If ``None``, the results for each source + file are placed in its directory. -A simple example demonstrating the use of the programming interface:: +A simple example demonstrating the use of the programmatic interface:: import sys import trace diff --git a/Doc/library/ttk.rst b/Doc/library/ttk.rst new file mode 100644 index 0000000..ebfdede --- /dev/null +++ b/Doc/library/ttk.rst @@ -0,0 +1,1407 @@ +:mod:`ttk` --- Tk themed widgets +================================ + +.. module:: ttk + :synopsis: Tk themed widget set +.. sectionauthor:: Guilherme Polo <ggpolo@gmail.com> + + +.. index:: single: ttk + +The :mod:`ttk` module provides access to the Tk themed widget set, which has +been introduced in Tk 8.5. If Python is not compiled against Tk 8.5 code may +still use this module as long as Tile is installed. However, some features +provided by the new Tk, like anti-aliased font rendering under X11, window +transparency (on X11 you will need a composition window manager) will be +missing. + +The basic idea of :mod:`ttk` is to separate, to the extent possible, the code +implementing a widget's behavior from the code implementing its appearance. + + +.. seealso:: + + `Tk Widget Styling Support <http://www.tcl.tk/cgi-bin/tct/tip/48>`_ + The document which brought up theming support for Tk + + +Using Ttk +--------- + +To start using Ttk, import its module:: + + import ttk + +But code like this:: + + from Tkinter import * + +may optionally want to use this:: + + from Tkinter import * + from ttk import * + +And then several :mod:`ttk` widgets (:class:`Button`, :class:`Checkbutton`, +:class:`Entry`, :class:`Frame`, :class:`Label`, :class:`LabelFrame`, +:class:`Menubutton`, :class:`PanedWindow`, :class:`Radiobutton`, :class:`Scale` +and :class:`Scrollbar`) will automatically substitute for the Tk widgets. + +This has the direct benefit of using the new widgets, giving better look & feel +across platforms, but be aware that they are not totally compatible. The main +difference is that widget options such as "fg", "bg" and others related to +widget styling are no longer present in Ttk widgets. Use :class:`ttk.Style` to +achieve the same (or better) styling. + +.. seealso:: + + `Converting existing applications to use the Tile widgets <http://tktable.sourceforge.net/tile/doc/converting.txt>`_ + A text which talks in Tcl terms about differences typically found when + converting applications to use the new widgets. + + +Ttk Widgets +----------- + +Ttk comes with 17 widgets, 11 of which already exist in Tkinter: +:class:`Button`, :class:`Checkbutton`, :class:`Entry`, :class:`Frame`, +:class:`Label`, :class:`LabelFrame`, :class:`Menubutton`, +:class:`PanedWindow`, :class:`Radiobutton`, :class:`Scale` and +:class:`Scrollbar`. The 6 new widget classes are: :class:`Combobox`, +:class:`Notebook`, :class:`Progressbar`, :class:`Separator`, +:class:`Sizegrip` and :class:`Treeview`. All of these classes are +subclasses of :class:`Widget`. + +As said previously, you will notice changes in look-and-feel as well in the +styling code. To demonstrate the latter, a very simple example is shown below. + +Tk code:: + + l1 = Tkinter.Label(text="Test", fg="black", bg="white") + l2 = Tkinter.Label(text="Test", fg="black", bg="white") + + +Corresponding Ttk code:: + + style = ttk.Style() + style.configure("BW.TLabel", foreground="black", background="white") + + l1 = ttk.Label(text="Test", style="BW.TLabel") + l2 = ttk.Label(text="Test", style="BW.TLabel") + +For more information about TtkStyling_ read the :class:`Style` class +documentation. + +Widget +------ + +:class:`ttk.Widget` defines standard options and methods supported by Tk +themed widgets and is not supposed to be directly instantiated. + + +Standard Options +^^^^^^^^^^^^^^^^ + +All the :mod:`ttk` widgets accept the following options: + + +-----------+--------------------------------------------------------------+ + | Option | Description | + +===========+==============================================================+ + | class | Specifies the window class. The class is used when querying | + | | the option database for the window's other options, to | + | | determine the default bindtags for the window, and to select | + | | the widget's default layout and style. This is a read-only | + | | option which may only be specified when the window is | + | | created. | + +-----------+--------------------------------------------------------------+ + | cursor | Specifies the mouse cursor to be used for the widget. If set | + | | to the empty string (the default), the cursor is inherited | + | | from the parent widget. | + +-----------+--------------------------------------------------------------+ + | takefocus | Determines whether the window accepts the focus during | + | | keyboard traversal. 0, 1 or an empty string is returned. | + | | If 0, the window should be skipped entirely | + | | during keyboard traversal. If 1, the window | + | | should receive the input focus as long as it is viewable. | + | | An empty string means that the traversal scripts make the | + | | decision about whether or not to focus on the window. | + +-----------+--------------------------------------------------------------+ + | style | May be used to specify a custom widget style. | + +-----------+--------------------------------------------------------------+ + + +Scrollable Widget Options +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following options are supported by widgets that are controlled by a +scrollbar. + + +----------------+---------------------------------------------------------+ + | option | description | + +================+=========================================================+ + | xscrollcommand | Used to communicate with horizontal scrollbars. | + | | | + | | When the view in the widget's window changes, the widget| + | | will generate a Tcl command based on the scrollcommand. | + | | | + | | Usually this option consists of the | + | | :meth:`Scrollbar.set` method of some scrollbar. This | + | | will cause | + | | the scrollbar to be updated whenever the view in the | + | | window changes. | + +----------------+---------------------------------------------------------+ + | yscrollcommand | Used to communicate with vertical scrollbars. | + | | For more information, see above. | + +----------------+---------------------------------------------------------+ + + +Label Options +^^^^^^^^^^^^^ + +The following options are supported by labels, buttons and other button-like +widgets. + +.. tabularcolumns:: |p{0.2\textwidth}|p{0.7\textwidth}| +.. + + +--------------+-----------------------------------------------------------+ + | option | description | + +==============+===========================================================+ + | text | Specifies a text string to be displayed inside the widget.| + +--------------+-----------------------------------------------------------+ + | textvariable | Specifies a name whose value will be used in place of the | + | | text option resource. | + +--------------+-----------------------------------------------------------+ + | underline | If set, specifies the index (0-based) of a character to | + | | underline in the text string. The underline character is | + | | used for mnemonic activation. | + +--------------+-----------------------------------------------------------+ + | image | Specifies an image to display. This is a list of 1 or more| + | | elements. The first element is the default image name. The| + | | rest of the list is a sequence of statespec/value pairs as| + | | defined by :meth:`Style.map`, specifying different images | + | | to use when the widget is in a particular state or a | + | | combination of states. All images in the list should have | + | | the same size. | + +--------------+-----------------------------------------------------------+ + | compound | Specifies how to display the image relative to the text, | + | | in the case both text and image options are present. | + | | Valid values are: | + | | | + | | * text: display text only | + | | * image: display image only | + | | * top, bottom, left, right: display image above, below, | + | | left of, or right of the text, respectively. | + | | * none: the default. display the image if present, | + | | otherwise the text. | + +--------------+-----------------------------------------------------------+ + | width | If greater than zero, specifies how much space, in | + | | character widths, to allocate for the text label; if less | + | | than zero, specifies a minimum width. If zero or | + | | unspecified, the natural width of the text label is used. | + +--------------+-----------------------------------------------------------+ + + +Compatibility Options +^^^^^^^^^^^^^^^^^^^^^ + + +--------+----------------------------------------------------------------+ + | option | description | + +========+================================================================+ + | state | May be set to "normal" or "disabled" to control the "disabled" | + | | state bit. This is a write-only option: setting it changes the | + | | widget state, but the :meth:`Widget.state` method does not | + | | affect this option. | + +--------+----------------------------------------------------------------+ + +Widget States +^^^^^^^^^^^^^ + +The widget state is a bitmap of independent state flags. + + +------------+-------------------------------------------------------------+ + | flag | description | + +============+=============================================================+ + | active | The mouse cursor is over the widget and pressing a mouse | + | | button will cause some action to occur. | + +------------+-------------------------------------------------------------+ + | disabled | Widget is disabled under program control. | + +------------+-------------------------------------------------------------+ + | focus | Widget has keyboard focus. | + +------------+-------------------------------------------------------------+ + | pressed | Widget is being pressed. | + +------------+-------------------------------------------------------------+ + | selected | "On", "true", or "current" for things like Checkbuttons and | + | | radiobuttons. | + +------------+-------------------------------------------------------------+ + | background | Windows and Mac have a notion of an "active" or foreground | + | | window. The *background* state is set for widgets in a | + | | background window, and cleared for those in the foreground | + | | window. | + +------------+-------------------------------------------------------------+ + | readonly | Widget should not allow user modification. | + +------------+-------------------------------------------------------------+ + | alternate | A widget-specific alternate display format. | + +------------+-------------------------------------------------------------+ + | invalid | The widget's value is invalid. | + +------------+-------------------------------------------------------------+ + +A state specification is a sequence of state names, optionally prefixed with +an exclamation point indicating that the bit is off. + + +ttk.Widget +^^^^^^^^^^ + +Besides the methods described below, the :class:`ttk.Widget` class supports the +:meth:`Tkinter.Widget.cget` and :meth:`Tkinter.Widget.configure` methods. + +.. class:: Widget + + .. method:: identify(x, y) + + Returns the name of the element at position *x* *y*, or the empty string + if the point does not lie within any element. + + *x* and *y* are pixel coordinates relative to the widget. + + + .. method:: instate(statespec[, callback=None[, *args[, **kw]]]) + + Test the widget's state. If a callback is not specified, returns True + if the widget state matches *statespec* and False otherwise. If callback + is specified then it is called with *args* if widget state matches + *statespec*. + + + .. method:: state([statespec=None]) + + Modify or read widget state. If *statespec* is specified, sets the + widget state accordingly and returns a new *statespec* indicating + which flags were changed. If *statespec* is not specified, returns + the currently-enabled state flags. + + *statespec* will usually be a list or a tuple. + + +Combobox +-------- + +The :class:`ttk.Combobox` widget combines a text field with a pop-down list of +values. This widget is a subclass of :class:`Entry`. + +Besides the methods inherited from :class:`Widget` (:meth:`Widget.cget`, +:meth:`Widget.configure`, :meth:`Widget.identify`, :meth:`Widget.instate` +and :meth:`Widget.state`) and those inherited from :class:`Entry` +(:meth:`Entry.bbox`, :meth:`Entry.delete`, :meth:`Entry.icursor`, +:meth:`Entry.index`, :meth:`Entry.inset`, :meth:`Entry.selection`, +:meth:`Entry.xview`), this class has some other methods, described at +:class:`ttk.Combobox`. + + +Options +^^^^^^^ + +This widget accepts the following options: + + +-----------------+--------------------------------------------------------+ + | option | description | + +=================+========================================================+ + | exportselection | Boolean value. If set, the widget selection is linked | + | | to the Window Manager selection (which can be returned | + | | by invoking :meth:`Misc.selection_get`, for example). | + +-----------------+--------------------------------------------------------+ + | justify | Specifies how the text is aligned within the widget. | + | | One of "left", "center", or "right". | + +-----------------+--------------------------------------------------------+ + | height | Specifies the height of the pop-down listbox, in rows. | + +-----------------+--------------------------------------------------------+ + | postcommand | A script (possibly registered with | + | | :meth:`Misc.register`) that | + | | is called immediately before displaying the values. It | + | | may specify which values to display. | + +-----------------+--------------------------------------------------------+ + | state | One of "normal", "readonly", or "disabled". In the | + | | "readonly" state, the value may not be edited directly,| + | | and the user can only select one of the values from the| + | | dropdown list. In the "normal" state, the text field is| + | | directly editable. In the "disabled" state, no | + | | interaction is possible. | + +-----------------+--------------------------------------------------------+ + | textvariable | Specifies a name whose value is linked to the widget | + | | value. Whenever the value associated with that name | + | | changes, the widget value is updated, and vice versa. | + | | See :class:`Tkinter.StringVar`. | + +-----------------+--------------------------------------------------------+ + | values | Specifies the list of values to display in the | + | | drop-down listbox. | + +-----------------+--------------------------------------------------------+ + | width | Specifies an integer value indicating the desired width| + | | of the entry window, in average-size characters of the | + | | widget's font. | + +-----------------+--------------------------------------------------------+ + + +Virtual events +^^^^^^^^^^^^^^ + +The combobox widget generates a **<<ComboboxSelected>>** virtual event +when the user selects an element from the list of values. + + +ttk.Combobox +^^^^^^^^^^^^ + +.. class:: Combobox + + .. method:: current([newindex=None]) + + If *newindex* is specified, sets the combobox value to the element + position *newindex*. Otherwise, returns the index of the current value or + -1 if the current value is not in the values list. + + + .. method:: get() + + Returns the current value of the combobox. + + + .. method:: set(value) + + Sets the value of the combobox to *value*. + + +Notebook +-------- + +The Ttk Notebook widget manages a collection of windows and displays a single +one at a time. Each child window is associated with a tab, which the user +may select to change the currently-displayed window. + + +Options +^^^^^^^ + +This widget accepts the following specific options: + + +---------+----------------------------------------------------------------+ + | option | description | + +=========+================================================================+ + | height | If present and greater than zero, specifies the desired height | + | | of the pane area (not including internal padding or tabs). | + | | Otherwise, the maximum height of all panes is used. | + +---------+----------------------------------------------------------------+ + | padding | Specifies the amount of extra space to add around the outside | + | | of the notebook. The padding is a list of up to four length | + | | specifications: left top right bottom. If fewer than four | + | | elements are specified, bottom defaults to top, right defaults | + | | to left, and top defaults to left. | + +---------+----------------------------------------------------------------+ + | width | If present and greater than zero, specifies the desired width | + | | of the pane area (not including internal padding). Otherwise, | + | | the maximum width of all panes is used. | + +---------+----------------------------------------------------------------+ + + +Tab Options +^^^^^^^^^^^ + +There are also specific options for tabs: + + +-----------+--------------------------------------------------------------+ + | option | description | + +===========+==============================================================+ + | state | Either "normal", "disabled" or "hidden". If "disabled", then | + | | the tab is not selectable. If "hidden", then the tab is not | + | | shown. | + +-----------+--------------------------------------------------------------+ + | sticky | Specifies how the child window is positioned within the pane | + | | area. Value is a string containing zero or more of the | + | | characters "n", "s", "e" or "w". Each letter refers to a | + | | side (north, south, east or west) that the child window will | + | | stick to, as per the :meth:`grid` geometry manager. | + +-----------+--------------------------------------------------------------+ + | padding | Specifies the amount of extra space to add between the | + | | notebook and this pane. Syntax is the same as for the option | + | | padding used by this widget. | + +-----------+--------------------------------------------------------------+ + | text | Specifies a text to be displayed in the tab. | + +-----------+--------------------------------------------------------------+ + | image | Specifies an image to display in the tab. See the option | + | | image described in :class:`Widget`. | + +-----------+--------------------------------------------------------------+ + | compound | Specifies how to display the image relative to the text, in | + | | the case both text and image options are present. See | + | | `Label Options`_ for legal values. | + +-----------+--------------------------------------------------------------+ + | underline | Specifies the index (0-based) of a character to underline in | + | | the text string. The underlined character is used for | + | | mnemonic activation if :meth:`Notebook.enable_traversal` is | + | | called. | + +-----------+--------------------------------------------------------------+ + + +Tab Identifiers +^^^^^^^^^^^^^^^ + +The *tab_id* present in several methods of :class:`ttk.Notebook` may take any +of the following forms: + +* An integer between zero and the number of tabs. +* The name of a child window. +* A positional specification of the form "@x,y", which identifies the tab. +* The literal string "current", which identifies the currently-selected tab. +* The literal string "end", which returns the number of tabs (only valid for + :meth:`Notebook.index`). + + +Virtual Events +^^^^^^^^^^^^^^ + +This widget generates a **<<NotebookTabChanged>>** virtual event after a new +tab is selected. + + +ttk.Notebook +^^^^^^^^^^^^ + +.. class:: Notebook + + .. method:: add(child, **kw) + + Adds a new tab to the notebook. + + If window is currently managed by the notebook but hidden, it is + restored to its previous position. + + See `Tab Options`_ for the list of available options. + + + .. method:: forget(tab_id) + + Removes the tab specified by *tab_id*, unmaps and unmanages the + associated window. + + + .. method:: hide(tab_id) + + Hides the tab specified by *tab_id*. + + The tab will not be displayed, but the associated window remains + managed by the notebook and its configuration remembered. Hidden tabs + may be restored with the :meth:`add` command. + + + .. method:: identify(x, y) + + Returns the name of the tab element at position *x*, *y*, or the empty + string if none. + + + .. method:: index(tab_id) + + Returns the numeric index of the tab specified by *tab_id*, or the total + number of tabs if *tab_id* is the string "end". + + + .. method:: insert(pos, child, **kw) + + Inserts a pane at the specified position. + + *pos* is either the string "end", an integer index, or the name of a + managed child. If *child* is already managed by the notebook, moves it to + the specified position. + + See `Tab Options`_ for the list of available options. + + + .. method:: select([tab_id]) + + Selects the specified *tab_id*. + + The associated child window will be displayed, and the + previously-selected window (if different) is unmapped. If *tab_id* is + omitted, returns the widget name of the currently selected pane. + + + .. method:: tab(tab_id[, option=None[, **kw]]) + + Query or modify the options of the specific *tab_id*. + + If *kw* is not given, returns a dictionary of the tab option values. If + *option* is specified, returns the value of that *option*. Otherwise, + sets the options to the corresponding values. + + + .. method:: tabs() + + Returns a list of windows managed by the notebook. + + + .. method:: enable_traversal() + + Enable keyboard traversal for a toplevel window containing this notebook. + + This will extend the bindings for the toplevel window containing the + notebook as follows: + + * Control-Tab: selects the tab following the currently selected one. + * Shift-Control-Tab: selects the tab preceding the currently selected one. + * Alt-K: where K is the mnemonic (underlined) character of any tab, will + select that tab. + + Multiple notebooks in a single toplevel may be enabled for traversal, + including nested notebooks. However, notebook traversal only works + properly if all panes have the notebook they are in as master. + + +Progressbar +----------- + +The :class:`ttk.Progressbar` widget shows the status of a long-running +operation. It can operate in two modes: determinate mode shows the amount +completed relative to the total amount of work to be done, and indeterminate +mode provides an animated display to let the user know that something is +happening. + + +Options +^^^^^^^ + +This widget accepts the following specific options: + + +----------+---------------------------------------------------------------+ + | option | description | + +==========+===============================================================+ + | orient | One of "horizontal" or "vertical". Specifies the orientation | + | | of the progress bar. | + +----------+---------------------------------------------------------------+ + | length | Specifies the length of the long axis of the progress bar | + | | (width if horizontal, height if vertical). | + +----------+---------------------------------------------------------------+ + | mode | One of "determinate" or "indeterminate". | + +----------+---------------------------------------------------------------+ + | maximum | A number specifying the maximum value. Defaults to 100. | + +----------+---------------------------------------------------------------+ + | value | The current value of the progress bar. In "determinate" mode, | + | | this represents the amount of work completed. In | + | | "indeterminate" mode, it is interpreted as modulo *maximum*; | + | | that is, the progress bar completes one "cycle" when its value| + | | increases by *maximum*. | + +----------+---------------------------------------------------------------+ + | variable | A name which is linked to the option value. If specified, the | + | | value of the progress bar is automatically set to the value of| + | | this name whenever the latter is modified. | + +----------+---------------------------------------------------------------+ + | phase | Read-only option. The widget periodically increments the value| + | | of this option whenever its value is greater than 0 and, in | + | | determinate mode, less than maximum. This option may be used | + | | by the current theme to provide additional animation effects. | + +----------+---------------------------------------------------------------+ + + +ttk.Progressbar +^^^^^^^^^^^^^^^ + +.. class:: Progressbar + + .. method:: start([interval]) + + Begin autoincrement mode: schedules a recurring timer event that calls + :meth:`Progressbar.step` every *interval* milliseconds. If omitted, + *interval* defaults to 50 milliseconds. + + + .. method:: step([amount]) + + Increments the progress bar's value by *amount*. + + *amount* defaults to 1.0 if omitted. + + + .. method:: stop() + + Stop autoincrement mode: cancels any recurring timer event initiated by + :meth:`Progressbar.start` for this progress bar. + + +Separator +--------- + +The :class:`ttk.Separator` widget displays a horizontal or vertical separator +bar. + +It has no other methods besides the ones inherited from :class:`ttk.Widget`. + + +Options +^^^^^^^ + +This widget accepts the following specific option: + + +--------+----------------------------------------------------------------+ + | option | description | + +========+================================================================+ + | orient | One of "horizontal" or "vertical". Specifies the orientation of| + | | the separator. | + +--------+----------------------------------------------------------------+ + + +Sizegrip +-------- + +The :class:`ttk.Sizegrip` widget (also known as a grow box) allows the user to +resize the containing toplevel window by pressing and dragging the grip. + +This widget has neither specific options nor specific methods, besides the +ones inherited from :class:`ttk.Widget`. + + +Platform-specific notes +^^^^^^^^^^^^^^^^^^^^^^^ + +* On Mac OS X, toplevel windows automatically include a built-in size grip + by default. Adding a :class:`Sizegrip` is harmless, since the built-in + grip will just mask the widget. + + +Bugs +^^^^ + +* If the containing toplevel's position was specified relative to the right + or bottom of the screen (e.g. ....), the :class:`Sizegrip` widget will + not resize the window. +* This widget supports only "southeast" resizing. + + +Treeview +-------- + +The :class:`ttk.Treeview` widget displays a hierarchical collection of items. +Each item has a textual label, an optional image, and an optional list of data +values. The data values are displayed in successive columns after the tree +label. + +The order in which data values are displayed may be controlled by setting +the widget option ``displaycolumns``. The tree widget can also display column +headings. Columns may be accessed by number or symbolic names listed in the +widget option columns. See `Column Identifiers`_. + +Each item is identified by an unique name. The widget will generate item IDs +if they are not supplied by the caller. There is a distinguished root item, +named ``{}``. The root item itself is not displayed; its children appear at the +top level of the hierarchy. + +Each item also has a list of tags, which can be used to associate event bindings +with individual items and control the appearance of the item. + +The Treeview widget supports horizontal and vertical scrolling, according to +the options described in `Scrollable Widget Options`_ and the methods +:meth:`Treeview.xview` and :meth:`Treeview.yview`. + + +Options +^^^^^^^ + +This widget accepts the following specific options: + +.. tabularcolumns:: |p{0.2\textwidth}|p{0.7\textwidth}| +.. + + +----------------+--------------------------------------------------------+ + | option | description | + +================+========================================================+ + | columns | A list of column identifiers, specifying the number of | + | | columns and their names. | + +----------------+--------------------------------------------------------+ + | displaycolumns | A list of column identifiers (either symbolic or | + | | integer indices) specifying which data columns are | + | | displayed and the order in which they appear, or the | + | | string "#all". | + +----------------+--------------------------------------------------------+ + | height | Specifies the number of rows which should be visible. | + | | Note: the requested width is determined from the sum | + | | of the column widths. | + +----------------+--------------------------------------------------------+ + | padding | Specifies the internal padding for the widget. The | + | | padding is a list of up to four length specifications. | + +----------------+--------------------------------------------------------+ + | selectmode | Controls how the built-in class bindings manage the | + | | selection. One of "extended", "browse" or "none". | + | | If set to "extended" (the default), multiple items may | + | | be selected. If "browse", only a single item will be | + | | selected at a time. If "none", the selection will not | + | | be changed. | + | | | + | | Note that the application code and tag bindings can set| + | | the selection however they wish, regardless of the | + | | value of this option. | + +----------------+--------------------------------------------------------+ + | show | A list containing zero or more of the following values,| + | | specifying which elements of the tree to display. | + | | | + | | * tree: display tree labels in column #0. | + | | * headings: display the heading row. | + | | | + | | The default is "tree headings", i.e., show all | + | | elements. | + | | | + | | **Note**: Column #0 always refers to the tree column, | + | | even if show="tree" is not specified. | + +----------------+--------------------------------------------------------+ + + +Item Options +^^^^^^^^^^^^ + +The following item options may be specified for items in the insert and item +widget commands. + + +--------+---------------------------------------------------------------+ + | option | description | + +========+===============================================================+ + | text | The textual label to display for the item. | + +--------+---------------------------------------------------------------+ + | image | A Tk Image, displayed to the left of the label. | + +--------+---------------------------------------------------------------+ + | values | The list of values associated with the item. | + | | | + | | Each item should have the same number of values as the widget | + | | option columns. If there are fewer values than columns, the | + | | remaining values are assumed empty. If there are more values | + | | than columns, the extra values are ignored. | + +--------+---------------------------------------------------------------+ + | open | True/False value indicating whether the item's children should| + | | be displayed or hidden. | + +--------+---------------------------------------------------------------+ + | tags | A list of tags associated with this item. | + +--------+---------------------------------------------------------------+ + + +Tag Options +^^^^^^^^^^^ + +The following options may be specified on tags: + + +------------+-----------------------------------------------------------+ + | option | description | + +============+===========================================================+ + | foreground | Specifies the text foreground color. | + +------------+-----------------------------------------------------------+ + | background | Specifies the cell or item background color. | + +------------+-----------------------------------------------------------+ + | font | Specifies the font to use when drawing text. | + +------------+-----------------------------------------------------------+ + | image | Specifies the item image, in case the item's image option | + | | is empty. | + +------------+-----------------------------------------------------------+ + + +Column Identifiers +^^^^^^^^^^^^^^^^^^ + +Column identifiers take any of the following forms: + +* A symbolic name from the list of columns option. +* An integer n, specifying the nth data column. +* A string of the form #n, where n is an integer, specifying the nth display + column. + +Notes: + +* Item's option values may be displayed in a different order than the order + in which they are stored. +* Column #0 always refers to the tree column, even if show="tree" is not + specified. + +A data column number is an index into an item's option values list; a display +column number is the column number in the tree where the values are displayed. +Tree labels are displayed in column #0. If option displaycolumns is not set, +then data column n is displayed in column #n+1. Again, **column #0 always +refers to the tree column**. + + +Virtual Events +^^^^^^^^^^^^^^ + +The Treeview widget generates the following virtual events. + + +--------------------+--------------------------------------------------+ + | event | description | + +====================+==================================================+ + | <<TreeviewSelect>> | Generated whenever the selection changes. | + +--------------------+--------------------------------------------------+ + | <<TreeviewOpen>> | Generated just before settings the focus item to | + | | open=True. | + +--------------------+--------------------------------------------------+ + | <<TreeviewClose>> | Generated just after setting the focus item to | + | | open=False. | + +--------------------+--------------------------------------------------+ + +The :meth:`Treeview.focus` and :meth:`Treeview.selection` methods can be used +to determine the affected item or items. + + +ttk.Treeview +^^^^^^^^^^^^ + +.. class:: Treeview + + .. method:: bbox(item[, column=None]) + + Returns the bounding box (relative to the treeview widget's window) of + the specified *item* in the form (x, y, width, height). + + If *column* is specified, returns the bounding box of that cell. If the + *item* is not visible (i.e., if it is a descendant of a closed item or is + scrolled offscreen), returns an empty string. + + + .. method:: get_children([item]) + + Returns the list of children belonging to *item*. + + If *item* is not specified, returns root children. + + + .. method:: set_children(item, *newchildren) + + Replaces *item*'s child with *newchildren*. + + Children present in *item* that are not present in *newchildren* are + detached from the tree. No items in *newchildren* may be an ancestor of + *item*. Note that not specifying *newchildren* results in detaching + *item*'s children. + + + .. method:: column(column[, option=None[, **kw]]) + + Query or modify the options for the specified *column*. + + If *kw* is not given, returns a dict of the column option values. If + *option* is specified then the value for that *option* is returned. + Otherwise, sets the options to the corresponding values. + + The valid options/values are: + + * id + Returns the column name. This is a read-only option. + * anchor: One of the standard Tk anchor values. + Specifies how the text in this column should be aligned with respect + to the cell. + * minwidth: width + The minimum width of the column in pixels. The treeview widget will + not make the column any smaller than specified by this option when + the widget is resized or the user drags a column. + * stretch: True/False + Specifies whether the column's width should be adjusted when + the widget is resized. + * width: width + The width of the column in pixels. + + To configure the tree column, call this with column = "#0" + + .. method:: delete(*items) + + Delete all specified *items* and all their descendants. + + The root item may not be deleted. + + + .. method:: detach(*items) + + Unlinks all of the specified *items* from the tree. + + The items and all of their descendants are still present, and may be + reinserted at another point in the tree, but will not be displayed. + + The root item may not be detached. + + + .. method:: exists(item) + + Returns True if the specified *item* is present in the tree. + + + .. method:: focus([item=None]) + + If *item* is specified, sets the focus item to *item*. Otherwise, returns + the current focus item, or '' if there is none. + + + .. method:: heading(column[, option=None[, **kw]]) + + Query or modify the heading options for the specified *column*. + + If *kw* is not given, returns a dict of the heading option values. If + *option* is specified then the value for that *option* is returned. + Otherwise, sets the options to the corresponding values. + + The valid options/values are: + + * text: text + The text to display in the column heading. + * image: imageName + Specifies an image to display to the right of the column heading. + * anchor: anchor + Specifies how the heading text should be aligned. One of the standard + Tk anchor values. + * command: callback + A callback to be invoked when the heading label is pressed. + + To configure the tree column heading, call this with column = "#0". + + + .. method:: identify(component, x, y) + + Returns a description of the specified *component* under the point given + by *x* and *y*, or the empty string if no such *component* is present at + that position. + + + .. method:: identify_row(y) + + Returns the item ID of the item at position *y*. + + + .. method:: identify_column(x) + + Returns the data column identifier of the cell at position *x*. + + The tree column has ID #0. + + + .. method:: identify_region(x, y) + + Returns one of: + + +-----------+--------------------------------------+ + | region | meaning | + +===========+======================================+ + | heading | Tree heading area. | + +-----------+--------------------------------------+ + | separator | Space between two columns headings. | + +-----------+--------------------------------------+ + | tree | The tree area. | + +-----------+--------------------------------------+ + | cell | A data cell. | + +-----------+--------------------------------------+ + + Availability: Tk 8.6. + + + .. method:: identify_element(x, y) + + Returns the element at position *x*, *y*. + + Availability: Tk 8.6. + + + .. method:: index(item) + + Returns the integer index of *item* within its parent's list of children. + + + .. method:: insert(parent, index[, iid=None[, **kw]]) + + Creates a new item and returns the item identifier of the newly created + item. + + *parent* is the item ID of the parent item, or the empty string to create + a new top-level item. *index* is an integer, or the value "end", + specifying where in the list of parent's children to insert the new item. + If *index* is less than or equal to zero, the new node is inserted at + the beginning; if *index* is greater than or equal to the current number + of children, it is inserted at the end. If *iid* is specified, it is used + as the item identifier; *iid* must not already exist in the tree. + Otherwise, a new unique identifier is generated. + + See `Item Options`_ for the list of available points. + + + .. method:: item(item[, option[, **kw]]) + + Query or modify the options for the specified *item*. + + If no options are given, a dict with options/values for the item is + returned. + If *option* is specified then the value for that option is returned. + Otherwise, sets the options to the corresponding values as given by *kw*. + + + .. method:: move(item, parent, index) + + Moves *item* to position *index* in *parent*'s list of children. + + It is illegal to move an item under one of its descendants. If *index* is + less than or equal to zero, *item* is moved to the beginning; if greater + than or equal to the number of children, it is moved to the end. If *item* + was detached it is reattached. + + + .. method:: next(item) + + Returns the identifier of *item*'s next sibling, or '' if *item* is the + last child of its parent. + + + .. method:: parent(item) + + Returns the ID of the parent of *item*, or '' if *item* is at the top + level of the hierarchy. + + + .. method:: prev(item) + + Returns the identifier of *item*'s previous sibling, or '' if *item* is + the first child of its parent. + + + .. method:: reattach(item, parent, index) + + An alias for :meth:`Treeview.move`. + + + .. method:: see(item) + + Ensure that *item* is visible. + + Sets all of *item*'s ancestors open option to True, and scrolls the + widget if necessary so that *item* is within the visible portion of + the tree. + + + .. method:: selection([selop=None[, items=None]]) + + If *selop* is not specified, returns selected items. Otherwise, it will + act according to the following selection methods. + + + .. method:: selection_set(items) + + *items* becomes the new selection. + + + .. method:: selection_add(items) + + Add *items* to the selection. + + + .. method:: selection_remove(items) + + Remove *items* from the selection. + + + .. method:: selection_toggle(items) + + Toggle the selection state of each item in *items*. + + + .. method:: set(item[, column=None[, value=None]]) + + With one argument, returns a dictionary of column/value pairs for the + specified *item*. With two arguments, returns the current value of the + specified *column*. With three arguments, sets the value of given + *column* in given *item* to the specified *value*. + + + .. method:: tag_bind(tagname[, sequence=None[, callback=None]]) + + Bind a callback for the given event *sequence* to the tag *tagname*. + When an event is delivered to an item, the callbacks for each of the + item's tags option are called. + + + .. method:: tag_configure(tagname[, option=None[, **kw]]) + + Query or modify the options for the specified *tagname*. + + If *kw* is not given, returns a dict of the option settings for + *tagname*. If *option* is specified, returns the value for that *option* + for the specified *tagname*. Otherwise, sets the options to the + corresponding values for the given *tagname*. + + + .. method:: tag_has(tagname[, item]) + + If *item* is specified, returns 1 or 0 depending on whether the specified + *item* has the given *tagname*. Otherwise, returns a list of all items + that have the specified tag. + + Availability: Tk 8.6 + + + .. method:: xview(*args) + + Query or modify horizontal position of the treeview. + + + .. method:: yview(*args) + + Query or modify vertical position of the treeview. + + +.. _TtkStyling: + +Ttk Styling +----------- + +Each widget in :mod:`ttk` is assigned a style, which specifies the set of +elements making up the widget and how they are arranged, along with dynamic and +default settings for element options. By default the style name is the same as +the widget's class name, but it may be overridden by the widget's style +option. If the class name of a widget is unknown, use the method +:meth:`Misc.winfo_class` (somewidget.winfo_class()). + +.. seealso:: + + `Tcl'2004 conference presentation <http://tktable.sourceforge.net/tile/tile-tcl2004.pdf>`_ + This document explains how the theme engine works + + +.. class:: Style + + This class is used to manipulate the style database. + + + .. method:: configure(style, query_opt=None, **kw) + + Query or set the default value of the specified option(s) in *style*. + + Each key in *kw* is an option and each value is a string identifying + the value for that option. + + For example, to change every default button to be a flat button with some + padding and a different background color do:: + + import ttk + import Tkinter + + root = Tkinter.Tk() + + ttk.Style().configure("TButton", padding=6, relief="flat", + background="#ccc") + + btn = ttk.Button(text="Sample") + btn.pack() + + root.mainloop() + + + .. method:: map(style, query_opt=None, **kw) + + Query or sets dynamic values of the specified option(s) in *style*. + + Each key in *kw* is an option and each value should be a list or a + tuple (usually) containing statespecs grouped in tuples, lists, or + something else of your preference. A statespec is a compound of one + or more states and then a value. + + An example:: + + import Tkinter + import ttk + + root = Tkinter.Tk() + + style = ttk.Style() + style.map("C.TButton", + foreground=[('pressed', 'red'), ('active', 'blue')], + background=[('pressed', '!disabled', 'black'), ('active', 'white')] + ) + + colored_btn = ttk.Button(text="Test", style="C.TButton").pack() + + root.mainloop() + + + Note that the order of the (states, value) sequences for an + option matters. In the previous example, if you change the + order to ``[('active', 'blue'), ('pressed', 'red')]`` in the + foreground option, for example, you would get a blue foreground + when the widget is in the active or pressed states. + + .. method:: lookup(style, option[, state=None[, default=None]]) + + Returns the value specified for *option* in *style*. + + If *state* is specified, it is expected to be a sequence of one or more + states. If the *default* argument is set, it is used as a fallback value + in case no specification for option is found. + + To check what font a Button uses by default, do:: + + import ttk + + print ttk.Style().lookup("TButton", "font") + + + .. method:: layout(style[, layoutspec=None]) + + Define the widget layout for given *style*. If *layoutspec* is omitted, + return the layout specification for given style. + + *layoutspec*, if specified, is expected to be a list or some other + sequence type (excluding strings), where each item should be a tuple and + the first item is the layout name and the second item should have the + format described described in `Layouts`_. + + To understand the format, see the following example (it is not + intended to do anything useful):: + + import ttk + import Tkinter + + root = Tkinter.Tk() + + style = ttk.Style() + style.layout("TMenubutton", [ + ("Menubutton.background", None), + ("Menubutton.button", {"children": + [("Menubutton.focus", {"children": + [("Menubutton.padding", {"children": + [("Menubutton.label", {"side": "left", "expand": 1})] + })] + })] + }), + ]) + + mbtn = ttk.Menubutton(text='Text') + mbtn.pack() + root.mainloop() + + + .. method:: element_create(elementname, etype, *args, **kw) + + Create a new element in the current theme, of the given *etype* which is + expected to be either "image", "from" or "vsapi". The latter is only + available in Tk 8.6a for Windows XP and Vista and is not described here. + + If "image" is used, *args* should contain the default image name followed + by statespec/value pairs (this is the imagespec), and *kw* may have the + following options: + + * border=padding + padding is a list of up to four integers, specifying the left, top, + right, and bottom borders, respectively. + + * height=height + Specifies a minimum height for the element. If less than zero, the + base image's height is used as a default. + + * padding=padding + Specifies the element's interior padding. Defaults to border's value + if not specified. + + * sticky=spec + Specifies how the image is placed within the final parcel. spec + contains zero or more characters “n”, “s”, “w”, or “e”. + + * width=width + Specifies a minimum width for the element. If less than zero, the + base image's width is used as a default. + + If "from" is used as the value of *etype*, + :meth:`element_create` will clone an existing + element. *args* is expected to contain a themename, from which + the element will be cloned, and optionally an element to clone from. + If this element to clone from is not specified, an empty element will + be used. *kw* is discarded. + + + .. method:: element_names() + + Returns the list of elements defined in the current theme. + + + .. method:: element_options(elementname) + + Returns the list of *elementname*'s options. + + + .. method:: theme_create(themename[, parent=None[, settings=None]]) + + Create a new theme. + + It is an error if *themename* already exists. If *parent* is specified, + the new theme will inherit styles, elements and layouts from the parent + theme. If *settings* are present they are expected to have the same + syntax used for :meth:`theme_settings`. + + + .. method:: theme_settings(themename, settings) + + Temporarily sets the current theme to *themename*, apply specified + *settings* and then restore the previous theme. + + Each key in *settings* is a style and each value may contain the keys + 'configure', 'map', 'layout' and 'element create' and they are expected + to have the same format as specified by the methods + :meth:`Style.configure`, :meth:`Style.map`, :meth:`Style.layout` and + :meth:`Style.element_create` respectively. + + As an example, let's change the Combobox for the default theme a bit:: + + import ttk + import Tkinter + + root = Tkinter.Tk() + + style = ttk.Style() + style.theme_settings("default", { + "TCombobox": { + "configure": {"padding": 5}, + "map": { + "background": [("active", "green2"), + ("!disabled", "green4")], + "fieldbackground": [("!disabled", "green3")], + "foreground": [("focus", "OliveDrab1"), + ("!disabled", "OliveDrab2")] + } + } + }) + + combo = ttk.Combobox().pack() + + root.mainloop() + + + .. method:: theme_names() + + Returns a list of all known themes. + + + .. method:: theme_use([themename]) + + If *themename* is not given, returns the theme in use. Otherwise, sets + the current theme to *themename*, refreshes all widgets and emits a + <<ThemeChanged>> event. + + +Layouts +^^^^^^^ + +A layout can be just None, if it takes no options, or a dict of +options specifying how to arrange the element. The layout mechanism +uses a simplified version of the pack geometry manager: given an +initial cavity, each element is allocated a parcel. Valid +options/values are: + + * side: whichside + Specifies which side of the cavity to place the element; one of + top, right, bottom or left. If omitted, the element occupies the + entire cavity. + + * sticky: nswe + Specifies where the element is placed inside its allocated parcel. + + * unit: 0 or 1 + If set to 1, causes the element and all of its descendants to be treated as + a single element for the purposes of :meth:`Widget.identify` et al. It's + used for things like scrollbar thumbs with grips. + + * children: [sublayout... ] + Specifies a list of elements to place inside the element. Each + element is a tuple (or other sequence type) where the first item is + the layout name, and the other is a `Layout`_. + +.. _Layout: `Layouts`_ diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index 249eb23..129ec0e 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -708,7 +708,10 @@ Settings for measurement >>> turtle.left(90) >>> turtle.heading() 90.0 - >>> turtle.degrees(400.0) # angle measurement in gon + + Change angle measurement unit to grad (also known as gon, + grade, or gradian and equals 1/100-th of the right angle.) + >>> turtle.degrees(400.0) >>> turtle.heading() 100.0 >>> turtle.degrees(360) @@ -875,7 +878,7 @@ Color control >>> tup = (0.2, 0.8, 0.55) >>> turtle.pencolor(tup) >>> turtle.pencolor() - (0.20000000000000001, 0.80000000000000004, 0.5490196078431373) + (0.2, 0.8, 0.5490196078431373) >>> colormode(255) >>> turtle.pencolor() (51, 204, 140) @@ -1455,6 +1458,7 @@ Window control :param args: a color string or three numbers in the range 0..colormode or a 3-tuple of such numbers + Set or return background color of the TurtleScreen. .. doctest:: @@ -1835,7 +1839,7 @@ Methods specific to Screen, not inherited from TurtleScreen .. function:: setup(width=_CFG["width"], height=_CFG["height"], startx=_CFG["leftright"], starty=_CFG["topbottom"]) Set the size and position of the main window. Default values of arguments - are stored in the configuration dicionary and can be changed via a + are stored in the configuration dictionary and can be changed via a :file:`turtle.cfg` file. :param width: if an integer, a size in pixels, if a float, a fraction of the @@ -2165,9 +2169,11 @@ The demoscripts are: | bytedesign | complex classical | :func:`tracer`, delay,| | | turtlegraphics pattern | :func:`update` | +----------------+------------------------------+-----------------------+ -| chaos | graphs verhust dynamics, | world coordinates | -| | proves that you must not | | -| | trust computers' computations| | +| chaos | graphs Verhulst dynamics, | world coordinates | +| | shows that computer's | | +| | computations can generate | | +| | results sometimes against the| | +| | common sense expectations | | +----------------+------------------------------+-----------------------+ | clock | analog clock showing time | turtles as clock's | | | of your computer | hands, ontimer | diff --git a/Doc/library/unicodedata.rst b/Doc/library/unicodedata.rst index 440a135..c765d74 100644 --- a/Doc/library/unicodedata.rst +++ b/Doc/library/unicodedata.rst @@ -16,12 +16,12 @@ This module provides access to the Unicode Character Database which defines character properties for all Unicode characters. The data in this database is -based on the :file:`UnicodeData.txt` file version 5.1.0 which is publicly +based on the :file:`UnicodeData.txt` file version 5.2.0 which is publicly available from ftp://ftp.unicode.org/. The module uses the same names and symbols as defined by the UnicodeData File -Format 5.1.0 (see http://www.unicode.org/Public/5.1.0/ucd/UCD.html). It defines -the following functions: +Format 5.2.0 (see http://www.unicode.org/reports/tr44/tr44-4.html). +It defines the following functions: .. function:: lookup(name) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 689f1f2..42ae8e2 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -1,4 +1,3 @@ - :mod:`unittest` --- Unit testing framework ========================================== @@ -9,9 +8,11 @@ .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> .. sectionauthor:: Raymond Hettinger <python@rcn.com> - .. versionadded:: 2.1 +(If you are already familiar with the basic concepts of testing, you might want +to skip to :ref:`the list of assert methods <assert-methods>`.) + The Python unit testing framework, sometimes referred to as "PyUnit," is a Python language version of JUnit, by Kent Beck and Erich Gamma. JUnit is, in turn, a Java version of Kent's Smalltalk testing framework. Each is the de @@ -49,26 +50,27 @@ The test case and test fixture concepts are supported through the :class:`TestCase` and :class:`FunctionTestCase` classes; the former should be used when creating new tests, and the latter can be used when integrating existing test code with a :mod:`unittest`\ -driven framework. When building test -fixtures using :class:`TestCase`, the :meth:`setUp` and :meth:`tearDown` methods -can be overridden to provide initialization and cleanup for the fixture. With -:class:`FunctionTestCase`, existing functions can be passed to the constructor -for these purposes. When the test is run, the fixture initialization is run -first; if it succeeds, the cleanup method is run after the test has been -executed, regardless of the outcome of the test. Each instance of the -:class:`TestCase` will only be used to run a single test method, so a new -fixture is created for each test. +fixtures using :class:`TestCase`, the :meth:`~TestCase.setUp` and +:meth:`~TestCase.tearDown` methods can be overridden to provide initialization +and cleanup for the fixture. With :class:`FunctionTestCase`, existing functions +can be passed to the constructor for these purposes. When the test is run, the +fixture initialization is run first; if it succeeds, the cleanup method is run +after the test has been executed, regardless of the outcome of the test. Each +instance of the :class:`TestCase` will only be used to run a single test method, +so a new fixture is created for each test. Test suites are implemented by the :class:`TestSuite` class. This class allows individual tests and test suites to be aggregated; when the suite is executed, all tests added directly to the suite and in "child" test suites are run. -A test runner is an object that provides a single method, :meth:`run`, which -accepts a :class:`TestCase` or :class:`TestSuite` object as a parameter, and -returns a result object. The class :class:`TestResult` is provided for use as -the result object. :mod:`unittest` provides the :class:`TextTestRunner` as an -example test runner which reports test results on the standard error stream by -default. Alternate runners can be implemented for other environments (such as -graphical environments) without any need to derive from a specific class. +A test runner is an object that provides a single method, +:meth:`~TestRunner.run`, which accepts a :class:`TestCase` or :class:`TestSuite` +object as a parameter, and returns a result object. The class +:class:`TestResult` is provided for use as the result object. :mod:`unittest` +provides the :class:`TextTestRunner` as an example test runner which reports +test results on the standard error stream by default. Alternate runners can be +implemented for other environments (such as graphical environments) without any +need to derive from a specific class. .. seealso:: @@ -76,16 +78,27 @@ graphical environments) without any need to derive from a specific class. Module :mod:`doctest` Another test-support module with a very different flavor. + `unittest2: A backport of new unittest features for Python 2.4-2.6 <http://pypi.python.org/pypi/unittest2>`_ + Many new features were added to unittest in Python 2.7, including test + discovery. unittest2 allows you to use these features with earlier + versions of Python. + `Simple Smalltalk Testing: With Patterns <http://www.XProgramming.com/testfram.htm>`_ - Kent Beck's original paper on testing frameworks using the pattern shared by - :mod:`unittest`. + Kent Beck's original paper on testing frameworks using the pattern shared + by :mod:`unittest`. `Nose <http://code.google.com/p/python-nose/>`_ and `py.test <http://pytest.org>`_ - Third-party unittest frameworks with a lighter-weight syntax - for writing tests. For example, ``assert func(10) == 42``. + Third-party unittest frameworks with a lighter-weight syntax for writing + tests. For example, ``assert func(10) == 42``. + + `The Python Testing Tools Taxonomy <http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy>`_ + An extensive list of Python testing tools including functional testing + frameworks and mock object libraries. + + `Testing in Python Mailing List <http://lists.idyll.org/listinfo/testing-in-python>`_ + A special-interest-group for discussion of testing, and testing tools, + in Python. - `python-mock <http://python-mock.sourceforge.net/>`_ and `minimock <http://blog.ianbicking.org/minimock.html>`_ - Tools for creating mock test objects (objects simulating external resources). .. _unittest-minimal-example: @@ -112,36 +125,41 @@ Here is a short script to test three functions from the :mod:`random` module:: self.seq.sort() self.assertEqual(self.seq, range(10)) + # should raise an exception for an immutable sequence + self.assertRaises(TypeError, random.shuffle, (1,2,3)) + def test_choice(self): element = random.choice(self.seq) self.assertTrue(element in self.seq) def test_sample(self): - self.assertRaises(ValueError, random.sample, self.seq, 20) + with self.assertRaises(ValueError): + random.sample(self.seq, 20) for element in random.sample(self.seq, 5): self.assertTrue(element in self.seq) if __name__ == '__main__': unittest.main() -A testcase is created by subclassing :class:`unittest.TestCase`. The three +A testcase is created by subclassing :class:`unittest.TestCase`. The three individual tests are defined with methods whose names start with the letters ``test``. This naming convention informs the test runner about which methods represent tests. -The crux of each test is a call to :meth:`assertEqual` to check for an expected -result; :meth:`assert_` to verify a condition; or :meth:`assertRaises` to verify -that an expected exception gets raised. These methods are used instead of the -:keyword:`assert` statement so the test runner can accumulate all test results -and produce a report. +The crux of each test is a call to :meth:`~TestCase.assertEqual` to check for an +expected result; :meth:`~TestCase.assertTrue` to verify a condition; or +:meth:`~TestCase.assertRaises` to verify that an expected exception gets raised. +These methods are used instead of the :keyword:`assert` statement so the test +runner can accumulate all test results and produce a report. -When a :meth:`setUp` method is defined, the test runner will run that method -prior to each test. Likewise, if a :meth:`tearDown` method is defined, the test -runner will invoke that method after each test. In the example, :meth:`setUp` -was used to create a fresh sequence for each test. +When a :meth:`~TestCase.setUp` method is defined, the test runner will run that +method prior to each test. Likewise, if a :meth:`~TestCase.tearDown` method is +defined, the test runner will invoke that method after each test. In the +example, :meth:`~TestCase.setUp` was used to create a fresh sequence for each +test. The final block shows a simple way to run the tests. :func:`unittest.main` -provides a command line interface to the test script. When run from the command +provides a command-line interface to the test script. When run from the command line, the above script produces an output that looks like this:: ... @@ -174,6 +192,137 @@ are sufficient to meet many everyday testing needs. The remainder of the documentation explores the full feature set from first principles. +.. _unittest-command-line-interface: + +Command-Line Interface +---------------------- + +The unittest module can be used from the command line to run tests from +modules, classes or even individual test methods:: + + python -m unittest test_module1 test_module2 + python -m unittest test_module.TestClass + python -m unittest test_module.TestClass.test_method + +You can pass in a list with any combination of module names, and fully +qualified class or method names. + +You can run tests with more detail (higher verbosity) by passing in the -v flag:: + + python -m unittest -v test_module + +For a list of all the command-line options:: + + python -m unittest -h + +.. versionchanged:: 2.7 + In earlier versions it was only possible to run individual test methods and + not modules or classes. + + +Command-line options +~~~~~~~~~~~~~~~~~~~~ + +:program:`unittest` supports these command-line options: + +.. program:: unittest + +.. cmdoption:: -b, --buffer + + The standard output and standard error streams are buffered during the test + run. Output during a passing test is discarded. Output is echoed normally + on test fail or error and is added to the failure messages. + +.. cmdoption:: -c, --catch + + Control-C during the test run waits for the current test to end and then + reports all the results so far. A second control-C raises the normal + :exc:`KeyboardInterrupt` exception. + + See `Signal Handling`_ for the functions that provide this functionality. + +.. cmdoption:: -f, --failfast + + Stop the test run on the first error or failure. + +.. versionadded:: 2.7 + The command-line options ``-b``, ``-c`` and ``-f`` were added. + +The command line can also be used for test discovery, for running all of the +tests in a project or just a subset. + + +.. _unittest-test-discovery: + +Test Discovery +-------------- + +.. versionadded:: 2.7 + +Unittest supports simple test discovery. In order to be compatible with test +discovery, all of the test files must be :ref:`modules <tut-modules>` or +:ref:`packages <tut-packages>` importable from the top-level directory of +the project (this means that their filenames must be valid +:ref:`identifiers <identifiers>`). + +Test discovery is implemented in :meth:`TestLoader.discover`, but can also be +used from the command line. The basic command-line usage is:: + + cd project_directory + python -m unittest discover + +The ``discover`` sub-command has the following options: + +.. program:: unittest discover + +.. cmdoption:: -v, --verbose + + Verbose output + +.. cmdoption:: -s directory + + Directory to start discovery ('.' default) + +.. cmdoption:: -p pattern + + Pattern to match test files ('test*.py' default) + +.. cmdoption:: -t directory + + Top level directory of project (defaults to start directory) + +The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in +as positional arguments in that order. The following two command lines +are equivalent:: + + python -m unittest discover -s project_directory -p '*_test.py' + python -m unittest discover project_directory '*_test.py' + +As well as being a path it is possible to pass a package name, for example +``myproject.subpackage.test``, as the start directory. The package name you +supply will then be imported and its location on the filesystem will be used +as the start directory. + +.. caution:: + + Test discovery loads tests by importing them. Once test discovery has + found all the test files from the start directory you specify it turns the + paths into package names to import. For example `foo/bar/baz.py` will be + imported as ``foo.bar.baz``. + + If you have a package installed globally and attempt test discovery on + a different copy of the package then the import *could* happen from the + wrong place. If this happens test discovery will warn you and exit. + + If you supply the start directory as a package name rather than a + path to a directory then discover assumes that whichever location it + imports from is the location you intended, so you will not get the + warning. + +Test modules and packages can customize test loading and discovery by through +the `load_tests protocol`_. + + .. _organizing-tests: Organizing test code @@ -193,8 +342,8 @@ The testing code of a :class:`TestCase` instance should be entirely self contained, such that it can be run either in isolation or in arbitrary combination with any number of other test cases. -The simplest :class:`TestCase` subclass will simply override the :meth:`runTest` -method in order to perform specific testing code:: +The simplest :class:`TestCase` subclass will simply override the +:meth:`~TestCase.runTest` method in order to perform specific testing code:: import unittest @@ -203,14 +352,13 @@ method in order to perform specific testing code:: widget = Widget('The widget') self.assertEqual(widget.size(), (50, 50), 'incorrect default size') -Note that in order to test something, we use the one of the :meth:`assert\*` or -:meth:`fail\*` methods provided by the :class:`TestCase` base class. If the -test fails, an exception will be raised, and :mod:`unittest` will identify the -test case as a :dfn:`failure`. Any other exceptions will be treated as -:dfn:`errors`. This helps you identify where the problem is: :dfn:`failures` are -caused by incorrect results - a 5 where you expected a 6. :dfn:`Errors` are -caused by incorrect code - e.g., a :exc:`TypeError` caused by an incorrect -function call. +Note that in order to test something, we use the one of the :meth:`assert\*` +methods provided by the :class:`TestCase` base class. If the test fails, an +exception will be raised, and :mod:`unittest` will identify the test case as a +:dfn:`failure`. Any other exceptions will be treated as :dfn:`errors`. This +helps you identify where the problem is: :dfn:`failures` are caused by incorrect +results - a 5 where you expected a 6. :dfn:`Errors` are caused by incorrect +code - e.g., a :exc:`TypeError` caused by an incorrect function call. The way to run a test case will be described later. For now, note that to construct an instance of such a test case, we call its constructor without @@ -223,8 +371,8 @@ the above case, constructing a :class:`Widget` in each of 100 Widget test case subclasses would mean unsightly duplication. Luckily, we can factor out such set-up code by implementing a method called -:meth:`setUp`, which the testing framework will automatically call for us when -we run the test:: +:meth:`~TestCase.setUp`, which the testing framework will automatically call for +us when we run the test:: import unittest @@ -243,12 +391,12 @@ we run the test:: self.assertEqual(self.widget.size(), (100,150), 'wrong size after resize') -If the :meth:`setUp` method raises an exception while the test is running, the -framework will consider the test to have suffered an error, and the -:meth:`runTest` method will not be executed. +If the :meth:`~TestCase.setUp` method raises an exception while the test is +running, the framework will consider the test to have suffered an error, and the +:meth:`~TestCase.runTest` method will not be executed. -Similarly, we can provide a :meth:`tearDown` method that tidies up after the -:meth:`runTest` method has been run:: +Similarly, we can provide a :meth:`~TestCase.tearDown` method that tidies up +after the :meth:`~TestCase.runTest` method has been run:: import unittest @@ -260,8 +408,8 @@ Similarly, we can provide a :meth:`tearDown` method that tidies up after the self.widget.dispose() self.widget = None -If :meth:`setUp` succeeded, the :meth:`tearDown` method will be run whether -:meth:`runTest` succeeded or not. +If :meth:`~TestCase.setUp` succeeded, the :meth:`~TestCase.tearDown` method will +be run whether :meth:`~TestCase.runTest` succeeded or not. Such a working environment for the testing code is called a :dfn:`fixture`. @@ -336,8 +484,9 @@ will create a test suite that will run ``WidgetTestCase.test_default_size()`` an ``WidgetTestCase.test_resize``. :class:`TestLoader` uses the ``'test'`` method name prefix to identify test methods automatically. -Note that the order in which the various test cases will be run is determined by -sorting the test function names with the built-in :func:`cmp` function. +Note that the order in which the various test cases will be run is +determined by sorting the test function names with respect to the +built-in ordering for strings. Often it is desirable to group suites of test cases together, so as to run tests for the whole system at once. This is easy, since :class:`TestSuite` instances @@ -409,10 +558,110 @@ may treat :exc:`AssertionError` differently. .. note:: - Even though :class:`FunctionTestCase` can be used to quickly convert an existing - test base over to a :mod:`unittest`\ -based system, this approach is not - recommended. Taking the time to set up proper :class:`TestCase` subclasses will - make future test refactorings infinitely easier. + Even though :class:`FunctionTestCase` can be used to quickly convert an + existing test base over to a :mod:`unittest`\ -based system, this approach is + not recommended. Taking the time to set up proper :class:`TestCase` + subclasses will make future test refactorings infinitely easier. + +In some cases, the existing tests may have been written using the :mod:`doctest` +module. If so, :mod:`doctest` provides a :class:`DocTestSuite` class that can +automatically build :class:`unittest.TestSuite` instances from the existing +:mod:`doctest`\ -based tests. + + +.. _unittest-skipping: + +Skipping tests and expected failures +------------------------------------ + +.. versionadded:: 2.7 + +Unittest supports skipping individual test methods and even whole classes of +tests. In addition, it supports marking a test as a "expected failure," a test +that is broken and will fail, but shouldn't be counted as a failure on a +:class:`TestResult`. + +Skipping a test is simply a matter of using the :func:`skip` :term:`decorator` +or one of its conditional variants. + +Basic skipping looks like this: :: + + class MyTestCase(unittest.TestCase): + + @unittest.skip("demonstrating skipping") + def test_nothing(self): + self.fail("shouldn't happen") + + @unittest.skipIf(mylib.__version__ < (1, 3), + "not supported in this library version") + def test_format(self): + # Tests that work for only a certain version of the library. + pass + + @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows") + def test_windows_support(self): + # windows specific testing code + pass + +This is the output of running the example above in verbose mode: :: + + test_format (__main__.MyTestCase) ... skipped 'not supported in this library version' + test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping' + test_windows_support (__main__.MyTestCase) ... skipped 'requires Windows' + + ---------------------------------------------------------------------- + Ran 3 tests in 0.005s + + OK (skipped=3) + +Classes can be skipped just like methods: :: + + @skip("showing class skipping") + class MySkippedTestCase(unittest.TestCase): + def test_not_run(self): + pass + +:meth:`TestCase.setUp` can also skip the test. This is useful when a resource +that needs to be set up is not available. + +Expected failures use the :func:`expectedFailure` decorator. :: + + class ExpectedFailureTestCase(unittest.TestCase): + @unittest.expectedFailure + def test_fail(self): + self.assertEqual(1, 0, "broken") + +It's easy to roll your own skipping decorators by making a decorator that calls +:func:`skip` on the test when it wants it to be skipped. This decorator skips +the test unless the passed object has a certain attribute: :: + + def skipUnlessHasattr(obj, attr): + if hasattr(obj, attr): + return lambda func: func + return unittest.skip("{0!r} doesn't have {1!r}".format(obj, attr)) + +The following decorators implement test skipping and expected failures: + +.. function:: skip(reason) + + Unconditionally skip the decorated test. *reason* should describe why the + test is being skipped. + +.. function:: skipIf(condition, reason) + + Skip the decorated test if *condition* is true. + +.. function:: skipUnless(condition, reason) + + Skip the decorated test unless *condition* is true. + +.. function:: expectedFailure + + Mark the test as an expected failure. If the test fails when run, the test + is not counted as a failure. + +Skipped tests will not have :meth:`setUp` or :meth:`tearDown` run around them. +Skipped classes will not have :meth:`setUpClass` or :meth:`tearDownClass` run. .. _unittest-contents: @@ -420,8 +669,15 @@ may treat :exc:`AssertionError` differently. Classes and functions --------------------- +This section describes in depth the API of :mod:`unittest`. + -.. class:: TestCase([methodName]) +.. _testcase-objects: + +Test cases +~~~~~~~~~~ + +.. class:: TestCase(methodName='runTest') Instances of the :class:`TestCase` class represent the smallest testable units in the :mod:`unittest` universe. This class is intended to be used as a base @@ -443,19 +699,654 @@ Classes and functions Here, we create two instances of :class:`WidgetTestCase`, each of which runs a single test. - *methodName* defaults to ``'runTest'``. + *methodName* defaults to :meth:`runTest`. + + :class:`TestCase` instances provide three groups of methods: one group used + to run the test, another used by the test implementation to check conditions + and report failures, and some inquiry methods allowing information about the + test itself to be gathered. + + Methods in the first group (running the test) are: + + + .. method:: setUp() + + Method called to prepare the test fixture. This is called immediately + before calling the test method; any exception raised by this method will + be considered an error rather than a test failure. The default + implementation does nothing. + + + .. method:: tearDown() + + Method called immediately after the test method has been called and the + result recorded. This is called even if the test method raised an + exception, so the implementation in subclasses may need to be particularly + careful about checking internal state. Any exception raised by this + method will be considered an error rather than a test failure. This + method will only be called if the :meth:`setUp` succeeds, regardless of + the outcome of the test method. The default implementation does nothing. + + + .. method:: setUpClass() + + A class method called before tests in an individual class run. + ``setUpClass`` is called with the class as the only argument + and must be decorated as a :func:`classmethod`:: + + @classmethod + def setUpClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 2.7 + + + .. method:: tearDownClass() + + A class method called after tests in an individual class have run. + ``tearDownClass`` is called with the class as the only argument + and must be decorated as a :meth:`classmethod`:: + + @classmethod + def tearDownClass(cls): + ... + + See `Class and Module Fixtures`_ for more details. + + .. versionadded:: 2.7 + + + .. method:: run(result=None) + + Run the test, collecting the result into the test result object passed as + *result*. If *result* is omitted or ``None``, a temporary result + object is created (by calling the :meth:`defaultTestResult` method) and + used. The result object is not returned to :meth:`run`'s caller. + + The same effect may be had by simply calling the :class:`TestCase` + instance. + + + .. method:: skipTest(reason) + + Calling this during a test method or :meth:`setUp` skips the current + test. See :ref:`unittest-skipping` for more information. + + .. versionadded:: 2.7 + + + .. method:: debug() + + Run the test without collecting the result. This allows exceptions raised + by the test to be propagated to the caller, and can be used to support + running tests under a debugger. + + .. _assert-methods: + + The :class:`TestCase` class provides a number of methods to check for and + report failures, such as: + + +-----------------------------------------+-----------------------------+---------------+ + | Method | Checks that | New in | + +=========================================+=============================+===============+ + | :meth:`assertEqual(a, b) | ``a == b`` | | + | <TestCase.assertEqual>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertNotEqual(a, b) | ``a != b`` | | + | <TestCase.assertNotEqual>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertTrue(x) | ``bool(x) is True`` | | + | <TestCase.assertTrue>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertFalse(x) | ``bool(x) is False`` | | + | <TestCase.assertFalse>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIs(a, b) | ``a is b`` | 2.7 | + | <TestCase.assertIs>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsNot(a, b) | ``a is not b`` | 2.7 | + | <TestCase.assertIsNot>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsNone(x) | ``x is None`` | 2.7 | + | <TestCase.assertIsNone>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsNotNone(x) | ``x is not None`` | 2.7 | + | <TestCase.assertIsNotNone>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIn(a, b) | ``a in b`` | 2.7 | + | <TestCase.assertIn>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertNotIn(a, b) | ``a not in b`` | 2.7 | + | <TestCase.assertNotIn>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertIsInstance(a, b) | ``isinstance(a, b)`` | 2.7 | + | <TestCase.assertIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ + | :meth:`assertNotIsInstance(a, b) | ``not isinstance(a, b)`` | 2.7 | + | <TestCase.assertNotIsInstance>` | | | + +-----------------------------------------+-----------------------------+---------------+ + + All the assert methods (except :meth:`assertRaises`, + :meth:`assertRaisesRegexp`) + accept a *msg* argument that, if specified, is used as the error message on + failure (see also :data:`longMessage`). + + .. method:: assertEqual(first, second, msg=None) + + Test that *first* and *second* are equal. If the values do not compare + equal, the test will fail. + + In addition, if *first* and *second* are the exact same type and one of + list, tuple, dict, set, frozenset or unicode or any type that a subclass + registers with :meth:`addTypeEqualityFunc` the type specific equality + function will be called in order to generate a more useful default + error message (see also the :ref:`list of type-specific methods + <type-specific-methods>`). + + .. versionchanged:: 2.7 + Added the automatic calling of type specific equality function. + + + .. method:: assertNotEqual(first, second, msg=None) + + Test that *first* and *second* are not equal. If the values do compare + equal, the test will fail. + + .. method:: assertTrue(expr, msg=None) + assertFalse(expr, msg=None) + + Test that *expr* is true (or false). + + Note that this is equivalent to ``bool(expr) is True`` and not to ``expr + is True`` (use ``assertIs(expr, True)`` for the latter). This method + should also be avoided when more specific methods are available (e.g. + ``assertEqual(a, b)`` instead of ``assertTrue(a == b)``), because they + provide a better error message in case of failure. + + + .. method:: assertIs(first, second, msg=None) + assertIsNot(first, second, msg=None) + + Test that *first* and *second* evaluate (or don't evaluate) to the same object. + + .. versionadded:: 2.7 + + + .. method:: assertIsNone(expr, msg=None) + assertIsNotNone(expr, msg=None) + + Test that *expr* is (or is not) None. + + .. versionadded:: 2.7 + + + .. method:: assertIn(first, second, msg=None) + assertNotIn(first, second, msg=None) + + Test that *first* is (or is not) in *second*. + + .. versionadded:: 2.7 + + + .. method:: assertIsInstance(obj, cls, msg=None) + assertNotIsInstance(obj, cls, msg=None) + + Test that *obj* is (or is not) an instance of *cls* (which can be a + class or a tuple of classes, as supported by :func:`isinstance`). + + .. versionadded:: 2.7 + + + It is also possible to check that exceptions and warnings are raised using + the following methods: + + +---------------------------------------------------------+--------------------------------------+------------+ + | Method | Checks that | New in | + +=========================================================+======================================+============+ + | :meth:`assertRaises(exc, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | | + | <TestCase.assertRaises>` | | | + +---------------------------------------------------------+--------------------------------------+------------+ + | :meth:`assertRaisesRegexp(exc, re, fun, *args, **kwds) | ``fun(*args, **kwds)`` raises `exc` | 2.7 | + | <TestCase.assertRaisesRegexp>` | and the message matches `re` | | + +---------------------------------------------------------+--------------------------------------+------------+ + + .. method:: assertRaises(exception, callable, *args, **kwds) + assertRaises(exception) + + Test that an exception is raised when *callable* is called with any + positional or keyword arguments that are also passed to + :meth:`assertRaises`. The test passes if *exception* is raised, is an + error if another exception is raised, or fails if no exception is raised. + To catch any of a group of exceptions, a tuple containing the exception + classes may be passed as *exception*. + + If only the *exception* argument is given, returns a context manager so + that the code under test can be written inline rather than as a function:: + + with self.assertRaises(SomeException): + do_something() + + The context manager will store the caught exception object in its + :attr:`exception` attribute. This can be useful if the intention + is to perform additional checks on the exception raised:: + + with self.assertRaises(SomeException) as cm: + do_something() + + the_exception = cm.exception + self.assertEqual(the_exception.error_code, 3) + + .. versionchanged:: 2.7 + Added the ability to use :meth:`assertRaises` as a context manager. + + + .. method:: assertRaisesRegexp(exception, regexp, callable, *args, **kwds) + assertRaisesRegexp(exception, regexp) + + Like :meth:`assertRaises` but also tests that *regexp* matches + on the string representation of the raised exception. *regexp* may be + a regular expression object or a string containing a regular expression + suitable for use by :func:`re.search`. Examples:: + + self.assertRaisesRegexp(ValueError, 'invalid literal for.*XYZ$', + int, 'XYZ') + + or:: + + with self.assertRaisesRegexp(ValueError, 'literal'): + int('XYZ') + + .. versionadded:: 2.7 + + + + There are also other methods used to perform more specific checks, such as: + + +---------------------------------------+--------------------------------+--------------+ + | Method | Checks that | New in | + +=======================================+================================+==============+ + | :meth:`assertAlmostEqual(a, b) | ``round(a-b, 7) == 0`` | | + | <TestCase.assertAlmostEqual>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertNotAlmostEqual(a, b) | ``round(a-b, 7) != 0`` | | + | <TestCase.assertNotAlmostEqual>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertGreater(a, b) | ``a > b`` | 2.7 | + | <TestCase.assertGreater>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertGreaterEqual(a, b) | ``a >= b`` | 2.7 | + | <TestCase.assertGreaterEqual>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertLess(a, b) | ``a < b`` | 2.7 | + | <TestCase.assertLess>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertLessEqual(a, b) | ``a <= b`` | 2.7 | + | <TestCase.assertLessEqual>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertRegexpMatches(s, re) | ``regex.search(s)`` | 2.7 | + | <TestCase.assertRegexpMatches>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertNotRegexpMatches(s, re) | ``not regex.search(s)`` | 2.7 | + | <TestCase.assertNotRegexpMatches>` | | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertItemsEqual(a, b) | sorted(a) == sorted(b) and | 2.7 | + | <TestCase.assertItemsEqual>` | works with unhashable objs | | + +---------------------------------------+--------------------------------+--------------+ + | :meth:`assertDictContainsSubset(a, b) | all the key/value pairs | 2.7 | + | <TestCase.assertDictContainsSubset>` | in `a` exist in `b` | | + +---------------------------------------+--------------------------------+--------------+ + + + .. method:: assertAlmostEqual(first, second, places=7, msg=None, delta=None) + assertNotAlmostEqual(first, second, places=7, msg=None, delta=None) + + Test that *first* and *second* are approximately (or not approximately) + equal by computing the difference, rounding to the given number of + decimal *places* (default 7), and comparing to zero. Note that these + methods round the values to the given number of *decimal places* (i.e. + like the :func:`round` function) and not *significant digits*. + + If *delta* is supplied instead of *places* then the difference + between *first* and *second* must be less (or more) than *delta*. + + Supplying both *delta* and *places* raises a ``TypeError``. + + .. versionchanged:: 2.7 + :meth:`assertAlmostEqual` automatically considers almost equal objects + that compare equal. :meth:`assertNotAlmostEqual` automatically fails + if the objects compare equal. Added the *delta* keyword argument. + + + + .. method:: assertGreater(first, second, msg=None) + assertGreaterEqual(first, second, msg=None) + assertLess(first, second, msg=None) + assertLessEqual(first, second, msg=None) + + Test that *first* is respectively >, >=, < or <= than *second* depending + on the method name. If not, the test will fail:: + + >>> self.assertGreaterEqual(3, 4) + AssertionError: "3" unexpectedly not greater than or equal to "4" + + .. versionadded:: 2.7 + + .. method:: assertRegexpMatches(text, regexp, msg=None) -.. class:: FunctionTestCase(testFunc[, setUp[, tearDown[, description]]]) + Test that a *regexp* search matches *text*. In case + of failure, the error message will include the pattern and the *text* (or + the pattern and the part of *text* that unexpectedly matched). *regexp* + may be a regular expression object or a string containing a regular + expression suitable for use by :func:`re.search`. + + .. versionadded:: 2.7 + + + .. method:: assertNotRegexpMatches(text, regexp, msg=None) + + Verifies that a *regexp* search does not match *text*. Fails with an error + message including the pattern and the part of *text* that matches. *regexp* + may be a regular expression object or a string containing a regular + expression suitable for use by :func:`re.search`. + + .. versionadded:: 2.7 + + + .. method:: assertItemsEqual(actual, expected, msg=None) + + Test that sequence *expected* contains the same elements as *actual*, + regardless of their order. When they don't, an error message listing the + differences between the sequences will be generated. + + Duplicate elements are *not* ignored when comparing *actual* and + *expected*. It verifies if each element has the same count in both + sequences. It is the equivalent of ``assertEqual(sorted(expected), + sorted(actual))`` but it works with sequences of unhashable objects as + well. + + .. versionadded:: 2.7 + + + .. method:: assertDictContainsSubset(expected, actual, msg=None) + + Tests whether the key/value pairs in dictionary *actual* are a + superset of those in *expected*. If not, an error message listing + the missing keys and mismatched values is generated. + + .. versionadded:: 2.7 + .. deprecated:: 3.2 + + + + .. _type-specific-methods: + + The :meth:`assertEqual` method dispatches the equality check for objects of + the same type to different type-specific methods. These methods are already + implemented for most of the built-in types, but it's also possible to + register new methods using :meth:`addTypeEqualityFunc`: + + .. method:: addTypeEqualityFunc(typeobj, function) + + Registers a type-specific method called by :meth:`assertEqual` to check + if two objects of exactly the same *typeobj* (not subclasses) compare + equal. *function* must take two positional arguments and a third msg=None + keyword argument just as :meth:`assertEqual` does. It must raise + :data:`self.failureException(msg) <failureException>` when inequality + between the first two parameters is detected -- possibly providing useful + information and explaining the inequalities in details in the error + message. + + .. versionadded:: 2.7 + + The list of type-specific methods automatically used by + :meth:`~TestCase.assertEqual` are summarized in the following table. Note + that it's usually not necessary to invoke these methods directly. + + +-----------------------------------------+-----------------------------+--------------+ + | Method | Used to compare | New in | + +=========================================+=============================+==============+ + | :meth:`assertMultiLineEqual(a, b) | strings | 2.7 | + | <TestCase.assertMultiLineEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + | :meth:`assertSequenceEqual(a, b) | sequences | 2.7 | + | <TestCase.assertSequenceEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + | :meth:`assertListEqual(a, b) | lists | 2.7 | + | <TestCase.assertListEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + | :meth:`assertTupleEqual(a, b) | tuples | 2.7 | + | <TestCase.assertTupleEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + | :meth:`assertSetEqual(a, b) | sets or frozensets | 2.7 | + | <TestCase.assertSetEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + | :meth:`assertDictEqual(a, b) | dicts | 2.7 | + | <TestCase.assertDictEqual>` | | | + +-----------------------------------------+-----------------------------+--------------+ + + + + .. method:: assertMultiLineEqual(first, second, msg=None) + + Test that the multiline string *first* is equal to the string *second*. + When not equal a diff of the two strings highlighting the differences + will be included in the error message. This method is used by default + when comparing strings with :meth:`assertEqual`. + + .. versionadded:: 2.7 + + + .. method:: assertSequenceEqual(seq1, seq2, msg=None, seq_type=None) + + Tests that two sequences are equal. If a *seq_type* is supplied, both + *seq1* and *seq2* must be instances of *seq_type* or a failure will + be raised. If the sequences are different an error message is + constructed that shows the difference between the two. + + This method is not called directly by :meth:`assertEqual`, but + it's used to implement :meth:`assertListEqual` and + :meth:`assertTupleEqual`. + + .. versionadded:: 2.7 + + + .. method:: assertListEqual(list1, list2, msg=None) + assertTupleEqual(tuple1, tuple2, msg=None) + + Tests that two lists or tuples are equal. If not an error message is + constructed that shows only the differences between the two. An error + is also raised if either of the parameters are of the wrong type. + These methods are used by default when comparing lists or tuples with + :meth:`assertEqual`. + + .. versionadded:: 2.7 + + + .. method:: assertSetEqual(set1, set2, msg=None) + + Tests that two sets are equal. If not, an error message is constructed + that lists the differences between the sets. This method is used by + default when comparing sets or frozensets with :meth:`assertEqual`. + + Fails if either of *set1* or *set2* does not have a :meth:`set.difference` + method. + + .. versionadded:: 2.7 + + + .. method:: assertDictEqual(expected, actual, msg=None) + + Test that two dictionaries are equal. If not, an error message is + constructed that shows the differences in the dictionaries. This + method will be used by default to compare dictionaries in + calls to :meth:`assertEqual`. + + .. versionadded:: 2.7 + + + + .. _other-methods-and-attrs: + + Finally the :class:`TestCase` provides the following methods and attributes: + + + .. method:: fail(msg=None) + + Signals a test failure unconditionally, with *msg* or ``None`` for + the error message. + + + .. attribute:: failureException + + This class attribute gives the exception raised by the test method. If a + test framework needs to use a specialized exception, possibly to carry + additional information, it must subclass this exception in order to "play + fair" with the framework. The initial value of this attribute is + :exc:`AssertionError`. + + + .. attribute:: longMessage + + If set to ``True`` then any explicit failure message you pass in to the + :ref:`assert methods <assert-methods>` will be appended to the end of the + normal failure message. The normal messages contain useful information + about the objects involved, for example the message from assertEqual + shows you the repr of the two unequal objects. Setting this attribute + to ``True`` allows you to have a custom error message in addition to the + normal one. + + This attribute defaults to ``False``, meaning that a custom message passed + to an assert method will silence the normal message. + + The class setting can be overridden in individual tests by assigning an + instance attribute to ``True`` or ``False`` before calling the assert methods. + + .. versionadded:: 2.7 + + + .. attribute:: maxDiff + + This attribute controls the maximum length of diffs output by assert + methods that report diffs on failure. It defaults to 80*8 characters. + Assert methods affected by this attribute are + :meth:`assertSequenceEqual` (including all the sequence comparison + methods that delegate to it), :meth:`assertDictEqual` and + :meth:`assertMultiLineEqual`. + + Setting ``maxDiff`` to None means that there is no maximum length of + diffs. + + .. versionadded:: 2.7 + + + Testing frameworks can use the following methods to collect information on + the test: + + + .. method:: countTestCases() + + Return the number of tests represented by this test object. For + :class:`TestCase` instances, this will always be ``1``. + + + .. method:: defaultTestResult() + + Return an instance of the test result class that should be used for this + test case class (if no other result instance is provided to the + :meth:`run` method). + + For :class:`TestCase` instances, this will always be an instance of + :class:`TestResult`; subclasses of :class:`TestCase` should override this + as necessary. + + + .. method:: id() + + Return a string identifying the specific test case. This is usually the + full name of the test method, including the module and class name. + + + .. method:: shortDescription() + + Returns a description of the test, or ``None`` if no description + has been provided. The default implementation of this method + returns the first line of the test method's docstring, if available, + or :const:`None`. + + + + .. method:: addCleanup(function, *args, **kwargs) + + Add a function to be called after :meth:`tearDown` to cleanup resources + used during the test. Functions will be called in reverse order to the + order they are added (LIFO). They are called with any arguments and + keyword arguments passed into :meth:`addCleanup` when they are + added. + + If :meth:`setUp` fails, meaning that :meth:`tearDown` is not called, + then any cleanup functions added will still be called. + + .. versionadded:: 2.7 + + + .. method:: doCleanups() + + This method is called unconditionally after :meth:`tearDown`, or + after :meth:`setUp` if :meth:`setUp` raises an exception. + + It is responsible for calling all the cleanup functions added by + :meth:`addCleanup`. If you need cleanup functions to be called + *prior* to :meth:`tearDown` then you can call :meth:`doCleanups` + yourself. + + :meth:`doCleanups` pops methods off the stack of cleanup + functions one at a time, so it can be called at any time. + + .. versionadded:: 2.7 + + +.. class:: FunctionTestCase(testFunc, setUp=None, tearDown=None, description=None) This class implements the portion of the :class:`TestCase` interface which - allows the test runner to drive the test, but does not provide the methods which - test code can use to check and report errors. This is used to create test cases - using legacy test code, allowing it to be integrated into a :mod:`unittest`\ - -based test framework. + allows the test runner to drive the test, but does not provide the methods + which test code can use to check and report errors. This is used to create + test cases using legacy test code, allowing it to be integrated into a + :mod:`unittest`-based test framework. + + +Deprecated aliases +################## + +For historical reasons, some of the :class:`TestCase` methods had one or more +aliases that are now deprecated. The following table lists the correct names +along with their deprecated aliases: + + ============================== =============================== + Method Name Deprecated alias(es) + ============================== =============================== + :meth:`.assertEqual` failUnlessEqual, assertEquals + :meth:`.assertNotEqual` failIfEqual + :meth:`.assertTrue` failUnless, assert\_ + :meth:`.assertFalse` failIf + :meth:`.assertRaises` failUnlessRaises + :meth:`.assertAlmostEqual` failUnlessAlmostEqual + :meth:`.assertNotAlmostEqual` failIfAlmostEqual + ============================== =============================== + + .. deprecated:: 2.7 + the aliases listed in the second column -.. class:: TestSuite([tests]) + +.. _testsuite-objects: + +Grouping tests +~~~~~~~~~~~~~~ + +.. class:: TestSuite(tests=()) This class represents an aggregation of individual tests cases and test suites. The class presents the interface needed by the test runner to allow it to be run @@ -466,483 +1357,676 @@ Classes and functions test suites that will be used to build the suite initially. Additional methods are provided to add test cases and suites to the collection later on. + :class:`TestSuite` objects behave much like :class:`TestCase` objects, except + they do not actually implement a test. Instead, they are used to aggregate + tests into groups of tests that should be run together. Some additional + methods are available to add tests to :class:`TestSuite` instances: + + + .. method:: TestSuite.addTest(test) + + Add a :class:`TestCase` or :class:`TestSuite` to the suite. + + + .. method:: TestSuite.addTests(tests) + + Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite` + instances to this test suite. + + This is equivalent to iterating over *tests*, calling :meth:`addTest` for + each element. + + :class:`TestSuite` shares the following methods with :class:`TestCase`: + + + .. method:: run(result) + + Run the tests associated with this suite, collecting the result into the + test result object passed as *result*. Note that unlike + :meth:`TestCase.run`, :meth:`TestSuite.run` requires the result object to + be passed in. + + + .. method:: debug() + + Run the tests associated with this suite without collecting the + result. This allows exceptions raised by the test to be propagated to the + caller and can be used to support running tests under a debugger. + + + .. method:: countTestCases() + + Return the number of tests represented by this test object, including all + individual tests and sub-suites. + + + .. method:: __iter__() + + Tests grouped by a :class:`TestSuite` are always accessed by iteration. + Subclasses can lazily provide tests by overriding :meth:`__iter__`. Note + that this method maybe called several times on a single suite + (for example when counting tests or comparing for equality) + so the tests returned must be the same for repeated iterations. + + .. versionchanged:: 2.7 + In earlier versions the :class:`TestSuite` accessed tests directly rather + than through iteration, so overriding :meth:`__iter__` wasn't sufficient + for providing tests. + + In the typical usage of a :class:`TestSuite` object, the :meth:`run` method + is invoked by a :class:`TestRunner` rather than by the end-user test harness. + + +Loading and running tests +~~~~~~~~~~~~~~~~~~~~~~~~~ .. class:: TestLoader() - This class is responsible for loading tests according to various criteria and - returning them wrapped in a :class:`TestSuite`. It can load all tests within a - given module or :class:`TestCase` subclass. + The :class:`TestLoader` class is used to create test suites from classes and + modules. Normally, there is no need to create an instance of this class; the + :mod:`unittest` module provides an instance that can be shared as + ``unittest.defaultTestLoader``. Using a subclass or instance, however, allows + customization of some configurable properties. + :class:`TestLoader` objects have the following methods: -.. class:: TestResult() - This class is used to compile information about which tests have succeeded and - which have failed. + .. method:: loadTestsFromTestCase(testCaseClass) + Return a suite of all tests cases contained in the :class:`TestCase`\ -derived + :class:`testCaseClass`. -.. data:: defaultTestLoader - Instance of the :class:`TestLoader` class intended to be shared. If no - customization of the :class:`TestLoader` is needed, this instance can be used - instead of repeatedly creating new instances. + .. method:: loadTestsFromModule(module) + Return a suite of all tests cases contained in the given module. This + method searches *module* for classes derived from :class:`TestCase` and + creates an instance of the class for each test method defined for the + class. -.. class:: TextTestRunner([stream[, descriptions[, verbosity]]]) + .. note:: - A basic test runner implementation which prints results on standard error. It - has a few configurable parameters, but is essentially very simple. Graphical - applications which run test suites should provide alternate implementations. + While using a hierarchy of :class:`TestCase`\ -derived classes can be + convenient in sharing fixtures and helper functions, defining test + methods on base classes that are not intended to be instantiated + directly does not play well with this method. Doing so, however, can + be useful when the fixtures are different and defined in subclasses. + If a module provides a ``load_tests`` function it will be called to + load the tests. This allows modules to customize test loading. + This is the `load_tests protocol`_. -.. function:: main([module[, defaultTest[, argv[, testRunner[, testLoader]]]]]) + .. versionchanged:: 2.7 + Support for ``load_tests`` added. - A command-line program that runs a set of tests; this is primarily for making - test modules conveniently executable. The simplest use for this function is to - include the following line at the end of a test script:: - if __name__ == '__main__': - unittest.main() + .. method:: loadTestsFromName(name, module=None) - The *testRunner* argument can either be a test runner class or an already - created instance of it. + Return a suite of all tests cases given a string specifier. -In some cases, the existing tests may have been written using the :mod:`doctest` -module. If so, that module provides a :class:`DocTestSuite` class that can -automatically build :class:`unittest.TestSuite` instances from the existing -:mod:`doctest`\ -based tests. + The specifier *name* is a "dotted name" that may resolve either to a + module, a test case class, a test method within a test case class, a + :class:`TestSuite` instance, or a callable object which returns a + :class:`TestCase` or :class:`TestSuite` instance. These checks are + applied in the order listed here; that is, a method on a possible test + case class will be picked up as "a test method within a test case class", + rather than "a callable object". -.. versionadded:: 2.3 + For example, if you have a module :mod:`SampleTests` containing a + :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test + methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the + specifier ``'SampleTests.SampleTestCase'`` would cause this method to + return a suite which will run all three test methods. Using the specifier + ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test + suite which will run only the :meth:`test_two` test method. The specifier + can refer to modules and packages which have not been imported; they will + be imported as a side-effect. + The method optionally resolves *name* relative to the given *module*. -.. _testcase-objects: -TestCase Objects ----------------- + .. method:: loadTestsFromNames(names, module=None) -Each :class:`TestCase` instance represents a single test, but each concrete -subclass may be used to define multiple tests --- the concrete class represents -a single test fixture. The fixture is created and cleaned up for each test -case. + Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather + than a single name. The return value is a test suite which supports all + the tests defined for each name. -:class:`TestCase` instances provide three groups of methods: one group used to -run the test, another used by the test implementation to check conditions and -report failures, and some inquiry methods allowing information about the test -itself to be gathered. -Methods in the first group (running the test) are: + .. method:: getTestCaseNames(testCaseClass) + Return a sorted sequence of method names found within *testCaseClass*; + this should be a subclass of :class:`TestCase`. -.. method:: TestCase.setUp() - Method called to prepare the test fixture. This is called immediately before - calling the test method; any exception raised by this method will be considered - an error rather than a test failure. The default implementation does nothing. + .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) + Find and return all test modules from the specified start directory, + recursing into subdirectories to find them. Only test files that match + *pattern* will be loaded. (Using shell style pattern matching.) Only + module names that are importable (i.e. are valid Python identifiers) will + be loaded. -.. method:: TestCase.tearDown() + All test modules must be importable from the top level of the project. If + the start directory is not the top level directory then the top level + directory must be specified separately. - Method called immediately after the test method has been called and the result - recorded. This is called even if the test method raised an exception, so the - implementation in subclasses may need to be particularly careful about checking - internal state. Any exception raised by this method will be considered an error - rather than a test failure. This method will only be called if the - :meth:`setUp` succeeds, regardless of the outcome of the test method. The - default implementation does nothing. + If importing a module fails, for example due to a syntax error, then this + will be recorded as a single error and discovery will continue. + If a test package name (directory with :file:`__init__.py`) matches the + pattern then the package will be checked for a ``load_tests`` + function. If this exists then it will be called with *loader*, *tests*, + *pattern*. -.. method:: TestCase.run([result]) + If load_tests exists then discovery does *not* recurse into the package, + ``load_tests`` is responsible for loading all tests in the package. - Run the test, collecting the result into the test result object passed as - *result*. If *result* is omitted or :const:`None`, a temporary result object is - created (by calling the :meth:`defaultTestCase` method) and used; this result - object is not returned to :meth:`run`'s caller. + The pattern is deliberately not stored as a loader attribute so that + packages can continue discovery themselves. *top_level_dir* is stored so + ``load_tests`` does not need to pass this argument in to + ``loader.discover()``. - The same effect may be had by simply calling the :class:`TestCase` instance. + *start_dir* can be a dotted module name as well as a directory. + .. versionadded:: 2.7 -.. method:: TestCase.debug() + The following attributes of a :class:`TestLoader` can be configured either by + subclassing or assignment on an instance: - Run the test without collecting the result. This allows exceptions raised by - the test to be propagated to the caller, and can be used to support running - tests under a debugger. -The test code can use any of the following methods to check for and report -failures. + .. attribute:: testMethodPrefix + String giving the prefix of method names which will be interpreted as test + methods. The default value is ``'test'``. -.. method:: TestCase.assert_(expr[, msg]) - TestCase.failUnless(expr[, msg]) - TestCase.assertTrue(expr[, msg]) + This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` + methods. - Signal a test failure if *expr* is false; the explanation for the error will be - *msg* if given, otherwise it will be :const:`None`. + .. attribute:: sortTestMethodsUsing -.. method:: TestCase.assertEqual(first, second[, msg]) - TestCase.failUnlessEqual(first, second[, msg]) + Function to be used to compare method names when sorting them in + :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods. The + default value is the built-in :func:`cmp` function; the attribute can also + be set to :const:`None` to disable the sort. - Test that *first* and *second* are equal. If the values do not compare equal, - the test will fail with the explanation given by *msg*, or :const:`None`. Note - that using :meth:`failUnlessEqual` improves upon doing the comparison as the - first parameter to :meth:`failUnless`: the default value for *msg* can be - computed to include representations of both *first* and *second*. + .. attribute:: suiteClass -.. method:: TestCase.assertNotEqual(first, second[, msg]) - TestCase.failIfEqual(first, second[, msg]) + Callable object that constructs a test suite from a list of tests. No + methods on the resulting object are needed. The default value is the + :class:`TestSuite` class. - Test that *first* and *second* are not equal. If the values do compare equal, - the test will fail with the explanation given by *msg*, or :const:`None`. Note - that using :meth:`failIfEqual` improves upon doing the comparison as the first - parameter to :meth:`failUnless` is that the default value for *msg* can be - computed to include representations of both *first* and *second*. + This affects all the :meth:`loadTestsFrom\*` methods. -.. method:: TestCase.assertAlmostEqual(first, second[, places[, msg]]) - TestCase.failUnlessAlmostEqual(first, second[, places[, msg]]) +.. class:: TestResult - Test that *first* and *second* are approximately equal by computing the - difference, rounding to the given number of decimal *places* (default 7), - and comparing to zero. - Note that comparing a given number of decimal places is not the same as - comparing a given number of significant digits. If the values do not compare - equal, the test will fail with the explanation given by *msg*, or :const:`None`. + This class is used to compile information about which tests have succeeded + and which have failed. + A :class:`TestResult` object stores the results of a set of tests. The + :class:`TestCase` and :class:`TestSuite` classes ensure that results are + properly recorded; test authors do not need to worry about recording the + outcome of tests. -.. method:: TestCase.assertNotAlmostEqual(first, second[, places[, msg]]) - TestCase.failIfAlmostEqual(first, second[, places[, msg]]) + Testing frameworks built on top of :mod:`unittest` may want access to the + :class:`TestResult` object generated by running a set of tests for reporting + purposes; a :class:`TestResult` instance is returned by the + :meth:`TestRunner.run` method for this purpose. - Test that *first* and *second* are not approximately equal by computing the - difference, rounding to the given number of decimal *places* (default 7), - and comparing to zero. - Note that comparing a given number of decimal places is not the same as - comparing a given number of significant digits. If the values do not compare - equal, the test will fail with the explanation given by *msg*, or :const:`None`. + :class:`TestResult` instances have the following attributes that will be of + interest when inspecting the results of running a set of tests: -.. method:: TestCase.assertRaises(exception, callable, ...) - TestCase.failUnlessRaises(exception, callable, ...) + .. attribute:: errors - Test that an exception is raised when *callable* is called with any positional - or keyword arguments that are also passed to :meth:`assertRaises`. The test - passes if *exception* is raised, is an error if another exception is raised, or - fails if no exception is raised. To catch any of a group of exceptions, a tuple - containing the exception classes may be passed as *exception*. + A list containing 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents a test which raised an + unexpected exception. + .. versionchanged:: 2.2 + Contains formatted tracebacks instead of :func:`sys.exc_info` results. -.. method:: TestCase.failIf(expr[, msg]) - TestCase.assertFalse(expr[, msg]) - The inverse of the :meth:`failUnless` method is the :meth:`failIf` method. This - signals a test failure if *expr* is true, with *msg* or :const:`None` for the - error message. + .. attribute:: failures + A list containing 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents a test where a failure + was explicitly signalled using the :meth:`TestCase.fail\*` or + :meth:`TestCase.assert\*` methods. -.. method:: TestCase.fail([msg]) + .. versionchanged:: 2.2 + Contains formatted tracebacks instead of :func:`sys.exc_info` results. - Signals a test failure unconditionally, with *msg* or :const:`None` for the - error message. + .. attribute:: skipped + A list containing 2-tuples of :class:`TestCase` instances and strings + holding the reason for skipping the test. -.. attribute:: TestCase.failureException + .. versionadded:: 2.7 - This class attribute gives the exception raised by the :meth:`test` method. If - a test framework needs to use a specialized exception, possibly to carry - additional information, it must subclass this exception in order to "play fair" - with the framework. The initial value of this attribute is - :exc:`AssertionError`. + .. attribute:: expectedFailures -Testing frameworks can use the following methods to collect information on the -test: + A list containing 2-tuples of :class:`TestCase` instances and strings + holding formatted tracebacks. Each tuple represents an expected failure + of the test case. + .. attribute:: unexpectedSuccesses -.. method:: TestCase.countTestCases() + A list containing :class:`TestCase` instances that were marked as expected + failures, but succeeded. - Return the number of tests represented by this test object. For - :class:`TestCase` instances, this will always be ``1``. + .. attribute:: shouldStop + Set to ``True`` when the execution of tests should stop by :meth:`stop`. -.. method:: TestCase.defaultTestResult() - Return an instance of the test result class that should be used for this test - case class (if no other result instance is provided to the :meth:`run` method). + .. attribute:: testsRun - For :class:`TestCase` instances, this will always be an instance of - :class:`TestResult`; subclasses of :class:`TestCase` should override this as - necessary. + The total number of tests run so far. -.. method:: TestCase.id() + .. attribute:: buffer - Return a string identifying the specific test case. This is usually the full - name of the test method, including the module and class name. + If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between + :meth:`startTest` and :meth:`stopTest` being called. Collected output will + only be echoed onto the real ``sys.stdout`` and ``sys.stderr`` if the test + fails or errors. Any output is also attached to the failure / error message. + .. versionadded:: 2.7 -.. method:: TestCase.shortDescription() - Returns a one-line description of the test, or :const:`None` if no description - has been provided. The default implementation of this method returns the first - line of the test method's docstring, if available, or :const:`None`. + .. attribute:: failfast + If set to true :meth:`stop` will be called on the first failure or error, + halting the test run. -.. _testsuite-objects: + .. versionadded:: 2.7 + + + .. method:: wasSuccessful() + + Return ``True`` if all tests run so far have passed, otherwise returns + ``False``. + + + .. method:: stop() + + This method can be called to signal that the set of tests being run should + be aborted by setting the :attr:`shouldStop` attribute to ``True``. + :class:`TestRunner` objects should respect this flag and return without + running any additional tests. + + For example, this feature is used by the :class:`TextTestRunner` class to + stop the test framework when the user signals an interrupt from the + keyboard. Interactive tools which provide :class:`TestRunner` + implementations can use this in a similar manner. + + The following methods of the :class:`TestResult` class are used to maintain + the internal data structures, and may be extended in subclasses to support + additional reporting requirements. This is particularly useful in building + tools which support interactive reporting while tests are being run. -TestSuite Objects ------------------ -:class:`TestSuite` objects behave much like :class:`TestCase` objects, except -they do not actually implement a test. Instead, they are used to aggregate -tests into groups of tests that should be run together. Some additional methods -are available to add tests to :class:`TestSuite` instances: + .. method:: startTest(test) + Called when the test case *test* is about to be run. -.. method:: TestSuite.addTest(test) + .. method:: stopTest(test) - Add a :class:`TestCase` or :class:`TestSuite` to the suite. + Called after the test case *test* has been executed, regardless of the + outcome. + .. method:: startTestRun(test) -.. method:: TestSuite.addTests(tests) + Called once before any tests are executed. - Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite` - instances to this test suite. + .. versionadded:: 2.7 - This is equivalent to iterating over *tests*, calling :meth:`addTest` for each - element. -:class:`TestSuite` shares the following methods with :class:`TestCase`: + .. method:: stopTestRun(test) + Called once after all tests are executed. -.. method:: TestSuite.run(result) + .. versionadded:: 2.7 - Run the tests associated with this suite, collecting the result into the test - result object passed as *result*. Note that unlike :meth:`TestCase.run`, - :meth:`TestSuite.run` requires the result object to be passed in. + .. method:: addError(test, err) -.. method:: TestSuite.debug() + Called when the test case *test* raises an unexpected exception *err* is a + tuple of the form returned by :func:`sys.exc_info`: ``(type, value, + traceback)``. - Run the tests associated with this suite without collecting the result. This - allows exceptions raised by the test to be propagated to the caller and can be - used to support running tests under a debugger. + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`errors` attribute, where *formatted_err* is a + formatted traceback derived from *err*. -.. method:: TestSuite.countTestCases() + .. method:: addFailure(test, err) - Return the number of tests represented by this test object, including all - individual tests and sub-suites. + Called when the test case *test* signals a failure. *err* is a tuple of + the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``. -In the typical usage of a :class:`TestSuite` object, the :meth:`run` method is -invoked by a :class:`TestRunner` rather than by the end-user test harness. + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`failures` attribute, where *formatted_err* is a + formatted traceback derived from *err*. -.. _testresult-objects: + .. method:: addSuccess(test) -TestResult Objects ------------------- + Called when the test case *test* succeeds. -A :class:`TestResult` object stores the results of a set of tests. The -:class:`TestCase` and :class:`TestSuite` classes ensure that results are -properly recorded; test authors do not need to worry about recording the outcome -of tests. + The default implementation does nothing. -Testing frameworks built on top of :mod:`unittest` may want access to the -:class:`TestResult` object generated by running a set of tests for reporting -purposes; a :class:`TestResult` instance is returned by the -:meth:`TestRunner.run` method for this purpose. -:class:`TestResult` instances have the following attributes that will be of -interest when inspecting the results of running a set of tests: + .. method:: addSkip(test, reason) + Called when the test case *test* is skipped. *reason* is the reason the + test gave for skipping. -.. attribute:: TestResult.errors + The default implementation appends a tuple ``(test, reason)`` to the + instance's :attr:`skipped` attribute. - A list containing 2-tuples of :class:`TestCase` instances and strings holding - formatted tracebacks. Each tuple represents a test which raised an unexpected - exception. - .. versionchanged:: 2.2 - Contains formatted tracebacks instead of :func:`sys.exc_info` results. + .. method:: addExpectedFailure(test, err) + Called when the test case *test* fails, but was marked with the + :func:`expectedFailure` decorator. -.. attribute:: TestResult.failures + The default implementation appends a tuple ``(test, formatted_err)`` to + the instance's :attr:`expectedFailures` attribute, where *formatted_err* + is a formatted traceback derived from *err*. - A list containing 2-tuples of :class:`TestCase` instances and strings holding - formatted tracebacks. Each tuple represents a test where a failure was - explicitly signalled using the :meth:`TestCase.fail\*` or - :meth:`TestCase.assert\*` methods. - .. versionchanged:: 2.2 - Contains formatted tracebacks instead of :func:`sys.exc_info` results. + .. method:: addUnexpectedSuccess(test) + Called when the test case *test* was marked with the + :func:`expectedFailure` decorator, but succeeded. -.. attribute:: TestResult.testsRun + The default implementation appends the test to the instance's + :attr:`unexpectedSuccesses` attribute. - The total number of tests run so far. +.. class:: TextTestResult(stream, descriptions, verbosity) + A concrete implementation of :class:`TestResult` used by the + :class:`TextTestRunner`. -.. method:: TestResult.wasSuccessful() + .. versionadded:: 2.7 + This class was previously named ``_TextTestResult``. The old name still + exists as an alias but is deprecated. - Returns :const:`True` if all tests run so far have passed, otherwise returns - :const:`False`. +.. data:: defaultTestLoader + Instance of the :class:`TestLoader` class intended to be shared. If no + customization of the :class:`TestLoader` is needed, this instance can be used + instead of repeatedly creating new instances. -.. method:: TestResult.stop() - This method can be called to signal that the set of tests being run should be - aborted by setting the :class:`TestResult`'s ``shouldStop`` attribute to - :const:`True`. :class:`TestRunner` objects should respect this flag and return - without running any additional tests. +.. class:: TextTestRunner(stream=sys.stderr, descriptions=True, verbosity=1) - For example, this feature is used by the :class:`TextTestRunner` class to stop - the test framework when the user signals an interrupt from the keyboard. - Interactive tools which provide :class:`TestRunner` implementations can use this - in a similar manner. + A basic test runner implementation which prints results on standard error. It + has a few configurable parameters, but is essentially very simple. Graphical + applications which run test suites should provide alternate implementations. -The following methods of the :class:`TestResult` class are used to maintain the -internal data structures, and may be extended in subclasses to support -additional reporting requirements. This is particularly useful in building -tools which support interactive reporting while tests are being run. + .. method:: _makeResult() + This method returns the instance of ``TestResult`` used by :meth:`run`. + It is not intended to be called directly, but can be overridden in + subclasses to provide a custom ``TestResult``. -.. method:: TestResult.startTest(test) + ``_makeResult()`` instantiates the class or callable passed in the + ``TextTestRunner`` constructor as the ``resultclass`` argument. It + defaults to :class:`TextTestResult` if no ``resultclass`` is provided. + The result class is instantiated with the following arguments:: - Called when the test case *test* is about to be run. + stream, descriptions, verbosity - The default implementation simply increments the instance's ``testsRun`` - counter. +.. function:: main([module[, defaultTest[, argv[, testRunner[, testLoader[, exit[, verbosity[, failfast[, catchbreak[,buffer]]]]]]]]]]) -.. method:: TestResult.stopTest(test) + A command-line program that runs a set of tests; this is primarily for making + test modules conveniently executable. The simplest use for this function is to + include the following line at the end of a test script:: - Called after the test case *test* has been executed, regardless of the outcome. + if __name__ == '__main__': + unittest.main() - The default implementation does nothing. + You can run tests with more detailed information by passing in the verbosity + argument:: + + if __name__ == '__main__': + unittest.main(verbosity=2) + + The *testRunner* argument can either be a test runner class or an already + created instance of it. By default ``main`` calls :func:`sys.exit` with + an exit code indicating success or failure of the tests run. + + ``main`` supports being used from the interactive interpreter by passing in the + argument ``exit=False``. This displays the result on standard output without + calling :func:`sys.exit`:: + + >>> from unittest import main + >>> main(module='test_module', exit=False) + + The ``failfast``, ``catchbreak`` and ``buffer`` parameters have the same + effect as the same-name `command-line options`_. + + Calling ``main`` actually returns an instance of the ``TestProgram`` class. + This stores the result of the tests run as the ``result`` attribute. + + .. versionchanged:: 2.7 + The ``exit``, ``verbosity``, ``failfast``, ``catchbreak`` and ``buffer`` + parameters were added. + + +load_tests Protocol +################### + +.. versionadded:: 2.7 + +Modules or packages can customize how tests are loaded from them during normal +test runs or test discovery by implementing a function called ``load_tests``. + +If a test module defines ``load_tests`` it will be called by +:meth:`TestLoader.loadTestsFromModule` with the following arguments:: + + load_tests(loader, standard_tests, None) + +It should return a :class:`TestSuite`. + +*loader* is the instance of :class:`TestLoader` doing the loading. +*standard_tests* are the tests that would be loaded by default from the +module. It is common for test modules to only want to add or remove tests +from the standard set of tests. +The third argument is used when loading packages as part of test discovery. + +A typical ``load_tests`` function that loads tests from a specific set of +:class:`TestCase` classes may look like:: + + test_cases = (TestCase1, TestCase2, TestCase3) + + def load_tests(loader, tests, pattern): + suite = TestSuite() + for test_class in test_cases: + tests = loader.loadTestsFromTestCase(test_class) + suite.addTests(tests) + return suite + +If discovery is started, either from the command line or by calling +:meth:`TestLoader.discover`, with a pattern that matches a package +name then the package :file:`__init__.py` will be checked for ``load_tests``. + +.. note:: + The default pattern is 'test*.py'. This matches all Python files + that start with 'test' but *won't* match any test directories. -.. method:: TestResult.addError(test, err) + A pattern like 'test*' will match test packages as well as + modules. - Called when the test case *test* raises an unexpected exception *err* is a tuple - of the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``. +If the package :file:`__init__.py` defines ``load_tests`` then it will be +called and discovery not continued into the package. ``load_tests`` +is called with the following arguments:: - The default implementation appends a tuple ``(test, formatted_err)`` to the - instance's ``errors`` attribute, where *formatted_err* is a formatted - traceback derived from *err*. + load_tests(loader, standard_tests, pattern) +This should return a :class:`TestSuite` representing all the tests +from the package. (``standard_tests`` will only contain tests +collected from :file:`__init__.py`.) -.. method:: TestResult.addFailure(test, err) +Because the pattern is passed into ``load_tests`` the package is free to +continue (and potentially modify) test discovery. A 'do nothing' +``load_tests`` function for a test package would look like:: - Called when the test case *test* signals a failure. *err* is a tuple of the form - returned by :func:`sys.exc_info`: ``(type, value, traceback)``. + def load_tests(loader, standard_tests, pattern): + # top level directory cached on loader instance + this_dir = os.path.dirname(__file__) + package_tests = loader.discover(start_dir=this_dir, pattern=pattern) + standard_tests.addTests(package_tests) + return standard_tests - The default implementation appends a tuple ``(test, formatted_err)`` to the - instance's ``failures`` attribute, where *formatted_err* is a formatted - traceback derived from *err*. +Class and Module Fixtures +------------------------- -.. method:: TestResult.addSuccess(test) +Class and module level fixtures are implemented in :class:`TestSuite`. When +the test suite encounters a test from a new class then :meth:`tearDownClass` +from the previous class (if there is one) is called, followed by +:meth:`setUpClass` from the new class. - Called when the test case *test* succeeds. +Similarly if a test is from a different module from the previous test then +``tearDownModule`` from the previous module is run, followed by +``setUpModule`` from the new module. - The default implementation does nothing. +After all the tests have run the final ``tearDownClass`` and +``tearDownModule`` are run. +Note that shared fixtures do not play well with [potential] features like test +parallelization and they break test isolation. They should be used with care. -.. _testloader-objects: +The default ordering of tests created by the unittest test loaders is to group +all tests from the same modules and classes together. This will lead to +``setUpClass`` / ``setUpModule`` (etc) being called exactly once per class and +module. If you randomize the order, so that tests from different modules and +classes are adjacent to each other, then these shared fixture functions may be +called multiple times in a single test run. -TestLoader Objects ------------------- +Shared fixtures are not intended to work with suites with non-standard +ordering. A ``BaseTestSuite`` still exists for frameworks that don't want to +support shared fixtures. -The :class:`TestLoader` class is used to create test suites from classes and -modules. Normally, there is no need to create an instance of this class; the -:mod:`unittest` module provides an instance that can be shared as -``unittest.defaultTestLoader``. Using a subclass or instance, however, allows -customization of some configurable properties. +If there are any exceptions raised during one of the shared fixture functions +the test is reported as an error. Because there is no corresponding test +instance an ``_ErrorHolder`` object (that has the same interface as a +:class:`TestCase`) is created to represent the error. If you are just using +the standard unittest test runner then this detail doesn't matter, but if you +are a framework author it may be relevant. -:class:`TestLoader` objects have the following methods: +setUpClass and tearDownClass +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. method:: TestLoader.loadTestsFromTestCase(testCaseClass) +These must be implemented as class methods:: - Return a suite of all tests cases contained in the :class:`TestCase`\ -derived - :class:`testCaseClass`. + import unittest + class Test(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls._connection = createExpensiveConnectionObject() -.. method:: TestLoader.loadTestsFromModule(module) + @classmethod + def tearDownClass(cls): + cls._connection.destroy() - Return a suite of all tests cases contained in the given module. This method - searches *module* for classes derived from :class:`TestCase` and creates an - instance of the class for each test method defined for the class. +If you want the ``setUpClass`` and ``tearDownClass`` on base classes called +then you must call up to them yourself. The implementations in +:class:`TestCase` are empty. - .. warning:: +If an exception is raised during a ``setUpClass`` then the tests in the class +are not run and the ``tearDownClass`` is not run. Skipped classes will not +have ``setUpClass`` or ``tearDownClass`` run. If the exception is a +``SkipTest`` exception then the class will be reported as having been skipped +instead of as an error. - While using a hierarchy of :class:`TestCase`\ -derived classes can be convenient - in sharing fixtures and helper functions, defining test methods on base classes - that are not intended to be instantiated directly does not play well with this - method. Doing so, however, can be useful when the fixtures are different and - defined in subclasses. +setUpModule and tearDownModule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. method:: TestLoader.loadTestsFromName(name[, module]) +These should be implemented as functions:: - Return a suite of all tests cases given a string specifier. + def setUpModule(): + createConnection() - The specifier *name* is a "dotted name" that may resolve either to a module, a - test case class, a test method within a test case class, a :class:`TestSuite` - instance, or a callable object which returns a :class:`TestCase` or - :class:`TestSuite` instance. These checks are applied in the order listed here; - that is, a method on a possible test case class will be picked up as "a test - method within a test case class", rather than "a callable object". + def tearDownModule(): + closeConnection() - For example, if you have a module :mod:`SampleTests` containing a - :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test - methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the - specifier ``'SampleTests.SampleTestCase'`` would cause this method to return a - suite which will run all three test methods. Using the specifier - ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test suite - which will run only the :meth:`test_two` test method. The specifier can refer - to modules and packages which have not been imported; they will be imported as a - side-effect. +If an exception is raised in a ``setUpModule`` then none of the tests in the +module will be run and the ``tearDownModule`` will not be run. If the exception is a +``SkipTest`` exception then the module will be reported as having been skipped +instead of as an error. - The method optionally resolves *name* relative to the given *module*. +Signal Handling +--------------- -.. method:: TestLoader.loadTestsFromNames(names[, module]) +The :option:`-c/--catch <unittest -c>` command-line option to unittest, +along with the ``catchbreak`` parameter to :func:`unittest.main()`, provide +more friendly handling of control-C during a test run. With catch break +behavior enabled control-C will allow the currently running test to complete, +and the test run will then end and report all the results so far. A second +control-c will raise a :exc:`KeyboardInterrupt` in the usual way. - Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather than - a single name. The return value is a test suite which supports all the tests - defined for each name. +The control-c handling signal handler attempts to remain compatible with code or +tests that install their own :const:`signal.SIGINT` handler. If the ``unittest`` +handler is called but *isn't* the installed :const:`signal.SIGINT` handler, +i.e. it has been replaced by the system under test and delegated to, then it +calls the default handler. This will normally be the expected behavior by code +that replaces an installed handler and delegates to it. For individual tests +that need ``unittest`` control-c handling disabled the :func:`removeHandler` +decorator can be used. +There are a few utility functions for framework authors to enable control-c +handling functionality within test frameworks. -.. method:: TestLoader.getTestCaseNames(testCaseClass) +.. function:: installHandler() - Return a sorted sequence of method names found within *testCaseClass*; this - should be a subclass of :class:`TestCase`. + Install the control-c handler. When a :const:`signal.SIGINT` is received + (usually in response to the user pressing control-c) all registered results + have :meth:`~TestResult.stop` called. -The following attributes of a :class:`TestLoader` can be configured either by -subclassing or assignment on an instance: + .. versionadded:: 2.7 +.. function:: registerResult(result) -.. attribute:: TestLoader.testMethodPrefix + Register a :class:`TestResult` object for control-c handling. Registering a + result stores a weak reference to it, so it doesn't prevent the result from + being garbage collected. - String giving the prefix of method names which will be interpreted as test - methods. The default value is ``'test'``. + Registering a :class:`TestResult` object has no side-effects if control-c + handling is not enabled, so test frameworks can unconditionally register + all results they create independently of whether or not handling is enabled. - This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` - methods. + .. versionadded:: 2.7 +.. function:: removeResult(result) -.. attribute:: TestLoader.sortTestMethodsUsing + Remove a registered result. Once a result has been removed then + :meth:`~TestResult.stop` will no longer be called on that result object in + response to a control-c. - Function to be used to compare method names when sorting them in - :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods. The - default value is the built-in :func:`cmp` function; the attribute can also be - set to :const:`None` to disable the sort. + .. versionadded:: 2.7 +.. function:: removeHandler(function=None) -.. attribute:: TestLoader.suiteClass + When called without arguments this function removes the control-c handler + if it has been installed. This function can also be used as a test decorator + to temporarily remove the handler whilst the test is being executed:: - Callable object that constructs a test suite from a list of tests. No methods on - the resulting object are needed. The default value is the :class:`TestSuite` - class. + @unittest.removeHandler + def test_signal_handling(self): + ... - This affects all the :meth:`loadTestsFrom\*` methods. + .. versionadded:: 2.7 diff --git a/Doc/library/urllib.rst b/Doc/library/urllib.rst index 98c55d5..9a6febc 100644 --- a/Doc/library/urllib.rst +++ b/Doc/library/urllib.rst @@ -23,6 +23,10 @@ built-in function :func:`open`, but accepts Universal Resource Locators (URLs) instead of filenames. Some restrictions apply --- it can only open URLs for reading, and no seek operations are available. +.. warning:: When opening HTTPS URLs, it is not attempted to validate the + server certificate. Use at your own risk! + + High-level interface -------------------- @@ -236,8 +240,8 @@ Utility functions .. function:: urlencode(query[, doseq]) - Convert a mapping object or a sequence of two-element tuples to a - "url-encoded" string, suitable to pass to :func:`urlopen` above as the + Convert a mapping object or a sequence of two-element tuples to a + "percent-encoded" string, suitable to pass to :func:`urlopen` above as the optional *data* argument. This is useful to pass a dictionary of form fields to a ``POST`` request. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` characters, where both *key* and @@ -262,7 +266,7 @@ Utility functions .. function:: url2pathname(path) - Convert the path component *path* from an encoded URL to the local syntax for a + Convert the path component *path* from an percent-encoded URL to the local syntax for a path. This does not accept a complete URL. This function uses :func:`unquote` to decode *path*. @@ -447,7 +451,7 @@ URL Opener objects you try to fetch a file whose read permissions make it inaccessible; the FTP code will try to read it, fail with a 550 error, and then perform a directory listing for the unreadable file. If fine-grained control is needed, consider - using the :mod:`ftplib` module, subclassing :class:`FancyURLOpener`, or changing + using the :mod:`ftplib` module, subclassing :class:`FancyURLopener`, or changing *_urlopener* to meet your needs. * This module does not support the use of proxies which require authentication. diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst index b8f2277..cef96a4 100644 --- a/Doc/library/urllib2.rst +++ b/Doc/library/urllib2.rst @@ -18,6 +18,7 @@ The :mod:`urllib2` module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world --- basic and digest authentication, redirections, cookies and more. + The :mod:`urllib2` module defines the following functions: @@ -25,18 +26,22 @@ The :mod:`urllib2` module defines the following functions: Open the URL *url*, which can be either a string or a :class:`Request` object. + .. warning:: + HTTPS requests do not do any verification of the server's certificate. + *data* may be a string specifying additional data to send to the server, or ``None`` if no such data is needed. Currently HTTP requests are the only ones that use *data*; the HTTP request will be a POST instead of a GET when the *data* parameter is provided. *data* should be a buffer in the standard :mimetype:`application/x-www-form-urlencoded` format. The :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and - returns a string in this format. + returns a string in this format. urllib2 module sends HTTP/1.1 requests with + `Connection:close` header included. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default - timeout setting will be used). This actually only works for HTTP, HTTPS, - FTP and FTPS connections. + timeout setting will be used). This actually only works for HTTP, HTTPS and + FTP connections. This function returns a file-like object with two additional methods: @@ -428,7 +433,7 @@ OpenerDirector Objects optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). The timeout feature actually works only for - HTTP, HTTPS, FTP and FTPS connections). + HTTP, HTTPS and FTP connections). .. versionchanged:: 2.6 *timeout* was added. diff --git a/Doc/library/urlparse.rst b/Doc/library/urlparse.rst index 47a7e57..88dafa9 100644 --- a/Doc/library/urlparse.rst +++ b/Doc/library/urlparse.rst @@ -33,6 +33,11 @@ following URL schemes: ``file``, ``ftp``, ``gopher``, ``hdl``, ``http``, .. versionadded:: 2.5 Support for the ``sftp`` and ``sips`` schemes. +.. seealso:: + + Latest version of the `urlparse module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/urlparse.py?view=markup>`_ + The :mod:`urlparse` module defines the following functions: @@ -58,6 +63,23 @@ The :mod:`urlparse` module defines the following functions: >>> o.geturl() 'http://www.cwi.nl:80/%7Eguido/Python.html' + + Following the syntax specifications in :rfc:`1808`, urlparse recognizes + a netloc only if it is properly introduced by '//'. Otherwise the + input is presumed to be a relative URL and thus to start with + a path component. + + >>> from urlparse import urlparse + >>> urlparse('//www.cwi.nl:80/%7Eguido/Python.html') + ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', + params='', query='', fragment='') + >>> urlparse('www.cwi.nl:80/%7Eguido/Python.html') + ParseResult(scheme='', netloc='', path='www.cwi.nl:80/%7Eguido/Python.html', + params='', query='', fragment='') + >>> urlparse('help/Python.html') + ParseResult(scheme='', netloc='', path='help/Python.html', params='', + query='', fragment='') + If the *scheme* argument is specified, it gives the default addressing scheme, to be used only if the URL does not specify one. The default value for this argument is the empty string. @@ -101,6 +123,10 @@ The :mod:`urlparse` module defines the following functions: .. versionchanged:: 2.5 Added attributes to return value. + .. versionchanged:: 2.7 + Added IPv6 URL parsing capabilities. + + .. function:: parse_qs(qs[, keep_blank_values[, strict_parsing]]) Parse a query string given as a string argument (data of type @@ -109,7 +135,7 @@ The :mod:`urlparse` module defines the following functions: values are lists of values for each name. The optional argument *keep_blank_values* is a flag indicating whether blank - values in URL encoded queries should be treated as blank strings. A true value + values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. @@ -132,7 +158,7 @@ The :mod:`urlparse` module defines the following functions: name, value pairs. The optional argument *keep_blank_values* is a flag indicating whether blank - values in URL encoded queries should be treated as blank strings. A true value + values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. @@ -254,9 +280,12 @@ The :mod:`urlparse` module defines the following functions: :rfc:`3986` - Uniform Resource Identifiers This is the current standard (STD66). Any changes to urlparse module should conform to this. Certain deviations could be observed, which are - mostly due backward compatiblity purposes and for certain to de-facto + mostly due backward compatiblity purposes and for certain de-facto parsing requirements as commonly observed in major browsers. + :rfc:`2732` - Format for Literal IPv6 Addresses in URL's. + This specifies the parsing requirements of IPv6 URLs. + :rfc:`2396` - Uniform Resource Identifiers (URI): Generic Syntax Document describing the generic syntactic requirements for both Uniform Resource Names (URNs) and Uniform Resource Locators (URLs). diff --git a/Doc/library/userdict.rst b/Doc/library/userdict.rst index 9ddfde5..ab9ab04 100644 --- a/Doc/library/userdict.rst +++ b/Doc/library/userdict.rst @@ -19,6 +19,11 @@ available starting with Python version 2.2). Prior to the introduction of sub-classes that obtained new behaviors by overriding existing methods or adding new ones. +.. seealso:: + + Latest version of the `UserDict Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/UserDict.py?view=markup>`_ + The :mod:`UserDict` module defines the :class:`UserDict` class and :class:`DictMixin`: diff --git a/Doc/library/uu.rst b/Doc/library/uu.rst index e2303c3..fa1477b 100644 --- a/Doc/library/uu.rst +++ b/Doc/library/uu.rst @@ -22,6 +22,11 @@ that, when required, the mode is ``'rb'`` or ``'wb'`` on Windows. This code was contributed by Lance Ellinghouse, and modified by Jack Jansen. +.. seealso:: + + Latest version of the `uu module Python source code + <http://svn.python.org/view/python/branches/release27-maint/Lib/uu.py?view=markup>`_ + The :mod:`uu` module defines the following functions: diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 86ca513..544d9f7 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -59,7 +59,7 @@ following warnings category classes are currently defined: | :exc:`UserWarning` | The default category for :func:`warn`. | +----------------------------------+-----------------------------------------------+ | :exc:`DeprecationWarning` | Base category for warnings about deprecated | -| | features. | +| | features (ignored by default). | +----------------------------------+-----------------------------------------------+ | :exc:`SyntaxWarning` | Base category for warnings about dubious | | | syntactic features. | @@ -89,6 +89,9 @@ User code can define additional warning categories by subclassing one of the standard warning categories. A warning category must always be a subclass of the :exc:`Warning` class. +.. versionchanged:: 2.7 + :exc:`DeprecationWarning` is ignored by default. + .. _warning-filter: @@ -148,13 +151,19 @@ interpreter command line. The interpreter saves the arguments for all :mod:`warnings` module parses these when it is first imported (invalid options are ignored, after printing a message to ``sys.stderr``). -The warnings that are ignored by default may be enabled by passing :option:`-Wd` -to the interpreter. This enables default handling for all warnings, including -those that are normally ignored by default. This is particular useful for -enabling ImportWarning when debugging problems importing a developed package. -ImportWarning can also be enabled explicitly in Python code using:: - warnings.simplefilter('default', ImportWarning) +Default Warning Filters +~~~~~~~~~~~~~~~~~~~~~~~ + +By default, Python installs several warning filters, which can be overridden by +the command-line options passed to :option:`-W` and calls to +:func:`filterwarnings`. + +* :exc:`PendingDeprecationWarning`, and :exc:`ImportWarning` are ignored. + +* :exc:`BytesWarning` is ignored unless the :option:`-b` option is given once or + twice; in this case this warning is either printed (``-b``) or turned into an + exception (``-bb``). .. _warning-suppress: @@ -231,6 +240,37 @@ continues to increase after each operation, or else delete the previous entries from the warnings list before each new operation). +Updating Code For New Versions of Python +---------------------------------------- + +Warnings that are only of interest to the developer are ignored by default. As +such you should make sure to test your code with typically ignored warnings +made visible. You can do this from the command-line by passing :option:`-Wd` +to the interpreter (this is shorthand for :option:`-W default`). This enables +default handling for all warnings, including those that are ignored by default. +To change what action is taken for encountered warnings you simply change what +argument is passed to :option:`-W`, e.g. :option:`-W error`. See the +:option:`-W` flag for more details on what is possible. + +To programmatically do the same as :option:`-Wd`, use:: + + warnings.simplefilter('default') + +Make sure to execute this code as soon as possible. This prevents the +registering of what warnings have been raised from unexpectedly influencing how +future warnings are treated. + +Having certain warnings ignored by default is done to prevent a user from +seeing warnings that are only of interest to the developer. As you do not +necessarily have control over what interpreter a user uses to run their code, +it is possible that a new version of Python will be released between your +release cycles. The new interpreter release could trigger new warnings in your +code that were not there in an older interpreter, e.g. +:exc:`DeprecationWarning` for a module that you are using. While you as a +developer want to be notified that your code is using a deprecated module, to a +user this information is essentially noise and provides no benefit to them. + + .. _warning-functions: Available Functions @@ -297,9 +337,8 @@ Available Functions message; if *line* is not supplied, :func:`showwarning` will try to read the line specified by *filename* and *lineno*. - .. versionchanged:: 2.6 - Added the *line* argument. Implementations that lack the new argument - will trigger a :exc:`DeprecationWarning`. + .. versionchanged:: 2.7 + The *line* argument is required to be supported. .. function:: formatwarning(message, category, filename, lineno[, line]) diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst index 3be0343..c1db523 100644 --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -14,8 +14,8 @@ The :mod:`wave` module defines the following function and exception: .. function:: open(file[, mode]) - If *file* is a string, open the file by that name, other treat it as a seekable - file-like object. *mode* can be any of + If *file* is a string, open the file by that name, otherwise treat it as a + seekable file-like object. *mode* can be any of ``'r'``, ``'rb'`` Read only mode. @@ -26,9 +26,14 @@ The :mod:`wave` module defines the following function and exception: Note that it does not allow read/write WAV files. A *mode* of ``'r'`` or ``'rb'`` returns a :class:`Wave_read` object, while a - *mode* of ``'w'`` or ``'wb'`` returns a :class:`Wave_write` object. If *mode* - is omitted and a file-like object is passed as *file*, ``file.mode`` is used as - the default value for *mode* (the ``'b'`` flag is still added if necessary). + *mode* of ``'w'`` or ``'wb'`` returns a :class:`Wave_write` object. If + *mode* is omitted and a file-like object is passed as *file*, ``file.mode`` + is used as the default value for *mode* (the ``'b'`` flag is still added if + necessary). + + If you pass in a file-like object, the wave object will not close it when its + :meth:`close` method is called; it is the caller's responsibility to close + the file object. .. function:: openfp(file, mode) @@ -52,8 +57,8 @@ Wave_read objects, as returned by :func:`.open`, have the following methods: .. method:: Wave_read.close() - Close the stream, and make the instance unusable. This is called automatically - on object collection. + Close the stream if it was opened by :mod:`wave`, and make the instance + unusable. This is called automatically on object collection. .. method:: Wave_read.getnchannels() @@ -139,8 +144,8 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: .. method:: Wave_write.close() - Make sure *nframes* is correct, and close the file. This method is called upon - deletion. + Make sure *nframes* is correct, and close the file if it was opened by + :mod:`wave`. This method is called upon object collection. .. method:: Wave_write.setnchannels(n) @@ -192,6 +197,7 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: Write audio frames and make sure *nframes* is correct. + Note that it is invalid to set any parameters after calling :meth:`writeframes` or :meth:`writeframesraw`, and any attempt to do so will raise :exc:`wave.Error`. diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index d403c3c..9c65587 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -59,11 +59,14 @@ Not all objects can be weakly referenced; those objects which can include class instances, functions written in Python (but not in C), methods (both bound and unbound), sets, frozensets, file objects, :term:`generator`\s, type objects, :class:`DBcursor` objects from the :mod:`bsddb` module, sockets, arrays, deques, -and regular expression pattern objects. +regular expression pattern objects, and code objects. .. versionchanged:: 2.4 Added support for files, sockets, arrays, and patterns. +.. versionchanged:: 2.7 + Added support for thread.lock, threading.Lock, and code objects. + Several built-in types such as :class:`list` and :class:`dict` do not directly support weak references but can add support through subclassing:: @@ -206,6 +209,14 @@ methods of :class:`WeakKeyDictionary` objects. .. versionadded:: 2.5 +.. class:: WeakSet([elements]) + + Set class that keeps weak references to its elements. An element will be + discarded when no strong reference to it exists any more. + + .. versionadded:: 2.7 + + .. data:: ReferenceType The type object for weak references objects. diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst index 20f6231..74e12a2 100644 --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -32,8 +32,8 @@ browser and wait. The script :program:`webbrowser` can be used as a command-line interface for the module. It accepts an URL as the argument. It accepts the following optional -parameters: :option:`-n` opens the URL in a new browser window, if possible; -:option:`-t` opens the URL in a new browser page ("tab"). The options are, +parameters: ``-n`` opens the URL in a new browser window, if possible; +``-t`` opens the URL in a new browser page ("tab"). The options are, naturally, mutually exclusive. The following exception is defined: @@ -68,7 +68,6 @@ The following functions are defined: Open *url* in a new window of the default browser, if possible, otherwise, open *url* in the only browser window. - .. function:: open_new_tab(url) Open *url* in a new page ("tab") of the default browser, if possible, otherwise diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index 517fc38..3163497 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -712,7 +712,7 @@ This is a working "Hello World" WSGI application:: # use a function (note that you're not limited to a function, you can # use a class for example). The first argument passed to the function # is a dictionary containing CGI-style envrironment variables and the - # second variable is the callable object (see :pep:`333`) + # second variable is the callable object (see PEP 333). def hello_world_app(environ, start_response): status = '200 OK' # HTTP Status headers = [('Content-type', 'text/plain')] # HTTP Headers diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index cfde5b9..064ddd4 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -118,7 +118,7 @@ module documentation. This section lists the differences between the API and to discard children of that node. -.. method:: Node.writexml(writer[, indent=""[, addindent=""[, newl=""[, encoding=""]]]]) +.. method:: Node.writexml(writer[, indent=""[, addindent=""[, newl=""]]]) Write XML to the writer object. The writer should have a :meth:`write` method which matches that of the file object interface. The *indent* parameter is the @@ -126,6 +126,9 @@ module documentation. This section lists the differences between the API and indentation to use for subnodes of the current one. The *newl* parameter specifies the string to use to terminate newlines. + For the :class:`Document` node, an additional keyword argument *encoding* can + be used to specify the encoding field of the XML header. + .. versionchanged:: 2.1 The optional keyword parameters *indent*, *addindent*, and *newl* were added to support pretty output. diff --git a/Doc/library/xml.dom.rst b/Doc/library/xml.dom.rst index 0f91723..1069615 100644 --- a/Doc/library/xml.dom.rst +++ b/Doc/library/xml.dom.rst @@ -705,18 +705,27 @@ Attr Objects .. attribute:: Attr.name - The attribute name. In a namespace-using document it may have colons in it. + The attribute name. + In a namespace-using document it may include a colon. .. attribute:: Attr.localName - The part of the name following the colon if there is one, else the entire name. + The part of the name following the colon if there is one, else the + entire name. This is a read-only attribute. .. attribute:: Attr.prefix - The part of the name preceding the colon if there is one, else the empty string. + The part of the name preceding the colon if there is one, else the + empty string. + + +.. attribute:: Attr.value + + The text value of the attribute. This is a synonym for the + :attr:`nodeValue` attribute. .. _dom-attributelist-objects: diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index e074d55..def4669 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -9,9 +9,9 @@ .. versionadded:: 2.5 -The Element type is a flexible container object, designed to store hierarchical -data structures in memory. The type can be described as a cross between a list -and a dictionary. +The :class:`Element` type is a flexible container object, designed to store +hierarchical data structures in memory. The type can be described as a cross +between a list and a dictionary. Each element has a number of properties associated with it: @@ -26,7 +26,8 @@ Each element has a number of properties associated with it: * a number of child elements, stored in a Python sequence -To create an element instance, use the Element or SubElement factory functions. +To create an element instance, use the :class:`Element` constructor or the +:func:`SubElement` factory function. The :class:`ElementTree` class can be used to wrap an element structure, and convert it from and to XML. @@ -34,8 +35,14 @@ convert it from and to XML. A C implementation of this API is available as :mod:`xml.etree.cElementTree`. See http://effbot.org/zone/element-index.htm for tutorials and links to other -docs. Fredrik Lundh's page is also the location of the development version of the -xml.etree.ElementTree. +docs. Fredrik Lundh's page is also the location of the development version of +the xml.etree.ElementTree. + +.. versionchanged:: 2.7 + The ElementTree API is updated to 1.3. For more information, see + `Introducing ElementTree 1.3 + <http://effbot.org/zone/elementtree-13-intro.htm>`_. + .. _elementtree-functions: @@ -43,18 +50,19 @@ Functions --------- -.. function:: Comment([text]) +.. function:: Comment(text=None) - Comment element factory. This factory function creates a special element that - will be serialized as an XML comment. The comment string can be either an 8-bit - ASCII string or a Unicode string. *text* is a string containing the comment - string. Returns an element instance representing a comment. + Comment element factory. This factory function creates a special element + that will be serialized as an XML comment by the standard serializer. The + comment string can be either a bytestring or a Unicode string. *text* is a + string containing the comment string. Returns an element instance + representing a comment. .. function:: dump(elem) - Writes an element tree or element structure to sys.stdout. This function should - be used for debugging only. + Writes an element tree or element structure to sys.stdout. This function + should be used for debugging only. The exact output format is implementation dependent. In this version, it's written as an ordinary XML file. @@ -62,37 +70,36 @@ Functions *elem* is an element tree or an individual element. -.. function:: Element(tag[, attrib][, **extra]) +.. function:: fromstring(text) - Element factory. This function returns an object implementing the standard - Element interface. The exact class or type of that object is implementation - dependent, but it will always be compatible with the _ElementInterface class in - this module. + Parses an XML section from a string constant. Same as :func:`XML`. *text* + is a string containing XML data. Returns an :class:`Element` instance. - The element name, attribute names, and attribute values can be either 8-bit - ASCII strings or Unicode strings. *tag* is the element name. *attrib* is an - optional dictionary, containing element attributes. *extra* contains additional - attributes, given as keyword arguments. Returns an element instance. +.. function:: fromstringlist(sequence, parser=None) -.. function:: fromstring(text) + Parses an XML document from a sequence of string fragments. *sequence* is a + list or other sequence containing XML data fragments. *parser* is an + optional parser instance. If not given, the standard :class:`XMLParser` + parser is used. Returns an :class:`Element` instance. - Parses an XML section from a string constant. Same as XML. *text* is a string - containing XML data. Returns an Element instance. + .. versionadded:: 2.7 .. function:: iselement(element) - Checks if an object appears to be a valid element object. *element* is an - element instance. Returns a true value if this is an element object. + Checks if an object appears to be a valid element object. *element* is an + element instance. Returns a true value if this is an element object. -.. function:: iterparse(source[, events]) +.. function:: iterparse(source, events=None, parser=None) Parses an XML section into an element tree incrementally, and reports what's - going on to the user. *source* is a filename or file object containing XML data. - *events* is a list of events to report back. If omitted, only "end" events are - reported. Returns an :term:`iterator` providing ``(event, elem)`` pairs. + going on to the user. *source* is a filename or file object containing XML + data. *events* is a list of events to report back. If omitted, only "end" + events are reported. *parser* is an optional parser instance. If not + given, the standard :class:`XMLParser` parser is used. Returns an + :term:`iterator` providing ``(event, elem)`` pairs. .. note:: @@ -105,198 +112,271 @@ Functions If you need a fully populated element, look for "end" events instead. -.. function:: parse(source[, parser]) +.. function:: parse(source, parser=None) + + Parses an XML section into an element tree. *source* is a filename or file + object containing XML data. *parser* is an optional parser instance. If + not given, the standard :class:`XMLParser` parser is used. Returns an + :class:`ElementTree` instance. + + +.. function:: ProcessingInstruction(target, text=None) + + PI element factory. This factory function creates a special element that + will be serialized as an XML processing instruction. *target* is a string + containing the PI target. *text* is a string containing the PI contents, if + given. Returns an element instance, representing a processing instruction. + + +.. function:: register_namespace(prefix, uri) + + Registers a namespace prefix. The registry is global, and any existing + mapping for either the given prefix or the namespace URI will be removed. + *prefix* is a namespace prefix. *uri* is a namespace uri. Tags and + attributes in this namespace will be serialized with the given prefix, if at + all possible. + + .. versionadded:: 2.7 - Parses an XML section into an element tree. *source* is a filename or file - object containing XML data. *parser* is an optional parser instance. If not - given, the standard XMLTreeBuilder parser is used. Returns an ElementTree - instance. +.. function:: SubElement(parent, tag, attrib={}, **extra) -.. function:: ProcessingInstruction(target[, text]) + Subelement factory. This function creates an element instance, and appends + it to an existing element. - PI element factory. This factory function creates a special element that will - be serialized as an XML processing instruction. *target* is a string containing - the PI target. *text* is a string containing the PI contents, if given. Returns - an element instance, representing a processing instruction. + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *parent* is the parent element. *tag* is + the subelement name. *attrib* is an optional dictionary, containing element + attributes. *extra* contains additional attributes, given as keyword + arguments. Returns an element instance. -.. function:: SubElement(parent, tag[, attrib[, **extra]]) +.. function:: tostring(element, encoding="us-ascii", method="xml") - Subelement factory. This function creates an element instance, and appends it - to an existing element. + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns an encoded string + containing the XML data. - The element name, attribute names, and attribute values can be either 8-bit - ASCII strings or Unicode strings. *parent* is the parent element. *tag* is the - subelement name. *attrib* is an optional dictionary, containing element - attributes. *extra* contains additional attributes, given as keyword arguments. - Returns an element instance. +.. function:: tostringlist(element, encoding="us-ascii", method="xml") -.. function:: tostring(element[, encoding]) + Generates a string representation of an XML element, including all + subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is + the output encoding (default is US-ASCII). *method* is either ``"xml"``, + ``"html"`` or ``"text"`` (default is ``"xml"``). Returns a list of encoded + strings containing the XML data. It does not guarantee any specific + sequence, except that ``"".join(tostringlist(element)) == + tostring(element)``. - Generates a string representation of an XML element, including all subelements. - *element* is an Element instance. *encoding* is the output encoding (default is - US-ASCII). Returns an encoded string containing the XML data. + .. versionadded:: 2.7 -.. function:: XML(text) +.. function:: XML(text, parser=None) Parses an XML section from a string constant. This function can be used to - embed "XML literals" in Python code. *text* is a string containing XML data. - Returns an Element instance. + embed "XML literals" in Python code. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns an :class:`Element` instance. -.. function:: XMLID(text) +.. function:: XMLID(text, parser=None) Parses an XML section from a string constant, and also returns a dictionary - which maps from element id:s to elements. *text* is a string containing XML - data. Returns a tuple containing an Element instance and a dictionary. + which maps from element id:s to elements. *text* is a string containing XML + data. *parser* is an optional parser instance. If not given, the standard + :class:`XMLParser` parser is used. Returns a tuple containing an + :class:`Element` instance and a dictionary. + + +.. _elementtree-element-objects: + +Element Objects +--------------- + + +.. class:: Element(tag, attrib={}, **extra) + + Element class. This class defines the Element interface, and provides a + reference implementation of this interface. + + The element name, attribute names, and attribute values can be either + bytestrings or Unicode strings. *tag* is the element name. *attrib* is + an optional dictionary, containing element attributes. *extra* contains + additional attributes, given as keyword arguments. + + + .. attribute:: tag + + A string identifying what kind of data this element represents (the + element type, in other words). + + + .. attribute:: text + The *text* attribute can be used to hold additional data associated with + the element. As the name implies this attribute is usually a string but + may be any application-specific object. If the element is created from + an XML file the attribute will contain any text found between the element + tags. -.. _elementtree-element-interface: -The Element Interface ---------------------- + .. attribute:: tail -Element objects returned by Element or SubElement have the following methods -and attributes. + The *tail* attribute can be used to hold additional data associated with + the element. This attribute is usually a string but may be any + application-specific object. If the element is created from an XML file + the attribute will contain any text found after the element's end tag and + before the next tag. -.. attribute:: Element.tag + .. attribute:: attrib - A string identifying what kind of data this element represents (the element - type, in other words). + A dictionary containing the element's attributes. Note that while the + *attrib* value is always a real mutable Python dictionary, an ElementTree + implementation may choose to use another internal representation, and + create the dictionary only if someone asks for it. To take advantage of + such implementations, use the dictionary methods below whenever possible. + The following dictionary-like methods work on the element attributes. -.. attribute:: Element.text - The *text* attribute can be used to hold additional data associated with the - element. As the name implies this attribute is usually a string but may be any - application-specific object. If the element is created from an XML file the - attribute will contain any text found between the element tags. + .. method:: clear() + Resets an element. This function removes all subelements, clears all + attributes, and sets the text and tail attributes to None. -.. attribute:: Element.tail - The *tail* attribute can be used to hold additional data associated with the - element. This attribute is usually a string but may be any application-specific - object. If the element is created from an XML file the attribute will contain - any text found after the element's end tag and before the next tag. + .. method:: get(key, default=None) + Gets the element attribute named *key*. -.. attribute:: Element.attrib + Returns the attribute value, or *default* if the attribute was not found. - A dictionary containing the element's attributes. Note that while the *attrib* - value is always a real mutable Python dictionary, an ElementTree implementation - may choose to use another internal representation, and create the dictionary - only if someone asks for it. To take advantage of such implementations, use the - dictionary methods below whenever possible. -The following dictionary-like methods work on the element attributes. + .. method:: items() + Returns the element attributes as a sequence of (name, value) pairs. The + attributes are returned in an arbitrary order. -.. method:: Element.clear() - Resets an element. This function removes all subelements, clears all - attributes, and sets the text and tail attributes to None. + .. method:: keys() + Returns the elements attribute names as a list. The names are returned + in an arbitrary order. -.. method:: Element.get(key[, default=None]) - Gets the element attribute named *key*. + .. method:: set(key, value) - Returns the attribute value, or *default* if the attribute was not found. + Set the attribute *key* on the element to *value*. + The following methods work on the element's children (subelements). -.. method:: Element.items() - Returns the element attributes as a sequence of (name, value) pairs. The - attributes are returned in an arbitrary order. + .. method:: append(subelement) + Adds the element *subelement* to the end of this elements internal list + of subelements. -.. method:: Element.keys() - Returns the elements attribute names as a list. The names are returned in an - arbitrary order. + .. method:: extend(subelements) + Appends *subelements* from a sequence object with zero or more elements. + Raises :exc:`AssertionError` if a subelement is not a valid object. -.. method:: Element.set(key, value) + .. versionadded:: 2.7 - Set the attribute *key* on the element to *value*. -The following methods work on the element's children (subelements). + .. method:: find(match) + Finds the first subelement matching *match*. *match* may be a tag name + or path. Returns an element instance or ``None``. -.. method:: Element.append(subelement) - Adds the element *subelement* to the end of this elements internal list of - subelements. + .. method:: findall(match) + Finds all matching subelements, by tag name or path. Returns a list + containing all matching elements in document order. -.. method:: Element.find(match) - Finds the first subelement matching *match*. *match* may be a tag name or path. - Returns an element instance or ``None``. + .. method:: findtext(match, default=None) + Finds text for the first subelement matching *match*. *match* may be + a tag name or path. Returns the text content of the first matching + element, or *default* if no element was found. Note that if the matching + element has no text content an empty string is returned. -.. method:: Element.findall(match) - Finds all subelements matching *match*. *match* may be a tag name or path. - Returns an iterable yielding all matching elements in document order. + .. method:: getchildren() + .. deprecated:: 2.7 + Use ``list(elem)`` or iteration. -.. method:: Element.findtext(condition[, default=None]) - Finds text for the first subelement matching *condition*. *condition* may be a - tag name or path. Returns the text content of the first matching element, or - *default* if no element was found. Note that if the matching element has no - text content an empty string is returned. + .. method:: getiterator(tag=None) + .. deprecated:: 2.7 + Use method :meth:`Element.iter` instead. -.. method:: Element.getchildren() - Returns all subelements. The elements are returned in document order. + .. method:: insert(index, element) + Inserts a subelement at the given position in this element. -.. method:: Element.getiterator([tag=None]) - Creates a tree iterator with the current element as the root. The iterator - iterates over this element and all elements below it, in document (depth first) - order. If *tag* is not ``None`` or ``'*'``, only elements whose tag equals - *tag* are returned from the iterator. + .. method:: iter(tag=None) + Creates a tree :term:`iterator` with the current element as the root. + The iterator iterates over this element and all elements below it, in + document (depth first) order. If *tag* is not ``None`` or ``'*'``, only + elements whose tag equals *tag* are returned from the iterator. If the + tree structure is modified during iteration, the result is undefined. -.. method:: Element.insert(index, element) - Inserts a subelement at the given position in this element. + .. method:: iterfind(match) + Finds all matching subelements, by tag name or path. Returns an iterable + yielding all matching elements in document order. -.. method:: Element.makeelement(tag, attrib) + .. versionadded:: 2.7 - Creates a new element object of the same type as this element. Do not call this - method, use the SubElement factory function instead. + .. method:: itertext() -.. method:: Element.remove(subelement) + Creates a text iterator. The iterator loops over this element and all + subelements, in document order, and returns all inner text. - Removes *subelement* from the element. Unlike the findXYZ methods this method - compares elements based on the instance identity, not on tag value or contents. + .. versionadded:: 2.7 -Element objects also support the following sequence type methods for working -with subelements: :meth:`__delitem__`, :meth:`__getitem__`, :meth:`__setitem__`, -:meth:`__len__`. -Caution: Because Element objects do not define a :meth:`__nonzero__` method, -elements with no subelements will test as ``False``. :: + .. method:: makeelement(tag, attrib) - element = root.find('foo') + Creates a new element object of the same type as this element. Do not + call this method, use the :func:`SubElement` factory function instead. - if not element: # careful! - print "element not found, or element has no subelements" - if element is None: - print "element not found" + .. method:: remove(subelement) + + Removes *subelement* from the element. Unlike the find\* methods this + method compares elements based on the instance identity, not on tag value + or contents. + + :class:`Element` objects also support the following sequence type methods + for working with subelements: :meth:`__delitem__`, :meth:`__getitem__`, + :meth:`__setitem__`, :meth:`__len__`. + + Caution: Elements with no subelements will test as ``False``. This behavior + will change in future versions. Use specific ``len(elem)`` or ``elem is + None`` test instead. :: + + element = root.find('foo') + + if not element: # careful! + print "element not found, or element has no subelements" + + if element is None: + print "element not found" .. _elementtree-elementtree-objects: @@ -305,52 +385,51 @@ ElementTree Objects ------------------- -.. class:: ElementTree([element,] [file]) +.. class:: ElementTree(element=None, file=None) - ElementTree wrapper class. This class represents an entire element hierarchy, - and adds some extra support for serialization to and from standard XML. + ElementTree wrapper class. This class represents an entire element + hierarchy, and adds some extra support for serialization to and from + standard XML. - *element* is the root element. The tree is initialized with the contents of the - XML *file* if given. + *element* is the root element. The tree is initialized with the contents + of the XML *file* if given. .. method:: _setroot(element) Replaces the root element for this tree. This discards the current contents of the tree, and replaces it with the given element. Use with - care. *element* is an element instance. + care. *element* is an element instance. - .. method:: find(path) + .. method:: find(match) - Finds the first toplevel element with given tag. Same as - getroot().find(path). *path* is the element to look for. Returns the - first matching element, or ``None`` if no element was found. + Finds the first toplevel element matching *match*. *match* may be a tag + name or path. Same as getroot().find(match). Returns the first matching + element, or ``None`` if no element was found. - .. method:: findall(path) + .. method:: findall(match) - Finds all toplevel elements with the given tag. Same as - getroot().findall(path). *path* is the element to look for. Returns a - list or :term:`iterator` containing all matching elements, in document - order. + Finds all matching subelements, by tag name or path. Same as + getroot().findall(match). *match* may be a tag name or path. Returns a + list containing all matching elements, in document order. - .. method:: findtext(path[, default]) + .. method:: findtext(match, default=None) Finds the element text for the first toplevel element with given tag. - Same as getroot().findtext(path). *path* is the toplevel element to look - for. *default* is the value to return if the element was not - found. Returns the text content of the first matching element, or the - default value no element was found. Note that if the element has is - found, but has no text content, this method returns an empty string. + Same as getroot().findtext(match). *match* may be a tag name or path. + *default* is the value to return if the element was not found. Returns + the text content of the first matching element, or the default value no + element was found. Note that if the element is found, but has no text + content, this method returns an empty string. - .. method:: getiterator([tag]) + .. method:: getiterator(tag=None) - Creates and returns a tree iterator for the root element. The iterator - loops over all elements in this tree, in section order. *tag* is the tag - to look for (default is to return all elements) + .. deprecated:: 2.7 + Use method :meth:`ElementTree.iter` instead. .. method:: getroot() @@ -358,19 +437,39 @@ ElementTree Objects Returns the root element for this tree. - .. method:: parse(source[, parser]) + .. method:: iter(tag=None) + + Creates and returns a tree iterator for the root element. The iterator + loops over all elements in this tree, in section order. *tag* is the tag + to look for (default is to return all elements) + + + .. method:: iterfind(match) + + Finds all matching subelements, by tag name or path. Same as + getroot().iterfind(match). Returns an iterable yielding all matching + elements in document order. + + .. versionadded:: 2.7 - Loads an external XML section into this element tree. *source* is a file - name or file object. *parser* is an optional parser instance. If not - given, the standard XMLTreeBuilder parser is used. Returns the section + + .. method:: parse(source, parser=None) + + Loads an external XML section into this element tree. *source* is a file + name or file object. *parser* is an optional parser instance. If not + given, the standard XMLParser parser is used. Returns the section root element. - .. method:: write(file[, encoding]) + .. method:: write(file, encoding="us-ascii", xml_declaration=None, method="xml") - Writes the element tree to a file, as XML. *file* is a file name, or a - file object opened for writing. *encoding* [1]_ is the output encoding - (default is US-ASCII). + Writes the element tree to a file, as XML. *file* is a file name, or a + file object opened for writing. *encoding* [1]_ is the output encoding + (default is US-ASCII). *xml_declaration* controls if an XML declaration + should be added to the file. Use False for never, True for always, None + for only if not US-ASCII or UTF-8 (default is None). *method* is either + ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``). Returns an + encoded string. This is the XML file that is going to be manipulated:: @@ -389,13 +488,13 @@ Example of changing the attribute "target" of every link in first paragraph:: >>> from xml.etree.ElementTree import ElementTree >>> tree = ElementTree() >>> tree.parse("index.xhtml") - <Element html at b7d3f1ec> + <Element 'html' at 0xb77e6fac> >>> p = tree.find("body/p") # Finds first occurrence of tag p in body >>> p - <Element p at 8416e0c> - >>> links = p.getiterator("a") # Returns list of all links + <Element 'p' at 0xb77ec26c> + >>> links = list(p.iter("a")) # Returns list of all links >>> links - [<Element a at b7d4f9ec>, <Element a at b7d4fb0c>] + [<Element 'a' at 0xb77ec2ac>, <Element 'a' at 0xb77ec1cc>] >>> for i in links: # Iterates through all found links ... i.attrib["target"] = "blank" >>> tree.write("output.xhtml") @@ -406,14 +505,14 @@ QName Objects ------------- -.. class:: QName(text_or_uri[, tag]) +.. class:: QName(text_or_uri, tag=None) - QName wrapper. This can be used to wrap a QName attribute value, in order to - get proper namespace handling on output. *text_or_uri* is a string containing - the QName value, in the form {uri}local, or, if the tag argument is given, the - URI part of a QName. If *tag* is given, the first argument is interpreted as an - URI, and this argument is interpreted as a local name. :class:`QName` instances - are opaque. + QName wrapper. This can be used to wrap a QName attribute value, in order + to get proper namespace handling on output. *text_or_uri* is a string + containing the QName value, in the form {uri}local, or, if the tag argument + is given, the URI part of a QName. If *tag* is given, the first argument is + interpreted as an URI, and this argument is interpreted as a local name. + :class:`QName` instances are opaque. .. _elementtree-treebuilder-objects: @@ -422,76 +521,91 @@ TreeBuilder Objects ------------------- -.. class:: TreeBuilder([element_factory]) +.. class:: TreeBuilder(element_factory=None) - Generic element structure builder. This builder converts a sequence of start, - data, and end method calls to a well-formed element structure. You can use this - class to build an element structure using a custom XML parser, or a parser for - some other XML-like format. The *element_factory* is called to create new - Element instances when given. + Generic element structure builder. This builder converts a sequence of + start, data, and end method calls to a well-formed element structure. You + can use this class to build an element structure using a custom XML parser, + or a parser for some other XML-like format. The *element_factory* is called + to create new :class:`Element` instances when given. .. method:: close() - Flushes the parser buffers, and returns the toplevel document - element. Returns an Element instance. + Flushes the builder buffers, and returns the toplevel document + element. Returns an :class:`Element` instance. .. method:: data(data) - Adds text to the current element. *data* is a string. This should be - either an 8-bit string containing ASCII text, or a Unicode string. + Adds text to the current element. *data* is a string. This should be + either a bytestring, or a Unicode string. .. method:: end(tag) - Closes the current element. *tag* is the element name. Returns the closed - element. + Closes the current element. *tag* is the element name. Returns the + closed element. .. method:: start(tag, attrs) - Opens a new element. *tag* is the element name. *attrs* is a dictionary - containing element attributes. Returns the opened element. + Opens a new element. *tag* is the element name. *attrs* is a dictionary + containing element attributes. Returns the opened element. + + In addition, a custom :class:`TreeBuilder` object can provide the + following method: -.. _elementtree-xmltreebuilder-objects: + .. method:: doctype(name, pubid, system) + + Handles a doctype declaration. *name* is the doctype name. *pubid* is + the public identifier. *system* is the system identifier. This method + does not exist on the default :class:`TreeBuilder` class. + + .. versionadded:: 2.7 -XMLTreeBuilder Objects ----------------------- +.. _elementtree-xmlparser-objects: -.. class:: XMLTreeBuilder([html,] [target]) +XMLParser Objects +----------------- - Element structure builder for XML source data, based on the expat parser. *html* - are predefined HTML entities. This flag is not supported by the current - implementation. *target* is the target object. If omitted, the builder uses an - instance of the standard TreeBuilder class. + +.. class:: XMLParser(html=0, target=None, encoding=None) + + :class:`Element` structure builder for XML source data, based on the expat + parser. *html* are predefined HTML entities. This flag is not supported by + the current implementation. *target* is the target object. If omitted, the + builder uses an instance of the standard TreeBuilder class. *encoding* [1]_ + is optional. If given, the value overrides the encoding specified in the + XML file. .. method:: close() - Finishes feeding data to the parser. Returns an element structure. + Finishes feeding data to the parser. Returns an element structure. .. method:: doctype(name, pubid, system) - Handles a doctype declaration. *name* is the doctype name. *pubid* is the - public identifier. *system* is the system identifier. + .. deprecated:: 2.7 + Define the :meth:`TreeBuilder.doctype` method on a custom TreeBuilder + target. .. method:: feed(data) - Feeds data to the parser. *data* is encoded data. + Feeds data to the parser. *data* is encoded data. -:meth:`XMLTreeBuilder.feed` calls *target*\'s :meth:`start` method +:meth:`XMLParser.feed` calls *target*\'s :meth:`start` method for each opening tag, its :meth:`end` method for each closing tag, -and data is processed by method :meth:`data`. :meth:`XMLTreeBuilder.close` +and data is processed by method :meth:`data`. :meth:`XMLParser.close` calls *target*\'s method :meth:`close`. -:class:`XMLTreeBuilder` can be used not only for building a tree structure. +:class:`XMLParser` can be used not only for building a tree structure. This is an example of counting the maximum depth of an XML file:: - >>> from xml.etree.ElementTree import XMLTreeBuilder + >>> from xml.etree.ElementTree import XMLParser >>> class MaxDepth: # The target object of the parser ... maxDepth = 0 ... depth = 0 @@ -507,7 +621,7 @@ This is an example of counting the maximum depth of an XML file:: ... return self.maxDepth ... >>> target = MaxDepth() - >>> parser = XMLTreeBuilder(target=target) + >>> parser = XMLParser(target=target) >>> exampleXml = """ ... <a> ... <b> @@ -527,7 +641,6 @@ This is an example of counting the maximum depth of an XML file:: .. rubric:: Footnotes .. [#] The encoding string included in XML output should conform to the - appropriate standards. For example, "UTF-8" is valid, but "UTF8" is - not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl + appropriate standards. For example, "UTF-8" is valid, but "UTF8" is + not. See http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-EncodingDecl and http://www.iana.org/assignments/character-sets. - diff --git a/Doc/library/xml.etree.rst b/Doc/library/xml.etree.rst deleted file mode 100644 index 3f85b3b..0000000 --- a/Doc/library/xml.etree.rst +++ /dev/null @@ -1,24 +0,0 @@ -:mod:`xml.etree` --- The ElementTree API for XML -================================================ - -.. module:: xml.etree - :synopsis: Package containing common ElementTree modules. -.. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> - - -.. versionadded:: 2.5 - -The ElementTree package is a simple, efficient, and quite popular library for -XML manipulation in Python. The :mod:`xml.etree` package contains the most -common components from the ElementTree API library. In the current release, -this package contains the :mod:`ElementTree`, :mod:`ElementPath`, and -:mod:`ElementInclude` modules from the full ElementTree distribution. - -.. XXX To be continued! - - -.. seealso:: - - `ElementTree Overview <http://effbot.org/tag/elementtree>`_ - The home page for :mod:`ElementTree`. This includes links to additional - documentation, alternative implementations, and other add-ons. diff --git a/Doc/library/xml.sax.handler.rst b/Doc/library/xml.sax.handler.rst index 832f1dc..23f429e 100644 --- a/Doc/library/xml.sax.handler.rst +++ b/Doc/library/xml.sax.handler.rst @@ -52,52 +52,57 @@ for the feature and property names. .. data:: feature_namespaces - Value: ``"http://xml.org/sax/features/namespaces"`` --- true: Perform Namespace - processing. --- false: Optionally do not perform Namespace processing (implies - namespace-prefixes; default). --- access: (parsing) read-only; (not parsing) - read/write + | value: ``"http://xml.org/sax/features/namespaces"`` + | true: Perform Namespace processing. + | false: Optionally do not perform Namespace processing (implies + namespace-prefixes; default). + | access: (parsing) read-only; (not parsing) read/write .. data:: feature_namespace_prefixes - Value: ``"http://xml.org/sax/features/namespace-prefixes"`` --- true: Report - the original prefixed names and attributes used for Namespace - declarations. --- false: Do not report attributes used for Namespace - declarations, and optionally do not report original prefixed names - (default). --- access: (parsing) read-only; (not parsing) read/write + | value: ``"http://xml.org/sax/features/namespace-prefixes"`` + | true: Report the original prefixed names and attributes used for Namespace + declarations. + | false: Do not report attributes used for Namespace declarations, and + optionally do not report original prefixed names (default). + | access: (parsing) read-only; (not parsing) read/write .. data:: feature_string_interning - Value: ``"http://xml.org/sax/features/string-interning"`` --- true: All element - names, prefixes, attribute names, Namespace URIs, and local names are interned - using the built-in intern function. --- false: Names are not necessarily - interned, although they may be (default). --- access: (parsing) read-only; (not - parsing) read/write + | value: ``"http://xml.org/sax/features/string-interning"`` + | true: All element names, prefixes, attribute names, Namespace URIs, and + local names are interned using the built-in intern function. + | false: Names are not necessarily interned, although they may be (default). + | access: (parsing) read-only; (not parsing) read/write .. data:: feature_validation - Value: ``"http://xml.org/sax/features/validation"`` --- true: Report all - validation errors (implies external-general-entities and - external-parameter-entities). --- false: Do not report validation errors. --- - access: (parsing) read-only; (not parsing) read/write + | value: ``"http://xml.org/sax/features/validation"`` + | true: Report all validation errors (implies external-general-entities and + external-parameter-entities). + | false: Do not report validation errors. + | access: (parsing) read-only; (not parsing) read/write .. data:: feature_external_ges - Value: ``"http://xml.org/sax/features/external-general-entities"`` --- true: - Include all external general (text) entities. --- false: Do not include - external general entities. --- access: (parsing) read-only; (not parsing) - read/write + | value: ``"http://xml.org/sax/features/external-general-entities"`` + | true: Include all external general (text) entities. + | false: Do not include external general entities. + | access: (parsing) read-only; (not parsing) read/write .. data:: feature_external_pes - Value: ``"http://xml.org/sax/features/external-parameter-entities"`` --- true: - Include all external parameter entities, including the external DTD subset. --- - false: Do not include any external parameter entities, even the external DTD - subset. --- access: (parsing) read-only; (not parsing) read/write + | value: ``"http://xml.org/sax/features/external-parameter-entities"`` + | true: Include all external parameter entities, including the external DTD + subset. + | false: Do not include any external parameter entities, even the external + DTD subset. + | access: (parsing) read-only; (not parsing) read/write .. data:: all_features @@ -107,34 +112,38 @@ for the feature and property names. .. data:: property_lexical_handler - Value: ``"http://xml.org/sax/properties/lexical-handler"`` --- data type: - xml.sax.sax2lib.LexicalHandler (not supported in Python 2) --- description: An - optional extension handler for lexical events like comments. --- access: - read/write + | value: ``"http://xml.org/sax/properties/lexical-handler"`` + | data type: xml.sax.sax2lib.LexicalHandler (not supported in Python 2) + | description: An optional extension handler for lexical events like + comments. + | access: read/write .. data:: property_declaration_handler - Value: ``"http://xml.org/sax/properties/declaration-handler"`` --- data type: - xml.sax.sax2lib.DeclHandler (not supported in Python 2) --- description: An - optional extension handler for DTD-related events other than notations and - unparsed entities. --- access: read/write + | value: ``"http://xml.org/sax/properties/declaration-handler"`` + | data type: xml.sax.sax2lib.DeclHandler (not supported in Python 2) + | description: An optional extension handler for DTD-related events other + than notations and unparsed entities. + | access: read/write .. data:: property_dom_node - Value: ``"http://xml.org/sax/properties/dom-node"`` --- data type: - org.w3c.dom.Node (not supported in Python 2) --- description: When parsing, - the current DOM node being visited if this is a DOM iterator; when not parsing, - the root DOM node for iteration. --- access: (parsing) read-only; (not parsing) - read/write + | value: ``"http://xml.org/sax/properties/dom-node"`` + | data type: org.w3c.dom.Node (not supported in Python 2) + | description: When parsing, the current DOM node being visited if this is + a DOM iterator; when not parsing, the root DOM node for iteration. + | access: (parsing) read-only; (not parsing) read/write .. data:: property_xml_string - Value: ``"http://xml.org/sax/properties/xml-string"`` --- data type: String --- - description: The literal string of characters that was the source for the - current event. --- access: read-only + | value: ``"http://xml.org/sax/properties/xml-string"`` + | data type: String + | description: The literal string of characters that was the source for the + current event. + | access: read-only .. data:: all_properties diff --git a/Doc/library/xml.sax.reader.rst b/Doc/library/xml.sax.reader.rst index 75c7d5b..4b3c18a 100644 --- a/Doc/library/xml.sax.reader.rst +++ b/Doc/library/xml.sax.reader.rst @@ -157,7 +157,7 @@ The :class:`XMLReader` interface supports the following methods: Allow an application to set the locale for errors and warnings. SAX parsers are not required to provide localization for errors and warnings; if - they cannot support the requested locale, however, they must throw a SAX + they cannot support the requested locale, however, they must raise a SAX exception. Applications may request a locale change in the middle of a parse. diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index b2bf730..51240ba 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -39,6 +39,7 @@ The module defines the following items: .. class:: ZipFile + :noindex: The class for reading and writing ZIP files. See section :ref:`zipfile-objects` for constructor details. @@ -64,8 +65,10 @@ The module defines the following items: .. function:: is_zipfile(filename) Returns ``True`` if *filename* is a valid ZIP file based on its magic number, - otherwise returns ``False``. + otherwise returns ``False``. *filename* may be a file or file-like object too. + .. versionchanged:: 2.7 + Support for file and file-like objects. .. data:: ZIP_STORED @@ -100,28 +103,40 @@ ZipFile Objects Open a ZIP file, where *file* can be either a path to a file (a string) or a file-like object. The *mode* parameter should be ``'r'`` to read an existing file, ``'w'`` to truncate and write a new file, or ``'a'`` to append to an - existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP file, - then additional files are added to it. If *file* does not refer to a ZIP file, - then a new ZIP archive is appended to the file. This is meant for adding a ZIP - archive to another file, such as :file:`python.exe`. Using :: - - cat myzip.zip >> python.exe - - also works, and at least :program:`WinZip` can read such files. If *mode* is - ``a`` and the file does not exist at all, it is created. *compression* is the - ZIP compression method to use when writing the archive, and should be - :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized values will cause - :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` is specified but the - :mod:`zlib` module is not available, :exc:`RuntimeError` is also raised. The - default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create - ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If - it is false (the default) :mod:`zipfile` will raise an exception when the ZIP - file would require ZIP64 extensions. ZIP64 extensions are disabled by default - because the default :program:`zip` and :program:`unzip` commands on Unix (the - InfoZIP utilities) don't support these extensions. + existing file. If *mode* is ``'a'`` and *file* refers to an existing ZIP + file, then additional files are added to it. If *file* does not refer to a + ZIP file, then a new ZIP archive is appended to the file. This is meant for + adding a ZIP archive to another file (such as :file:`python.exe`). .. versionchanged:: 2.6 - If the file does not exist, it is created if the mode is 'a'. + If *mode* is ``a`` and the file does not exist at all, it is created. + + *compression* is the ZIP compression method to use when writing the archive, + and should be :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` + is specified but the :mod:`zlib` module is not available, :exc:`RuntimeError` + is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is + ``True`` zipfile will create ZIP files that use the ZIP64 extensions when + the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` + will raise an exception when the ZIP file would require ZIP64 extensions. + ZIP64 extensions are disabled by default because the default :program:`zip` + and :program:`unzip` commands on Unix (the InfoZIP utilities) don't support + these extensions. + + .. versionchanged:: 2.7.1 + If the file is created with mode ``'a'`` or ``'w'`` and then + :meth:`close`\ d without adding any files to the archive, the appropriate + ZIP structures for an empty archive will be written to the file. + + ZipFile is also a context manager and therefore supports the + :keyword:`with` statement. In the example, *myzip* is closed after the + :keyword:`with` statement's suite is finished---even if an exception occurs:: + + with ZipFile('spam.zip', 'w') as myzip: + myzip.write('eggs.txt') + + .. versionadded:: 2.7 + Added the ability to use :class:`ZipFile` as a context manager. .. method:: ZipFile.close() @@ -161,8 +176,8 @@ ZipFile Objects .. note:: The file-like object is read-only and provides the following methods: - :meth:`read`, :meth:`readline`, :meth:`readlines`, :meth:`__iter__`, - :meth:`next`. + :meth:`!read`, :meth:`!readline`, :meth:`!readlines`, :meth:`!__iter__`, + :meth:`!next`. .. note:: @@ -272,7 +287,7 @@ ZipFile Objects byte, the name of the file in the archive will be truncated at the null byte. -.. method:: ZipFile.writestr(zinfo_or_arcname, bytes) +.. method:: ZipFile.writestr(zinfo_or_arcname, bytes[, compress_type]) Write the string *bytes* to the archive; *zinfo_or_arcname* is either the file name it will be given in the archive, or a :class:`ZipInfo` instance. If it's @@ -282,13 +297,20 @@ ZipFile Objects created with mode ``'r'`` will raise a :exc:`RuntimeError`. Calling :meth:`writestr` on a closed ZipFile will raise a :exc:`RuntimeError`. + If given, *compress_type* overrides the value given for the *compression* + parameter to the constructor for the new entry, or in the *zinfo_or_arcname* + (if that is a :class:`ZipInfo` instance). + .. note:: - When passing a :class:`ZipInfo` instance as the *zinfo_or_acrname* parameter, + When passing a :class:`ZipInfo` instance as the *zinfo_or_arcname* parameter, the compression method used will be that specified in the *compress_type* member of the given :class:`ZipInfo` instance. By default, the :class:`ZipInfo` constructor sets this member to :const:`ZIP_STORED`. + .. versionchanged:: 2.7 + The *compression_type* argument. + The following data attributes are also available: diff --git a/Doc/library/zipimport.rst b/Doc/library/zipimport.rst index 4ce9a35..43e12b1 100644 --- a/Doc/library/zipimport.rst +++ b/Doc/library/zipimport.rst @@ -96,6 +96,15 @@ zipimporter Objects file wasn't found. + .. method:: get_filename(fullname) + + Return the value ``__file__`` would be set to if the specified module + was imported. Raise :exc:`ZipImportError` if the module couldn't be + found. + + .. versionadded:: 2.7 + + .. method:: get_source(fullname) Return the source code for the specified module. Raise diff --git a/Doc/license.rst b/Doc/license.rst index 6ab673c..4c654ae 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -100,9 +100,9 @@ been GPL-compatible; the table below summarizes the various releases. +----------------+--------------+-----------+------------+-----------------+ | 2.6.3 | 2.6.2 | 2009 | PSF | yes | +----------------+--------------+-----------+------------+-----------------+ -| 2.6.4 | 2.6.3 | 2009 | PSF | yes | +| 2.6.4 | 2.6.3 | 2010 | PSF | yes | +----------------+--------------+-----------+------------+-----------------+ -| 2.6.5 | 2.6.4 | 2010 | PSF | yes | +| 2.7 | 2.6 | 2010 | PSF | yes | +----------------+--------------+-----------+------------+-----------------+ .. note:: @@ -716,3 +716,260 @@ The :mod:`select` and contains the following notice for the kqueue interface:: LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +strtod and dtoa +--------------- + +The file :file:`Python/dtoa.c`, which supplies C functions dtoa and +strtod for conversion of C doubles to and from strings, is derived +from the file of the same name by David M. Gay, currently available +from http://www.netlib.org/fp/. The original file, as retrieved on +March 16, 2009, contains the following copyright and licensing +notice:: + + /**************************************************************** + * + * The author of this software is David M. Gay. + * + * Copyright (c) 1991, 2000, 2001 by Lucent Technologies. + * + * Permission to use, copy, modify, and distribute this software for any + * purpose without fee is hereby granted, provided that this entire notice + * is included in all copies of any software which is or includes a copy + * or modification of this software and in all copies of the supporting + * documentation for such software. + * + * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED + * WARRANTY. IN PARTICULAR, NEITHER THE AUTHOR NOR LUCENT MAKES ANY + * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY + * OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. + * + ***************************************************************/ + + +OpenSSL +------- + +The modules :mod:`hashlib`, :mod:`posix`, :mod:`ssl`, :mod:`crypt` use +the OpenSSL library for added performance if made available by the +operating system. Additionally, the Windows installers for Python +include a copy of the OpenSSL libraries, so we include a copy of the +OpenSSL license here:: + + + LICENSE ISSUES + ============== + + The OpenSSL toolkit stays under a dual license, i.e. both the conditions of + the OpenSSL License and the original SSLeay license apply to the toolkit. + See below for the actual license texts. Actually both licenses are BSD-style + Open Source licenses. In case of any license issues related to OpenSSL + please contact openssl-core@openssl.org. + + OpenSSL License + --------------- + + /* ==================================================================== + * Copyright (c) 1998-2008 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + + Original SSLeay License + ----------------------- + + /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are aheared to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the rouines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publically available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ + + +expat +----- + +The :mod:`pyexpat` extension is built using an included copy of the expat +sources unless the build is configured :option:`--with-system-expat`:: + + Copyright (c) 1998, 1999, 2000 Thai Open Source Software Center Ltd + and Clark Cooper + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be included + in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY + CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, + TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE + SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +libffi +------ + +The :mod:`_ctypes` extension is built using an included copy of the libffi +sources unless the build is configured :option:`--with-system-libffi`:: + + Copyright (c) 1996-2008 Red Hat, Inc and others. + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + ``Software''), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be included + in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + DEALINGS IN THE SOFTWARE. + + +zlib +---- + +The :mod:`zlib` extension is built using an included copy of the zlib +sources unless the zlib version found on the system is too old to be +used for the build:: + + Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + diff --git a/Doc/make.bat b/Doc/make.bat index 80ac806..d234e77 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -34,7 +34,7 @@ echo. goto end :checkout -svn co %SVNROOT%/external/Sphinx-0.6.5/sphinx tools/sphinx +svn co %SVNROOT%/external/Sphinx-0.6.7/sphinx tools/sphinx svn co %SVNROOT%/external/docutils-0.6/docutils tools/docutils svn co %SVNROOT%/external/Jinja-2.3.1/jinja2 tools/jinja2 svn co %SVNROOT%/external/Pygments-1.3.1/pygments tools/pygments diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 3c716b2..9dfa31f 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -333,11 +333,15 @@ allows common :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` usage patterns to be encapsulated for convenient reuse. .. productionlist:: - with_stmt: "with" `expression` ["as" `target`] ":" `suite` + with_stmt: "with" with_item ("," with_item)* ":" `suite` + with_item: `expression` ["as" `target`] -The execution of the :keyword:`with` statement proceeds as follows: +The execution of the :keyword:`with` statement with one "item" proceeds as follows: -#. The context expression is evaluated to obtain a context manager. +#. The context expression (the expression given in the :token:`with_item`) is + evaluated to obtain a context manager. + +#. The context manager's :meth:`__exit__` is loaded for later use. #. The context manager's :meth:`__enter__` method is invoked. @@ -349,7 +353,7 @@ The execution of the :keyword:`with` statement proceeds as follows: The :keyword:`with` statement guarantees that if the :meth:`__enter__` method returns without an error, then :meth:`__exit__` will always be called. Thus, if an error occurs during the assignment to the target list, it will be treated the - same as an error occurring within the suite would be. See step 5 below. + same as an error occurring within the suite would be. See step 6 below. #. The suite is executed. @@ -367,12 +371,27 @@ The execution of the :keyword:`with` statement proceeds as follows: from :meth:`__exit__` is ignored, and execution proceeds at the normal location for the kind of exit that was taken. +With more than one item, the context managers are processed as if multiple +:keyword:`with` statements were nested:: + + with A() as a, B() as b: + suite + +is equivalent to :: + + with A() as a: + with B() as b: + suite + .. note:: In Python 2.5, the :keyword:`with` statement is only allowed when the ``with_statement`` feature has been enabled. It is always enabled in Python 2.6. +.. versionchanged:: 2.7 + Support for multiple context expressions. + .. seealso:: :pep:`0343` - The "with" statement diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index f1743d9..7c2c9af 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -66,6 +66,8 @@ are still reachable. containing circular references. See the documentation of the :mod:`gc` module for information on controlling the collection of cyclic garbage. Other implementations act differently and CPython may change. + Do not depend on immediate finalization of objects when they become + unreachable (ex: always close files). Note that the use of the implementation's tracing or debugging facilities may keep objects alive that would normally be collectable. Also note that catching @@ -689,7 +691,7 @@ Callable types an object passed to the C function as an implicit extra argument. An example of a built-in method is ``alist.append()``, assuming *alist* is a list object. In this case, the special read-only attribute :attr:`__self__` is set to the object - denoted by *list*. + denoted by *alist*. Class Types Class types, or "new-style classes," are callable. These objects normally act @@ -739,6 +741,13 @@ Modules Special read-only attribute: :attr:`__dict__` is the module's namespace as a dictionary object. + .. impl-detail:: + + Because of the way CPython clears module dictionaries, the module + dictionary will be cleared when the module falls out of scope even if the + dictionary still has live references. To avoid this, copy the dictionary + or keep the module around while using its dictionary directly. + .. index:: single: __name__ (module attribute) single: __doc__ (module attribute) @@ -908,6 +917,22 @@ Internal types objects, code objects are immutable and contain no references (directly or indirectly) to mutable objects. + .. index:: + single: co_argcount (code object attribute) + single: co_code (code object attribute) + single: co_consts (code object attribute) + single: co_filename (code object attribute) + single: co_firstlineno (code object attribute) + single: co_flags (code object attribute) + single: co_lnotab (code object attribute) + single: co_name (code object attribute) + single: co_names (code object attribute) + single: co_nlocals (code object attribute) + single: co_stacksize (code object attribute) + single: co_varnames (code object attribute) + single: co_cellvars (code object attribute) + single: co_freevars (code object attribute) + Special read-only attributes: :attr:`co_name` gives the function name; :attr:`co_argcount` is the number of positional arguments (including arguments with default values); :attr:`co_nlocals` is the number of local variables used @@ -925,22 +950,6 @@ Internal types :attr:`co_stacksize` is the required stack size (including local variables); :attr:`co_flags` is an integer encoding a number of flags for the interpreter. - .. index:: - single: co_argcount (code object attribute) - single: co_code (code object attribute) - single: co_consts (code object attribute) - single: co_filename (code object attribute) - single: co_firstlineno (code object attribute) - single: co_flags (code object attribute) - single: co_lnotab (code object attribute) - single: co_name (code object attribute) - single: co_names (code object attribute) - single: co_nlocals (code object attribute) - single: co_stacksize (code object attribute) - single: co_varnames (code object attribute) - single: co_cellvars (code object attribute) - single: co_freevars (code object attribute) - .. index:: object: generator The following flag bits are defined for :attr:`co_flags`: bit ``0x04`` is set if @@ -1353,8 +1362,7 @@ Basic customization Arguments to rich comparison methods are never coerced. To automatically generate ordering operations from a single root operation, - see the `Total Ordering recipe in the ASPN cookbook - <http://code.activestate.com/recipes/576529/>`_\. + see :func:`functools.total_ordering`. .. method:: object.__cmp__(self, other) @@ -1531,11 +1539,11 @@ Implementing Descriptors ^^^^^^^^^^^^^^^^^^^^^^^^ The following methods only apply when an instance of the class containing the -method (a so-called *descriptor* class) appears in the class dictionary of -another new-style class, known as the *owner* class. In the examples below, "the -attribute" refers to the attribute whose name is the key of the property in the -owner class' ``__dict__``. Descriptors can only be implemented as new-style -classes themselves. +method (a so-called *descriptor* class) appears in an *owner* class (the +descriptor must be in either the owner's class dictionary or in the class +dictionary for one of its parents). In the examples below, "the attribute" +refers to the attribute whose name is the key of the property in the owner +class' :attr:`__dict__`. .. method:: object.__get__(self, instance, owner) @@ -1600,7 +1608,7 @@ Super Binding If ``a`` is an instance of :class:`super`, then the binding ``super(B, obj).m()`` searches ``obj.__class__.__mro__`` for the base class ``A`` immediately preceding ``B`` and then invokes the descriptor with the call: - ``A.__dict__['m'].__get__(obj, A)``. + ``A.__dict__['m'].__get__(obj, obj.__class__)``. For instance bindings, the precedence of descriptor invocation depends on the which descriptor methods are defined. A descriptor can define any combination @@ -1773,7 +1781,7 @@ The following methods are used to override the default behavior of the In particular, the metaclass :class:`abc.ABCMeta` implements these methods in order to allow the addition of Abstract Base Classes (ABCs) as "virtual base -classes" to any class or type (including built-in types), and including to other +classes" to any class or type (including built-in types), including other ABCs. .. method:: class.__instancecheck__(self, instance) @@ -1792,7 +1800,7 @@ ABCs. Note that these methods are looked up on the type (metaclass) of a class. They cannot be defined as class methods in the actual class. This is consistent with -the lookup of special methods that are called on instances, only that in this +the lookup of special methods that are called on instances, only in this case the instance is itself a class. .. seealso:: @@ -2313,13 +2321,15 @@ will not be supported. * In the current implementation, the built-in numeric types :class:`int`, - :class:`long` and :class:`float` do not use coercion; the type :class:`complex` - however does use coercion for binary operators and rich comparisons, despite - the above rules. The difference can become apparent when subclassing these - types. Over time, the type :class:`complex` may be fixed to avoid coercion. + :class:`long`, :class:`float`, and :class:`complex` do not use coercion. All these types implement a :meth:`__coerce__` method, for use by the built-in :func:`coerce` function. + .. versionchanged:: 2.7 + + The complex type no longer makes implicit calls to the :meth:`__coerce__` + method for mixed-type binary arithmetic operations. + .. _context-managers: diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 4e38536..91ac72f 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -140,9 +140,9 @@ weak form of restricted execution. The namespace for a module is automatically created the first time a module is imported. The main module for a script is always called :mod:`__main__`. -The global statement has the same scope as a name binding operation in the same -block. If the nearest enclosing scope for a free variable contains a global -statement, the free variable is treated as a global. +The :keyword:`global` statement has the same scope as a name binding operation +in the same block. If the nearest enclosing scope for a free variable contains +a global statement, the free variable is treated as a global. A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution. The namespace of diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 8bfc8d9..067a3d6 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -65,7 +65,7 @@ atoms is: .. productionlist:: atom: `identifier` | `literal` | `enclosure` enclosure: `parenth_form` | `list_display` - : | `generator_expression` | `dict_display` + : | `generator_expression` | `dict_display` | `set_display` : | `string_conversion` | `yield_atom` @@ -206,74 +206,100 @@ block, nesting from left to right, and evaluating the expression to produce a list element each time the innermost block is reached [#]_. +.. _comprehensions: + +Displays for sets and dictionaries +---------------------------------- + +For constructing a set or a dictionary Python provides special syntax +called "displays", each of them in two flavors: + +* either the container contents are listed explicitly, or + +* they are computed via a set of looping and filtering instructions, called a + :dfn:`comprehension`. + +Common syntax elements for comprehensions are: + +.. productionlist:: + comprehension: `expression` `comp_for` + comp_for: "for" `target_list` "in" `or_test` [`comp_iter`] + comp_iter: `comp_for` | `comp_if` + comp_if: "if" `expression_nocond` [`comp_iter`] + +The comprehension consists of a single expression followed by at least one +:keyword:`for` clause and zero or more :keyword:`for` or :keyword:`if` clauses. +In this case, the elements of the new container are those that would be produced +by considering each of the :keyword:`for` or :keyword:`if` clauses a block, +nesting from left to right, and evaluating the expression to produce an element +each time the innermost block is reached. + +Note that the comprehension is executed in a separate scope, so names assigned +to in the target list don't "leak" in the enclosing scope. + + .. _genexpr: Generator expressions --------------------- .. index:: pair: generator; expression + object: generator A generator expression is a compact generator notation in parentheses: .. productionlist:: - generator_expression: "(" `expression` `genexpr_for` ")" - genexpr_for: "for" `target_list` "in" `or_test` [`genexpr_iter`] - genexpr_iter: `genexpr_for` | `genexpr_if` - genexpr_if: "if" `old_expression` [`genexpr_iter`] + generator_expression: "(" `expression` `comp_for` ")" -.. index:: object: generator +A generator expression yields a new generator object. Its syntax is the same as +for comprehensions, except that it is enclosed in parentheses instead of +brackets or curly braces. -A generator expression yields a new generator object. It consists of a single -expression followed by at least one :keyword:`for` clause and zero or more -:keyword:`for` or :keyword:`if` clauses. The iterating values of the new -generator are those that would be produced by considering each of the -:keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and -evaluating the expression to yield a value that is reached the innermost block -for each iteration. - -Variables used in the generator expression are evaluated lazily in a separate -scope when the :meth:`next` method is called for the generator object (in the -same fashion as for normal generators). However, the :keyword:`in` expression -of the leftmost :keyword:`for` clause is immediately evaluated in the current -scope so that an error produced by it can be seen before any other possible +Variables used in the generator expression are evaluated lazily when the +:meth:`__next__` method is called for generator object (in the same fashion as +normal generators). However, the leftmost :keyword:`for` clause is immediately +evaluated, so that an error produced by it can be seen before any other possible error in the code that handles the generator expression. Subsequent -:keyword:`for` and :keyword:`if` clauses cannot be evaluated immediately since -they may depend on the previous :keyword:`for` loop. For example: -``(x*y for x in range(10) for y in bar(x))``. +:keyword:`for` clauses cannot be evaluated immediately since they may depend on +the previous :keyword:`for` loop. For example: ``(x*y for x in range(10) for y +in bar(x))``. -The parentheses can be omitted on calls with only one argument. See section +The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for the detail. - .. _dict: Dictionary displays ------------------- .. index:: pair: dictionary; display - -.. index:: - single: key - single: datum - single: key/datum pair + key, datum, key/datum pair + object: dictionary A dictionary display is a possibly empty series of key/datum pairs enclosed in curly braces: .. productionlist:: - dict_display: "{" [`key_datum_list`] "}" + dict_display: "{" [`key_datum_list` | `dict_comprehension`] "}" key_datum_list: `key_datum` ("," `key_datum`)* [","] key_datum: `expression` ":" `expression` - -.. index:: object: dictionary + dict_comprehension: `expression` ":" `expression` `comp_for` A dictionary display yields a new dictionary object. -The key/datum pairs are evaluated from left to right to define the entries of -the dictionary: each key object is used as a key into the dictionary to store -the corresponding datum. +If a comma-separated sequence of key/datum pairs is given, they are evaluated +from left to right to define the entries of the dictionary: each key object is +used as a key into the dictionary to store the corresponding datum. This means +that you can specify the same key multiple times in the key/datum list, and the +final dictionary's value for that key will be the last one given. + +A dict comprehension, in contrast to list and set comprehensions, needs two +expressions separated with a colon followed by the usual "for" and "if" clauses. +When the comprehension is run, the resulting key and value elements are inserted +in the new dictionary in the order they are produced. .. index:: pair: immutable; object + hashable Restrictions on the types of the key values are listed earlier in section :ref:`types`. (To summarize, the key type should be :term:`hashable`, which excludes @@ -282,6 +308,30 @@ datum (textually rightmost in the display) stored for a given key value prevails. +.. _set: + +Set displays +------------ + +.. index:: pair: set; display + object: set + +A set display is denoted by curly braces and distinguishable from dictionary +displays by the lack of colons separating keys and values: + +.. productionlist:: + set_display: "{" (`expression_list` | `comprehension`) "}" + +A set display yields a new mutable set object, the contents being specified by +either a sequence of expressions or a comprehension. When a comma-separated +list of expressions is supplied, its elements are evaluated from left to right +and added to the set object. When a comprehension is supplied, the set is +constructed from the elements resulting from the comprehension. + +An empty set cannot be constructed with ``{}``; this literal constructs an empty +dictionary. + + .. _string-conversions: String conversions @@ -367,7 +417,7 @@ All of this makes generator functions quite similar to coroutines; they yield multiple times, they have more than one entry point and their execution can be suspended. The only difference is that a generator function cannot control where should the execution continue after it yields; the control is always -transfered to the generator's caller. +transferred to the generator's caller. .. index:: object: generator @@ -950,6 +1000,11 @@ A right shift by *n* bits is defined as division by ``pow(2, n)``. A left shift by *n* bits is defined as multiplication with ``pow(2, n)``. Negative shift counts raise a :exc:`ValueError` exception. +.. note:: + + In the current implementation, the right-hand operand is required + to be at most :attr:`sys.maxsize`. If the right-hand operand is larger than + :attr:`sys.maxsize` an :exc:`OverflowError` exception is raised. .. _bitwise: @@ -1312,6 +1367,7 @@ groups from right to left). | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ | ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder | +| | [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ @@ -1336,8 +1392,8 @@ groups from right to left). true numerically due to roundoff. For example, and assuming a platform on which a Python float is an IEEE 754 double-precision number, in order that ``-1e-100 % 1e100`` have the same sign as ``1e100``, the computed result is ``-1e-100 + - 1e100``, which is numerically exactly equal to ``1e100``. Function :func:`fmod` - in the :mod:`math` module returns a result whose sign matches the sign of the + 1e100``, which is numerically exactly equal to ``1e100``. The function + :func:`math.fmod` returns a result whose sign matches the sign of the first argument instead, and so returns ``-1e-100`` in this case. Which approach is more appropriate depends on the application. @@ -1367,5 +1423,8 @@ groups from right to left). the :keyword:`is` operator, like those involving comparisons between instance methods, or constants. Check their documentation for more info. +.. [#] The ``%`` operator is also used for string formatting; the same + precedence applies. + .. [#] The power operator ``**`` binds less tightly than an arithmetic or bitwise unary operator on its right, that is, ``2**-1`` is ``0.5``. diff --git a/Doc/reference/introduction.rst b/Doc/reference/introduction.rst index 69bd41d..36f824f 100644 --- a/Doc/reference/introduction.rst +++ b/Doc/reference/introduction.rst @@ -68,12 +68,12 @@ IronPython more information, see `the IronPython website <http://www.ironpython.com/>`_. PyPy - An implementation of Python written in Python; even the bytecode interpreter is - written in Python. This is executed using CPython as the underlying - interpreter. One of the goals of the project is to encourage experimentation - with the language itself by making it easier to modify the interpreter (since it - is written in Python). Additional information is available on `the PyPy - project's home page <http://codespeak.net/pypy/>`_. + An implementation of Python written completely in Python. It supports several + advanced features not found in other implementations like stackless support + and a Just in Time compiler. One of the goals of the project is to encourage + experimentation with the language itself by making it easier to modify the + interpreter (since it is written in Python). Additional information is + available on `the PyPy project's home page <http://pypy.org/>`_. Each of these implementations varies in some way from the language as documented in this manual, or introduces specific information beyond what's covered in the diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index d4c7990..d716139 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -426,6 +426,7 @@ String literals are described by the following lexical definitions: .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) stringprefix: "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR" + : | "b" | "B" | "br" | "Br" | "bR" | "BR" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' longstring: "'''" `longstringitem`* "'''" : | '"""' `longstringitem`* '"""' @@ -458,8 +459,10 @@ different rules for interpreting backslash escape sequences. A prefix of ``'u'`` or ``'U'`` makes the string a Unicode string. Unicode strings use the Unicode character set as defined by the Unicode Consortium and ISO 10646. Some additional escape sequences, described below, are available in Unicode strings. -The two prefix characters may be combined; in this case, ``'u'`` must appear -before ``'r'``. +A prefix of ``'b'`` or ``'B'`` is ignored in Python 2; it indicates that the +literal should become a bytes literal in Python 3 (e.g. when code is +automatically converted with 2to3). A ``'u'`` or ``'b'`` prefix may be followed +by an ``'r'`` prefix. In triple-quoted strings, unescaped newlines and quotes are allowed (and are retained), except that three unescaped quotes in a row terminate the string. (A diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 73d98ca..60e0e99 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -121,9 +121,6 @@ Assignment of an object to a target list is recursively defined as follows. * If the target list is a comma-separated list of targets: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets. - (This rule is relaxed as of Python 1.5; in earlier versions, the object had to - be a tuple. Since strings are sequences, an assignment like ``a, b = "xy"`` is - now legal as long as the string has the right length.) Assignment of an object to a single target is recursively defined as follows. @@ -836,15 +833,11 @@ leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 -imprt mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. +import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. The specification for relative imports is contained within :pep:`328`. - -.. index:: builtin: __import__ - -The built-in function :func:`__import__` is provided to support applications -that determine which modules need to be loaded dynamically; refer to -:ref:`built-in-funcs` for additional information. +:func:`importlib.import_module` is provided to support applications that +determine which modules need to be loaded dynamically. .. _future: diff --git a/Doc/tools/dailybuild.py b/Doc/tools/dailybuild.py new file mode 100755 index 0000000..1a471e6 --- /dev/null +++ b/Doc/tools/dailybuild.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Runs the daily build of the Python docs on dinsdale.python.org. +# +# Usages: +# +# dailybuild.py +# +# without any arguments builds docs for all branches configured in the global +# BRANCHES value. +# +# dailybuild.py [-d] <checkout> <target> +# +# builds one version, where <checkout> is an SVN checkout directory of the +# Python branch to build docs for, and <target> is the directory where the +# result should be placed. If -d is given, the docs are built even if the +# branch is in development mode (i.e. version contains a, b or c). +# +# This script is not run from the checkout, so if you want to change how the +# daily build is run, you must replace it on dinsdale. This is necessary, for +# example, after the release of a new minor version. +# +# 03/2010, Georg Brandl + +import os +import sys +import getopt + + +BUILDROOT = '/home/gbrandl/docbuild' +WWWROOT = '/data/ftp.python.org/pub/docs.python.org' + +BRANCHES = [ + # checkout, target, isdev + (BUILDROOT + '/python33', WWWROOT + '/dev', True), + (BUILDROOT + '/python27', WWWROOT, False), + (BUILDROOT + '/python32', WWWROOT + '/py3k', False), +] + + +def build_one(checkout, target, isdev): + print 'Doc autobuild started in %s' % checkout + os.chdir(checkout) + print 'Running svn update' + os.system('svn update') + print 'Running make autobuild' + if os.WEXITSTATUS(os.system( + 'cd Doc; make autobuild-%s' % (isdev and 'dev' or 'stable'))) == 2: + print '*' * 80 + return + print 'Copying HTML files' + os.system('cp -a Doc/build/html/* %s' % target) + print 'Copying dist files' + os.system('mkdir -p %s/archives' % target) + os.system('cp -a Doc/dist/* %s/archives' % target) + print 'Finished' + print '=' * 80 + +def usage(): + print 'Usage:' + print ' %s' % sys.argv[0] + print 'or' + print ' %s [-d] <checkout> <target>' % sys.argv[0] + sys.exit(1) + + +if __name__ == '__main__': + try: + opts, args = getopt.getopt(sys.argv[1:], 'd') + except getopt.error: + usage() + if opts and not args: + usage() + if args: + if len(args) != 2: + usage() + build_one(args[0], args[1], bool(opts)) + else: + for branch in BRANCHES: + build_one(*branch) diff --git a/Doc/tools/sphinx-build.py b/Doc/tools/sphinx-build.py index 94a8df8..a0bd7fa 100644 --- a/Doc/tools/sphinx-build.py +++ b/Doc/tools/sphinx-build.py @@ -3,11 +3,15 @@ Sphinx - Python documentation toolchain ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - :copyright: 2007 by Georg Brandl. + :copyright: 2007-2010 by Georg Brandl. :license: Python license. """ import sys +import warnings + +# Get rid of UserWarnings reported by pkg_resources. +warnings.filterwarnings('ignore', category=UserWarning, module='jinja2') if __name__ == '__main__': diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html index bcc47e5..30963c3 100644 --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -7,12 +7,12 @@ <span class="linkdescr">or <a href="{{ pathto("whatsnew/index") }}">all "What's new" documents</a> since 2.0</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("tutorial/index") }}">Tutorial</a><br/> <span class="linkdescr">start here</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("using/index") }}">Using Python</a><br/> - <span class="linkdescr">how to use Python on different platforms</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("library/index") }}">Library Reference</a><br/> <span class="linkdescr">keep this under your pillow</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("reference/index") }}">Language Reference</a><br/> <span class="linkdescr">describes syntax and language elements</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("using/index") }}">Python Setup and Usage</a><br/> + <span class="linkdescr">how to use Python on different platforms</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("howto/index") }}">Python HOWTOs</a><br/> <span class="linkdescr">in-depth documents on specific topics</span></p> </td><td width="50%"> diff --git a/Doc/tools/sphinxext/indexsidebar.html b/Doc/tools/sphinxext/indexsidebar.html index 7daa0c6..8d3feef 100644 --- a/Doc/tools/sphinxext/indexsidebar.html +++ b/Doc/tools/sphinxext/indexsidebar.html @@ -2,9 +2,9 @@ <p><a href="{{ pathto('download') }}">Download these documents</a></p> <h3>Docs for other versions</h3> <ul> - <li><a href="http://docs.python.org/2.7/">Python 2.7 (stable)</a></li> - <li><a href="http://docs.python.org/3.1/">Python 3.1 (stable)</a></li> - <li><a href="http://docs.python.org/dev/py3k/">Python 3.2 (in development)</a></li> + <li><a href="http://docs.python.org/2.6/">Python 2.6 (stable)</a></li> + <li><a href="http://docs.python.org/3.2/">Python 3.2 (stable)</a></li> + <li><a href="http://docs.python.org/dev/">Python 3.3 (in development)</a></li> <li><a href="http://www.python.org/doc/versions/">Old versions</a></li> </ul> diff --git a/Doc/tools/sphinxext/patchlevel.py b/Doc/tools/sphinxext/patchlevel.py index cb9e35c..082858e 100644 --- a/Doc/tools/sphinxext/patchlevel.py +++ b/Doc/tools/sphinxext/patchlevel.py @@ -41,7 +41,7 @@ def get_header_version_info(srcdir): suffixes = { 'PY_RELEASE_LEVEL_ALPHA': 'a', 'PY_RELEASE_LEVEL_BETA': 'b', - 'PY_RELEASE_LEVEL_GAMMA': 'c', + 'PY_RELEASE_LEVEL_GAMMA': 'rc', } if level != 'PY_RELEASE_LEVEL_FINAL': release += suffixes[level] + str(int(d['PY_RELEASE_SERIAL'])) diff --git a/Doc/tools/sphinxext/pyspecific.py b/Doc/tools/sphinxext/pyspecific.py index ff07adb..415310d 100644 --- a/Doc/tools/sphinxext/pyspecific.py +++ b/Doc/tools/sphinxext/pyspecific.py @@ -34,6 +34,7 @@ from sphinx.writers.html import HTMLTranslator from sphinx.locale import versionlabels HTMLTranslator.visit_versionmodified = new_visit_versionmodified + # Support for marking up and linking to bugs.python.org issues def issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): @@ -99,19 +100,8 @@ from pprint import pformat from docutils.io import StringOutput from docutils.utils import new_document -try: - from sphinx.builders import Builder -except ImportError: - # using Sphinx < 0.6, which has a different package layout - from sphinx.builder import Builder - # monkey-patch toctree directive to accept (and ignore) the :numbered: flag - from sphinx.directives.other import toctree_directive - toctree_directive.options['numbered'] = toctree_directive.options['glob'] - -try: - from sphinx.writers.text import TextWriter -except ImportError: - from sphinx.textwriter import TextWriter +from sphinx.builders import Builder +from sphinx.writers.text import TextWriter class PydocTopicsBuilder(Builder): @@ -128,7 +118,9 @@ class PydocTopicsBuilder(Builder): def write(self, *ignored): writer = TextWriter(self) - for label in self.status_iterator(pydoc_topic_labels, 'building topics... '): + for label in self.status_iterator(pydoc_topic_labels, + 'building topics... ', + length=len(pydoc_topic_labels)): if label not in self.env.labels: self.warn('label %r not in documentation' % label) continue @@ -141,7 +133,7 @@ class PydocTopicsBuilder(Builder): self.topics[label] = writer.output def finish(self): - f = open(path.join(self.outdir, 'pydoc_topics.py'), 'w') + f = open(path.join(self.outdir, 'topics.py'), 'w') try: f.write('# Autogenerated by Sphinx on %s\n' % asctime()) f.write('topics = ' + pformat(self.topics) + '\n') diff --git a/Doc/tools/sphinxext/static/basic.css b/Doc/tools/sphinxext/static/basic.css index aeb1381..2b47622 100644 --- a/Doc/tools/sphinxext/static/basic.css +++ b/Doc/tools/sphinxext/static/basic.css @@ -257,6 +257,10 @@ table.docutils td, table.docutils th { background-color: #eef; } +table.docutils td p.last, table.docutils th p.last { + margin-bottom: 0; +} + table.field-list td, table.field-list th { border: 0 !important; } @@ -342,7 +346,7 @@ p.deprecated { } .footnote:target { - background-color: #ffa + background-color: #ffa; } .impl-detail { @@ -364,6 +368,7 @@ p.deprecated { pre { overflow: auto; + overflow-y: hidden; } td.linenos pre { diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv index 3f987c7..a97b764 100644 --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -164,3 +164,139 @@ whatsnew/2.4,,:System, whatsnew/2.5,,:memory,:memory: whatsnew/2.5,,:step,[start:stop:step] whatsnew/2.5,,:stop,[start:stop:step] +distutils/examples,267,`,This is the description of the ``foobar`` package. +documenting/fromlatex,39,:func,:func:`str(object)` +documenting/fromlatex,39,`,:func:`str(object)` +documenting/fromlatex,39,`,``str(object)`` +documenting/fromlatex,55,.. deprecated:,.. deprecated:: 2.5 +documenting/fromlatex,66,.. note:,.. note:: +documenting/fromlatex,76,:samp,":samp:`open({filename}, {mode})`" +documenting/fromlatex,76,`,":samp:`open({filename}, {mode})`" +documenting/fromlatex,80,`,``'c'`` +documenting/fromlatex,80,`,`Title <URL>`_ +documenting/fromlatex,80,`,``code`` +documenting/fromlatex,80,`,`Title <URL>`_ +documenting/fromlatex,99,:file,:file:`C:\\Temp\\my.tmp` +documenting/fromlatex,99,`,:file:`C:\\Temp\\my.tmp` +documenting/fromlatex,99,`,"``open(""C:\Temp\my.tmp"")``" +documenting/fromlatex,129,.. function:,.. function:: do_foo(bar) +documenting/fromlatex,141,.. function:,".. function:: open(filename[, mode[, buffering]])" +documenting/fromlatex,152,.. function:,.. function:: foo_* +documenting/fromlatex,152,:noindex,:noindex: +documenting/fromlatex,162,.. describe:,.. describe:: a == b +documenting/fromlatex,168,.. cmdoption:,.. cmdoption:: -O +documenting/fromlatex,168,.. envvar:,.. envvar:: PYTHONINSPECT +documenting/rest,33,`,``text`` +documenting/rest,47,:rolename,:rolename:`content` +documenting/rest,47,`,:rolename:`content` +documenting/rest,103,::,This is a normal text paragraph. The next paragraph is a code sample:: +documenting/rest,130,`,`Link text <http://target>`_ +documenting/rest,187,.. function:,.. function:: foo(x) +documenting/rest,187,:bar,:bar: no +documenting/rest,208,.. rubric:,.. rubric:: Footnotes +faq/programming,,:reduce,"print (lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y," +faq/programming,,:reduce,"Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro," +faq/programming,,:chr,">=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr(" +faq/programming,,::,for x in sequence[::-1]: +faq/windows,229,:EOF,@setlocal enableextensions & python -x %~f0 %* & goto :EOF +faq/windows,393,:REG,.py :REG_SZ: c:\<path to python>\python.exe -u %s %s +library/bisect,,:hi,all(val >= x for val in a[i:hi]) +library/bisect,,:hi,all(val > x for val in a[i:hi]) +library/http.client,52,:port,host:port +library/nntplib,,:bytes,:bytes +library/nntplib,,:lines,:lines +library/nntplib,,:lines,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" +library/nntplib,,:bytes,"['xref', 'from', ':lines', ':bytes', 'references', 'date', 'message-id', 'subject']" +library/pickle,567,:memory,"conn = sqlite3.connect("":memory:"")" +library/profile,293,:lineno,"(sort by filename:lineno)," +library/socket,,::,"(10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))]" +library/stdtypes,,:end,s[start:end] +library/stdtypes,,:end,s[start:end] +license,,`,* THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY +license,,`,* THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND +license,,`,"``Software''), to deal in the Software without restriction, including" +license,,`,"THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND," +reference/lexical_analysis,704,`,$ ? ` +whatsnew/2.7,735,:Sunday,'2009:4:Sunday' +whatsnew/2.7,862,::,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,862,:Cookie,"export PYTHONWARNINGS=all,error:::Cookie:0" +whatsnew/2.7,,::,>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') +whatsnew/2.7,,::,"ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]'," +documenting/markup,33,.. sectionauthor:,.. sectionauthor:: Guido van Rossum <guido@python.org> +documenting/markup,42,:mod,:mod:`parrot` -- Dead parrot access +documenting/markup,42,`,:mod:`parrot` -- Dead parrot access +documenting/markup,42,.. module:,.. module:: parrot +documenting/markup,42,:platform,":platform: Unix, Windows" +documenting/markup,42,:synopsis,:synopsis: Analyze and reanimate dead parrots. +documenting/markup,42,.. moduleauthor:,.. moduleauthor:: Eric Cleese <eric@python.invalid> +documenting/markup,42,.. moduleauthor:,.. moduleauthor:: John Idle <john@python.invalid> +documenting/markup,88,:noindex,:noindex: +documenting/markup,95,.. function:,.. function:: spam(eggs) +documenting/markup,95,:noindex,:noindex: +documenting/markup,101,.. method:,.. method:: FileInput.input(...) +documenting/markup,121,.. cfunction:,".. cfunction:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)" +documenting/markup,131,.. cmember:,.. cmember:: PyObject* PyTypeObject.tp_bases +documenting/markup,150,.. cvar:,.. cvar:: PyObject* PyClass_Type +documenting/markup,179,.. function:,".. function:: Timer.repeat([repeat=3[, number=1000000]])" +documenting/markup,209,.. cmdoption:,.. cmdoption:: -m <module> +documenting/markup,227,.. describe:,.. describe:: opcode +documenting/markup,256,.. highlightlang:,.. highlightlang:: c +documenting/markup,276,.. literalinclude:,.. literalinclude:: example.py +documenting/markup,291,:rolename,:rolename:`content` +documenting/markup,291,`,:rolename:`content` +documenting/markup,296,:role,:role:`title <target>` +documenting/markup,296,`,:role:`title <target>` +documenting/markup,302,:meth,:meth:`~Queue.Queue.get` +documenting/markup,302,`,:meth:`~Queue.Queue.get` +documenting/markup,350,:func,:func:`filter` +documenting/markup,350,`,:func:`filter` +documenting/markup,350,:func,:func:`foo.filter` +documenting/markup,350,`,:func:`foo.filter` +documenting/markup,356,:func,:func:`open` +documenting/markup,356,`,:func:`open` +documenting/markup,356,:func,:func:`.open` +documenting/markup,356,`,:func:`.open` +documenting/markup,435,:file,... is installed in :file:`/usr/lib/python2.{x}/site-packages` ... +documenting/markup,435,`,... is installed in :file:`/usr/lib/python2.{x}/site-packages` ... +documenting/markup,454,:kbd,:kbd:`C-x C-f` +documenting/markup,454,`,:kbd:`C-x C-f` +documenting/markup,454,:kbd,:kbd:`Control-x Control-f` +documenting/markup,454,`,:kbd:`Control-x Control-f` +documenting/markup,468,:mailheader,:mailheader:`Content-Type` +documenting/markup,468,`,:mailheader:`Content-Type` +documenting/markup,477,:manpage,:manpage:`ls(1)` +documenting/markup,477,`,:manpage:`ls(1)` +documenting/markup,493,:menuselection,:menuselection:`Start --> Programs` +documenting/markup,493,`,:menuselection:`Start --> Programs` +documenting/markup,508,`,``code`` +documenting/markup,526,:file,:file: +documenting/markup,526,`,``code`` +documenting/markup,561,:ref,:ref:`label-name` +documenting/markup,561,`,:ref:`label-name` +documenting/markup,565,:ref,"It refers to the section itself, see :ref:`my-reference-label`." +documenting/markup,565,`,"It refers to the section itself, see :ref:`my-reference-label`." +documenting/markup,574,:ref,:ref: +documenting/markup,595,.. note:,.. note:: +documenting/markup,622,.. versionadded:,.. versionadded:: 2.5 +documenting/markup,647,::,.. impl-detail:: +documenting/markup,647,::,.. impl-detail:: This shortly mentions an implementation detail. +documenting/markup,667,.. seealso:,.. seealso:: +documenting/markup,667,:mod,Module :mod:`zipfile` +documenting/markup,667,`,Module :mod:`zipfile` +documenting/markup,667,:mod,Documentation of the :mod:`zipfile` standard module. +documenting/markup,667,`,Documentation of the :mod:`zipfile` standard module. +documenting/markup,667,`,"`GNU tar manual, Basic Tar Format <http://link>`_" +documenting/markup,681,.. centered:,.. centered:: +documenting/markup,726,.. toctree:,.. toctree:: +documenting/markup,726,:maxdepth,:maxdepth: 2 +documenting/markup,742,.. index:,.. index:: +documenting/markup,772,.. index:,".. index:: BNF, grammar, syntax, notation" +documenting/markup,803,`,"unaryneg ::= ""-"" `integer`" +documenting/markup,808,.. productionlist:,.. productionlist:: +documenting/markup,808,`,"try1_stmt: ""try"" "":"" `suite`" +documenting/markup,808,`,": (""except"" [`expression` ["","" `target`]] "":"" `suite`)+" +documenting/markup,808,`,": [""else"" "":"" `suite`]" +documenting/markup,808,`,": [""finally"" "":"" `suite`]" +documenting/markup,808,`,"try2_stmt: ""try"" "":"" `suite`" +documenting/markup,808,`,": ""finally"" "":"" `suite`" +library/urllib2,67,:close,Connection:close diff --git a/Doc/tools/sphinxext/suspicious.py b/Doc/tools/sphinxext/suspicious.py index 7684f65..f15e931b 100644 --- a/Doc/tools/sphinxext/suspicious.py +++ b/Doc/tools/sphinxext/suspicious.py @@ -41,16 +41,13 @@ Copyright 2009 Gabriel A. Genellina """ -import os, sys -import csv +import os import re -from docutils import nodes - -try: - from sphinx.builders import Builder -except ImportError: - from sphinx.builder import Builder +import csv +import sys +from docutils import nodes +from sphinx.builders import Builder detect_all = re.compile(ur''' ::(?=[^=])| # two :: (but NOT ::=) @@ -59,9 +56,10 @@ detect_all = re.compile(ur''' (?<!\.)\.\.[ \t]*\w+: # .. foo: (but NOT ... else:) ''', re.UNICODE | re.VERBOSE).finditer + class Rule: def __init__(self, docname, lineno, issue, line): - "A rule for ignoring issues" + """A rule for ignoring issues""" self.docname = docname # document to which this rule applies self.lineno = lineno # line number in the original source; # this rule matches only near that. @@ -70,9 +68,15 @@ class Rule: self.line = line # text of the container element (single line only) + +class dialect(csv.excel): + """Our dialect: uses only linefeed as newline.""" + lineterminator = '\n' + + class CheckSuspiciousMarkupBuilder(Builder): """ - Checks for possibly invalid markup that may leak into the output + Checks for possibly invalid markup that may leak into the output. """ name = 'suspicious' @@ -81,7 +85,8 @@ class CheckSuspiciousMarkupBuilder(Builder): self.log_file_name = os.path.join(self.outdir, 'suspicious.csv') open(self.log_file_name, 'w').close() # load database of previously ignored issues - self.load_rules(os.path.join(os.path.dirname(__file__), 'susp-ignored.csv')) + self.load_rules(os.path.join(os.path.dirname(__file__), + 'susp-ignored.csv')) def get_outdated_docs(self): return self.env.found_docs @@ -90,14 +95,11 @@ class CheckSuspiciousMarkupBuilder(Builder): return '' def prepare_writing(self, docnames): - ### PYTHON PROJECT SPECIFIC ### - for name in set(docnames): - if name.split('/', 1)[0] == 'documenting': - docnames.remove(name) - ### PYTHON PROJECT SPECIFIC ### + pass def write_doc(self, docname, doctree): - self.any_issue = False # set when any issue is encountered in this document + # set when any issue is encountered in this document + self.any_issue = False self.docname = docname visitor = SuspiciousVisitor(doctree, self) doctree.walk(visitor) @@ -110,8 +112,7 @@ class CheckSuspiciousMarkupBuilder(Builder): self.report_issue(line, lineno, issue) def is_ignored(self, line, lineno, issue): - """Determine whether this issue should be ignored. - """ + """Determine whether this issue should be ignored.""" docname = self.docname for rule in self.rules: if rule.docname != docname: continue @@ -144,12 +145,11 @@ class CheckSuspiciousMarkupBuilder(Builder): def write_log_entry(self, lineno, issue, text): f = open(self.log_file_name, 'ab') - writer = csv.writer(f) + writer = csv.writer(f, dialect) writer.writerow([self.docname.encode('utf-8'), - lineno, - issue.encode('utf-8'), - text.strip().encode('utf-8')]) - del writer + lineno, + issue.encode('utf-8'), + text.strip().encode('utf-8')]) f.close() def load_rules(self, filename): @@ -164,7 +164,8 @@ class CheckSuspiciousMarkupBuilder(Builder): except IOError: return for i, row in enumerate(csv.reader(f)): if len(row) != 4: - raise ValueError("wrong format in %s, line %d: %s" % (filename, i+1, row)) + raise ValueError( + "wrong format in %s, line %d: %s" % (filename, i+1, row)) docname, lineno, issue, text = row docname = docname.decode('utf-8') if lineno: lineno = int(lineno) @@ -178,7 +179,7 @@ class CheckSuspiciousMarkupBuilder(Builder): def get_lineno(node): - "Obtain line number information for a node" + """Obtain line number information for a node.""" lineno = None while lineno is None and node: node = node.parent @@ -203,7 +204,8 @@ def extract_line(text, index): """ p = text.rfind('\n', 0, index) + 1 q = text.find('\n', index) - if q<0: q = len(text) + if q < 0: + q = len(text) return text[p:q] @@ -222,7 +224,6 @@ class SuspiciousVisitor(nodes.GenericNodeVisitor): self.lastlineno = lineno = max(get_lineno(node) or 0, self.lastlineno) seen = set() # don't report the same issue more than only once per line for match in detect_all(text): - #import pdb; pdb.set_trace() issue = match.group() line = extract_line(text, match.start()) if (issue, line) not in seen: diff --git a/Doc/tutorial/classes.rst b/Doc/tutorial/classes.rst index d729955..ed0e655 100644 --- a/Doc/tutorial/classes.rst +++ b/Doc/tutorial/classes.rst @@ -4,25 +4,26 @@ Classes ******* -Python's class mechanism adds classes to the language with a minimum of new -syntax and semantics. It is a mixture of the class mechanisms found in C++ and -Modula-3. As is true for modules, classes in Python do not put an absolute -barrier between definition and user, but rather rely on the politeness of the -user not to "break into the definition." The most important features of classes -are retained with full power, however: the class inheritance mechanism allows +Compared with other programming languages, Python's class mechanism adds classes +with a minimum of new syntax and semantics. It is a mixture of the class +mechanisms found in C++ and Modula-3. Python classes provide all the standard +features of Object Oriented Programming: the class inheritance mechanism allows multiple base classes, a derived class can override any methods of its base class or classes, and a method can call the method of a base class with the same -name. Objects can contain an arbitrary amount of data. - -In C++ terminology, all class members (including the data members) are *public*, -and all member functions are *virtual*. As in Modula-3, there are no shorthands -for referencing the object's members from its methods: the method function is -declared with an explicit first argument representing the object, which is -provided implicitly by the call. As in Smalltalk, classes themselves are -objects. This provides semantics for importing and renaming. Unlike C++ and -Modula-3, built-in types can be used as base classes for extension by the user. -Also, like in C++, most built-in operators with special syntax (arithmetic -operators, subscripting etc.) can be redefined for class instances. +name. Objects can contain arbitrary amounts and kinds of data. As is true for +modules, classes partake of the dynamic nature of Python: they are created at +runtime, and can be modified further after creation. + +In C++ terminology, normally class members (including the data members) are +*public* (except see below :ref:`tut-private`), and all member functions are +*virtual*. As in Modula-3, there are no shorthands for referencing the object's +members from its methods: the method function is declared with an explicit first +argument representing the object, which is provided implicitly by the call. As +in Smalltalk, classes themselves are objects. This provides semantics for +importing and renaming. Unlike C++ and Modula-3, built-in types can be used as +base classes for extension by the user. Also, like in C++, most built-in +operators with special syntax (arithmetic operators, subscripting etc.) can be +redefined for class instances. (Lacking universally accepted terminology to talk about classes, I will make occasional use of Smalltalk and C++ terms. I would use Modula-3 terms, since @@ -517,7 +518,7 @@ other multiple-inheritance languages as call-next-method and is more powerful than the super call found in single-inheritance languages. With new-style classes, dynamic ordering is necessary because all cases of -multiple inheritance exhibit one or more diamond relationships (where one at +multiple inheritance exhibit one or more diamond relationships (where at least one of the parent classes can be accessed through multiple paths from the bottommost class). For example, all new-style classes inherit from :class:`object`, so any case of multiple inheritance provides more than one path @@ -537,7 +538,7 @@ Private Variables ================= "Private" instance variables that cannot be accessed except from inside an -object, don't exist in Python. However, there is a convention that is followed +object don't exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. ``_spam``) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject diff --git a/Doc/tutorial/controlflow.rst b/Doc/tutorial/controlflow.rst index a9247cd..de93beb 100644 --- a/Doc/tutorial/controlflow.rst +++ b/Doc/tutorial/controlflow.rst @@ -429,11 +429,12 @@ function like this:: def cheeseshop(kind, *arguments, **keywords): print "-- Do you have any", kind, "?" print "-- I'm sorry, we're all out of", kind - for arg in arguments: print arg + for arg in arguments: + print arg print "-" * 40 - keys = keywords.keys() - keys.sort() - for kw in keys: print kw, ":", keywords[kw] + keys = sorted(keywords.keys()) + for kw in keys: + print kw, ":", keywords[kw] It could be called like this:: diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index dfc2b33..226eadd 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -170,7 +170,8 @@ There are three built-in functions that are very useful when used with lists: ``filter(function, sequence)`` returns a sequence consisting of those items from the sequence for which ``function(item)`` is true. If *sequence* is a :class:`string` or :class:`tuple`, the result will be of the same type; -otherwise, it is always a :class:`list`. For example, to compute some primes:: +otherwise, it is always a :class:`list`. For example, to compute primes up +to 25:: >>> def f(x): return x % 2 != 0 and x % 3 != 0 ... @@ -481,8 +482,8 @@ using a non-existent key. The :meth:`keys` method of a dictionary object returns a list of all the keys used in the dictionary, in arbitrary order (if you want it sorted, just apply -the :meth:`sort` method to the list of keys). To check whether a single key is -in the dictionary, use the :keyword:`in` keyword. +the :func:`sorted` function to it). To check whether a single key is in the +dictionary, use the :keyword:`in` keyword. Here is a small example using a dictionary:: diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index a0069f5..1351957 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -218,11 +218,9 @@ exception to occur. For example:: File "<stdin>", line 1, in ? NameError: HiThere -The argument to :keyword:`raise` is an exception class or instance to be -raised. There is a deprecated alternate syntax that separates class and -constructor arguments; the above could be written as ``raise NameError, -'HiThere'``. Since it once was the only one available, the latter form is -prevalent in older code. +The sole argument to :keyword:`raise` indicates the exception to be raised. +This must be either an exception instance or an exception class (a class that +derives from :class:`Exception`). If you need to determine whether an exception was raised but don't intend to handle it, a simpler form of the :keyword:`raise` statement allows you to diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 29c7a66..4b95570 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -48,74 +48,82 @@ decimal value 0.1 cannot be represented exactly as a base 2 fraction. In base 0.0001100110011001100110011001100110011001100110011... -Stop at any finite number of bits, and you get an approximation. This is why -you see things like:: +Stop at any finite number of bits, and you get an approximation. - >>> 0.1 - 0.10000000000000001 +On a typical machine running Python, there are 53 bits of precision available +for a Python float, so the value stored internally when you enter the decimal +number ``0.1`` is the binary fraction :: + + 0.00011001100110011001100110011001100110011001100110011010 -On most machines today, that is what you'll see if you enter 0.1 at a Python -prompt. You may not, though, because the number of bits used by the hardware to -store floating-point values can vary across machines, and Python only prints a -decimal approximation to the true decimal value of the binary approximation -stored by the machine. On most machines, if Python were to print the true -decimal value of the binary approximation stored for 0.1, it would have to -display :: +which is close to, but not exactly equal to, 1/10. + +It's easy to forget that the stored value is an approximation to the original +decimal fraction, because of the way that floats are displayed at the +interpreter prompt. Python only prints a decimal approximation to the true +decimal value of the binary approximation stored by the machine. If Python +were to print the true decimal value of the binary approximation stored for +0.1, it would have to display :: >>> 0.1 0.1000000000000000055511151231257827021181583404541015625 -instead! The Python prompt uses the built-in :func:`repr` function to obtain a -string version of everything it displays. For floats, ``repr(float)`` rounds -the true decimal value to 17 significant digits, giving :: +That is more digits than most people find useful, so Python keeps the number +of digits manageable by displaying a rounded value instead :: - 0.10000000000000001 + >>> 0.1 + 0.1 -``repr(float)`` produces 17 significant digits because it turns out that's -enough (on most machines) so that ``eval(repr(x)) == x`` exactly for all finite -floats *x*, but rounding to 16 digits is not enough to make that true. +It's important to realize that this is, in a real sense, an illusion: the value +in the machine is not exactly 1/10, you're simply rounding the *display* of the +true machine value. This fact becomes apparent as soon as you try to do +arithmetic with these values :: -Note that this is in the very nature of binary floating-point: this is not a bug -in Python, and it is not a bug in your code either. You'll see the same kind of -thing in all languages that support your hardware's floating-point arithmetic -(although some languages may not *display* the difference by default, or in all -output modes). + >>> 0.1 + 0.2 + 0.30000000000000004 -Python's built-in :func:`str` function produces only 12 significant digits, and -you may wish to use that instead. It's unusual for ``eval(str(x))`` to -reproduce *x*, but the output may be more pleasant to look at:: +Note that this is in the very nature of binary floating-point: this is not a +bug in Python, and it is not a bug in your code either. You'll see the same +kind of thing in all languages that support your hardware's floating-point +arithmetic (although some languages may not *display* the difference by +default, or in all output modes). - >>> print str(0.1) - 0.1 +Other surprises follow from this one. For example, if you try to round the +value 2.675 to two decimal places, you get this :: -It's important to realize that this is, in a real sense, an illusion: the value -in the machine is not exactly 1/10, you're simply rounding the *display* of the -true machine value. + >>> round(2.675, 2) + 2.67 -Other surprises follow from this one. For example, after seeing :: +The documentation for the built-in :func:`round` function says that it rounds +to the nearest value, rounding ties away from zero. Since the decimal fraction +2.675 is exactly halfway between 2.67 and 2.68, you might expect the result +here to be (a binary approximation to) 2.68. It's not, because when the +decimal string ``2.675`` is converted to a binary floating-point number, it's +again replaced with a binary approximation, whose exact value is :: - >>> 0.1 - 0.10000000000000001 + 2.67499999999999982236431605997495353221893310546875 -you may be tempted to use the :func:`round` function to chop it back to the -single digit you expect. But that makes no difference:: +Since this approximation is slightly closer to 2.67 than to 2.68, it's rounded +down. - >>> round(0.1, 1) - 0.10000000000000001 +If you're in a situation where you care which way your decimal halfway-cases +are rounded, you should consider using the :mod:`decimal` module. +Incidentally, the :mod:`decimal` module also provides a nice way to "see" the +exact value that's stored in any particular Python float :: -The problem is that the binary floating-point value stored for "0.1" was already -the best possible binary approximation to 1/10, so trying to round it again -can't make it better: it was already as good as it gets. + >>> from decimal import Decimal + >>> Decimal(2.675) + Decimal('2.67499999999999982236431605997495353221893310546875') -Another consequence is that since 0.1 is not exactly 1/10, summing ten values of -0.1 may not yield exactly 1.0, either:: +Another consequence is that since 0.1 is not exactly 1/10, summing ten values +of 0.1 may not yield exactly 1.0, either:: >>> sum = 0.0 >>> for i in range(10): ... sum += 0.1 ... >>> sum - 0.99999999999999989 + 0.9999999999999999 Binary floating-point arithmetic holds many surprises like this. The problem with "0.1" is explained in precise detail below, in the "Representation Error" @@ -131,9 +139,9 @@ that every float operation can suffer a new rounding error. While pathological cases do exist, for most casual use of floating-point arithmetic you'll see the result you expect in the end if you simply round the -display of your final results to the number of decimal digits you expect. -:func:`str` usually suffices, and for finer control see the :meth:`str.format` -method's format specifiers in :ref:`formatstrings`. +display of your final results to the number of decimal digits you expect. For +fine control over how a float is displayed see the :meth:`str.format` method's +format specifiers in :ref:`formatstrings`. .. _tut-fp-error: @@ -141,24 +149,24 @@ method's format specifiers in :ref:`formatstrings`. Representation Error ==================== -This section explains the "0.1" example in detail, and shows how you can perform -an exact analysis of cases like this yourself. Basic familiarity with binary -floating-point representation is assumed. +This section explains the "0.1" example in detail, and shows how you can +perform an exact analysis of cases like this yourself. Basic familiarity with +binary floating-point representation is assumed. :dfn:`Representation error` refers to the fact that some (most, actually) decimal fractions cannot be represented exactly as binary (base 2) fractions. This is the chief reason why Python (or Perl, C, C++, Java, Fortran, and many others) often won't display the exact decimal number you expect:: - >>> 0.1 - 0.10000000000000001 + >>> 0.1 + 0.2 + 0.30000000000000004 -Why is that? 1/10 is not exactly representable as a binary fraction. Almost all -machines today (November 2000) use IEEE-754 floating point arithmetic, and -almost all platforms map Python floats to IEEE-754 "double precision". 754 -doubles contain 53 bits of precision, so on input the computer strives to -convert 0.1 to the closest fraction it can of the form *J*/2**\ *N* where *J* is -an integer containing exactly 53 bits. Rewriting :: +Why is that? 1/10 and 2/10 are not exactly representable as a binary +fraction. Almost all machines today (July 2010) use IEEE-754 floating point +arithmetic, and almost all platforms map Python floats to IEEE-754 "double +precision". 754 doubles contain 53 bits of precision, so on input the computer +strives to convert 0.1 to the closest fraction it can of the form *J*/2**\ *N* +where *J* is an integer containing exactly 53 bits. Rewriting :: 1 / 10 ~= J / (2**N) @@ -170,24 +178,24 @@ and recalling that *J* has exactly 53 bits (is ``>= 2**52`` but ``< 2**53``), the best value for *N* is 56:: >>> 2**52 - 4503599627370496L + 4503599627370496 >>> 2**53 - 9007199254740992L + 9007199254740992 >>> 2**56/10 - 7205759403792793L + 7205759403792793 -That is, 56 is the only value for *N* that leaves *J* with exactly 53 bits. The -best possible value for *J* is then that quotient rounded:: +That is, 56 is the only value for *N* that leaves *J* with exactly 53 bits. +The best possible value for *J* is then that quotient rounded:: >>> q, r = divmod(2**56, 10) >>> r - 6L + 6 Since the remainder is more than half of 10, the best approximation is obtained by rounding up:: >>> q+1 - 7205759403792794L + 7205759403792794 Therefore the best possible approximation to 1/10 in 754 double precision is that over 2\*\*56, or :: @@ -195,8 +203,8 @@ that over 2\*\*56, or :: 7205759403792794 / 72057594037927936 Note that since we rounded up, this is actually a little bit larger than 1/10; -if we had not rounded up, the quotient would have been a little bit smaller than -1/10. But in no case can it be *exactly* 1/10! +if we had not rounded up, the quotient would have been a little bit smaller +than 1/10. But in no case can it be *exactly* 1/10! So the computer never "sees" 1/10: what it sees is the exact fraction given above, the best 754 double approximation it can get:: @@ -207,13 +215,12 @@ above, the best 754 double approximation it can get:: If we multiply that fraction by 10\*\*30, we can see the (truncated) value of its 30 most significant decimal digits:: - >>> 7205759403792794 * 10**30 / 2**56 + >>> 7205759403792794 * 10**30 // 2**56 100000000000000005551115123125L meaning that the exact number stored in the computer is approximately equal to -the decimal value 0.100000000000000005551115123125. Rounding that to 17 -significant digits gives the 0.10000000000000001 that Python displays (well, -will display on any 754-conforming platform that does best-possible input and -output conversions in its C library --- yours may not!). - - +the decimal value 0.100000000000000005551115123125. In versions prior to +Python 2.7 and Python 3.1, Python rounded this value to 17 significant digits, +giving '0.10000000000000001'. In current versions, Python displays a value +based on the shortest decimal fraction that rounds correctly back to the true +binary value, resulting simply in '0.1'. diff --git a/Doc/tutorial/inputoutput.rst b/Doc/tutorial/inputoutput.rst index a2dda3b..3441f54 100644 --- a/Doc/tutorial/inputoutput.rst +++ b/Doc/tutorial/inputoutput.rst @@ -19,16 +19,17 @@ the :keyword:`print` statement. (A third way is using the :meth:`write` method of file objects; the standard output file can be referenced as ``sys.stdout``. See the Library Reference for more information on this.) -.. index:: module: string - Often you'll want more control over the formatting of your output than simply printing space-separated values. There are two ways to format your output; the first way is to do all the string handling yourself; using string slicing and concatenation operations you can create any layout you can imagine. The -standard module :mod:`string` contains some useful operations for padding +string types have some methods that perform useful operations for padding strings to a given column width; these will be discussed shortly. The second way is to use the :meth:`str.format` method. +The :mod:`string` module contains a :class:`~string.Template` class which offers +yet another way to substitute values into strings. + One question remains, of course: how do you convert values to strings? Luckily, Python has ways to convert any value to a string: pass it to the :func:`repr` or :func:`str` functions. @@ -49,10 +50,10 @@ Some examples:: 'Hello, world.' >>> repr(s) "'Hello, world.'" - >>> str(0.1) - '0.1' - >>> repr(0.1) - '0.10000000000000001' + >>> str(1.0/7.0) + '0.142857142857' + >>> repr(1.0/7.0) + '0.14285714285714285' >>> x = 10 * 3.25 >>> y = 200 * 200 >>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...' @@ -102,17 +103,18 @@ Here are two ways to write a table of squares and cubes:: (Note that in the first example, one space between each column was added by the way :keyword:`print` works: it always adds spaces between its arguments.) -This example demonstrates the :meth:`rjust` method of string objects, which -right-justifies a string in a field of a given width by padding it with spaces -on the left. There are similar methods :meth:`ljust` and :meth:`center`. These -methods do not write anything, they just return a new string. If the input -string is too long, they don't truncate it, but return it unchanged; this will -mess up your column lay-out but that's usually better than the alternative, -which would be lying about a value. (If you really want truncation you can -always add a slice operation, as in ``x.ljust(n)[:n]``.) +This example demonstrates the :meth:`str.rjust` method of string +objects, which right-justifies a string in a field of a given width by padding +it with spaces on the left. There are similar methods :meth:`str.ljust` and +:meth:`str.center`. These methods do not write anything, they just return a +new string. If the input string is too long, they don't truncate it, but +return it unchanged; this will mess up your column lay-out but that's usually +better than the alternative, which would be lying about a value. (If you +really want truncation you can always add a slice operation, as in +``x.ljust(n)[:n]``.) -There is another method, :meth:`zfill`, which pads a numeric string on the left -with zeros. It understands about plus and minus signs:: +There is another method, :meth:`str.zfill`, which pads a numeric string on the +left with zeros. It understands about plus and minus signs:: >>> '12'.zfill(5) '00012' @@ -123,20 +125,20 @@ with zeros. It understands about plus and minus signs:: Basic usage of the :meth:`str.format` method looks like this:: - >>> print 'We are the {0} who say "{1}!"'.format('knights', 'Ni') + >>> print 'We are the {} who say "{}!"'.format('knights', 'Ni') We are the knights who say "Ni!" The brackets and characters within them (called format fields) are replaced with -the objects passed into the :meth:`~str.format` method. A number in the +the objects passed into the :meth:`str.format` method. A number in the brackets refers to the position of the object passed into the -:meth:`~str.format` method. :: +:meth:`str.format` method. :: >>> print '{0} and {1}'.format('spam', 'eggs') spam and eggs >>> print '{1} and {0}'.format('spam', 'eggs') eggs and spam -If keyword arguments are used in the :meth:`~str.format` method, their values +If keyword arguments are used in the :meth:`str.format` method, their values are referred to by using the name of the argument. :: >>> print 'This {food} is {adjective}.'.format( @@ -153,14 +155,14 @@ Positional and keyword arguments can be arbitrarily combined:: convert the value before it is formatted. :: >>> import math - >>> print 'The value of PI is approximately {0}.'.format(math.pi) + >>> print 'The value of PI is approximately {}.'.format(math.pi) The value of PI is approximately 3.14159265359. - >>> print 'The value of PI is approximately {0!r}.'.format(math.pi) + >>> print 'The value of PI is approximately {!r}.'.format(math.pi) The value of PI is approximately 3.141592653589793. An optional ``':'`` and format specifier can follow the field name. This allows greater control over how the value is formatted. The following example -truncates Pi to three places after the decimal. +rounds Pi to three places after the decimal. >>> import math >>> print 'The value of PI is approximately {0:.3f}.'.format(math.pi) @@ -194,8 +196,8 @@ notation. :: >>> print 'Jack: {Jack:d}; Sjoerd: {Sjoerd:d}; Dcab: {Dcab:d}'.format(**table) Jack: 4098; Sjoerd: 4127; Dcab: 8637678 -This is particularly useful in combination with the new built-in :func:`vars` -function, which returns a dictionary containing all local variables. +This is particularly useful in combination with the built-in function +:func:`vars`, which returns a dictionary containing all local variables. For a complete overview of string formatting with :meth:`str.format`, see :ref:`formatstrings`. diff --git a/Doc/tutorial/interactive.rst b/Doc/tutorial/interactive.rst index ca0cfaf..5faaf96 100644 --- a/Doc/tutorial/interactive.rst +++ b/Doc/tutorial/interactive.rst @@ -123,10 +123,7 @@ interpreter. :: # bound to the Esc key by default (you can change it - see readline docs). # # Store the file in ~/.pystartup, and set an environment variable to point - # to it: "export PYTHONSTARTUP=/home/user/.pystartup" in bash. - # - # Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the - # full path to your home directory. + # to it: "export PYTHONSTARTUP=~/.pystartup" in bash. import atexit import os diff --git a/Doc/tutorial/interpreter.rst b/Doc/tutorial/interpreter.rst index 311b409..df93af2 100644 --- a/Doc/tutorial/interpreter.rst +++ b/Doc/tutorial/interpreter.rst @@ -22,11 +22,11 @@ guru or system administrator. (E.g., :file:`/usr/local/python` is a popular alternative location.) On Windows machines, the Python installation is usually placed in -:file:`C:\\Python26`, though you can change this when you're running the +:file:`C:\\Python27`, though you can change this when you're running the installer. To add this directory to your path, you can type the following command into the command prompt in a DOS box:: - set path=%path%;C:\python26 + set path=%path%;C:\python27 Typing an end-of-file character (:kbd:`Control-D` on Unix, :kbd:`Control-Z` on Windows) at the primary prompt causes the interpreter to exit with a zero exit @@ -58,14 +58,6 @@ Some Python modules are also useful as scripts. These can be invoked using ``python -m module [arg] ...``, which executes the source file for *module* as if you had spelled out its full name on the command line. -Note that there is a difference between ``python file`` and ``python <file``. -In the latter case, input requests from the program, such as calls to -:func:`input` and :func:`raw_input`, are satisfied from *file*. Since this file -has already been read until the end by the parser before the program starts -executing, the program will encounter end-of-file immediately. In the former -case (which is usually what you want) they are satisfied from whatever file or -device is connected to standard input of the Python interpreter. - When a script file is used, it is sometimes useful to be able to run the script and enter interactive mode afterwards. This can be done by passing :option:`-i` before the script. (This does not work if the script is read from standard @@ -78,8 +70,9 @@ Argument Passing ---------------- When known to the interpreter, the script name and additional arguments -thereafter are passed to the script in the variable ``sys.argv``, which is a -list of strings. Its length is at least one; when no script and no arguments +thereafter are turned into a list of strings and assigned to the ``argv`` +variable in the ``sys`` module. You can access this list by executing ``import +sys``. The length of the list is at least one; when no script and no arguments are given, ``sys.argv[0]`` is an empty string. When the script name is given as ``'-'`` (meaning standard input), ``sys.argv[0]`` is set to ``'-'``. When :option:`-c` *command* is used, ``sys.argv[0]`` is set to ``'-c'``. When @@ -102,7 +95,7 @@ prints a welcome message stating its version number and a copyright notice before printing the first prompt:: python - Python 2.6 (#1, Feb 28 2007, 00:02:06) + Python 2.7 (#1, Feb 28 2010, 00:02:06) Type "help", "copyright", "credits" or "license" for more information. >>> diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index c953394..6d8f890 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -178,6 +178,13 @@ several ways. They can be enclosed in single quotes or double quotes:: >>> '"Isn\'t," she said.' '"Isn\'t," she said.' +The interpreter prints the result of string operations in the same way as they +are typed for input: inside quotes, and with quotes and other funny characters +escaped by backslashes, to show the precise value. The string is enclosed in +double quotes if the string contains a single quote and no double quotes, else +it's enclosed in single quotes. The :keyword:`print` statement produces a more +readable output for such input strings. + String literals can span multiple lines in several ways. Continuation lines can be used, with a backslash as the last character on the line indicating that the next line is a logical continuation of the line:: @@ -189,7 +196,7 @@ next line is a logical continuation of the line:: print hello -Note that newlines still need to be embedded in the string using ``\n``; the +Note that newlines still need to be embedded in the string using ``\n`` -- the newline following the trailing backslash is discarded. This example would print the following: diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index 48858f3..6f51357 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -332,8 +332,8 @@ want a list of those, they are defined in the standard module 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', - 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'min', - 'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range', + 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview', + 'min', 'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip'] diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index 9dedd96..844f8bc 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -14,11 +14,11 @@ The :mod:`os` module provides dozens of functions for interacting with the operating system:: >>> import os - >>> os.system('time 0:02') - 0 >>> os.getcwd() # Return the current working directory 'C:\\Python26' - >>> os.chdir('/server/accesslogs') + >>> os.chdir('/server/accesslogs') # Change current working directory + >>> os.system('mkdir today') # Run the command mkdir in the system shell + 0 Be sure to use the ``import os`` style instead of ``from os import *``. This will keep :func:`os.open` from shadowing the built-in :func:`open` function which @@ -72,7 +72,7 @@ three`` at the command line:: The :mod:`getopt` module processes *sys.argv* using the conventions of the Unix :func:`getopt` function. More powerful and flexible command line processing is -provided by the :mod:`optparse` module. +provided by the :mod:`argparse` module. .. _tut-stderr: diff --git a/Doc/tutorial/stdlib2.rst b/Doc/tutorial/stdlib2.rst index ed56364..514d583 100644 --- a/Doc/tutorial/stdlib2.rst +++ b/Doc/tutorial/stdlib2.rst @@ -362,10 +362,13 @@ results in decimal floating point and binary floating point. The difference becomes significant if the results are rounded to the nearest cent:: >>> from decimal import * - >>> Decimal('0.70') * Decimal('1.05') + >>> x = Decimal('0.70') * Decimal('1.05') + >>> x Decimal('0.7350') - >>> .70 * 1.05 - 0.73499999999999999 + >>> x.quantize(Decimal('0.01')) # round to nearest cent + Decimal('0.74') + >>> round(.70 * 1.05, 2) # same calculation with floats + 0.73 The :class:`Decimal` result keeps a trailing zero, automatically inferring four place significance from multiplicands with two place significance. Decimal diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index b41d244..a10ab1c 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -1,5 +1,8 @@ .. highlightlang:: none +.. ATTENTION: You probably should update Misc/python.man, too, if you modify +.. this file. + .. _using-on-general: Command line and environment @@ -78,6 +81,12 @@ source. the implementation may not always enforce this (e.g. it may allow you to use a name that includes a hyphen). + Package names are also permitted. When a package name is supplied instead + of a normal module, the interpreter will execute ``<pkg>.__main__`` as + the main module. This behaviour is deliberately similar to the handling + of directories and zipfiles that are passed to the interpreter as the + script argument. + .. note:: This option cannot be used with built-in modules and extension modules @@ -97,7 +106,7 @@ source. .. seealso:: :func:`runpy.run_module` - The actual implementation of this feature. + Equivalent functionality directly available to Python code :pep:`338` -- Executing modules as scripts @@ -106,6 +115,11 @@ source. .. versionchanged:: 2.5 The named module can now be located inside a package. + .. versionchanged:: 2.7 + Supply the package name to run a ``__main__`` submodule. + sys.argv[0] is now set to ``"-m"`` while searching for the module + (it was previously incorrectly set to ``"-c"``) + .. describe:: - @@ -301,6 +315,10 @@ Miscellaneous options :option:`-W` options are ignored (though, a warning message is printed about invalid options when the first warning is issued). + Starting from Python 2.7, :exc:`DeprecationWarning` and its descendants + are ignored by default. The :option:`-Wd` option can be used to re-enable + them. + Warnings can also be controlled from within a Python program using the :mod:`warnings` module. @@ -331,7 +349,7 @@ Miscellaneous options the remaining fields. Empty fields match all values; trailing empty fields may be omitted. The *message* field matches the start of the warning message printed; this match is case-insensitive. The *category* field matches the - warning category. This must be a class name; the match test whether the + warning category. This must be a class name; the match tests whether the actual warning category of the message is a subclass of the specified warning category. The full class name must be given. The *module* field matches the (fully-qualified) module name; this match is case-sensitive. The *line* @@ -343,6 +361,8 @@ Miscellaneous options :pep:`230` -- Warning framework + :envvar:`PYTHONWARNINGS` + .. cmdoption:: -x @@ -538,6 +558,12 @@ These environment variables influence Python's behavior. value instead of the value got through the C runtime. Only works on Mac OS X. +.. envvar:: PYTHONWARNINGS + + This is equivalent to the :option:`-W` option. If set to a comma + separated string, it is equivalent to specifying :option:`-W` multiple + times. + Debug-mode variables ~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/using/index.rst b/Doc/using/index.rst index 6ce5a00..1201153 100644 --- a/Doc/using/index.rst +++ b/Doc/using/index.rst @@ -1,8 +1,8 @@ .. _using-index: -################ - Using Python -################ +########################## + Python Setup and Usage +########################## This part of the documentation is devoted to general information on the setup diff --git a/Doc/using/mac.rst b/Doc/using/mac.rst index 18bfb82..c5f7f2d 100644 --- a/Doc/using/mac.rst +++ b/Doc/using/mac.rst @@ -122,8 +122,8 @@ The IDE ======= MacPython ships with the standard IDLE development environment. A good -introduction to using IDLE can be found at http://hkn.eecs.berkeley.edu/ -dyoo/python/idle_intro/index.html. +introduction to using IDLE can be found at +http://hkn.eecs.berkeley.edu/~dyoo/python/idle_intro/index.html. .. _mac-package-manager: diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 52349d7..8a91ef7 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -158,23 +158,48 @@ installation directory. So, if you had installed Python to :file:`C:\\Python\\Lib\\` and third-party modules should be stored in :file:`C:\\Python\\Lib\\site-packages\\`. -.. `` this fixes syntax highlighting errors in some editors due to the \\ hackery - -You can add folders to your search path to make Python's import mechanism search -in these directories as well. Use :envvar:`PYTHONPATH`, as described in -:ref:`using-on-envvars`, to modify :data:`sys.path`. On Windows, paths are -separated by semicolons, though, to distinguish them from drive identifiers -(:file:`C:\\` etc.). - -.. `` - -Modifying the module search path can also be done through the Windows registry -under the key :file:`HKLM\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath`. -Subkeys which have semicolon-delimited path strings as their default value will -cause each path to be searched. Multiple subkeys can be created and are -appended to the path in alphabetical order. A convenient registry editor is -:program:`regedit` (start it by typing "regedit" into :menuselection:`Start --> -Run`). +This is how :data:`sys.path` is populated on Windows: + +* An empty entry is added at the start, which corresponds to the current + directory. + +* If the environment variable :envvar:`PYTHONPATH` exists, as described in + :ref:`using-on-envvars`, its entries are added next. Note that on Windows, + paths in this variable must be separated by semicolons, to distinguish them + from the colon used in drive identifiers (``C:\`` etc.). + +* Additional "application paths" can be added in the registry as subkeys of + :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the + ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives. Subkeys which have + semicolon-delimited path strings as their default value will cause each path + to be added to :data:`sys.path`. (Note that all known installers only use + HKLM, so HKCU is typically empty.) + +* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as + "Python Home". Otherwise, the path of the main Python executable is used to + locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home". If a + Python home is found, the relevant sub-directories added to :data:`sys.path` + (``Lib``, ``plat-win``, etc) are based on that folder. Otherwise, the core + Python path is constructed from the PythonPath stored in the registry. + +* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in + the environment, and no registry entries can be found, a default path with + relative entries is used (e.g. ``.\Lib;.\plat-win``, etc). + +The end result of all this is: + +* When running :file:`python.exe`, or any other .exe in the main Python + directory (either an installed version, or directly from the PCbuild + directory), the core path is deduced, and the core paths in the registry are + ignored. Other "application paths" in the registry are always read. + +* When Python is hosted in another .exe (different directory, embedded via COM, + etc), the "Python Home" will not be deduced, so the core path from the + registry is used. Other "application paths" in the registry are always read. + +* If Python can't find its home and there is no registry (eg, frozen .exe, some + very strange installation setup) you get a path with some default, but + relative, paths. Executing scripts diff --git a/Doc/whatsnew/2.4.rst b/Doc/whatsnew/2.4.rst index 27e412e..fadde50 100644 --- a/Doc/whatsnew/2.4.rst +++ b/Doc/whatsnew/2.4.rst @@ -1066,7 +1066,7 @@ complete list of changes, or look through the CVS logs for all the details. deprecated APIs and removes support for Python versions earlier than 2.3. The 3.0 version of the package uses a new incremental parser for MIME messages, available in the :mod:`email.FeedParser` module. The new parser doesn't require - reading the entire message into memory, and doesn't throw exceptions if a + reading the entire message into memory, and doesn't raise exceptions if a message is malformed; instead it records any problems in the :attr:`defect` attribute of the message. (Developed by Anthony Baxter, Barry Warsaw, Thomas Wouters, and others.) diff --git a/Doc/whatsnew/2.6.rst b/Doc/whatsnew/2.6.rst index da1256c..ccc97b8 100644 --- a/Doc/whatsnew/2.6.rst +++ b/Doc/whatsnew/2.6.rst @@ -2992,33 +2992,6 @@ Changes to Python's build process and to the C API include: architectures (x86, PowerPC), 64-bit (x86-64 and PPC-64), or both. (Contributed by Ronald Oussoren.) -* A new function added in Python 2.6.6, :cfunc:`PySys_SetArgvEx`, sets - the value of ``sys.argv`` and can optionally update ``sys.path`` to - include the directory containing the script named by ``sys.argv[0]`` - depending on the value of an *updatepath* parameter. - - This function was added to close a security hole for applications - that embed Python. The old function, :cfunc:`PySys_SetArgv`, would - always update ``sys.path``, and sometimes it would add the current - directory. This meant that, if you ran an application embedding - Python in a directory controlled by someone else, attackers could - put a Trojan-horse module in the directory (say, a file named - :file:`os.py`) that your application would then import and run. - - If you maintain a C/C++ application that embeds Python, check - whether you're calling :cfunc:`PySys_SetArgv` and carefully consider - whether the application should be using :cfunc:`PySys_SetArgvEx` - with *updatepath* set to false. Note that using this function will - break compatibility with Python versions 2.6.5 and earlier; if you - have to continue working with earlier versions, you can leave - the call to :cfunc:`PySys_SetArgv` alone and call - ``PyRun_SimpleString("sys.path.pop(0)\n")`` afterwards to discard - the first ``sys.path`` component. - - Security issue reported as `CVE-2008-5983 - <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_; - discussed in :issue:`5753`, and fixed by Antoine Pitrou. - * The BerkeleyDB module now has a C API object, available as ``bsddb.db.api``. This object can be used by other C extensions that wish to use the :mod:`bsddb` module for their own purposes. @@ -3321,15 +3294,6 @@ that may require changes to your code: scoping rules, also cause warnings because such comparisons are forbidden entirely in 3.0. -For applications that embed Python: - -* The :cfunc:`PySys_SetArgvEx` function was added in Python 2.6.6, - letting applications close a security hole when the existing - :cfunc:`PySys_SetArgv` function was used. Check whether you're - calling :cfunc:`PySys_SetArgv` and carefully consider whether the - application should be using :cfunc:`PySys_SetArgvEx` with - *updatepath* set to false. - .. ====================================================================== diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst new file mode 100644 index 0000000..6cd3aba --- /dev/null +++ b/Doc/whatsnew/2.7.rst @@ -0,0 +1,2475 @@ +**************************** + What's New in Python 2.7 +**************************** + +:Author: A.M. Kuchling (amk at amk.ca) +:Release: |release| +:Date: |today| + +.. hyperlink all the methods & functions. + +.. T_STRING_INPLACE not described in main docs + +.. $Id$ + Rules for maintenance: + + * Anyone can add text to this document. Do not spend very much time + on the wording of your changes, because your text will probably + get rewritten to some degree. + + * The maintainer will go through Misc/NEWS periodically and add + changes; it's therefore more important to add your changes to + Misc/NEWS than to this file. + + * This is not a complete list of every single change; completeness + is the purpose of Misc/NEWS. Some changes I consider too small + or esoteric to include. If such a change is added to the text, + I'll just remove it. (This is another reason you shouldn't spend + too much time on writing your addition.) + + * If you want to draw your new text to the attention of the + maintainer, add 'XXX' to the beginning of the paragraph or + section. + + * It's OK to just add a fragmentary note about a change. For + example: "XXX Describe the transmogrify() function added to the + socket module." The maintainer will research the change and + write the necessary text. + + * You can comment out your additions if you like, but it's not + necessary (especially when a final release is some months away). + + * Credit the author of a patch or bugfix. Just the name is + sufficient; the e-mail address isn't necessary. + + * It's helpful to add the bug/patch number in a parenthetical comment. + + XXX Describe the transmogrify() function added to the socket + module. + (Contributed by P.Y. Developer; :issue:`12345`.) + + This saves the maintainer some effort going through the SVN logs + when researching a change. + +This article explains the new features in Python 2.7. Python 2.7 was released +on July 3, 2010. + +Numeric handling has been improved in many ways, for both +floating-point numbers and for the :class:`~decimal.Decimal` class. +There are some useful additions to the standard library, such as a +greatly enhanced :mod:`unittest` module, the :mod:`argparse` module +for parsing command-line options, convenient :class:`~collections.OrderedDict` +and :class:`~collections.Counter` classes in the :mod:`collections` module, +and many other improvements. + +Python 2.7 is planned to be the last of the 2.x releases, so we worked +on making it a good release for the long term. To help with porting +to Python 3, several new features from the Python 3.x series have been +included in 2.7. + +This article doesn't attempt to provide a complete specification of +the new features, but instead provides a convenient overview. For +full details, you should refer to the documentation for Python 2.7 at +http://docs.python.org. If you want to understand the rationale for +the design and implementation, refer to the PEP for a particular new +feature or the issue on http://bugs.python.org in which a change was +discussed. Whenever possible, "What's New in Python" links to the +bug/patch item for each change. + +.. _whatsnew27-python31: + +The Future for Python 2.x +========================= + +Python 2.7 is intended to be the last major release in the 2.x series. +The Python maintainers are planning to focus their future efforts on +the Python 3.x series. + +This means that 2.7 will remain in place for a long time, running +production systems that have not been ported to Python 3.x. +Two consequences of the long-term significance of 2.7 are: + +* It's very likely the 2.7 release will have a longer period of + maintenance compared to earlier 2.x versions. Python 2.7 will + continue to be maintained while the transition to 3.x continues, and + the developers are planning to support Python 2.7 with bug-fix + releases beyond the typical two years. + +* A policy decision was made to silence warnings only of interest to + developers. :exc:`DeprecationWarning` and its + descendants are now ignored unless otherwise requested, preventing + users from seeing warnings triggered by an application. This change + was also made in the branch that will become Python 3.2. (Discussed + on stdlib-sig and carried out in :issue:`7319`.) + + In previous releases, :exc:`DeprecationWarning` messages were + enabled by default, providing Python developers with a clear + indication of where their code may break in a future major version + of Python. + + However, there are increasingly many users of Python-based + applications who are not directly involved in the development of + those applications. :exc:`DeprecationWarning` messages are + irrelevant to such users, making them worry about an application + that's actually working correctly and burdening application developers + with responding to these concerns. + + You can re-enable display of :exc:`DeprecationWarning` messages by + running Python with the :option:`-Wdefault <-W>` (short form: + :option:`-Wd <-W>`) switch, or by setting the :envvar:`PYTHONWARNINGS` + environment variable to ``"default"`` (or ``"d"``) before running + Python. Python code can also re-enable them + by calling ``warnings.simplefilter('default')``. + + +Python 3.1 Features +======================= + +Much as Python 2.6 incorporated features from Python 3.0, +version 2.7 incorporates some of the new features +in Python 3.1. The 2.x series continues to provide tools +for migrating to the 3.x series. + +A partial list of 3.1 features that were backported to 2.7: + +* The syntax for set literals (``{1,2,3}`` is a mutable set). +* Dictionary and set comprehensions (``{i: i*2 for i in range(3)}``). +* Multiple context managers in a single :keyword:`with` statement. +* A new version of the :mod:`io` library, rewritten in C for performance. +* The ordered-dictionary type described in :ref:`pep-0372`. +* The new ``","`` format specifier described in :ref:`pep-0378`. +* The :class:`memoryview` object. +* A small subset of the :mod:`importlib` module, + `described below <#importlib-section>`__. +* The :func:`repr` of a float ``x`` is shorter in many cases: it's now + based on the shortest decimal string that's guaranteed to round back + to ``x``. As in previous versions of Python, it's guaranteed that + ``float(repr(x))`` recovers ``x``. +* Float-to-string and string-to-float conversions are correctly rounded. + The :func:`round` function is also now correctly rounded. +* The :ctype:`PyCapsule` type, used to provide a C API for extension modules. +* The :cfunc:`PyLong_AsLongAndOverflow` C API function. + +Other new Python3-mode warnings include: + +* :func:`operator.isCallable` and :func:`operator.sequenceIncludes`, + which are not supported in 3.x, now trigger warnings. +* The :option:`-3` switch now automatically + enables the :option:`-Qwarn <-Q>` switch that causes warnings + about using classic division with integers and long integers. + + + +.. ======================================================================== +.. Large, PEP-level features and changes should be described here. +.. ======================================================================== + +.. _pep-0372: + +PEP 372: Adding an Ordered Dictionary to collections +==================================================== + +Regular Python dictionaries iterate over key/value pairs in arbitrary order. +Over the years, a number of authors have written alternative implementations +that remember the order that the keys were originally inserted. Based on +the experiences from those implementations, 2.7 introduces a new +:class:`~collections.OrderedDict` class in the :mod:`collections` module. + +The :class:`~collections.OrderedDict` API provides the same interface as regular +dictionaries but iterates over keys and values in a guaranteed order +depending on when a key was first inserted:: + + >>> from collections import OrderedDict + >>> d = OrderedDict([('first', 1), + ... ('second', 2), + ... ('third', 3)]) + >>> d.items() + [('first', 1), ('second', 2), ('third', 3)] + +If a new entry overwrites an existing entry, the original insertion +position is left unchanged:: + + >>> d['second'] = 4 + >>> d.items() + [('first', 1), ('second', 4), ('third', 3)] + +Deleting an entry and reinserting it will move it to the end:: + + >>> del d['second'] + >>> d['second'] = 5 + >>> d.items() + [('first', 1), ('third', 3), ('second', 5)] + +The :meth:`~collections.OrderedDict.popitem` method has an optional *last* +argument that defaults to True. If *last* is True, the most recently +added key is returned and removed; if it's False, the +oldest key is selected:: + + >>> od = OrderedDict([(x,0) for x in range(20)]) + >>> od.popitem() + (19, 0) + >>> od.popitem() + (18, 0) + >>> od.popitem(last=False) + (0, 0) + >>> od.popitem(last=False) + (1, 0) + +Comparing two ordered dictionaries checks both the keys and values, +and requires that the insertion order was the same:: + + >>> od1 = OrderedDict([('first', 1), + ... ('second', 2), + ... ('third', 3)]) + >>> od2 = OrderedDict([('third', 3), + ... ('first', 1), + ... ('second', 2)]) + >>> od1 == od2 + False + >>> # Move 'third' key to the end + >>> del od2['third']; od2['third'] = 3 + >>> od1 == od2 + True + +Comparing an :class:`~collections.OrderedDict` with a regular dictionary +ignores the insertion order and just compares the keys and values. + +How does the :class:`~collections.OrderedDict` work? It maintains a +doubly-linked list of keys, appending new keys to the list as they're inserted. +A secondary dictionary maps keys to their corresponding list node, so +deletion doesn't have to traverse the entire linked list and therefore +remains O(1). + +The standard library now supports use of ordered dictionaries in several +modules. + +* The :mod:`ConfigParser` module uses them by default, meaning that + configuration files can now be read, modified, and then written back + in their original order. + +* The :meth:`~collections.somenamedtuple._asdict()` method for + :func:`collections.namedtuple` now returns an ordered dictionary with the + values appearing in the same order as the underlying tuple indices. + +* The :mod:`json` module's :class:`~json.JSONDecoder` class + constructor was extended with an *object_pairs_hook* parameter to + allow :class:`OrderedDict` instances to be built by the decoder. + Support was also added for third-party tools like + `PyYAML <http://pyyaml.org/>`_. + +.. seealso:: + + :pep:`372` - Adding an ordered dictionary to collections + PEP written by Armin Ronacher and Raymond Hettinger; + implemented by Raymond Hettinger. + +.. _pep-0378: + +PEP 378: Format Specifier for Thousands Separator +================================================= + +To make program output more readable, it can be useful to add +separators to large numbers, rendering them as +18,446,744,073,709,551,616 instead of 18446744073709551616. + +The fully general solution for doing this is the :mod:`locale` module, +which can use different separators ("," in North America, "." in +Europe) and different grouping sizes, but :mod:`locale` is complicated +to use and unsuitable for multi-threaded applications where different +threads are producing output for different locales. + +Therefore, a simple comma-grouping mechanism has been added to the +mini-language used by the :meth:`str.format` method. When +formatting a floating-point number, simply include a comma between the +width and the precision:: + + >>> '{:20,.2f}'.format(18446744073709551616.0) + '18,446,744,073,709,551,616.00' + +When formatting an integer, include the comma after the width: + + >>> '{:20,d}'.format(18446744073709551616) + '18,446,744,073,709,551,616' + +This mechanism is not adaptable at all; commas are always used as the +separator and the grouping is always into three-digit groups. The +comma-formatting mechanism isn't as general as the :mod:`locale` +module, but it's easier to use. + +.. seealso:: + + :pep:`378` - Format Specifier for Thousands Separator + PEP written by Raymond Hettinger; implemented by Eric Smith. + +PEP 389: The argparse Module for Parsing Command Lines +====================================================== + +The :mod:`argparse` module for parsing command-line arguments was +added as a more powerful replacement for the +:mod:`optparse` module. + +This means Python now supports three different modules for parsing +command-line arguments: :mod:`getopt`, :mod:`optparse`, and +:mod:`argparse`. The :mod:`getopt` module closely resembles the C +library's :cfunc:`getopt` function, so it remains useful if you're writing a +Python prototype that will eventually be rewritten in C. +:mod:`optparse` becomes redundant, but there are no plans to remove it +because there are many scripts still using it, and there's no +automated way to update these scripts. (Making the :mod:`argparse` +API consistent with :mod:`optparse`'s interface was discussed but +rejected as too messy and difficult.) + +In short, if you're writing a new script and don't need to worry +about compatibility with earlier versions of Python, use +:mod:`argparse` instead of :mod:`optparse`. + +Here's an example:: + + import argparse + + parser = argparse.ArgumentParser(description='Command-line example.') + + # Add optional switches + parser.add_argument('-v', action='store_true', dest='is_verbose', + help='produce verbose output') + parser.add_argument('-o', action='store', dest='output', + metavar='FILE', + help='direct output to FILE instead of stdout') + parser.add_argument('-C', action='store', type=int, dest='context', + metavar='NUM', default=0, + help='display NUM lines of added context') + + # Allow any number of additional arguments. + parser.add_argument(nargs='*', action='store', dest='inputs', + help='input filenames (default is stdin)') + + args = parser.parse_args() + print args.__dict__ + +Unless you override it, :option:`-h` and :option:`--help` switches +are automatically added, and produce neatly formatted output:: + + -> ./python.exe argparse-example.py --help + usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]] + + Command-line example. + + positional arguments: + inputs input filenames (default is stdin) + + optional arguments: + -h, --help show this help message and exit + -v produce verbose output + -o FILE direct output to FILE instead of stdout + -C NUM display NUM lines of added context + +As with :mod:`optparse`, the command-line switches and arguments +are returned as an object with attributes named by the *dest* parameters:: + + -> ./python.exe argparse-example.py -v + {'output': None, + 'is_verbose': True, + 'context': 0, + 'inputs': []} + + -> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2 + {'output': '/tmp/output', + 'is_verbose': True, + 'context': 4, + 'inputs': ['file1', 'file2']} + +:mod:`argparse` has much fancier validation than :mod:`optparse`; you +can specify an exact number of arguments as an integer, 0 or more +arguments by passing ``'*'``, 1 or more by passing ``'+'``, or an +optional argument with ``'?'``. A top-level parser can contain +sub-parsers to define subcommands that have different sets of +switches, as in ``svn commit``, ``svn checkout``, etc. You can +specify an argument's type as :class:`~argparse.FileType`, which will +automatically open files for you and understands that ``'-'`` means +standard input or output. + +.. seealso:: + + :mod:`argparse` documentation + The documentation page of the argparse module. + + :ref:`argparse-from-optparse` + Part of the Python documentation, describing how to convert + code that uses :mod:`optparse`. + + :pep:`389` - argparse - New Command Line Parsing Module + PEP written and implemented by Steven Bethard. + +PEP 391: Dictionary-Based Configuration For Logging +==================================================== + +The :mod:`logging` module is very flexible; applications can define +a tree of logging subsystems, and each logger in this tree can filter +out certain messages, format them differently, and direct messages to +a varying number of handlers. + +All this flexibility can require a lot of configuration. You can +write Python statements to create objects and set their properties, +but a complex set-up requires verbose but boring code. +:mod:`logging` also supports a :func:`~logging.fileConfig` +function that parses a file, but the file format doesn't support +configuring filters, and it's messier to generate programmatically. + +Python 2.7 adds a :func:`~logging.dictConfig` function that +uses a dictionary to configure logging. There are many ways to +produce a dictionary from different sources: construct one with code; +parse a file containing JSON; or use a YAML parsing library if one is +installed. For more information see :ref:`logging-config-api`. + +The following example configures two loggers, the root logger and a +logger named "network". Messages sent to the root logger will be +sent to the system log using the syslog protocol, and messages +to the "network" logger will be written to a :file:`network.log` file +that will be rotated once the log reaches 1MB. + +:: + + import logging + import logging.config + + configdict = { + 'version': 1, # Configuration schema in use; must be 1 for now + 'formatters': { + 'standard': { + 'format': ('%(asctime)s %(name)-15s ' + '%(levelname)-8s %(message)s')}}, + + 'handlers': {'netlog': {'backupCount': 10, + 'class': 'logging.handlers.RotatingFileHandler', + 'filename': '/logs/network.log', + 'formatter': 'standard', + 'level': 'INFO', + 'maxBytes': 1000000}, + 'syslog': {'class': 'logging.handlers.SysLogHandler', + 'formatter': 'standard', + 'level': 'ERROR'}}, + + # Specify all the subordinate loggers + 'loggers': { + 'network': { + 'handlers': ['netlog'] + } + }, + # Specify properties of the root logger + 'root': { + 'handlers': ['syslog'] + }, + } + + # Set up configuration + logging.config.dictConfig(configdict) + + # As an example, log two error messages + logger = logging.getLogger('/') + logger.error('Database not found') + + netlogger = logging.getLogger('network') + netlogger.error('Connection failed') + +Three smaller enhancements to the :mod:`logging` module, all +implemented by Vinay Sajip, are: + +.. rev79293 + +* The :class:`~logging.handlers.SysLogHandler` class now supports + syslogging over TCP. The constructor has a *socktype* parameter + giving the type of socket to use, either :const:`socket.SOCK_DGRAM` + for UDP or :const:`socket.SOCK_STREAM` for TCP. The default + protocol remains UDP. + +* :class:`~logging.Logger` instances gained a :meth:`~logging.Logger.getChild` + method that retrieves a descendant logger using a relative path. + For example, once you retrieve a logger by doing ``log = getLogger('app')``, + calling ``log.getChild('network.listen')`` is equivalent to + ``getLogger('app.network.listen')``. + +* The :class:`~logging.LoggerAdapter` class gained a + :meth:`~logging.LoggerAdapter.isEnabledFor` method that takes a + *level* and returns whether the underlying logger would + process a message of that level of importance. + +.. XXX: Logger objects don't have a class declaration so the link don't work + +.. seealso:: + + :pep:`391` - Dictionary-Based Configuration For Logging + PEP written and implemented by Vinay Sajip. + +PEP 3106: Dictionary Views +==================================================== + +The dictionary methods :meth:`~dict.keys`, :meth:`~dict.values`, and +:meth:`~dict.items` are different in Python 3.x. They return an object +called a :dfn:`view` instead of a fully materialized list. + +It's not possible to change the return values of :meth:`~dict.keys`, +:meth:`~dict.values`, and :meth:`~dict.items` in Python 2.7 because +too much code would break. Instead the 3.x versions were added +under the new names :meth:`~dict.viewkeys`, :meth:`~dict.viewvalues`, +and :meth:`~dict.viewitems`. + +:: + + >>> d = dict((i*10, chr(65+i)) for i in range(26)) + >>> d + {0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'} + >>> d.viewkeys() + dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250]) + +Views can be iterated over, but the key and item views also behave +like sets. The ``&`` operator performs intersection, and ``|`` +performs a union:: + + >>> d1 = dict((i*10, chr(65+i)) for i in range(26)) + >>> d2 = dict((i**.5, i) for i in range(1000)) + >>> d1.viewkeys() & d2.viewkeys() + set([0.0, 10.0, 20.0, 30.0]) + >>> d1.viewkeys() | range(0, 30) + set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250]) + +The view keeps track of the dictionary and its contents change as the +dictionary is modified:: + + >>> vk = d.viewkeys() + >>> vk + dict_keys([0, 130, 10, ..., 250]) + >>> d[260] = '&' + >>> vk + dict_keys([0, 130, 260, 10, ..., 250]) + +However, note that you can't add or remove keys while you're iterating +over the view:: + + >>> for k in vk: + ... d[k*2] = k + ... + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + RuntimeError: dictionary changed size during iteration + +You can use the view methods in Python 2.x code, and the 2to3 +converter will change them to the standard :meth:`~dict.keys`, +:meth:`~dict.values`, and :meth:`~dict.items` methods. + +.. seealso:: + + :pep:`3106` - Revamping dict.keys(), .values() and .items() + PEP written by Guido van Rossum. + Backported to 2.7 by Alexandre Vassalotti; :issue:`1967`. + + +PEP 3137: The memoryview Object +==================================================== + +The :class:`memoryview` object provides a view of another object's +memory content that matches the :class:`bytes` type's interface. + + >>> import string + >>> m = memoryview(string.letters) + >>> m + <memory at 0x37f850> + >>> len(m) # Returns length of underlying object + 52 + >>> m[0], m[25], m[26] # Indexing returns one byte + ('a', 'z', 'A') + >>> m2 = m[0:26] # Slicing returns another memoryview + >>> m2 + <memory at 0x37f080> + +The content of the view can be converted to a string of bytes or +a list of integers: + + >>> m2.tobytes() + 'abcdefghijklmnopqrstuvwxyz' + >>> m2.tolist() + [97, 98, 99, 100, 101, 102, 103, ... 121, 122] + >>> + +:class:`memoryview` objects allow modifying the underlying object if +it's a mutable object. + + >>> m2[0] = 75 + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + TypeError: cannot modify read-only memory + >>> b = bytearray(string.letters) # Creating a mutable object + >>> b + bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ') + >>> mb = memoryview(b) + >>> mb[0] = '*' # Assign to view, changing the bytearray. + >>> b[0:5] # The bytearray has been changed. + bytearray(b'*bcde') + >>> + +.. seealso:: + + :pep:`3137` - Immutable Bytes and Mutable Buffer + PEP written by Guido van Rossum. + Implemented by Travis Oliphant, Antoine Pitrou and others. + Backported to 2.7 by Antoine Pitrou; :issue:`2396`. + + + +Other Language Changes +====================== + +Some smaller changes made to the core Python language are: + +* The syntax for set literals has been backported from Python 3.x. + Curly brackets are used to surround the contents of the resulting + mutable set; set literals are + distinguished from dictionaries by not containing colons and values. + ``{}`` continues to represent an empty dictionary; use + ``set()`` for an empty set. + + >>> {1, 2, 3, 4, 5} + set([1, 2, 3, 4, 5]) + >>> set() # empty set + set([]) + >>> {} # empty dict + {} + + Backported by Alexandre Vassalotti; :issue:`2335`. + +* Dictionary and set comprehensions are another feature backported from + 3.x, generalizing list/generator comprehensions to use + the literal syntax for sets and dictionaries. + + >>> {x: x*x for x in range(6)} + {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25} + >>> {('a'*x) for x in range(6)} + set(['', 'a', 'aa', 'aaa', 'aaaa', 'aaaaa']) + + Backported by Alexandre Vassalotti; :issue:`2333`. + +* The :keyword:`with` statement can now use multiple context managers + in one statement. Context managers are processed from left to right + and each one is treated as beginning a new :keyword:`with` statement. + This means that:: + + with A() as a, B() as b: + ... suite of statements ... + + is equivalent to:: + + with A() as a: + with B() as b: + ... suite of statements ... + + The :func:`contextlib.nested` function provides a very similar + function, so it's no longer necessary and has been deprecated. + + (Proposed in http://codereview.appspot.com/53094; implemented by + Georg Brandl.) + +* Conversions between floating-point numbers and strings are + now correctly rounded on most platforms. These conversions occur + in many different places: :func:`str` on + floats and complex numbers; the :class:`float` and :class:`complex` + constructors; + numeric formatting; serializing and + deserializing floats and complex numbers using the + :mod:`marshal`, :mod:`pickle` + and :mod:`json` modules; + parsing of float and imaginary literals in Python code; + and :class:`~decimal.Decimal`-to-float conversion. + + Related to this, the :func:`repr` of a floating-point number *x* + now returns a result based on the shortest decimal string that's + guaranteed to round back to *x* under correct rounding (with + round-half-to-even rounding mode). Previously it gave a string + based on rounding x to 17 decimal digits. + + .. maybe add an example? + + The rounding library responsible for this improvement works on + Windows and on Unix platforms using the gcc, icc, or suncc + compilers. There may be a small number of platforms where correct + operation of this code cannot be guaranteed, so the code is not + used on such systems. You can find out which code is being used + by checking :data:`sys.float_repr_style`, which will be ``short`` + if the new code is in use and ``legacy`` if it isn't. + + Implemented by Eric Smith and Mark Dickinson, using David Gay's + :file:`dtoa.c` library; :issue:`7117`. + +* Conversions from long integers and regular integers to floating + point now round differently, returning the floating-point number + closest to the number. This doesn't matter for small integers that + can be converted exactly, but for large numbers that will + unavoidably lose precision, Python 2.7 now approximates more + closely. For example, Python 2.6 computed the following:: + + >>> n = 295147905179352891391 + >>> float(n) + 2.9514790517935283e+20 + >>> n - long(float(n)) + 65535L + + Python 2.7's floating-point result is larger, but much closer to the + true value:: + + >>> n = 295147905179352891391 + >>> float(n) + 2.9514790517935289e+20 + >>> n - long(float(n)) + -1L + + (Implemented by Mark Dickinson; :issue:`3166`.) + + Integer division is also more accurate in its rounding behaviours. (Also + implemented by Mark Dickinson; :issue:`1811`.) + +* Implicit coercion for complex numbers has been removed; the interpreter + will no longer ever attempt to call a :meth:`__coerce__` method on complex + objects. (Removed by Meador Inge and Mark Dickinson; :issue:`5211`.) + +* The :meth:`str.format` method now supports automatic numbering of the replacement + fields. This makes using :meth:`str.format` more closely resemble using + ``%s`` formatting:: + + >>> '{}:{}:{}'.format(2009, 04, 'Sunday') + '2009:4:Sunday' + >>> '{}:{}:{day}'.format(2009, 4, day='Sunday') + '2009:4:Sunday' + + The auto-numbering takes the fields from left to right, so the first ``{...}`` + specifier will use the first argument to :meth:`str.format`, the next + specifier will use the next argument, and so on. You can't mix auto-numbering + and explicit numbering -- either number all of your specifier fields or none + of them -- but you can mix auto-numbering and named fields, as in the second + example above. (Contributed by Eric Smith; :issue:`5237`.) + + Complex numbers now correctly support usage with :func:`format`, + and default to being right-aligned. + Specifying a precision or comma-separation applies to both the real + and imaginary parts of the number, but a specified field width and + alignment is applied to the whole of the resulting ``1.5+3j`` + output. (Contributed by Eric Smith; :issue:`1588` and :issue:`7988`.) + + The 'F' format code now always formats its output using uppercase characters, + so it will now produce 'INF' and 'NAN'. + (Contributed by Eric Smith; :issue:`3382`.) + + A low-level change: the :meth:`object.__format__` method now triggers + a :exc:`PendingDeprecationWarning` if it's passed a format string, + because the :meth:`__format__` method for :class:`object` converts + the object to a string representation and formats that. Previously + the method silently applied the format string to the string + representation, but that could hide mistakes in Python code. If + you're supplying formatting information such as an alignment or + precision, presumably you're expecting the formatting to be applied + in some object-specific way. (Fixed by Eric Smith; :issue:`7994`.) + +* The :func:`int` and :func:`long` types gained a ``bit_length`` + method that returns the number of bits necessary to represent + its argument in binary:: + + >>> n = 37 + >>> bin(n) + '0b100101' + >>> n.bit_length() + 6 + >>> n = 2**123-1 + >>> n.bit_length() + 123 + >>> (n+1).bit_length() + 124 + + (Contributed by Fredrik Johansson and Victor Stinner; :issue:`3439`.) + +* The :keyword:`import` statement will no longer try a relative import + if an absolute import (e.g. ``from .os import sep``) fails. This + fixes a bug, but could possibly break certain :keyword:`import` + statements that were only working by accident. (Fixed by Meador Inge; + :issue:`7902`.) + +* It's now possible for a subclass of the built-in :class:`unicode` type + to override the :meth:`__unicode__` method. (Implemented by + Victor Stinner; :issue:`1583863`.) + +* The :class:`bytearray` type's :meth:`~bytearray.translate` method now accepts + ``None`` as its first argument. (Fixed by Georg Brandl; + :issue:`4759`.) + + .. XXX bytearray doesn't seem to be documented + +* When using ``@classmethod`` and ``@staticmethod`` to wrap + methods as class or static methods, the wrapper object now + exposes the wrapped function as their :attr:`__func__` attribute. + (Contributed by Amaury Forgeot d'Arc, after a suggestion by + George Sakkis; :issue:`5982`.) + +* When a restricted set of attributes were set using ``__slots__``, + deleting an unset attribute would not raise :exc:`AttributeError` + as you would expect. Fixed by Benjamin Peterson; :issue:`7604`.) + +* Two new encodings are now supported: "cp720", used primarily for + Arabic text; and "cp858", a variant of CP 850 that adds the euro + symbol. (CP720 contributed by Alexander Belchenko and Amaury + Forgeot d'Arc in :issue:`1616979`; CP858 contributed by Tim Hatch in + :issue:`8016`.) + +* The :class:`file` object will now set the :attr:`filename` attribute + on the :exc:`IOError` exception when trying to open a directory + on POSIX platforms (noted by Jan Kaliszewski; :issue:`4764`), and + now explicitly checks for and forbids writing to read-only file objects + instead of trusting the C library to catch and report the error + (fixed by Stefan Krah; :issue:`5677`). + +* The Python tokenizer now translates line endings itself, so the + :func:`compile` built-in function now accepts code using any + line-ending convention. Additionally, it no longer requires that the + code end in a newline. + +* Extra parentheses in function definitions are illegal in Python 3.x, + meaning that you get a syntax error from ``def f((x)): pass``. In + Python3-warning mode, Python 2.7 will now warn about this odd usage. + (Noted by James Lingard; :issue:`7362`.) + +* It's now possible to create weak references to old-style class + objects. New-style classes were always weak-referenceable. (Fixed + by Antoine Pitrou; :issue:`8268`.) + +* When a module object is garbage-collected, the module's dictionary is + now only cleared if no one else is holding a reference to the + dictionary (:issue:`7140`). + +.. ====================================================================== + +.. _new-27-interpreter: + +Interpreter Changes +------------------------------- + +A new environment variable, :envvar:`PYTHONWARNINGS`, +allows controlling warnings. It should be set to a string +containing warning settings, equivalent to those +used with the :option:`-W` switch, separated by commas. +(Contributed by Brian Curtin; :issue:`7301`.) + +For example, the following setting will print warnings every time +they occur, but turn warnings from the :mod:`Cookie` module into an +error. (The exact syntax for setting an environment variable varies +across operating systems and shells.) + +:: + + export PYTHONWARNINGS=all,error:::Cookie:0 + +.. ====================================================================== + + +Optimizations +------------- + +Several performance enhancements have been added: + +* A new opcode was added to perform the initial setup for + :keyword:`with` statements, looking up the :meth:`__enter__` and + :meth:`__exit__` methods. (Contributed by Benjamin Peterson.) + +* The garbage collector now performs better for one common usage + pattern: when many objects are being allocated without deallocating + any of them. This would previously take quadratic + time for garbage collection, but now the number of full garbage collections + is reduced as the number of objects on the heap grows. + The new logic only performs a full garbage collection pass when + the middle generation has been collected 10 times and when the + number of survivor objects from the middle generation exceeds 10% of + the number of objects in the oldest generation. (Suggested by Martin + von Löwis and implemented by Antoine Pitrou; :issue:`4074`.) + +* The garbage collector tries to avoid tracking simple containers + which can't be part of a cycle. In Python 2.7, this is now true for + tuples and dicts containing atomic types (such as ints, strings, + etc.). Transitively, a dict containing tuples of atomic types won't + be tracked either. This helps reduce the cost of each + garbage collection by decreasing the number of objects to be + considered and traversed by the collector. + (Contributed by Antoine Pitrou; :issue:`4688`.) + +* Long integers are now stored internally either in base 2**15 or in base + 2**30, the base being determined at build time. Previously, they + were always stored in base 2**15. Using base 2**30 gives + significant performance improvements on 64-bit machines, but + benchmark results on 32-bit machines have been mixed. Therefore, + the default is to use base 2**30 on 64-bit machines and base 2**15 + on 32-bit machines; on Unix, there's a new configure option + :option:`--enable-big-digits` that can be used to override this default. + + Apart from the performance improvements this change should be + invisible to end users, with one exception: for testing and + debugging purposes there's a new structseq :data:`sys.long_info` that + provides information about the internal format, giving the number of + bits per digit and the size in bytes of the C type used to store + each digit:: + + >>> import sys + >>> sys.long_info + sys.long_info(bits_per_digit=30, sizeof_digit=4) + + (Contributed by Mark Dickinson; :issue:`4258`.) + + Another set of changes made long objects a few bytes smaller: 2 bytes + smaller on 32-bit systems and 6 bytes on 64-bit. + (Contributed by Mark Dickinson; :issue:`5260`.) + +* The division algorithm for long integers has been made faster + by tightening the inner loop, doing shifts instead of multiplications, + and fixing an unnecessary extra iteration. + Various benchmarks show speedups of between 50% and 150% for long + integer divisions and modulo operations. + (Contributed by Mark Dickinson; :issue:`5512`.) + Bitwise operations are also significantly faster (initial patch by + Gregory Smith; :issue:`1087418`). + +* The implementation of ``%`` checks for the left-side operand being + a Python string and special-cases it; this results in a 1-3% + performance increase for applications that frequently use ``%`` + with strings, such as templating libraries. + (Implemented by Collin Winter; :issue:`5176`.) + +* List comprehensions with an ``if`` condition are compiled into + faster bytecode. (Patch by Antoine Pitrou, back-ported to 2.7 + by Jeffrey Yasskin; :issue:`4715`.) + +* Converting an integer or long integer to a decimal string was made + faster by special-casing base 10 instead of using a generalized + conversion function that supports arbitrary bases. + (Patch by Gawain Bolton; :issue:`6713`.) + +* The :meth:`split`, :meth:`replace`, :meth:`rindex`, + :meth:`rpartition`, and :meth:`rsplit` methods of string-like types + (strings, Unicode strings, and :class:`bytearray` objects) now use a + fast reverse-search algorithm instead of a character-by-character + scan. This is sometimes faster by a factor of 10. (Added by + Florent Xicluna; :issue:`7462` and :issue:`7622`.) + +* The :mod:`pickle` and :mod:`cPickle` modules now automatically + intern the strings used for attribute names, reducing memory usage + of the objects resulting from unpickling. (Contributed by Jake + McGuire; :issue:`5084`.) + +* The :mod:`cPickle` module now special-cases dictionaries, + nearly halving the time required to pickle them. + (Contributed by Collin Winter; :issue:`5670`.) + +.. ====================================================================== + +New and Improved Modules +======================== + +As in every release, Python's standard library received a number of +enhancements and bug fixes. Here's a partial list of the most notable +changes, sorted alphabetically by module name. Consult the +:file:`Misc/NEWS` file in the source tree for a more complete list of +changes, or look through the Subversion logs for all the details. + +* The :mod:`bdb` module's base debugging class :class:`~bdb.Bdb` + gained a feature for skipping modules. The constructor + now takes an iterable containing glob-style patterns such as + ``django.*``; the debugger will not step into stack frames + from a module that matches one of these patterns. + (Contributed by Maru Newby after a suggestion by + Senthil Kumaran; :issue:`5142`.) + +* The :mod:`binascii` module now supports the buffer API, so it can be + used with :class:`memoryview` instances and other similar buffer objects. + (Backported from 3.x by Florent Xicluna; :issue:`7703`.) + +* Updated module: the :mod:`bsddb` module has been updated from 4.7.2devel9 + to version 4.8.4 of + `the pybsddb package <http://www.jcea.es/programacion/pybsddb.htm>`__. + The new version features better Python 3.x compatibility, various bug fixes, + and adds several new BerkeleyDB flags and methods. + (Updated by Jesús Cea Avión; :issue:`8156`. The pybsddb + changelog can be read at http://hg.jcea.es/pybsddb/file/tip/ChangeLog.) + +* The :mod:`bz2` module's :class:`~bz2.BZ2File` now supports the context + management protocol, so you can write ``with bz2.BZ2File(...) as f:``. + (Contributed by Hagen Fürstenau; :issue:`3860`.) + +* New class: the :class:`~collections.Counter` class in the :mod:`collections` + module is useful for tallying data. :class:`~collections.Counter` instances + behave mostly like dictionaries but return zero for missing keys instead of + raising a :exc:`KeyError`: + + .. doctest:: + :options: +NORMALIZE_WHITESPACE + + >>> from collections import Counter + >>> c = Counter() + >>> for letter in 'here is a sample of english text': + ... c[letter] += 1 + ... + >>> c + Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2, + 'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1, + 'p': 1, 'r': 1, 'x': 1}) + >>> c['e'] + 5 + >>> c['z'] + 0 + + There are three additional :class:`~collections.Counter` methods. + :meth:`~collections.Counter.most_common` returns the N most common + elements and their counts. :meth:`~collections.Counter.elements` + returns an iterator over the contained elements, repeating each + element as many times as its count. + :meth:`~collections.Counter.subtract` takes an iterable and + subtracts one for each element instead of adding; if the argument is + a dictionary or another :class:`Counter`, the counts are + subtracted. :: + + >>> c.most_common(5) + [(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)] + >>> c.elements() -> + 'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ', + 'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i', + 'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's', + 's', 's', 'r', 't', 't', 'x' + >>> c['e'] + 5 + >>> c.subtract('very heavy on the letter e') + >>> c['e'] # Count is now lower + -1 + + Contributed by Raymond Hettinger; :issue:`1696199`. + + .. revision 79660 + + New class: :class:`~collections.OrderedDict` is described in the earlier + section :ref:`pep-0372`. + + New method: The :class:`~collections.deque` data type now has a + :meth:`~collections.deque.count` method that returns the number of + contained elements equal to the supplied argument *x*, and a + :meth:`~collections.deque.reverse` method that reverses the elements + of the deque in-place. :class:`~collections.deque` also exposes its maximum + length as the read-only :attr:`~collections.deque.maxlen` attribute. + (Both features added by Raymond Hettinger.) + + The :class:`~collections.namedtuple` class now has an optional *rename* parameter. + If *rename* is true, field names that are invalid because they've + been repeated or aren't legal Python identifiers will be + renamed to legal names that are derived from the field's + position within the list of fields: + + >>> from collections import namedtuple + >>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True) + >>> T._fields + ('field1', '_1', '_2', 'field2') + + (Added by Raymond Hettinger; :issue:`1818`.) + + Finally, the :class:`~collections.Mapping` abstract base class now + returns :const:`NotImplemented` if a mapping is compared to + another type that isn't a :class:`Mapping`. + (Fixed by Daniel Stutzbach; :issue:`8729`.) + +* Constructors for the parsing classes in the :mod:`ConfigParser` module now + take a *allow_no_value* parameter, defaulting to false; if true, + options without values will be allowed. For example:: + + >>> import ConfigParser, StringIO + >>> sample_config = """ + ... [mysqld] + ... user = mysql + ... pid-file = /var/run/mysqld/mysqld.pid + ... skip-bdb + ... """ + >>> config = ConfigParser.RawConfigParser(allow_no_value=True) + >>> config.readfp(StringIO.StringIO(sample_config)) + >>> config.get('mysqld', 'user') + 'mysql' + >>> print config.get('mysqld', 'skip-bdb') + None + >>> print config.get('mysqld', 'unknown') + Traceback (most recent call last): + ... + NoOptionError: No option 'unknown' in section: 'mysqld' + + (Contributed by Mats Kindahl; :issue:`7005`.) + +* Deprecated function: :func:`contextlib.nested`, which allows + handling more than one context manager with a single :keyword:`with` + statement, has been deprecated, because the :keyword:`with` statement + now supports multiple context managers. + +* The :mod:`cookielib` module now ignores cookies that have an invalid + version field, one that doesn't contain an integer value. (Fixed by + John J. Lee; :issue:`3924`.) + +* The :mod:`copy` module's :func:`~copy.deepcopy` function will now + correctly copy bound instance methods. (Implemented by + Robert Collins; :issue:`1515`.) + +* The :mod:`ctypes` module now always converts ``None`` to a C NULL + pointer for arguments declared as pointers. (Changed by Thomas + Heller; :issue:`4606`.) The underlying `libffi library + <http://sourceware.org/libffi/>`__ has been updated to version + 3.0.9, containing various fixes for different platforms. (Updated + by Matthias Klose; :issue:`8142`.) + +* New method: the :mod:`datetime` module's :class:`~datetime.timedelta` class + gained a :meth:`~datetime.timedelta.total_seconds` method that returns the + number of seconds in the duration. (Contributed by Brian Quinlan; :issue:`5788`.) + +* New method: the :class:`~decimal.Decimal` class gained a + :meth:`~decimal.Decimal.from_float` class method that performs an exact + conversion of a floating-point number to a :class:`~decimal.Decimal`. + This exact conversion strives for the + closest decimal approximation to the floating-point representation's value; + the resulting decimal value will therefore still include the inaccuracy, + if any. + For example, ``Decimal.from_float(0.1)`` returns + ``Decimal('0.1000000000000000055511151231257827021181583404541015625')``. + (Implemented by Raymond Hettinger; :issue:`4796`.) + + Comparing instances of :class:`~decimal.Decimal` with floating-point + numbers now produces sensible results based on the numeric values + of the operands. Previously such comparisons would fall back to + Python's default rules for comparing objects, which produced arbitrary + results based on their type. Note that you still cannot combine + :class:`Decimal` and floating-point in other operations such as addition, + since you should be explicitly choosing how to convert between float and + :class:`~decimal.Decimal`. (Fixed by Mark Dickinson; :issue:`2531`.) + + The constructor for :class:`~decimal.Decimal` now accepts + floating-point numbers (added by Raymond Hettinger; :issue:`8257`) + and non-European Unicode characters such as Arabic-Indic digits + (contributed by Mark Dickinson; :issue:`6595`). + + Most of the methods of the :class:`~decimal.Context` class now accept integers + as well as :class:`~decimal.Decimal` instances; the only exceptions are the + :meth:`~decimal.Context.canonical` and :meth:`~decimal.Context.is_canonical` + methods. (Patch by Juan José Conti; :issue:`7633`.) + + When using :class:`~decimal.Decimal` instances with a string's + :meth:`~str.format` method, the default alignment was previously + left-alignment. This has been changed to right-alignment, which is + more sensible for numeric types. (Changed by Mark Dickinson; :issue:`6857`.) + + Comparisons involving a signaling NaN value (or ``sNAN``) now signal + :const:`InvalidOperation` instead of silently returning a true or + false value depending on the comparison operator. Quiet NaN values + (or ``NaN``) are now hashable. (Fixed by Mark Dickinson; + :issue:`7279`.) + +* The :mod:`difflib` module now produces output that is more + compatible with modern :command:`diff`/:command:`patch` tools + through one small change, using a tab character instead of spaces as + a separator in the header giving the filename. (Fixed by Anatoly + Techtonik; :issue:`7585`.) + +* The Distutils ``sdist`` command now always regenerates the + :file:`MANIFEST` file, since even if the :file:`MANIFEST.in` or + :file:`setup.py` files haven't been modified, the user might have + created some new files that should be included. + (Fixed by Tarek Ziadé; :issue:`8688`.) + +* The :mod:`doctest` module's :const:`IGNORE_EXCEPTION_DETAIL` flag + will now ignore the name of the module containing the exception + being tested. (Patch by Lennart Regebro; :issue:`7490`.) + +* The :mod:`email` module's :class:`~email.message.Message` class will + now accept a Unicode-valued payload, automatically converting the + payload to the encoding specified by :attr:`output_charset`. + (Added by R. David Murray; :issue:`1368247`.) + +* The :class:`~fractions.Fraction` class now accepts a single float or + :class:`~decimal.Decimal` instance, or two rational numbers, as + arguments to its constructor. (Implemented by Mark Dickinson; + rationals added in :issue:`5812`, and float/decimal in + :issue:`8294`.) + + Ordering comparisons (``<``, ``<=``, ``>``, ``>=``) between + fractions and complex numbers now raise a :exc:`TypeError`. + This fixes an oversight, making the :class:`~fractions.Fraction` + match the other numeric types. + + .. revision 79455 + +* New class: :class:`~ftplib.FTP_TLS` in + the :mod:`ftplib` module provides secure FTP + connections using TLS encapsulation of authentication as well as + subsequent control and data transfers. + (Contributed by Giampaolo Rodola; :issue:`2054`.) + + The :meth:`~ftplib.FTP.storbinary` method for binary uploads can now restart + uploads thanks to an added *rest* parameter (patch by Pablo Mouzo; + :issue:`6845`.) + +* New class decorator: :func:`~functools.total_ordering` in the :mod:`functools` + module takes a class that defines an :meth:`__eq__` method and one of + :meth:`__lt__`, :meth:`__le__`, :meth:`__gt__`, or :meth:`__ge__`, + and generates the missing comparison methods. Since the + :meth:`__cmp__` method is being deprecated in Python 3.x, + this decorator makes it easier to define ordered classes. + (Added by Raymond Hettinger; :issue:`5479`.) + + New function: :func:`~functools.cmp_to_key` will take an old-style comparison + function that expects two arguments and return a new callable that + can be used as the *key* parameter to functions such as + :func:`sorted`, :func:`min` and :func:`max`, etc. The primary + intended use is to help with making code compatible with Python 3.x. + (Added by Raymond Hettinger.) + +* New function: the :mod:`gc` module's :func:`~gc.is_tracked` returns + true if a given instance is tracked by the garbage collector, false + otherwise. (Contributed by Antoine Pitrou; :issue:`4688`.) + +* The :mod:`gzip` module's :class:`~gzip.GzipFile` now supports the context + management protocol, so you can write ``with gzip.GzipFile(...) as f:`` + (contributed by Hagen Fürstenau; :issue:`3860`), and it now implements + the :class:`io.BufferedIOBase` ABC, so you can wrap it with + :class:`io.BufferedReader` for faster processing + (contributed by Nir Aides; :issue:`7471`). + It's also now possible to override the modification time + recorded in a gzipped file by providing an optional timestamp to + the constructor. (Contributed by Jacques Frechet; :issue:`4272`.) + + Files in gzip format can be padded with trailing zero bytes; the + :mod:`gzip` module will now consume these trailing bytes. (Fixed by + Tadek Pietraszek and Brian Curtin; :issue:`2846`.) + +* New attribute: the :mod:`hashlib` module now has an :attr:`~hashlib.hashlib.algorithms` + attribute containing a tuple naming the supported algorithms. + In Python 2.7, ``hashlib.algorithms`` contains + ``('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')``. + (Contributed by Carl Chenet; :issue:`7418`.) + +* The default :class:`~httplib.HTTPResponse` class used by the :mod:`httplib` module now + supports buffering, resulting in much faster reading of HTTP responses. + (Contributed by Kristján Valur Jónsson; :issue:`4879`.) + + The :class:`~httplib.HTTPConnection` and :class:`~httplib.HTTPSConnection` classes + now support a *source_address* parameter, a ``(host, port)`` 2-tuple + giving the source address that will be used for the connection. + (Contributed by Eldon Ziegler; :issue:`3972`.) + +* The :mod:`ihooks` module now supports relative imports. Note that + :mod:`ihooks` is an older module for customizing imports, + superseded by the :mod:`imputil` module added in Python 2.0. + (Relative import support added by Neil Schemenauer.) + + .. revision 75423 + +* The :mod:`imaplib` module now supports IPv6 addresses. + (Contributed by Derek Morr; :issue:`1655`.) + +* New function: the :mod:`inspect` module's :func:`~inspect.getcallargs` + takes a callable and its positional and keyword arguments, + and figures out which of the callable's parameters will receive each argument, + returning a dictionary mapping argument names to their values. For example:: + + >>> from inspect import getcallargs + >>> def f(a, b=1, *pos, **named): + ... pass + >>> getcallargs(f, 1, 2, 3) + {'a': 1, 'b': 2, 'pos': (3,), 'named': {}} + >>> getcallargs(f, a=2, x=4) + {'a': 2, 'b': 1, 'pos': (), 'named': {'x': 4}} + >>> getcallargs(f) + Traceback (most recent call last): + ... + TypeError: f() takes at least 1 argument (0 given) + + Contributed by George Sakkis; :issue:`3135`. + +* Updated module: The :mod:`io` library has been upgraded to the version shipped with + Python 3.1. For 3.1, the I/O library was entirely rewritten in C + and is 2 to 20 times faster depending on the task being performed. The + original Python version was renamed to the :mod:`_pyio` module. + + One minor resulting change: the :class:`io.TextIOBase` class now + has an :attr:`errors` attribute giving the error setting + used for encoding and decoding errors (one of ``'strict'``, ``'replace'``, + ``'ignore'``). + + The :class:`io.FileIO` class now raises an :exc:`OSError` when passed + an invalid file descriptor. (Implemented by Benjamin Peterson; + :issue:`4991`.) The :meth:`~io.IOBase.truncate` method now preserves the + file position; previously it would change the file position to the + end of the new file. (Fixed by Pascal Chambon; :issue:`6939`.) + +* New function: ``itertools.compress(data, selectors)`` takes two + iterators. Elements of *data* are returned if the corresponding + value in *selectors* is true:: + + itertools.compress('ABCDEF', [1,0,1,0,1,1]) => + A, C, E, F + + .. maybe here is better to use >>> list(itertools.compress(...)) instead + + New function: ``itertools.combinations_with_replacement(iter, r)`` + returns all the possible *r*-length combinations of elements from the + iterable *iter*. Unlike :func:`~itertools.combinations`, individual elements + can be repeated in the generated combinations:: + + itertools.combinations_with_replacement('abc', 2) => + ('a', 'a'), ('a', 'b'), ('a', 'c'), + ('b', 'b'), ('b', 'c'), ('c', 'c') + + Note that elements are treated as unique depending on their position + in the input, not their actual values. + + The :func:`itertools.count` function now has a *step* argument that + allows incrementing by values other than 1. :func:`~itertools.count` also + now allows keyword arguments, and using non-integer values such as + floats or :class:`~decimal.Decimal` instances. (Implemented by Raymond + Hettinger; :issue:`5032`.) + + :func:`itertools.combinations` and :func:`itertools.product` + previously raised :exc:`ValueError` for values of *r* larger than + the input iterable. This was deemed a specification error, so they + now return an empty iterator. (Fixed by Raymond Hettinger; :issue:`4816`.) + +* Updated module: The :mod:`json` module was upgraded to version 2.0.9 of the + simplejson package, which includes a C extension that makes + encoding and decoding faster. + (Contributed by Bob Ippolito; :issue:`4136`.) + + To support the new :class:`collections.OrderedDict` type, :func:`json.load` + now has an optional *object_pairs_hook* parameter that will be called + with any object literal that decodes to a list of pairs. + (Contributed by Raymond Hettinger; :issue:`5381`.) + +* The :mod:`mailbox` module's :class:`~mailbox.Maildir` class now records the + timestamp on the directories it reads, and only re-reads them if the + modification time has subsequently changed. This improves + performance by avoiding unneeded directory scans. (Fixed by + A.M. Kuchling and Antoine Pitrou; :issue:`1607951`, :issue:`6896`.) + +* New functions: the :mod:`math` module gained + :func:`~math.erf` and :func:`~math.erfc` for the error function and the complementary error function, + :func:`~math.expm1` which computes ``e**x - 1`` with more precision than + using :func:`~math.exp` and subtracting 1, + :func:`~math.gamma` for the Gamma function, and + :func:`~math.lgamma` for the natural log of the Gamma function. + (Contributed by Mark Dickinson and nirinA raseliarison; :issue:`3366`.) + +* The :mod:`multiprocessing` module's :class:`Manager*` classes + can now be passed a callable that will be called whenever + a subprocess is started, along with a set of arguments that will be + passed to the callable. + (Contributed by lekma; :issue:`5585`.) + + The :class:`~multiprocessing.Pool` class, which controls a pool of worker processes, + now has an optional *maxtasksperchild* parameter. Worker processes + will perform the specified number of tasks and then exit, causing the + :class:`~multiprocessing.Pool` to start a new worker. This is useful if tasks may leak + memory or other resources, or if some tasks will cause the worker to + become very large. + (Contributed by Charles Cazabon; :issue:`6963`.) + +* The :mod:`nntplib` module now supports IPv6 addresses. + (Contributed by Derek Morr; :issue:`1664`.) + +* New functions: the :mod:`os` module wraps the following POSIX system + calls: :func:`~os.getresgid` and :func:`~os.getresuid`, which return the + real, effective, and saved GIDs and UIDs; + :func:`~os.setresgid` and :func:`~os.setresuid`, which set + real, effective, and saved GIDs and UIDs to new values; + :func:`~os.initgroups`, which initialize the group access list + for the current process. (GID/UID functions + contributed by Travis H.; :issue:`6508`. Support for initgroups added + by Jean-Paul Calderone; :issue:`7333`.) + + The :func:`os.fork` function now re-initializes the import lock in + the child process; this fixes problems on Solaris when :func:`~os.fork` + is called from a thread. (Fixed by Zsolt Cserna; :issue:`7242`.) + +* In the :mod:`os.path` module, the :func:`~os.path.normpath` and + :func:`~os.path.abspath` functions now preserve Unicode; if their input path + is a Unicode string, the return value is also a Unicode string. + (:meth:`~os.path.normpath` fixed by Matt Giuca in :issue:`5827`; + :meth:`~os.path.abspath` fixed by Ezio Melotti in :issue:`3426`.) + +* The :mod:`pydoc` module now has help for the various symbols that Python + uses. You can now do ``help('<<')`` or ``help('@')``, for example. + (Contributed by David Laban; :issue:`4739`.) + +* The :mod:`re` module's :func:`~re.split`, :func:`~re.sub`, and :func:`~re.subn` + now accept an optional *flags* argument, for consistency with the + other functions in the module. (Added by Gregory P. Smith.) + +* New function: :func:`~runpy.run_path` in the :mod:`runpy` module + will execute the code at a provided *path* argument. *path* can be + the path of a Python source file (:file:`example.py`), a compiled + bytecode file (:file:`example.pyc`), a directory + (:file:`./package/`), or a zip archive (:file:`example.zip`). If a + directory or zip path is provided, it will be added to the front of + ``sys.path`` and the module :mod:`__main__` will be imported. It's + expected that the directory or zip contains a :file:`__main__.py`; + if it doesn't, some other :file:`__main__.py` might be imported from + a location later in ``sys.path``. This makes more of the machinery + of :mod:`runpy` available to scripts that want to mimic the way + Python's command line processes an explicit path name. + (Added by Nick Coghlan; :issue:`6816`.) + +* New function: in the :mod:`shutil` module, :func:`~shutil.make_archive` + takes a filename, archive type (zip or tar-format), and a directory + path, and creates an archive containing the directory's contents. + (Added by Tarek Ziadé.) + + :mod:`shutil`'s :func:`~shutil.copyfile` and :func:`~shutil.copytree` + functions now raise a :exc:`~shutil.SpecialFileError` exception when + asked to copy a named pipe. Previously the code would treat + named pipes like a regular file by opening them for reading, and + this would block indefinitely. (Fixed by Antoine Pitrou; :issue:`3002`.) + +* The :mod:`signal` module no longer re-installs the signal handler + unless this is truly necessary, which fixes a bug that could make it + impossible to catch the EINTR signal robustly. (Fixed by + Charles-Francois Natali; :issue:`8354`.) + +* New functions: in the :mod:`site` module, three new functions + return various site- and user-specific paths. + :func:`~site.getsitepackages` returns a list containing all + global site-packages directories, + :func:`~site.getusersitepackages` returns the path of the user's + site-packages directory, and + :func:`~site.getuserbase` returns the value of the :envvar:`USER_BASE` + environment variable, giving the path to a directory that can be used + to store data. + (Contributed by Tarek Ziadé; :issue:`6693`.) + + The :mod:`site` module now reports exceptions occurring + when the :mod:`sitecustomize` module is imported, and will no longer + catch and swallow the :exc:`KeyboardInterrupt` exception. (Fixed by + Victor Stinner; :issue:`3137`.) + +* The :func:`~socket.create_connection` function + gained a *source_address* parameter, a ``(host, port)`` 2-tuple + giving the source address that will be used for the connection. + (Contributed by Eldon Ziegler; :issue:`3972`.) + + The :meth:`~socket.socket.recv_into` and :meth:`~socket.socket.recvfrom_into` + methods will now write into objects that support the buffer API, most usefully + the :class:`bytearray` and :class:`memoryview` objects. (Implemented by + Antoine Pitrou; :issue:`8104`.) + +* The :mod:`SocketServer` module's :class:`~SocketServer.TCPServer` class now + supports socket timeouts and disabling the Nagle algorithm. + The :attr:`~SocketServer.TCPServer.disable_nagle_algorithm` class attribute + defaults to False; if overridden to be True, + new request connections will have the TCP_NODELAY option set to + prevent buffering many small sends into a single TCP packet. + The :attr:`~SocketServer.BaseServer.timeout` class attribute can hold + a timeout in seconds that will be applied to the request socket; if + no request is received within that time, :meth:`~SocketServer.BaseServer.handle_timeout` + will be called and :meth:`~SocketServer.BaseServer.handle_request` will return. + (Contributed by Kristján Valur Jónsson; :issue:`6192` and :issue:`6267`.) + +* Updated module: the :mod:`sqlite3` module has been updated to + version 2.6.0 of the `pysqlite package <http://code.google.com/p/pysqlite/>`__. Version 2.6.0 includes a number of bugfixes, and adds + the ability to load SQLite extensions from shared libraries. + Call the ``enable_load_extension(True)`` method to enable extensions, + and then call :meth:`~sqlite3.Connection.load_extension` to load a particular shared library. + (Updated by Gerhard Häring.) + +* The :mod:`ssl` module's :class:`~ssl.SSLSocket` objects now support the + buffer API, which fixed a test suite failure (fix by Antoine Pitrou; + :issue:`7133`) and automatically set + OpenSSL's :cmacro:`SSL_MODE_AUTO_RETRY`, which will prevent an error + code being returned from :meth:`recv` operations that trigger an SSL + renegotiation (fix by Antoine Pitrou; :issue:`8222`). + + The :func:`ssl.wrap_socket` constructor function now takes a + *ciphers* argument that's a string listing the encryption algorithms + to be allowed; the format of the string is described + `in the OpenSSL documentation + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. + (Added by Antoine Pitrou; :issue:`8322`.) + + Another change makes the extension load all of OpenSSL's ciphers and + digest algorithms so that they're all available. Some SSL + certificates couldn't be verified, reporting an "unknown algorithm" + error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; + :issue:`8484`.) + + The version of OpenSSL being used is now available as the module + attributes :data:`ssl.OPENSSL_VERSION` (a string), + :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and + :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by Antoine + Pitrou; :issue:`8321`.) + +* The :mod:`struct` module will no longer silently ignore overflow + errors when a value is too large for a particular integer format + code (one of ``bBhHiIlLqQ``); it now always raises a + :exc:`struct.error` exception. (Changed by Mark Dickinson; + :issue:`1523`.) The :func:`~struct.pack` function will also + attempt to use :meth:`__index__` to convert and pack non-integers + before trying the :meth:`__int__` method or reporting an error. + (Changed by Mark Dickinson; :issue:`8300`.) + +* New function: the :mod:`subprocess` module's + :func:`~subprocess.check_output` runs a command with a specified set of arguments + and returns the command's output as a string when the command runs without + error, or raises a :exc:`~subprocess.CalledProcessError` exception otherwise. + + :: + + >>> subprocess.check_output(['df', '-h', '.']) + 'Filesystem Size Used Avail Capacity Mounted on\n + /dev/disk0s2 52G 49G 3.0G 94% /\n' + + >>> subprocess.check_output(['df', '-h', '/bogus']) + ... + subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1 + + (Contributed by Gregory P. Smith.) + + The :mod:`subprocess` module will now retry its internal system calls + on receiving an :const:`EINTR` signal. (Reported by several people; final + patch by Gregory P. Smith in :issue:`1068268`.) + +* New function: :func:`~symtable.Symbol.is_declared_global` in the :mod:`symtable` module + returns true for variables that are explicitly declared to be global, + false for ones that are implicitly global. + (Contributed by Jeremy Hylton.) + +* The :mod:`syslog` module will now use the value of ``sys.argv[0]`` as the + identifier instead of the previous default value of ``'python'``. + (Changed by Sean Reifschneider; :issue:`8451`.) + +* The ``sys.version_info`` value is now a named tuple, with attributes + named :attr:`major`, :attr:`minor`, :attr:`micro`, + :attr:`releaselevel`, and :attr:`serial`. (Contributed by Ross + Light; :issue:`4285`.) + + :func:`sys.getwindowsversion` also returns a named tuple, + with attributes named :attr:`major`, :attr:`minor`, :attr:`build`, + :attr:`platform`, :attr:`service_pack`, :attr:`service_pack_major`, + :attr:`service_pack_minor`, :attr:`suite_mask`, and + :attr:`product_type`. (Contributed by Brian Curtin; :issue:`7766`.) + +* The :mod:`tarfile` module's default error handling has changed, to + no longer suppress fatal errors. The default error level was previously 0, + which meant that errors would only result in a message being written to the + debug log, but because the debug log is not activated by default, + these errors go unnoticed. The default error level is now 1, + which raises an exception if there's an error. + (Changed by Lars Gustäbel; :issue:`7357`.) + + :mod:`tarfile` now supports filtering the :class:`~tarfile.TarInfo` + objects being added to a tar file. When you call :meth:`~tarfile.TarFile.add`, + you may supply an optional *filter* argument + that's a callable. The *filter* callable will be passed the + :class:`~tarfile.TarInfo` for every file being added, and can modify and return it. + If the callable returns ``None``, the file will be excluded from the + resulting archive. This is more powerful than the existing + *exclude* argument, which has therefore been deprecated. + (Added by Lars Gustäbel; :issue:`6856`.) + The :class:`~tarfile.TarFile` class also now supports the context manager protocol. + (Added by Lars Gustäbel; :issue:`7232`.) + +* The :meth:`~threading.Event.wait` method of the :class:`threading.Event` class + now returns the internal flag on exit. This means the method will usually + return true because :meth:`~threading.Event.wait` is supposed to block until the + internal flag becomes true. The return value will only be false if + a timeout was provided and the operation timed out. + (Contributed by Tim Lesher; :issue:`1674032`.) + +* The Unicode database provided by the :mod:`unicodedata` module is + now used internally to determine which characters are numeric, + whitespace, or represent line breaks. The database also + includes information from the :file:`Unihan.txt` data file (patch + by Anders Chrigström and Amaury Forgeot d'Arc; :issue:`1571184`) + and has been updated to version 5.2.0 (updated by + Florent Xicluna; :issue:`8024`). + +* The :mod:`urlparse` module's :func:`~urlparse.urlsplit` now handles + unknown URL schemes in a fashion compliant with :rfc:`3986`: if the + URL is of the form ``"<something>://..."``, the text before the + ``://`` is treated as the scheme, even if it's a made-up scheme that + the module doesn't know about. This change may break code that + worked around the old behaviour. For example, Python 2.6.4 or 2.5 + will return the following: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', '', '//host/filename?query', '', '') + + Python 2.7 (and Python 2.6.5) will return: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', 'host', '/filename?query', '', '') + + (Python 2.7 actually produces slightly different output, since it + returns a named tuple instead of a standard tuple.) + + The :mod:`urlparse` module also supports IPv6 literal addresses as defined by + :rfc:`2732` (contributed by Senthil Kumaran; :issue:`2987`). :: + + >>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo') + ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]', + path='/foo', params='', query='', fragment='') + +* New class: the :class:`~weakref.WeakSet` class in the :mod:`weakref` + module is a set that only holds weak references to its elements; elements + will be removed once there are no references pointing to them. + (Originally implemented in Python 3.x by Raymond Hettinger, and backported + to 2.7 by Michael Foord.) + +* The ElementTree library, :mod:`xml.etree`, no longer escapes + ampersands and angle brackets when outputting an XML processing + instruction (which looks like ``<?xml-stylesheet href="#style1"?>``) + or comment (which looks like ``<!-- comment -->``). + (Patch by Neil Muller; :issue:`2746`.) + +* The XML-RPC client and server, provided by the :mod:`xmlrpclib` and + :mod:`SimpleXMLRPCServer` modules, have improved performance by + supporting HTTP/1.1 keep-alive and by optionally using gzip encoding + to compress the XML being exchanged. The gzip compression is + controlled by the :attr:`encode_threshold` attribute of + :class:`SimpleXMLRPCRequestHandler`, which contains a size in bytes; + responses larger than this will be compressed. + (Contributed by Kristján Valur Jónsson; :issue:`6267`.) + +* The :mod:`zipfile` module's :class:`~zipfile.ZipFile` now supports the context + management protocol, so you can write ``with zipfile.ZipFile(...) as f:``. + (Contributed by Brian Curtin; :issue:`5511`.) + + :mod:`zipfile` now also supports archiving empty directories and + extracts them correctly. (Fixed by Kuba Wieczorek; :issue:`4710`.) + Reading files out of an archive is faster, and interleaving + :meth:`~zipfile.ZipFile.read` and :meth:`~zipfile.ZipFile.readline` now works correctly. + (Contributed by Nir Aides; :issue:`7610`.) + + The :func:`~zipfile.is_zipfile` function now + accepts a file object, in addition to the path names accepted in earlier + versions. (Contributed by Gabriel Genellina; :issue:`4756`.) + + The :meth:`~zipfile.ZipFile.writestr` method now has an optional *compress_type* parameter + that lets you override the default compression method specified in the + :class:`~zipfile.ZipFile` constructor. (Contributed by Ronald Oussoren; + :issue:`6003`.) + + +.. ====================================================================== +.. whole new modules get described in subsections here + + +.. _importlib-section: + +New module: importlib +------------------------------ + +Python 3.1 includes the :mod:`importlib` package, a re-implementation +of the logic underlying Python's :keyword:`import` statement. +:mod:`importlib` is useful for implementors of Python interpreters and +to users who wish to write new importers that can participate in the +import process. Python 2.7 doesn't contain the complete +:mod:`importlib` package, but instead has a tiny subset that contains +a single function, :func:`~importlib.import_module`. + +``import_module(name, package=None)`` imports a module. *name* is +a string containing the module or package's name. It's possible to do +relative imports by providing a string that begins with a ``.`` +character, such as ``..utils.errors``. For relative imports, the +*package* argument must be provided and is the name of the package that +will be used as the anchor for +the relative import. :func:`~importlib.import_module` both inserts the imported +module into ``sys.modules`` and returns the module object. + +Here are some examples:: + + >>> from importlib import import_module + >>> anydbm = import_module('anydbm') # Standard absolute import + >>> anydbm + <module 'anydbm' from '/p/python/Lib/anydbm.py'> + >>> # Relative import + >>> file_util = import_module('..file_util', 'distutils.command') + >>> file_util + <module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'> + +:mod:`importlib` was implemented by Brett Cannon and introduced in +Python 3.1. + + +New module: sysconfig +--------------------------------- + +The :mod:`sysconfig` module has been pulled out of the Distutils +package, becoming a new top-level module in its own right. +:mod:`sysconfig` provides functions for getting information about +Python's build process: compiler switches, installation paths, the +platform name, and whether Python is running from its source +directory. + +Some of the functions in the module are: + +* :func:`~sysconfig.get_config_var` returns variables from Python's + Makefile and the :file:`pyconfig.h` file. +* :func:`~sysconfig.get_config_vars` returns a dictionary containing + all of the configuration variables. +* :func:`~sysconfig.get_path` returns the configured path for + a particular type of module: the standard library, + site-specific modules, platform-specific modules, etc. +* :func:`~sysconfig.is_python_build` returns true if you're running a + binary from a Python source tree, and false otherwise. + +Consult the :mod:`sysconfig` documentation for more details and for +a complete list of functions. + +The Distutils package and :mod:`sysconfig` are now maintained by Tarek +Ziadé, who has also started a Distutils2 package (source repository at +http://hg.python.org/distutils2/) for developing a next-generation +version of Distutils. + + +ttk: Themed Widgets for Tk +-------------------------- + +Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk +widgets but have a more customizable appearance and can therefore more +closely resemble the native platform's widgets. This widget +set was originally called Tile, but was renamed to Ttk (for "themed Tk") +on being added to Tcl/Tck release 8.5. + +To learn more, read the :mod:`ttk` module documentation. You may also +wish to read the Tcl/Tk manual page describing the +Ttk theme engine, available at +http://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some +screenshots of the Python/Ttk code in use are at +http://code.google.com/p/python-ttk/wiki/Screenshots. + +The :mod:`ttk` module was written by Guilherme Polo and added in +:issue:`2983`. An alternate version called ``Tile.py``, written by +Martin Franklin and maintained by Kevin Walzer, was proposed for +inclusion in :issue:`2618`, but the authors argued that Guilherme +Polo's work was more comprehensive. + + +.. _unittest-section: + +Updated module: unittest +--------------------------------- + +The :mod:`unittest` module was greatly enhanced; many +new features were added. Most of these features were implemented +by Michael Foord, unless otherwise noted. The enhanced version of +the module is downloadable separately for use with Python versions 2.4 to 2.6, +packaged as the :mod:`unittest2` package, from +http://pypi.python.org/pypi/unittest2. + +When used from the command line, the module can automatically discover +tests. It's not as fancy as `py.test <http://pytest.org>`__ or +`nose <http://code.google.com/p/python-nose/>`__, but provides a simple way +to run tests kept within a set of package directories. For example, +the following command will search the :file:`test/` subdirectory for +any importable test files named ``test*.py``:: + + python -m unittest discover -s test + +Consult the :mod:`unittest` module documentation for more details. +(Developed in :issue:`6001`.) + +The :func:`~unittest.main` function supports some other new options: + +* :option:`-b` or :option:`--buffer` will buffer the standard output + and standard error streams during each test. If the test passes, + any resulting output will be discarded; on failure, the buffered + output will be displayed. + +* :option:`-c` or :option:`--catch` will cause the control-C interrupt + to be handled more gracefully. Instead of interrupting the test + process immediately, the currently running test will be completed + and then the partial results up to the interruption will be reported. + If you're impatient, a second press of control-C will cause an immediate + interruption. + + This control-C handler tries to avoid causing problems when the code + being tested or the tests being run have defined a signal handler of + their own, by noticing that a signal handler was already set and + calling it. If this doesn't work for you, there's a + :func:`~unittest.removeHandler` decorator that can be used to mark tests that + should have the control-C handling disabled. + +* :option:`-f` or :option:`--failfast` makes + test execution stop immediately when a test fails instead of + continuing to execute further tests. (Suggested by Cliff Dyer and + implemented by Michael Foord; :issue:`8074`.) + +The progress messages now show 'x' for expected failures +and 'u' for unexpected successes when run in verbose mode. +(Contributed by Benjamin Peterson.) + +Test cases can raise the :exc:`~unittest.SkipTest` exception to skip a +test (:issue:`1034053`). + +The error messages for :meth:`~unittest.TestCase.assertEqual`, +:meth:`~unittest.TestCase.assertTrue`, and :meth:`~unittest.TestCase.assertFalse` +failures now provide more information. If you set the +:attr:`~unittest.TestCase.longMessage` attribute of your :class:`~unittest.TestCase` classes to +True, both the standard error message and any additional message you +provide will be printed for failures. (Added by Michael Foord; :issue:`5663`.) + +The :meth:`~unittest.TestCase.assertRaises` method now +returns a context handler when called without providing a callable +object to run. For example, you can write this:: + + with self.assertRaises(KeyError): + {}['foo'] + +(Implemented by Antoine Pitrou; :issue:`4444`.) + +.. rev 78774 + +Module- and class-level setup and teardown fixtures are now supported. +Modules can contain :func:`~unittest.setUpModule` and :func:`~unittest.tearDownModule` +functions. Classes can have :meth:`~unittest.TestCase.setUpClass` and +:meth:`~unittest.TestCase.tearDownClass` methods that must be defined as class methods +(using ``@classmethod`` or equivalent). These functions and +methods are invoked when the test runner switches to a test case in a +different module or class. + +The methods :meth:`~unittest.TestCase.addCleanup` and +:meth:`~unittest.TestCase.doCleanups` were added. +:meth:`~unittest.TestCase.addCleanup` lets you add cleanup functions that +will be called unconditionally (after :meth:`~unittest.TestCase.setUp` if +:meth:`~unittest.TestCase.setUp` fails, otherwise after :meth:`~unittest.TestCase.tearDown`). This allows +for much simpler resource allocation and deallocation during tests +(:issue:`5679`). + +A number of new methods were added that provide more specialized +tests. Many of these methods were written by Google engineers +for use in their test suites; Gregory P. Smith, Michael Foord, and +GvR worked on merging them into Python's version of :mod:`unittest`. + +* :meth:`~unittest.TestCase.assertIsNone` and :meth:`~unittest.TestCase.assertIsNotNone` take one + expression and verify that the result is or is not ``None``. + +* :meth:`~unittest.TestCase.assertIs` and :meth:`~unittest.TestCase.assertIsNot` + take two values and check whether the two values evaluate to the same object or not. + (Added by Michael Foord; :issue:`2578`.) + +* :meth:`~unittest.TestCase.assertIsInstance` and + :meth:`~unittest.TestCase.assertNotIsInstance` check whether + the resulting object is an instance of a particular class, or of + one of a tuple of classes. (Added by Georg Brandl; :issue:`7031`.) + +* :meth:`~unittest.TestCase.assertGreater`, :meth:`~unittest.TestCase.assertGreaterEqual`, + :meth:`~unittest.TestCase.assertLess`, and :meth:`~unittest.TestCase.assertLessEqual` compare + two quantities. + +* :meth:`~unittest.TestCase.assertMultiLineEqual` compares two strings, and if they're + not equal, displays a helpful comparison that highlights the + differences in the two strings. This comparison is now used by + default when Unicode strings are compared with :meth:`~unittest.TestCase.assertEqual`. + +* :meth:`~unittest.TestCase.assertRegexpMatches` and + :meth:`~unittest.TestCase.assertNotRegexpMatches` checks whether the + first argument is a string matching or not matching the regular + expression provided as the second argument (:issue:`8038`). + +* :meth:`~unittest.TestCase.assertRaisesRegexp` checks whether a particular exception + is raised, and then also checks that the string representation of + the exception matches the provided regular expression. + +* :meth:`~unittest.TestCase.assertIn` and :meth:`~unittest.TestCase.assertNotIn` + tests whether *first* is or is not in *second*. + +* :meth:`~unittest.TestCase.assertItemsEqual` tests whether two provided sequences + contain the same elements. + +* :meth:`~unittest.TestCase.assertSetEqual` compares whether two sets are equal, and + only reports the differences between the sets in case of error. + +* Similarly, :meth:`~unittest.TestCase.assertListEqual` and :meth:`~unittest.TestCase.assertTupleEqual` + compare the specified types and explain any differences without necessarily + printing their full values; these methods are now used by default + when comparing lists and tuples using :meth:`~unittest.TestCase.assertEqual`. + More generally, :meth:`~unittest.TestCase.assertSequenceEqual` compares two sequences + and can optionally check whether both sequences are of a + particular type. + +* :meth:`~unittest.TestCase.assertDictEqual` compares two dictionaries and reports the + differences; it's now used by default when you compare two dictionaries + using :meth:`~unittest.TestCase.assertEqual`. :meth:`~unittest.TestCase.assertDictContainsSubset` checks whether + all of the key/value pairs in *first* are found in *second*. + +* :meth:`~unittest.TestCase.assertAlmostEqual` and :meth:`~unittest.TestCase.assertNotAlmostEqual` test + whether *first* and *second* are approximately equal. This method + can either round their difference to an optionally-specified number + of *places* (the default is 7) and compare it to zero, or require + the difference to be smaller than a supplied *delta* value. + +* :meth:`~unittest.TestLoader.loadTestsFromName` properly honors the + :attr:`~unittest.TestLoader.suiteClass` attribute of + the :class:`~unittest.TestLoader`. (Fixed by Mark Roddy; :issue:`6866`.) + +* A new hook lets you extend the :meth:`~unittest.TestCase.assertEqual` method to handle + new data types. The :meth:`~unittest.TestCase.addTypeEqualityFunc` method takes a type + object and a function. The function will be used when both of the + objects being compared are of the specified type. This function + should compare the two objects and raise an exception if they don't + match; it's a good idea for the function to provide additional + information about why the two objects aren't matching, much as the new + sequence comparison methods do. + +:func:`unittest.main` now takes an optional ``exit`` argument. If +False, :func:`~unittest.main` doesn't call :func:`sys.exit`, allowing +:func:`~unittest.main` to be used from the interactive interpreter. +(Contributed by J. Pablo Fernández; :issue:`3379`.) + +:class:`~unittest.TestResult` has new :meth:`~unittest.TestResult.startTestRun` and +:meth:`~unittest.TestResult.stopTestRun` methods that are called immediately before +and after a test run. (Contributed by Robert Collins; :issue:`5728`.) + +With all these changes, the :file:`unittest.py` was becoming awkwardly +large, so the module was turned into a package and the code split into +several files (by Benjamin Peterson). This doesn't affect how the +module is imported or used. + +.. seealso:: + + http://www.voidspace.org.uk/python/articles/unittest2.shtml + Describes the new features, how to use them, and the + rationale for various design decisions. (By Michael Foord.) + +.. _elementtree-section: + +Updated module: ElementTree 1.3 +--------------------------------- + +The version of the ElementTree library included with Python was updated to +version 1.3. Some of the new features are: + +* The various parsing functions now take a *parser* keyword argument + giving an :class:`XMLParser` instance that will + be used. This makes it possible to override the file's internal encoding:: + + p = ET.XMLParser(encoding='utf-8') + t = ET.XML("""<root/>""", parser=p) + + Errors in parsing XML now raise a :exc:`ParseError` exception, whose + instances have a :attr:`position` attribute + containing a (*line*, *column*) tuple giving the location of the problem. + +* ElementTree's code for converting trees to a string has been + significantly reworked, making it roughly twice as fast in many + cases. The :class:`ElementTree` :meth:`write` and :class:`Element` + :meth:`write` methods now have a *method* parameter that can be + "xml" (the default), "html", or "text". HTML mode will output empty + elements as ``<empty></empty>`` instead of ``<empty/>``, and text + mode will skip over elements and only output the text chunks. If + you set the :attr:`tag` attribute of an element to ``None`` but + leave its children in place, the element will be omitted when the + tree is written out, so you don't need to do more extensive rearrangement + to remove a single element. + + Namespace handling has also been improved. All ``xmlns:<whatever>`` + declarations are now output on the root element, not scattered throughout + the resulting XML. You can set the default namespace for a tree + by setting the :attr:`default_namespace` attribute and can + register new prefixes with :meth:`register_namespace`. In XML mode, + you can use the true/false *xml_declaration* parameter to suppress the + XML declaration. + +* New :class:`Element` method: :meth:`extend` appends the items from a + sequence to the element's children. Elements themselves behave like + sequences, so it's easy to move children from one element to + another:: + + from xml.etree import ElementTree as ET + + t = ET.XML("""<list> + <item>1</item> <item>2</item> <item>3</item> + </list>""") + new = ET.XML('<root/>') + new.extend(t) + + # Outputs <root><item>1</item>...</root> + print ET.tostring(new) + +* New :class:`Element` method: :meth:`iter` yields the children of the + element as a generator. It's also possible to write ``for child in + elem:`` to loop over an element's children. The existing method + :meth:`getiterator` is now deprecated, as is :meth:`getchildren` + which constructs and returns a list of children. + +* New :class:`Element` method: :meth:`itertext` yields all chunks of + text that are descendants of the element. For example:: + + t = ET.XML("""<list> + <item>1</item> <item>2</item> <item>3</item> + </list>""") + + # Outputs ['\n ', '1', ' ', '2', ' ', '3', '\n'] + print list(t.itertext()) + +* Deprecated: using an element as a Boolean (i.e., ``if elem:``) would + return true if the element had any children, or false if there were + no children. This behaviour is confusing -- ``None`` is false, but + so is a childless element? -- so it will now trigger a + :exc:`FutureWarning`. In your code, you should be explicit: write + ``len(elem) != 0`` if you're interested in the number of children, + or ``elem is not None``. + +Fredrik Lundh develops ElementTree and produced the 1.3 version; +you can read his article describing 1.3 at +http://effbot.org/zone/elementtree-13-intro.htm. +Florent Xicluna updated the version included with +Python, after discussions on python-dev and in :issue:`6472`.) + +.. ====================================================================== + + +Build and C API Changes +======================= + +Changes to Python's build process and to the C API include: + +* The latest release of the GNU Debugger, GDB 7, can be `scripted + using Python + <http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html>`__. + When you begin debugging an executable program P, GDB will look for + a file named ``P-gdb.py`` and automatically read it. Dave Malcolm + contributed a :file:`python-gdb.py` that adds a number of + commands useful when debugging Python itself. For example, + ``py-up`` and ``py-down`` go up or down one Python stack frame, + which usually corresponds to several C stack frames. ``py-print`` + prints the value of a Python variable, and ``py-bt`` prints the + Python stack trace. (Added as a result of :issue:`8032`.) + +* If you use the :file:`.gdbinit` file provided with Python, + the "pyo" macro in the 2.7 version now works correctly when the thread being + debugged doesn't hold the GIL; the macro now acquires it before printing. + (Contributed by Victor Stinner; :issue:`3632`.) + +* :cfunc:`Py_AddPendingCall` is now thread-safe, letting any + worker thread submit notifications to the main Python thread. This + is particularly useful for asynchronous IO operations. + (Contributed by Kristján Valur Jónsson; :issue:`4293`.) + +* New function: :cfunc:`PyCode_NewEmpty` creates an empty code object; + only the filename, function name, and first line number are required. + This is useful for extension modules that are attempting to + construct a more useful traceback stack. Previously such + extensions needed to call :cfunc:`PyCode_New`, which had many + more arguments. (Added by Jeffrey Yasskin.) + +* New function: :cfunc:`PyErr_NewExceptionWithDoc` creates a new + exception class, just as the existing :cfunc:`PyErr_NewException` does, + but takes an extra ``char *`` argument containing the docstring for the + new exception class. (Added by 'lekma' on the Python bug tracker; + :issue:`7033`.) + +* New function: :cfunc:`PyFrame_GetLineNumber` takes a frame object + and returns the line number that the frame is currently executing. + Previously code would need to get the index of the bytecode + instruction currently executing, and then look up the line number + corresponding to that address. (Added by Jeffrey Yasskin.) + +* New functions: :cfunc:`PyLong_AsLongAndOverflow` and + :cfunc:`PyLong_AsLongLongAndOverflow` approximates a Python long + integer as a C :ctype:`long` or :ctype:`long long`. + If the number is too large to fit into + the output type, an *overflow* flag is set and returned to the caller. + (Contributed by Case Van Horsen; :issue:`7528` and :issue:`7767`.) + +* New function: stemming from the rewrite of string-to-float conversion, + a new :cfunc:`PyOS_string_to_double` function was added. The old + :cfunc:`PyOS_ascii_strtod` and :cfunc:`PyOS_ascii_atof` functions + are now deprecated. + +* New function: :cfunc:`PySys_SetArgvEx` sets the value of + ``sys.argv`` and can optionally update ``sys.path`` to include the + directory containing the script named by ``sys.argv[0]`` depending + on the value of an *updatepath* parameter. + + This function was added to close a security hole for applications + that embed Python. The old function, :cfunc:`PySys_SetArgv`, would + always update ``sys.path``, and sometimes it would add the current + directory. This meant that, if you ran an application embedding + Python in a directory controlled by someone else, attackers could + put a Trojan-horse module in the directory (say, a file named + :file:`os.py`) that your application would then import and run. + + If you maintain a C/C++ application that embeds Python, check + whether you're calling :cfunc:`PySys_SetArgv` and carefully consider + whether the application should be using :cfunc:`PySys_SetArgvEx` + with *updatepath* set to false. + + Security issue reported as `CVE-2008-5983 + <http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_; + discussed in :issue:`5753`, and fixed by Antoine Pitrou. + +* New macros: the Python header files now define the following macros: + :cmacro:`Py_ISALNUM`, + :cmacro:`Py_ISALPHA`, + :cmacro:`Py_ISDIGIT`, + :cmacro:`Py_ISLOWER`, + :cmacro:`Py_ISSPACE`, + :cmacro:`Py_ISUPPER`, + :cmacro:`Py_ISXDIGIT`, + :cmacro:`Py_TOLOWER`, and :cmacro:`Py_TOUPPER`. + All of these functions are analogous to the C + standard macros for classifying characters, but ignore the current + locale setting, because in + several places Python needs to analyze characters in a + locale-independent way. (Added by Eric Smith; + :issue:`5793`.) + + .. XXX these macros don't seem to be described in the c-api docs. + +* Removed function: :cmacro:`PyEval_CallObject` is now only available + as a macro. A function version was being kept around to preserve + ABI linking compatibility, but that was in 1997; it can certainly be + deleted by now. (Removed by Antoine Pitrou; :issue:`8276`.) + +* New format codes: the :cfunc:`PyFormat_FromString`, + :cfunc:`PyFormat_FromStringV`, and :cfunc:`PyErr_Format` functions now + accept ``%lld`` and ``%llu`` format codes for displaying + C's :ctype:`long long` types. + (Contributed by Mark Dickinson; :issue:`7228`.) + +* The complicated interaction between threads and process forking has + been changed. Previously, the child process created by + :func:`os.fork` might fail because the child is created with only a + single thread running, the thread performing the :func:`os.fork`. + If other threads were holding a lock, such as Python's import lock, + when the fork was performed, the lock would still be marked as + "held" in the new process. But in the child process nothing would + ever release the lock, since the other threads weren't replicated, + and the child process would no longer be able to perform imports. + + Python 2.7 acquires the import lock before performing an + :func:`os.fork`, and will also clean up any locks created using the + :mod:`threading` module. C extension modules that have internal + locks, or that call :cfunc:`fork()` themselves, will not benefit + from this clean-up. + + (Fixed by Thomas Wouters; :issue:`1590864`.) + +* The :cfunc:`Py_Finalize` function now calls the internal + :func:`threading._shutdown` function; this prevents some exceptions from + being raised when an interpreter shuts down. + (Patch by Adam Olsen; :issue:`1722344`.) + +* When using the :ctype:`PyMemberDef` structure to define attributes + of a type, Python will no longer let you try to delete or set a + :const:`T_STRING_INPLACE` attribute. + + .. rev 79644 + +* Global symbols defined by the :mod:`ctypes` module are now prefixed + with ``Py``, or with ``_ctypes``. (Implemented by Thomas + Heller; :issue:`3102`.) + +* New configure option: the :option:`--with-system-expat` switch allows + building the :mod:`pyexpat` module to use the system Expat library. + (Contributed by Arfrever Frehtes Taifersar Arahesis; :issue:`7609`.) + +* New configure option: the + :option:`--with-valgrind` option will now disable the pymalloc + allocator, which is difficult for the Valgrind memory-error detector + to analyze correctly. + Valgrind will therefore be better at detecting memory leaks and + overruns. (Contributed by James Henstridge; :issue:`2422`.) + +* New configure option: you can now supply an empty string to + :option:`--with-dbmliborder=` in order to disable all of the various + DBM modules. (Added by Arfrever Frehtes Taifersar Arahesis; + :issue:`6491`.) + +* The :program:`configure` script now checks for floating-point rounding bugs + on certain 32-bit Intel chips and defines a :cmacro:`X87_DOUBLE_ROUNDING` + preprocessor definition. No code currently uses this definition, + but it's available if anyone wishes to use it. + (Added by Mark Dickinson; :issue:`2937`.) + + :program:`configure` also now sets a :envvar:`LDCXXSHARED` Makefile + variable for supporting C++ linking. (Contributed by Arfrever + Frehtes Taifersar Arahesis; :issue:`1222585`.) + +* The build process now creates the necessary files for pkg-config + support. (Contributed by Clinton Roy; :issue:`3585`.) + +* The build process now supports Subversion 1.7. (Contributed by + Arfrever Frehtes Taifersar Arahesis; :issue:`6094`.) + + +.. _whatsnew27-capsules: + +Capsules +------------------- + +Python 3.1 adds a new C datatype, :ctype:`PyCapsule`, for providing a +C API to an extension module. A capsule is essentially the holder of +a C ``void *`` pointer, and is made available as a module attribute; for +example, the :mod:`socket` module's API is exposed as ``socket.CAPI``, +and :mod:`unicodedata` exposes ``ucnhash_CAPI``. Other extensions +can import the module, access its dictionary to get the capsule +object, and then get the ``void *`` pointer, which will usually point +to an array of pointers to the module's various API functions. + +There is an existing data type already used for this, +:ctype:`PyCObject`, but it doesn't provide type safety. Evil code +written in pure Python could cause a segmentation fault by taking a +:ctype:`PyCObject` from module A and somehow substituting it for the +:ctype:`PyCObject` in module B. Capsules know their own name, +and getting the pointer requires providing the name:: + + void *vtable; + + if (!PyCapsule_IsValid(capsule, "mymodule.CAPI") { + PyErr_SetString(PyExc_ValueError, "argument type invalid"); + return NULL; + } + + vtable = PyCapsule_GetPointer(capsule, "mymodule.CAPI"); + +You are assured that ``vtable`` points to whatever you're expecting. +If a different capsule was passed in, :cfunc:`PyCapsule_IsValid` would +detect the mismatched name and return false. Refer to +:ref:`using-capsules` for more information on using these objects. + +Python 2.7 now uses capsules internally to provide various +extension-module APIs, but the :cfunc:`PyCObject_AsVoidPtr` was +modified to handle capsules, preserving compile-time compatibility +with the :ctype:`CObject` interface. Use of +:cfunc:`PyCObject_AsVoidPtr` will signal a +:exc:`PendingDeprecationWarning`, which is silent by default. + +Implemented in Python 3.1 and backported to 2.7 by Larry Hastings; +discussed in :issue:`5630`. + + +.. ====================================================================== + +Port-Specific Changes: Windows +----------------------------------- + +* The :mod:`msvcrt` module now contains some constants from + the :file:`crtassem.h` header file: + :data:`CRT_ASSEMBLY_VERSION`, + :data:`VC_ASSEMBLY_PUBLICKEYTOKEN`, + and :data:`LIBRARIES_ASSEMBLY_NAME_PREFIX`. + (Contributed by David Cournapeau; :issue:`4365`.) + +* The :mod:`_winreg` module for accessing the registry now implements + the :func:`~_winreg.CreateKeyEx` and :func:`~_winreg.DeleteKeyEx` + functions, extended versions of previously-supported functions that + take several extra arguments. The :func:`~_winreg.DisableReflectionKey`, + :func:`~_winreg.EnableReflectionKey`, and :func:`~_winreg.QueryReflectionKey` + were also tested and documented. + (Implemented by Brian Curtin: :issue:`7347`.) + +* The new :cfunc:`_beginthreadex` API is used to start threads, and + the native thread-local storage functions are now used. + (Contributed by Kristján Valur Jónsson; :issue:`3582`.) + +* The :func:`os.kill` function now works on Windows. The signal value + can be the constants :const:`CTRL_C_EVENT`, + :const:`CTRL_BREAK_EVENT`, or any integer. The first two constants + will send Control-C and Control-Break keystroke events to + subprocesses; any other value will use the :cfunc:`TerminateProcess` + API. (Contributed by Miki Tebeka; :issue:`1220212`.) + +* The :func:`os.listdir` function now correctly fails + for an empty path. (Fixed by Hirokazu Yamamoto; :issue:`5913`.) + +* The :mod:`mimelib` module will now read the MIME database from + the Windows registry when initializing. + (Patch by Gabriel Genellina; :issue:`4969`.) + +.. ====================================================================== + +Port-Specific Changes: Mac OS X +----------------------------------- + +* The path ``/Library/Python/2.7/site-packages`` is now appended to + ``sys.path``, in order to share added packages between the system + installation and a user-installed copy of the same version. + (Changed by Ronald Oussoren; :issue:`4865`.) + +Port-Specific Changes: FreeBSD +----------------------------------- + +* FreeBSD 7.1's :const:`SO_SETFIB` constant, used with + :func:`~socket.getsockopt`/:func:`~socket.setsockopt` to select an + alternate routing table, is now available in the :mod:`socket` + module. (Added by Kyle VanderBeek; :issue:`8235`.) + +Other Changes and Fixes +======================= + +* Two benchmark scripts, :file:`iobench` and :file:`ccbench`, were + added to the :file:`Tools` directory. :file:`iobench` measures the + speed of the built-in file I/O objects returned by :func:`open` + while performing various operations, and :file:`ccbench` is a + concurrency benchmark that tries to measure computing throughput, + thread switching latency, and IO processing bandwidth when + performing several tasks using a varying number of threads. + +* The :file:`Tools/i18n/msgfmt.py` script now understands plural + forms in :file:`.po` files. (Fixed by Martin von Löwis; + :issue:`5464`.) + +* When importing a module from a :file:`.pyc` or :file:`.pyo` file + with an existing :file:`.py` counterpart, the :attr:`co_filename` + attributes of the resulting code objects are overwritten when the + original filename is obsolete. This can happen if the file has been + renamed, moved, or is accessed through different paths. (Patch by + Ziga Seilnacht and Jean-Paul Calderone; :issue:`1180193`.) + +* The :file:`regrtest.py` script now takes a :option:`--randseed=` + switch that takes an integer that will be used as the random seed + for the :option:`-r` option that executes tests in random order. + The :option:`-r` option also reports the seed that was used + (Added by Collin Winter.) + +* Another :file:`regrtest.py` switch is :option:`-j`, which + takes an integer specifying how many tests run in parallel. This + allows reducing the total runtime on multi-core machines. + This option is compatible with several other options, including the + :option:`-R` switch which is known to produce long runtimes. + (Added by Antoine Pitrou, :issue:`6152`.) This can also be used + with a new :option:`-F` switch that runs selected tests in a loop + until they fail. (Added by Antoine Pitrou; :issue:`7312`.) + +* When executed as a script, the :file:`py_compile.py` module now + accepts ``'-'`` as an argument, which will read standard input for + the list of filenames to be compiled. (Contributed by Piotr + Ożarowski; :issue:`8233`.) + +.. ====================================================================== + +Porting to Python 2.7 +===================== + +This section lists previously described changes and other bugfixes +that may require changes to your code: + +* The :func:`range` function processes its arguments more + consistently; it will now call :meth:`__int__` on non-float, + non-integer arguments that are supplied to it. (Fixed by Alexander + Belopolsky; :issue:`1533`.) + +* The string :meth:`format` method changed the default precision used + for floating-point and complex numbers from 6 decimal + places to 12, which matches the precision used by :func:`str`. + (Changed by Eric Smith; :issue:`5920`.) + +* Because of an optimization for the :keyword:`with` statement, the special + methods :meth:`__enter__` and :meth:`__exit__` must belong to the object's + type, and cannot be directly attached to the object's instance. This + affects new-style classes (derived from :class:`object`) and C extension + types. (:issue:`6101`.) + +* Due to a bug in Python 2.6, the *exc_value* parameter to + :meth:`__exit__` methods was often the string representation of the + exception, not an instance. This was fixed in 2.7, so *exc_value* + will be an instance as expected. (Fixed by Florent Xicluna; + :issue:`7853`.) + +* When a restricted set of attributes were set using ``__slots__``, + deleting an unset attribute would not raise :exc:`AttributeError` + as you would expect. Fixed by Benjamin Peterson; :issue:`7604`.) + +In the standard library: + +* Operations with :class:`~datetime.datetime` instances that resulted in a year + falling outside the supported range didn't always raise + :exc:`OverflowError`. Such errors are now checked more carefully + and will now raise the exception. (Reported by Mark Leander, patch + by Anand B. Pillai and Alexander Belopolsky; :issue:`7150`.) + +* When using :class:`~decimal.Decimal` instances with a string's + :meth:`format` method, the default alignment was previously + left-alignment. This has been changed to right-alignment, which might + change the output of your programs. + (Changed by Mark Dickinson; :issue:`6857`.) + + Comparisons involving a signaling NaN value (or ``sNAN``) now signal + :const:`~decimal.InvalidOperation` instead of silently returning a true or + false value depending on the comparison operator. Quiet NaN values + (or ``NaN``) are now hashable. (Fixed by Mark Dickinson; + :issue:`7279`.) + +* The ElementTree library, :mod:`xml.etree`, no longer escapes + ampersands and angle brackets when outputting an XML processing + instruction (which looks like `<?xml-stylesheet href="#style1"?>`) + or comment (which looks like `<!-- comment -->`). + (Patch by Neil Muller; :issue:`2746`.) + +* The :meth:`~StringIO.StringIO.readline` method of :class:`~StringIO.StringIO` objects now does + nothing when a negative length is requested, as other file-like + objects do. (:issue:`7348`). + +* The :mod:`syslog` module will now use the value of ``sys.argv[0]`` as the + identifier instead of the previous default value of ``'python'``. + (Changed by Sean Reifschneider; :issue:`8451`.) + +* The :mod:`tarfile` module's default error handling has changed, to + no longer suppress fatal errors. The default error level was previously 0, + which meant that errors would only result in a message being written to the + debug log, but because the debug log is not activated by default, + these errors go unnoticed. The default error level is now 1, + which raises an exception if there's an error. + (Changed by Lars Gustäbel; :issue:`7357`.) + +* The :mod:`urlparse` module's :func:`~urlparse.urlsplit` now handles + unknown URL schemes in a fashion compliant with :rfc:`3986`: if the + URL is of the form ``"<something>://..."``, the text before the + ``://`` is treated as the scheme, even if it's a made-up scheme that + the module doesn't know about. This change may break code that + worked around the old behaviour. For example, Python 2.6.4 or 2.5 + will return the following: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', '', '//host/filename?query', '', '') + + Python 2.7 (and Python 2.6.5) will return: + + >>> import urlparse + >>> urlparse.urlsplit('invented://host/filename?query') + ('invented', 'host', '/filename?query', '', '') + + (Python 2.7 actually produces slightly different output, since it + returns a named tuple instead of a standard tuple.) + +For C extensions: + +* C extensions that use integer format codes with the ``PyArg_Parse*`` + family of functions will now raise a :exc:`TypeError` exception + instead of triggering a :exc:`DeprecationWarning` (:issue:`5080`). + +* Use the new :cfunc:`PyOS_string_to_double` function instead of the old + :cfunc:`PyOS_ascii_strtod` and :cfunc:`PyOS_ascii_atof` functions, + which are now deprecated. + +For applications that embed Python: + +* The :cfunc:`PySys_SetArgvEx` function was added, letting + applications close a security hole when the existing + :cfunc:`PySys_SetArgv` function was used. Check whether you're + calling :cfunc:`PySys_SetArgv` and carefully consider whether the + application should be using :cfunc:`PySys_SetArgvEx` with + *updatepath* set to false. + +.. ====================================================================== + + +.. _acks27: + +Acknowledgements +================ + +The author would like to thank the following people for offering +suggestions, corrections and assistance with various drafts of this +article: Nick Coghlan, Philip Jenvey, Ryan Lovett, R. David Murray, +Hugh Secker-Walker. + diff --git a/Doc/whatsnew/index.rst b/Doc/whatsnew/index.rst index f82b58c..ec45756 100644 --- a/Doc/whatsnew/index.rst +++ b/Doc/whatsnew/index.rst @@ -11,6 +11,7 @@ anyone wishing to stay up-to-date after a new release. .. toctree:: :maxdepth: 2 + 2.7.rst 2.6.rst 2.5.rst 2.4.rst |