diff options
Diffstat (limited to 'Doc')
92 files changed, 2024 insertions, 564 deletions
diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst index bf13eed..2f02241 100644 --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -45,6 +45,7 @@ in any early abort case). Unless otherwise stated, buffers are not NUL-terminated. .. note:: + For all ``#`` variants of formats (``s#``, ``y#``, etc.), the type of the length argument (int or :c:type:`Py_ssize_t`) is controlled by defining the macro :c:macro:`PY_SSIZE_T_CLEAN` before including diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index 989166f..7067448 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -525,11 +525,11 @@ Exception Objects reference, as accessible from Python through :attr:`__cause__`. -.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *ctx) +.. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause) - Set the cause associated with the exception to *ctx*. Use *NULL* to clear - it. There is no type check to make sure that *ctx* is either an exception - instance or :const:`None`. This steals a reference to *ctx*. + Set the cause associated with the exception to *cause*. Use *NULL* to clear + it. There is no type check to make sure that *cause* is either an exception + instance or :const:`None`. This steals a reference to *cause*. :attr:`__suppress_context__` is implicitly set to ``True`` by this function. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 6439d7f..0587e15 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -582,6 +582,7 @@ code, or when embedding the Python interpreter: .. index:: module: _thread .. note:: + When only the main thread exists, no GIL operations are needed. This is a common situation (most Python programs do not use threads), and the lock operations slow the interpreter down a bit. Therefore, the lock is not diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 26c4384..985a347 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -120,7 +120,7 @@ There are only a few functions special to module objects. Return a pointer to the :c:type:`PyModuleDef` struct from which the module was created, or *NULL* if the module wasn't created with - :c:func:`PyModule_Create`.i + :c:func:`PyModule_Create`. .. c:function:: PyObject* PyState_FindModule(PyModuleDef *def) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index ad84301..509ca3f 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -357,9 +357,9 @@ attribute is considered sufficient for this determination. .. c:function:: Py_ssize_t PyObject_LengthHint(PyObject *o, Py_ssize_t default) - Return an estimated length for the object *o*. First trying to return its - actual length, then an estimate using ``__length_hint__``, and finally - returning the default value. On error ``-1`` is returned. This is the + Return an estimated length for the object *o*. First try to return its + actual length, then an estimate using :meth:`~object.__length_hint__`, and + finally return the default value. On error return ``-1``. This is the equivalent to the Python expression ``operator.length_hint(o, default)``. .. versionadded:: 3.4 diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index 184affb..3922d50 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -129,6 +129,14 @@ type. Initializes a struct sequence type *type* from *desc* in place. +.. c:function:: int PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc *desc) + + The same as ``PyStructSequence_InitType``, but returns ``0`` on success and ``-1`` on + failure. + + .. versionadded:: 3.4 + + .. c:type:: PyStructSequence_Desc Contains the meta information of a struct sequence type to create. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 48b13e5..efd8574 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -205,9 +205,8 @@ type objects) *must* have the :attr:`ob_size` field. bit currently defined is :const:`Py_PRINT_RAW`. When the :const:`Py_PRINT_RAW` flag bit is set, the instance should be printed the same way as :c:member:`~PyTypeObject.tp_str` would format it; when the :const:`Py_PRINT_RAW` flag bit is clear, the instance - should be printed the same was as :c:member:`~PyTypeObject.tp_repr` would format it. It should - return ``-1`` and set an exception condition when an error occurred during the - comparison. + should be printed the same way as :c:member:`~PyTypeObject.tp_repr` would format it. It should + return ``-1`` and set an exception condition when an error occurs. It is possible that the :c:member:`~PyTypeObject.tp_print` field will be deprecated. In any case, it is recommended not to define :c:member:`~PyTypeObject.tp_print`, but instead to rely on diff --git a/Doc/conf.py b/Doc/conf.py index 5b63cad..c0ed8c6 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -61,6 +61,8 @@ add_module_names = True # By default, highlight as Python 3. highlight_language = 'python3' +needs_sphinx = '1.1' + # Options for HTML output # ----------------------- @@ -118,11 +120,11 @@ _stdauthor = r'Guido van Rossum\\Fred L. Drake, Jr., editor' latex_documents = [ ('c-api/index', 'c-api.tex', 'The Python/C API', _stdauthor, 'manual'), - ('distutils/index', 'distutils.tex', + ('distributing/index', 'distributing.tex', 'Distributing Python Modules', _stdauthor, 'manual'), ('extending/index', 'extending.tex', 'Extending and Embedding Python', _stdauthor, 'manual'), - ('install/index', 'install.tex', + ('installing/index', 'installing.tex', 'Installing Python Modules', _stdauthor, 'manual'), ('library/index', 'library.tex', 'The Python Library Reference', _stdauthor, 'manual'), diff --git a/Doc/contents.rst b/Doc/contents.rst index c0c6af3..29b07db 100644 --- a/Doc/contents.rst +++ b/Doc/contents.rst @@ -11,8 +11,8 @@ library/index.rst extending/index.rst c-api/index.rst - distutils/index.rst - install/index.rst + distributing/index.rst + installing/index.rst howto/index.rst faq/index.rst glossary.rst @@ -21,3 +21,11 @@ bugs.rst copyright.rst license.rst + +.. include legacy packaging docs in build + +.. toctree:: + :hidden: + + distutils/index.rst + install/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst new file mode 100644 index 0000000..d671041 --- /dev/null +++ b/Doc/distributing/index.rst @@ -0,0 +1,143 @@ +.. _distributing-index: + +############################### + Distributing Python Modules +############################### + +:Email: distutils-sig@python.org + + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the distribution part of the process. For a guide to +installing other Python projects, refer to the +:ref:`installation guide <installing-index>`. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* the `Python Package Index <https://pypi.python.org/pypi>`__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + <http://packaging.python.org/en/latest/future.html>`__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub <https://github.com/pypa>`__ and + `BitBucket <https://bitbucket.org/pypa/>`__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Open source licensing and collaboration +======================================= + +In most parts of the world, software is automatically covered by copyright. +This means that other developers require explicit permission to copy, use, +modify and redistribute the software. + +Open source licensing is a way of explicitly granting such permission in a +relatively consistent way, allowing developers to share and collaborate +efficiently by making common solutions to various problems freely available. +This leaves many developers free to spend more time focusing on the problems +that are relatively unique to their specific situation. + +The distribution tools provided with Python are designed to make it +reasonably straightforward for developers to make their own contributions +back to that common pool of software if they choose to do so. + +The same distribution tools can also be used to distribute software within +an organisation, regardless of whether that software is published as open +source software or not. + + +Installing the tools +==================== + +The standard library does not include build tools that support modern +Python packaging standards, as the core development team has found that it +is important to have standard tools that work consistently, even on older +versions of Python. + +The currently recommended build and distribution tools can be installed +using ``pip``:: + + pip install setuptools wheel twine + + +Reading the guide +================= + +The Python Packaging User Guide covers the various key steps and elements +involved in creating a project + +* `Project structure`_ +* `Building and packaging the project`_ +* `Uploading the project to the Python Package Index`_ + +.. _Project structure: \ + http://packaging.python.org/en/latest/tutorial.html#creating-your-own-project +.. _Building and packaging the project: \ + http://packaging.python.org/en/latest/tutorial.html#building-packaging-your-project +.. _Uploading the project to the Python Package Index: \ + http://packaging.python.org/en/latest/tutorial.html#uploading-your-project-to-pypi + + +How do I...? +============ + +These are quick answers or links for some common tasks. + +... choose a name for my project? +--------------------------------- + +This isn't an easy topic, but here are a few tips: + +* check the Python Package Index to see if the name is already in use +* check popular hosting sites like GitHub, BitBucket, etc to see if there + is already a project with that name +* check what comes up in a web search for the name you're considering +* avoid particularly common words, especially ones with multiple meanings, + as they can make it difficult for users to find your software when + searching for it + + +... create and distribute binary extensions? +-------------------------------------------- + +This is actually quite a complex topic, with a variety of alternatives +available depending on exactly what you're aiming to achieve. See the +Python Packaging User Guide for more information and recommendations. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + <http://packaging.python.org/en/latest/extensions.html>`__ + +.. other topics: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) diff --git a/Doc/distutils/apiref.rst b/Doc/distutils/apiref.rst index 54f0a4e..36a911e 100644 --- a/Doc/distutils/apiref.rst +++ b/Doc/distutils/apiref.rst @@ -853,17 +853,6 @@ Windows. It also contains the Mingw32CCompiler class which handles the mingw32 port of GCC (same as cygwin in no-cygwin mode). -:mod:`distutils.emxccompiler` --- OS/2 EMX Compiler -=================================================== - -.. module:: distutils.emxccompiler - :synopsis: OS/2 EMX Compiler support - - -This module provides the EMXCCompiler class, a subclass of -:class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2. - - :mod:`distutils.archive_util` --- Archiving utilities ====================================================== @@ -1171,15 +1160,6 @@ other utility module. underscore. No { } or ( ) style quoting is available. -.. function:: grok_environment_error(exc[, prefix='error: ']) - - Generate a useful error message from an :exc:`OSError` exception object. - Handles Python 1.5.1 and later styles, and does what it can to deal with - exception objects that don't have a filename (which happens when the error - is due to a two-file operation, such as :func:`~os.rename` or :func:`~os.link`). - Returns the error message as a string prefixed with *prefix*. - - .. function:: split_quoted(s) Split a string up according to Unix shell-like rules for quotes and backslashes. @@ -1943,8 +1923,12 @@ Subclasses of :class:`Command` must define the following methods. .. module:: distutils.command.clean :synopsis: Clean a package build area +This command removes the temporary files created by :command:`build` +and its subcommands, like intermediary compiled object files. With +the ``--all`` option, the complete build directory will be removed. -.. % todo +Extension modules built :ref:`in place <distutils-build-ext-inplace>` +will not be cleaned, as they are not in the build directory. :mod:`distutils.command.config` --- Perform package configuration diff --git a/Doc/distutils/configfile.rst b/Doc/distutils/configfile.rst index 890047c..ac79671 100644 --- a/Doc/distutils/configfile.rst +++ b/Doc/distutils/configfile.rst @@ -69,6 +69,8 @@ universal :option:`--help` option, e.g. :: Note that an option spelled :option:`--foo-bar` on the command-line is spelled :option:`foo_bar` in configuration files. +.. _distutils-build-ext-inplace: + For example, say you want your extensions to be built "in-place"---that is, you have an extension :mod:`pkg.ext`, and you want the compiled extension file (:file:`ext.so` on Unix, say) to be put in the same source directory as your diff --git a/Doc/distutils/index.rst b/Doc/distutils/index.rst index 1a6f04c..90d1c1a 100644 --- a/Doc/distutils/index.rst +++ b/Doc/distutils/index.rst @@ -1,8 +1,8 @@ .. _distutils-index: -############################### - Distributing Python Modules -############################### +############################################## + Distributing Python Modules (Legacy version) +############################################## :Authors: Greg Ward, Anthony Baxter :Email: distutils-sig@python.org diff --git a/Doc/distutils/setupscript.rst b/Doc/distutils/setupscript.rst index ee96302..fbe4290 100644 --- a/Doc/distutils/setupscript.rst +++ b/Doc/distutils/setupscript.rst @@ -685,6 +685,8 @@ include the following code fragment in your :file:`setup.py` before the DistributionMetadata.download_url = None +.. _debug-setup-script: + Debugging the setup script ========================== @@ -700,7 +702,8 @@ installation is broken because they don't read all the way down to the bottom and see that it's a permission problem. On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set +failure. For this purpose, the :envvar:`DISTUTILS_DEBUG` environment variable can be set to anything except an empty string, and distutils will now print detailed -information what it is doing, and prints the full traceback in case an exception -occurs. +information about what it is doing, dump the full traceback when an exception +occurs, and print the whole command line when an external program (like a C +compiler) fails. diff --git a/Doc/extending/index.rst b/Doc/extending/index.rst index 44a7f92..dd43926 100644 --- a/Doc/extending/index.rst +++ b/Doc/extending/index.rst @@ -21,14 +21,31 @@ Python) that give the language its wide application range. For a detailed description of the whole Python/C API, see the separate :ref:`c-api-index`. -.. note:: - This guide only covers the basic tools for creating extensions provided - as part of this version of CPython. Third party tools may offer simpler - alternatives. Refer to the `binary extensions section - <https://python-packaging-user-guide.readthedocs.org/en/latest/extensions.html>`__ - in the Python Packaging User Guide for more information. +Recommended third party tools +============================= +This guide only covers the basic tools for creating extensions provided +as part of this version of CPython. Third party tools like Cython, +``cffi``, SWIG and Numba offer both simpler and more sophisticated +approaches to creating C and C++ extensions for Python. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions <https://packaging.python.org/en/latest/extensions.html>`_ + The Python Packaging User Guide not only covers several available + tools that simplify the creation of binary extensions, but also + discusses the various reasons why creating an extension module may be + desirable in the first place. + + +Creating extensions without third party tools +============================================= + +This section of the guide covers creating C and C++ extensions without +assistance from third party tools. It is intended primarily for creators +of those tools, rather than being a recommended way to create your own +C extensions. .. toctree:: :maxdepth: 2 @@ -38,4 +55,17 @@ For a detailed description of the whole Python/C API, see the separate newtypes.rst building.rst windows.rst + +Embedding the CPython runtime in a larger application +===================================================== + +Sometimes, rather than creating an extension that runs inside the Python +interpreter as the main application, it is desirable to instead embed +the CPython runtime inside a larger application. This section covers +some of the details involved in doing that successfully. + +.. toctree:: + :maxdepth: 2 + :numbered: + embedding.rst diff --git a/Doc/faq/general.rst b/Doc/faq/general.rst index da2c933..6f4733f 100644 --- a/Doc/faq/general.rst +++ b/Doc/faq/general.rst @@ -471,7 +471,8 @@ that is written in Python using Tkinter. PythonWin is a Windows-specific IDE. Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult -http://www.python.org/editors/ for a full list of Python editing environments. +`the Python wiki <https://wiki.python.org/moin/PythonEditors>`_ for a full list +of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst index 34e2fdf..5f4ff17 100644 --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -513,6 +513,7 @@ For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the :mod:`array` module. .. note:: + To read and write binary data, it is mandatory to open the file in binary mode (here, passing ``"rb"`` to :func:`open`). If you use ``"r"`` instead (the default), the file will be open in text mode diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 79c7289..d514a80 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1103,6 +1103,7 @@ Use a list comprehension:: result = [obj.method() for obj in mylist] +.. _faq-augmented-assignment-tuple-error: Why does a_tuple[i] += ['item'] raise an exception when the addition works? --------------------------------------------------------------------------- diff --git a/Doc/glossary.rst b/Doc/glossary.rst index 2f658ac..f71a1f7 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -783,6 +783,14 @@ Glossary mapping rather than a sequence because the lookups use arbitrary :term:`immutable` keys rather than integers. + The :class:`collections.abc.Sequence` abstract base class + defines a much richer interface that goes beyond just + :meth:`__getitem__` and :meth:`__len__`, adding :meth:`count`, + :meth:`index`, :meth:`__contains__`, and + :meth:`__reversed__`. Types that implement this expanded + interface can be registered explicitly using + :func:`~abc.register`. + single dispatch A form of :term:`generic function` dispatch where the implementation is chosen based on the type of a single argument. diff --git a/Doc/howto/pyporting.rst b/Doc/howto/pyporting.rst index d6ad401..9d7e859 100644 --- a/Doc/howto/pyporting.rst +++ b/Doc/howto/pyporting.rst @@ -26,6 +26,32 @@ Porting Python 2 Code to Python 3 For help with porting, you can email the python-porting_ mailing list with questions. +The Short Version +================= + +* Decide what's the oldest version of Python 2 you want to support (if at all) +* Make sure you have a thorough test suite and use continuous integration + testing to make sure you stay compatible with the versions of Python you care + about +* If you have dependencies, check their Python 3 status using caniusepython3 + (`command-line tool <https://pypi.python.org/pypi/caniusepython3>`__, + `web app <https://caniusepython3.com/>`__) + +With that done, your options are: + +* If you are dropping Python 2 support, use 2to3_ to port to Python 3 +* If you are keeping Python 2 support, then start writing Python 2/3-compatible + code starting **TODAY** + + + If you have dependencies that have not been ported, reach out to them to port + their project while working to make your code compatible with Python 3 so + you're ready when your dependencies are all ported + + If all your dependencies have been ported (or you have none), go ahead and + port to Python 3 + +* If you are creating a new project that wants to have 2/3 compatibility, + code in Python 3 and then backport to Python 2 + Before You Begin ================ @@ -85,7 +111,7 @@ between Python 2 and 3 easier. Projects to Consider -------------------- -The lowest level library for suppoting Python 2 & 3 simultaneously is six_. +The lowest level library for supporting Python 2 & 3 simultaneously is six_. Reading through its documentation will give you an idea of where exactly the Python language changed between versions 2 & 3 and thus what you will want the library to help you continue to support. @@ -548,7 +574,10 @@ Backporting Python 3 code to Python 2 If you have Python 3 code and have little interest in supporting Python 2 you can use 3to2_ to translate from Python 3 code to Python 2 code. This is only -recommended if you don't plan to heavily support Python 2 users. +recommended if you don't plan to heavily support Python 2 users. Otherwise +write your code for Python 3 and then backport as far back as you want. This +is typically easier than going from Python 2 to 3 as you will have worked out +any difficulties with e.g. bytes/strings, etc. Other Resources diff --git a/Doc/includes/email-alternative-new-api.py b/Doc/includes/email-alternative-new-api.py index eeabf34..c1255a6 100644 --- a/Doc/includes/email-alternative-new-api.py +++ b/Doc/includes/email-alternative-new-api.py @@ -19,7 +19,7 @@ Cela ressemble à un excellent recipie[1] déjeuner. [1] http://www.yummly.com/recipe/Roasted-Asparagus-Epicurious-203718 ---Éric +--Pepé """) # Add the html version. This converts the message into a multipart/alternative diff --git a/Doc/install/index.rst b/Doc/install/index.rst index 779e923..d0fea3d 100644 --- a/Doc/install/index.rst +++ b/Doc/install/index.rst @@ -2,9 +2,9 @@ .. _install-index: -***************************** - Installing Python Modules -***************************** +******************************************** + Installing Python Modules (Legacy version) +******************************************** :Author: Greg Ward @@ -58,7 +58,8 @@ new goodies to their toolbox. You don't need to know Python to read this document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. +the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of +interest. .. _inst-trivial-install: diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst new file mode 100644 index 0000000..1284613 --- /dev/null +++ b/Doc/installing/index.rst @@ -0,0 +1,210 @@ +.. highlightlang:: none + +.. _installing-index: + +***************************** + Installing Python Modules +***************************** + +:Email: distutils-sig@python.org + +As a popular open source development project, Python has an active +supporting community of contributors and users that also make their software +available for other Python developers to use under open source license terms. + +This allows Python users to share and collaborate effectively, benefiting +from the solutions others have already created to common (and sometimes +even rare!) problems, as well as potentially contributing their own +solutions to the common pool. + +This guide covers the installation part of the process. For a guide to +creating and sharing your own Python projects, refer to the +:ref:`distribution guide <distributing-index>`. + +.. note:: + + For corporate and other institutional users, be aware that many + organisations have their own policies around using and contributing to + open source software. Please take such policies into account when making + use of the distribution and installation tools provided with Python. + + +Key terms +========= + +* ``pip`` is the preferred installer program. Starting with Python 3.4, it + is included by default with the Python binary installers. +* a virtual environment is a semi-isolated Python environment that allows + packages to be installed for use by a particular application, rather than + being installed system wide +* ``pyvenv`` is the standard tool for creating virtual environments, and has + been part of Python since Python 3.3. Starting with Python 3.4, it + defaults to installing ``pip`` into all created virtual environments +* the `Python Package Index <https://pypi.python.org/pypi>`__ is a public + repository of open source licensed packages made available for use by + other Python users +* the `Python Packaging Authority + <http://packaging.python.org/en/latest/future.html>`__ are the group of + developers and documentation authors responsible for the maintenance and + evolution of the standard packaging tools and the associated metadata and + file format standards. They maintain a variety of tools, documentation + and issue trackers on both `GitHub <https://github.com/pypa>`__ and + `BitBucket <https://bitbucket.org/pypa/>`__. +* ``distutils`` is the original build and distribution system first added to + the Python standard library in 1998. While direct use of ``distutils`` is + being phased out, it still laid the foundation for the current packaging + and distribution infrastructure, and it not only remains part of the + standard library, but its name lives on in other ways (such as the name + of the mailing list used to coordinate Python packaging standards + development). + + +Basic usage +=========== + +The standard packaging tools are all designed to be used from the command +line. For Windows users, the examples below assume that the option to +adjust the system PATH environment variable was selected when installing +Python. For Linux users, the command to install into the system version of +Python 3 is likely to be ``pip3`` rather than ``pip``. + +The following command will install the latest version of a module and its +dependencies from the Python Package Index:: + + pip install SomePackage + +It's also possible to specify an exact or minimum version directly on the +command line:: + + pip install SomePackage==1.0.4 # specific version + pip install 'SomePackage>=1.0.4' # minimum version + +Normally, if a suitable module is already installed, attempting to install +it again will have no effect. Upgrading existing modules must be requested +explicitly:: + + pip install --upgrade SomePackage + +More information and resources regarding ``pip`` and its capabilities can be +found in the `Python Packaging User Guide <http://packaging.python.org>`__. + +``pyvenv`` has its own documentation at :ref:`scripts-pyvenv`. Installing +into an active virtual environment uses the commands shown above. + +.. seealso:: + + `Python Packaging User Guide: Installing Python packages + <http://packaging.python.org/en/latest/tutorial.html#installing-python-packages>`__ + + +How do I ...? +============= + +These are quick answers or links for some common tasks. + +... install ``pip`` in versions of Python prior to Python 3.4? +-------------------------------------------------------------- + +Python only started bundling ``pip`` with Python 3.4. For earlier versions, +``pip`` needs to be "bootstrapped" as described in the Python Packaging +User Guide. + +.. seealso:: + + `Python Packaging User Guide: Installing the Tools + <http://packaging.python.org/en/latest/tutorial.html#installing-the-tools>`__ + + +.. installing-per-user-installation: + +... install packages just for the current user? +----------------------------------------------- + +Passing the ``--user`` option to ``pip install`` will install a package +just for the current user, rather than for all users of the system. + + +... install scientific Python packages? +--------------------------------------- + +A number of scientific Python packages have complex binary dependencies, and +aren't currently easy to install using ``pip`` directly. At this point in +time, it will often be easier for users to install these packages by +`other means +<http://packaging.python.org/en/latest/platforms.html#installing-scientific-packages>`__ +rather than attempting to install them with ``pip``. + +.. seealso:: + + `Python Packaging User Guide: Installing Scientific Packages + <http://packaging.python.org/en/latest/platforms.html#installing-scientific-packages>`__ + + +... work with multiple versions of Python installed in parallel? +---------------------------------------------------------------- + +On Linux, Mac OS X and other POSIX systems, use the versioned Python commands +in combination with the ``-m`` switch to run the appropriate copy of +``pip``:: + + python2 -m pip install SomePackage # default Python 2 + python2.7 -m pip install SomePackage # specifically Python 2.7 + python3 -m pip install SomePackage # default Python 3 + python3.4 -m pip install SomePackage # specifically Python 3.4 + +(appropriately versioned ``pip`` commands may also be available) + +On Windows, use the ``py`` Python launcher in combination with the ``-m`` +switch:: + + py -2 -m pip install SomePackage # default Python 2 + py -2.7 -m pip install SomePackage # specifically Python 2.7 + py -3 -m pip install SomePackage # default Python 3 + py -3.4 -m pip install SomePackage # specifically Python 3.4 + +.. other questions: + + Once the Development & Deployment part of PPUG is fleshed out, some of + those sections should be linked from new questions here (most notably, + we should have a question about avoiding depending on PyPI that links to + http://packaging.python.org/en/latest/deployment.html#pypi-mirrors-and-caches) + + +Common installation issues +========================== + +Installing into the system Python on Linux +------------------------------------------ + +On Linux systems, a Python installation will typically be included as part +of the distribution. Installing into this Python installation requires +root access to the system, and may interfere with the operation of the +system package manager and other components of the system if a component +is unexpectedly upgraded using ``pip``. + +On such systems, it is often better to use a virtual environment or a +per-user installation when installing packages with ``pip``. + + +Installing binary extensions +---------------------------- + +Python has typically relied heavily on source based distribution, with end +users being expected to compile extension modules from source as part of +the installation process. + +With the introduction of support for the binary ``wheel`` format, and the +ability to publish wheels for at least Windows and Mac OS X through the +Python Package Index, this problem is expected to diminish over time, +as users are more regularly able to install pre-built extensions rather +than needing to build them themselves. + +Some of the solutions for installing `scientific software +<http://packaging.python.org/en/latest/platforms.html#installing-scientific-packages>`__ +that is not yet available as pre-built ``wheel`` files may also help with +obtaining other binary extensions without needing to build them locally. + +.. seealso:: + + `Python Packaging User Guide: Binary Extensions + <http://packaging.python.org/en/latest/extensions.html>`__ diff --git a/Doc/library/aifc.rst b/Doc/library/aifc.rst index caecfbc..6fbcf28 100644 --- a/Doc/library/aifc.rst +++ b/Doc/library/aifc.rst @@ -226,7 +226,7 @@ number of frames must be filled in. file parameters have been set. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.writeframesraw(data) @@ -235,7 +235,7 @@ number of frames must be filled in. updated. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: aifc.close() diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst index 9d54964..df84169 100644 --- a/Doc/library/asyncio-eventloop.rst +++ b/Doc/library/asyncio-eventloop.rst @@ -488,7 +488,7 @@ Run subprocesses asynchronously using the :mod:`subprocess` module. * *stdout*: Either a file-like object representing the pipe to be connected to the subprocess's standard output stream using - :meth:`~BaseEventLoop.connect_write_pipe`, or the constant + :meth:`~BaseEventLoop.connect_read_pipe`, or the constant :const:`subprocess.PIPE` (the default). By default a new pipe will be created and connected. @@ -589,8 +589,8 @@ pool of processes). By default, an event loop uses a thread pool executor Arrange for a callback to be called in the specified executor. - *executor* is a :class:`~concurrent.futures.Executor` instance, - the default executor is used if *executor* is ``None``. + The *executor* argument should be an :class:`~concurrent.futures.Executor` + instance. The default executor is used if *executor* is ``None``. This method is a :ref:`coroutine <coroutine>`. diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst index dabe7d8..80c974a 100644 --- a/Doc/library/asyncio-protocol.rst +++ b/Doc/library/asyncio-protocol.rst @@ -376,8 +376,8 @@ The following callbacks are called on :class:`DatagramProtocol` instances. Flow control callbacks ---------------------- -These callbacks may be called on :class:`Protocol` and -:class:`SubprocessProtocol` instances: +These callbacks may be called on :class:`Protocol`, +:class:`DatagramProtocol` and :class:`SubprocessProtocol` instances: .. method:: BaseProtocol.pause_writing() @@ -402,6 +402,15 @@ buffer size reaches the low-water mark. are important to ensure that things go as expected when either mark is zero. +.. note:: + On BSD systems (OS X, FreeBSD, etc.) flow control is not supported + for :class:`DatagramProtocol`, because send failures caused by + writing too many packets cannot be detected easily. The socket + always appears 'ready' and excess packets are dropped; an + :class:`OSError` with errno set to :const:`errno.ENOBUFS` may or + may not be raised; if it is raised, it will be reported to + :meth:`DatagramProtocol.error_received` but otherwise ignored. + Coroutines and protocols ------------------------ @@ -488,6 +497,6 @@ TCP echo server example, send back received data and close the connection:: :meth:`Transport.close` can be called immediately after :meth:`WriteTransport.write` even if data are not sent yet on the socket: both methods are asynchronous. ``yield from`` is not needed because these transport -methods don't return coroutines. +methods are not coroutines. diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst index 5e6e657..861bcc0 100644 --- a/Doc/library/asyncio-subprocess.rst +++ b/Doc/library/asyncio-subprocess.rst @@ -8,16 +8,16 @@ Create a subprocess .. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - Run the shell command *cmd* given as a string. Return a :class:`Process` + Run the shell command *cmd* given as a string. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object <coroutine>`. + This function is a :ref:`coroutine <coroutine>`. .. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds) - Create a subprocess. Return a :class:`Process` instance. + Create a subprocess. Return a :class:`~asyncio.subprocess.Process` instance. - This function returns a :ref:`coroutine object <coroutine>`. + This function is a :ref:`coroutine <coroutine>`. Use the :meth:`BaseEventLoop.connect_read_pipe` and :meth:`BaseEventLoop.connect_write_pipe` methods to connect pipes. diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst index de38131..a299f09 100644 --- a/Doc/library/asyncio-sync.rst +++ b/Doc/library/asyncio-sync.rst @@ -64,7 +64,7 @@ Lock .. method:: locked() - Return ``True`` if lock is acquired. + Return ``True`` if the lock is acquired. .. method:: acquire() @@ -73,7 +73,7 @@ Lock This method blocks until the lock is unlocked, then sets it to locked and returns ``True``. - This method returns a :ref:`coroutine object <coroutine>`. + This method is a :ref:`coroutine <coroutine>`. .. method:: release() @@ -141,6 +141,15 @@ Condition A new :class:`Lock` object is created and used as the underlying lock. + .. method:: acquire() + + Acquire the underlying lock. + + This method blocks until the lock is unlocked, then sets it to locked and + returns ``True``. + + This method is a :ref:`coroutine <coroutine>`. + .. method:: notify(n=1) By default, wake up one coroutine waiting on this condition, if any. @@ -156,6 +165,10 @@ Condition call until it can reacquire the lock. Since :meth:`notify` does not release the lock, its caller should. + .. method:: locked() + + Return ``True`` if the underlying lock is acquired. + .. method:: notify_all() Wake up all threads waiting on this condition. This method acts like @@ -163,6 +176,18 @@ Condition calling thread has not acquired the lock when this method is called, a :exc:`RuntimeError` is raised. + .. method:: release() + + Release the underlying lock. + + When the lock is locked, reset it to unlocked, and return. If any other + coroutines are blocked waiting for the lock to become unlocked, allow + exactly one of them to proceed. + + When invoked on an unlocked lock, a :exc:`RuntimeError` is raised. + + There is no return value. + .. method:: wait() Wait until notified. diff --git a/Doc/library/audioop.rst b/Doc/library/audioop.rst index fbb7fc6..ce127aa 100644 --- a/Doc/library/audioop.rst +++ b/Doc/library/audioop.rst @@ -12,10 +12,8 @@ integers, unless specified otherwise. .. versionchanged:: 3.4 Support for 24-bit samples was added. - -.. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted by all functions in this - module. Strings no more supported. + All functions now accept any :term:`bytes-like object`. + String input now results in an immediate error. .. index:: single: Intel/DVI ADPCM @@ -82,7 +80,7 @@ The module defines the following variables and functions: "Byteswap" all samples in a fragment and returns the modified fragment. Converts big-endian samples to little-endian and vice versa. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: cross(fragment, width) diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 3d1f087..02b4d7b 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -1,27 +1,33 @@ -:mod:`base64` --- RFC 3548: Base16, Base32, Base64 Data Encodings -================================================================= +:mod:`base64` --- Base16, Base32, Base64, Base85 Data Encodings +=============================================================== .. module:: base64 - :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings + :synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings; + Base85 and Ascii85 .. index:: pair: base64; encoding single: MIME; base64 encoding -This module provides data encoding and decoding as specified in :rfc:`3548`. -This standard defines the Base16, Base32, and Base64 algorithms for encoding -and decoding arbitrary binary strings into ASCII-only byte strings that can be +This module provides functions for encoding binary data to printable +ASCII characters and decoding such encodings back to binary data. +It provides encoding and decoding functions for the encodings specified in +in :rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms, +and for the de-facto standard Ascii85 and Base85 encodings. + +The :rfc:`3548` encodings are suitable for encoding binary data so that it can safely sent by email, used as parts of URLs, or included as part of an HTTP POST request. The encoding algorithm is not the same as the :program:`uuencode` program. -There are two interfaces provided by this module. The modern interface -supports encoding and decoding ASCII byte string objects using all three -alphabets. Additionally, the decoding functions of the modern interface also -accept Unicode strings containing only ASCII characters. The legacy interface -provides for encoding and decoding to and from file-like objects as well as -byte strings, but only using the Base64 standard alphabet. +There are two :rfc:`3548` interfaces provided by this module. The modern +interface supports encoding and decoding ASCII byte string objects using all +three :rfc:`3548` defined alphabets (normal, URL-safe, and filesystem-safe). +Additionally, the decoding functions of the modern interface also accept +Unicode strings containing only ASCII characters. The legacy interface provides +for encoding and decoding to and from file-like objects as well as byte +strings, but only using the Base64 standard alphabet. .. versionchanged:: 3.3 ASCII-only Unicode strings are now accepted by the decoding functions of @@ -29,7 +35,7 @@ byte strings, but only using the Base64 standard alphabet. .. versionchanged:: 3.4 Any :term:`bytes-like object`\ s are now accepted by all - encoding and decoding functions in this module. + encoding and decoding functions in this module. Ascii85/Base85 support added. The modern interface provides: diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index a20c4be..ccc9dc6 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -142,36 +142,43 @@ The :mod:`csv` module defines the following functions: The :mod:`csv` module defines the following classes: -.. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds) - - Create an object which operates like a regular reader but maps the information - read into a dict whose keys are given by the optional *fieldnames* parameter. - If the *fieldnames* parameter is omitted, the values in the first row of the - *csvfile* will be used as the fieldnames. If the row read has more fields - than the fieldnames sequence, the remaining data is added as a sequence - keyed by the value of *restkey*. If the row read has fewer fields than the - fieldnames sequence, the remaining keys take the value of the optional - *restval* parameter. Any other optional or keyword arguments are passed to - the underlying :class:`reader` instance. - - -.. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds) - - Create an object which operates like a regular writer but maps dictionaries onto - output rows. The *fieldnames* parameter identifies the order in which values in - the dictionary passed to the :meth:`writerow` method are written to the - *csvfile*. The optional *restval* parameter specifies the value to be written - if the dictionary is missing a key in *fieldnames*. If the dictionary passed to - the :meth:`writerow` method contains a key not found in *fieldnames*, the - optional *extrasaction* parameter indicates what action to take. If it is set - to ``'raise'`` a :exc:`ValueError` is raised. If it is set to ``'ignore'``, - extra values in the dictionary are ignored. Any other optional or keyword - arguments are passed to the underlying :class:`writer` instance. - - Note that unlike the :class:`DictReader` class, the *fieldnames* parameter of - the :class:`DictWriter` is not optional. Since Python's :class:`dict` objects - are not ordered, there is not enough information available to deduce the order - in which the row should be written to the *csvfile*. +.. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, \ + dialect='excel', *args, **kwds) + + Create an object which operates like a regular reader but maps the + information read into a dict whose keys are given by the optional + *fieldnames* parameter. The *fieldnames* parameter is a :mod:`sequence + <collections.abc>` whose elements are associated with the fields of the + input data in order. These elements become the keys of the resulting + dictionary. If the *fieldnames* parameter is omitted, the values in the + first row of the *csvfile* will be used as the fieldnames. If the row read + has more fields than the fieldnames sequence, the remaining data is added as + a sequence keyed by the value of *restkey*. If the row read has fewer + fields than the fieldnames sequence, the remaining keys take the value of + the optional *restval* parameter. Any other optional or keyword arguments + are passed to the underlying :class:`reader` instance. + + +.. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', \ + dialect='excel', *args, **kwds) + + Create an object which operates like a regular writer but maps dictionaries + onto output rows. The *fieldnames* parameter is a :mod:`sequence + <collections.abc>` of keys that identify the order in which values in the + dictionary passed to the :meth:`writerow` method are written to the + *csvfile*. The optional *restval* parameter specifies the value to be + written if the dictionary is missing a key in *fieldnames*. If the + dictionary passed to the :meth:`writerow` method contains a key not found in + *fieldnames*, the optional *extrasaction* parameter indicates what action to + take. If it is set to ``'raise'`` a :exc:`ValueError` is raised. If it is + set to ``'ignore'``, extra values in the dictionary are ignored. Any other + optional or keyword arguments are passed to the underlying :class:`writer` + instance. + + Note that unlike the :class:`DictReader` class, the *fieldnames* parameter + of the :class:`DictWriter` is not optional. Since Python's :class:`dict` + objects are not ordered, there is not enough information available to deduce + the order in which the row should be written to the *csvfile*. .. class:: Dialect diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 314636e..f3e60b4 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -12,7 +12,7 @@ The :mod:`curses` module provides an interface to the curses library, the de-facto standard for portable advanced terminal handling. While curses is most widely used in the Unix environment, versions are available -for DOS, OS/2, and possibly other systems as well. This extension module is +for Windows, DOS, and possibly other systems as well. This extension module is designed to match the API of ncurses, an open-source curses library hosted on Linux and the BSD variants of Unix. diff --git a/Doc/library/distutils.rst b/Doc/library/distutils.rst index 6666a9b..8e3732b 100644 --- a/Doc/library/distutils.rst +++ b/Doc/library/distutils.rst @@ -12,14 +12,15 @@ additional modules into a Python installation. The new modules may be either 100%-pure Python, or may be extension modules written in C, or may be collections of Python packages which include modules coded in both Python and C. +Most Python users will *not* want to use this module directly, but instead +use the cross-version tools maintained by the Python Packaging Authority. +Refer to the `Python Packaging User Guide <http://packaging.python.org>`_ +for more information. -User documentation and API reference are provided in another document: +For the benefits of packaging tool authors and users seeking a deeper +understanding of the details of the current packaging and distribution +system, the legacy :mod:`distutils` based user documentation and API +reference remain available: -.. seealso:: - - :ref:`distutils-index` - The manual for developers and packagers of Python modules. This describes - how to prepare :mod:`distutils`\ -based packages so that they may be - easily installed into an existing Python installation. It also contains - instructions for end-users wanting to install a distutils-based package, - :ref:`install-index`. +* :ref:`install-index` +* :ref:`distutils-index` diff --git a/Doc/library/email-examples.rst b/Doc/library/email-examples.rst index 294e131..cbbcb78 100644 --- a/Doc/library/email-examples.rst +++ b/Doc/library/email-examples.rst @@ -40,8 +40,10 @@ text version: [2]_ .. literalinclude:: ../includes/email-alternative.py -Examples using the Provision API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. _email-contentmanager-api-examples: + +Examples using the Provisional API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Here is a reworking of the last example using the provisional API. To make things a bit more interesting, we include a related image in the html part, and we save a copy of what we are going to send to disk, as well as sending it. diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst index 5162da1..8f0bfdb 100644 --- a/Doc/library/email.contentmanager.rst +++ b/Doc/library/email.contentmanager.rst @@ -54,6 +54,7 @@ this module. documented in this module because of the provisional nature of the code, the implementation lives in the :mod:`email.message` module. +.. currentmodule:: email.message .. class:: EmailMessage(policy=default) @@ -235,6 +236,16 @@ this module. all other headers intact and in their original order. +.. class:: MIMEPart(policy=default) + + This class represents a subpart of a MIME message. It is identical to + :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are + added when :meth:`~EmailMessage.set_content` is called, since sub-parts do + not need their own :mailheader:`MIME-Version` headers. + + +.. currentmodule:: email.contentmanager + .. class:: ContentManager() Base class for content managers. Provides the standard registry mechanisms @@ -305,14 +316,6 @@ this module. values of *typekey*, see :meth:`set_content`. -.. class:: MIMEPart(policy=default) - - This class represents a subpart of a MIME message. It is identical to - :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are - added when :meth:`~EmailMessage.set_content` is called, since sub-parts do - not need their own :mailheader:`MIME-Version` headers. - - Content Manager Instances ~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 84a5f51..58f708c 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -466,7 +466,7 @@ Here are the methods of the :class:`Message` class: to ``False``. - .. method:: set_param(param, value, header='Content-Type', requote=True, + .. method:: set_param(param, value, header='Content-Type', requote=True, \ charset=None, language='', replace=False) Set a parameter in the :mailheader:`Content-Type` header. If the @@ -488,7 +488,7 @@ Here are the methods of the :class:`Message` class: end of the list of headers. If *replace* is ``True``, the header will be updated in place. - .. versionchanged: 3.4 ``replace`` keyword was added. + .. versionchanged:: 3.4 ``replace`` keyword was added. .. method:: del_param(param, header='content-type', requote=True) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst index c2f9e6a..97358f0 100644 --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -419,7 +419,7 @@ added matters. To illustrate:: additional arguments. By default ``content_manager`` is set to :data:`~email.contentmanager.raw_data_manager`. - .. versionadded 3.4 + .. versionadded:: 3.4 The class provides the following concrete implementations of the abstract diff --git a/Doc/library/ensurepip.rst b/Doc/library/ensurepip.rst index 0d82135..8012f51 100644 --- a/Doc/library/ensurepip.rst +++ b/Doc/library/ensurepip.rst @@ -28,7 +28,7 @@ when creating a virtual environment) or after explicitly uninstalling .. seealso:: - :ref:`install-index` + :ref:`installing-index` The end user guide for installing Python packages :pep:`453`: Explicit bootstrapping of pip in Python installations diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 628f91c..fc7267b 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -139,7 +139,7 @@ If you want to access enum members by *name*, use item access:: >>> Color['green'] <Color.green: 2> -If have an enum member and need its :attr:`name` or :attr:`value`:: +If you have an enum member and need its :attr:`name` or :attr:`value`:: >>> member = Color.red >>> member.name diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index c2017cc..0aac1bf 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -265,7 +265,6 @@ The following exceptions are the exceptions that are usually raised. :exc:`mmap.error` have been merged into :exc:`OSError`. .. versionchanged:: 3.4 - The :attr:`filename` attribute is now the original file name passed to the function, instead of the name encoded to or decoded from the filesystem encoding. Also, the :attr:`filename2` attribute was added. diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index fba199b..c2c7401 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -99,7 +99,9 @@ another rational number, or from a string. value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)`` - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`float`. @@ -108,7 +110,9 @@ another rational number, or from a string. This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. - .. note:: From Python 3.2 onwards, you can also construct a + .. note:: + + From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`decimal.Decimal` instance. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 4371969..2b37069 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -540,12 +540,13 @@ are always available. They are listed here in alphabetical order. A call to ``format(value, format_spec)`` is translated to ``type(value).__format__(format_spec)`` which bypasses the instance dictionary when searching for the value's :meth:`__format__` method. A - :exc:`TypeError` exception is raised if the method is not found or if either - the *format_spec* or the return value are not strings. + :exc:`TypeError` exception is raised if the method search reaches + :mod:`object` and the *format_spec* is non-empty, or if either the + *format_spec* or the return value are not strings. - .. versionadded:: 3.4 + .. versionchanged:: 3.4 ``object().__format__(format_spec)`` raises :exc:`TypeError` - if *format_spec* is not empty string. + if *format_spec* is not an empty string. .. _func-frozenset: @@ -609,12 +610,26 @@ are always available. They are listed here in alphabetical order. This function is added to the built-in namespace by the :mod:`site` module. + .. versionchanged:: 3.4 + Changes to :mod:`pydoc` and :mod:`inspect` mean that the reported + signatures for callables are now more comprehensive and consistent. + .. function:: hex(x) - Convert an integer number to a hexadecimal string. The result is a valid Python - expression. If *x* is not a Python :class:`int` object, it has to define an - :meth:`__index__` method that returns an integer. + Convert an integer number to a lowercase hexadecimal string + prefixed with "0x", for example: + + >>> hex(255) + '0xff' + >>> hex(-42) + '-0x2a' + + If x is not a Python :class:`int` object, it has to define an __index__() + method that returns an integer. + + See also :func:`int` for converting a hexadecimal string to an + integer using a base of 16. .. note:: @@ -780,6 +795,9 @@ are always available. They are listed here in alphabetical order. such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and ``heapq.nlargest(1, iterable, key=keyfunc)``. + .. versionadded:: 3.4 + The *default* keyword-only argument. + .. _func-memoryview: .. function:: memoryview(obj) @@ -811,6 +829,9 @@ are always available. They are listed here in alphabetical order. such as ``sorted(iterable, key=keyfunc)[0]`` and ``heapq.nsmallest(1, iterable, key=keyfunc)``. + .. versionadded:: 3.4 + The *default* keyword-only argument. + .. function:: next(iterator[, default]) Retrieve the next item from the *iterator* by calling its diff --git a/Doc/library/getopt.rst b/Doc/library/getopt.rst index b6ab3df..f9a1e53 100644 --- a/Doc/library/getopt.rst +++ b/Doc/library/getopt.rst @@ -10,6 +10,7 @@ -------------- .. note:: + The :mod:`getopt` module is a parser for command line options whose API is designed to be familiar to users of the C :c:func:`getopt` function. Users who are unfamiliar with the C :c:func:`getopt` function or who would like to write diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 8f4cd70..d27902e 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -101,18 +101,18 @@ Hashlib provides the following constant attributes: .. data:: algorithms_guaranteed - Contains the names of the hash algorithms guaranteed to be supported + A set containing the names of the hash algorithms guaranteed to be supported by this module on all platforms. .. versionadded:: 3.2 .. data:: algorithms_available - Contains the names of the hash algorithms that are available - in the running Python interpreter. These names will be recognized - when passed to :func:`new`. :attr:`algorithms_guaranteed` - will always be a subset. Duplicate algorithms with different - name formats may appear in this set (thanks to OpenSSL). + A set containing the names of the hash algorithms that are available in the + running Python interpreter. These names will be recognized when passed to + :func:`new`. :attr:`algorithms_guaranteed` will always be a subset. The + same algorithm may appear multiple times in this set under different names + (thanks to OpenSSL). .. versionadded:: 3.2 diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst index 2e9b0b2..4858235 100644 --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -23,9 +23,8 @@ This module implements the HMAC algorithm as described by :rfc:`2104`. defaults to the :data:`hashlib.md5` constructor. .. versionchanged:: 3.4 - Parameter *key* can be a bytes or bytearray object. Parameter *msg* can - be of any type supported by :mod:`hashlib`. - + Parameter *key* can be a bytes or bytearray object. + Parameter *msg* can be of any type supported by :mod:`hashlib`. Paramter *digestmod* can be the name of a hash algorithm. .. deprecated:: 3.4 diff --git a/Doc/library/http.server.rst b/Doc/library/http.server.rst index 0f7c8b3..dd19fb4 100644 --- a/Doc/library/http.server.rst +++ b/Doc/library/http.server.rst @@ -351,7 +351,7 @@ of which this module provides three different variants: The :class:`SimpleHTTPRequestHandler` class can be used in the following manner in order to create a very basic webserver serving files relative to -the current directory. :: +the current directory:: import http.server import socketserver @@ -365,15 +365,17 @@ the current directory. :: print("serving at port", PORT) httpd.serve_forever() +.. _http-server-cli: + :mod:`http.server` can also be invoked directly using the :option:`-m` switch of the interpreter with a ``port number`` argument. Similar to -the previous example, this serves files relative to the current directory. :: +the previous example, this serves files relative to the current directory:: python -m http.server 8000 -By default, server binds itself to all interfaces. To restrict it to bind to a -particular interface only, ``--bind ADDRESS`` argument can be used. For e.g, to -restrict the server to bind only to localhost. :: +By default, server binds itself to all interfaces. The option ``-b/--bind`` +specifies a specific address to which it should bind. For example, the +following command causes the server to bind to localhost only:: python -m http.server 8000 --bind 127.0.0.1 @@ -422,7 +424,7 @@ restrict the server to bind only to localhost. :: reasons. Problems with the CGI script will be translated to error 403. :class:`CGIHTTPRequestHandler` can be enabled in the command line by passing -the ``--cgi`` option.:: +the ``--cgi`` option:: python -m http.server --cgi 8000 diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index a81ccfd..3b55b06 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -134,7 +134,7 @@ Functions When :func:`reload` is executed: - * Python modules' code is recompiled and the module-level code re-executed, + * Python module's code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module's dictionary by reusing the :term:`loader` which originally loaded the module. The ``init`` function of extension modules is not called a second diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index ccb2bd7..0c08712 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -729,6 +729,11 @@ Classes and functions Consider using the new :ref:`Signature Object <inspect-signature-object>` interface, which provides a better way of introspecting functions. + .. versionchanged:: 3.4 + This function is now based on :func:`signature`, but still ignores + ``__wrapped__`` attributes and includes the already bound first + parameter in the signature output for bound methods. + .. function:: getargvalues(frame) diff --git a/Doc/library/io.rst b/Doc/library/io.rst index c80198b..79f65e0 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -686,6 +686,7 @@ than raw I/O does. :exc:`UnsupportedOperation`. .. warning:: + :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 9ccf262..9625e71 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -154,20 +154,20 @@ write code that handles both IP versions correctly. .. attribute:: is_private ``True`` if the address is allocated for private networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). .. attribute:: is_global ``True`` if the address is allocated for public networks. See - iana-ipv4-special-registry (for IPv4) or iana-ipv6-special-registry + iana-ipv4-special-registry_ (for IPv4) or iana-ipv6-special-registry_ (for IPv6). - .. versionadded:: 3.4 + .. versionadded:: 3.4 .. attribute:: is_unspecified - ``True`` if the address is unspecified. See :RFC:`5375` (for IPv4) + ``True`` if the address is unspecified. See :RFC:`5735` (for IPv4) or :RFC:`2373` (for IPv6). .. attribute:: is_reserved @@ -184,6 +184,9 @@ write code that handles both IP versions correctly. ``True`` if the address is reserved for link-local usage. See :RFC:`3927`. +.. _iana-ipv4-special-registry: http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml +.. _iana-ipv6-special-registry: http://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml + .. class:: IPv6Address(address) @@ -218,18 +221,23 @@ write code that handles both IP versions correctly. The long form of the address representation, with all leading zeroes and groups consisting entirely of zeroes included. + + For the following attributes, see the corresponding documention of the + :class:`IPv4Address` class: + .. attribute:: packed .. attribute:: version .. attribute:: max_prefixlen .. attribute:: is_multicast .. attribute:: is_private + .. attribute:: is_global .. attribute:: is_unspecified .. attribute:: is_reserved .. attribute:: is_loopback .. attribute:: is_link_local - Refer to the corresponding attribute documentation in - :class:`IPv4Address` + .. versionadded:: 3.4 + is_global .. attribute:: is_site_local diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst index 5fb1b7a..171b0a3 100644 --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -148,7 +148,9 @@ in :mod:`logging` itself) and defining handlers which are declared either in send it to the socket as a string of bytes preceded by a four-byte length string packed in binary using ``struct.pack('>L', n)``. - .. note:: Because portions of the configuration are passed through + .. note:: + + Because portions of the configuration are passed through :func:`eval`, use of this function may open its users to a security risk. While the function only binds to a socket on ``localhost``, and so does not accept connections from remote machines, there are scenarios where @@ -752,7 +754,9 @@ The ``class`` entry is optional. It indicates the name of the formatter's class :class:`~logging.Formatter` can present exception tracebacks in an expanded or condensed format. -.. note:: Due to the use of :func:`eval` as described above, there are +.. note:: + + Due to the use of :func:`eval` as described above, there are potential security risks which result from using the :func:`listen` to send and receive configurations via sockets. The risks are limited to where multiple users with no mutual trust run code on the same machine; see the diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 7c3ab59..3c41672 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -36,9 +36,9 @@ Number-theoretic and representation functions .. function:: copysign(x, y) - Return *x* with the sign of *y*. On a platform that supports - signed zeros, ``copysign(1.0, -0.0)`` returns *-1.0*. - + Return a float with the magnitude (absolute value) of *x* but the sign of + *y*. On platforms that support signed zeros, ``copysign(1.0, -0.0)`` + returns *-1.0*. .. function:: fabs(x) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index a8872e7..b303e16 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -136,9 +136,11 @@ to start a process. These *start methods* are Available on Unix platforms which support passing file descriptors over Unix pipes. -Before Python 3.4 *fork* was the only option available on Unix. Also, -prior to Python 3.4, child processes would inherit all the parents -inheritable handles on Windows. +.. versionchanged:: 3.4 + *spawn* added on all unix platforms, and *forkserver* added for + some unix platforms. + Child processes no longer inherit all of the parents inheritable + handles on Windows. On Unix using the *spawn* or *forkserver* start methods will also start a *semaphore tracker* process which tracks the unlinked named @@ -1474,7 +1476,7 @@ their parent process exits. The manager classes are defined in the *exposed* is used to specify a sequence of method names which proxies for this typeid should be allowed to access using - :meth:`BaseProxy._callMethod`. (If *exposed* is ``None`` then + :meth:`BaseProxy._callmethod`. (If *exposed* is ``None`` then :attr:`proxytype._exposed_` is used instead if it exists.) In the case where no exposed list is specified, all "public methods" of the shared object will be accessible. (Here a "public method" means any attribute @@ -1853,25 +1855,30 @@ with the :class:`Pool` class. callbacks and has a parallel map implementation. *processes* is the number of worker processes to use. If *processes* is - ``None`` then the number returned by :func:`os.cpu_count` is used. If - *initializer* is not ``None`` then each worker process will call + ``None`` then the number returned by :func:`os.cpu_count` is used. + + If *initializer* is not ``None`` then each worker process will call ``initializer(*initargs)`` when it starts. + *maxtasksperchild* is the number of tasks a worker process can complete + before it will exit and be replaced with a fresh worker process, to enable + unused resources to be freed. The default *maxtasksperchild* is None, which + means worker processes will live as long as the pool. + + *context* can be used to specify the context used for starting + the worker processes. Usually a pool is created using the + function :func:`multiprocessing.Pool` or the :meth:`Pool` method + of a context object. In both cases *context* is set + appropriately. + Note that the methods of the pool object should only be called by the process which created the pool. .. versionadded:: 3.2 - *maxtasksperchild* is the number of tasks a worker process can complete - before it will exit and be replaced with a fresh worker process, to enable - unused resources to be freed. The default *maxtasksperchild* is None, which - means worker processes will live as long as the pool. + *maxtasksperchild* .. versionadded:: 3.4 - *context* can be used to specify the context used for starting - the worker processes. Usually a pool is created using the - function :func:`multiprocessing.Pool` or the :meth:`Pool` method - of a context object. In both cases *context* is set - appropriately. + *context* .. note:: diff --git a/Doc/library/operator.rst b/Doc/library/operator.rst index fa3bcdd..3bcbaa4 100644 --- a/Doc/library/operator.rst +++ b/Doc/library/operator.rst @@ -240,9 +240,9 @@ their character equivalents. .. function:: length_hint(obj, default=0) - Return an estimated length for the object *o*. First trying to return its + Return an estimated length for the object *o*. First try to return its actual length, then an estimate using :meth:`object.__length_hint__`, and - finally returning the default value. + finally return the default value. .. versionadded:: 3.4 diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 269856c..3771ab0 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -193,11 +193,17 @@ the :mod:`glob` module.) .. function:: ismount(path) - Return ``True`` if pathname *path* is a :dfn:`mount point`: a point in a file - system where a different file system has been mounted. The function checks - whether *path*'s parent, :file:`path/..`, is on a different device than *path*, - or whether :file:`path/..` and *path* point to the same i-node on the same - device --- this should detect mount points for all Unix and POSIX variants. + Return ``True`` if pathname *path* is a :dfn:`mount point`: a point in a + file system where a different file system has been mounted. On POSIX, the + function checks whether *path*'s parent, :file:`path/..`, is on a different + device than *path*, or whether :file:`path/..` and *path* point to the same + i-node on the same device --- this should detect mount points for all Unix + and POSIX variants. On Windows, a drive letter root and a share UNC are + always mount points, and for any other path ``GetVolumePathName`` is called + to see if it is different from the input path. + + .. versionadded:: 3.4 + Support for detecting non-root mount points on Windows. .. function:: join(path1[, path2[, ...]]) @@ -251,7 +257,7 @@ the :mod:`glob` module.) .. function:: samefile(path1, path2) Return ``True`` if both pathname arguments refer to the same file or directory. - On Unix, this is determined by the device number and i-node number and raises an + This is determined by the device number and i-node number and raises an exception if a :func:`os.stat` call on either pathname fails. Availability: Unix, Windows. diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 1fc8de9..f50e5d9 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -260,7 +260,9 @@ process and user. Availability: Unix. - .. note:: On Mac OS X, :func:`getgroups` behavior differs somewhat from + .. note:: + + On Mac OS X, :func:`getgroups` behavior differs somewhat from other Unix platforms. If the Python interpreter was built with a deployment target of :const:`10.5` or earlier, :func:`getgroups` returns the list of effective group ids associated with the current user process; @@ -940,8 +942,9 @@ or `the MSDN <http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Window the C library. .. versionchanged:: 3.4 - Add :data:`O_TMPFILE` constant. It's only available on Linux Kernel 3.11 - or newer. + Add :data:`O_PATH` on systems that support it. + Add :data:`O_TMPFILE`, only available on Linux Kernel 3.11 + or newer. .. function:: openpty() @@ -2653,7 +2656,7 @@ written in Python, such as a mail server's external command delivery program. Fork a child process. Return ``0`` in the child and the child's process id in the parent. If an error occurs :exc:`OSError` is raised. - Note that some platforms including FreeBSD <= 6.3, Cygwin and OS/2 EMX have + Note that some platforms including FreeBSD <= 6.3 and Cygwin have known issues when using fork() from a thread. .. warning:: @@ -2899,7 +2902,6 @@ written in Python, such as a mail server's external command delivery program. :manpage:`times(2)` or the corresponding Windows Platform API documentation. On Windows, only :attr:`user` and :attr:`system` are known; the other attributes are zero. - On OS/2, only :attr:`elapsed` is known; the other attributes are zero. Availability: Unix, Windows. diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 3aa9d4b..ec1dc4f 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -522,6 +522,36 @@ Pure paths provide the following methods and properties: ValueError: '/etc/passwd' does not start with '/usr' +.. method:: PurePath.with_name(name) + + Return a new path with the :attr:`name` changed. If the original path + doesn't have a name, ValueError is raised:: + + >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz') + >>> p.with_name('setup.py') + PureWindowsPath('c:/Downloads/setup.py') + >>> p = PureWindowsPath('c:/') + >>> p.with_name('setup.py') + Traceback (most recent call last): + File "<stdin>", line 1, in <module> + File "/home/antoine/cpython/default/Lib/pathlib.py", line 751, in with_name + raise ValueError("%r has an empty name" % (self,)) + ValueError: PureWindowsPath('c:/') has an empty name + + +.. method:: PurePath.with_suffix(suffix) + + Return a new path with the :attr:`suffix` changed. If the original path + doesn't have a suffix, the new *suffix* is appended instead:: + + >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz') + >>> p.with_suffix('.bz2') + PureWindowsPath('c:/Downloads/pathlib.tar.bz2') + >>> p = PureWindowsPath('README') + >>> p.with_suffix('.txt') + PureWindowsPath('README.txt') + + .. _concrete-paths: diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 10b7848..13ea7b9 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -147,6 +147,7 @@ support. *prefix* is a string to output on the front of every module name on output. .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and @@ -185,6 +186,7 @@ support. walk_packages(ctypes.__path__, ctypes.__name__ + '.') .. note:: + Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst index 7301d49..6a2d6b4 100644 --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -32,6 +32,9 @@ Values can be strings, integers, floats, booleans, tuples, lists, dictionaries (but only with string keys), :class:`Data`, :class:`bytes`, :class:`bytesarray` or :class:`datetime.datetime` objects. +.. versionchanged:: 3.4 + New API, old API deprecated. Support for binary format plists added. + .. seealso:: `PList manual page <http://developer.apple.com/documentation/Darwin/Reference/ManPages/man5/plist.5.html>`_ @@ -107,7 +110,7 @@ This module defines the following functions: An :exc:`OverflowError` will be raised for integer values that cannot be represented in (binary) plist files. - .. versionadded: 3.4 + .. versionadded:: 3.4 .. function:: dumps(value, \*, fmt=FMT_XML, sort_keys=True, skipkeys=False) @@ -116,7 +119,7 @@ This module defines the following functions: the documentation for :func:`dump` for an explanation of the keyword arguments of this function. - .. versionadded: 3.4 + .. versionadded:: 3.4 The following functions are deprecated: @@ -135,7 +138,7 @@ The following functions are deprecated: to ``__getitem_``. This means that you can use attribute access to access items of these dictionaries. - .. deprecated: 3.4 Use :func:`load` instead. + .. deprecated:: 3.4 Use :func:`load` instead. .. function:: writePlist(rootObject, pathOrFile) @@ -143,7 +146,7 @@ The following functions are deprecated: Write *rootObject* to an XML plist file. *pathOrFile* may be either a file name or a (writable and binary) file object - .. deprecated: 3.4 Use :func:`dump` instead. + .. deprecated:: 3.4 Use :func:`dump` instead. .. function:: readPlistFromBytes(data) @@ -194,7 +197,7 @@ The following classes are available: .. deprecated:: 3.4 Use a :class:`bytes` object instead -The following constants are avaiable: +The following constants are available: .. data:: FMT_XML diff --git a/Doc/library/pydoc.rst b/Doc/library/pydoc.rst index e100865..3f520e8 100644 --- a/Doc/library/pydoc.rst +++ b/Doc/library/pydoc.rst @@ -84,3 +84,8 @@ Reference Manual pages. .. versionchanged:: 3.2 Added the ``-b`` option, deprecated the ``-g`` option. + +.. versionchanged:: 3.4 + :mod:`pydoc` now uses :func:`inspect.signature` rather than + :func:`inspect.getfullargspec` to extract signature information from + callables. diff --git a/Doc/library/site.rst b/Doc/library/site.rst index d93e938..2fdf303 100644 --- a/Doc/library/site.rst +++ b/Doc/library/site.rst @@ -123,9 +123,13 @@ On systems that support :mod:`readline`, this module will also import and configure the :mod:`rlcompleter` module, if Python is started in :ref:`interactive mode <tut-interactive>` and without the :option:`-S` option. The default behavior is enable tab-completion and to use -:file:`~/.python_history` as the history save file. To disable it, override -the :data:`sys.__interactivehook__` attribute in your :mod:`sitecustomize` -or :mod:`usercustomize` module or your :envvar:`PYTHONSTARTUP` file. +:file:`~/.python_history` as the history save file. To disable it, delete (or +override) the :data:`sys.__interactivehook__` attribute in your +:mod:`sitecustomize` or :mod:`usercustomize` module or your +:envvar:`PYTHONSTARTUP` file. + +.. versionchanged:: 3.4 + Activation of rlcompleter and history was made automatic. Module contents diff --git a/Doc/library/smtplib.rst b/Doc/library/smtplib.rst index eba8ae9..ec8dc9d 100644 --- a/Doc/library/smtplib.rst +++ b/Doc/library/smtplib.rst @@ -117,6 +117,9 @@ A nice selection of exceptions is defined as well: Subclass of :exc:`OSError` that is the base exception class for all the other exceptions provided by this module. + .. versionchanged:: 3.4 + SMTPException became subclass of :exc:`OSError` + .. exception:: SMTPServerDisconnected diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index c2e9f00..54c6bad 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -6,8 +6,7 @@ This module provides access to the BSD *socket* interface. It is available on -all modern Unix systems, Windows, MacOS, OS/2, and probably additional -platforms. +all modern Unix systems, Windows, MacOS, and probably additional platforms. .. note:: @@ -468,7 +467,7 @@ The :mod:`socket` module also offers various network-related services: (10, 1, 6, '', ('2001:888:2000:d::a2', 80, 0, 0))] .. versionchanged:: 3.2 - parameters can now be passed as single keyword arguments. + parameters can now be passed using keyword arguments. .. function:: getfqdn([name]) @@ -634,6 +633,9 @@ The :mod:`socket` module also offers various network-related services: Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. function:: inet_ntop(address_family, packed_ip) @@ -650,6 +652,9 @@ The :mod:`socket` module also offers various network-related services: Availability: Unix (maybe not all platforms), Windows. + .. versionchanged:: 3.4 + Windows support added + .. XXX: Are sendmsg(), recvmsg() and CMSG_*() available on any @@ -794,6 +799,7 @@ to sockets. :keyword:`with` statement around them. .. note:: + :meth:`close()` releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call :meth:`shutdown()` @@ -1512,4 +1518,3 @@ the :data:`SO_REUSEADDR` flag tells the kernel to reuse a local socket in details of socket semantics. For Unix, refer to the manual pages; for Windows, see the WinSock (or Winsock 2) specification. For IPv6-ready APIs, readers may want to refer to :rfc:`3493` titled Basic Socket Interface Extensions for IPv6. - diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 76bb432..0ed5fb2 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -818,6 +818,7 @@ SSL sockets also have the following additional methods and attributes: 'version': 3} .. note:: + To validate a certificate for a particular service, you can use the :func:`match_hostname` function. @@ -841,10 +842,8 @@ SSL sockets also have the following additional methods and attributes: .. versionchanged:: 3.4 :exc:`ValueError` is raised when the handshake isn't done. - - .. versionchanged:: 3.4 The returned dictionary includes additional X509v3 extension items - such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. + such as ``crlDistributionPoints``, ``caIssuers`` and ``OCSP`` URIs. .. method:: SSLSocket.cipher() diff --git a/Doc/library/stringprep.rst b/Doc/library/stringprep.rst index 47144a6..fc890cb 100644 --- a/Doc/library/stringprep.rst +++ b/Doc/library/stringprep.rst @@ -3,7 +3,6 @@ .. module:: stringprep :synopsis: String preparation, as per RFC 3453 - :deprecated: .. moduleauthor:: Martin v. Löwis <martin@v.loewis.de> .. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index d2893b6..cdcbe82 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -629,6 +629,12 @@ Instances of the :class:`Popen` class have the following methods: :exc:`TimeoutExpired` exception. It is safe to catch this exception and retry the wait. + .. note:: + + The function is implemented using a busy loop (non-blocking call and + short sleeps). Use the :mod:`asyncio` module for an asynchronous wait: + see :class:`asyncio.create_subprocess_exec`. + .. warning:: This will deadlock when using ``stdout=PIPE`` and/or @@ -639,6 +645,11 @@ Instances of the :class:`Popen` class have the following methods: .. versionchanged:: 3.3 *timeout* was added. + .. deprecated:: 3.4 + + Do not use the undocumented *endtime* parameter. It is was + unintentionally exposed in 3.3 but was intended to be private + for internal use. Use *timeout* instead. .. method:: Popen.communicate(input=None, timeout=None) @@ -1080,8 +1091,10 @@ handling consistency are valid for these functions. >>> subprocess.getstatusoutput('/bin/junk') (256, 'sh: /bin/junk: not found') - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added .. function:: getoutput(cmd) @@ -1094,8 +1107,10 @@ handling consistency are valid for these functions. >>> subprocess.getoutput('ls /bin/ls') '/bin/ls' - .. versionchanged:: 3.3 - Availability: Unix & Windows + Availability: Unix & Windows + + .. versionchanged:: 3.3.4 + Windows support added Notes diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst index 15c06b5..a94ae08 100644 --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -251,7 +251,7 @@ AU_write objects, as returned by :func:`.open` above, have the following methods Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.writeframes(data) @@ -259,7 +259,7 @@ AU_write objects, as returned by :func:`.open` above, have the following methods Write audio frames and make sure *nframes* is correct. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: AU_write.close() diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 5ff7178..e3cc866 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -227,7 +227,9 @@ always available. installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* is the version number of Python, for example ``3.2``. - .. note:: If a :ref:`virtual environment <venv-def>` is in effect, this + .. note:: + + If a :ref:`virtual environment <venv-def>` is in effect, this value will be changed in ``site.py`` to point to the virtual environment. The value for the Python installation will still be available, via :data:`base_exec_prefix`. @@ -692,10 +694,11 @@ always available. .. data:: __interactivehook__ - When present, this function is automatically called (with no arguments) - when the interpreter is launched in :ref:`interactive mode <tut-interactive>`. - This is done after the :envvar:`PYTHONSTARTUP` file is read, so that you - can set this hook there. + When this attribute exists, its value is automatically called (with no + arguments) when the interpreter is launched in :ref:`interactive mode + <tut-interactive>`. This is done after the :envvar:`PYTHONSTARTUP` file is + read, so that you can set this hook there. The :mod:`site` module + :ref:`sets this <rlcompleter-config>`. .. versionadded:: 3.4 diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 2c51549..83026d8 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -199,6 +199,7 @@ The :mod:`test.support` module provides support for Python's regression test suite. .. note:: + :mod:`test.support` is not a public module. It is documented here to help Python developers write tests. The API of this module is subject to change without backwards compatibility concerns between releases. diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst index 1ba42a3..edf1fd6 100644 --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -40,13 +40,14 @@ functions should be good enough; otherwise, you should use an instance of :func:`wrap`. -.. function:: shorten(text, width=70, *, placeholder=" [...]") +.. function:: shorten(text, width, **kwargs) - Collapse and truncate the given text to fit in the given width. + Collapse and truncate the given *text* to fit in the given *width*. - The text first has its whitespace collapsed. If it then fits in - the *width*, it is returned unchanged. Otherwise, as many words - as possible are joined and then the *placeholder* is appended:: + First the whitespace in *text* is collapsed (all whitespace is replaced by + single spaces). If the result fits in the *width*, it is returned. + Otherwise, enough words are dropped from the end so that the remaining words + plus the :attr:`placeholder` fit within :attr:`width`:: >>> textwrap.shorten("Hello world!", width=12) 'Hello world!' @@ -55,6 +56,12 @@ functions should be good enough; otherwise, you should use an instance of >>> textwrap.shorten("Hello world", width=10, placeholder="...") 'Hello...' + Optional keyword arguments correspond to the instance attributes of + :class:`TextWrapper`, documented below. Note that the whitespace is + collapsed before the text is passed to the :class:`TextWrapper` :meth:`fill` + function, so changing the value of :attr:`.tabsize`, :attr:`.expand_tabs`, + :attr:`.drop_whitespace`, and :attr:`.replace_whitespace` will have no effect. + .. versionadded:: 3.4 @@ -110,8 +117,8 @@ functions should be good enough; otherwise, you should use an instance of :func:`wrap`, :func:`fill` and :func:`shorten` work by creating a :class:`TextWrapper` instance and calling a single method on it. That instance is not reused, so for applications that process many text -strings, it may be more efficient to create your own -:class:`TextWrapper` object. +strings using :func:`wrap` and/or :func:`fill`, it may be more efficient to +create your own :class:`TextWrapper` object. Text is preferably wrapped on whitespaces and right after the hyphens in hyphenated words; only then will long words be broken if necessary, unless @@ -252,16 +259,16 @@ hyphenated words; only then will long words be broken if necessary, unless .. attribute:: max_lines - (default: ``None``) If not ``None``, then the text be will truncated to - *max_lines* lines. + (default: ``None``) If not ``None``, then the output will contain at most + *max_lines* lines, with *placeholder* appearing at the end of the output. .. versionadded:: 3.4 .. attribute:: placeholder - (default: ``' [...]'``) String that will be appended to the last line of - text if it will be truncated. + (default: ``' [...]'``) String that will appear at the end of the output + text if it has been truncated. .. versionadded:: 3.4 diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst index c12ef87..3405518 100644 --- a/Doc/library/tracemalloc.rst +++ b/Doc/library/tracemalloc.rst @@ -184,6 +184,7 @@ Pretty top Code to display the 10 lines allocating the most memory with a pretty output, ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: + import linecache import os import tracemalloc @@ -201,6 +202,9 @@ ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)) + line = linecache.getline(frame.filename, frame.lineno).strip() + if line: + print(' %s' % line) other = top_stats[limit:] if other: @@ -218,19 +222,28 @@ ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: Example of output of the Python test suite:: - 2013-11-08 14:16:58.149320: Top 10 lines - #1: collections/__init__.py:368: 291.9 KiB - #2: Lib/doctest.py:1291: 200.2 KiB - #3: unittest/case.py:571: 160.3 KiB - #4: Lib/abc.py:133: 99.8 KiB - #5: urllib/parse.py:476: 71.8 KiB - #6: <string>:5: 62.7 KiB - #7: Lib/base64.py:140: 59.8 KiB - #8: Lib/_weakrefset.py:37: 51.8 KiB - #9: collections/__init__.py:362: 50.6 KiB - #10: test/test_site.py:56: 48.0 KiB - 7496 other: 4161.9 KiB - Total allocated size: 5258.8 KiB + Top 10 lines + #1: Lib/base64.py:414: 419.8 KiB + _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] + #2: Lib/base64.py:306: 419.8 KiB + _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] + #3: collections/__init__.py:368: 293.6 KiB + exec(class_definition, namespace) + #4: Lib/abc.py:133: 115.2 KiB + cls = super().__new__(mcls, name, bases, namespace) + #5: unittest/case.py:574: 103.1 KiB + testMethod() + #6: Lib/linecache.py:127: 95.4 KiB + lines = fp.readlines() + #7: urllib/parse.py:476: 71.8 KiB + for a in _hexdig for b in _hexdig} + #8: <string>:5: 62.0 KiB + #9: Lib/_weakrefset.py:37: 60.0 KiB + self.data = set() + #10: Lib/base64.py:142: 59.8 KiB + _b32tab2 = [a + b for a in _b32tab for b in _b32tab] + 6220 other: 3602.8 KiB + Total allocated size: 5303.1 KiB See :meth:`Snapshot.statistics` for more options. diff --git a/Doc/library/types.rst b/Doc/library/types.rst index c4f57e4..abdb939 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -15,6 +15,9 @@ It also defines names for some object types that are used by the standard Python interpreter, but not exposed as builtins like :class:`int` or :class:`str` are. +Finally, it provides some additional type-related utility classes and functions +that are not fundamental enough to be builtins. + Dynamic Type Creation --------------------- @@ -220,6 +223,9 @@ Standard names are defined for the following types: Return a new view of the underlying mapping's values. +Additional Utility Classes and Functions +---------------------------------------- + .. class:: SimpleNamespace A simple :class:`object` subclass that provides attribute access to its @@ -246,3 +252,18 @@ Standard names are defined for the following types: instead. .. versionadded:: 3.3 + + +.. function:: DynamicClassAttribute(fget=None, fset=None, fdel=None, doc=None) + + Route attribute access on a class to __getattr__. + + This is a descriptor, used to define attributes that act differently when + accessed through an instance and through a class. Instance access remains + normal, but access to an attribute through a class will be routed to the + class's __getattr__ method; this is done by raising AttributeError. + + This allows one to have properties active on an instance, and have virtual + attributes on the class with the same name (see Enum for an example). + + .. versionadded:: 3.4 diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst index 80386d2..20830e2 100644 --- a/Doc/library/undoc.rst +++ b/Doc/library/undoc.rst @@ -20,7 +20,7 @@ These modules are used to implement the :mod:`os.path` module, and are not documented beyond this mention. There's little need to document these. :mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32, Win64, WinCE, and OS/2 platforms. + --- Implementation of :mod:`os.path` on Win32, Win64, and WinCE platforms. :mod:`posixpath` --- Implementation of :mod:`os.path` on POSIX. diff --git a/Doc/library/unittest.mock-examples.rst b/Doc/library/unittest.mock-examples.rst index 98bfcc5..c5fa365 100644 --- a/Doc/library/unittest.mock-examples.rst +++ b/Doc/library/unittest.mock-examples.rst @@ -426,7 +426,7 @@ mock using the "as" form of the with statement: As an alternative `patch`, `patch.object` and `patch.dict` can be used as class decorators. When used in this way it is the same as applying the -decorator indvidually to every method whose name starts with "test". +decorator individually to every method whose name starts with "test". .. _further-examples: diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index b5cf7b5..cb72a68 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -958,7 +958,7 @@ method: .. [#] The only exceptions are magic methods and attributes (those that have leading and trailing double underscores). Mock doesn't create these but - instead of raises an ``AttributeError``. This is because the interpreter + instead raises an ``AttributeError``. This is because the interpreter will often implicitly request these methods, and gets *very* confused to get a new Mock object when it expects a magic method. If you need magic method support see :ref:`magic methods <magic-methods>`. @@ -1509,7 +1509,7 @@ Patching Descriptors and Proxy Objects Both patch_ and patch.object_ correctly patch and restore descriptors: class methods, static methods and properties. You should patch these on the *class* rather than an instance. They also work with *some* objects -that proxy attribute access, like the `django setttings object +that proxy attribute access, like the `django settings object <http://www.voidspace.org.uk/python/weblog/arch_d7_2010_12_04.shtml#e1198>`_. diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 6ec3609..690c98b 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -239,9 +239,10 @@ Test Discovery Unittest supports simple test discovery. In order to be compatible with test discovery, all of the test files must be :ref:`modules <tut-modules>` or -:ref:`packages <tut-packages>` importable from the top-level directory of -the project (this means that their filenames must be valid -:ref:`identifiers <identifiers>`). +:ref:`packages <tut-packages>` (including :term:`namespace packages +<namespace package>`) importable from the top-level directory of +the project (this means that their filenames must be valid :ref:`identifiers +<identifiers>`). Test discovery is implemented in :meth:`TestLoader.discover`, but can also be used from the command line. The basic command-line usage is:: @@ -306,6 +307,9 @@ as the start directory. Test modules and packages can customize test loading and discovery by through the `load_tests protocol`_. +.. versionchanged:: 3.4 + Test discovery supports :term:`namespace packages <namespace package>`. + .. _organizing-tests: @@ -1620,11 +1624,11 @@ Loading and running tests .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) - Find and return all test modules from the specified start directory, - recursing into subdirectories to find them. Only test files that match - *pattern* will be loaded. (Using shell style pattern matching.) Only - module names that are importable (i.e. are valid Python identifiers) will - be loaded. + Find all the test modules by recursing into subdirectories from the + specified start directory, and return a TestSuite object containing them. + Only test files that match *pattern* will be loaded. (Using shell style + pattern matching.) Only module names that are importable (i.e. are valid + Python identifiers) will be loaded. All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level @@ -1654,12 +1658,11 @@ Loading and running tests .. versionchanged:: 3.4 Modules that raise :exc:`SkipTest` on import are recorded as skips, - not errors. - - .. versionchanged:: 3.4 - Paths are sorted before being imported to ensure execution order for a - given test suite is the same even if the underlying file system's ordering - is not dependent on file name like in ext3/4. + not errors. + Discovery works for :term:`namespace packages <namespace package>`. + Paths are sorted before being imported so that execution order is + the same even if the underlying file system's ordering is not + dependent on file name. The following attributes of a :class:`TestLoader` can be configured either by diff --git a/Doc/library/urllib.error.rst b/Doc/library/urllib.error.rst index 9fb58f5..f7f0c97 100644 --- a/Doc/library/urllib.error.rst +++ b/Doc/library/urllib.error.rst @@ -48,7 +48,7 @@ The following exceptions are raised by :mod:`urllib.error` as appropriate: .. attribute:: headers - The HTTP response headers for the HTTP request that cause the + The HTTP response headers for the HTTP request that caused the :exc:`HTTPError`. .. versionadded:: 3.4 diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 20449d9..6ac9081 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -218,7 +218,7 @@ The following classes are provided: fetching of the image, this should be true. *method* should be a string that indicates the HTTP request method that - will be used (e.g. ``'HEAD'``). Its value is stored in the + will be used (e.g. ``'HEAD'``). If provided, its value is stored in the :attr:`~Request.method` attribute and is used by :meth:`get_method()`. Subclasses may indicate a default method by setting the :attr:`~Request.method` attribute in the class itself. @@ -440,13 +440,20 @@ request. .. attribute:: Request.method - The HTTP request method to use. This value is used by - :meth:`~Request.get_method` to override the computed HTTP request - method that would otherwise be returned. This attribute is initialized with - the value of the *method* argument passed to the constructor. + The HTTP request method to use. By default its value is :const:`None`, + which means that :meth:`~Request.get_method` will do its normal computation + of the method to be used. Its value can be set (thus overriding the default + computation in :meth:`~Request.get_method`) either by providing a default + value by setting it at the class level in a :class:`Request` subclass, or by + passing a value in to the :class:`Request` constructor via the *method* + argument. .. versionadded:: 3.3 + .. versionchanged:: 3.4 + A default value can now be set in subclasses; previously it could only + be set via the constructor argument. + .. method:: Request.get_method() diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index d29a957..e93f48e 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -76,6 +76,8 @@ Creating virtual environments without there needing to be any reference to its venv in ``PATH``. +.. _venv-api: + API --- diff --git a/Doc/library/wave.rst b/Doc/library/wave.rst index c32e1fc..ab64978 100644 --- a/Doc/library/wave.rst +++ b/Doc/library/wave.rst @@ -150,14 +150,30 @@ them, and is otherwise implementation dependent. Wave_write Objects ------------------ +For seekable output streams, the ``wave`` header will automatically be updated +to reflect the number of frames actually written. For unseekable streams, the +*nframes* value must be accurate when the first frame data is written. An +accurate *nframes* value can be achieved either by calling +:meth:`~Wave_write.setnframes` or :meth:`~Wave_write.setparams` with the number +of frames that will be written before :meth:`~Wave_write.close` is called and +then using :meth:`~Wave_write.writeframesraw` to write the frame data, or by +calling :meth:`~Wave_write.writeframes` with all of the frame data to be +written. In the latter case :meth:`~Wave_write.writeframes` will calculate +the number of frames in the data and set *nframes* accordingly before writing +the frame data. + Wave_write objects, as returned by :func:`.open`, have the following methods: +.. versionchanged:: 3.4 + Added support for unseekable files. + .. method:: Wave_write.close() Make sure *nframes* is correct, and close the file if it was opened by - :mod:`wave`. This method is called upon object collection. Can raise an - exception if *nframes* is not correct and a file is not seekable. + :mod:`wave`. This method is called upon object collection. It will raise + an exception if the output stream is not seekable and *nframes* does not + match the number of frames actually written. .. method:: Wave_write.setnchannels(n) @@ -181,8 +197,9 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: .. method:: Wave_write.setnframes(n) - Set the number of frames to *n*. This will be changed later if more frames are - written. + Set the number of frames to *n*. This will be changed later if the number + of frames actually written is different (this update attempt will + raise an error if the output stream is not seekable). .. method:: Wave_write.setcomptype(type, name) @@ -209,16 +226,18 @@ Wave_write objects, as returned by :func:`.open`, have the following methods: Write audio frames, without correcting *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. .. method:: Wave_write.writeframes(data) - Write audio frames and make sure *nframes* is correct. Can raise an - exception if a file is not seekable. + Write audio frames and make sure *nframes* is correct. It will raise an + error if the output stream is not seekable and the total number of frames + that have been written after *data* has been written does not match the + previously set value for *nframes*. .. versionchanged:: 3.4 - Any :term:`bytes-like object`\ s are now accepted. + Any :term:`bytes-like object` is now accepted. Note that it is invalid to set any parameters after calling :meth:`writeframes` diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst index dfa2b71..c6dbce0 100644 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -952,7 +952,8 @@ XMLParser Objects specified in the XML file. .. deprecated:: 3.4 - The *html* argument. + The *html* argument. The remaining arguments should be passed via + keywword to prepare for the removal of the *html* argument. .. method:: close() diff --git a/Doc/library/xml.rst b/Doc/library/xml.rst index f793bae..0188219 100644 --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@ -14,9 +14,9 @@ Python's interfaces for processing XML are grouped in the ``xml`` package. .. warning:: The XML modules are not secure against erroneous or maliciously - constructed data. If you need to parse untrusted or unauthenticated data see - :ref:`xml-vulnerabilities`. - + constructed data. If you need to parse untrusted or + unauthenticated data see the :ref:`xml-vulnerabilities` and + :ref:`defused-packages` sections. It is important to note that modules in the :mod:`xml` package require that there be at least one SAX-compliant XML parser available. The Expat parser is @@ -46,16 +46,15 @@ The XML handling submodules are: .. _xml-vulnerabilities: XML vulnerabilities -=================== +------------------- The XML processing modules are not secure against maliciously constructed data. -An attacker can abuse vulnerabilities for e.g. denial of service attacks, to -access local files, to generate network connections to other machines, or -to or circumvent firewalls. The attacks on XML abuse unfamiliar features -like inline `DTD`_ (document type definition) with entities. +An attacker can abuse XML features to carry out denial of service attacks, +access local files, generate network connections to other machines, or +circumvent firewalls. -The following table gives an overview of the known attacks and if the various -modules are vulnerable to them. +The following table gives an overview of the known attacks and whether +the various modules are vulnerable to them. ========================= ======== ========= ========= ======== ========= kind sax etree minidom pulldom xmlrpc @@ -68,7 +67,7 @@ decompression bomb No No No No **Yes** ========================= ======== ========= ========= ======== ========= 1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a - ParserError when an entity occurs. + :exc:`ParserError` when an entity occurs. 2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns the unexpanded entity verbatim. 3. :mod:`xmlrpclib` doesn't expand external entities and omits them. @@ -77,23 +76,21 @@ decompression bomb No No No No **Yes** billion laughs / exponential entity expansion The `Billion Laughs`_ attack -- also known as exponential entity expansion -- uses multiple levels of nested entities. Each entity refers to another entity - several times, the final entity definition contains a small string. Eventually - the small string is expanded to several gigabytes. The exponential expansion - consumes lots of CPU time, too. + several times, and the final entity definition contains a small string. + The exponential expansion results in several gigabytes of text and + consumes lots of memory and CPU time. quadratic blowup entity expansion A quadratic blowup attack is similar to a `Billion Laughs`_ attack; it abuses entity expansion, too. Instead of nested entities it repeats one large entity with a couple of thousand chars over and over again. The attack isn't as - efficient as the exponential case but it avoids triggering countermeasures of - parsers against heavily nested entities. + efficient as the exponential case but it avoids triggering parser countermeasures + that forbid deeply-nested entities. external entity expansion Entity declarations can contain more than just text for replacement. They can - also point to external resources by public identifiers or system identifiers. - System identifiers are standard URIs or can refer to local files. The XML - parser retrieves the resource with e.g. HTTP or FTP requests and embeds the - content into the XML document. + also point to external resources or local files. The XML + parser accesses the resource and embeds the content into the XML document. DTD retrieval Some XML libraries like Python's :mod:`xml.dom.pulldom` retrieve document type @@ -101,31 +98,32 @@ DTD retrieval implications as the external entity expansion issue. decompression bomb - The issue of decompression bombs (aka `ZIP bomb`_) apply to all XML libraries - that can parse compressed XML stream like gzipped HTTP streams or LZMA-ed + Decompression bombs (aka `ZIP bomb`_) apply to all XML libraries + that can parse compressed XML streams such as gzipped HTTP streams or + LZMA-compressed files. For an attacker it can reduce the amount of transmitted data by three magnitudes or more. -The documentation of `defusedxml`_ on PyPI has further information about +The documentation for `defusedxml`_ on PyPI has further information about all known attack vectors with examples and references. -defused packages ----------------- +.. _defused-packages: -`defusedxml`_ is a pure Python package with modified subclasses of all stdlib -XML parsers that prevent any potentially malicious operation. The courses of -action are recommended for any server code that parses untrusted XML data. The -package also ships with example exploits and an extended documentation on more -XML exploits like xpath injection. +The :mod:`defusedxml` and :mod:`defusedexpat` Packages +------------------------------------------------------ -`defusedexpat`_ provides a modified libexpat and patched replacment -:mod:`pyexpat` extension module with countermeasures against entity expansion -DoS attacks. Defusedexpat still allows a sane and configurable amount of entity -expansions. The modifications will be merged into future releases of Python. +`defusedxml`_ is a pure Python package with modified subclasses of all stdlib +XML parsers that prevent any potentially malicious operation. Use of this +package is recommended for any server code that parses untrusted XML data. The +package also ships with example exploits and extended documentation on more +XML exploits such as XPath injection. -The workarounds and modifications are not included in patch releases as they -break backward compatibility. After all inline DTD and entity expansion are -well-definied XML features. +`defusedexpat`_ provides a modified libexpat and a patched +:mod:`pyexpat` module that have countermeasures against entity expansion +DoS attacks. The :mod:`defusedexpat` module still allows a sane and configurable amount of entity +expansions. The modifications may be included in some future release of Python, +but will not be included in any bugfix releases of +Python because they break backward compatibility. .. _defusedxml: https://pypi.python.org/pypi/defusedxml/ @@ -133,4 +131,3 @@ well-definied XML features. .. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs .. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb .. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition - diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index 969a536..1d23a7c 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -401,18 +401,32 @@ The :class:`PyZipFile` constructor takes the same parameters as the ``2``, only files with that optimization level (see :func:`compile`) are added to the archive, compiling if necessary. - If the pathname is a file, the filename must end with :file:`.py`, and + If *pathname* is a file, the filename must end with :file:`.py`, and just the (corresponding :file:`\*.py[co]`) file is added at the top level - (no path information). If the pathname is a file that does not end with + (no path information). If *pathname* is a file that does not end with :file:`.py`, a :exc:`RuntimeError` will be raised. If it is a directory, and the directory is not a package directory, then all the files :file:`\*.py[co]` are added at the top level. If the directory is a package directory, then all :file:`\*.py[co]` are added under the package name as a file path, and if any subdirectories are package directories, - all of these are added recursively. *basename* is intended for internal - use only. When *filterfunc(pathname)* is given, it will be called for every - invocation. When it returns a false value, that path and its subpaths will - be ignored. + all of these are added recursively. + + *basename* is intended for internal use only. + + *filterfunc*, if given, must be a function taking a single string + argument. It will be passed each path (including each individual full + file path) before it is added to the archive. If *filterfunc* returns a + false value, the path will not be added, and if it is a directory its + contents will be ignored. For example, if our test files are all either + in ``test`` directories or start with the string ``test_``, we can use a + *filterfunc* to exclude them:: + + >>> zf = PyZipFile('myprog.zip') + >>> def notests(s): + ... fn = os.path.basename(s) + ... return (not (fn == 'test' or fn.startswith('test_'))) + >>> zf.writepy('myprog', filterfunc=notests) + The :meth:`writepy` method makes archives with file names like this:: diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 4f19b37..8204dc3 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1226,6 +1226,10 @@ Basic customization The return value must be a string object. + .. versionchanged:: 3.4 + The __format__ method of ``object`` itself raises a :exc:`TypeError` + if passed any non-empty string. + .. _richcmpfuncs: .. method:: object.__lt__(self, other) @@ -1643,6 +1647,8 @@ of these candidate metaclasses. If none of the candidate metaclasses meets that criterion, then the class definition will fail with ``TypeError``. +.. _prepare: + Preparing the class namespace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -2044,11 +2050,13 @@ left undefined. ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation in-place (modifying *self*) and return the result (which could be, but does not have to be, *self*). If a specific method is not defined, the augmented - assignment falls back to the normal methods. For instance, to execute the - statement ``x += y``, where *x* is an instance of a class that has an - :meth:`__iadd__` method, ``x.__iadd__(y)`` is called. If *x* is an instance - of a class that does not define a :meth:`__iadd__` method, ``x.__add__(y)`` - and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. + assignment falls back to the normal methods. For instance, if *x* is an + instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent + to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` + are considered, as with the evaluation of ``x + y``. In certain situations, + augmented assignment can result in unexpected errors (see + :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in + fact part of the data model. .. method:: object.__neg__(self) diff --git a/Doc/tools/sphinxext/indexcontent.html b/Doc/tools/sphinxext/indexcontent.html index 7f85470..969099a 100644 --- a/Doc/tools/sphinxext/indexcontent.html +++ b/Doc/tools/sphinxext/indexcontent.html @@ -16,14 +16,14 @@ <p class="biglink"><a class="biglink" href="{{ pathto("howto/index") }}">Python HOWTOs</a><br/> <span class="linkdescr">in-depth documents on specific topics</span></p> </td><td width="50%"> + <p class="biglink"><a class="biglink" href="{{ pathto("installing/index") }}">Installing Python Modules</a><br/> + <span class="linkdescr">installing from the Python Package Index & other sources</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("distributing/index") }}">Distributing Python Modules</a><br/> + <span class="linkdescr">publishing modules for installation by others</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("extending/index") }}">Extending and Embedding</a><br/> <span class="linkdescr">tutorial for C/C++ programmers</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("c-api/index") }}">Python/C API</a><br/> <span class="linkdescr">reference for C/C++ programmers</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("install/index") }}">Installing Python Modules</a><br/> - <span class="linkdescr">information for installers & sys-admins</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("distutils/index") }}">Distributing Python Modules</a><br/> - <span class="linkdescr">sharing modules with others</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("faq/index") }}">FAQs</a><br/> <span class="linkdescr">frequently asked questions (with answers!)</span></p> </td></tr> diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv index 1769023..f9d6bbf 100644 --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -282,3 +282,4 @@ whatsnew/changelog,,:PythonCmd,"With Tk < 8.5 _tkinter.c:PythonCmd() raised Unic whatsnew/changelog,,::,": Fix FTP tests for IPv6, bind to ""::1"" instead of ""localhost""." whatsnew/changelog,,::,": Use ""127.0.0.1"" or ""::1"" instead of ""localhost"" as much as" whatsnew/changelog,,:password,user:password +whatsnew/changelog,,:gz,w:gz diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 1225e20..9efd1ac 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -371,9 +371,9 @@ values. The most versatile is the *list*, which can be written as a list of comma-separated values (items) between square brackets. Lists might contain items of different types, but usually the items all have the same type. :: - >>> squares = [1, 2, 4, 9, 16, 25] + >>> squares = [1, 4, 9, 16, 25] >>> squares - [1, 2, 4, 9, 16, 25] + [1, 4, 9, 16, 25] Like strings (and all other built-in :term:`sequence` type), lists can be indexed and sliced:: @@ -389,12 +389,12 @@ All slice operations return a new list containing the requested elements. This means that the following slice returns a new (shallow) copy of the list:: >>> squares[:] - [1, 2, 4, 9, 16, 25] + [1, 4, 9, 16, 25] Lists also supports operations like concatenation:: >>> squares + [36, 49, 64, 81, 100] - [1, 2, 4, 9, 16, 25, 36, 49, 64, 81, 100] + [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] Unlike strings, which are :term:`immutable`, lists are a :term:`mutable` type, i.e. it is possible to change their content:: diff --git a/Doc/tutorial/whatnow.rst b/Doc/tutorial/whatnow.rst index 7fcbdc3f..979f587 100644 --- a/Doc/tutorial/whatnow.rst +++ b/Doc/tutorial/whatnow.rst @@ -21,8 +21,8 @@ the set are: and many other tasks. Skimming through the Library Reference will give you an idea of what's available. -* :ref:`install-index` explains how to install external modules written by other - Python users. +* :ref:`installing-index` explains how to install additional modules written + by other Python users. * :ref:`reference-index`: A detailed explanation of Python's syntax and semantics. It's heavy reading, but is useful as a complete guide to the diff --git a/Doc/using/unix.rst b/Doc/using/unix.rst index 40635c6..3c7fc41 100644 --- a/Doc/using/unix.rst +++ b/Doc/using/unix.rst @@ -60,6 +60,8 @@ To install the newest Python versions on OpenSolaris, install `blastwave prompt. +.. _building-python-on-unix: + Building Python =============== diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index 997e24b..52cdda0 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -11,6 +11,11 @@ containing a copy of the ``python`` binary (or binaries, in the case of Windows). It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` subdirectory (on Windows, this is ``Lib\site-packages``). +.. seealso:: + + `Python Packaging User Guide: Creating and using virtual environments + <http://packaging.python.org/en/latest/tutorial.html#creating-and-using-virtual-environments>`__ + .. highlight:: none On Windows, you may have to invoke the ``pyvenv`` script as follows, if you @@ -38,6 +43,8 @@ The command, if run with ``-h``, will show the available options:: virtual environment. --symlinks Try to use symlinks rather than copies, when symlinks are not the default for the platform. + --copies Try to use copies rather than symlinks, even when + symlinks are the default for the platform. --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. @@ -47,7 +54,8 @@ The command, if run with ``-h``, will show the available options:: environment (pip is bootstrapped by default) .. versionchanged:: 3.4 - Installs pip by default, added the ``--without-pip`` option + Installs pip by default, added the ``--without-pip`` and ``--copies`` + options If the target directory already exists an error will be raised, unless the ``--clear`` or ``--upgrade`` option was provided. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index b1e7898..42a92af 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -11,6 +11,10 @@ This document aims to give an overview of Windows-specific behaviour you should know about when using Python on Microsoft Windows. +.. XXX (ncoghlan) + + This looks rather stale to me... + Installing Python ================= diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 939d491..9cf51bd 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -2,8 +2,7 @@ What's New In Python 3.4 **************************** -.. :Author: Someone <email> - (uncomment if there is a principal author) +:Author: R. David Murray <rdmurray@bitdance.com> (Editor) .. Rules for maintenance: @@ -68,11 +67,6 @@ This article explains the new features in Python 3.4, compared to 3.3. For full details, see the `changelog <http://docs.python.org/3.4/whatsnew/changelog.html>`_. -.. note:: Prerelease users should be aware that this document is currently in - draft form. While it should be close to complete for the Python 3.4 - release candidates, adjustments and additions to the document may be made - up until the final release. - .. seealso:: @@ -90,17 +84,19 @@ New syntax features: * No new syntax features were added in Python 3.4. -New expected features for Python implementations: +Other new features: -* :ref:`pip should always be "available" <whatsnew-pep-453>` (:pep:`453`). -* :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` +* :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`). +* :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>` (:pep:`446`). -* command line option for :ref:`isolated mode <using-on-misc-options>`, +* command line option for :ref:`isolated mode <whatsnew-isolated-mode>` (:issue:`16499`). * :ref:`improvements in the handling of codecs <codec-handling-improvements>` that are not text encodings (multiple issues). * :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System (:pep:`451`). (Affects importer authors.) +* The :mod:`marshal` format has been made :ref:`more compact and efficient + <whatsnew-marshal-3>` (:issue:`16475`). New library modules: @@ -120,33 +116,57 @@ New library modules: * :mod:`tracemalloc`: :ref:`Trace Python memory allocations <whatsnew-tracemalloc>` (:pep:`454`). -Significantly Improved Library Modules: +Significantly improved library modules: * :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in :mod:`functools` (:pep:`443`). * New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`). -* :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl` - (:issue:`16692`). * :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`). * :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and a new :mod:`~email.message.Message` subclass (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME handling <whatsnew_email_contentmanager>` (:issue:`18891`). +* The :mod:`inspect` and :mod:`pydoc` modules are now capable of + correct introspection of a much wider variety of callable objects, + which improves the output of the Python :func:`help` system. * The :mod:`ipaddress` module API has been declared stable +Security improvements: + +* :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` + (:pep:`456`). +* :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` + (:pep:`446`) to avoid leaking file descriptors to child processes. +* New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`, + (:issue:`16499`). +* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork + on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are + more secure because they avoid sharing data with child processes. +* :mod:`multiprocessing` child processes on Windows no longer inherit + all of the parent's inheritable handles, only the necessary ones. +* A new :func:`hashlib.pbkdf2_hmac` function provides + the `PKCS#5 password-based key derivation function 2 + <http://en.wikipedia.org/wiki/PBKDF2>`_. +* :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. +* :ref:`Retrieving certificates from the Windows system cert store support + <whatsnew34-win-cert-store>` for :mod:`ssl`. +* :ref:`Server-side SNI (Server Name Indication) support + <whatsnew34-sni>` for :mod:`ssl`. +* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements + <whatsnew34-sslcontext>`. +* All modules in the standard library that support SSL now support server + certificate verification, including hostname matching + (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see + :func:`ssl.SSLContext.load_verify_locations`). CPython implementation improvements: * :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`). -* Leveraging :pep:`442`, :ref:`module globals are no longer set to None - during finalization <whatsnew-pep-442>`, in most cases (:issue:`18214`). +* Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set + to None during finalization <whatsnew-pep-442>` (:issue:`18214`). * :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`). -* :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` - (:pep:`456`). * :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`). -* The :mod:`marshal` format has been made :ref:`more compact and efficient - <whatsnew-marshal-3>` (:issue:`16475`). Please read on for a comprehensive list of user-facing changes, including many other smaller improvements, CPython optimizations, deprecations, and potential @@ -154,38 +174,54 @@ porting issues. -New Expected Features for Python Implementations -================================================ +New Features +============ .. _whatsnew-pep-453: PEP 453: Explicit Bootstrapping of PIP in Python Installations -------------------------------------------------------------- +Bootstrapping pip By Default +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard cross-platform mechanism to bootstrap the pip installer into Python -installations and virtual environments. - -By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where -X.Y stands for the version of the Python installation), along with the -``pip`` Python package and its dependencies. - -The :mod:`venv` module and the :command:`pyvenv` utility make use of this -module to make ``pip`` readily available in virtual environments. When -using the command line interface, ``pip`` is installed by default, while -for the module API installation of ``pip`` must be requested explicitly. - -For CPython source builds on POSIX systems, the ``make install`` and -``make altinstall`` commands bootstrap ``pip`` by default. This behaviour -can be controlled through configure options, and overridden through -Makefile options. - -On Windows and Mac OS X, the CPython installers now offer the option to -install ``pip`` along with CPython itself. +installations and virtual environments. The version of ``pip`` included +with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases +will update the bundled version to the latest version of ``pip`` that is +available at the time of creating the release candidate. + +By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all +platforms (where X.Y stands for the version of the Python installation), +along with the ``pip`` Python package and its dependencies. On Windows and +in virtual environments on all platforms, the unversioned ``pip`` command +will also be installed. On other platforms, the system wide unversioned +``pip`` command typically refers to the separately installed Python 2 +version. + +The :ref:`pyvenv <scripts-pyvenv>` command line utility and the :mod:`venv` +module make use of the :mod:`ensurepip` module to make ``pip`` readily +available in virtual environments. When using the command line utility, +``pip`` is installed by default, while when using the :mod:`venv` module +:ref:`venv-api` installation of ``pip`` must be requested explicitly. + +For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`, +the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by +default. This behaviour can be controlled through configure options, and +overridden through Makefile options. + +On Windows and Mac OS X, the CPython installers now default to installing +``pip`` along with CPython itself (users may opt out of installing it +during the installation process). Window users will need to opt in to the +automatic ``PATH`` modifications to have ``pip`` available from the command +line by default, otherwise it can still be accessed through the Python +launcher for Windows as ``py -m pip``. As `discussed in the PEP`__, platform packagers may choose not to install -``pip`` by default, as long as the command ``pip``, when invoked, provides -clear and simple directions on how to install ``pip`` on the platform. +these commands by default, as long as, when invoked, they provide clear and +simple directions on how to install them on that platform (usually using +the system package manager). __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors @@ -193,12 +229,29 @@ __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distr To avoid conflicts between parallel Python 2 and Python 3 installations, only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by - default when ``ensurepip`` is invoked directly (including by the CPython - installers). ``pyvenv`` ensures that the unqualified ``pip`` command is - made available in virtual environments, and ``pip`` can always be + default when ``ensurepip`` is invoked directly - the ``--default-pip`` + option is needed to also request the unversioned ``pip`` command. + ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` + command is made available in those environments, and ``pip`` can always be invoked via the ``-m`` switch rather than directly to avoid ambiguity on systems with multiple Python installations. + +Documentation Changes +~~~~~~~~~~~~~~~~~~~~~ + +As part of this change, the :ref:`installing-index` and +:ref:`distributing-index` sections of the documentation have been +completely redesigned as short getting started and FAQ documents. Most +packaging documentation has now been moved out to the Python Packaging +Authority maintained `Python Packaging User Guide +<http://packaging.python.org>`__ and the documentation of the individual +projects. + +However, as this migration is currently still incomplete, the legacy +versions of those guides remaining available as :ref:`install-index` +and :ref:`distutils-index`. + .. seealso:: :pep:`453` -- Explicit bootstrapping of pip in Python installations @@ -208,11 +261,17 @@ __ http://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distr .. _whatsnew-pep-446: -PEP 446: Make Newly Created File Descriptors Non-Inheritable ------------------------------------------------------------- +PEP 446: Newly Created File Descriptors Are Non-Inheritable +----------------------------------------------------------- :pep:`446` makes newly created file descriptors :ref:`non-inheritable -<fd_inheritance>`. New functions and methods: +<fd_inheritance>`. In general, this is the behavior an application will +want: when launching a new process, having currently open files also +open in the new process can lead to all sorts of hard to find bugs, +and potentially to security issues. + +However, there are occasions when inheritance is desired. To support +these cases, the following new functions and methods are available: * :func:`os.get_inheritable`, :func:`os.set_inheritable` * :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` @@ -243,10 +302,10 @@ module (and have been covered by the regression test suite) since Python 2.4, but were previously only discoverable through runtime introspection. Unlike the convenience methods on :class:`str`, :class:`bytes` and -:class:`bytearray`, these convenience functions support arbitrary codecs -in both Python 2 and Python 3, rather than being limited to Unicode text -encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions -(in Python 2). +:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary +codecs in both Python 2 and Python 3, rather than being limited to Unicode text +encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in +Python 2). In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these @@ -269,7 +328,7 @@ general purpose convenience functions when appropriate:: In a related change, whenever it is feasible without breaking backwards compatibility, exceptions raised during encoding and decoding operations -will be wrapped in a chained exception of the same type that mentions the +are wrapped in a chained exception of the same type that mentions the name of the codec responsible for producing the error:: >>> import codecs @@ -315,7 +374,7 @@ as:: The binary and text transforms provided in the standard library are detailed in :ref:`binary-transforms` and :ref:`text-transforms`. -(Contributed by Nick Coghlan in :issue:`7475`, , :issue:`17827`, +(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, :issue:`17828` and :issue:`19619`) @@ -336,7 +395,9 @@ The public-facing changes from the PEP are entirely backward-compatible. Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing -importers should be updated to implement the new methods. +importers should be updated to implement the new methods. See the +:ref:`deprecated-3.4` section for a list of methods that should be replaced and +their replacements. Other Language Changes @@ -346,9 +407,10 @@ Some smaller changes made to the core Python language are: * Unicode database updated to UCD version 6.3. -* :func:`min` and :func:`max` now accept a *default* argument that can be used - to specify the value they return if the iterable they are evaluating has no - elements. Contributed by Julian Berman in :issue:`18111`. +* :func:`min` and :func:`max` now accept a *default* keyword-only argument that + can be used to specify the value they return if the iterable they are + evaluating has no elements. (Contributed by Julian Berman in + :issue:`18111`.) * Module objects are now :mod:`weakref`'able. @@ -357,16 +419,42 @@ Some smaller changes made to the core Python language are: ``__main__.__file__`` when a script has been executed directly using a relative path (Contributed by Brett Cannon in :issue:`18416`). -* Now all the UTF-\* codecs (except UTF-7) reject surrogates during both +* All the UTF-\* codecs (except UTF-7) now reject surrogates during both encoding and decoding unless the ``surrogatepass`` error handler is used, - with the exception of the UTF-16 decoder that accepts valid surrogate pairs, - and the UTF-16 encoder that produces them while encoding non-BMP characters. + with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) + and the UTF-16 encoder (which produces them while encoding non-BMP characters). Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in :issue:`12892`. -* New EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed by - Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) +* New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed + by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) + +* New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by + Serhiy Storchaka in :issue:`19668`.) + +* :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary + buffer objects as arguments. (Contributed by Antoine Pitrou in + :issue:`15958`.) +* The :class:`int` constructor now accepts any object that has an ``__index__`` + method for its *base* argument. (Contributed by Mark Dickinson in + :issue:`16772`.) + +* Frame objects now have a :func:`~frame.clear` method that clears all + references to local variables from the frame. (Contributed by Antoine Pitrou + in :issue:`17934`.) + +* :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`, + and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan + and Claudiu Popa in :issue:`18690` and :issue:`19078`.) + +* Signatures reported by :func:`help` have been modified and improved in + several cases as a result of the introduction of Argument Clinic and other + changes to the :mod:`inspect` and :mod:`pydoc` modules. + +* :meth:`~object.__length_hint__` is now part of the formal language + specification (see :pep:`424`). (Contributed by Armin Ronacher in + :issue:`16148`.) New Modules @@ -405,9 +493,11 @@ environment was declined. :mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ``ensurepip`` does not -access the internet. (If the installation has Internet access, it is of course -possible to upgrade ``pip`` to a release more recent than the bundled ``pip`` -by using the bundled ``pip`` command itself once it is installed.) +access the internet. If the installation has Internet access, after +``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a +more recent release than the bundled one. (Note that such an upgraded version +of ``pip`` is considered to be a separately installed package and will not be +removed if Python is uninstalled.) The module is named *ensure*\ pip because if called when ``pip`` is already installed, it does nothing. It also has an ``--upgrade`` option that will @@ -486,7 +576,7 @@ tracemalloc The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to trace memory blocks allocated by Python. It provides the following information: -* Traceback where an object was allocated +* Trace where an object was allocated * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Compute the differences between two snapshots to detect memory leaks @@ -518,7 +608,7 @@ Using ``ABC`` as a base class has essentially the same effect as specifying aifc ---- -The :meth:`~aifc.getparams` method now returns a namedtuple rather than a +The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) :func:`aifc.open` now supports the context manager protocol: when used in a @@ -526,6 +616,10 @@ plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in :issue:`16486`.) +The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` +methods now accept any :term:`bytes-like object`. (Contributed by Serhiy +Storchaka in :issue:`8311`.) + argparse -------- @@ -538,10 +632,16 @@ by Lucas Maystre in :issue:`11175`.) audioop ------- -Added support for 24-bit samples (:issue:`12866`). +:mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka +in :issue:`12866`.) -Added the :func:`~audioop.byteswap` function to convert big-endian samples -to little-endian and vice versa (:issue:`19641`). +New :func:`~audioop.byteswap` function converts big-endian samples to +little-endian and vice versa (Contributed by Serhiy Storchaka in +:issue:`19641`). + +All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings +are not accepted: they didn't work before, now they raise an error right away. +(Contributed by Serhiy Storchaka in :issue:`16685`.) base64 @@ -549,7 +649,25 @@ base64 The encoding and decoding functions in :mod:`base64` now accept any :term:`bytes-like object` in cases where it previously required a -:class:`bytes` or :class:`bytearray` instance (:issue:`17839`). +:class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in +:issue:`17839`.) + +New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, +:func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to +encode and decode binary data from and to ``Ascii85`` and the git/mercurial +``Base85`` formats, respectively. The ``a85`` functions have options that can +be used to make them compatible with the variants of the ``Ascii85`` encoding, +including the Adobe variant. (Contributed by Martin Morrison, the Mercurial +project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) + + +collections +----------- + +The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying +the child map to add to the chain. This allows an existing mapping and/or a +custom mapping type to be used for the child. (Contributed by Vinay Sajip in +:issue:`16613`.) colorsys @@ -570,17 +688,30 @@ statement. (Contributed by Raymond Hettinger in :issue:`15806` and Zero Piraeus in :issue:`19266`) The new :func:`contextlib.redirect_stdout` context manager makes it easier -for utility scripts to handle inflexible APIs that don't provide any -options to retrieve their output as a string or direct it to somewhere -other than :data:`sys.stdout`. In conjunction with :class:`io.StringIO`, -this context manager is also useful for checking expected output from -command line utilities. (Contribute by Raymond Hettinger in :issue:`15805`) +for utility scripts to handle inflexible APIs that write their output to +:data:`sys.stdout` and don't provide any options to redirect it. Using the +context manager, the :data:`sys.stdout` output can be redirected to any +other stream or, in conjunction with :class:`io.StringIO`, to a string. +The latter can be especially useful, for example, to capture output +from a function that was written to implement a command line interface. +It is recommended only for utility scripts because it affects the +global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger +in :issue:`15805`) The :mod:`contextlib` documentation has also been updated to include a :ref:`discussion <single-use-reusable-and-reentrant-cms>` of the differences between single use, reusable and reentrant context managers. +dbm +--- + +:func:`dbm.open` objects now support the context management protocol. When +used in a :keyword:`with` statement, the ``close`` method of the database +object will be called automatically at the end of the block. (Contributed by +Claudiu Popa and Nick Coghlan in :issue:`19282`.) + + dis --- @@ -639,6 +770,10 @@ to ``distb(tb)``). (Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` and Claudiu Popa in :issue:`17916`) +New function :func:`~dis.stack_effect` computes the effect on the Python stack +of a given opcode and argument, information that is not otherwise available. +(Contributed by Larry Hastings in :issue:`19722`.) + doctest ------- @@ -653,6 +788,9 @@ new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the :mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) +:mod:`doctest` will now find doctests in extension module ``__doc__`` strings. +(Contributed by Zachary Ware in :issue:`3158`.) + email ----- @@ -661,7 +799,8 @@ email override the default policy of the message when generating a string representation of it. This means that ``as_string`` can now be used in more circumstances, instead of having to create and use a :mod:`~email.generator` in -order to pass formatting parameters to its ``flatten`` method. +order to pass formatting parameters to its ``flatten`` method. (Contributed by +R. David Murray in :issue:`18600`.) New method :meth:`~email.message.Message.as_bytes` added to produce a bytes representation of the message in a fashion similar to how ``as_string`` @@ -669,26 +808,30 @@ produces a string representation. It does not accept the *maxheaderlen* argument, but does accept the *unixfrom* and *policy* arguments. The :class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive -result: a bytes object containing the fully formatted message. +result: a bytes object containing the fully formatted message. (Contributed +by R. David Murray in :issue:`18600`.) -(Contributed by R. David Murray in :issue:`18600`.) +The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. +When specified, the associated header will be updated without changing +its location in the list of headers. For backward compatibility, the default +is ``False``. (Contributed by R. David Murray in :issue:`18891`.) -.. _whatsnew_email_contentmanager: - -A pair of new subclasses of :class:`~email.message.Message` have been added, -along with a new sub-module, :mod:`~email.contentmanager`. All documentation -is currently in the new module, which is being added as part of the new -:term:`provisional <provisional package>` email API. These classes provide a -number of new methods that make extracting content from and inserting content -into email messages much easier. See the :mod:`~email.contentmanager` -documentation for details. -These API additions complete the bulk of the work that was planned as part of -the email6 project. The currently provisional API is scheduled to become final -in Python 3.5 (possibly with a few minor additions in the area of error -handling). +.. _whatsnew_email_contentmanager: -(Contributed by R. David Murray in :issue:`18891`.) +A pair of new subclasses of :class:`~email.message.Message` have been added +(:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, +:mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute +:attr:`~email.policy.EmailPolicy.content_manager`. All documentation is +currently in the new module, which is being added as part of email's new +:term:`provisional API`. These classes provide a number of new methods that +make extracting content from and inserting content into email messages much +easier. For details, see the :mod:`~email.contentmanager` documentation and +the :ref:`email-contentmanager-api-examples`. These API additions complete the +bulk of the work that was planned as part of the email6 project. The currently +provisional API is scheduled to become final in Python 3.5 (possibly with a few +minor additions in the area of error handling). (Contributed by R. David +Murray in :issue:`18891`.) filecmp @@ -701,6 +844,11 @@ for example, if the file might have been changed and re-checked in less time than the resolution of a particular filesystem's file modification time field. (Contributed by Mark Levitt in :issue:`18149`.) +New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of +directories that are used as the default value for the *ignore* parameter of +the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in +:issue:`15442`.) + functools --------- @@ -726,6 +874,10 @@ multiple implementations of an operation that allows it to work with :pep:`443` -- Single-dispatch generic functions PEP written and implemented by Łukasz Langa. +:func:`~functools.total_ordering` now supports a return value of +:const:`NotImplemented` from the underlying comparison function. (Contributed +by Katie Miller in :issue:`10042`.) + A pure-python version of the :func:`~functools.partial` function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in @@ -740,11 +892,28 @@ dictionaries containing the collections statistics since interpreter startup. (Contributed by Antoine Pitrou in :issue:`16351`.) +glob +---- + +A new function :func:`~glob.escape` provides a way to escape special characters +in a filename so that they do not become part of the globbing expansion but are +instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) + + hashlib ------- -New :func:`hashlib.pbkdf2_hmac` function. -(Contributed by Christian Heimes in :issue:`18582`) +A new :func:`hashlib.pbkdf2_hmac` function provides +the `PKCS#5 password-based key derivation function 2 +<http://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian +Heimes in :issue:`18582`) + +The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now +a formally supported interface. It has always existed in CPython's +:mod:`hashlib` (although it did not return lower case names for all supported +hashes), but it was not a public interface and so some other Python +implementations have not previously supported it. (Contributed by Jason R. +Coombs in :issue:`18532`.) hmac @@ -756,20 +925,31 @@ argument to the :func:`~hmac.new` function, and the *msg* parameter to both the accepts any type supported by the :mod:`hashlib` module. (Contributed by Jonas Borgström in :issue:`18240`.) +The *digestmod* argument to the :func:`hmac.new` function may now be any hash +digest name recognized by :mod:`hashlib`. In addition, the current behavior in +which the value of *digestmod* defaults to ``MD5`` is deprecated: in a +future version of Python there will be no default value. (Contributed by +Christian Heimes in :issue:`17276`.) + +With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` +attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` +attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. +(Contributed by Christian Heimes in :issue:`18775`.) + html ---- -Added a new :func:`html.unescape` function that converts HTML5 character -references to the corresponding Unicode characters. -(Contributed by Ezio Melotti in :issue:`2927`) +New function :func:`~html.unescape` function converts HTML5 character references to +the corresponding Unicode characters. (Contributed by Ezio Melotti in +:issue:`2927`) -Added a new *convert_charrefs* keyword argument to -:class:`~html.parser.HTMLParser` that, when ``True``, automatically converts -all character references. For backward-compatibility, its value defaults -to ``False``, but it will change to ``True`` in future versions, so you -are invited to set it explicitly and update your code to use this new feature. -(Contributed by Ezio Melotti in :issue:`13633`) +:class:`~html.parser.HTMLParser` accepts a new keyword argument +*convert_charrefs* that, when ``True``, automatically converts all character +references. For backward-compatibility, its value defaults to ``False``, but +it will change to ``True`` in a future version of Python, so you are invited to +set it explicitly and update your code to use this new feature. (Contributed +by Ezio Melotti in :issue:`13633`) The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. (Contributed by Ezio Melotti in :issue:`15114`) @@ -785,6 +965,10 @@ This extended error description will be formatted using the :attr:`~http.server.HTTP.error_message_format` attribute and sent as the body of the error response. (Contributed by Karl Cow in :issue:`12921`.) +The :mod:`http.server` :ref:`command line interface <http-server-cli>` now has +a ``-b/--bind`` option that causes the server to listen on a specific address. +(Contributed by Malte Swart in :issue:`17764`.) + importlib --------- @@ -823,11 +1007,16 @@ by Brett Cannon in :issue:`18058`.) that decodes source from bytes using universal newline processing. This is useful for implementing :meth:`.InspectLoader.get_source` methods. +:class:`importlib.machinery.ExtensionFileLoader` now has a +:meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was +inadvertently omitted in the original implementation. (Contributed by Eric +Snow in :issue:`19152`.) + inspect ------- -The inspect module now offers a basic :ref:`command line interface +The :mod:`inspect` module now offers a basic :ref:`command line interface <inspect-module-cli>` to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in :issue:`18626`) @@ -845,10 +1034,14 @@ metaclasses (Contributed by Ethan Furman in :issue:`18929` and :func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` now use the :func:`~inspect.signature` API. This allows them to -support much broader range of functions, including some builtins and -callables that follow ``__signature__`` protocol. It is still -recommended to update your code to use :func:`~inspect.signature` -directly. (Contributed by Yury Selivanov in :issue:`17481`) +support a much broader range of callables, including those with +``__signature__`` attributes, those with metadata provided by argument +clinic, :func:`functools.partial` objects and more. Note that, unlike +:func:`~inspect.signature`, these functions still ignore ``__wrapped__`` +attributes, and report the already bound first argument for bound methods, +so it is still necessary to update your code to use +:func:`~inspect.signature` directly if those features are desired. +(Contributed by Yury Selivanov in :issue:`17481`) :func:`~inspect.signature` now supports duck types of CPython functions, which adds support for functions compiled with Cython. (Contributed @@ -864,6 +1057,10 @@ has been removed: :mod:`ipaddress` is now considered a stable API, covered by the normal standard library requirements to maintain backwards compatibility. +A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if +an address is globally routeable. (Contributed by Peter Moody in +:issue:`17400`.) + logging ------- @@ -872,6 +1069,24 @@ The :class:`~logging.handlers.TimedRotatingFileHandler` has a new *atTime* parameter that can be used to specify the time of day when rollover should happen. (Contributed by Ronald Oussoren in :issue:`9556`.) +:class:`~logging.handlers.SocketHandler` and +:class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by +setting *port* to ``None``). (Contributed by Vinay Sajip in commit +ce46195b56a9.) + +:func:`~logging.config.fileConfig` now accepts a +:class:`configparser.RawConfigParser` subclass instance for the *fname* +parameter. This facilitates using a configuration file when logging +configuration is just a part of the overall application configuration, or where +the application modifies the configuration before passing it to +:func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in +:issue:`16110`.) + +Logging configuration data received from a socket via the +:func:`logging.config.listen` function can now be validated before being +processed by supplying a verification function as the argument to the new +*verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) + .. _whatsnew-marshal-3: @@ -884,26 +1099,15 @@ interned strings and preserving the interning on deserialization, and extends this "one copy" ability to any object type (including handling recursive references). This reduces both the size of ``.pyc`` files and the amount of memory a module occupies in memory when it is loaded from a ``.pyc`` (or -``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`.) +``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`, +with additional speedups by Antoine Pitrou in :issue:`19219`.) mmap ---- -mmap objects can now be weakref'ed. -(Contributed by Valerie Lambert in :issue:`4885`.) - - -mock ----- - -:mod:`~unittest.mock` objects now inspect their specification signatures when -matching calls, which means an argument can now be matched by either position -or name, instead of only by position. (Contributed by Antoine Pitrou in -:issue:`17015`.) - -:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` -methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) +mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in +:issue:`4885`.) multiprocessing @@ -911,20 +1115,34 @@ multiprocessing .. _whatsnew-multiprocessing-no-fork: -On Unix, two new :ref:`start methods <multiprocessing-start-methods>` -(``spawn`` and ``forkserver``) have been added for starting processes using +On Unix two new :ref:`start methods <multiprocessing-start-methods>`, +(``spawn`` and ``forkserver``, have been added for starting processes using :mod:`multiprocessing`. These make the mixing of processes with threads more robust, and the ``spawn`` method matches the semantics that multiprocessing has -always used on Windows. (Contributed by Richard Oudkerk in :issue:`8713`). - -Also, except when using the old *fork* start method, child processes -will no longer inherit unneeded handles/file descriptors from their parents -(part of :issue:`8713`). +always used on Windows. New function +:func:`~multiprocessing.get_all_start_methods` reports all start methods +available on the platform, :func:`~multiprocessing.get_start_method` reports +the current start method, and :func:`~multiprocessing.set_start_method` sets +the start method. (Contributed by Richard Oudkerk in :issue:`8713`). + +:mod:`multiprocessing` also now has the concept of a ``context``, which +determines how child processes are created. New function +:func:`~multiprocessing.get_context` returns a context that uses a specified +start method. It has the same API as the :mod:`multiprocessing` module itself, +so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other +objects that will operate within that context. This allows a framework and an +application or different parts of the same application to use multiprocessing +without interfering with each other. (Contributed by Richard Oudkerk in +:issue:`18999`.) + +Except when using the old *fork* start method, child processes no longer +inherit unneeded handles/file descriptors from their parents (part of +:issue:`8713`). :mod:`multiprocessing` now relies on :mod:`runpy` (which implements the ``-m`` switch) to initialise ``__main__`` appropriately in child processes when using the ``spawn`` or ``forkserver`` start methods. This resolves some -edge cases where combining multiprocessing, the ``-m`` command line switch +edge cases where combining multiprocessing, the ``-m`` command line switch, and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in :issue:`19946`) @@ -932,6 +1150,11 @@ processes. (Contributed by Nick Coghlan in :issue:`19946`) operator -------- +New function :func:`~operator.length_hint` provides an implementation of the +specification for how the :meth:`~object.__length_hint__` special method should +be used, as part of the :pep:`424` formal specification of this language +feature. (Contributed by Armin Ronacher in :issue:`16148`.) + There is now a pure-python version of the :mod:`operator` module available for reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in :issue:`16694`.) @@ -940,32 +1163,48 @@ Zachary Ware in :issue:`16694`.) os -- -New functions to get and set the :ref:`inheritable flag <fd_inheritance>` of a file -descriptors or a Windows handle: +There are new functions to get and set the :ref:`inheritable flag +<fd_inheritance>` of a file descriptor (:func:`os.get_inheritable`, +:func:`os.set_inheritable`) or a Windows handle +(:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). -* :func:`os.get_inheritable`, :func:`os.set_inheritable` -* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` +New function :func:`~os.cpu_count` reports the number of CPUs available on the +platform on which Python is running (or ``None`` if the count can't be +determined). The :func:`multiprocessing.cpu_count` function is now implemented +in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, +Victor Stinner, and Charles-François Natali in :issue:`17914`.) + +:func:`os.path.samestat` is now available on the Windows platform (and the +:func:`os.path.samefile` implementation is now shared between Unix and +Windows). (Contributed by Brian Curtin in :issue:`11939`.) -The :mod:`os` module now provides a :func:`~os.cpu_count` function, analogous to -the :func:`multiprocessing.cpu_count` function (which is now implemented in -terms of the new :mod:`os` function). (Contributed by Trent Nelson, Yogesh -Chaudhari, Victor Stinner, and Charles-François Natali in :issue:`17914`.) +:func:`os.path.ismount` now recognizes volumes mounted below a drive +root on Windows. (Contributed by Tim Golden in :issue:`9035`.) + +:func:`os.open` supports two new flags on platforms that provide them, +:data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` +(unnamed temporary file; as of 3.4.0 release available only on Linux systems +with a kernel version of 3.11 or newer that have uapi headers). (Contributed +by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) pdb --- -The ``print`` command has been removed from :mod:`pdb`, restoring access to the -``print`` function. - -Rationale: Python2's ``pdb`` did not have a ``print`` command; instead, -entering ``print`` executed the ``print`` statement. In Python3 ``print`` was -mistakenly made an alias for the pdb :pdbcmd:`p` command. ``p``, however, -prints the ``repr`` of its argument, not the ``str`` like the Python2 ``print`` -command did. Worse, the Python3 ``pdb print`` command shadowed the Python3 -``print`` function, making it inaccessible at the ``pdb`` prompt. +:mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and +``yield from`` in a more useful fashion. This is especially helpful when +debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and +Xavier de Gaye in :issue:`16596`.) -(Contributed by Connor Osborn in :issue:`18764`.) +The ``print`` command has been removed from :mod:`pdb`, restoring access to the +Python :func:`print` function from the pdb command line. Python2's ``pdb`` did +not have a ``print`` command; instead, entering ``print`` executed the +``print`` statement. In Python3 ``print`` was mistakenly made an alias for the +pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument, +not the ``str`` like the Python2 ``print`` command did. Worse, the Python3 +``pdb print`` command shadowed the Python3 ``print`` function, making it +inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in +:issue:`18764`.) .. _whatsnew-protocol-4: @@ -976,7 +1215,7 @@ pickle :mod:`pickle` now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large -strings and containers, or classes whose :meth:`__new__` method takes +strings and containers, and classes whose :meth:`__new__` method takes keyword-only arguments. It also provides some efficiency improvements. .. seealso:: @@ -988,9 +1227,13 @@ keyword-only arguments. It also provides some efficiency improvements. plistlib -------- -:mod:`plistlib` now supports binary plist files, and offers the common -``load``/``loads``/``dump``/``dumps`` API pattern for serialization formats -(Contributed by Ronald Oussoren and others in :issue:`14455`). +:mod:`plistlib` now has an API that is similar to the standard pattern for +stdlib serialization protocols, with new :func:`~plistlib.load`, +:func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` +functions. (The older API is now deprecated.) In addition to the already +supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports +the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald +Oussoren and others in :issue:`14455`). poplib @@ -1006,11 +1249,15 @@ Catucci in :issue:`4473`.) pprint ------ -The :mod:`pprint` module now supports *compact* mode for formatting long -sequences (:issue:`19132`). +The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its +:func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new +option, *compact*, that controls how the output is formatted. Currently +setting *compact* to ``True`` means that sequences will be printed with as many +sequence elements as will fit within *width* on each (indented) line. +(Contributed by Serhiy Storchaka in :issue:`19132`.) Long strings are now wrapped using Python's normal line continuation -syntax (Contributed by Antoine Pitrou in :issue:`17150`.) +syntax. (Contributed by Antoine Pitrou in :issue:`17150`). pty @@ -1023,30 +1270,61 @@ the child process, instead of ``None``. (Contributed by Gregory P. Smith.) pydoc ----- -While significant changes have not been made to :mod:`pydoc` directly, +The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` +introspection API, allowing it to provide signature information for a wider +variety of callable objects. This change also means that ``__wrapped__`` +attributes are now taken into account when displaying help information +(Contributed by Larry Hastings in :issue:`19674`) + +The :mod:`pydoc` module no longer displays the ``self`` parameter for +already bound methods. Instead, it aims to always display the exact current +signature of the supplied callable (Contributed by Larry Hastings in +:issue:`20710`) + +In addition to the changes that have been made to :mod:`pydoc` directly, its handling of custom ``__dir__`` methods and various descriptor -behaviours has been improved substantially by the underlying changes in +behaviours has also been improved substantially by the underlying changes in the :mod:`inspect` module. +As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also +affect the behaviour of :func:`help`. + re -- -Added :func:`re.fullmatch` function and :meth:`regex.fullmatch` method, -which anchor the pattern at both ends of the string to match. -(Contributed by Matthew Barnett in :issue:`16203`.) +New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor +the pattern at both ends of the string to match. This provides a way to be +explicit about the goal of the match, which avoids a class of subtle bugs where +``$`` characters get lost during code changes or the addition of alternatives +to an existing regular expression. (Contributed by Matthew Barnett in +:issue:`16203`.) The repr of :ref:`regex objects <re-objects>` now includes the pattern and the flags; the repr of :ref:`match objects <match-objects>` now -includes the start, end, and the part of the string that matched. -(Contributed by Serhiy Storchaka in :issue:`13592` and :issue:`17087`.) +includes the start, end, and the part of the string that matched. (Contributed +by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and +:issue:`17087`.) resource -------- -New :func:`resource.prlimit` function and Linux specific constants. -(Contributed by Christian Heimes in :issue:`16595` and :issue:`19324`.) +New :func:`~resource.prlimit` function, available on Linux platforms with a +kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the +ability to query or set the resource limits for processes other than the one +making the call. (Contributed by Christian Heimes in :issue:`16595`.) + +On Linux kernel version 2.6.36 or later, there are there are also some new +Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, +:attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, +:attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. +(Contributed by Christian Heimes in :issue:`19324`.) + +On FreeBSD version 9 and later, there some new FreeBSD specific constants: +:attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and +:attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in +:issue:`19343`.) select @@ -1057,6 +1335,11 @@ When used in a :keyword:`with` statement, the :meth:`~select.epoll.close` method will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in :issue:`16488`.) +:class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and +:meth:`~select.devpoll.close` methods, as well as a new attribute +:attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in +:issue:`18794`.) + shelve ------ @@ -1066,11 +1349,21 @@ and will be automatically closed at the end of the :keyword:`with` block. (Contributed by Filip Gruszczyński in :issue:`13896`.) +shutil +------ + +:func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, +:exc:`~shutil.SameFileError`, when the source and destination are the same +file, which allows an application to take appropriate action on this specific +error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in +:issue:`1492704`.) + + smtpd ----- The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now -accept a *map* keyword argument, which if specified is passed in to +accept a *map* keyword argument which, if specified, is passed in to :class:`asynchat.async_chat` as its *map* argument. This allows an application to avoid affecting the global socket map. (Contributed by Vinay Sajip in :issue:`11959`.) @@ -1082,7 +1375,7 @@ smtplib :exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. -(:issue:`2118`). +(Contributed by Ned Jackson Lovely in :issue:`2118`). socket @@ -1101,14 +1394,17 @@ during debugging, instead of integer "magic numbers". The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. +:func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported +on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) + sqlite3 ------- -A new boolean parameter, *uri*, to the :func:`~sqlite3.connect` function can -be used to indicate that the *database* parameter is a ``uri`` (see -the `SQLite URI documentation <http://www.sqlite.org/uri.html>`_). -(Contributed by poq in :issue:`13773`.) +A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be +used to indicate that the *database* parameter is a ``uri`` (see the `SQLite +URI documentation <http://www.sqlite.org/uri.html>`_). (Contributed by poq in +:issue:`13773`.) ssl @@ -1121,6 +1417,25 @@ TLSv1.2 support) have been added; support for these protocols is only available Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and Antoine Pitrou in :issue:`16692`) +.. _whatsnew34-sslcontext: + +New function :func:`~ssl.create_default_context` provides a standard way to +obtain an :class:`~ssl.SSLContext` whose settings are intended to be a +reasonable balance between compatibility and security. These settings are +more stringent than the defaults provided by the :class:`~ssl.SSLContext` +constructor, and may be adjusted in the future, without prior deprecation, if +best-practice security requirements change. The new recommended best +practice for using stdlib libraries that support SSL is to use +:func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` +object, modify it if needed, and then pass it as the *context* argument +of the appropriate stdlib API. (Contributed by Christian Heimes +in :issue:`19689`.) + +:class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` +accepts a new optional argument *cadata*, which can be used to provide PEM or +DER encoded certificates directly via strings or bytes, respectively. +(Contributed by Christian Heimes in :issue:`18138`.) + New function :func:`~ssl.get_default_verify_paths` returns a named tuple of the paths and environment variables that the :meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set @@ -1133,36 +1448,62 @@ in :issue:`18143`.) ``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists (``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that returns a list of the loaded ``CA`` certificates. (Contributed by Christian Heimes in -and :issue:`18147`.) +:issue:`18147`.) + +If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has an new +attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the +certificate verification process by setting it to some combination of the new +constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, +:data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. +OpenSSL does not do any CRL verification by default. (Contributed by +Christien Heimes in :issue:`8813`.) + +New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` +loads a set of dfault "certificate authority" (CA) certificates from default +locations, which vary according to the platform. It can be used to load both +TLS web server authentication certificates +(``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a +server, and certificates for a server to use in verifying client certificates +(``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian +Heimes in :issue:`19292`.) + +.. _whatsnew34-win-cert-store: Two new windows-only functions, :func:`~ssl.enum_certificates` and :func:`~ssl.enum_crls` provide the ability to retrieve certificates, certificate information, and CRLs from the Windows cert store. (Contributed by Christian Heimes in :issue:`17134`.) -Support for server-side SNI using the new +.. _whatsnew34-sni: + +Support for server-side SNI (Server Name Indication) using the new :meth:`ssl.SSLContext.set_servername_callback` method. (Contributed by Daniel Black in :issue:`8109`.) +The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional +``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and +``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) + stat ---- The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C implementation is required as most of the values aren't standardized and -platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) +are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) -The module supports new file types: door, event port and whiteout. +The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, +:attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by +Christian Hiemes in :issue:`11016`.) struct ------ -:mod:`struct` now supports the streamed unpacking of a buffer containing -repeated instances of a given format of data. Both a module level -:mod:`~struct.iter_unpack` function and a :meth:`struct.Struct.iter_unpack` -method on compiled formats have been added. (Contributed by Antoine Pitrou in -:issue:`17804`.) +New function :mod:`~struct.iter_unpack` and a new +:meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed +unpacking of a buffer containing repeated instances of a given format of data. +(Contributed by Antoine Pitrou in :issue:`17804`.) subprocess @@ -1172,6 +1513,10 @@ subprocess be used to provide the contents of ``stdin`` for the command that is run. (Contributed by Zack Weinberg in :issue:`16624`.) +:func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now +work on Windows. This change was actually inadvertently made in 3.3.4. +(Contributed by Tim Golden in :issue:`10197`.) + sunau ----- @@ -1179,26 +1524,93 @@ sunau The :meth:`~sunau.getparams` method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) -:meth:`sunau.open` now supports the context manager protocol (:issue:`18878`). +:meth:`sunau.open` now supports the context manager protocol: when used in a +:keyword:`with` block, the ``close`` method of the returned object will be +called automatically at the end of the block. (Contributed by Serhiy Storchaka +in :issue:`18878`.) + +:meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding +support for writing 24 sample using the module. (Contributed by +Serhiy Storchaka in :issue:`19261`.) + +The :meth:`~sunau.AU_write.writeframesraw` and +:meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) sys --- New function :func:`sys.getallocatedblocks` returns the current number of -blocks allocated by the interpreter (in CPython with the default +blocks allocated by the interpreter. (In CPython with the default ``--with-pymalloc`` setting, this is allocations made through the -:c:func:`PyObject_Malloc` API). This can be useful for tracking memory leaks, +:c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in :issue:`13390`.) +When the Python interpreter starts in :ref:`interactive mode +<tut-interactive>`, it checks for an :data:`~sys.__interactivehook__` attribute +on the :mod:`sys` module. If the attribute exists, its value is called with no +arguments just before interactive mode is started. The check is made after the +:envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` +module :ref:`sets it <rlcompleter-config>` to a function that enables tab +completion and history saving (in :file:`~/.python-history`) if the platform +supports :mod:`readline`. If you do not want this (new) behavior, you can +override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or +:mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it +to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in +:issue:`5845`.) + + +tarfile +------- + +The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when +called as a script directly or via ``-m``. This can be used to create and +extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) + + +textwrap +-------- + +The :class:`~textwrap.TextWrapper` class has two new attributes/constructor +arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of +lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a +string that will appear at the end of the output if it has been truncated +because of *max_lines*. Building on these capabilities, a new convenience +function :func:`~textwrap.shorten` collapses all of the whitespace in the input +to single spaces and produces a single line of a given *width* that ends with +the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and +Serhiy Storchaka in :issue:`18585` and :issue:`18725`.) + + +threading +--------- + +The :class:`~threading.Thread` object representing the main thread can be +obtained from the new :func:`~threading.main_thread` function. In normal +conditions this will be the thread from which the Python interpreter was +started. (Contributed by Andrew Svetlov in :issue:`18882`.) + traceback --------- A new :func:`traceback.clear_frames` function takes a traceback object and clears the local variables in all of the frames it references, -reducing the amount of memory consumed (:issue:`1565525`). +reducing the amount of memory consumed. (Contributed by Andrew Kuchling in +:issue:`1565525`). + + +types +----- + +A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define +an attribute that acts normally when looked up through an instance object, but +which is routed to the *class* ``__getattr__`` when looked up through the +class. This allows one to have properties active on a class, and have virtual +attributes on the class with the same name (see :mod:`Enum` for an example). +(Contributed by Ethan Furman in :issue:`19030`.) urllib @@ -1208,6 +1620,29 @@ urllib :class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenböck in :issue:`16423`.) +The http method that will be used by a :class:`~urllib.request.Request` class +can now be specified by setting a :class:`~urllib.request.Request.method` +class attribute on the subclass. (Contributed by Jason R Coombs in +:issue:`18978`.) + +:class:`~urllib.request.Request` objects are now reusable: if the +:attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` +attributes are modified, all relevant internal properties are updated. This +means, for example, that it is now possible to use the same +:class:`~urllib.request.Request` object in more than one +:meth:`.OpenerDirector.open` call with different *data* arguments, or to +modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it +from scratch. There is also a new +:meth:`~urllib.request.Request.remove_header` method that can be used to remove +headers from a :class:`~urllib.request.Request`. (Contributed by Alexey +Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht +and Senthil Kumaran in :issue:`17272`.) + +:class:`~urllib.error.HTTPError` objects now have a +:attr:`~urllib.error.HTTPError.headers` attribute that provides access to the +HTTP response headers associated with the error. (Contributed by +Berker Peksag in :issue:`15701`.) + unittest -------- @@ -1223,7 +1658,7 @@ which will run even if one or more of them fail. For example:: class NumbersTest(unittest.TestCase): def test_even(self): for i in range(6): - with self.subTest(i=1): + with self.subTest(i=i): self.assertEqual(i % 2, 0) will result in six subtests, each identified in the unittest verbose output @@ -1243,6 +1678,34 @@ error. (Contributed by Zach Ware in :issue:`16935`.) consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in :issue:`16709`.) +:class:`~unittest.TestSuite` now drops references to tests as soon as the test +has been run, if the test is successful. On Python interpreters that do +garbage collection, this allows the tests to be garbage collected if nothing +else is holding a reference to the test. It is possible to override this +behavior by creating a :class:`~unittest.TestSuite` subclass that defines a +custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt +McClure, and Andrew Svetlov in :issue:`11798`.) + +A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`, +will ensure that a given block of code emits a log message using the +:mod:`logging` module. By default the message can come from any logger and +have a priority of ``INFO`` or higher, but both the logger name and an +alternative minimum logging level may be specified. The object returned by the +context manager can be queried for the :class:`~logging.LogRecord`\ s and/or +formatted messages that were logged. (Contributed by Antoine Pitrou in +:issue:`18937`.) + +Test discovery now works with namespace packages (Contributed by Claudiu Popa +in :issue:`17457`.) + +:mod:`unittest.mock` objects now inspect their specification signatures when +matching calls, which means an argument can now be matched by either position +or name, instead of only by position. (Contributed by Antoine Pitrou in +:issue:`17015`.) + +:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` +methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) + venv ---- @@ -1250,6 +1713,12 @@ venv :mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` shells (Contributed by Andrew Svetlov in :issue:`15417`.) +:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function +take a new keyword argument *with_pip*, which defaults to ``False``, that +controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is +installed in the virtual environment. (Contributed by Nick Coghlan in +:issue:`19552` as part of the :pep:`453` implementation.) + wave ---- @@ -1260,6 +1729,14 @@ plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.) :meth:`wave.open` now supports the context manager protocol. (Contributed by Claudiu Popa in :issue:`17616`.) +:mod:`wave` can now :ref:`write output to unseekable files +<wave-write-objects>`. (Contributed by David Jones, Guilherme Polo, and Serhiy +Storchaka in :issue:`5202`.) + +The :meth:`~wave.Wave_write.writeframesraw` and +:meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like +object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) + weakref ------- @@ -1295,13 +1772,19 @@ abbreviated (``<tag />``) or expanded (``<tag></tag>``) form. (Contributed by Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) -zipfile.PyZipfile ------------------ +zipfile +------- -Add a filter function to ignore some packages (tests for instance), -:meth:`~zipfile.PyZipFile.writepy`. +The :meth:`~zipfile.PyZipFile.writepy` method of the +:class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be +used to control which directories and files are added to the archive. For +example, this could be used to exclude test files from the archive. (Contributed by Christian Tismer in :issue:`19274`.) +The *allowZip64* parameter to :class:`~zipfile.ZipFile` and +:class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by +William Mallard in :issue:`17201`.) + CPython Implementation Changes @@ -1375,7 +1858,12 @@ accurate signatures for builtins and standard library extension modules implemented in C. Some standard library extension modules have been converted to use Argument -Clinic in Python 3.4, and :mod:`inspect` has been updated accordingly. +Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated +accordingly. + +It is expected that signature metadata for programmatic introspection will +be added to additional callables implemented in C as part of Python 3.4 +maintenance releases. .. note:: The Argument Clinic PEP is not fully up to date with the state of the @@ -1405,15 +1893,53 @@ Other Build and C API Changes marked as accepting ``const char *`` rather than ``char *`` (Contributed by Serhiy Storchaka in :issue:`1772673`). -* New shell version of ``python-config``; can be used even when a python +* A new shell version of ``python-config`` can be used even when a python interpreter is not available (for example, in cross compilation scenarios). +* :c:func:`PyUnicode_FromFormat` now supports width and precision + specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``. + (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.) +* New function :c:func:`PyStructSequence_InitType2` supplements the + existing :c:func:`PyStructSequence_InitType` function. The difference + is that it returns ``0`` on success and ``-1`` on failure. -Other Improvements -================== +* The CPython source can now be compiled using the address sanity checking + features of recent versions of GCC and clang: the false alarms in the small + object allocator have been silenced. (Contributed by Dhiru Kholia in + :issue:`18596`.) -* Tab-completion is now enabled by default in the interactive interpreter. +* The Windows build now uses `Address Space Layout Randomization + <http://en.wikipedia.org/wiki/ASLR>`_ and `Data Execution Prevention + <http://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by + Christian Heimes in :issue:`16632`.) + +* New function :c:func:`PyObject_LengthHint` is the C API equivalent + of :func:`operator.length_hint`. (Contributed by Armin Ronacher in + :issue:`16148`.) + + +.. _other-improvements-3.4: + +Other Improvements +------------------ + +.. _whatsnew-isolated-mode: + +* The :ref:`python <using-on-cmdline>` command has a new :ref:`option + <using-on-misc-options>`, ``-I``, which causes it to run in "isolated mode", + which means that :data:`sys.path` contains neither the script's directory nor + the user's ``site-packages`` directory, and all :envvar:`PYTHON*` environment + variables are ignored (it implies both ``-s`` and ``-E``). Other + restrictions may also be applied in the future, with the goal being to + isolate the execution of a script from the user's environment. This is + appropriate, for example, when Python is used to run a system script. On + most POSIX systems it can and should be used in the ``#!`` line of system + scripts. (Contributed by Christian Heimes in :issue:`16499`.) + +* Tab-completion is now enabled by default in the interactive interpreter + on systems that support :mod:`readline`. History is also enabled by default, + and is written to (and read from) the file :file:`~/.python-history`. (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.) * Invoking the Python interpreter with ``--version`` now outputs the version to @@ -1443,10 +1969,40 @@ Other Improvements values for its constants from the C header files, instead of having the values hard-coded in the python module as was previously the case. +* Loading multiple python modules from a single OS module (``.so``, ``.dll``) + now works correctly (previously it silently returned the first python + module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.) + +* A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the + loading of free variables in class bodies that could be triggered by certain + uses of :ref:`__prepare__ <prepare>`. (Contributed by Benjamin Peterson in + :issue:`17853`.) + +* A number of MemoryError-related crashes were identified and fixed by Victor + Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, + :issue:`18520`). + +* The :ref:`pyvenv <scripts-pyvenv>` command now accepts a ``--copies`` option + to use copies rather than symlinks even on systems where symlinks are the + default. (Contributed by Vinay Sajip in :issue:`18807`.) + +* The :ref:`pyvenv <scripts-pyvenv>` command also accepts a ``--without-pip`` + option to suppress the otherwise-automatic bootstrapping of pip into + the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` + as part of the :pep:`453` implementation.) + +* The encoding name is now optional in the value set for the + :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to + set just the error handler, without changing the default encoding. + (Contributed by Serhiy Storchaka in :issue:`18818`.) + +* The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now + support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and + Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) Significant Optimizations -========================= +------------------------- * The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka in :issue:`14625`.) @@ -1489,6 +2045,16 @@ Significant Optimizations * :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in :issue:`18020`.) +* On Windows, the native ``VirtualAlloc`` is now used instead of the CRT + ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory + savings. + +* :func:`os.urandom` now uses a lazily-opened persistent file descriptor + so as to avoid using many file descriptors when run in parallel from + multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) + + +.. _deprecated-3.4: Deprecated ========== @@ -1500,8 +2066,8 @@ when the interpreter is run with deprecation warnings enabled (for example, by using ``-Wd``). -Deprecated Python Modules, Functions and Methods ------------------------------------------------- +Deprecations in the Python API +------------------------------ * As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importilb` methods and functions are deprecated: :meth:`importlib.find_loader` is @@ -1534,18 +2100,39 @@ Deprecated Python Modules, Functions and Methods * The :mod:`formatter` module is pending deprecation and is slated for removal in Python 3.6. -* MD5 as default digestmod for :mod:`hmac` is deprecated. Python 3.6 will - require an explicit digest name or constructor as *digestmod* argument. +* ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is + deprecated. Python 3.6 will require an explicit digest name or constructor as + *digestmod* argument. * The internal ``Netrc`` class in the :mod:`ftplib` module has been documented as deprecated in its docstring for quite some time. It now emits a :exc:`DeprecationWarning` and will be removed completely in Python 3.5. +* The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should + not have been exposed and is hopefully not in use; it is deprecated and + will mostly likely be removed in Python 3.5. + +* The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. + +* The :mod:`plistlib` :func:`~plistlib.readPlist`, + :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and + :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the + corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, + :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` + is deprecated in favor of just using the :class:`bytes` constructor. + +* The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by + ``EXT_SUFFIX``. -Deprecated Functions and Types in the C API -------------------------------------------- +* The ``U`` mode accepted by various ``open`` functions is deprecated. + In Python3 it does not do anything useful, and should be replaced by + appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* + argument. -XXX: None so far +* The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has + been deprecated, as has the *html* argument of + :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the + latter, all arguments to ``XMLParser`` should be passed by keyword. Deprecated Features @@ -1571,6 +2158,7 @@ and build tools: * OS/2 (:issue:`16135`). * Windows 2000 (changeset e52df05b496a). +* Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`). * VMS (:issue:`16136`). @@ -1588,7 +2176,8 @@ removed: ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). * The ``PyThreadState.tick_counter`` field has been removed; its value has - been meaningless since Python 3.2, when the "new GIL" was introduced. + been meaningless since Python 3.2, when the "new GIL" was introduced + (:issue:`19199`). * ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. (Contributed by Taras Lyapun in :issue:`15641`.) @@ -1608,6 +2197,15 @@ removed: * :class:`inspect.Signature`: positional-only parameters are now required to have a valid name. +* :meth:`object.__format__` no longer accepts non-empty format strings, it now + raises a :exc:`TypeError` instead. Using a non-empty string has been + deprecated since Python 3.2. This change has been made to prevent a + situation where previously working (but incorrect) code would start failing + if an object gained a __format__ method, which means that your code may now + raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects + that do not have a __format__ method that handles it. See :issue:`7994` for + background. + * :meth:`difflib.SequenceMatcher.isbjunk` and :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have now been removed: use ``x in sm.bjunk`` and @@ -1626,6 +2224,9 @@ Code Cleanups ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called on badly broken OSX systems (see :issue:`18393`). +* The hardcoded copies of certain :mod:`stat` constants that were included in + the :mod:`tarfile` module namespace have been removed. + Porting to Python 3.4 @@ -1634,6 +2235,27 @@ Porting to Python 3.4 This section lists previously described changes and other bugfixes that may require changes to your code. + +Changes in 'python' Command Behavior +------------------------------------ + +* In a posix shell, setting the :envvar:`PATH` environment variable to + an empty value is equivalent to not setting it at all. However, setting + :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it + at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to + setting it to ``.``, which leads to confusion when reasoning by analogy to + how :envvar:`PATH` works. The behavior now conforms to the posix convention + for :envvar:`PATH`. + +* The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the + CPython interpreter is now off by default. It can be re-enabled using the + ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.) + +* The python command and most stdlib scripts (as well as :mod:`argparse`) now + output ``--version`` information to ``stdout`` instead of ``stderr`` (for + issue list see :ref:`other-improvements-3.4` above). + + Changes in the Python API ------------------------- @@ -1646,7 +2268,7 @@ Changes in the Python API * The module type now initializes the :attr:`__package__` and :attr:`__loader__` attributes to ``None`` by default. To determine if these attributes were set in a backwards-compatible fashion, use e.g. - ``getattr(module, '__loader__', None) is not None``. + ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.) * :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and ``__package__`` unconditionally to properly support reloading. If this is not @@ -1655,10 +2277,23 @@ Changes in the Python API * Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. + Note that this restores a pre-3.3 behavior in that it means a module is + re-found when re-loaded (:issue:`19413`). * Frozen packages no longer set ``__path__`` to a list containing the package - name but an empty list instead. Determing if a module is a package should be - done using ``hasattr(module, '__path__')``. + name, they now set it to an empty list. The previous behavior could cause + the import system to do the wrong thing on submodule imports if there was + also a directory with the same name as the frozen package. The correct way + to determine if a module is a package or not is to use``hasattr(module, + '__path__')`` (:issue:`18065`). + +* Frozen modules no longer define a ``__file__`` attribute. It's semantically + incorrect for frozen modules to set the attribute as they are not loaded from + any explicit location. If you must know that a module comes from frozen code + then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, + check if the loader is a subclass of + :class:`importlib.machinery.FrozenImporter`, + or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. * :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path it would write to is a symlink or a non-regular file. This is to act as a @@ -1684,14 +2319,24 @@ Changes in the Python API :func:`inspect.unwrap` to access the first function in the chain that has no ``__wrapped__`` attribute. +* :func:`inspect.getfullargspec` has been reimplemented on top of + :func:`inspect.signature` and hence handles a much wider variety of callable + objects than it did in the past. It is expected that additional builtin and + extension module callables will gain signature metadata over the course of + the Python 3.4 series. Code that assumes that + :func:`inspect.getfullargspec` will fail on non-Python callables may need + to be adjusted accordingly. + * :class:`importlib.machinery.PathFinder` now passes on the current working directory to objects in :data:`sys.path_hooks` for the empty string. This results in :data:`sys.path_importer_cache` never containing ``''``, thus iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` will not find all keys. A module's ``__file__`` when imported in the current working directory will also now have an absolute path, including when using - ``-m`` with the interpreter (this does not influence when the path to a file - is specified on the command-line). + ``-m`` with the interpreter (except for ``__main__.__file__`` when a script + has been executed directly using a relative path) (Contributed by Brett + Cannon in :issue:`18416`). is specified on the command-line) + (:issue:`18416`). * The removal of the *strict* argument to :class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` changes the meaning of the @@ -1729,10 +2374,106 @@ Changes in the Python API informative :exc:`ValueError` rather than the previous more mysterious :exc:`AttributeError` (:issue:`9177`). -* Parameter names in ``__annotations__`` dict are now mangled properly, +* :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge + values. As a consequence of this fix, :meth:`slice.indices` now raises a + :exc:`ValueError` if given a negative length; previously it returned nonsense + values (:issue:`14794`). + +* The :class:`complex` constructor, unlike the :mod:`cmath` functions, was + incorrectly accepting :class:`float` values if an object's ``__complex__`` + special method returned one. This now raises a :exc:`TypeError`. + (:issue:`16290`.) + +* The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float` + values for the *base* parameter. It is unlikely anyone was doing this, but + if so, it will now raise a :exc:`TypeError` (:issue:`16772`). + +* Defaults for keyword-only arguments are now evaluated *after* defaults for + regular keyword arguments, instead of before. Hopefully no one wrote any + code that depends on the previous buggy behavior (:issue:`16967`). + +* Stale thread states are now cleared after :func:`~os.fork`. This may cause + some system resources to be released that previously were incorrectly kept + perpetually alive (for example, database connections kept in thread-local + storage). (:issue:`17094`.) + +* Parameter names in ``__annotations__`` dicts are now mangled properly, similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in :issue:`20625`). +* :attr:`hashlib.hash.name` now always returns the identifier in lower case. + Previously some builtin hashes had uppercase names, but now that it is a + formal public interface the naming has been made consistent (:issue:`18532`). + +* Because :mod:`unittest.TestSuite` now drops references to tests after they + are run, test harnesses that re-use a :class:`~unittest.TestSuite` to re-run + a set of tests may fail. Test suites should not be re-used in this fashion + since it means state is retained between test runs, breaking the test + isolation that :mod:`unittest` is designed to provide. However, if the lack + of isolation is considered acceptable, the old behavior can be restored by + creating a :mod:`~unittest.TestSuite` subclass that defines a + ``_removeTestAtIndex`` method that does nothing (see + :meth:`.TestSuite.__iter__`) (:issue:`11798`). + +* :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are + certain invalid command forms that used to work that are no longer allowed; + in theory this should not cause backward compatibility issues since the + disallowed command forms didn't make any sense and are unlikely to be in use. + +* The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and + the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of + ``match`` objects now always return a *bytes* object when the string + to be matched is a :term:`bytes-like object`. Previously the return type + matched the input type, so if your code was depending on the return value + being, say, a ``bytearray``, you will need to change your code. + +* :mod:`audioop` functions now raise an error immediately if passed string + input, instead of failing randomly later on (:issue:`16685`). + +* The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` + currently defaults to ``False`` for backward compatibility, but will + eventually be changed to default to ``True``. It is recommended that you add + this keyword, with the appropriate value, to any + :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). + +* Since the *digestmod* argument to the :func:`hmac.new` function will in the + future have no default, all calls to :func:`hmac.new` should be changed to + explicitly specify a *digestmod* (:issue:`17276`). + +* Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking + ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` + is deprecated. This key should be replaced by ``EXT_SUFFIX`` or + ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). + +* Any calls to ``open`` functions that specify ``U`` should be modified. + ``U`` is ineffective in Python3 and will eventually raise an error if used. + Depending on the function, the equivalent of its old Python2 behavior can be + achieved using either a *newline* argument, or if necessary by wrapping the + stream in :mod:`~io.TextIOWrapper` to use its *newline* argument + (:issue:`15204`). + +* If you use :ref:`pyvenv <scripts-pyvenv>` in a script and desire that pip + *not* be installed, you must add ``--without-pip`` to your command + invocation. + +* The default behavior of :func:`json.dump` and :func:`json.dumps` when + an indent is specified has changed: it no longer produces trailing + spaces after the item separating commas at the ends of lines. This + will matter only if you have tests that are doing white-space-sensitive + comparisons of such output (:issue:`16333`). + +* :mod:`doctest` now looks for doctests in extension module ``__doc__`` + strings, so if your doctest test discovery includes extension modules that + have things that look like doctests in them you may see test failures you've + never seen before when running your tests (:issue:`3158`). + +* The :mod:`collections.abc` module has been slightly refactored as + part of the Python startup improvements. As a consequence of this, it is no + longer the case that importing :mod:`collections` automatically imports + :mod:`collections.abc`. If your program depended on the (undocumented) + implicit import, you will need to add an explicit ``import collections.abc`` + (:issue:`20784`). + Changes in the C API -------------------- @@ -1756,7 +2497,8 @@ Changes in the C API * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must now be a string allocated by :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a - string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. + string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` + (:issue:`16742`) * :c:func:`PyThread_set_key_value` now always set the value. In Python 3.3, the function did nothing if the key already exists (if the current |